Data Streaming Engine

StreamNative Cloud offers two data streaming engines to run your StreamNative clusters: Classic Engine and Ursa Engine.

Classic Engine

Classic Engine refers to the original ZooKeeper & BookKeeper based Apache Pulsar engine that StreamNative Cloud uses.

It uses ZooKeeper for metadata storage and cluster membership, and BookKeeper for low-latency data persistence.

The Classic Engine is the default engine for StreamNative Cloud, which is available across all deployment options, including Serverless, Dedicated, and BYOC clusters.

Besides Pulsar protocol, the Classic Engine also supports Kafka and MQTT protocols.

The Classic Engine only supports low-latency BookKeeper based storage, which is suitable for all latency-sensitive workloads. Lakehouse storage can be enabled for your Classic Engine clusters to support advanced data processing and analytics workloads.

Ursa Engine

Ursa Engine is the next-generation data streaming engine that StreamNative Cloud offers.

It is built on top of Apache Pulsar but evolves many of the core design and implementation details to support a more flexible and scalable architecture.

The Ursa Engine uses Oxia for metadata storage and uses Object Storage (such as S3, GCS, and Azure Blob Storage) as the underlying data persistence layer, making BookKeeper storage optional only for low-latency workloads.

Key pillars of the Ursa Engine include:

• Fully Kafka API-compatible: The Ursa Engine is fully compatible with the Kafka API, making it easier to migrate existing Kafka-based applications to Pulsar.
• Lakehouse storage for long-term durability and open standards-based storage.
• Oxia as a scalable and durable metadata storage.
• Support for both Latency-Optimized and Cost-Optimized storage options.

The Ursa Engine has two storage options:

• Latency-Optimized storage: This storage option is based on Apache BookKeeper and provides low-latency data persistence. It is suitable for latency-sensitive workloads that require high-throughput and low-latency messaging and streaming.
• Cost-Optimized storage: This storage option is based on Object Storage (such as S3, GCS, and Azure Blob Storage) and provides a more cost-effective storage solution for workloads that can tolerate sub-second latencies in exchange for lower total cost of ownership.

What is included in the Ursa Engine Public Preview?

With the Public Preview, users can access core Ursa Engine Features such as Oxia-based metadata management and S3-based WAL, targeting cost-sensitive, latency-relaxed workloads. However, there are certain limitations of this Public Preview as shown below:

• Only Kafka protocol is enabled during Public Preview. Pulsar & MQTT protocols are disabled.
• Transactions and topic compaction are not yet supported.
• Only S3-based Cost-Optimized storage is available, making it ideal for latency-relaxed use cases.
• For ultra-low-latency applications, we recommend continuing with the Classic Engine until the Ursa Engine reaches full GA.

These limitations allow us to focus on gathering feedback and optimizing the experience for broader production use.

Ursa Stream Storage

At the heart of the Ursa Engine is the concept of Ursa Stream Storage. It is a headless, multi-modal data storage layer built on lakehouse formats.

At the heart of Ursa Stream Storage is a WAL (Write-Ahead Log) implementation based on S3. This design writes records directly to object storage services like S3, bypassing BookKeeper and eliminating the need for replication between brokers. As a result, Ursa Engine-powered clusters replace expensive inter-AZ replication with cost-efficient, direct-to-object-storage writes. This trade-off introduces a slight increase in latency (from 200ms to 500ms) but results in significantly lower network costs—on average 10x cheaper.

In the S3-based WAL implementation, brokers create batches of produce requests and write them directly to object storage before acknowledging the client. These brokers are stateless and leaderless, meaning any broker can handle produce or fetch requests for any partition. For improved batch and fetch performance, however, specific partitions may still be routed to designated brokers. This architecture eliminates inter-AZ replication traffic between brokers, while maintaining—and even improving—the durability and availability that customers expect from StreamNative. As with any engineering trade-off, these savings come at a cost: Produce requests must now wait for acknowledgments from object storage, introducing some additional latency. However, this trade-off can result in up to 90% cost savings, making it a compelling choice for cost-sensitive workloads.

Comparison between Classic Engine and Ursa Engine

Below is a feature comparison between the Classic and Ursa Engines:

The Kafka API compatibility and Lakehouse Storage are available as add-ons for Classic Engine users. However, Oxia and S3-based WAL are integral to the Ursa Engine and cannot be retrofitted onto existing clusters.

Choose the right engine for your workload

Currently, the Classic Engine is the default engine for StreamNative Cloud. If you are looking for a more cost-effective storage solution for your latency-relaxed workloads, you can try the Ursa Engine Public Preview.

The engine is associated with an instance, and you can choose the engine when you create an instance.