kafka
and pulsar
. Each format has different performance characteristics:
kafka
format: Provides the best performance with Kafka clients. However, Pulsar consumers cannot consume this format unless a payload processor is employed.pulsar
format: Provides the highest interoperability between protocols. However, it incurs a performance penalty as it requires transforming data from Kafka producers into the Pulsar format before storage.kafka
format.
batch.size
parameter to increase the maximum size in bytes of each message batch. To give more time for batches to fill, you can configure the linger.ms
parameter to have the producer wait longer before sending. This delay allows the producer to wait for the batch to reach the configured batch.size
. The trade-off is higher latency since messages aren’t sent immediately when they’re ready.
For Ursa Engine clusters, it is recommended to use large batch sizes and higher linger.ms
values to achieve better throughput.
compression.type
parameter to one of the following standard compression codecs:
lz4
(recommended for performance)snappy
zstd
gzip
none
(default, meaning no compression)lz4
for optimal performance instead of gzip
, which is more compute-intensive and may impact application performance. Compression is applied on full batches of data, so better batching results in better compression ratios.
acks=0
is configured, the producer sends messages without waiting for acknowledgment.
The acks
configuration parameter controls how many acknowledgments the designated broker must receive before responding to the producer:
acks=0
: Producer sends messages without waiting for any acknowledgmentacks=1
: Designated broker acknowledges after receiving at least one acknowledgment from the underlying storage. In StreamNative Cloud, due to its storage architecture differing from Apache Kafka, this setting behaves the same as acks=all
acks=all
: Designated broker waits for all acknowledgments from the underlying storageacks
setting (acks=1
or acks=all
). Instead, durability is controlled by the underlying storage settings and namespace policies that define write quorum and ack quorum sizes. From a Kafka protocol perspective, both acks=1
and acks=all
behave identically in StreamNative Cloud - they wait for acknowledgments based on the cluster and namespace configurations.
To adjust throughput, modify the write quorum size and acknowledgment quorum size in either the namespace policies or cluster configuration. Cluster-level settings will apply globally to all namespaces.
max.block.ms
time passes. You can adjust how much memory is allocated with the buffer.memory
configuration parameter.
If you don’t have many partitions, you may not need to adjust this parameter at all. However, if you have many partitions, you can tune buffer.memory
—while taking into account the message size, linger time, and partition count—to maintain pipelines across more partitions. This enables better use of bandwidth across more brokers.
fetch.min.bytes
. This parameter sets the minimum number of bytes expected for a fetch response from a consumer.
Increasing fetch.min.bytes
reduces the number of fetch requests made to StreamNative Cloud, reducing the broker CPU overhead to process each fetch, thereby improving throughput. Similar to increasing batching on the producer side, there may be a resulting trade-off with higher latency when increasing this parameter on the consumer. This is because the broker won’t send the consumer new messages until either:
fetch.min.bytes
)fetch.max.wait.ms
)Configuration | Recommended Value | Default Value | Description |
---|---|---|---|
batch.size | 100000-200000 | 16384 | Maximum size in bytes for message batches |
linger.ms | 10-100 | 0 | Time to wait for batches to fill |
compression.type | lz4 | none | Compression codec to use |
acks | all | all | Number of acknowledgments required |
buffer.memory | Increase if many partitions | 33554432 | Memory buffer size for unsent messages |
Configuration | Recommended Value | Default Value | Description |
---|---|---|---|
fetch.min.bytes | ~100000 | 1 | Minimum data size for fetch responses |
fetch.max.wait.ms | 500 | 500 | Maximum time to wait for fetch response |