kafka
and pulsar
. Each format has different performance characteristics:
kafka
format: The default format for StreamNative Cloud. It provides the best performance with Kafka clients. However, Pulsar consumers cannot consume this format unless a payload processor is employed.pulsar
format: Provides the highest interoperability between protocols. However, it incurs a performance penalty as it requires transforming data from Kafka producers into the Pulsar format before storage.kafka
format.
linger.ms
parameter set to 0, meaning the producer sends data as soon as it’s available. While batching is always enabled—messages are always sent in batches—with linger.ms=0
, a batch may contain only one message (unless messages arrive faster than the producer can send them).
compression.type=none
) spares CPU cycles but increases network bandwidth usage. While a good compression codec may potentially reduce latency by decreasing network transfer time, the CPU overhead of compression could offset those gains. Evaluate your specific use case - if CPU is your bottleneck, consider disabling compression; if network bandwidth is constrained, compression may help reduce overall latency.
acks
parameter:
acks=0
: Producer doesn’t wait for any acknowledgment, providing lowest latency but no durability guaranteesacks=1
: Producer waits for acknowledgment from the designated broker after receiving at least one acknowledgment from storageacks=all
: Producer waits for acknowledgment from the designated broker after receiving all acknowledgments from storageacks=all
provides the strongest durability guarantees but higher latency. For latency-sensitive applications that can tolerate potential message loss, you can set acks=0
, but be aware that messages may be lost silently if broker failures occur.
fetch.min.bytes
defaults to 1
, which means fetch requests are answered as soon as a single byte of data is available or the fetch request times out (controlled by fetch.max.wait.ms
). These two parameters work together to control both the size of fetch requests (fetch.min.bytes
) and how long to wait for data (fetch.max.wait.ms
).
For lowest latency, keep fetch.min.bytes
at its default of 1
and reduce fetch.max.wait.ms
from its default of 500ms
. This ensures consumers receive data as soon as it’s available, though at the cost of potentially more frequent fetch requests.
Configuration | Recommended Value | Default Value | Description |
---|---|---|---|
linger.ms | 0 | 0 | Time to wait for batches to fill |
compression.type | none | none | Compression codec to use |
acks | 1 | all | Number of acknowledgments required |
Configuration | Recommended Value | Default Value | Description |
---|---|---|---|
fetch.min.bytes | 1 | 1 | Minimum data size for fetch responses |