- For Classic Engine clusters, messages are replicated across multiple storage nodes to protect against data loss
- For Ursa Engine clusters, messages are persisted to object storage
Producer Acknowledgments
Producers can control the durability of messages written to Kafka through theacks
configuration parameter. Although you can use the acks
parameter for throughput and latency optimization, it is primarily used to ensure message durability.
To optimize for high durability, set acks=all
(equivalent to acks=-1
). With this setting:
- For Classic Engine clusters, the broker waits for acknowledgment from all storage nodes before responding to the producer
- For Ursa Engine clusters, the broker waits for acknowledgment from object storage before responding to the producer
Duplication and Ordering
Producers can increase durability by retrying failed message sends to prevent data loss. The producer automatically retries sending messages up to the number specified by theretries
parameter (default MAX_INT
) and up to the time duration specified by delivery.timeout.ms
(default 120000ms
). You can tune delivery.timeout.ms
to set an upper bound on the total time between sending a message and receiving an acknowledgment from the broker, which should align with your business requirements for message validity.
There are two key considerations with automatic producer retries:
- Duplication: Transient failures in StreamNative Cloud may cause the producer to send duplicate messages when retrying.
- Ordering: Multiple sends may be “in flight” simultaneously, and a retry of a failed message may occur after a newer message has succeeded.
enable.idempotence=true
. With idempotency enabled, brokers track messages using incrementing sequence numbers (similar to TCP). This prevents message duplication because brokers ignore duplicate sequence numbers, and preserves message ordering because on failures, the producer temporarily constrains to a single in-flight message until sequencing is restored. If idempotency guarantees cannot be satisfied, the producer raises a fatal error and rejects further sends. Applications should catch and handle these fatal errors appropriately.
If you don’t configure producer idempotency but require these guarantees, you must handle potential duplication and ordering issues differently:
- For duplication: Build consumer application logic to handle duplicate messages
- For ordering: Either:
- Set
max.in.flight.requests.per.connection=1
to allow only one request at a time - Set
retries=0
to preserve order while allowing pipelining (accepting potential message loss)
- Set
onCompletion()
method in the Java client’s Callback interface). For manual retry handling, disable automatic retries with retries=0
. Note that producer idempotency only works with automatic retries enabled - manual retries generate new sequence numbers that bypass duplication detection. While disabling automatic retries may create message gaps from individual failures, the broker still preserves the order of received writes.
Consumer Offsets and Auto Commit
When optimizing for durability, you need to carefully consider how consumer offsets are managed, especially in the case of unexpected consumer failures. Consumer offsets track which messages have been consumed, making the timing and method of offset commits crucial for durability. A key scenario to avoid is when a consumer commits an offset for a message, begins processing that message, but then fails unexpectedly. In this case, when a new consumer takes over the partition, it won’t reprocess any messages with offsets that were already committed, potentially leading to data loss. By default, consumer offsets are automatically committed during the consumer’spoll()
call at regular intervals. While this default behavior works well for many use cases, you may need stronger guarantees if your consumer is part of a transactional chain. In such cases, you might want to ensure offsets are only committed after messages are fully processed.
You can control whether offset commits happen automatically or manually using the enable.auto.commit
parameter:
- With
enable.auto.commit=true
(default), offsets are committed automatically during polling - With
enable.auto.commit=false
, you must explicitly commit offsets in your consumer code using either:commitSync()
for synchronous commitscommitAsync()
for asynchronous commits
Exactly Once Semantics (EOS)
EOS transactions are supported for Classic Engine clusters only.
isolation.level
configuration parameter:
- Setting
isolation.level=read_committed
ensures consumers only receive:- Non-transactional messages
- Committed transactional messages
- No messages from open or aborted transactions
- Set
enable.auto.commit=false
on the consumer - Manually commit offsets using the
sendOffsetsToTransaction()
method in theKafkaProducer
interface - For event streaming applications, configure the
processing.guarantee
parameter for exactly-once processing
Summary
Here’s a summary of key configurations for optimizing durability:Producer Configurations
Configuration | Recommended Value | Default Value | Description |
---|---|---|---|
replication.factor | 3 | - | Number of replicas for each partition |
acks | all | all , default prior to Kafka 3.0: 1 | Number of acknowledgments required |
enable.idempotence | true | true , default prior to Kafka 3.0: false | Enable exactly-once delivery semantics |
max.in.flight.requests.per.connection | 1 | 5 | Maximum number of unacknowledged requests |
Consumer Configurations
Configuration | Recommended Value | Default Value | Description |
---|---|---|---|
enable.auto.commit | false | true | Enable automatic offset commits |
isolation.level | read_committed (Ursa Engine doesn’t support read_committed yet) | read_uncommitted for Java client and read_committed for librdkafka based clients | Transaction isolation level for consumers |