acks
configuration parameter. Although you can use the acks
parameter for throughput and latency optimization, it is primarily used to ensure message durability.
To optimize for high durability, set acks=all
(equivalent to acks=-1
). With this setting:
retries
parameter (default MAX_INT
) and up to the time duration specified by delivery.timeout.ms
(default 120000ms
). You can tune delivery.timeout.ms
to set an upper bound on the total time between sending a message and receiving an acknowledgment from the broker, which should align with your business requirements for message validity.
There are two key considerations with automatic producer retries:
enable.idempotence=true
. With idempotency enabled, brokers track messages using incrementing sequence numbers (similar to TCP). This prevents message duplication because brokers ignore duplicate sequence numbers, and preserves message ordering because on failures, the producer temporarily constrains to a single in-flight message until sequencing is restored. If idempotency guarantees cannot be satisfied, the producer raises a fatal error and rejects further sends. Applications should catch and handle these fatal errors appropriately.
If you don’t configure producer idempotency but require these guarantees, you must handle potential duplication and ordering issues differently:
max.in.flight.requests.per.connection=1
to allow only one request at a timeretries=0
to preserve order while allowing pipelining (accepting potential message loss)onCompletion()
method in the Java client’s Callback interface). For manual retry handling, disable automatic retries with retries=0
. Note that producer idempotency only works with automatic retries enabled - manual retries generate new sequence numbers that bypass duplication detection. While disabling automatic retries may create message gaps from individual failures, the broker still preserves the order of received writes.
poll()
call at regular intervals. While this default behavior works well for many use cases, you may need stronger guarantees if your consumer is part of a transactional chain. In such cases, you might want to ensure offsets are only committed after messages are fully processed.
You can control whether offset commits happen automatically or manually using the enable.auto.commit
parameter:
enable.auto.commit=true
(default), offsets are committed automatically during pollingenable.auto.commit=false
, you must explicitly commit offsets in your consumer code using either:
commitSync()
for synchronous commitscommitAsync()
for asynchronous commitsisolation.level
configuration parameter:
isolation.level=read_committed
ensures consumers only receive:
enable.auto.commit=false
on the consumersendOffsetsToTransaction()
method in the KafkaProducer
interfaceprocessing.guarantee
parameter for exactly-once processingConfiguration | Recommended Value | Default Value | Description |
---|---|---|---|
replication.factor | 3 | - | Number of replicas for each partition |
acks | all | all , default prior to Kafka 3.0: 1 | Number of acknowledgments required |
enable.idempotence | true | true , default prior to Kafka 3.0: false | Enable exactly-once delivery semantics |
max.in.flight.requests.per.connection | 1 | 5 | Maximum number of unacknowledged requests |
Configuration | Recommended Value | Default Value | Description |
---|---|---|---|
enable.auto.commit | false | true | Enable automatic offset commits |
isolation.level | read_committed (Ursa Engine doesn’t support read_committed yet) | read_uncommitted for Java client and read_committed for librdkafka based clients | Transaction isolation level for consumers |