Configuring Kafka Producer

An Apache Kafka producer is a client application that publishes (sends) messages to StreamNative Cloud. This section gives an overview of the Kafka producer and an introduction to the configuration settings for Producers.

Producer Configuration

The following sections describe the key configuration settings for Kafka producers and explain how they affect producer behavior.

For common client settings, such as networking and authentication settings, see Configuring Kafka Clients.

Message Durability

acks

Controls the durability of messages written to StreamNative Cloud. This setting has three possible values:

• all (default): Provides the strongest durability guarantee. The broker will wait for the message to be successfully persisted to the storage layer (either BookKeeper or object storage) before sending an acknowledgment response.

• 1: Requires acknowledgement only from the owner broker. Provides moderate durability with better performance than all.

• 0: Provides no durability guarantees but maximum throughput. The broker does not send any response, so you cannot verify message delivery or determine message offsets.

For C/C++, Python, Go and .NET clients, this is configured per-topic. To apply globally, use:

• C/C++: default_topic_conf sub-configuration
• Python, Go, .NET: default.topic.config sub-configuration

Message Ordering

By default, messages are written to the broker in the same order that they are received by the producer client. However, certain configuration settings can affect this ordering.

retries

This setting controls message retry attempts when a send fails. When set to a value greater than 0, retries are enabled (the default is 0, which disables retries). With retries enabled, message re-ordering can occur if a retry succeeds after subsequent messages have already been written successfully.

max.in.flight.requests.per.connection

To maintain strict message ordering while using retries, set this value to 1. This ensures only one request can be sent to the broker at a time, preventing any potential re-ordering. Note that when retries are disabled, the broker preserves the order of writes it receives, but failed sends can create gaps in the message sequence.

Batching and Compression

Kafka producers attempt to collect sent messages into batches to improve throughput and efficiency.

Use the following settings to control batching and compression:

batch.size

Controls the maximum size in bytes of each message batch for the Java client. When a batch reaches this size, it is sent to the broker.

linger.ms

Use this setting to control how long the producer waits to allow more messages to accumulate in a batch before sending it. A longer linger time increases the likelihood of filling batches but adds latency.

compression.type

Enable compression with this setting. Compression covers full message batches, so larger batches typically achieve higher compression ratios. When using Snappy compression, you need write access to the /tmp directory. If you don't have write access to the /tmp directory because it's set to noexec, you can specify an alternate directory path that you have write access to:

-Dorg.xerial.snappy.tempdir=<path>

batch.num.messages

Use this setting with the C/C++, Python, Go, and .NET clients to set a limit on the number of messages contained in each batch.

Queuing Limit

buffer.memory

Controls the total memory available to the Java client for collecting unsent messages. When this limit is reached, the producer will block additional sends for up to max.block.ms before raising an exception.

request.timeout.ms

Sets a timeout to prevent records from being queued indefinitely. If this timeout expires before a message is successfully sent, the message will be removed from the queue and an exception will be thrown. The C/C++, Python, Go, and .NET clients have similar settings.