Tune Kafka Client Performance

StreamNative is dedicated to providing Kafka users with a performance experience equivalent to that of native Pulsar. This is achieved because StreamNative builds upon the foundation of Pulsar topics, ensuring consistency with native Pulsar in terms of data transmission, storage, and other aspects.

In addition to CPU, network bandwidth, and disk I/O resources, there are two crucial factors that impact the performance of Kafka clients:

Batch Size

This determination is typically made on the client side, and it is a widely employed practice that significantly enhances throughput. Aggregating multiple small messages into a larger message can significantly reduce the overhead of network packet transmission and disk IO operations. Indeed, this approach is achieved by introducing an acceptable increase in write latency. Typically, this would entail adding a few milliseconds of latency. You can adjust the batch size within an acceptable publish latency by setting the client's linger.ms and batch.size parameters.

Data Format

StreamNative supports two primary data formats, pulsar and kafka, for storing data in the storage layer. The Pulsar format is specifically designed to build interoperability between Kafka and Pulsar clients. As a result, it is imperative to undergo a transformation process, converting data from Kafka producers into the Pulsar format before storage in the storage layer. In contrast, the Kafka format is intended to provide exceptional performance experiences for users of Kafka clients. However, when using Pulsar clients for consumption, it becomes necessary to introduce a Kafka format data processor within the client.