Performance

KSN is dedicated to providing users with a performance experience equivalent to native Pulsar. This is achieved because KSN itself builds upon the foundation of Pulsar Topics, while ensuring consistency with native Pulsar in terms of data transmission, storage, and other aspects.

In addition to CPU, network bandwidth, and disk I/O resources, there are two most crucial factors that impact KSN's performance:

Batch Size

This determination is typically made on the client side, and it is a widely employed practice that significantly enhances throughput. Aggregating multiple small messages into a larger message can significantly reduce the overhead of network packet transmission and disk IO operations. Indeed, this approach is achieved by introducing an acceptable increase in write latency. Typically, this would entail adding a few milliseconds of latency. You can adjust the batch size within an acceptable publish latency by setting the client's linger.ms and batch.size parameters.

Data Format

KSN offers support for two primary data formats, pulsar, and kafka, when it comes to storing data in the storage layer. The Pulsar format has been specifically designed to build interoperability between Kafka clients and Pulsar clients. As a result, it is imperative to undergo the transformation process, converting the data from Kafka producers into the Pulsar format, prior to its storage in the storage layer. On the contrary, the Kafka format is intended to provide exceptional performance experiences for users of Kafka clients. However, when utilizing Pulsar clients for consumption, it becomes necessary to introduce a Kafka format data processor within the client.