Benchmarking
Benchmark testing is essential because there is no one-size-fits-all configuration for Kafka applications. The optimal configuration depends on your specific use case, enabled features, data profile, and other factors. You should run benchmark tests when planning to tune Kafka clients beyond the default settings. Understanding your application’s performance profile is crucial, especially when choosing the right data streaming engine and optimizing for throughput or latency. Benchmark test results can also help determine the right size of your StreamNative Cloud cluster and the appropriate number of partitions and producer/consumer processes.If you need help with sizing your StreamNative Cloud cluster, you can always contact us for assistance.
Initial Performance Baseline
Start by measuring baseline performance using:- Kafka tools
kafka-producer-perf-test
andkafka-consumer-perf-test
that are bundled in the Kafka distribution for JVM clients rdkafka_performance
interface for non-JVM clients using librdkafka
Application Testing
- Test your application using default Kafka configuration parameters first
-
Establish producer baseline performance:
- Remove upstream dependencies
- Use mock data generation or sanitized production data
- Ensure test data reflects production data characteristics
- When testing with compression, be mindful that unrealistic mock data (repeated patterns, zero padding) may show better compression than production data
-
Producer benchmarking:
- Start with a single producer on one server
- Measure throughput using producer metrics
- Incrementally increase producer processes to find optimal count per server
-
Consumer benchmarking:
- Follow similar process as producer testing
- Start with single consumer, then increase processes
- Determine optimal number of consumer processes per server
Tuning Process
- Run benchmark tests with different configuration parameters aligned with your application’s SLAs
- Focus on a subset of parameters - avoid changing defaults without understanding system impact
- Iterate through: adjust settings, test, analyze results, and repeat
- Continue until meeting throughput and latency requirements
Defining Application SLAs
While getting a Kafka client application running is relatively quick, proper tuning is essential before production deployment. Different use cases have different requirements, so you must identify your primary service goals and align them with your application’s SLAs. For a modern cloud data streaming platform, it is impossible to achieve all three properties ofCost
, Availability
, and Performance
based on the New CAP Theorem, so you need to find the right balance among them.
Considerations
Consider these factors when determining service goals to align with your application’s SLAs:- The specific use cases your Kafka applications serve
- Critical application and business requirements
- Kafka’s role in your business applications and services
- Best for: High-volume data processing applications that need to handle millions of writes per second
- Example: Log aggregation systems, batch processing pipelines
- Best for: Real-time applications requiring immediate data delivery
- Examples: Chat applications, interactive websites, IoT device monitoring
- Best for: Systems where data integrity is critical
- Examples: Financial transactions, audit logging, event sourcing systems
- Best for: Mission-critical applications that cannot tolerate downtime
- Examples: Payment processing systems, user authentication services