Kafka Compacted Topic

StreamNative and Apache Kafka both support topic compaction, a key-based data retention mechanism. Unlike the segment-based data retention mechanism, which removes old segments based on time or size, the key-based mechanism retains the latest value for a given key. However, there are subtle differences in how data compaction is handled between StreamNative and Apache Kafka because the topic compaction mechanism on StreamNative Cloud is based on Apache Pulsar.

• Apache Kafka supports a delete+compact retention policy, which can remove the record of an old key even if it is the latest value for that key and applies the segment-based data cleanup policy (1 day by default). However, StreamNative cannot remove compacted keys by retention time or size unless a tombstone (null value) message is written into the topic.
• The topic config options max.compaction.lag.ms and min.compaction.lag.ms are not supported.
• Topic compaction for Kafka currently cannot work with transactions, but support for this feature is planned for the future.
• In Kafka, a tombstone (a key with a null value) is retained for a period set by delete.retention.ms. In contrast, StreamNative/Pulsar removes the tombstone immediately.
• StreamNative supports manually triggering compaction, whereas Kafka does not.

Use Compacted Topic

Create Compacted Topic

To create a compact topic, you can follow the CLI Tools tutorial and use the following command line to create the compact topic:

./bin/kafka-topics.sh --create --bootstrap-server <YOUR-BOOTSTRAP-SERVER-ADDRESS> --replication-factor 1 --partitions 1 --topic my_compact_topic --config "cleanup.policy=compact"

Configure Compaction Policy

You can change the compaction-threshold policy to control how often compression is triggered (default 100MB) it specifies how large the topic backlog can grow before compaction is triggered, or you can manually trigger compaction using the Pulsar administrative API. For more information, see Topic Compaction Cookbook.