- Build Applications
- Kafka Clients
Migrating to StreamNative
StreamNative adds native support for the Kafka protocol, but it does not mean StreamNative is an exact copy of Apache Kafka. StreamNative not only maintains compatibility with the Kafka protocol but also incorporates many of Pulsar's excellent features. It is crucial for Kafka users to understand these differences before migrating to StreamNative. By doing so, they can avoid any potential losses resulting from the disparities between the two systems.
Data Retention
Pulsar has a different data retention policy by default. In Pulsar, consumed and acknowledged data from all subscriptions, or data from topics with no subscriptions, is systematically removed from the topic segment by segment. In contrast, Kafka retains data within the topic for a fixed duration of 7 days, regardless of whether it has been consumed or not. However, StreamNative adopts Pulsar's approach, offering users a data retention policy that can discern data consumption patterns. Therefore, when migrating to StreamNative, it is necessary to proactively adjust the data retention policy to prevent the deletion of data that has been written, after consumption, or in the absence of subscriptions. Certainly, if Pulsar's behavior aligns with your expectations, there would be no need to modify the policy.
You can follow the Pulsar Admin CLI, Pulsar Admin API or Pulsar Admin REST API to set the data retention policy for the namespace or topic.
In practice, you can see that the retention policy is not configured in the public/default namespace via the Pulsar admin CLI:
$ ./bin/pulsar-admin namespaces get-retention public/default
null
null means the retention policy is not set. The default behavior is that all messages could be deleted after the retention period. You can set the retention policy for the namespace with the following command:
$ ./bin/pulsar-admin namespaces set-retention -t 7d -s -1 public/default
This will result in the same behavior as Kafka, where messages will be deleted after 7 days. You can confirm this by running the following command:
$ ./bin/pulsar-admin namespaces get-retention public/default
{
"retentionTimeInMinutes" : 10080,
"retentionSizeInMB" : -1
}
See more details here for how Pulsar's retention policy works.