Kafka Compatibility Overview

Warning

If you are using a Ursa-Engine powered cluster, please note that transactions and topic compaction are not supported in Ursa Engine.

StreamNative Cloud is compatible with Apache Kafka versions 0.9 and later, with specific exceptions noted on this page.

TLS SNI extension requirements

StreamNative Cloud uses TLS Service Name Indication (SNI) to route the Kafka or Pulsar requests to the correct brokers. In order to connect to a StreamNative Cloud cluster, the client libraries you are using must include a SNI extension in the TLS handshake. Most of the Kafka client libraries do include the SNI extension in the TLS handshake.

For Kafka protocol connections (which use port 9093 on StreamNative Cloud), the SNI extension must be set to the DNS hostname of the cluster endpoint or one of the brokers.

For initial connections to the cluster, they include the configured bootstrap endpoint as the SNI extension.
For the subsequent connections to individual brokers which are made after bootstrapping, they include that specific broker's endpoint as the SNI extension.

These broker endpoints are automatically discovered as part of Kafka client bootstrapping process and do not need to be configured.

For REST/HTTPS connections (which use port 443), the SNI extension must be set to the DNS hostname of the cluster endpoint.

Any forward proxies deployed in your environment that are in the path of the network traffic to StreamNative Cloud must be configured to forward the SNI extension unchanged.

For testing conformance with TLS SNI requirements, see Test Connectivity.

See TLS Extensions: Server Name Indication and TLS Protocol Version 1.3: Server Name Indication for more details.

Kafka client compatibility

Clients developed for Kafka versions 0.9 or later are compatible with StreamNative Cloud. Modern clients will automatically negotiate protocol versions or utilize an earlier protocol version that StreamNative Cloud accepts.

Note

We recommend always using the latest supported version of a client.

The following clients have been validated with StreamNative Cloud:


Language	Client
Java	Apache Kafka Java Client
C/C++	librdkafka
Go	confluent-kafka-go franz-go sarama
Python	confluent-kafka-python
Node.js	KafkaJS
.NET	confluent-kafka-dotnet
Rust	rust-rdkafka

Other clients that use the Kafka protocol but have not been validated by StreamNative may still be compatible with StreamNative Cloud, subject to the limitations described below. This is particularly true for clients based on librdkafka.

If you encounter any compatibility issues with a client not listed above, please reach out to the StreamNative team through the StreamNative Cloud console chat or Support Portal for assistance.

Note

Specifically, librdkafka 2.5.0 and 2.5.3, and all other client SDKs based on these two versions are not supported (e.g. confluent-kafka-python 2.5.0 and 2.5.3). It's caused by a backward compatibility regression in librdkafka, which is fixed in 2.6.0 (see librdkafka #4871).

Unsupported Kafka features

StreamNative Cloud does not currently support the following Apache Kafka features:

Managing SASL users with Kafka Admin APIs: Use Pulsar Admin instead.
Managing Kafka ACLs with Kafka Admin APIs: Use Pulsar ACLs instead. For more details, see Kafka ACLs.
Limited support for Topic Configuration: Since StreamNative Cloud is based on Apache Pulsar, the storage layer differs from Apache Kafka. The topic configuration only supports cleanup.policy to set the compact or delete retention policy.

If you encounter any issues while working with a Kafka tool, you can file a support ticket through the Support Portal.