Kafka Protocol on StreamNative Cloud vs. KoP
Kafka-on-Pulsar (KoP) is an open-source protocol handler that enables Apache Pulsar brokers to support the Kafka protocol. Originally developed by StreamNative, KoP provides the foundation for StreamNative Cloud’s Kafka protocol support. StreamNative Cloud enhances KoP by providing an enterprise-grade Kafka-compatible experience with additional features and capabilities. The comparison tables below highlight the key differences between StreamNative Cloud’s Kafka Protocol implementation and the open-source KoP. StreamNative Cloud offers two data streaming engines: the Classic Engine and the Ursa Engine (currently in Public Preview). The following tables detail the features supported by each implementation and the differences between StreamNative’s implementation and KoP.Kafka Protocol Support
Please note that KStreams and KSqlDB support in Ursa Engine has certain limitations. It doesn’t support functionalities that require transactions and topic compaction.
| Classic Engine | Ursa Engine | KoP | |
|---|---|---|---|
| Publish & Consume | YES | YES | YES |
| Unified Schema Registry | Coming Soon | Coming Soon | - |
| Kafka Schema Registry | YES | YES | YES |
| Transactions | YES | Coming Soon | YES |
| Compacted Topic | YES | Coming Soon | |
| KStreams Integration | YES | Yes, with limitations | |
| KSqlDB Integration | YES | Yes, with limitations |
Production Readiness
| Classic Engine | Ursa Engine | KoP | |
|---|---|---|---|
| OAuth Authentication | YES | YES | - |
| Kubernetes Authentication | YES | YES | - |
| RBAC | Coming Soon | Coming Soon | - |
| Schema Registry OAuth Authentication | YES | YES | - |
| Schema Registry RBAC | Coming Soon | Coming Soon | - |
| TLS | YES | YES | YES |
| Authorization | YES | YES | YES |
| Multi-AZ / Multi-Region Clusters | YES | YES | - |
| Geo-replication | YES | YES | - |
Deployments and Efficient Operations
| Classic Engine | Ursa Engine | KoP | |
|---|---|---|---|
| Serverless / Dedicated | YES | Comming Soon | - |
| BYOC | YES | YES | - |
| Private Cloud | YES | Coming Soon | - |
| On-prem (self-managed) | YES | Coming Soon | YES |
| Auto-scaling | YES | YES | - |
| Cloud Console / UI | YES | YES | - |
Get started
To get started with StreamNative Cloud using Kafka, see the QuickStart guide to learn how to set up a cluster with Kafka protocol enabled and configure a Kafka client for producing and consuming messages. For language-specific setup instructions, refer to our Kafka Client Guides which provide QuickStart tutorials for your preferred programming language.Use Kafka Client Setup Wizard
After provisioning your cluster, StreamNative Console provides a step-by-step wizard to help you set up Kafka client libraries and tools. The wizard guides you through the basic setup and configuration process, including selecting or creating service accounts, downloading key files or tokens, installing client libraries, and generating sample code to run. To get started with the Kafka client setup wizard, follow these steps.- Navigate to your StreamNative Cloud cluster.
-
On the left navigation pane, in the Clients section, click Kafka Clients.
- Follow the wizard to generate the sample code you need for connecting to your StreamNative cluster.
Kafka Clients
| Language | References | Description |
|---|---|---|
| Kafka Java client | QuickStart | Client Guide | Tutorial | Java producer and consumer shipped with Apache Kafka. |
| Kafka C/C++ client | QuickStart | Tutorial | librdkafka, a C/C++ library for Kafka. |
| Kafka Python client | QuickStart | Tutorial | Python client that provides high-level producer, consumer and AdminClient. |
| Kafka Go client | QuickStart | Tutorial | Go client that offers a producer and consumer for Kafka. |
| Kafka Node.js client | QuickStart | Tutorial | Node.js client that provides Kafka APIs |
| Kafka .NET client | Tutorial | .NET client that provides a high-level producer, consumer and AdminClient. |