Build Kafka Client Applications on StreamNative Cloud
StreamNative Cloud provides a fully managed data streaming service with complete Kafka protocol compatibility. This enables developers to build streaming applications using their preferred programming language and standard Kafka clients. You can also seamlessly migrate existing Kafka applications to StreamNative Cloud without modifying any code.
Kafka Protocol on StreamNative Cloud vs. KoP
Kafka-on-Pulsar (KoP) is an open-source protocol handler that enables Apache Pulsar brokers to support the Kafka protocol. Originally developed by StreamNative, KoP provides the foundation for StreamNative Cloud’s Kafka protocol support.
StreamNative Cloud enhances KoP by providing an enterprise-grade Kafka-compatible experience with additional features and capabilities. The comparison tables below highlight the key differences between StreamNative Cloud’s Kafka Protocol implementation and the open-source KoP.
StreamNative Cloud offers two data streaming engines: the Classic Engine and the Ursa Engine (currently in Public Preview). The following tables detail the features supported by each implementation and the differences between StreamNative’s implementation and KoP.
Kafka Protocol Support
Please note that KStreams and KSqlDB support in Ursa Engine has certain limitations. It doesn’t support functionalities that require transactions and topic compaction.
| Classic Engine | Ursa Engine | KoP | |
|---|---|---|---|
| Publish & Consume | YES | YES | YES |
| Unified Schema Registry | Coming Soon | Coming Soon | - |
| Kafka Schema Registry | YES | YES | YES |
| Transactions | YES | Coming Soon | YES |
| Compacted Topic | YES | Coming Soon | |
| KStreams Integration | YES | Yes, with limitations | |
| KSqlDB Integration | YES | Yes, with limitations |
Production Readiness
| Classic Engine | Ursa Engine | KoP | |
|---|---|---|---|
| OAuth Authentication | YES | YES | - |
| Kubernetes Authentication | YES | YES | - |
| RBAC | Coming Soon | Coming Soon | - |
| Schema Registry OAuth Authentication | YES | YES | - |
| Schema Registry RBAC | Coming Soon | Coming Soon | - |
| TLS | YES | YES | YES |
| Authorization | YES | YES | YES |
| Multi-AZ / Multi-Region Clusters | YES | YES | - |
| Geo-replication | YES | YES | - |
Deployments and Efficient Operations
| Classic Engine | Ursa Engine | KoP | |
|---|---|---|---|
| Serverless / Dedicated | YES | Comming Soon | - |
| BYOC | YES | YES | - |
| Private Cloud | YES | Coming Soon | - |
| On-prem (self-managed) | YES | Coming Soon | YES |
| Auto-scaling | YES | YES | - |
| Cloud Console / UI | YES | YES | - |
You can use StreamNative Cloud for your existing Kafka applications and services without migrating the code. See a full list of supported Kafka clients and Kafka Compatibility.
Get started
To get started with StreamNative Cloud using Kafka, see the QuickStart guide to learn how to set up a cluster with Kafka protocol enabled and configure a Kafka client for producing and consuming messages.
For language-specific setup instructions, refer to our Kafka Client Guides which provide QuickStart tutorials for your preferred programming language.
Use Kafka Client Setup Wizard
After provisioning your cluster, StreamNative Console provides a step-by-step wizard to help you set up Kafka client libraries and tools. The wizard guides you through the basic setup and configuration process, including selecting or creating service accounts, downloading key files or tokens, installing client libraries, and generating sample code to run.
To get started with the Kafka client setup wizard, follow these steps.
-
On the left navigation pane, in the Clients section, click Kafka Clients.
-
Follow the wizard to generate the sample code you need for connecting to your StreamNative cluster.
With a copy-and-paste, you can run the given sample code to produce and consume messages.
Kafka Clients
| Language | References | Description |
|---|---|---|
| Kafka Java client | QuickStart | Client Guide | Tutorial | Java producer and consumer shipped with Apache Kafka. |
| Kafka C/C++ client | QuickStart | Tutorial | librdkafka, a C/C++ library for Kafka. |
| Kafka Python client | QuickStart | Tutorial | Python client that provides high-level producer, consumer and AdminClient. |
| Kafka Go client | QuickStart | Tutorial | Go client that offers a producer and consumer for Kafka. |
| Kafka Node.js client | QuickStart | Tutorial | Node.js client that provides Kafka APIs |
| Kafka .NET client | Tutorial | .NET client that provides a high-level producer, consumer and AdminClient. |
Kafka CLI & Tools
You can use the Kafka CLI tools to connect to your StreamNative cluster. You can see a quickstart guide here. For more information, see Use Kafka Tools with StreamNative Cloud.
Kafka Connect
StreamNative Cloud provides full compatibility with Kafka Connect, offering two deployment options: you can self-host Kafka Connect connectors in your own environment (see examples here), or leverage fully managed Kafka Connect connectors running directly in StreamNative Cloud.
Kafka Streams
You can also build data streaming applications using Kafka Streams. See the Kafka Streams QuickStart for more information. Please note that StreamNative Cloud doesn’t host any Kafka Streams applications.
KSQL
You can also build data streaming applications using KSQL. See the KSQL QuickStart for more information. Please note that StreamNative Cloud doesn’t host the KSQL service.