Configure Kafka Protocol

StreamNative Private Cloud supports the Kafka protocol via the Ursa engine, a Kafka-compatible data streaming engine that has been refined through our extensive experience developing Pulsar, KoP (Kafka-on-Pulsar), and managing both Pulsar and Kafka at scale. Designed for today's cost-conscious economy, Ursa offers a genuinely cloud-native experience and enterprise-grade features that boost developer productivity and operational efficiency.

• All of the Kafka features you're used to: Ursa supports all the Kafka features you're accustomed to, including KStreams, KSQL, KTables with Topic Compaction, Schema Registry for the Java Client, and Kerberos Authentication for Kafka Clients.
• Great developer experience: Ursa features streamlined local testing, with its own testcontainers module.
• Robust and reliable: Architecturally designed for scale and subjected to rigorous resilience testing. We've fortified Ursa through comprehensive testing, ensuring it meets and exceeds the demands of large-scale Kafka operations.

Enable Kafka Protocol

To enable the Kafka Protocol on StreamNative Private Cloud, you need to configure the PulsarBroker CR as follows:

spec:
  config:
    protocolHandlers:
      kop:
        enabled: true
    transactionEnabled: true

• [1] spec.config.protocolHandlers.kop.enabled: Required. A boolean to control whether the Kafka Protocol is enabled or disabled in a StreamNative cluster.
• [2] spec.config.transactionEnabled: Required. A boolean to control whether the Kafka Transaction is enabled or disabled in a StreamNative cluster.

Enable Kafka Proxy

StreamNative supports a proxy extension which allows Kafka clients to connect and route Kafka traffic to a StreamNative cluster.

To use Kafka Proxy on StreamNative Private Cloud, you need to configure on the PulsarProxy CR as follows:

spec:
  config:
    kopProxy:
      enabled: true
      kafkaAdvertisedListener: 'a93bbde015ab54756ab4ce46154d7e8f-a6056a28f6b1b492.elb.us-west-2.amazonaws.com'

• [1] spec.config.kopProxy.enabled: Required. A boolean to control whether the Kafka Proxy is enabled or disabled in the StreamNative cluster.
• [2] spec.config.kopProxy.kafkaAdvertisedListener: Required. Specifies the listeners for Kafka clients. In Kubernetes environments, you should configure the external Proxy LoadBalancer type service address or NodePort type service address for this field.

For example, on AWS EKS, you should config the following annotations to expose an external LoadBalancer type service, then use the LoadBalancer hostname as the kafkaAdvertisedListener:

metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-type: external

Use Kafka Client to connect to Kafka Proxy

After enabling the Kafka Protocol on PulsarBroker and Kafka Proxy on PulsarProxy, we can follow below steps to have a quick tutorial:

Download the Kafka release

tar -xzf kafka_2.13-3.6.0.tgz
cd kafka_2.13-3.6.0

Click Produce messages through the Kafka Proxy

bin/kafka-console-producer.sh --topic Kafka-on-StreamNative --bootstrap-server a93bbde015ab54756ab4ce46154d7e8f-a6056a28f6b1b492.elb.us-west-2.amazonaws.com:9092

>First message to Kafka on StreamNative
>Second message to Kafka on StreamNative
>Third message to Kafka on StreamNative

Consume messages through the Kafka Proxy

bin/kafka-console-consumer.sh --topic Kafka-on-StreamNative --from-beginning --bootstrap-server a93bbde015ab54756ab4ce46154d7e8f-a6056a28f6b1b492.elb.us-west-2.amazonaws.com:9092

First message to Kafka on StreamNative
Second message to Kafka on StreamNative
Third message to Kafka on StreamNative