1. StreamNative Cloud
  2. Connect

Connect to cluster through Kafka CLI

This document assumes that you have created a Pulsar cluster and a service account, and have granted the service account produce and consume permissions to a namespace on your Pulsar cluster. For details, see get started with the Kafka protocol.

This document describes how to connect to your Pulsar cluster using the Kafka CLI tool (v3.1.0) through the OAuth2 authentication plugin.

Prerequisites

Steps

  1. Download Kafka 3.1.0 release and extract it to the ~/kafka folder.

    mkdir -p ~/kafka && cd ~/kafka
    # download Kafka 3.1.0
    curl -O https://archive.apache.org/dist/kafka/3.1.0/kafka_2.13-3.1.0.tgz
    tar xzf ./kafka_2.13-3.1.0.tgz
    
  2. Download the supplementary libraries for the Kafka client.

    cd ~/kafka/kafka_2.13-3.1.0
    # download supplementary libraries
    curl -O https://repo1.maven.org/maven2/io/streamnative/pulsar/handlers/oauth-client/2.9.1.5/oauth-client-2.9.1.5.jar --output-dir ./libs
    curl -O https://repo1.maven.org/maven2/org/apache/pulsar/pulsar-client-admin-api/2.9.2/pulsar-client-admin-api-2.9.2.jar --output-dir ./libs
    curl -O https://repo1.maven.org/maven2/org/apache/pulsar/pulsar-client/2.9.2/pulsar-client-2.9.2.jar --output-dir ./libs
    curl -O https://repo1.maven.org/maven2/org/apache/pulsar/pulsar-client-api/2.9.2/pulsar-client-api-2.9.2.jar --output-dir ./libs
    
  3. Create an OAuth2 configuration file.

    This example creates a file named kafka.properties, substituting the path to your downloaded OAuth2 credential file and the audience respectively.

    # configure kafka.properties file.
    echo 'sasl.login.callback.handler.class=io.streamnative.pulsar.handlers.kop.security.oauth.OauthLoginCallbackHandler
    security.protocol=SASL_SSL
    sasl.mechanism=OAUTHBEARER
    sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule \
      required oauth.issuer.url="https://auth.streamnative.cloud/"\
      oauth.credentials.url="file:///YOUR-KEY-FILE-PATH"\
      oauth.audience="YOUR-AUDIENCE-STRING";' > ~/kafka/kafka.properties
    
    • required oauth.issuer.url: the OAuth2 authentication provider. You can get the value from your downloaded OAuth2 credential file.
    • oauth.credentials.url: the path to your downloaded OAuth2 credential file. For details about how to get the OAuth2 credential file, see get an OAuth2 credential file.
    • oauth.audience: the audience parameter is a combination of the urn:sn:pulsar, your organization name, and your Pulsar instance name.
  4. Connect to the cluster through the OAuth2 authentication plugin.

    a. Open a terminal and run a Kafka consumer to receive a message from the test-topic topic.

    # run consumer
    ~/kafka/kafka_2.13-3.1.0/bin/kafka-console-consumer.sh \
        --bootstrap-server "your-pulsar-service-url" \
        --consumer.config ~/kafka/kafka.properties \
        --topic test-topic
    
    • bootstrap-server: the service URL of your Pulsar cluster. For details, see get a service URL.

    b. Open another terminal and run a Kafka producer to send a message to the test-topic topic.

    # run producer
    ~/kafka/kafka_2.13-3.1.0/bin/kafka-console-producer.sh \
        --bootstrap-server "your-pulsar-service-url" \
        --producer.config ~/kafka/kafka.properties \
        --topic test-topic
    

    You can type some messages, for example Hello, kafka on Pulsar! and then press the Enter key to produce the message to the test-topic topic. Then, you should see this message on the consumer terminal.

Previous
CLI - pulsar-perf