Use Kafka Tools With StreamNative Cloud

Apache Kafka provides a suite of command-line interface (CLI) tools that can be accessed from the bin/ directory after downloading and extracting the Kafka distribution. These tools offer a range of capabilities, including starting and stopping Kafka, managing topics, and handling partitions. To learn how to use each tool, simply run it with no argument or use the --help argument for detailed instructions.

You can use these tools by creating a configuration file that contains basic connectivity details such as the bootstrap server and a API key. You can use this file with any Kafka tool that accepts a configuration file. Some of the tools that provide a configuration option, and the option to specify the configuration file are listsed in the table that follows:

Create a configuration file

1. Create a file named cloud.properties and save it in a secure location. You will populate this file with credentials to access your StreamNative Cloud account so you must keep in a safe place. Add the following content to the file:

   bootstrap.servers=<broker-endpoint>
   security.protocol=SASL_SSL
   sasl.mechanism=PLAIN
   sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="unused" password="token:<api-key>";
   

2. Next, you will populate the file with your StreamNative Cloud cluster information.

   1. Sign in to your StreamNative Cloud account.

   2. In the Cloud Console, navigate to the cluster you want to connect to.

   3. In the Cluster Dashboard page, select the Details tab.

   4. Copy the Kafka Service URL (TCP) value and paste it into the bootstrap.servers property in the cloud.properties file.

   5. Follow the instructions in Create an API key to create an API key and paste the API Key value into the cloud.properties file.

Use the tool

After you have set up the configuration file that references your cluster, you can use it with some of the Kafka tools. You will also need the bootstrap server of the cluster when you run the tool.

The following example demonstrates how to write messages to a topic named test in your StreamNative Cloud cluster using the kafka-console-producer.sh tool. The command requires both the configuration file and bootstrap server to be specified as options. Before running this command:

1. Ensure the topic test exists in your cluster
2. Replace <bootstrap-server> with your cluster's actual bootstrap server address
3. Update the path to point to your cloud.properties file location

./bin/kafka-console-producer.sh --producer.config /path/to/secure/location/cloud.properties --bootstrap-server <bootstrap-server> --topic test

Now enter some messages at the prompt (>):

>hello world
>another message
>3rd message

You can also use the kafka-console-consumer.sh tool to read messages from the topic.

./bin/kafka-console-consumer.sh --consumer.config /path/to/secure/location/cloud.properties --bootstrap-server <bootstrap-server> --from-beginning --group test-group --topic test

You should see the following output:

hello world
another message
3rd message