streamnative.io
Help Center
Login
  1. StreamNative Private Cloud

QuickStart - StreamNative Platform

Note

By installing the StreamNative Platform you agree to and are in compliance with the StreamNative Cloud Subscription Agreement. Please contact us to learn more or request a trial license.

This QuickStart guides you through every step of installing and running StreamNative Platform with operators on Kubernetes.

Prerequisites

Step 1: Install StreamNative Platform

Note

This step uses the original images within the operators.

  1. Install the StreamNative repositories.

    helm repo add streamnative https://charts.streamnative.io
helm repo update

  2. Create a Kubernetes namespace.

    kubectl create namespace sn-system
kubectl create namespace sn-platform

  3. Install the sn-operator.

    The sn-operator is used to manage Pulsar components, including the Pulsar broker, BookKeeper, ZooKeeper, and Pulsar proxy.

    helm upgrade --install sn-operator streamnative/sn-operator -n sn-system

  4. Install the Function Mesh operator.

    The Function Mesh operator is used to configure and manage Pulsar IO connectors and Pulsar Functions.

    Function Mesh is a serverless and purpose-built framework for orchestrating multiple Pulsar Functions and Pulsar IO connectors for stream processing applications.

    helm upgrade --install function-mesh-operator function-mesh/function-mesh-operator -n sn-system

  5. Deploy a Pulsar cluster.

    a. Deploy the Pulsar cluster.

    helm install sn-platform streamnative/sn-platform-slim -n sn-platform

Step 2: Create Pulsar tenants/namespaces/topics

pulsarctl is a Command-Line Interface (CLI) tool for Pulsar. In this section, you can use the pulsarctl CLI tool to create tenants, namespaces, and topics.

  1. Log in to pulsarctl and create a tenant.

    pulsarctl \
--admin-service-url <WEB_SERVICE_URL> \
--token <AUTH_PARAMS> \
tenants create <tenant_name>

    Replace WEB_SERVICE_URL with the Web service URL of your Pulsar cluster. Replace AUTH_PARAMS with the token that you get from StreamNative Console. For details, see the prepare to connect to a Pulsar cluster user guide.

  2. Create a namespace.

    pulsarctl namespaces create <namespace_name> -c <cluster_name>

  3. Create a topic.

    pulsarctl topics create <topic_name>

  4. List all the topics.

    pulsarctl topics list <namespace_name>

Step 3: Use pulsar-client to produce and consume events

StreamNative Platform supports all the official Pulsar clients. You can use the pulsar-client CLI tool to create producers and consumers to simulate a simple production and consumption model.

  1. Enter the toolset Pod.

    To facilitate the usage of the official Pulsar CLI tools, such as pulsar-client, StreamNative Platform provides a toolset Pod, which you can use to execute the kubectl exec command to connect to the official Pulsar CLI tools.

    kubectl exec -n <k8s_namespace> -it <release_name>-sn-platform-toolset-0 -- bash

  2. Create a consumer.

    pulsar-client consume -s <subscription_name> <topic_name> -n 0

  3. Create a producer.

    pulsar-client produce TOPIC_NAME  -m "---------hello streamnative platform-------" -n 10

    • From the producer side, you can see that the messages have been produced successfully.

      23:04:25.652 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 10 messages successfully produced

    • From the consumer side, you can receive the following messages.

      ----- got message -----
---------hello streamnative platform-------
----- got message -----
---------hello streamnative platform-------
----- got message -----
---------hello streamnative platform-------
----- got message -----
---------hello streamnative platform-------
----- got message -----
---------hello streamnative platform-------
----- got message -----
---------hello streamnative platform-------
----- got message -----
---------hello streamnative platform-------
----- got message -----
---------hello streamnative platform-------
----- got message -----
---------hello streamnative platform-------
----- got message -----
---------hello streamnative platform-------

Step 4: Use Kafka client to produce and consume events

StreamNative Platform brings native Kafka protocol support to Pulsar brokers using Kafka on Pulsar (KoP). Therefore, you can migrate your existing Kafka applications and services to Apache Pulsar without modifying the codes.

Currently, StreamNative Platform supports Kafka Client v1.0.0 - v2.6.0.

  1. Grant produce and consume permission to the Admin role on the namespace.

    pulsarctl namespaces grant-permission --role <role_name> --actions produce,consume <namespace_name>

  2. Run the Kafka image.

    kubectl run kafka --rm -it -n <k8s_namespace> --image bitnami/kafka -- bash

  3. Start a Kafka producer and send a message from the Kafka producer.

    kafka-console-producer.sh \
--producer-property security.protocol=SASL_PLAINTEXT \
--producer-property sasl.mechanism=PLAIN \
--producer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="TENANT/NAMESPACE" password="token:'YOUR_TOKEN'";' \
--broker-list <release_name>-sn-platform-broker:9092 --topic <topic_name>

    Here are security-related options to be configured.

    OptionDescriptionDefault
    usernameThe username for the Kafka client connecting to the Pulsar cluster. It is set to the name of the Pulsar tenant and namespace (TENANT_NAME/NAMESPACE_NAME) where Kafka topics are stored.public/default
    passwordThe password for the Kafka client connecting to the Pulsar cluster. It is set to the token that you get from StreamNative Console. For details, see the prepare to connect to a Pulsar cluster user guide.N/A

  4. Open a new terminal window and enter the Kafka Pod.

    kubectl exec -it -n <k8s_namespace> kafka -- bash

  5. Start a Kafka consumer.

    kafka-console-consumer.sh \
--consumer-property security.protocol=SASL_PLAINTEXT \
--consumer-property sasl.mechanism=PLAIN \
--consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="TENANT/NAMESPACE" password="token:'YOUR_TOKEN'";' \
--bootstrap-server sn-platform-broker:9092 --topic <topic_name>

Step 5: Verify interoperability between Pulsar and Kafka

As shown in Step 3 and Step 4, Pulsar producer, Pulsar consumer, Kafka producer, and Kafka consumer run normally. The Pulsar consumer can get the message from the Pulsar producer and the Kafka consumer can receive the message from the Kafka producer. This section further verifies the interoperability between Pulsar and Kafka.

  1. Start a Pulsar consumer.

    a. Enter the toolset Pod.

    kubectl exec -n <k8s_namespace> -it <release_name>-sn-platform-toolset-0 -- bash

    b. Create a Pulsar consumer.

    pulsar-client consume -s <subscription_name> <topic_name>  -n 0

  2. Start a Kafka consumer.

    a. Enter the Kafka Pod.

    kubectl exec -it -n <k8s_namespace> kafka -- bash

    b. Start a Kafka consumer.

    kafka-console-consumer.sh \
--consumer-property security.protocol=SASL_PLAINTEXT \
--consumer-property sasl.mechanism=PLAIN \
--consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="TENANT/NAMESPACE" password="token:'YOUR_TOKEN'";' \
--bootstrap-server sn-platform-broker:9092 --topic <topic_name>

  3. Start a Kafka producer.

    a. Enter the Kafka Pod.

    kubectl exec -it -n <k8s_namespace> kafka -- bash

    b. Start a Kafka producer and send a message.

    kafka-console-producer.sh \
--producer-property security.protocol=SASL_PLAINTEXT \
--producer-property sasl.mechanism=PLAIN \
--producer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="TENANT/NAMESPACE" password="token:'YOUR_TOKEN'";' \
--broker-list <release_name>-sn-platform-broker:9092 --topic <topic_name>
> message-for-both-pulsar-and-kafka-client

    At the same time, you can receive messages from both Pulsar and Kafka consumers.

    • From Pulsar consumer side

      Output

      ----- got message -----
message-for-both-pulsar-and-kafka-client

    • From Kafka consumer side

      Output

      > message-for-both-pulsar-and-kafka-client

  4. Start a Pulsar producer and send the message message-from-pulsar-producer.

    pulsar-client producer -m "message-from-pulsar-producer" <topic_name>

    At the same time, you can receive messages from both Pulsar and Kafka consumers.

    • From Pulsar consumer side

      Output

      ----- got message -----
message-from-pulsar-producer

    • From Kafka consumer side

      Output

      > message-from-pulsar-producer

Step 6: Use StreamNative Console to manage Pulsar cluster

The StreamNative Console is a web-based GUI management tool for managing and monitoring Pulsar.

This section describes how to manage a Pulsar cluster, including creating and managing tenants, namespaces, and topics.

  1. Log in to the StreamNative Console. For details, see log in to StreamNative Console.

  2. Create a tenant. For a full list of operations available for tenants on the StreamNative Console, see create a tenant.

    a. On the left navigation pane, in the Admin section, click Tenants/Namespaces.

    b. On the Tenants page, click New Tenant. A dialog box displays.

    c. Configure the tenant and then click Confirm.

  3. Create a namespace. For a full list of operations available for namespaces on the StreamNative Console, see create a namespace.

    a. On the left navigation pane, under ADMIN, click Tenants/Namespaces.

    b. Select the name of the tenant you want to associate with the new namespace, and click New Namespace. A dialog box displays.

    c. Enter a name for the namespace and then click Confirm. The namespace name is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).

  4. Create a topic. For a full list of operations available for topics on the StreamNative Console, see create a topic.

    a. On the left navigation pane, under RESOURCES, click Topics.

    b. Click New Topic and a dialog box displays.

    c. Configure the topic and then click Confirm.

Step 7: Use Prometheus and Grafana to monitor status

This section describes how to use the Prometheus and Grafana to monitor Pulsar status.

Prometheus

Prometheus is an open-source system monitoring and alerting toolkit, which is used to collect and store Pulsar related metrics.

  1. Expose Prometheus service.

    Before accessing the Prometheus website, you need to expose Prometheus service.

    kubectl expose service <prometheus_service_name> --type=LoadBalancer --name=<prometheus_name> --port=9090 -n <k8s_namespace>

  2. Get the external IP address of Prometheus service.

    kubectl get service -n <k8s_namespace>

    Output

    sn-prometheus is exposed and the external IP address is {PROM-EXTERNAL-IP}.

    NAME                                        TYPE           CLUSTER-IP     EXTERNAL-IP        PORT
sn-prometheus                               LoadBalancer   10.12.3.133    {PROM-EXTERNAL-IP} 9090:32556/TCP               9s

  3. Navigate to the Prometheus website at http://{PROM-EXTERNAL-IP}:9090/targets.

    Then you can use Prometheus to check metrics of all Pulsar components.

Grafana

Apache Pulsar Grafana dashboard is an open-source visualization tool, containing a unique Graphite target parser that enables easy metric and function editing, which is used to visualize time series data of different monitoring indexes.

  1. Expose Grafana service.

    Before accessing Grafana, you need to expose Grafana service.

    kubectl expose service <grafana_service_name> --type=LoadBalancer --name=<grafana> --port=3000 -n <k8s_namespace>

  2. Get the external IP address of Grafana service.

    kubectl get service -n <k8s_namespace>

    Output

    sn-grafana is exposed and the external IP address is {GRAFANA-EXTERNAL-IP}.

    NAME                                        TYPE           CLUSTER-IP     EXTERNAL-IP            PORT(S)                      AGE
sn-grafana                                  LoadBalancer   10.12.5.204    {GRAFANA-EXTERNAL-IP}  3000:30307/TCP               5s

  3. Navigate to the Grafana website at http://{GRAFANA-EXTERNAL-IP}:3000 and log into using the default credentials as below.

    • Account: pulsar
    • Password: pulsar

    Now you can see the Grafana dashboard showing detailed metrics of components, such as bookie, JVM, messaging, node, proxy, Zookeeper, pulsar topics and so on.

Step 8: Uninstallation

This section describes how to uninstall Pulsar cluster and StreamNative Platform.

  • Execute the following command to uninstall the Pulsar cluster.

    helm uninstall streamnative/sn-platform -n <k8s_namespace>

  • Execute the following command to uninstall the StreamNative Platform.

    helm uninstall vault-operator -n <k8s_namespace>
helm uninstall cert-manager -n <k8s_namespace>
helm uninstall sn-operator -n <k8s_namespace>
helm uninstall function-mesh-operator -n <k8s_namespace>

Note

If you want to delete the PVCs or the Secret, you need to delete them together. Otherwise, you will fail to reinstall the StreamNative Platform. It is recommended that you exercise caution when deleting the PVCs or the Secret because some of your significant information cannot be restored once you have deleted them.

Previous
Overview
Next
Plan for Deployment