1. StreamNative Platform

Get Started

This document guides you through every step of installing and running StreamNative Platform with operators on Kubernetes quickly. It includes the following tasks:

  • Install the StreamNative Platform on Kubernetes using operators.
  • Start and stop StreamNative Platform.
  • Create tenants, namespaces, and topics using the pulsarctl CLI tool.
  • Produce and consume events using pulsar-client.
  • Produce and consume events using Kafka client.
  • Verify interoperability between Pulsar and Kafka.
  • Monitor StreamNative Platform status with Prometheus and Grafana.

Prerequisites

  • Kubernetes server v1.16 or higher
  • Install kubectl v1.16 or higher.
  • Install Helm 3.0 or higher.
  • Install pulsarctl 2.8.0 or higher.
  • Deploy a Kubernetes cluster.

Step 1: Install StreamNative Platform

Note

This step uses the original images within the operators. If you want to use customized images to install StreamNative Platform, see install StreamNative Platform using customized image.

  1. Install the StreamNative repositories.

    helm repo add streamnative https://charts.streamnative.io
    helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
    helm repo add jetstack https://charts.jetstack.io
    helm repo add function-mesh http://charts.functionmesh.io/
    helm repo update
    
  2. Create a Kubernetes namespace.

    kubectl create namespace <k8s_namespace>
    
  3. Install the Vault operator.

    The Vault operator creates and maintains highly-available Vault clusters on Kubernetes, allowing users to easily deploy and manage Vault clusters for their applications.

    helm upgrade --install vault-operator banzaicloud-stable/vault-operator -n <k8s_namespace>
    
  4. Install the cert-manager.

    The cert-manager is a native Kubernetes certificate management controller. It helps issue certificates from HashiCorp Vault. The cert-manager ensures that certificates are valid and up-to-date, and attempts to renew certificates at a configured time before expiry.

    The cert-manager requires a number of CRD resources to be installed into your cluster as part of installation. To automatically install and manage the CRDs as part of your Helm release, you must add the --set installCRDs=true flag to your Helm installation command.

    helm upgrade --install cert-manager jetstack/cert-manager -n <k8s_namespace> --set installCRDs=true
    
  5. Install the Pulsar operator.

    The Pulsar operator is used to manage Pulsar components, including the Pulsar broker, BookKeeper, ZooKeeper, and Pulsar proxy.

    helm upgrade --install pulsar-operator streamnative/pulsar-operator -n <k8s_namespace>
    
  6. Install the Function Mesh operator.

    The Function Mesh operator is used to configure and manage Pulsar IO connectors and Pulsar Functions.

    Function Mesh is a serverless and purpose-built framework for orchestrating multiple Pulsar Functions and Pulsar IO connectors for stream processing applications.

    helm upgrade --install function-mesh-operator function-mesh/function-mesh-operator -n <k8s_namespace>
    
  7. Deploy a Pulsar cluster.

    a. Define a Pulsar cluster configuration file.

    b. Deploy the Pulsar cluster using the YAML file.

    helm install <release_name> streamnative/sn-platform --set initialize=true -n <namespace>
    

Step 2: Create Pulsar tenants/namespaces/topics

pulsarctl is a Command-Line Interface (CLI) tool for Pulsar. In this section, you can use the pulsarctl CLI tool to create tenants, namespaces, and topics.

  1. Log in to pulsarctl and create a tenant.

    pulsarctl \
    --admin-service-url <WEB_SERVICE_URL> \
    --token <AUTH_PARAMS> \
    tenants create <tenant_name>
    

    Replace WEB_SERVICE_URL with the Web service URL of your Pulsar cluster. Replace AUTH_PARAMS with the token that you get from StreamNative Console. For details, see the prepare to connect to a Pulsar cluster user guide.

  2. Create a namespace.

    pulsarctl namespaces create <namespace_name> -c <cluster_name>
    
  3. Create a topic.

    pulsarctl topics create <topic_name>
    
  4. List all the topics.

    pulsarctl topics list <namespace_name>
    

Step 3: Use pulsar-client to produce and consume events

StreamNative Platform supports all the official Pulsar clients. You can use the pulsar-client CLI tool to create producers and consumers to simulate a simple production and consumption model.

  1. Enter the toolset Pod.

    To facilitate the usage of the official Pulsar CLI tools, such as pulsar-client, StreamNative Platform provides a toolset Pod, which you can use to execute the kubectl exec command to connect to the official Pulsar CLI tools.

    kubectl exec -n <k8s_namespace> -it <release_name>-sn-platform-toolset-0 -- bash
    
  2. Create a consumer.

    pulsar-client consume -s <subscription_name> <topic_name> -n 0
    
  3. Create a producer.

    pulsar-client produce TOPIC_NAME  -m "---------hello streamnative platform-------" -n 10
    
    • From the producer side, you can see that the messages have been produced successfully.

      23:04:25.652 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 10 messages successfully produced
      
    • From the consumer side, you can receive the following messages.

      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      ----- got message -----
      ---------hello streamnative platform-------
      

Step 4: Use Kafka client to produce and consume events

StreamNative Platform brings native Kafka protocol support to Pulsar brokers using Kafka on Pulsar (KoP). Therefore, you can migrate your existing Kafka applications and services to Apache Pulsar without modifying the codes.

Currently, StreamNative Platform supports Kafka Client v1.0.0 - v2.6.0.

  1. Grant produce and consume permission to the Admin role on the namespace.

    pulsarctl namespaces grant-permission --role <role_name> --actions produce,consume <namespace_name>
    
  2. Run the Kafka image.

    kubectl run kafka --rm -it -n <k8s_namespace> --image bitnami/kafka -- bash
    
  3. Start a Kafka producer and send a message from the Kafka producer.

    kafka-console-producer.sh \
    --producer-property security.protocol=SASL_PLAINTEXT \
    --producer-property sasl.mechanism=PLAIN \
    --producer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="TENANT/NAMESPACE" password="token:'YOUR_TOKEN'";' \
    --broker-list <release_name>-sn-platform-broker:9092 --topic <topic_name>
    

    Here are security-related options to be configured.

    OptionDescriptionDefault
    usernameThe username for the Kafka client connecting to the Pulsar cluster. It is set to the name of the Pulsar tenant and namespace (TENANT_NAME/NAMESPACE_NAME) where Kafka topics are stored.public/default
    passwordThe password for the Kafka client connecting to the Pulsar cluster. It is set to the token that you get from StreamNative Console. For details, see the prepare to connect to a Pulsar cluster user guide.N/A
  4. Open a new terminal window and enter the Kafka Pod.

    kubectl exec -it -n <k8s_namespace> kafka -- bash
    
  5. Start a Kafka consumer.

    kafka-console-consumer.sh \
    --consumer-property security.protocol=SASL_PLAINTEXT \
    --consumer-property sasl.mechanism=PLAIN \
    --consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="TENANT/NAMESPACE" password="token:'YOUR_TOKEN'";' \
    --bootstrap-server sn-platform-broker:9092 --topic <topic_name>
    

Step 5: Verify interoperability between Pulsar and Kafka

As shown in Step 3 and Step 4, Pulsar producer, Pulsar consumer, Kafka producer, and Kafka consumer run normally. The Pulsar consumer can get the message from the Pulsar producer and the Kafka consumer can receive the message from the Kafka producer. This section further verifies the interoperability between Pulsar and Kafka.

  1. Start a Pulsar consumer.

    a. Enter the toolset Pod.

    kubectl exec -n <k8s_namespace> -it <release_name>-sn-platform-toolset-0 -- bash
    

    b. Create a Pulsar consumer.

    pulsar-client consume -s <subscription_name> <topic_name>  -n 0
    
  2. Start a Kafka consumer.

    a. Enter the Kafka Pod.

    kubectl exec -it -n <k8s_namespace> kafka -- bash
    

    b. Start a Kafka consumer.

    kafka-console-consumer.sh \
    --consumer-property security.protocol=SASL_PLAINTEXT \
    --consumer-property sasl.mechanism=PLAIN \
    --consumer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="TENANT/NAMESPACE" password="token:'YOUR_TOKEN'";' \
    --bootstrap-server sn-platform-broker:9092 --topic <topic_name>
    
  3. Start a Kafka producer.

    a. Enter the Kafka Pod.

    kubectl exec -it -n <k8s_namespace> kafka -- bash
    

    b. Start a Kafka producer and send a message.

    kafka-console-producer.sh \
    --producer-property security.protocol=SASL_PLAINTEXT \
    --producer-property sasl.mechanism=PLAIN \
    --producer-property 'sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="TENANT/NAMESPACE" password="token:'YOUR_TOKEN'";' \
    --broker-list <release_name>-sn-platform-broker:9092 --topic <topic_name>
    > message-for-both-pulsar-and-kafka-client
    

    At the same time, you can receive messages from both Pulsar and Kafka consumers.

    • From Pulsar consumer side

      Output

      ----- got message -----
      message-for-both-pulsar-and-kafka-client
      
    • From Kafka consumer side

      Output

      > message-for-both-pulsar-and-kafka-client
      
  4. Start a Pulsar producer and send the message message-from-pulsar-producer.

    pulsar-client producer -m "message-from-pulsar-producer" <topic_name>
    

    At the same time, you can receive messages from both Pulsar and Kafka consumers.

    • From Pulsar consumer side

      Output

      ----- got message -----
      message-from-pulsar-producer
      
    • From Kafka consumer side

      Output

      > message-from-pulsar-producer
      

Step 6: Use StreamNative Console to manage Pulsar cluster

The StreamNative Console is a web-based GUI management tool for managing and monitoring Pulsar.

This section describes how to manage a Pulsar cluster, including creating and managing tenants, namespaces, and topics.

  1. Log in to the StreamNative Console. For details, see here.

  2. Create a tenant. For a full list of operations available for tenants on the StreamNative Console, see here.

    a. On the left navigation pane, under TENANT/NAMESPACE, select the current default tenant/namespace, and then click NEW TENANT.

    b. On the Tenants page, click New Tenant. A dialog box displays.

    c. Configure the tenant and then click Confirm.

  3. Create a namespace. For a full list of operations available for namespaces on the StreamNative Console, see here.

    a. On the left navigation pane, under ADMIN, click Tenants/Namespaces.

    b. Select the name of the tenant you want to associate with the new namespace, and click New Namespace. A dialog box displays.

    c. Enter a name for the namespace and then click Confirm. The namespace name is a string of up to 40 characters, supporting lowercase letters (a-z), numeric characters (0-9), and the special character hyphen (-).

  4. Create a topic. For a full list of operations available for topics on the StreamNative Console, see here.

    a. On the left navigation pane, under RESOURCES, click Topics.

    b. Click New Topic and a dialog box displays.

    c. Configure the topic and then click Confirm.

Step 7: Use Prometheus and Grafana to monitor status

This section describes how to use the Prometheus and Grafana to monitor Pulsar status.

Prometheus

Prometheus is an open-source system monitoring and alerting toolkit, which is used to collect and store Pulsar related metrics.

  1. Expose Prometheus service.

    Before accessing the Prometheus website, you need to expose Prometheus service.

    kubectl expose service <prometheus_service_name> --type=LoadBalancer --name=<prometheus_name> --port=9090 -n <k8s_namespace>
    
  2. Get the external IP address of Prometheus service.

    kubectl get service -n <k8s_namespace>
    

    Output

    sn-prometheus is exposed and the external IP address is {PROM-EXTERNAL-IP}.

    NAME                                        TYPE           CLUSTER-IP     EXTERNAL-IP        PORT
    sn-prometheus                               LoadBalancer   10.12.3.133    {PROM-EXTERNAL-IP} 9090:32556/TCP               9s
    
  3. Navigate to the Prometheus website at http://{PROM-EXTERNAL-IP}:9090/targets.

    Then you can use Prometheus to check metrics of all Pulsar components.

Grafana

Apache Pulsar Grafana dashboard is an open-source visualization tool, containing a unique Graphite target parser that enables easy metric and function editing, which is used to visualize time series data of different monitoring indexes.

  1. Expose Grafana service.

    Before accessing Grafana, you need to expose Grafana service.

    kubectl expose service <grafana_service_name> --type=LoadBalancer --name=<grafana> --port=3000 -n <k8s_namespace>
    
  2. Get the external IP address of Grafana service.

    kubectl get service -n <k8s_namespace>
    

    Output

    sn-grafana is exposed and the external IP address is {GRAFANA-EXTERNAL-IP}.

    NAME                                        TYPE           CLUSTER-IP     EXTERNAL-IP            PORT(S)                      AGE
    sn-grafana                                  LoadBalancer   10.12.5.204    {GRAFANA-EXTERNAL-IP}  3000:30307/TCP               5s
    
  3. Navigate to the Grafana website at http://{GRAFANA-EXTERNAL-IP}:3000 and log into using the default credentials as below.

    • Account: pulsar
    • Password: pulsar

    Now you can see the Grafana dashboard showing detailed metrics of components, such as bookie, JVM, messaging, node, proxy, Zookeeper, pulsar topics and so on.

Step 8: Uninstallation

This section describes how to uninstall Pulsar cluster and StreamNative Platform.

  • Execute the following command to uninstall the Pulsar cluster.

    helm uninstall streamnative/sn-platform -n <k8s_namespace>
    
  • Execute the following command to uninstall the StreamNative Platform.

    helm uninstall vault-operator -n <k8s_namespace>
    helm uninstall cert-manager -n <k8s_namespace>
    helm uninstall pulsar-operator -n <k8s_namespace>
    helm uninstall function-mesh-operator -n <k8s_namespace>
    

Note

If you want to delete the PVCs or the Secret, you need to delete them together. Otherwise, you will fail to reinstall the StreamNative Platform. It is recommended that you exercise caution when deleting the PVCs or the Secret because some of your significant information cannot be restored once you have deleted them.

Previous
Overview