Deploy Pulsar cluster

This document describes how to deploy BookKeeper clusters, ZooKeeper clusters, Pulsar brokers, and Pulsar proxies using the Pulsar Operator for a Pulsar cluster.

The Pulsar Operator provides full lifecycle management for all the resources within a Pulsar cluster. It can set up, upgrade, as well as scale up and down a cluster. In addition, it can use CRDs for provisioning resources, such as tenants and topics, within a Pulsar cluster.

In general, a Pulsar cluster consists of the following components:

One or more Pulsar brokers that handle and balance messages from producers, dispatch messages to consumers, communicate with the Pulsar configuration store to handle various coordination tasks, etc.
A BookKeeper cluster that consists of one or more bookies handling persistent storage of messages.
A ZooKeeper cluster that handles coordination tasks between Pulsar clusters.

Sometimes, Pulsar proxy is used when direct connections between clients and Pulsar brokers are either unfeasible or undesirable.

Prerequisites

Install kubectl (v1.16 or higher), compatible with your cluster (+/- 1 minor release from your cluster).
Install Helm (v3.0.2 or higher).
Prepare a Kubernetes cluster (v1.16 or higher).
Install the Pulsar Operator.
Preload images to your local directory from StreamNative Docker Hub.

Steps

Add the StreamNative chart repository.

helm repo add streamnative https://charts.streamnative.io
helm repo update

Add the environment variables for the Pulsar Chart directory, Pulsar cluster name, and Kubernetes namespace.

# If you use the offline version, export PULSAR_CHART first.
export PULSAR_CHART=/path/to/pulsar/chart
# Define your Pulsar cluster name
export RELEASE_NAME=<release_name>
# Define the K8s namespace to install the pulsar cluster
export NAMESPACE=<k8s_namespace>

Create a Kubernetes namespace for your Pulsar cluster.
```
kubectl create namespace $NAMESPACE
```

Deploy a ZooKeeper cluster.

a. Define a ZooKeeper cluster using a YAML file.

This example creates a ZooKeeper cluster named test-zookeeper in the test Kubernetes namespace and saves the YAML file as zookeeper-sample.yaml.

apiVersion: zookeeper.streamnative.io/v1alpha1
kind: ZooKeeperCluster
metadata:
  name: test-zookeeper
  namespace: test
spec:
  image: streamnative/pulsar:2.8.0.9
  pod:
    resources:
      requests:
        cpu: 50m
        memory: 256Mi
  persistence:
    data:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 8Gi
    dataLog:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 2Gi
    reclaimPolicy: Delete
  replicas: 3

If you want faster performance, you should use the solid-state drive (SSD) equivalent storage class for data logs in your Kubernetes namespace.

b. Apply the YAML file to create the ZooKeeper cluster.

kubectl apply -f /path/to/zookeeper-sample.yaml

c. Check whether the ZooKeeper cluster is created successfully.

kubectl get all -n test

Deploy a BookKeeper cluster.

a. Define a Pulsar BookKeeper cluster using a YAML file.

This example creates a BookKeeper cluster named test-bookie in the test Kubernetes namespace and saves the YAML file as bookie-sample.yaml.

apiVersion: bookkeeper.streamnative.io/v1alpha1
kind: BookKeeperCluster
metadata:
  name: test-bookie
  namespace: test
spec:
  image: streamnative/pulsar:2.8.0.9
  replicas: 3
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
  storage:
    journal:
      numDirsPerVolume: 1
      numVolumes: 1
      volumeClaimTemplate:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 8Gi
    ledger:
      numDirsPerVolume: 1
      numVolumes: 1
      volumeClaimTemplate:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 16Gi
    reclaimPolicy: Delete
  zkServers: test-zookeeper:2181

If you want faster performance, you should use the SSD equivalent storage class for data logs in your Kubernetes namespace.

b. Apply the YAML file to create the BookKeeper cluster.

kubectl apply -f /path/to/bookie-sample.yaml

c. Check whether the BookKeeper cluster is created successfully.

kubectl get all -n test

Deploy a Pulsar broker.

a. Define a Pulsar broker using a YAML file.

This example creates a Pulsar broker named test-broker in the test Kubernetes namespace and saves the YAML file as broker-sample.yaml.

apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarBroker
metadata:
  name: test-broker
  namespace: test
spec:
  image: streamnative/pulsar:2.8.0.9
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
    serviceAccountName: broker
    terminationGracePeriodSeconds: 30
  replicas: 1
  zkServers: test-zookeeper:2181

b. Apply the YAML file to create the Pulsar broker.

kubectl apply -f /path/to/broker-sample.yaml

c. Check whether the Pulsar broker is created successfully.

kubectl get all -n test

Deploy a Pulsar Proxy.

a. Define a Pulsar proxy using a YAML file.

This example creates a Pulsar proxy named test-proxy in the test Kubernetes namespace and saves the YAML file as proxy-sample.yaml.

apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarProxy
metadata:
  name: test-proxy
  namespace: test
spec:
  brokerAddress: broker.test.svc.cluster.local
  dnsNames: []
  image: streamnative/pulsar:2.8.0.9
  issuerRef:
    name: ''
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 256Mi
  replicas: 1

b. Apply the YAML file to create the Pulsar proxy.

kubectl apply -f /path/to/proxy-sample.yaml

c. Check whether the Pulsar proxy is created successfully.

kubectl get all -n test