1. How-to guides

Manually scale a Pulsar cluster using the Pulsar Operators

When the traffic handled by your Pulsar cluster increases or decreases, you can scale out or scale in the cluster accordingly. Depending on the specific use case, you may choose to increase or decrease the number of nodes for different components, including Pulsar brokers, Pulsar proxies, bookies, and ZooKeeper.

When resizing the Pulsar cluster on Kubernetes created by the Pulsar Operators, you need to perform the scaling action on Custom Resources (CRs) instead of the individual workloads deployed through them. For example, when deploying Pulsar broker workloads, the Pulsar Operators create a broker StatefulSet with 2 Pod replicas by default. To scale the StatefulSet, you should adjust the broker CR instead of manually changing the StatefulSet.

This document describes how to manually scale Pulsar components on Kubernetes.

Prerequisites

You have deployed a Pulsar cluster using the Pulsar Operators.

Scale Pulsar brokers

If you use the default broker CR, you should have 2 broker replicas running in the pulsar namespace. This example scales the number of Pods from 2 to 5.

  1. View the existing broker CR.

    kubectl get pp -n pulsar
    

    You should see the following output:

    NAME     REPLICAS   READY REPLICAS   DESIRED IMAGE                            ENDPOINT   AGE
    brokers  2          2                streamnative/sn-platform-slim:2.10.3.4              10m
    
  2. Run the following command to scale brokers.

    kubectl scale pb brokers --replicas=5 -n pulsar
    
  3. View the newly-created Pods.

    kubectl get pods -n pulsar
    

    You should see the following output:

    NAME                         READY   STATUS    RESTARTS   AGE
    bookies-bk-0                 1/1     Running   0          10m
    bookies-bk-1                 1/1     Running   0          10m
    bookies-bk-2                 1/1     Running   0          10m
    bookies-bk-auto-recovery-0   1/1     Running   0          10m
    brokers-broker-0             1/1     Running   0          10m
    brokers-broker-1             1/1     Running   0          10m
    brokers-broker-2             1/1     Running   0          10s
    brokers-broker-3             1/1     Running   0          10s
    brokers-broker-4             1/1     Running   0          10s
    proxys-proxy-0               1/1     Running   0          10m
    proxys-proxy-1               1/1     Running   0          10m
    zookeepers-zk-0              1/1     Running   0          10m
    zookeepers-zk-1              1/1     Running   0          10m
    zookeepers-zk-2              1/1     Running   0          10m
    
  4. Alternatively, you can create a broker YAML file with the following configuration and use the kubectl apply -f command to apply the file.

    apiVersion: pulsar.streamnative.io/v1alpha1
    kind: PulsarBroker
    metadata:
    name: brokers
    namespace: pulsar
    spec:
    image: streamnative/sn-platform-slim:2.10.3.4
    pod:
      resources:
        requests:
        cpu: 200m
        memory: 512Mi
      securityContext:
        runAsNonRoot: true
    replicas: 5 # Change the replica count
    zkServers: zookeepers-zk:2181
    

Scale Pulsar proxies

If you use the default proxy CR, you should have 2 proxy replicas running in the pulsar namespace. This example scales the number of Pods from 2 to 5.

  1. View the existing proxy CR.

    kubectl get pp -n pulsar
    

    You should see the following output:

    NAME     REPLICAS   READY REPLICAS   DESIRED IMAGE                            ENDPOINT   AGE
    proxys   2          2                streamnative/sn-platform-slim:2.10.3.4              10m
    
  2. Run the following command to scale Pulsar proxies.

    kubectl scale pp proxys --replicas=5 -n pulsar
    
  3. View the newly-created Pods.

    kubectl get pods -n pulsar
    

    You should see the following output:

    NAME                         READY   STATUS    RESTARTS   AGE
    bookies-bk-0                 1/1     Running   0          10m
    bookies-bk-1                 1/1     Running   0          10m
    bookies-bk-2                 1/1     Running   0          10m
    bookies-bk-auto-recovery-0   1/1     Running   0          10m
    brokers-broker-0             1/1     Running   0          10m
    brokers-broker-1             1/1     Running   0          10m
    proxys-proxy-0               1/1     Running   0          10m
    proxys-proxy-1               1/1     Running   0          10m
    proxys-proxy-2               1/1     Running   0          42s
    proxys-proxy-3               1/1     Running   0          42s
    proxys-proxy-4               1/1     Running   0          42s
    zookeepers-zk-0              1/1     Running   0          10m
    zookeepers-zk-1              1/1     Running   0          10m
    zookeepers-zk-2              1/1     Running   0          10m
    
  4. Alternatively, you can create a proxy YAML file with the following configuration and use the kubectl apply -f command to apply the file.

    apiVersion: pulsar.streamnative.io/v1alpha1
    kind: PulsarProxy
    metadata:
    name: proxys
    namespace: pulsar
    spec:
    image: 'streamnative/sn-platform-slim:2.10.3.4'
    pod:
      resources:
        requests:
        cpu: 200m
        memory: 512Mi
      securityContext:
        runAsNonRoot: true
    brokerAddress: brokers-broker
    replicas: 5 # Change the replica count
    config:
      tls:
        enabled: false
    dnsNames: []
    issuerRef:
      name: ''
    

Scale bookies

If you use the default BookKeeper CR, you should have 3 bookie replicas running in the pulsar namespace. This example scales the number of bookie Pods from 3 to 7.

  1. View the existing BookKeeper CR.

    kubectl get pp -n pulsar
    

    You should see the following output:

    NAME      REPLICAS   READY REPLICAS   ZKSERVERS            DESIRED IMAGE                            AGE
    bookies   3          3                zookeepers-zk:2181   streamnative/sn-platform-slim:2.10.3.4   13m
    
  2. Run the following command to scale bookies.

    kubectl scale bk bookies --replicas=7 -n pulsar
    
  3. View the newly-created Pods.

    kubectl get pods -n pulsar
    

    You should see the following output:

    NAME                         READY   STATUS    RESTARTS   AGE
    bookies-bk-0                 1/1     Running   0          10m
    bookies-bk-1                 1/1     Running   0          10m
    bookies-bk-2                 1/1     Running   0          10m
    bookies-bk-3                 1/1     Running   0          40s
    bookies-bk-4                 1/1     Running   0          40s
    bookies-bk-5                 1/1     Running   0          40s
    bookies-bk-6                 1/1     Running   0          40s
    bookies-bk-auto-recovery-0   1/1     Running   0          10m
    brokers-broker-0             1/1     Running   0          10m
    brokers-broker-1             1/1     Running   0          10m
    proxys-proxy-0               1/1     Running   0          10m
    proxys-proxy-1               1/1     Running   0          10m
    zookeepers-zk-0              1/1     Running   0          10m
    zookeepers-zk-1              1/1     Running   0          10m
    zookeepers-zk-2              1/1     Running   0          10m
    
  4. Alternatively, you can create a BookKeeper YAML file with the following configuration and use the kubectl apply -f command to apply the file.

    apiVersion: bookkeeper.streamnative.io/v1alpha1
    kind: BookKeeperCluster
    metadata:
    name: bookies
    namespace: pulsar
    spec:
    image: streamnative/sn-platform-slim:2.10.3.4
    replicas: 7 # Change the replica count
    pod:
      resources:
        requests:
        cpu: 200m
        memory: 512Mi
      securityContext:
        runAsNonRoot: true
    storage:
      journal:
        numDirsPerVolume: 1
        numVolumes: 1
        volumeClaimTemplate:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 8Gi
      ledger:
        numDirsPerVolume: 1
        numVolumes: 1
        volumeClaimTemplate:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 16Gi
      reclaimPolicy: Delete
    zkServers: zookeepers-zk:2181
    

Scale ZooKeeper

If you use the default ZooKeeper CR, you should have 3 ZooKeeper replicas running in the pulsar namespace. This example scales the number of Pods from 3 to 5.

  1. View the existing ZooKeeper CR.

    kubectl get zk -n pulsar
    

    You should see the following output:

    NAME         REPLICAS   READY REPLICAS   DESIRED IMAGE                            AGE
    zookeepers   3          3                streamnative/sn-platform-slim:2.10.3.4   16m
    
  2. Run the following command to scale ZooKeeper.

    kubectl scale zk zookeepers --replicas=5 -n pulsar
    
  3. View the newly-created Pods.

    kubectl get pods -n pulsar
    

    You should see the following output:

    NAME                         READY   STATUS    RESTARTS   AGE
    bookies-bk-0                 1/1     Running   0          10m
    bookies-bk-1                 1/1     Running   0          10m
    bookies-bk-2                 1/1     Running   0          10m
    bookies-bk-auto-recovery-0   1/1     Running   0          10m
    brokers-broker-0             1/1     Running   0          10m
    brokers-broker-1             1/1     Running   0          10m
    proxys-proxy-0               1/1     Running   0          10m
    proxys-proxy-1               1/1     Running   0          10m
    zookeepers-zk-0              1/1     Running   0          54s
    zookeepers-zk-1              1/1     Running   0          31s
    zookeepers-zk-2              1/1     Running   0          71s
    zookeepers-zk-3              1/1     Running   0          98s
    zookeepers-zk-4              1/1     Running   0          98s
    
  4. Alternatively, you can create a ZooKeeper YAML file with the following configuration and use the kubectl apply -f command to apply the file.

    apiVersion: zookeeper.streamnative.io/v1alpha1
    kind: ZooKeeperCluster
    metadata:
    name: zookeepers
    namespace: pulsar
    spec:
    image: streamnative/sn-platform-slim:2.10.3.4
    pod:
      resources:
        requests:
        cpu: 50m
        memory: 256Mi
      securityContext:
        runAsNonRoot: true
    persistence:
      data:
        accessModes:
          - ReadWriteOnce
        resources:
        requests:
          storage: 8Gi
      dataLog:
        accessModes:
          - ReadWriteOnce
        resources:
        requests:
          storage: 2Gi
      reclaimPolicy: Delete
    replicas: 5 # Change the replica count
    
Previous
Deploy a Pulsar cluster