- How-to guides
Manually scale a Pulsar cluster using the Pulsar Operators
When the traffic handled by your Pulsar cluster increases or decreases, you can scale out or scale in the cluster accordingly. Depending on the specific use case, you may choose to increase or decrease the number of nodes for different components, including Pulsar brokers, Pulsar proxies, bookies, and ZooKeeper.
When resizing the Pulsar cluster on Kubernetes created by the Pulsar Operators, you need to perform the scaling action on Custom Resources (CRs) instead of the individual workloads deployed through them. For example, when deploying Pulsar broker workloads, the Pulsar Operators create a broker StatefulSet with 2 Pod replicas by default. To scale the StatefulSet, you should adjust the broker CR instead of manually changing the StatefulSet.
This document describes how to manually scale Pulsar components on Kubernetes.
Prerequisites
You have deployed a Pulsar cluster using the Pulsar Operators.
Scale Pulsar brokers
If you use the default broker CR, you should have 2 broker replicas running in the pulsar
namespace. This example scales the number of Pods from 2 to 5.
View the existing broker CR.
kubectl get pb -n pulsar
You should see the following output:
NAME REPLICAS READY REPLICAS DESIRED IMAGE ENDPOINT AGE brokers 2 2 streamnative/sn-platform-slim:2.10.3.4 10m
Run the following command to scale brokers.
kubectl scale pb brokers --replicas=5 -n pulsar
View the newly-created Pods.
kubectl get pods -n pulsar
You should see the following output:
NAME READY STATUS RESTARTS AGE bookies-bk-0 1/1 Running 0 10m bookies-bk-1 1/1 Running 0 10m bookies-bk-2 1/1 Running 0 10m bookies-bk-auto-recovery-0 1/1 Running 0 10m brokers-broker-0 1/1 Running 0 10m brokers-broker-1 1/1 Running 0 10m brokers-broker-2 1/1 Running 0 10s brokers-broker-3 1/1 Running 0 10s brokers-broker-4 1/1 Running 0 10s proxys-proxy-0 1/1 Running 0 10m proxys-proxy-1 1/1 Running 0 10m zookeepers-zk-0 1/1 Running 0 10m zookeepers-zk-1 1/1 Running 0 10m zookeepers-zk-2 1/1 Running 0 10m
Alternatively, you can create a broker YAML file with the following configuration and use the
kubectl apply -f
command to apply the file.apiVersion: pulsar.streamnative.io/v1alpha1 kind: PulsarBroker metadata: name: brokers namespace: pulsar spec: image: streamnative/sn-platform-slim:2.10.3.4 pod: resources: requests: cpu: 200m memory: 512Mi securityContext: runAsNonRoot: true replicas: 5 # Change the replica count zkServers: zookeepers-zk:2181
Scale Pulsar proxies
If you use the default proxy CR, you should have 2 proxy replicas running in the pulsar
namespace. This example scales the number of Pods from 2 to 5.
View the existing proxy CR.
kubectl get pp -n pulsar
You should see the following output:
NAME REPLICAS READY REPLICAS DESIRED IMAGE ENDPOINT AGE proxys 2 2 streamnative/sn-platform-slim:2.10.3.4 10m
Run the following command to scale Pulsar proxies.
kubectl scale pp proxys --replicas=5 -n pulsar
View the newly created Pods.
kubectl get pods -n pulsar
You should see the following output:
NAME READY STATUS RESTARTS AGE bookies-bk-0 1/1 Running 0 10m bookies-bk-1 1/1 Running 0 10m bookies-bk-2 1/1 Running 0 10m bookies-bk-auto-recovery-0 1/1 Running 0 10m brokers-broker-0 1/1 Running 0 10m brokers-broker-1 1/1 Running 0 10m proxys-proxy-0 1/1 Running 0 10m proxys-proxy-1 1/1 Running 0 10m proxys-proxy-2 1/1 Running 0 42s proxys-proxy-3 1/1 Running 0 42s proxys-proxy-4 1/1 Running 0 42s zookeepers-zk-0 1/1 Running 0 10m zookeepers-zk-1 1/1 Running 0 10m zookeepers-zk-2 1/1 Running 0 10m
Alternatively, you can create a proxy YAML file with the following configuration and use the
kubectl apply -f
command to apply the file.apiVersion: pulsar.streamnative.io/v1alpha1 kind: PulsarProxy metadata: name: proxys namespace: pulsar spec: image: 'streamnative/sn-platform-slim:2.10.3.4' pod: resources: requests: cpu: 200m memory: 512Mi securityContext: runAsNonRoot: true brokerAddress: brokers-broker replicas: 5 # Change the replica count config: tls: enabled: false dnsNames: [] issuerRef: name: ''
Scale bookies
If you use the default BookKeeper CR, you should have 3 bookie replicas running in the pulsar
namespace. This example scales the number of bookie Pods from 3 to 7.
View the existing BookKeeper CR.
kubectl get bk -n pulsar
You should see the following output:
NAME REPLICAS READY REPLICAS ZKSERVERS DESIRED IMAGE AGE bookies 3 3 zookeepers-zk:2181 streamnative/sn-platform-slim:2.10.3.4 13m
Run the following command to scale bookies.
kubectl scale bk bookies --replicas=7 -n pulsar
View the newly-created Pods.
kubectl get pods -n pulsar
You should see the following output:
NAME READY STATUS RESTARTS AGE bookies-bk-0 1/1 Running 0 10m bookies-bk-1 1/1 Running 0 10m bookies-bk-2 1/1 Running 0 10m bookies-bk-3 1/1 Running 0 40s bookies-bk-4 1/1 Running 0 40s bookies-bk-5 1/1 Running 0 40s bookies-bk-6 1/1 Running 0 40s bookies-bk-auto-recovery-0 1/1 Running 0 10m brokers-broker-0 1/1 Running 0 10m brokers-broker-1 1/1 Running 0 10m proxys-proxy-0 1/1 Running 0 10m proxys-proxy-1 1/1 Running 0 10m zookeepers-zk-0 1/1 Running 0 10m zookeepers-zk-1 1/1 Running 0 10m zookeepers-zk-2 1/1 Running 0 10m
Alternatively, you can create a BookKeeper YAML file with the following configuration and use the
kubectl apply -f
command to apply the file.apiVersion: bookkeeper.streamnative.io/v1alpha1 kind: BookKeeperCluster metadata: name: bookies namespace: pulsar spec: image: streamnative/sn-platform-slim:2.10.3.4 replicas: 7 # Change the replica count pod: resources: requests: cpu: 200m memory: 512Mi securityContext: runAsNonRoot: true storage: journal: numDirsPerVolume: 1 numVolumes: 1 volumeClaimTemplate: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi ledger: numDirsPerVolume: 1 numVolumes: 1 volumeClaimTemplate: accessModes: - ReadWriteOnce resources: requests: storage: 16Gi reclaimPolicy: Delete zkServers: zookeepers-zk:2181
Scale ZooKeeper
If you use the default ZooKeeper CR, you should have 3 ZooKeeper replicas running in the pulsar
namespace. This example scales the number of Pods from 3 to 5.
View the existing ZooKeeper CR.
kubectl get zk -n pulsar
You should see the following output:
NAME REPLICAS READY REPLICAS DESIRED IMAGE AGE zookeepers 3 3 streamnative/sn-platform-slim:2.10.3.4 16m
Run the following command to scale ZooKeeper.
kubectl scale zk zookeepers --replicas=5 -n pulsar
View the newly-created Pods.
kubectl get pods -n pulsar
You should see the following output:
NAME READY STATUS RESTARTS AGE bookies-bk-0 1/1 Running 0 10m bookies-bk-1 1/1 Running 0 10m bookies-bk-2 1/1 Running 0 10m bookies-bk-auto-recovery-0 1/1 Running 0 10m brokers-broker-0 1/1 Running 0 10m brokers-broker-1 1/1 Running 0 10m proxys-proxy-0 1/1 Running 0 10m proxys-proxy-1 1/1 Running 0 10m zookeepers-zk-0 1/1 Running 0 54s zookeepers-zk-1 1/1 Running 0 31s zookeepers-zk-2 1/1 Running 0 71s zookeepers-zk-3 1/1 Running 0 98s zookeepers-zk-4 1/1 Running 0 98s
Alternatively, you can create a ZooKeeper YAML file with the following configuration and use the
kubectl apply -f
command to apply the file.apiVersion: zookeeper.streamnative.io/v1alpha1 kind: ZooKeeperCluster metadata: name: zookeepers namespace: pulsar spec: image: streamnative/sn-platform-slim:2.10.3.4 pod: resources: requests: cpu: 50m memory: 256Mi securityContext: runAsNonRoot: true persistence: data: accessModes: - ReadWriteOnce resources: requests: storage: 8Gi dataLog: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi reclaimPolicy: Delete replicas: 5 # Change the replica count