When the traffic handled by your Pulsar cluster increases or decreases, you can scale out or scale in the cluster accordingly. Depending on the specific use case, you may choose to increase or decrease the number of nodes for different components, including Pulsar brokers, Pulsar proxies, bookies, and ZooKeeper.
When resizing the Pulsar cluster on Kubernetes created by the Pulsar Operators, you need to perform the scaling action on Custom Resources (CRs) instead of the individual workloads deployed through them. For example, when deploying Pulsar broker workloads, the Pulsar Operators create a broker StatefulSet with 2 Pod replicas by default. To scale the StatefulSet, you should adjust the broker CR instead of manually changing the StatefulSet.
This document describes how to manually scale Pulsar components on Kubernetes.
Prerequisites
You have deployed a Pulsar cluster using the Pulsar Operators.
Scale Pulsar brokers
If you use the default broker CR, you should have 2 broker replicas running in the pulsar namespace. This example scales the number of Pods from 2 to 5.
-
View the existing broker CR.
You should see the following output:
NAME REPLICAS READY REPLICAS DESIRED IMAGE ENDPOINT AGE
brokers 2 2 streamnative/sn-platform-slim:2.10.3.4 10m
-
Run the following command to scale brokers.
kubectl scale pb brokers --replicas=5 -n pulsar
-
View the newly-created Pods.
kubectl get pods -n pulsar
You should see the following output:
NAME READY STATUS RESTARTS AGE
bookies-bk-0 1/1 Running 0 10m
bookies-bk-1 1/1 Running 0 10m
bookies-bk-2 1/1 Running 0 10m
bookies-bk-auto-recovery-0 1/1 Running 0 10m
brokers-broker-0 1/1 Running 0 10m
brokers-broker-1 1/1 Running 0 10m
brokers-broker-2 1/1 Running 0 10s
brokers-broker-3 1/1 Running 0 10s
brokers-broker-4 1/1 Running 0 10s
proxys-proxy-0 1/1 Running 0 10m
proxys-proxy-1 1/1 Running 0 10m
zookeepers-zk-0 1/1 Running 0 10m
zookeepers-zk-1 1/1 Running 0 10m
zookeepers-zk-2 1/1 Running 0 10m
-
Alternatively, you can create a broker YAML file with the following configuration and use the kubectl apply -f command to apply the file.
apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarBroker
metadata:
name: brokers
namespace: pulsar
spec:
image: streamnative/sn-platform-slim:2.10.3.4
pod:
resources:
requests:
cpu: 200m
memory: 512Mi
securityContext:
runAsNonRoot: true
replicas: 5 # Change the replica count
zkServers: zookeepers-zk:2181
Scale Pulsar proxies
If you use the default proxy CR, you should have 2 proxy replicas running in the pulsar namespace. This example scales the number of Pods from 2 to 5.
-
View the existing proxy CR.
You should see the following output:
NAME REPLICAS READY REPLICAS DESIRED IMAGE ENDPOINT AGE
proxys 2 2 streamnative/sn-platform-slim:2.10.3.4 10m
-
Run the following command to scale Pulsar proxies.
kubectl scale pp proxys --replicas=5 -n pulsar
-
View the newly created Pods.
kubectl get pods -n pulsar
You should see the following output:
NAME READY STATUS RESTARTS AGE
bookies-bk-0 1/1 Running 0 10m
bookies-bk-1 1/1 Running 0 10m
bookies-bk-2 1/1 Running 0 10m
bookies-bk-auto-recovery-0 1/1 Running 0 10m
brokers-broker-0 1/1 Running 0 10m
brokers-broker-1 1/1 Running 0 10m
proxys-proxy-0 1/1 Running 0 10m
proxys-proxy-1 1/1 Running 0 10m
proxys-proxy-2 1/1 Running 0 42s
proxys-proxy-3 1/1 Running 0 42s
proxys-proxy-4 1/1 Running 0 42s
zookeepers-zk-0 1/1 Running 0 10m
zookeepers-zk-1 1/1 Running 0 10m
zookeepers-zk-2 1/1 Running 0 10m
-
Alternatively, you can create a proxy YAML file with the following configuration and use the kubectl apply -f command to apply the file.
apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarProxy
metadata:
name: proxys
namespace: pulsar
spec:
image: 'streamnative/sn-platform-slim:2.10.3.4'
pod:
resources:
requests:
cpu: 200m
memory: 512Mi
securityContext:
runAsNonRoot: true
brokerAddress: brokers-broker
replicas: 5 # Change the replica count
config:
tls:
enabled: false
dnsNames: []
issuerRef:
name: ''
Scale bookies
If you use the default BookKeeper CR, you should have 3 bookie replicas running in the pulsar namespace. This example scales the number of bookie Pods from 3 to 7.
-
View the existing BookKeeper CR.
You should see the following output:
NAME REPLICAS READY REPLICAS ZKSERVERS DESIRED IMAGE AGE
bookies 3 3 zookeepers-zk:2181 streamnative/sn-platform-slim:2.10.3.4 13m
-
Run the following command to scale bookies.
kubectl scale bk bookies --replicas=7 -n pulsar
-
View the newly-created Pods.
kubectl get pods -n pulsar
You should see the following output:
NAME READY STATUS RESTARTS AGE
bookies-bk-0 1/1 Running 0 10m
bookies-bk-1 1/1 Running 0 10m
bookies-bk-2 1/1 Running 0 10m
bookies-bk-3 1/1 Running 0 40s
bookies-bk-4 1/1 Running 0 40s
bookies-bk-5 1/1 Running 0 40s
bookies-bk-6 1/1 Running 0 40s
bookies-bk-auto-recovery-0 1/1 Running 0 10m
brokers-broker-0 1/1 Running 0 10m
brokers-broker-1 1/1 Running 0 10m
proxys-proxy-0 1/1 Running 0 10m
proxys-proxy-1 1/1 Running 0 10m
zookeepers-zk-0 1/1 Running 0 10m
zookeepers-zk-1 1/1 Running 0 10m
zookeepers-zk-2 1/1 Running 0 10m
-
Alternatively, you can create a BookKeeper YAML file with the following configuration and use the kubectl apply -f command to apply the file.
apiVersion: bookkeeper.streamnative.io/v1alpha1
kind: BookKeeperCluster
metadata:
name: bookies
namespace: pulsar
spec:
image: streamnative/sn-platform-slim:2.10.3.4
replicas: 7 # Change the replica count
pod:
resources:
requests:
cpu: 200m
memory: 512Mi
securityContext:
runAsNonRoot: true
storage:
journal:
numDirsPerVolume: 1
numVolumes: 1
volumeClaimTemplate:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
ledger:
numDirsPerVolume: 1
numVolumes: 1
volumeClaimTemplate:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 16Gi
reclaimPolicy: Delete
zkServers: zookeepers-zk:2181
Scale ZooKeeper
If you use the default ZooKeeper CR, you should have 3 ZooKeeper replicas running in the pulsar namespace. This example scales the number of Pods from 3 to 5.
-
View the existing ZooKeeper CR.
You should see the following output:
NAME REPLICAS READY REPLICAS DESIRED IMAGE AGE
zookeepers 3 3 streamnative/sn-platform-slim:2.10.3.4 16m
-
Run the following command to scale ZooKeeper.
kubectl scale zk zookeepers --replicas=5 -n pulsar
-
View the newly-created Pods.
kubectl get pods -n pulsar
You should see the following output:
NAME READY STATUS RESTARTS AGE
bookies-bk-0 1/1 Running 0 10m
bookies-bk-1 1/1 Running 0 10m
bookies-bk-2 1/1 Running 0 10m
bookies-bk-auto-recovery-0 1/1 Running 0 10m
brokers-broker-0 1/1 Running 0 10m
brokers-broker-1 1/1 Running 0 10m
proxys-proxy-0 1/1 Running 0 10m
proxys-proxy-1 1/1 Running 0 10m
zookeepers-zk-0 1/1 Running 0 54s
zookeepers-zk-1 1/1 Running 0 31s
zookeepers-zk-2 1/1 Running 0 71s
zookeepers-zk-3 1/1 Running 0 98s
zookeepers-zk-4 1/1 Running 0 98s
-
Alternatively, you can create a ZooKeeper YAML file with the following configuration and use the kubectl apply -f command to apply the file.
apiVersion: zookeeper.streamnative.io/v1alpha1
kind: ZooKeeperCluster
metadata:
name: zookeepers
namespace: pulsar
spec:
image: streamnative/sn-platform-slim:2.10.3.4
pod:
resources:
requests:
cpu: 50m
memory: 256Mi
securityContext:
runAsNonRoot: true
persistence:
data:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
dataLog:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
reclaimPolicy: Delete
replicas: 5 # Change the replica count
