> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Replace ZooKeeper storage
Replace the persistent volumes backing a running ZooKeeper cluster, without downtime. Use this to change disk type (for example, HDD to SSD), move ZooKeeper to a different StorageClass, or reduce over-provisioned volume sizes.
Volume expansion does not require this procedure. To grow a volume, increase the storage request on the `ZooKeeperCluster` resource and `sn-operator` expands the existing volumes in place, provided the StorageClass sets `allowVolumeExpansion: true`.
## How it works
ZooKeeper replicates its entire state to every ensemble member. A ZooKeeper server that starts with empty data directories automatically downloads a full snapshot and transaction log from the current leader. ZooKeeper state is typically well under 1 GB, so this sync completes in seconds.
This makes volume replacement straightforward: delete the volumes for one server, restart it, and let ZooKeeper refill it from the remaining quorum. Replacing one server at a time keeps a quorum available throughout, so clients stay connected and no data is lost.
Replace one ZooKeeper server at a time, and confirm the ensemble is fully synced before moving to the next one. Replacing volumes on two servers of a three-server ensemble at the same time destroys quorum and loses data.
## Prerequisites
* `kubectl` access to the namespace running your Pulsar cluster, with permission to delete PersistentVolumeClaims and pods
* Permission to create or modify StorageClasses, depending on which approach you choose below
* A recent backup, or a maintenance window in which you can take one
* No in-flight scaling operation or rolling upgrade on the ZooKeeper cluster
### Inspect your environment first
Determine the following three things before you begin.
**1. Which volumes ZooKeeper uses.** ZooKeeper may run with a single data volume, or with a dedicated transaction log volume in addition:
```bash theme={null}
kubectl get sts -zk -n \
-o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}{"\n"}{end}'
```
A `data` entry alone means one volume per server. A `data` and a `data-log` entry means two, which is the case when `spec.persistence.dataLog.resources.requests.storage` is set on the `ZooKeeperCluster` resource. The rest of this page refers to "each volume" — apply every step to all volumes the command lists.
When your cluster uses a dedicated transaction log volume, always replace the `data` and `data-log` volumes for a server **together**, in the same step. Replacing only the data volume leaves stale transaction logs behind. ZooKeeper then fails to start with `Committed proposal cached out of order`, or silently drops znodes on a later replacement.
**2. Which StorageClass they use, and its reclaim policy.**
```bash theme={null}
kubectl get pvc -n -l cloud.streamnative.io/cluster= \
-o custom-columns=NAME:.metadata.name,SC:.spec.storageClassName,SIZE:.spec.resources.requests.storage
kubectl get sc \
-o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner,RECLAIM:.reclaimPolicy
```
A reclaim policy of `Delete` means the old disk is released automatically when you delete a claim. `Retain` means it is not, and you must [clean up the old volumes](#clean-up-retained-volumes) yourself or continue paying for them.
**3. That the ensemble is healthy.** Every ZooKeeper pod must be Running and Ready, with a leader elected:
```bash theme={null}
for i in 0 1 2; do
echo "=== zk-$i ==="
kubectl -n exec -zk-$i -- \
bash -c "echo stat | nc localhost 2181 | grep Mode"
done
```
Exactly one server reports `Mode: leader`. Note which one — you will replace it last.
## Step 1: Point ZooKeeper at the new storage
New volumes are created from the StatefulSet's volume claim templates, so those templates must describe the storage you want before you replace anything. Choose the approach that matches your environment.
### Option A: Change the StorageClass on the ZooKeeperCluster
Use this approach whenever you can create a new StorageClass. It confines the change to ZooKeeper and leaves every other component untouched.
1. Create a new StorageClass for the target disk type, under a new name. The following example provisions AWS EBS gp3 volumes:
```yaml theme={null}
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: zookeeper-ssd
provisioner: ebs.csi.aws.com
parameters:
type: gp3
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
```
2. Set the new StorageClass on the `ZooKeeperCluster` resource, for every volume the cluster uses:
```yaml theme={null}
spec:
persistence:
data:
storageClassName: zookeeper-ssd
dataLog:
storageClassName: zookeeper-ssd
```
3. `sn-operator` detects the change and replaces the StatefulSet so it carries the new templates. The StatefulSet is deleted with an orphan propagation policy, so the running pods and existing volumes survive and are adopted by the replacement. Confirm the new templates are in place before continuing:
```bash theme={null}
kubectl get sts -zk -n \
-o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}={.spec.storageClassName}{"\n"}{end}'
```
`sn-operator` performs this replacement only when the StatefulSet is Ready and its replica count is not changing. If the templates do not update, check for an in-flight scaling operation or a pod that is not yet Ready, then check the `sn-operator` logs.
Existing volumes keep their original StorageClass and are unaffected at this point. Only volumes created from now on use the new one, which is what makes the per-server replacement in Step 2 effective.
### Option B: Recreate the StorageClass in place
Use this approach when you cannot introduce a new StorageClass — for example, when ZooKeeper uses the cluster default StorageClass and StorageClass definitions are owned by a platform team or a GitOps pipeline — or when you deliberately want every component to move to the new disk type.
A StatefulSet's `volumeClaimTemplates` are immutable, so if the operator cannot replace the StatefulSet for you, redefining the StorageClass under the same name is the only way to change what new volumes get.
This changes the disk type for **every** volume subsequently created from that StorageClass, not just ZooKeeper's. A BookKeeper volume created later by a scale-up or a pod replacement also uses the new parameters. Confirm this is acceptable before proceeding.
StorageClass `parameters` are immutable, so the StorageClass must be deleted and recreated under the same name. Existing volumes are already bound and provisioned, and are unaffected by the deletion. New claims referencing the StorageClass fail until it is recreated, so perform both operations back to back.
1. Back up the current definition:
```bash theme={null}
kubectl get sc -o yaml > storageclass-backup.yaml
```
2. Copy the backup and modify only the `parameters` section. Keep `provisioner`, `reclaimPolicy`, and `volumeBindingMode` identical — changing those has consequences beyond disk type.
3. Delete and immediately recreate:
```bash theme={null}
kubectl delete sc
kubectl apply -f storageclass-updated.yaml
```
4. Verify the new parameters, and that all pods are still Running:
```bash theme={null}
kubectl get sc -o jsonpath='{.parameters}'
kubectl get pods -n -l cloud.streamnative.io/cluster=
```
To roll back, delete the StorageClass and re-apply `storageclass-backup.yaml`.
## Step 2: Replace each server's volumes
Repeat this section once per ZooKeeper server. **Process followers first and the leader last** — replacing the leader triggers an election, and there is no reason to incur one before the final server.
### Back up the server's data
```bash theme={null}
kubectl -n exec -zk- -- \
tar czf - /pulsar/data/zookeeper/ /pulsar/data-log/ \
> zk-backup--zk--$(date +%Y%m%d%H%M%S).tar.gz
```
### Delete the volume claims, then the pod
Delete the claims first. They remain in `Terminating` because the `pvc-protection` finalizer holds them until the pod using them is gone:
```bash theme={null}
kubectl -n delete pvc data--zk- --wait=false
kubectl -n delete pvc data-log--zk- --wait=false
```
Delete the claims before the pod, never the other way round. If the pod goes first, the StatefulSet recreates it and immediately rebinds the old volumes.
Then delete the pod. This is a graceful shutdown, so ZooKeeper flushes cleanly. The StatefulSet recreates the pod, creates new claims from the updated templates, and ZooKeeper syncs from the leader:
```bash theme={null}
kubectl -n delete pod -zk-
```
If your cluster provisions volumes statically rather than through a dynamic provisioner, nothing creates the replacement claim for you. Create it yourself, bound to the new volume, before deleting the pod — the StatefulSet adopts an existing claim rather than creating one, provided the name matches what it expects.
### Confirm the server rejoined and fully synced
Wait for the pod to become Ready:
```bash theme={null}
kubectl -n wait --timeout=10m --for=condition=Ready pod/-zk-
```
Readiness alone is not sufficient. Confirm from the leader that every follower has finished syncing. On a three-server ensemble, `zk_synced_followers` must read `2`:
```bash theme={null}
kubectl -n exec -zk- -- \
bash -c "echo mntr | nc localhost 2181 | grep zk_synced_followers"
```
Do not continue to the next server until `zk_synced_followers` equals the ensemble size minus one. Continuing early risks data loss, because a newly elected leader may hold incomplete data if its followers have not caught up.
Confirm the new volumes have the properties you intended:
```bash theme={null}
kubectl -n get pvc data--zk- data-log--zk- \
-o custom-columns=NAME:.metadata.name,SC:.spec.storageClassName,SIZE:.spec.resources.requests.storage
```
Then move to the next server.
### Replacing the leader
Deleting the leader triggers an election among the followers, whose volumes you have already replaced. Afterwards:
1. One of the already-replaced followers becomes the leader.
2. The old leader's pod restarts, rejoins as a follower, and syncs from the new leader.
3. Re-run the checks above against the **new** leader.
## Clean up retained volumes
If the StorageClass reclaim policy is `Retain`, each deleted claim leaves its PersistentVolume behind in the `Released` phase, along with the disk backing it. Those disks continue to incur cost until you remove them.
```bash theme={null}
kubectl get pv --sort-by=.metadata.creationTimestamp \
-o custom-columns=NAME:.metadata.name,STATUS:.status.phase,CLAIM:.spec.claimRef.name,SC:.spec.storageClassName
```
Confirm each `Released` volume corresponds to a claim you replaced, and that you no longer need its contents, before deleting it and its underlying disk.
## Troubleshooting
### A pod stays Pending after replacement
```bash theme={null}
kubectl -n describe pod -zk-
kubectl -n get pvc -n | grep -zk-
```
* **Claim is `Pending`** — nothing provisioned it. Check the claim's events, and confirm the StorageClass named in the StatefulSet templates exists.
* **Claim was created with the old StorageClass** — the StatefulSet templates were not updated before you deleted the volumes. Re-check Step 1, then replace that server's volumes again.
### ZooKeeper fails to start with `Committed proposal cached out of order`
A stale transaction log volume was left in place while the data volume was replaced. Replace both volumes for that server together, as described above.
### The StorageClass change is not reflected in the StatefulSet
`sn-operator` replaces the StatefulSet only while it is Ready and its replica count is unchanged. Confirm all ZooKeeper pods are Ready, that no scaling operation is in progress, and check the `sn-operator` logs.
## Related documentation
* [Configure storage](/private-cloud/v2/configure-private-cloud/storage/private-cloud-storage)
* [Graceful cluster rollout](/private-cloud/v2/configure-private-cloud/advanced/private-cloud-graceful-cluster-rollout)