Replace BookKeeper storage

Replace the persistent volumes backing a running BookKeeper cluster, without downtime and without data loss. Use this to change disk type (for example, HDD to SSD), move BookKeeper to a different StorageClass, or reduce over-provisioned volume sizes. This procedure uses the operator’s built-in decommission mechanism, which replicates a bookie’s ledgers to the remaining bookies before its volumes are discarded. It is the safest approach, and it is slower than alternatives because all data is copied over the network. Volume expansion does not require this procedure. To grow a volume, increase the storage request on the BookKeeperCluster resource and sn-operator expands the existing volumes in place, provided the StorageClass sets allowVolumeExpansion: true.

How it works

Unlike ZooKeeper, BookKeeper does not replicate its full state to every node, so a bookie cannot simply be emptied and restarted. Its ledgers must first be re-replicated elsewhere. Decommissioning does this. When you mark a bookie for decommission and delete its pod, the operator holds the pod via a finalizer and drives auto-recovery to replicate every ledger the bookie holds onto the remaining bookies. Once replication completes, the operator deletes the pod’s volume claims for you and lets the StatefulSet recreate the pod. The new claims are created from the StatefulSet’s volume claim templates, so they pick up whatever storage you configured in Step 1.

You do not delete volume claims yourself in this procedure. The operator does it as part of decommissioning, after it has confirmed the data is safe elsewhere. This is the main safety property of the decommission approach.

Because the whole pod is replaced, the journal and ledger volumes are recreated together. You cannot change one and leave the other on its original storage.

Decommission one bookie at a time, and confirm the cluster has no under-replicated ledgers before moving to the next. Decommissioning several bookies concurrently can leave ledgers without enough replicas to recover from.

Prerequisites

kubectl access to the namespace running your Pulsar cluster, with permission to annotate pods, delete pods, and edit the BookKeeperCluster resource
Permission to create or modify StorageClasses, depending on which approach you choose below
Enough spare capacity on the remaining bookies to hold the decommissioned bookie’s data
No in-flight scaling operation or rolling upgrade on the BookKeeper cluster

Inspect your environment first

Determine the following four things before you begin. 1. Which volumes your bookies use. BookKeeper supports multiple journal and ledger volumes per bookie, so the set of claims depends on your configuration:

kubectl get sts <cluster>-bk -n <namespace> \
  -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}{"\n"}{end}'

Templates are named journal-<index> and ledgers-<index>, one per volume configured in spec.storage.journal.numVolumes and spec.storage.ledger.numVolumes. Apply every step to all volumes the command lists. 2. Which StorageClass they use, and its reclaim policy.

kubectl get pvc -n <namespace> -l cloud.streamnative.io/cluster=<cluster>,cloud.streamnative.io/component=bookie \
  -o custom-columns=NAME:.metadata.name,SC:.spec.storageClassName,SIZE:.spec.resources.requests.storage

kubectl get sc <storageclass-name> \
  -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner,RECLAIM:.reclaimPolicy

A reclaim policy of Delete means each old disk is released automatically. Retain means it is not, and you must clean up the old volumes yourself or continue paying for them. 3. That the cluster is healthy, with no under-replicated ledgers.

kubectl -n <namespace> exec <cluster>-bk-0 -- \
  bin/bookkeeper shell listunderreplicated

All broker, bookie, and ZooKeeper pods must also be Running and Ready. 4. That you have enough bookies to decommission one. Decommissioning removes a bookie from service until its replacement is ready. If the number of remaining bookies drops below the largest ensemble size configured on any namespace, auto-recovery cannot place the replicas it needs and the decommission stalls with ledgers permanently under-replicated. Compare your bookie count against the highest ensembleSize in use across your namespaces, and confirm the cluster can serve writes with one fewer bookie before starting.

Step 1: Point BookKeeper at the new storage

New claims are created from the StatefulSet’s volume claim templates, so those templates must describe the storage you want before any bookie is decommissioned. Choose the approach that matches your environment.

Option A: Change the StorageClass on the BookKeeperCluster

Use this approach whenever you can create a new StorageClass. It confines the change to BookKeeper and leaves every other component untouched.

Create a new StorageClass for the target disk type, under a new name. The following example provisions AWS EBS gp3 volumes:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: bookkeeper-ssd
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
allowVolumeExpansion: true
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Set the new StorageClass on the BookKeeperCluster resource, for both the journal and the ledger volumes:

spec:
  storage:
    journal:
      volumeClaimTemplate:
        storageClassName: bookkeeper-ssd
    ledger:
      volumeClaimTemplate:
        storageClassName: bookkeeper-ssd

sn-operator detects the change and replaces the StatefulSet so it carries the new templates. The StatefulSet is deleted with an orphan propagation policy, so the running pods and existing volumes survive and are adopted by the replacement. Confirm the new templates are in place before continuing:
kubectl get sts <cluster>-bk -n <namespace> \ -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}={.spec.storageClassName}{"\n"}{end}'

sn-operator performs this replacement only while the StatefulSet is Ready and its replica count is not changing. If the templates do not update, see The StorageClass change is not reflected in the StatefulSet.

Existing volumes keep their original StorageClass and are unaffected at this point. Only volumes created from now on use the new one, which is what makes the per-bookie decommission in Step 2 effective.

Option B: Recreate the StorageClass in place

Use this approach when you cannot introduce a new StorageClass — for example, when BookKeeper uses the cluster default StorageClass and StorageClass definitions are owned by a platform team or a GitOps pipeline — or when you deliberately want every component to move to the new disk type. A StatefulSet’s volumeClaimTemplates are immutable, so if the operator cannot replace the StatefulSet for you, redefining the StorageClass under the same name is the only way to change what new volumes get.

This changes the disk type for every volume subsequently created from that StorageClass, not just BookKeeper’s. A ZooKeeper volume created later by a pod replacement also uses the new parameters. Confirm this is acceptable before proceeding.

StorageClass parameters are immutable, so the StorageClass must be deleted and recreated under the same name. Existing volumes are already bound and provisioned, and are unaffected by the deletion. New claims referencing the StorageClass fail until it is recreated, so perform both operations back to back.

Back up the current definition:

kubectl get sc <storageclass-name> -o yaml > storageclass-backup.yaml

Copy the backup and modify only the parameters section. Keep provisioner, reclaimPolicy, and volumeBindingMode identical — changing those has consequences beyond disk type.

Delete and immediately recreate:

kubectl delete sc <storageclass-name>
kubectl apply -f storageclass-updated.yaml

Verify the new parameters, and that all pods are still Running:

kubectl get sc <storageclass-name> -o jsonpath='{.parameters}'
kubectl get pods -n <namespace> -l cloud.streamnative.io/cluster=<cluster>

To roll back, delete the StorageClass and re-apply storageclass-backup.yaml.

Step 2: Decommission each bookie in turn

Repeat this section once per bookie. Process from the highest ordinal down to the lowest — for example bk-4, bk-3, bk-2, bk-1, bk-0. This keeps the ordering consistent with how the StatefulSet would scale in, and avoids repeatedly disturbing the lowest-ordinal pod that administrative commands are typically run against.

Mark the bookie for decommission

kubectl -n <namespace> annotate pod <cluster>-bk-<N> \
  bookkeeper.streamnative.io/decommission-ctrl="true"

The same annotation can be set on the BookKeeperCluster resource to apply to all bookies, and the pod-level annotation takes precedence over it. Setting the annotation to "false" prevents decommissioning instead — useful as a guard while other maintenance is in progress.

Delete the pod

kubectl -n <namespace> delete pod <cluster>-bk-<N>

The operator holds the pod with a finalizer, triggers auto-recovery to replicate this bookie’s ledgers onto the remaining bookies, then deletes the pod’s volume claims and allows the StatefulSet to recreate it.

The replacement pod may briefly appear Pending and unschedulable while its old claims are being removed. The operator deletes the pod a second time to work around a Kubernetes StatefulSet issue in this situation. This is expected and requires no action.

Monitor decommission progress

# Find the decommission job
kubectl -n <namespace> get jobs -l cloud.streamnative.io/component=bookie

# Follow its logs
kubectl -n <namespace> logs -f job/<decommission-job-name>

Watch for the count of ledgers needing re-replication to reach 0. On a bookie holding hundreds of gigabytes this can take hours.

Verify the bookie returned to service

# Wait for the replacement pod to be Ready
kubectl -n <namespace> wait --timeout=60m --for=condition=Ready pod/<cluster>-bk-<N>

# Confirm the bookie is writable, not in read-only mode
kubectl -n <namespace> exec <cluster>-bk-<N> -- \
  curl -s http://localhost:8000/api/v1/bookie/state/readonly
# Expected: {"readOnly":false}

# Confirm no ledgers are left under-replicated
kubectl -n <namespace> exec <cluster>-bk-0 -- \
  bin/bookkeeper shell listunderreplicated

# Confirm the new claims use the intended StorageClass
kubectl -n <namespace> get pvc -l cloud.streamnative.io/cluster=<cluster>,cloud.streamnative.io/component=bookie \
  -o custom-columns=NAME:.metadata.name,SC:.spec.storageClassName,SIZE:.spec.resources.requests.storage

Do not decommission the next bookie until listunderreplicated reports nothing and the replacement pod is Ready and writable. Proceeding while ledgers are still under-replicated reduces the number of intact copies and risks data loss.

Then move to the next bookie.

Speed up re-replication

Re-replication throughput is limited by the number of auto-recovery workers. If decommissioning is slower than you can accept, you can increase them:

kubectl -n <namespace> get pods -l cloud.streamnative.io/component=recovery

Additional recovery workers generate write load on the remaining bookies, which competes with client traffic. Increase them gradually and watch client write latency. Restore the original count once the procedure is complete.

Clean up retained volumes

If the StorageClass reclaim policy is Retain, each claim the operator deleted leaves its PersistentVolume behind in the Released phase, along with the disk backing it. Those disks continue to incur cost until you remove them.

kubectl get pv --sort-by=.metadata.creationTimestamp \
  -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,CLAIM:.spec.claimRef.name,SC:.spec.storageClassName

Confirm each Released volume corresponds to a bookie you decommissioned, and that the cluster reports no under-replicated ledgers, before deleting it and its underlying disk.

Troubleshooting

Decommission is taking too long

Check the number of auto-recovery workers and consider increasing them.
Check free capacity on the remaining bookies. Replication slows sharply as they fill.
Check network throughput between bookie nodes.

Decommission stalls with ledgers permanently under-replicated

Auto-recovery cannot place replicas when too few bookies remain. If the bookie count minus one is below the largest configured ensemble size, decommissioning cannot complete. Scale the cluster up before continuing, then retry.

The StorageClass change is not reflected in the StatefulSet

sn-operator replaces the StatefulSet only while it is Ready and its replica count is unchanged. Check that all bookie pods are Ready and no scaling operation is in progress. Also confirm the StatefulSet is not opted out of management or replacement — spec.apiObjects.bookieStatefulSet.managed: false or replaceEnabled: false on the BookKeeperCluster both prevent the templates from being updated. Then check the sn-operator logs.

A replacement pod stays Pending

kubectl -n <namespace> describe pod <cluster>-bk-<N>

Claim is Pending — nothing provisioned it. Confirm the StorageClass named in the StatefulSet templates exists.
Insufficient capacity — the new volumes could not be provisioned on the available nodes. Check node capacity and any topology constraints on the StorageClass.

Overview

Quick Start

Configure Private Cloud

Operate Private Cloud

References

How it works

Prerequisites

Inspect your environment first

Step 1: Point BookKeeper at the new storage

Option A: Change the StorageClass on the BookKeeperCluster

Option B: Recreate the StorageClass in place

Step 2: Decommission each bookie in turn

Mark the bookie for decommission

Delete the pod

Monitor decommission progress

Verify the bookie returned to service

Speed up re-replication

Clean up retained volumes

Troubleshooting

Decommission is taking too long

Decommission stalls with ledgers permanently under-replicated

The StorageClass change is not reflected in the StatefulSet

A replacement pod stays Pending

​How it works

​Prerequisites

​Inspect your environment first

​Step 1: Point BookKeeper at the new storage

​Option A: Change the StorageClass on the BookKeeperCluster

​Option B: Recreate the StorageClass in place

​Step 2: Decommission each bookie in turn

​Mark the bookie for decommission

​Delete the pod

​Monitor decommission progress

​Verify the bookie returned to service

​Speed up re-replication

​Clean up retained volumes

​Troubleshooting

​Decommission is taking too long

​Decommission stalls with ledgers permanently under-replicated

​The StorageClass change is not reflected in the StatefulSet

​A replacement pod stays Pending

​Related documentation

How it works

Prerequisites

Inspect your environment first

Step 1: Point BookKeeper at the new storage

Option A: Change the StorageClass on the BookKeeperCluster

Option B: Recreate the StorageClass in place

Step 2: Decommission each bookie in turn

Mark the bookie for decommission

Delete the pod

Monitor decommission progress

Verify the bookie returned to service

Speed up re-replication

Clean up retained volumes

Troubleshooting

Decommission is taking too long

Decommission stalls with ledgers permanently under-replicated

The StorageClass change is not reflected in the StatefulSet

A replacement pod stays Pending

Related documentation