> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Replace BookKeeper storage

Replace the persistent volumes backing a running BookKeeper cluster, without downtime and without data loss. Use this to change disk type (for example, HDD to SSD), move BookKeeper to a different StorageClass, or reduce over-provisioned volume sizes.

This procedure uses the operator's built-in decommission mechanism, which replicates a bookie's ledgers to the remaining bookies before its volumes are discarded. It is the safest approach, and it is slower than alternatives because all data is copied over the network.

Volume expansion does not require this procedure. To grow a volume, increase the storage request on the `BookKeeperCluster` resource and `sn-operator` expands the existing volumes in place, provided the StorageClass sets `allowVolumeExpansion: true`.

## How it works

Unlike ZooKeeper, BookKeeper does not replicate its full state to every node, so a bookie cannot simply be emptied and restarted. Its ledgers must first be re-replicated elsewhere.

Decommissioning does this. When you mark a bookie for decommission and delete its pod, the operator holds the pod via a finalizer and drives auto-recovery to replicate every ledger the bookie holds onto the remaining bookies. Once replication completes, **the operator deletes the pod's volume claims for you** and lets the StatefulSet recreate the pod. The new claims are created from the StatefulSet's volume claim templates, so they pick up whatever storage you configured in Step 1.

<Note>
  You do not delete volume claims yourself in this procedure. The operator does it as part of decommissioning, after it has confirmed the data is safe elsewhere. This is the main safety property of the decommission approach.
</Note>

Because the whole pod is replaced, the journal and ledger volumes are recreated together. You cannot change one and leave the other on its original storage.

<Warning>
  Decommission one bookie at a time, and confirm the cluster has no under-replicated ledgers before moving to the next. Decommissioning several bookies concurrently can leave ledgers without enough replicas to recover from.
</Warning>

## Prerequisites

* `kubectl` access to the namespace running your Pulsar cluster, with permission to annotate pods, delete pods, and edit the `BookKeeperCluster` resource
* Permission to create or modify StorageClasses, depending on which approach you choose below
* Enough spare capacity on the remaining bookies to hold the decommissioned bookie's data
* No in-flight scaling operation or rolling upgrade on the BookKeeper cluster

### Inspect your environment first

Determine the following four things before you begin.

**1. Which volumes your bookies use.** BookKeeper supports multiple journal and ledger volumes per bookie, so the set of claims depends on your configuration:

```bash theme={null}
kubectl get sts <cluster>-bk -n <namespace> \
  -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}{"\n"}{end}'
```

Templates are named `journal-<index>` and `ledgers-<index>`, one per volume configured in `spec.storage.journal.numVolumes` and `spec.storage.ledger.numVolumes`. Apply every step to all volumes the command lists.

**2. Which StorageClass they use, and its reclaim policy.**

```bash theme={null}
kubectl get pvc -n <namespace> -l cloud.streamnative.io/cluster=<cluster>,cloud.streamnative.io/component=bookie \
  -o custom-columns=NAME:.metadata.name,SC:.spec.storageClassName,SIZE:.spec.resources.requests.storage

kubectl get sc <storageclass-name> \
  -o custom-columns=NAME:.metadata.name,PROVISIONER:.provisioner,RECLAIM:.reclaimPolicy
```

A reclaim policy of `Delete` means each old disk is released automatically. `Retain` means it is not, and you must [clean up the old volumes](#clean-up-retained-volumes) yourself or continue paying for them.

**3. That the cluster is healthy, with no under-replicated ledgers.**

```bash theme={null}
kubectl -n <namespace> exec <cluster>-bk-0 -- \
  bin/bookkeeper shell listunderreplicated
```

All broker, bookie, and ZooKeeper pods must also be Running and Ready.

**4. That you have enough bookies to decommission one.** Decommissioning removes a bookie from service until its replacement is ready. If the number of remaining bookies drops below the largest ensemble size configured on any namespace, auto-recovery cannot place the replicas it needs and the decommission stalls with ledgers permanently under-replicated.

Compare your bookie count against the highest `ensembleSize` in use across your namespaces, and confirm the cluster can serve writes with one fewer bookie before starting.

## Step 1: Point BookKeeper at the new storage

New claims are created from the StatefulSet's volume claim templates, so those templates must describe the storage you want before any bookie is decommissioned. Choose the approach that matches your environment.

### Option A: Change the StorageClass on the BookKeeperCluster

Use this approach whenever you can create a new StorageClass. It confines the change to BookKeeper and leaves every other component untouched.

1. Create a new StorageClass for the target disk type, under a new name. The following example provisions AWS EBS gp3 volumes:

   ```yaml theme={null}
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: bookkeeper-ssd
   provisioner: ebs.csi.aws.com
   parameters:
     type: gp3
   allowVolumeExpansion: true
   reclaimPolicy: Delete
   volumeBindingMode: WaitForFirstConsumer
   ```

2. Set the new StorageClass on the `BookKeeperCluster` resource, for both the journal and the ledger volumes:

   ```yaml theme={null}
   spec:
     storage:
       journal:
         volumeClaimTemplate:
           storageClassName: bookkeeper-ssd
       ledger:
         volumeClaimTemplate:
           storageClassName: bookkeeper-ssd
   ```

3. `sn-operator` detects the change and replaces the StatefulSet so it carries the new templates. The StatefulSet is deleted with an orphan propagation policy, so the running pods and existing volumes survive and are adopted by the replacement. Confirm the new templates are in place before continuing:

   ```bash theme={null}
   kubectl get sts <cluster>-bk -n <namespace> \
     -o jsonpath='{range .spec.volumeClaimTemplates[*]}{.metadata.name}={.spec.storageClassName}{"\n"}{end}'
   ```

<Note>
  `sn-operator` performs this replacement only while the StatefulSet is Ready and its replica count is not changing. If the templates do not update, see [The StorageClass change is not reflected in the StatefulSet](#the-storageclass-change-is-not-reflected-in-the-statefulset).
</Note>

Existing volumes keep their original StorageClass and are unaffected at this point. Only volumes created from now on use the new one, which is what makes the per-bookie decommission in Step 2 effective.

### Option B: Recreate the StorageClass in place

Use this approach when you cannot introduce a new StorageClass — for example, when BookKeeper uses the cluster default StorageClass and StorageClass definitions are owned by a platform team or a GitOps pipeline — or when you deliberately want every component to move to the new disk type.

A StatefulSet's `volumeClaimTemplates` are immutable, so if the operator cannot replace the StatefulSet for you, redefining the StorageClass under the same name is the only way to change what new volumes get.

<Warning>
  This changes the disk type for **every** volume subsequently created from that StorageClass, not just BookKeeper's. A ZooKeeper volume created later by a pod replacement also uses the new parameters. Confirm this is acceptable before proceeding.
</Warning>

StorageClass `parameters` are immutable, so the StorageClass must be deleted and recreated under the same name. Existing volumes are already bound and provisioned, and are unaffected by the deletion. New claims referencing the StorageClass fail until it is recreated, so perform both operations back to back.

1. Back up the current definition:

   ```bash theme={null}
   kubectl get sc <storageclass-name> -o yaml > storageclass-backup.yaml
   ```

2. Copy the backup and modify only the `parameters` section. Keep `provisioner`, `reclaimPolicy`, and `volumeBindingMode` identical — changing those has consequences beyond disk type.

3. Delete and immediately recreate:

   ```bash theme={null}
   kubectl delete sc <storageclass-name>
   kubectl apply -f storageclass-updated.yaml
   ```

4. Verify the new parameters, and that all pods are still Running:

   ```bash theme={null}
   kubectl get sc <storageclass-name> -o jsonpath='{.parameters}'
   kubectl get pods -n <namespace> -l cloud.streamnative.io/cluster=<cluster>
   ```

To roll back, delete the StorageClass and re-apply `storageclass-backup.yaml`.

## Step 2: Decommission each bookie in turn

Repeat this section once per bookie. **Process from the highest ordinal down to the lowest** — for example `bk-4`, `bk-3`, `bk-2`, `bk-1`, `bk-0`. This keeps the ordering consistent with how the StatefulSet would scale in, and avoids repeatedly disturbing the lowest-ordinal pod that administrative commands are typically run against.

### Mark the bookie for decommission

```bash theme={null}
kubectl -n <namespace> annotate pod <cluster>-bk-<N> \
  bookkeeper.streamnative.io/decommission-ctrl="true"
```

<Note>
  The same annotation can be set on the `BookKeeperCluster` resource to apply to all bookies, and the pod-level annotation takes precedence over it. Setting the annotation to `"false"` prevents decommissioning instead — useful as a guard while other maintenance is in progress.
</Note>

### Delete the pod

```bash theme={null}
kubectl -n <namespace> delete pod <cluster>-bk-<N>
```

The operator holds the pod with a finalizer, triggers auto-recovery to replicate this bookie's ledgers onto the remaining bookies, then deletes the pod's volume claims and allows the StatefulSet to recreate it.

<Note>
  The replacement pod may briefly appear `Pending` and unschedulable while its old claims are being removed. The operator deletes the pod a second time to work around a Kubernetes StatefulSet issue in this situation. This is expected and requires no action.
</Note>

### Monitor decommission progress

```bash theme={null}
# Find the decommission job
kubectl -n <namespace> get jobs -l cloud.streamnative.io/component=bookie

# Follow its logs
kubectl -n <namespace> logs -f job/<decommission-job-name>
```

Watch for the count of ledgers needing re-replication to reach `0`. On a bookie holding hundreds of gigabytes this can take hours.

### Verify the bookie returned to service

```bash theme={null}
# Wait for the replacement pod to be Ready
kubectl -n <namespace> wait --timeout=60m --for=condition=Ready pod/<cluster>-bk-<N>

# Confirm the bookie is writable, not in read-only mode
kubectl -n <namespace> exec <cluster>-bk-<N> -- \
  curl -s http://localhost:8000/api/v1/bookie/state/readonly
# Expected: {"readOnly":false}

# Confirm no ledgers are left under-replicated
kubectl -n <namespace> exec <cluster>-bk-0 -- \
  bin/bookkeeper shell listunderreplicated

# Confirm the new claims use the intended StorageClass
kubectl -n <namespace> get pvc -l cloud.streamnative.io/cluster=<cluster>,cloud.streamnative.io/component=bookie \
  -o custom-columns=NAME:.metadata.name,SC:.spec.storageClassName,SIZE:.spec.resources.requests.storage
```

<Warning>
  Do not decommission the next bookie until `listunderreplicated` reports nothing and the replacement pod is Ready and writable. Proceeding while ledgers are still under-replicated reduces the number of intact copies and risks data loss.
</Warning>

Then move to the next bookie.

## Speed up re-replication

Re-replication throughput is limited by the number of auto-recovery workers. If decommissioning is slower than you can accept, you can increase them:

```bash theme={null}
kubectl -n <namespace> get pods -l cloud.streamnative.io/component=recovery
```

<Warning>
  Additional recovery workers generate write load on the remaining bookies, which competes with client traffic. Increase them gradually and watch client write latency. Restore the original count once the procedure is complete.
</Warning>

## Clean up retained volumes

If the StorageClass reclaim policy is `Retain`, each claim the operator deleted leaves its PersistentVolume behind in the `Released` phase, along with the disk backing it. Those disks continue to incur cost until you remove them.

```bash theme={null}
kubectl get pv --sort-by=.metadata.creationTimestamp \
  -o custom-columns=NAME:.metadata.name,STATUS:.status.phase,CLAIM:.spec.claimRef.name,SC:.spec.storageClassName
```

Confirm each `Released` volume corresponds to a bookie you decommissioned, and that the cluster reports no under-replicated ledgers, before deleting it and its underlying disk.

## Troubleshooting

### Decommission is taking too long

* Check the number of auto-recovery workers and consider increasing them.
* Check free capacity on the remaining bookies. Replication slows sharply as they fill.
* Check network throughput between bookie nodes.

### Decommission stalls with ledgers permanently under-replicated

Auto-recovery cannot place replicas when too few bookies remain. If the bookie count minus one is below the largest configured ensemble size, decommissioning cannot complete. Scale the cluster up before continuing, then retry.

### The StorageClass change is not reflected in the StatefulSet

`sn-operator` replaces the StatefulSet only while it is Ready and its replica count is unchanged. Check that all bookie pods are Ready and no scaling operation is in progress.

Also confirm the StatefulSet is not opted out of management or replacement — `spec.apiObjects.bookieStatefulSet.managed: false` or `replaceEnabled: false` on the `BookKeeperCluster` both prevent the templates from being updated. Then check the `sn-operator` logs.

### A replacement pod stays Pending

```bash theme={null}
kubectl -n <namespace> describe pod <cluster>-bk-<N>
```

* **Claim is `Pending`** — nothing provisioned it. Confirm the StorageClass named in the StatefulSet templates exists.
* **Insufficient capacity** — the new volumes could not be provisioned on the available nodes. Check node capacity and any topology constraints on the StorageClass.

## Related documentation

* [Configure storage](/private-cloud/v2/configure-private-cloud/storage/private-cloud-storage)
* [Replace ZooKeeper storage](/private-cloud/v2/operate-private-cloud/storage/replace-zookeeper-storage)