> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Configure ZooKeeper observers
> Add non-voting ZooKeeper replicas to a StreamNative Private Cloud cluster.
A ZooKeeper observer is a non-voting member of a ZooKeeper ensemble. An observer receives committed updates and serves client requests, but it doesn't vote in leader elections or proposals. Add observers when you need to serve more ZooKeeper clients without increasing the size of the voting quorum. For more information about observer behavior, see [ZooKeeper Observers](https://zookeeper.apache.org/doc/current/zookeeperObservers.html).
Observers don't improve quorum availability. Maintain an odd number of voting participants, typically three or five, and add observers separately. For example, an ensemble with three participants and one observer can tolerate the loss of one participant. An ensemble with two participants and one observer can't tolerate the loss of either participant because the observer doesn't vote.
## Prerequisites
Before you configure observers, make sure that you have:
* A StreamNative Private Cloud v2 deployment that uses ZooKeeper for metadata storage.
* Permission to update the `ZooKeeperCluster` resource and inspect Pods, Services, and ConfigMaps in its namespace.
* The name and namespace of the `ZooKeeperCluster` resource.
* A current backup of the ZooKeeper data.
`spec.advancedOptions.staticServerList` replaces the server list that the StreamNative Operator generates. An incorrect server ID, hostname, or role can prevent the ensemble from forming a quorum. Test the configuration in a non-production environment before you update a production cluster.
## Configure an observer
The following example configures three participants and one observer for a `ZooKeeperCluster` named `private-cloud` in the `pulsar` namespace.
1. Confirm the names of the ZooKeeper StatefulSet and headless Service:
```bash theme={null}
kubectl get statefulset,service -n pulsar | grep private-cloud-zk
```
2. In your existing `ZooKeeperCluster` manifest, set `spec.replicas` to the total number of participants and observers. Then add one server entry for each replica under `spec.advancedOptions.staticServerList`:
```yaml theme={null}
apiVersion: zookeeper.streamnative.io/v1alpha1
kind: ZooKeeperCluster
metadata:
name: private-cloud
namespace: pulsar
spec:
replicas: 4
# Keep the existing image, persistence, Pod, and other settings.
advancedOptions:
staticServerList:
servers:
- id: 1
listeners:
- hostname: private-cloud-zk-0.private-cloud-zk-headless.pulsar.svc.cluster.local
role: participant
- id: 2
listeners:
- hostname: private-cloud-zk-1.private-cloud-zk-headless.pulsar.svc.cluster.local
role: participant
- id: 3
listeners:
- hostname: private-cloud-zk-2.private-cloud-zk-headless.pulsar.svc.cluster.local
role: participant
- id: 4
listeners:
- hostname: private-cloud-zk-3.private-cloud-zk-headless.pulsar.svc.cluster.local
role: observer
```
Use these rules when you adapt the example:
* Set `replicas` to the number of entries in `servers`.
* Add one entry for every Pod ordinal, starting with ordinal `0`.
* Set each server `id` to the Pod ordinal plus 1. ZooKeeper server IDs start at `1`, while StatefulSet Pod ordinals start at `0`. For example, Pod `private-cloud-zk-0` uses `id: 1`, and Pod `private-cloud-zk-3` uses `id: 4`. The latter appears as `server.4` in the generated ZooKeeper configuration.
* Set each voting member's role to `participant` and each non-voting member's role to `observer`.
* Use the fully qualified Pod hostname generated by the StatefulSet and headless Service. The default format is `-zk-.-zk-headless..svc.cluster.local`.
3. Apply the complete manifest:
Applying this change restarts the ZooKeeper Pods. Schedule the change for a maintenance window, and confirm that the ensemble is healthy before you continue.
```bash theme={null}
kubectl apply -f .yaml
```
4. Wait for the `ZooKeeperCluster` to become ready:
```bash theme={null}
kubectl wait -n pulsar \
--for=condition=Ready \
zookeepercluster/private-cloud \
--timeout=300s
```
## Verify the observer
1. Confirm that all ZooKeeper Pods are ready:
```bash theme={null}
kubectl get pods -n pulsar | grep 'private-cloud-zk-'
```
2. Inspect the server configuration generated by the StreamNative Operator:
```bash theme={null}
kubectl get configmap private-cloud-zk-config -n pulsar \
-o jsonpath='{.data.ZOOKEEPER_SERVER_CONFIG}'
```
The configuration contains three `participant` entries and one `observer` entry. The observer entry resembles the following output:
```text theme={null}
server.4=private-cloud-zk-3.private-cloud-zk-headless.pulsar.svc.cluster.local:2888:3888:observer;0.0.0.0:2181
```
3. Check the observer Pod logs:
```bash theme={null}
kubectl logs -n pulsar private-cloud-zk-3 \
| grep -E 'ObserverZooKeeperServer|observing|Observing new leader'
```
Log entries that contain `ObserverZooKeeperServer`, `observing`, or `Observing new leader` confirm that the Pod is running as an observer and has connected to a leader.
## Troubleshoot the configuration
If the ZooKeeper Pods don't become ready, check the following settings:
* `spec.replicas` equals the number of entries in `staticServerList.servers`.
* Every server ID is unique.
* Every hostname matches an existing Pod and the ZooKeeper headless Service.
* Enough servers have the `participant` role to form a voting majority. Observers don't count toward this majority.
Inspect the resource status and Pod logs for more details:
```bash theme={null}
kubectl describe zookeepercluster private-cloud -n pulsar
kubectl logs private-cloud-zk-0 -n pulsar
```
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.