Configure ZooKeeper observers

A ZooKeeper observer is a non-voting member of a ZooKeeper ensemble. An observer receives committed updates and serves client requests, but it doesn’t vote in leader elections or proposals. Add observers when you need to serve more ZooKeeper clients without increasing the size of the voting quorum. For more information about observer behavior, see ZooKeeper Observers. Observers don’t improve quorum availability. Maintain an odd number of voting participants, typically three or five, and add observers separately. For example, an ensemble with three participants and one observer can tolerate the loss of one participant. An ensemble with two participants and one observer can’t tolerate the loss of either participant because the observer doesn’t vote.

Prerequisites

Before you configure observers, make sure that you have:

A StreamNative Private Cloud v2 deployment that uses ZooKeeper for metadata storage.
Permission to update the ZooKeeperCluster resource and inspect Pods, Services, and ConfigMaps in its namespace.
The name and namespace of the ZooKeeperCluster resource.
A current backup of the ZooKeeper data.

spec.advancedOptions.staticServerList replaces the server list that the StreamNative Operator generates. An incorrect server ID, hostname, or role can prevent the ensemble from forming a quorum. Test the configuration in a non-production environment before you update a production cluster.

Configure an observer

The following example configures three participants and one observer for a ZooKeeperCluster named private-cloud in the pulsar namespace.

Confirm the names of the ZooKeeper StatefulSet and headless Service:

kubectl get statefulset,service -n pulsar | grep private-cloud-zk

In your existing ZooKeeperCluster manifest, set spec.replicas to the total number of participants and observers. Then add one server entry for each replica under spec.advancedOptions.staticServerList:

apiVersion: zookeeper.streamnative.io/v1alpha1
kind: ZooKeeperCluster
metadata:
  name: private-cloud
  namespace: pulsar
spec:
  replicas: 4
  # Keep the existing image, persistence, Pod, and other settings.
  advancedOptions:
    staticServerList:
      servers:
        - id: 1
          listeners:
            - hostname: private-cloud-zk-0.private-cloud-zk-headless.pulsar.svc.cluster.local
              role: participant
        - id: 2
          listeners:
            - hostname: private-cloud-zk-1.private-cloud-zk-headless.pulsar.svc.cluster.local
              role: participant
        - id: 3
          listeners:
            - hostname: private-cloud-zk-2.private-cloud-zk-headless.pulsar.svc.cluster.local
              role: participant
        - id: 4
          listeners:
            - hostname: private-cloud-zk-3.private-cloud-zk-headless.pulsar.svc.cluster.local
              role: observer

Use these rules when you adapt the example:

Set replicas to the number of entries in servers.
Add one entry for every Pod ordinal, starting with ordinal 0.
Set each server id to the Pod ordinal plus 1. ZooKeeper server IDs start at 1, while StatefulSet Pod ordinals start at 0. For example, Pod private-cloud-zk-0 uses id: 1, and Pod private-cloud-zk-3 uses id: 4. The latter appears as server.4 in the generated ZooKeeper configuration.
Set each voting member’s role to participant and each non-voting member’s role to observer.
Use the fully qualified Pod hostname generated by the StatefulSet and headless Service. The default format is <cluster-name>-zk-<ordinal>.<cluster-name>-zk-headless.<namespace>.svc.cluster.local.

Apply the complete manifest:
Applying this change restarts the ZooKeeper Pods. Schedule the change for a maintenance window, and confirm that the ensemble is healthy before you continue.
kubectl apply -f <zookeeper-cluster-manifest>.yaml

Wait for the ZooKeeperCluster to become ready:

kubectl wait -n pulsar \
  --for=condition=Ready \
  zookeepercluster/private-cloud \
  --timeout=300s

Verify the observer

Confirm that all ZooKeeper Pods are ready:

kubectl get pods -n pulsar | grep 'private-cloud-zk-'

Inspect the server configuration generated by the StreamNative Operator:

kubectl get configmap private-cloud-zk-config -n pulsar \
  -o jsonpath='{.data.ZOOKEEPER_SERVER_CONFIG}'

The configuration contains three participant entries and one observer entry. The observer entry resembles the following output:

server.4=private-cloud-zk-3.private-cloud-zk-headless.pulsar.svc.cluster.local:2888:3888:observer;0.0.0.0:2181

Check the observer Pod logs:
kubectl logs -n pulsar private-cloud-zk-3 \ | grep -E 'ObserverZooKeeperServer|observing|Observing new leader'
Log entries that contain ObserverZooKeeperServer, observing, or Observing new leader confirm that the Pod is running as an observer and has connected to a leader.

Troubleshoot the configuration

If the ZooKeeper Pods don’t become ready, check the following settings:

spec.replicas equals the number of entries in staticServerList.servers.
Every server ID is unique.
Every hostname matches an existing Pod and the ZooKeeper headless Service.
Enough servers have the participant role to form a voting majority. Observers don’t count toward this majority.

Inspect the resource status and Pod logs for more details:

kubectl describe zookeepercluster private-cloud -n pulsar
kubectl logs private-cloud-zk-0 -n pulsar

Overview

Quick Start

Configure Private Cloud

Operate Private Cloud

References

Prerequisites

Configure an observer

Verify the observer

Troubleshoot the configuration

​Prerequisites

​Configure an observer

​Verify the observer

​Troubleshoot the configuration

Prerequisites

Configure an observer

Verify the observer

Troubleshoot the configuration