Configure Broker Load Balancer

sn-platform 1.11 uses the Pulsar 3.0 as the default which introduces a new load manager implementation, aiming to balance cluster utilization as evenly as possible while minimizing latency and reducing reliance on ZooKeeper. The new load manager stores this load data in non-persistent topics, while topic ownership information is saved in system topics and maintained with a state machine for eventual consistency. Clients can now connect to any broker for lookup without redirection.

With the new Broker Load Balancer can improve the:

• Efficiency:
  • Reduce idle brokers and save cluster resources.
  • Distribute data loads evenly and efficiently.
• Performance
  • Reduce hot spots and maintain message pub/sub latency.
  • Minimize the likelihood of Pulsar cluster downtime.
• Availability
  • Minimize topic unavailable time by shifting pub/sub sessions from unavailable brokers to available brokers.
  • Perform cluster maintenance without causing service disruption since pub/sub connections get rerouted to other brokers during maintenance.
• Scalability
  • Unload topic loads automatically to new brokers when scaling up.
  • Detect orphan topics and automatically and reassign them to available brokers when scaling down.

How to enable the new Load Balancer

Pulsar 3.0 by default uses the previous ModularLoadManagerImpl loadManagerClass and the new Load Balancer extra steps to manually enable.

After your Pulsar is in 3.0, you can enable the new Load Balancer by adding the configuraitons under the broker.configData field in the helm values. And use the helm upgrade command to update the configuration.

broker:
  configData:
    PULSAR_PREFIX_loadManagerClassName: "org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl"
    PULSAR_PREFIX_loadBalancerLoadSheddingStrategy: "org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder"

More resources

• Broker Load Balancer documentation overview
• Achieving Broker Load Balancing with Apache Pulsar blog