Pulsar 3.0 introduces a new load manager implementation, aiming to balance cluster utilization as evenly as possible while minimizing latency and reducing reliance on ZooKeeper. The new load manager stores this load data in non-persistent topics, while topic ownership information is saved in system topics and maintained with a state machine for eventual consistency. Clients can now connect to any broker for lookup without redirection.

With the new Broker Load Balancer can improve the:

  • Efficiency:
    • Reduce idle brokers and save cluster resources.
    • Distribute data loads evenly and efficiently.
  • Performance
    • Reduce hot spots and maintain message pub/sub latency.
    • Minimize the likelihood of Pulsar cluster downtime.
  • Availability
    • Minimize topic unavailable time by shifting pub/sub sessions from unavailable brokers to available brokers.
    • Perform cluster maintenance without causing service disruption since pub/sub connections get rerouted to other brokers during maintenance.
  • Scalability
    • Unload topic loads automatically to new brokers when scaling up.
    • Detect orphan topics and automatically and reassign them to available brokers when scaling down.

​
How to enable the new Load Balancer

Pulsar 3.0 by default uses the previous ModularLoadManagerImpl loadManagerClass and the new Load Balancer extra steps to manually enable.

Before using the new Load Balancer, please make sure upgrading the Pulsar to 3.0. Don’t recommand to upgrade Pulsar from 2.x and enable the the Load Balancer in the same process which may cause unexpected problems.

After your Pulsar is in 3.0, you can enable the new Load Balancer by adding the configuraitons under the PulsarBroker CR:

spec:
  config:
    custom:
      PULSAR_PREFIX_loadManagerClassName: "org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl"
      PULSAR_PREFIX_loadBalancerLoadSheddingStrategy: "org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder"

​
More resources

Pulsar 3.0 introduces a new load manager implementation, aiming to balance cluster utilization as evenly as possible while minimizing latency and reducing reliance on ZooKeeper. The new load manager stores this load data in non-persistent topics, while topic ownership information is saved in system topics and maintained with a state machine for eventual consistency. Clients can now connect to any broker for lookup without redirection.

With the new Broker Load Balancer can improve the:

  • Efficiency:
    • Reduce idle brokers and save cluster resources.
    • Distribute data loads evenly and efficiently.
  • Performance
    • Reduce hot spots and maintain message pub/sub latency.
    • Minimize the likelihood of Pulsar cluster downtime.
  • Availability
    • Minimize topic unavailable time by shifting pub/sub sessions from unavailable brokers to available brokers.
    • Perform cluster maintenance without causing service disruption since pub/sub connections get rerouted to other brokers during maintenance.
  • Scalability
    • Unload topic loads automatically to new brokers when scaling up.
    • Detect orphan topics and automatically and reassign them to available brokers when scaling down.

​
How to enable the new Load Balancer

Pulsar 3.0 by default uses the previous ModularLoadManagerImpl loadManagerClass and the new Load Balancer extra steps to manually enable.

Before using the new Load Balancer, please make sure upgrading the Pulsar to 3.0. Don’t recommand to upgrade Pulsar from 2.x and enable the the Load Balancer in the same process which may cause unexpected problems.

After your Pulsar is in 3.0, you can enable the new Load Balancer by adding the configuraitons under the PulsarBroker CR:

spec:
  config:
    custom:
      PULSAR_PREFIX_loadManagerClassName: "org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl"
      PULSAR_PREFIX_loadBalancerLoadSheddingStrategy: "org.apache.pulsar.broker.loadbalance.extensions.scheduler.TransferShedder"

​
More resources