Configure Auto-Scaling

Auto-Scaling automatically adjusts the available resources of your deployments, and eliminates the need for scripts or consulting services to make scaling decisions. It currently supports scaling for Pulsar Broker nodes only and works on a rolling basis, meaning the process doesn't incur any downtime.

You can specify a range of minimum and maximum Pulsar Broker nodes that your Pulsar cluster can automatically scale to, our Cloud scaler will monitor the CPU workload of Broker nodes and adjust their nodes based on the scaling rule.

Overview of the autoscaling

One of the significant challenges faced by organizations dealing with real-time data is ensuring that the underlying infrastructure can handle varying workloads. Traditional scaling methods often involve manual intervention, leading to inefficiency, and increased operational costs. Pulsar Broker Autoscaling addresses these challenges by providing an intelligent, automated solution.

The Power of Pulsar Broker Autoscaling:

• Dynamic Resource Allocation: Pulsar Broker Autoscaling dynamically adjusts resources based on the incoming workload. Whether it's handling a sudden spike in traffic or scaling down during periods of low activity, Pulsar ensures optimal resource utilization, leading to cost savings and improved performance.

• Efficient Load Balancing: Autoscaling in Pulsar ensures that the message processing load is evenly distributed across brokers. This prevents any single broker from becoming a bottleneck, allowing the system to maintain high throughput and low latency even under heavy loads.

• Cost-Effective Scaling: Traditional scaling methods often result in over-provisioning to handle peak loads, leading to unnecessary costs. Pulsar Broker Autoscaling optimizes resource allocation, ensuring that organizations pay only for the resources they need, making it a cost-effective solution for real-time data processing.

When does autoscaling occur?

StreamNative Cloud has observability stack to collect the Pulsar cluster workload status in real-time. After enabling the Broker nodes Auto-Scaling feature, the scaling controller will track the average CPU usage of the Broker nodes and adjusts the Brokers to keep at the target CPU usage level. If the average CPU usage for the Brokers is over the target, scaling controller will scale out to bring more Brokers and if the average CPU for the Brokers is less than the target, scaling controller will downscale brokers to save resources.

Enable or disable autoscaling

To enable or disable the Broker autoscaling:

1. Log in to the StreamNative Cloud Console

2. Select into your target Pulsar cluster.

3. Go to the Pulsar Clusters page on the sidebar and click into the Configuration tab.

4. Click the Edit Cluster button and find the Cluster Autoscaling feature, enable that switch and choose the minimum nodes and maximum nodes.

5. After enabling the the Broker autoscaling, you can go to the Dashboard tab on the Pulsar Clusters page to view the panel of Broker node counts with time series.