Configure Auto-Scaling

Note

Serverless clusters are elastic and automatically scale based on workloads. There is no need to configure autoscaling.

Auto-Scaling automatically adjusts the available resources of your deployments, and eliminates the need for scripts or consulting services to make scaling decisions. It currently supports scaling for Pulsar Broker nodes only and works on a rolling basis, meaning the process doesn't incur any downtime.

Note

Auto-scaling for BookKeeper clusters is not supported at this time.

You can specify a range of minimum and maximum Pulsar Broker nodes that your Pulsar cluster can automatically scale to, our Cloud auto-scaler will monitor the CPU workload of Broker nodes and adjust their nodes based on the scaling rule.

Overview of autoscaling

One of the significant challenges faced by organizations dealing with real-time data is ensuring that the underlying infrastructure can handle varying workloads. Traditional scaling methods often involve manual intervention, leading to inefficiency and increased operational costs. Pulsar Broker Autoscaling addresses these challenges by providing an intelligent, automated solution.

The Power of Pulsar Broker Autoscaling:

Dynamic Resource Allocation: Pulsar Broker Autoscaling dynamically adjusts resources based on the incoming workload. Whether it's handling a sudden spike in traffic or scaling down during periods of low activity, Pulsar ensures optimal resource utilization, leading to cost savings and improved performance.
Efficient Load Balancing: Autoscaling in Pulsar ensures that the message processing load is evenly distributed across brokers. This prevents any single broker from becoming a bottleneck, allowing the system to maintain high throughput and low latency even under heavy loads.
Cost-Effective Scaling: Traditional scaling methods often result in over-provisioning to handle peak loads, leading to unnecessary costs. Pulsar Broker Autoscaling optimizes resource allocation, ensuring that organizations pay only for the resources they need, making it a cost-effective solution for real-time data processing.

When does autoscaling occur?

StreamNative Cloud has an observability stack to collect the Pulsar cluster workload status in real-time. After enabling the Broker nodes Auto-Scaling feature, the scaling controller will track the average CPU usage of the Broker nodes and adjust the Brokers to keep them at the target CPU usage level. If the average CPU usage for the Brokers is over the target, the scaling controller will scale out to bring in more Brokers. If the average CPU usage for the Brokers is less than the target, the scaling controller will downscale brokers to save resources.

Enable or disable autoscaling

Auto-scaling is a cluster-level feature that allows you to dynamically adjust the number of Broker nodes. Here's how you can enable or disable Broker auto-scaling for your cluster:

Log in to the StreamNative Cloud Console.
Navigate to your target Cluster Workspace.
Click Configuration in the left navigation pane to access Cluster Configuration.
On the Cluster Configuration page, click the Edit Cluster button in the top-right corner. You will be redirected to the Cluster Provisioning page where you can update the cluster configuration.
Find the Cluster Autoscaling feature, enable the switch, and choose the minimum and maximum number of nodes.
Click the Cluster Size button to move to the next step. Skip the Cluster Size page by clicking the Finish button to complete the operation.
After enabling Broker autoscaling, you can go back to the Cluster Dashboard page to check the number of Broker nodes.

For other cluster features, You can follow Configure Cluster Features to enable or disable Broker autoscaling.