Auto-Scaling automatically adjusts the available resources of your deployments, and eliminates the need for scripts or consulting services to make scaling decisions. It currently supports scaling for Pulsar Broker nodes only and works on a rolling basis, meaning the process doesn't incur any downtime.
You can specify a range of minimum and maximum Pulsar Broker nodes that your Pulsar cluster can automatically scale to, our Cloud scaler will monitor the CPU workload of Broker nodes and adjust their nodes based on the scaling rule.
Hosted users can experience Auto-Scaling after the feature launch, but please note that Auto-Scaling is currently in alpha stage.
BYOC users want to try it out, submit a ticket to the support team.
Overview of the autoscaling
One of the significant challenges faced by organizations dealing with real-time data is ensuring that the underlying infrastructure can handle varying workloads. Traditional scaling methods often involve manual intervention, leading to inefficiency, and increased operational costs. Pulsar Broker Autoscaling addresses these challenges by providing an intelligent, automated solution.
The Power of Pulsar Broker Autoscaling:
Dynamic Resource Allocation: Pulsar Broker Autoscaling dynamically adjusts resources based on the incoming workload. Whether it's handling a sudden spike in traffic or scaling down during periods of low activity, Pulsar ensures optimal resource utilization, leading to cost savings and improved performance.
Efficient Load Balancing: Autoscaling in Pulsar ensures that the message processing load is evenly distributed across brokers. This prevents any single broker from becoming a bottleneck, allowing the system to maintain high throughput and low latency even under heavy loads.
Cost-Effective Scaling: Traditional scaling methods often result in over-provisioning to handle peak loads, leading to unnecessary costs. Pulsar Broker Autoscaling optimizes resource allocation, ensuring that organizations pay only for the resources they need, making it a cost-effective solution for real-time data processing.
When does autoscaling occur?
StreamNative Cloud has observability stack to collect the Pulsar cluster workload status in real-time. After enabling the Broker nodes Auto-Scaling feature, the scaling controller will track the average CPU usage of the Broker nodes and adjusts the Brokers to keep at the target CPU usage level. If the average CPU usage for the Brokers is over the target, scaling controller will scale out to bring more Brokers and if the average CPU for the Brokers is less than the target, scaling controller will downscale brokers to save resources.
Enable or disable autoscaling
To enable or disable the Broker autoscaling:
Log in to the StreamNative Cloud Console
Select into your target Pulsar cluster.
Go to the
Pulsar Clusterspage on the sidebar and click into the
Edit Clusterbutton and find the
Cluster Autoscalingfeature, enable that switch and choose the minimum nodes and maximum nodes.
After enabling the the Broker autoscaling, you can go to the
Dashboardtab on the
Pulsar Clusterspage to view the panel of Broker node counts with time series.