Configure Auto-Scaling

Auto-Scaling automatically adjusts the available resources of your deployments, and eliminates the need for scripts or consulting services to make scaling decisions. It currently supports scaling for Pulsar nodes only and works on a rolling basis, meaning the process doesn't incur any downtime.

You can specify a range of minimum and maximum Pulsar nodes that your Pulsar cluster can automatically scale to, our Cloud scaler will monitor the CPU workload of Broker nodes and adjust their nodes based on the scaling rule.

Overview of Auto-Scaling

One of the significant challenges faced by organizations dealing with real-time data is ensuring that the underlying infrastructure can handle varying workloads. Traditional scaling methods often involve manual intervention, leading to inefficiency, and increased operational costs. Private Cloud Auto-Scaling addresses these challenges by providing an intelligent, automated solution.

The Power of Private Cloud Auto-Scaling:

Dynamic Resource Allocation: Private Cloud Auto-Scaling dynamically adjusts resources based on the incoming workload. Whether it's handling a sudden spike in traffic or scaling down during periods of low activity, Pulsar ensures optimal resource utilization, leading to cost savings and improved performance.
Efficient Load Balancing: Auto-Scaling in Pulsar ensures that the message processing load is evenly distributed across brokers. This prevents any single broker from becoming a bottleneck, allowing the system to maintain high throughput and low latency even under heavy loads.
Cost-Effective Scaling: Traditional scaling methods often result in over-provisioning to handle peak loads, leading to unnecessary costs. Private Cloud Auto-Scaling optimizes resource allocation, ensuring that organizations pay only for the resources they need, making it a cost-effective solution for real-time data processing.

After enabling the Auto-Scaling, the scaling controller will track the average metrics usage of the Pulsar nodes and adjusts the nodes to keep at the target metrics usage level. If the average metrics usage for nodes is over the target, scaling controller will scale out to bring more nodes and if the average metrics for the nodes is less than the target, scaling controller will downscale nodes to save resources.

Auto-Scaling with resource metrics

Prerequisites

Private Cloud Auto-Scaling by default uses resource metrics for Pod scaling which requires the Kubernetes Metrics Server to provide metrics.

Deploy the Metrics Server with the following command:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Verify that the metrics-server deployment is running

kubectl get deployment metrics-server -n kube-system

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           7m14s

Enable Auto-Scaling with resource metrics

Private Cloud Auto-Scaling has default scaling policy basd on cpu Utilization as

type: Resource
resource:
  name: cpu
  target:
    type: Utilization
    averageUtilization: 80

After installing themetrics-server, you can easily enable the Auto-Scaling with spec.autoScalingPolicy field on PulsarBroker CR and PulsarProxy CR:

apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarBroker
metadata:
  name: brokers
  namespace: pulsar
  labels:
    k8s.streamnative.io/coordinator-name: private-cloud
spec:
  image: streamnative/private-cloud:3.0.1.4
  replicas: 3
  zkServers: zookeepers-zk:2181
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
    securityContext:
      runAsNonRoot: true
  autoScalingPolicy:
    minReplicas: 1
    maxReplicas: 5
---
apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarProxy
metadata:
  name: proxys
  namespace: pulsar
  labels:
    k8s.streamnative.io/coordinator-name: private-cloud
spec:
  image: streamnative/private-cloud:3.0.1.4
  replicas: 3
  brokerAddress: brokers-broker
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
    securityContext:
      runAsNonRoot: true
  autoScalingPolicy:
    minReplicas: 1
    maxReplicas: 5

Auto-Scaling with custom metrics

Private Cloud Auto-Scaling supports configuring scaling policy based on custom metrics and multiple metrics to cover more complex scenario.

Prerequisites

To use the custom metrics Auto-Scaling, we need to install the Prometheus and Prometheus Adapter to provide metrics.

Create Kubernetes namespace for Promtheus and Prometheus Adapter:

kubectl create ns monitor

Add prometheus-community Helm repo

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Install the Prometheus chart

helm install prometheus prometheus-community/prometheus -n monitor --set alertmanager.enabled=false --set kube-state-metrics.enabled=false --set prometheus-node-exporter.enabled=false --set prometheus-pushgateway.enabled=false

Preapre a values config sample-config.yaml for Prometheus Adapter chart

prometheus:
  url: http://prometheus-server.monitor.svc
  port: 80
listenPort: 8443
rules:
  default: false
  custom:
    - seriesQuery: '{namespace!="",__name__!~"^container_.*"}'
      resources:
        template: "<<.Resource>>"
      name:
        matches: "^(.*)_total"
        as: ""
      metricsQuery: |
        sum by (<<.GroupBy>>) (
          irate (
            <<.Series>>{<<.LabelMatchers>>}[1m]
          )
        )
    - seriesQuery: 'container_cpu_usage_seconds_total{namespace!~"(sn-system|kube-system|olm|cert-manager)"}'
      seriesFilters: []
      resources:
        overrides:
          pod:
            resource: pod
          namespace:
            resource: namespace
      name:
        matches: "container_cpu_usage_seconds_total"
        as: "cpu_usage"
      metricsQuery: sum(rate(container_cpu_usage_seconds_total{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>) / (sum(container_spec_cpu_shares{<<.LabelMatchers>>}/1000) by (<<.GroupBy>>))  * 100
    - seriesQuery: 'container_network_receive_bytes_total{namespace!~"(sn-system|kube-system|olm|cert-manager)"}'
      seriesFilters: []
      resources:
        overrides:
          pod:
            resource: pod
          namespace:
            resource: namespace
      name:
        matches: "container_network_receive_bytes_total"
        as: "network_in_rate_kb"
      metricsQuery: rate(container_network_receive_bytes_total{<<.LabelMatchers>>}[5m]) / 1024
    - seriesQuery: 'container_network_transmit_bytes_total{namespace!~"(sn-system|kube-system|olm|cert-manager)"}'
      seriesFilters: []
      resources:
        overrides:
          pod:
            resource: pod
          namespace:
            resource: namespace
      name:
        matches: "container_network_transmit_bytes_total"
        as: "network_out_rate_kb"
      metricsQuery: rate(container_network_transmit_bytes_total{<<.LabelMatchers>>}[5m]) / 1024
    - seriesQuery: 'container_fs_reads_bytes_total{namespace!~"(sn-system|kube-system|olm|cert-manager)"}'
      seriesFilters: []
      resources:
        overrides:
          pod:
            resource: pod
          namespace:
            resource: namespace
      name:
        matches: "container_fs_reads_bytes_total"
        as: "disk_read_rate_kb"
      metricsQuery: sum(rate(container_fs_reads_bytes_total{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>) / 1024
    - seriesQuery: 'container_fs_writes_bytes_total{namespace!~"(sn-system|kube-system|olm|cert-manager)"}'
      seriesFilters: []
      resources:
        overrides:
          pod:
            resource: pod
          namespace:
            resource: namespace
      name:
        matches: "container_fs_writes_bytes_total"
        as: "disk_write_rate_kb"
      metricsQuery: sum(rate(container_fs_writes_bytes_total{<<.LabelMatchers>>}[5m])) by (<<.GroupBy>>) / 1024

Install the Prometheus Adapter chart

helm install prometheus-adapter -f sample-config.yaml prometheus-community/prometheus-adapter -n monitor

Enable Auto-Scaling with custom metrics

After installing the Promtheus and Prometheus Adapter, you can configure to use custom metrics at the spec.autoScalingPolicy.metrics:

spec:
  autoScalingPolicy:
    minReplicas: 1
    maxReplicas: 5
    metrics:
    - pods:
        metric:
          name: cpu_usage
        target:
          averageValue: "75"
          type: AverageValue
      type: Pods
    - pods:
        metric:
          name: network_in_rate_kb
        target:
          averageValue: "204800"
          type: AverageValue
      type: Pods
    - pods:
        metric:
          name: network_out_rate_kb
        target:
          averageValue: "204800"
          type: AverageValue
      type: Pods

This configuration will leverage cpu_usage, network_in_rate_kb and network_out_rate_kb for Auto-Scaling.

To customize the scaling policy for Auto-Scaling, you can configure at the spec.autoScalingPolicy.behavior::

spec:
  Auto-ScalingPolicy:
    minReplicas: 1
    maxReplicas: 5
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300
        policies:
        - type: Percent
          value: 100
          periodSeconds: 15
      scaleUp:
        stabilizationWindowSeconds: 0
        policies:
        - type: Percent
          value: 100
          periodSeconds: 15
        - type: Pods
          value: 4
          periodSeconds: 15
        selectPolicy: Max

[1] scaleDown: Define the behavior while scaling down.
[2] scaleUp: Define the behavior while scaling up.
[3] stabilizationWindowSeconds: Stabilization window is used to restrict the flapping of replica count when the metrics used for scaling keep fluctuating.
[4] policies: Define the scaling policy, you can limit the scaling number or percent in the periodSeconds.