Resize a Cluster

You can expand the number of CUs and SUs in the StreamNative Cloud Console.

1. Log in to the StreamNative Cloud Console.

2. Navigate to the Cluster Workspace.

3. Click Configuration in the left navigation pane to access Cluster Configuration.

4. On the Cluster Configuration page, click the Edit Cluster button in the top-right corner. You will be redirected to the Cluster Provisioning page where you can update the cluster configuration and adjust the sizing of the cluster.

5. Skip the Cluster Configuration page by clicking the Cluster Size button. You will be redirected to the Cluster Size page.

6. On the Cluster Size page, you can choose between Basic or Advanced mode to expand the cluster.

   • Basic Mode: Use the slider to adjust the required throughput. StreamNative Cloud will automatically calculate the number of brokers and bookies, as well as their respective CUs and SUs per node.

   • Advanced Mode: Manually adjust the number of brokers, bookies, CUs, and SUs per node.

7. Click the Finish button to complete the cluster expansion.

The cluster will start scaling up, and you will be redirected to a Deploying page that shows the progress of the deployment process. The deployment might take a few minutes to complete, depending on how many resources you are scaling up. You can always click Go To The Dashboard to return to the Cluster Dashboard page. You can see the number of brokers and bookies in real-time on the Cluster Dashboard page.

CUs and SUs are reflected as the number of CPU and Memory used by the brokers and bookies. When expanding a cluster via snctl, you can modify the number of bookies and brokers, as well as their corresponding CPU, Memory, and Disk resources. This is done by updating the PulsarCluster Custom Resource Definition (CRD).

To expand your cluster using snctl, follow these steps:

1. Retrieve the current cluster configuration:

   Copy

   Ask AI

   snctl get -O <organization-id> pulsarcluster <cluster-name> -o yaml > cluster-config.yaml
   

2. Edit the cluster-config.yaml file to increase the resources. For example:

   • Increase the number of brokers or bookies:

     Copy

     Ask AI

     spec:
       broker:
         replicas: 5 # Increase from default 3 to 5
       bookkeeper:
         replicas: 4 # Increase from default 3 to 4
     

   • Adjust the CPU, memory, or storage for brokers or bookies:

     Copy

     Ask AI

     spec:
       broker:
         resources:
           cpu: '4'
           memory: '8Gi'
           heapPercentage: 50
           directPercentage: 50
       bookkeeper:
         resources:
           cpu: '4'
           memory: '8Gi'
           heapPercentage: 50
           directPercentage: 50
           ledgerDisk: '100Gi'
           journalDisk: '50Gi'
     

   These examples demonstrate how to modify the PulsarCluster CRD to scale your cluster’s resources.

3. Apply the updated configuration:

   Copy

   Ask AI

   snctl apply -f cluster-config.yaml
   

After snctl apply is completed, the cluster will initiate the expansion process. The operation may take several minutes to complete, depending on the scale of the expansion.

You can check the status of your PulsarCluster to monitor the progress. The PulsarCluster resource includes a status block that provides valuable information about the current state of your cluster components.

To check the status, you can use the following command:

snctl clusters -O <organization-id> get <cluster-name> -o yaml

Here’s an example of what the status block might look like:

status:
  bookkeeper:
    readyReplicas: 3
    replicas: 3
    updatedReplicas: 3
  broker:
    readyReplicas: 3
    replicas: 3
    updatedReplicas: 3

When all the readyReplicas, replicas, and updatedReplicas are the same, the cluster expansion is complete.

CUs (Compute Units) and SUs (Storage Units) are reflected in the CPU and memory resources used by brokers and bookies, respectively. When expanding a cluster via the Terraform Provider, you can modify the number of bookies and brokers, as well as the CUs per broker and SUs per bookie. This is accomplished by updating the streamnative_pulsar_cluster resource.

To resize a cluster using the StreamNative Terraform Provider, follow these steps:

1. Update your Terraform configuration file (e.g., main.tf) to modify the cluster resources. Here’s an example of how to update the cluster configuration:

   Copy

   Ask AI

   resource "streamnative_pulsar_cluster" "my_cluster" {
     # ... other configuration ...
   
     broker_replicas = 5  # Increase the number of broker replicas
     bookie_replicas = 5  # Increase the number of bookkeeper replicas
     compute_unit    = 0.4  # Increase compute units per broker
     storage_unit    = 0.4  # Increase storage units per bookie
   
     config {
       # ... other configuration ...
     }
   }
   

   Copy

   Ask AI

   
   This example demonstrates how to modify the `streamnative_pulsar_cluster` resource to scale your cluster's resources.
   
   

2. After updating the configuration, run the following Terraform commands:

   Copy

   Ask AI

   terraform plan
   

   This command will show you the planned changes to your StreamNative cluster.

3. If the plan looks correct, apply the changes:

   Copy

   Ask AI

   terraform apply
   

   Terraform will then initiate the cluster expansion process. The operation may take several minutes to complete, depending on the scale of the expansion.

4. You can monitor the progress of the cluster scaling operation by checking the Terraform output or by using the Cluster Dashboard on StreamNative Cloud console.

5. To verify the status of your cluster after the operation, you can use the terraform show command or check the Cluster Dashboard on StreamNative Cloud console for the most up-to-date information on your cluster’s resources and status.

Remember to always review the changes carefully before applying them to your production environment. It’s recommended to test any scaling operations in a non-production environment first.