Work with clusters

A cluster is a set of Pulsar brokers and bookies. After creating an instance, you can create a cluster for the instance.

Only one cluster is supported for an instance.

Work with clusters through snctl

Before you can create a cluster through snctl, you need to create a billing subscription. For more information, see subscribe to StreamNative Cloud.

In this document, we named the organization matrix for an example.

Create a cluster through snctl

To create a cluster, follow these steps.

Define a cluster named neo-1 by using a manifest file and save the manifest file clusterneo1.yaml.

apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarCluster
metadata:
  namespace: matrix
  name: neo-1
spec:
  instanceName: neo
  location: us-east4
  bookkeeper:
    replicas: 3
    resourceSpec:
      nodeType: tiny-1
  broker:
    replicas: 1
    resourceSpec:
      nodeType: tiny-1
  config:
    custom:
      backlogQuotaDefaultLimitBytes: '1000000000'
    websocketEnabled: true

The following table lists fields in the manifest file.

Field	Description
apiVersion	Specify the version of Pulsar API server.
kind	Specify the component to be created.
metadata	Configure the metadata information about the cluster. - `namespace`: specify the name of the organization. - `name`: specify the name of the cluster.
spec	Configure the specifications about the cluster. - `instanceName`: specify the name of the instance where the cluster belongs to. - `location`: specify the location of the cluster. - `broker`: configure the broker, including the number of brokers and the resource type (node type). - `bookkeeper`: configure the bookie, including the number of bookies and the resource type (node type). - `config`: specify customized cluster configurations and configure cluster features.

Apply the manifest file to create the cluster.
```
snctl apply -f /path/to/clusterneo1.yaml
```

Check whether the cluster is created successfully.

snctl describe pulsarcluster neo-1

Output

Name:         neo-1
Namespace:    matrix
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cloud.streamnative.io/v1alpha1","kind":"PulsarCluster","metadata":{"annotations":{},"name":"neo-1","namespace":"matrix"},...
API Version:  cloud.streamnative.io/v1alpha1
Kind:         PulsarCluster
Metadata:
  Creation Timestamp:  2021-10-12T01:49:30Z
  Finalizers:
    pulsarcluster.finalizers.cloud.streamnative.io
  Generation:  1
  Managed Fields:
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:bookkeeper:
          .:
          f:replicas:
          f:resourceSpec:
            .:
            f:nodeType:
        f:broker:
          f:replicas:
          f:resourceSpec:
            .:
            f:nodeType:
        f:config:
          .:
          f:custom:
            .:
            f:backlogQuotaDefaultLimitBytes:
          f:websocketEnabled:
        f:instanceName:
        f:location:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2021-10-12T01:49:29Z
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"pulsarcluster.finalizers.cloud.streamnative.io":
    Manager:         controller-manager
    Operation:       Update
    Time:            2021-10-12T01:49:30Z
  Resource Version:  11761383
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarclusters/neo-1
  UID:               b1409658-e9f5-4056-9dae-0d484b230607
Spec:
  Bookkeeper:
    Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
    Replicas:  3
    Resource Spec:
      Node Type:  tiny-1
  Broker:
    Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
    Replicas:  1
    Resource Spec:
      Node Type:  tiny-1
  Config:
    Custom:
      Backlog Quota Default Limit Bytes:  1000000000
    Websocket Enabled:                    true
  Instance Name:                          ins-2
  Location:                               us-east4
  Pool Member Ref:
    Name:       us-east4
    Namespace:  streamnative
  Service Endpoints:
    Dns Name:  neo-1.matrix.sn2.dev
    Type:      service
    Dns Name:  neo-1.matrix.us-east4.streamnative.g.sn2.dev
Status:
Events:  <none>

Check the cluster through snctl

You can use the following command to list all created clusters.

snctl get pulsarcluster

Output

NAME            CREATED AT
cluster-test   2020-08-14T01:48:46Z
neo-1          2021-10-12T01:49:29Z
test11         2020-08-12T13:24:19Z

From the output of this command, you can see all created clusters and the time when these clusters were created.

Check cluster details through snctl

Before checking the details about a cluster, you should use the following command to confirm whether the target cluster is available.

snctl get pulsarcluster

Then, you can use the following command to check details about a specific cluster.

snctl describe pulsarcluster <cluster_name>

The following example checks details about the instance neo-1.

snctl describe pulsarcluster neo-1

Output

Name:         neo-1
Namespace:    matrix
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cloud.streamnative.io/v1alpha1","kind":"PulsarCluster","metadata":{"annotations":{},"name":"neo-1","namespace":"matrix"},...
API Version:  cloud.streamnative.io/v1alpha1
Kind:         PulsarCluster
Metadata:
  Creation Timestamp:  2021-10-12T01:49:30Z
  Finalizers:
    pulsarcluster.finalizers.cloud.streamnative.io
  Generation:  1
  Managed Fields:
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:bookkeeper:
          .:
          f:replicas:
          f:resourceSpec:
            .:
            f:nodeType:
        f:broker:
          f:replicas:
          f:resourceSpec:
            .:
            f:nodeType:
        f:config:
          .:
          f:custom:
            .:
            f:backlogQuotaDefaultLimitBytes:
          f:websocketEnabled:
        f:instanceName:
        f:location:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2021-10-12T01:49:29Z
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"pulsarcluster.finalizers.cloud.streamnative.io":
    Manager:         controller-manager
    Operation:       Update
    Time:            2021-10-12T01:49:30Z
  Resource Version:  11761383
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarclusters/neo-1
  UID:               b1409658-e9f5-4056-9dae-0d484b230607
Spec:
  Bookkeeper:
    Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
    Replicas:  3
    Resource Spec:
      Node Type:  tiny-1
  Broker:
    Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
    Replicas:  1
    Resource Spec:
      Node Type:  tiny-1
  Config:
    Custom:
      Backlog Quota Default Limit Bytes:  1000000000
    Websocket Enabled:                    true
  Instance Name:                          ins-2
  Location:                               us-east4
  Pool Member Ref:
    Name:       us-east4
    Namespace:  streamnative
  Service Endpoints:
    Dns Name:  neo-1.matrix.sn2.dev
    Type:      service
    Dns Name:  neo-1.matrix.us-east4.streamnative.g.sn2.dev
Status:
Events:  <none>

Add clusters to the Pulsar configuration file

This section describes how to add a cluster to the Pulsar client configuration file by defining a context for that cluster. Then, you can access the cluster using the pulsarctl CLI tool.

Add a cluster to the Pulsar client configuration file.
This example adds a neo-1 cluster to the Pulsar client configuration file.
Input
```
snctl x update-pulsar-config --cluster-name <neo-1>
```
Output
```
Updated Pulsar client configuration for context 'neo-1'.
```
Verify that the current context has been changed.
Input
```
pulsarctl context current
```
Output
```
neo-1
```

Then, you can use the pulsarctl CLI tool to interact with the target cluster. For details, see Connect to cluster through pulsarctl and perform other operations.

Delete a cluster through snctl

You cannot delete a cluster if there are resources associated with the cluster.

You can use the following command to delete a specific cluster based on the cluster name.

snctl delete pulsarcluster <cluster_name>

In addition, you can use the following command to delete the cluster based on the cluster name specified in the manifest file.

snctl delete -f ./clusterneo1.yaml

Work with clusters through StreamNative Cloud Console

This section describes how to work with clusters through the StreamNative Cloud Console.

Create a cluster through StreamNative Cloud Console

To create a cluster, follow these steps.

On the left navigation pane, click Dashboard.
On the Provision a cluster card, click Create a cluster.
Select Deploy Hosted to start the instance creation process.
On the Instance Configuration page, enter a name for your instance, select an infrastructure pool, and select the multi Availability Zone (AZ).
Click Cluster Location to start the cluster creation process.
Enter a name for your cluster, select the cluster location, and then click Cluster Size.
Use the slider to select your cluster size. There are 5 pre-selected sizes you can choose from. The Approximate Cost section updates as you move the slider.
(Optional) Click the Advanced tab to select custom sizing options.
(Optional) In the Features section, enable the features you want on your cluster. You can enable or disable these features at any time by editing your cluster after it has been created.

(Optional) Expand the Add optional custom configurations section to specify optional custom configuration parameters.

Item	Description
Backlog Quota Size	Enter the backlog quota size.
Backlog Retention Policy	Enter the retention policy when the backlog quota threshold is reached. - `producer_request_hold`: the broker holds producers' sent requests until the resource becomes available or until the holding times out. - `producer_exception`: the broker rejects producers' send requests. - `consumer_backlog_eviction`: the broker evicts the oldest messages from the slowest consumer's backlog.
Max Producers Per Topic	Enter the maximum number of producers allowed to connect to a topic.
Max Consumers Per Topic	Enter the maximum number of consumers allowed to connect to a topic.
Max Consumers Per Subscription	Enter the maximum number of consumers allowed to connect to a subscription.
Dispatch Rate Per Topic	Enter the topic-based dispatch rate. - Throughput (bytes/second): specify the topic-based dispatch rate (in bytes/second). - Rate (messages/second): specify the topic-based dispatch rate (in messages/second).
Dispatch Rate Per Subscription	Enter the subscription-based dispatch rate. - Throughput (bytes/second): specify the subscription-based dispatch rate (in bytes/second). - Rate (messages/second): specify the subscription-based dispatch rate (in messages/second).
Subscribe Rate Per Consumer	Enter the subscribe rate for each consumer. Rate (subscribes/second): specify the subscribe rate for each consumer (in subscribes/second).

Click Finish. The cluster page displays, showing the cluster creation process. The cluster is ready for use after all components have been successfully deployed.

Check cluster details through StreamNative Cloud Console

You can check the cluster for a specific instance, including the cluster name and service URLs. Currently, only one cluster is available for each instance.

On the left navigation pane, in the Admin area, click Pulsar Clusters.

On the Dashboard tab, you can view some metrics about the cluster, including how many topics, subscriptions, consumers, and produces are cluster, the throughput, storage size, and backlog size.
On the Overview tab, you can view the details about the cluster, including the cluster name, location, availability mode, cloud provider, cluster type, service URLs, and the features you enable on the cluster.

Update a cluster through StreamNative Cloud Console

On the left navigation pane, in the Admin area, click Pulsar Clusters.
On the Dashboard tab, click Edit Cluster.
Update the cluster.
- Select the Basic tab, and then select a different cluster size.
- Select the Advanced tab to change advanced cluster settings.
Click Finish when you are done updating your cluster.

Delete a cluster through StreamNative Cloud Console

You cannot delete a cluster if there are resources associated with the cluster.

To delete a cluster, follow these steps.

On the left navigation pane, in the Admin area, click Pulsar Clusters.
Select the Overview tab.
In the Warning section, click Delete Cluster. A dialog box displays asking, Are you sure you want to delete this?
Enter the cluster name and then click Confirm.

Next step

Work with tenants