1. Tools
  2. StreamNative CLI (snctl)

Tutorials: Manage StreamNative Cloud resources using snctl

Work with organizations

An organization is a team account configured perfectly for your use case on the cloud provider of your choice.

Currently, snctl does not support creating organizations. You can create an organization through StreamNative Cloud Console. For details about how to create an organization, see create an organization.

After you have created an organization through StreamNative Cloud Console, you can use the snctl config set --namespace command to set your default organization.

Note

With the namespace option, you need to use the random string for the organization name, not the descriptive name. You can find the random string on the Dashboard, next to the organization's descriptive name. For an example, see organizations.

After setting the default organization, you can use snctl commands to perform other operations on StreamNative Cloud resources in the default organization without specifying the organization name.

Work with instances

Note

In this section, we named the organization matrix for an example.

Create an instance

A Pulsar instance is a group of clusters that acts together as a single unit.

To create an instance through snctl, follow these steps.

  1. Define an instance named neo by using a manifest file and save the manifest file instance-neo.yaml.

    • This example shows how to create an instance on Google Cloud.

      apiVersion: cloud.streamnative.io/v1alpha1
      kind: PulsarInstance
      metadata:
        namespace: matrix
        name: neo
      spec:
        availabilityMode: zonal
        poolRef:
          namespace: streamnative
          name: shared
      
    • This example shows how to create an instance on AWS.

      apiVersion: cloud.streamnative.io/v1alpha1
      kind: PulsarInstance
      metadata:
        namespace: matrix
        name: neo
      spec:
        availabilityMode: zonal
        poolRef:
          namespace: streamnative
          name: shared-aws
      

    The following table lists fields in the manifest file.

    FieldDescription
    apiVersionSpecify the version of Pulsar API server.
    kindSpecify the component to be created.
    metadataConfigure the metadata information about the Pulsar instance.
    -namespace: specify the name of the organization.
    -name: specify the name of the Pulsar instance.
    specConfigure the specifications about the Pulsar instance.
    -availabilityMode: specify the availability mode of the Pulsar instance. You can set the parameter to zonal or regional. By default, it is set to zonal.
    - poolRef: an abstract definition of the compute, storage, and networking needed to host Pulsar instances. It is set to shared for instances hosted on Google Cloud and is set to shared-aws for instances hosted on AWS.
  2. Apply the manifest file to create the instance.

    snctl apply -f /path/to/instance-neo.yaml
    
  3. Check whether the instance is created successfully. This example shows whether the instance is successfully created on AWS.

    snctl describe pulsarinstance neo
    

    Output

    Name:         neo
    Namespace:    matrix
    Labels:       <none>
    Annotations:  <none>
    API Version:  cloud.streamnative.io/v1alpha1
    Kind:         PulsarInstance
    Metadata:
      Creation Timestamp:  2020-08-11T07:37:49Z
      Finalizers:
        pulsarinstance.finalizers.cloud.streamnative.io
      Generation:        1
      Resource Version:  367947
      Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarinstances/neo
      UID:               53f2958d-c7c9-4628-9290-a34e36cdca3b
    Spec:
      Availability Mode:  zonal
      poolRef:
        namespace: streamnative
        name: shared-aws
    Status:
      Auth:
        oauth2:
          Audience:    urn:sn:pulsar:test:test
          Issuer URL:  https://auth.streamnative.cloud
        Type:          oauth2
      Conditions:
        Last Transition Time:  2020-08-11T07:38:00Z
        Status:                True
        Type:                  SubscriptionReady
        Last Transition Time:  2020-08-11T07:37:51Z
        Reason:                Created
        Status:                True
        Type:                  ResourceServerReady
        Last Transition Time:  2020-08-11T07:37:51Z
        Reason:                Created
        Status:                True
        Type:                  ServiceAccountReady
        Last Transition Time:  2020-08-11T07:38:01Z
        Reason:                AllConditionStatusTrue
        Status:                True
        Type:                  Ready
    Events:                    <none>
    

    From the outputs, you can see that the status and type parameters for items under Conditions are set to true and ready. This means that the instance neo is created successfully.

In addition, you can use the snctl create pulsarinstance PULSAR_INSTANCE_NAME command to create an instance. For details, see snctl reference.

Check the instance

You can use the following command to list all created instances.

snctl get pulsarinstance

Output

NAME            CREATED AT
instance-test   2020-08-13T14:14:43Z
neo            2020-08-11T07:37:49Z
test-orga-a     2020-08-12T05:32:33Z

From the output of this command, you can see all created instances and the time when these instances were created.

Check instance details

Before checking details about an instance, you should use the following command to confirm whether the target instance is available.

snctl get pulsarinstance

Then, you can use the following command to check details about a specific instance.

snctl describe pulsarinstance PULSAR_INSTANCE_NAME

The following example checks details about the instance neo.

snctl describe pulsarinstance neo

Output

Name:         neo
Namespace:    matrix
Labels:       <none>
Annotations:  <none>
API Version:  cloud.streamnative.io/v1alpha1
Kind:         PulsarInstance
Metadata:
  Creation Timestamp:  2020-08-11T07:37:49Z
  Finalizers:
    pulsarinstance.finalizers.cloud.streamnative.io
  Generation:        1
  Resource Version:  367947
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarinstances/neo
  UID:               53f2958d-c7c9-4628-9290-a34e36cdca3b
Spec:
  Availability Mode:  zonal
Status:
  Auth:
    oauth2:
      Audience:    urn:sn:pulsar:test:test
      Issuer URL:  https://auth.streamnative.cloud
    Type:          oauth2
  Conditions:
    Last Transition Time:  2020-08-11T07:38:00Z
    Status:                True
    Type:                  SubscriptionReady
    Last Transition Time:  2020-08-11T07:37:51Z
    Reason:                Created
    Status:                True
    Type:                  ResourceServerReady
    Last Transition Time:  2020-08-11T07:37:51Z
    Reason:                Created
    Status:                True
    Type:                  ServiceAccountReady
    Last Transition Time:  2020-08-11T07:38:01Z
    Reason:                AllConditionStatusTrue
    Status:                True
    Type:                  Ready
Events:                    <none>

Delete an instance

You can use the following command to delete a specific instance based on the instance name.

snctl delete pulsarinstance PULSAR_INSTANCE_NAME

In addition, you can use the following command to delete a instance based on the type and name specified in the manifest file.

snctl delete -f ./instance-neo.yaml

Work with clusters

A cluster is a secure messaging environment within Pulsar. Each Pulsar cluster consists a set of 3 components in a geographical location.

  • Pulsar brokers - set of brokers handling all the data going in and out of Pulsar (or client requests)
  • Metadata storage - providing coordination and service discovery between services
  • Bookie ensemble - set of bookies that retain copies of the messages

A cluster has two layers: a stateless serving layer (made up of brokers) and a stateful storage layer (made up of bookies). See also Pulsar Architecture and Design.

Note

In StreamNative Console, you can create one and only one cluster for an instance.

The code examples in this section use an organization called, matrix.

Create a cluster

Note

Before you can create a cluster through snctl, you need to create a billing subscription. For more information, see subscribe to StreamNative Cloud.

To create a cluster, follow these steps.

  1. Define a cluster named neo-1 by using a manifest file and save the manifest file clusterneo1.yaml.

    apiVersion: cloud.streamnative.io/v1alpha1
    kind: PulsarCluster
    metadata:
      namespace: matrix
      name: neo-1
    spec:
      instanceName: neo
      location: us-east4
      bookkeeper:
        replicas: 3
        resourceSpec:
          nodeType: tiny-1
      broker:
        replicas: 1
        resourceSpec:
          nodeType: tiny-1
      config:
        custom:
          backlogQuotaDefaultLimitBytes: '1000000000'
        websocketEnabled: true
    

    The following table lists fields in the manifest file.

    FieldDescription
    apiVersionSpecify the version of Pulsar API server.
    kindSpecify the component to be created.
    metadataConfigure the metadata information about the cluster.
    - namespace: specify the name of the organization.
    - name: specify the name of the cluster.
    specConfigure the specifications about the cluster.
    - instanceName: specify the name of the instance where the cluster belongs to.
    - location: specify the location of the cluster.
    - broker: configure the broker, including the number of brokers and the resource type (node type).
    - bookkeeper: configure the bookie, including the number of bookies and the resource type (node type).
    - config: specify customized cluster configurations and configure cluster features.
  2. Apply the manifest file to create the cluster.

    snctl apply -f /path/to/clusterneo1.yaml
    
  3. Check whether the cluster is created successfully.

    snctl describe pulsarcluster neo-1
    

    Output

    Name:         neo-1
    Namespace:    matrix
    Labels:       <none>
    Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                    {"apiVersion":"cloud.streamnative.io/v1alpha1","kind":"PulsarCluster","metadata":{"annotations":{},"name":"neo-1","namespace":"matrix"},...
    API Version:  cloud.streamnative.io/v1alpha1
    Kind:         PulsarCluster
    Metadata:
      Creation Timestamp:  2021-10-12T01:49:30Z
      Finalizers:
        pulsarcluster.finalizers.cloud.streamnative.io
      Generation:  1
      Managed Fields:
        API Version:  cloud.streamnative.io/v1alpha1
        Fields Type:  FieldsV1
        fieldsV1:
          f:metadata:
            f:annotations:
              .:
              f:kubectl.kubernetes.io/last-applied-configuration:
          f:spec:
            f:bookkeeper:
              .:
              f:replicas:
              f:resourceSpec:
                .:
                f:nodeType:
            f:broker:
              f:replicas:
              f:resourceSpec:
                .:
                f:nodeType:
            f:config:
              .:
              f:custom:
                .:
                f:backlogQuotaDefaultLimitBytes:
              f:websocketEnabled:
            f:instanceName:
            f:location:
        Manager:      Go-http-client
        Operation:    Update
        Time:         2021-10-12T01:49:29Z
        API Version:  cloud.streamnative.io/v1alpha1
        Fields Type:  FieldsV1
        fieldsV1:
          f:metadata:
            f:finalizers:
              .:
              v:"pulsarcluster.finalizers.cloud.streamnative.io":
        Manager:         controller-manager
        Operation:       Update
        Time:            2021-10-12T01:49:30Z
      Resource Version:  11761383
      Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarclusters/neo-1
      UID:               b1409658-e9f5-4056-9dae-0d484b230607
    Spec:
      Bookkeeper:
        Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
        Replicas:  3
        Resource Spec:
          Node Type:  tiny-1
      Broker:
        Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
        Replicas:  1
        Resource Spec:
          Node Type:  tiny-1
      Config:
        Custom:
          Backlog Quota Default Limit Bytes:  1000000000
        Websocket Enabled:                    true
      Instance Name:                          ins-2
      Location:                               us-east4
      Pool Member Ref:
        Name:       us-east4
        Namespace:  streamnative
      Service Endpoints:
        Dns Name:  neo-1.matrix.sn2.dev
        Type:      service
        Dns Name:  neo-1.matrix.us-east4.streamnative.g.sn2.dev
    Status:
    Events:  <none>
    

Check the cluster

You can use the following command to list all created clusters.

snctl get pulsarcluster

Output

NAME            CREATED AT
cluster-test   2020-08-14T01:48:46Z
neo-1          2021-10-12T01:49:29Z
test11         2020-08-12T13:24:19Z

From the output of this command, you can see all created clusters and the time when these clusters were created.

Check cluster details

Before checking the details about a cluster, you should use the following command to confirm whether the target cluster is available.

snctl get pulsarcluster

Then, you can use the following command to check details about a specific cluster.

snctl describe pulsarcluster <cluster_name>

The following example checks details about the instance neo-1.

snctl describe pulsarcluster neo-1

Output

Name:         neo-1
Namespace:    matrix
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"cloud.streamnative.io/v1alpha1","kind":"PulsarCluster","metadata":{"annotations":{},"name":"neo-1","namespace":"matrix"},...
API Version:  cloud.streamnative.io/v1alpha1
Kind:         PulsarCluster
Metadata:
  Creation Timestamp:  2021-10-12T01:49:30Z
  Finalizers:
    pulsarcluster.finalizers.cloud.streamnative.io
  Generation:  1
  Managed Fields:
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        f:bookkeeper:
          .:
          f:replicas:
          f:resourceSpec:
            .:
            f:nodeType:
        f:broker:
          f:replicas:
          f:resourceSpec:
            .:
            f:nodeType:
        f:config:
          .:
          f:custom:
            .:
            f:backlogQuotaDefaultLimitBytes:
          f:websocketEnabled:
        f:instanceName:
        f:location:
    Manager:      Go-http-client
    Operation:    Update
    Time:         2021-10-12T01:49:29Z
    API Version:  cloud.streamnative.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"pulsarcluster.finalizers.cloud.streamnative.io":
    Manager:         controller-manager
    Operation:       Update
    Time:            2021-10-12T01:49:30Z
  Resource Version:  11761383
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarclusters/neo-1
  UID:               b1409658-e9f5-4056-9dae-0d484b230607
Spec:
  Bookkeeper:
    Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
    Replicas:  3
    Resource Spec:
      Node Type:  tiny-1
  Broker:
    Image:     gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
    Replicas:  1
    Resource Spec:
      Node Type:  tiny-1
  Config:
    Custom:
      Backlog Quota Default Limit Bytes:  1000000000
    Websocket Enabled:                    true
  Instance Name:                          ins-2
  Location:                               us-east4
  Pool Member Ref:
    Name:       us-east4
    Namespace:  streamnative
  Service Endpoints:
    Dns Name:  neo-1.matrix.sn2.dev
    Type:      service
    Dns Name:  neo-1.matrix.us-east4.streamnative.g.sn2.dev
Status:
Events:  <none>

Add clusters to the Pulsar configuration file

This section describes how to add a cluster to the Pulsar client configuration file by defining a context for that cluster. Then, you can access the cluster using the pulsarctl CLI tool.

  1. Add a cluster to the Pulsar client configuration file.

    This example adds a neo-1 cluster to the Pulsar client configuration file.

    Input

    snctl x update-pulsar-config --cluster-name <neo-1>
    

    Output

    Updated Pulsar client configuration for context 'neo-1'.
    
  2. Verify that the current context has been changed.

    Input

    pulsarctl context current
    

    Output

    neo-1
    

Then, you can use the pulsarctl CLI tool to interact with the target cluster. For details, see Connect to cluster through pulsarctl and perform other operations.

Delete a cluster

Note

You cannot delete a cluster if there are resources associated with the cluster.

You can use the following command to delete a specific cluster based on the cluster name.

snctl delete pulsarcluster <cluster_name>

In addition, you can use the following command to delete the cluster based on the cluster name specified in the manifest file.

snctl delete -f ./clusterneo1.yaml

Work with service accounts

Note

In this section, the organization has the name matrix as an example name.

Create a service account

To create a service account through snctl, follow these steps.

  1. Define a service account resource named bot by using a manifest file and save the manifest file sa-bot.yaml.

    apiVersion: cloud.streamnative.io/v1alpha1
    kind: ServiceAccount
    metadata:
      namespace: matrix
      name: bot
    

    The following table lists fields in the manifest file.

    FieldDescription
    apiVersionSpecify the version of Pulsar API server.
    kindSpecify the component to be created.
    metadataConfigure the metadata information about the service account.
    - namespace: specify the name of the organization.
    - name: specify the name of the service account.
  2. Apply the manifest file to create the service account.

    snctl apply -f /path/to/sa-bot.yaml
    

    Output

    serviceaccount.cloud.streamnative.io/bot created
    
  3. Check whether the service account was created successfully.

    snctl describe serviceaccount bot
    

    Output

    Name:         bot
    Namespace:    matrix
    Labels:       <none>
    Annotations:  <none>
    API Version:  cloud.streamnative.io/v1alpha1
    Kind:         ServiceAccount
    Metadata:
      Creation Timestamp:  2020-08-11T16:25:10Z
      Finalizers:
        serviceaccount.finalizers.cloud.streamnative.io
      Generation:        1
      Resource Version:  396516
      Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/serviceaccounts/bot
      UID:               874b226b-ea01-41c2-9a7b-059fdcc0d5c1
    Spec:
    Status:
      Conditions:
        Last Transition Time:  2020-08-14T06:25:52Z
        Reason:                Provisioned
        Status:                True
        Type:                  Ready
      Private Key Data:
      Private Key Type:        TYPE_SN_CREDENTIALS_FILE
    Events:                    <none>
    

    From the output, you can see that the status and type parameters for items under Conditions are set to true and ready. This means that the service account bot was created successfully.

In addition, you can use the snctl create serviceaccount SERVICE_ACCOUNT_NAME command to create a service account. For details, see snctl reference.

Download service account credentials

To use a service account, you first download its associated credentials to a JSON file. The information is made available through the status block of the ServiceAccount resource.

The following example shows how to download the service account credentials to a file called bot.json.

snctl auth export-service-account bot --key-file bot.json

The file contents will be similar to the following:

{
  "type": "SN_SERVICE_ACCOUNT",
  "client_id": "CLIENT_ID",
  "client_secret": "CLIENT_SECRET",
  "client_email": "[email protected]"
}

The following table lists two fields in the JSON file.

FieldDescription
client_idIt is an Auth0 Application that has been created.
client_secretIt is used to authenticate to Auth0 for accessing snctl.

Note

The file contains credentials information and should be well protected.

Download service account tokens

To get a token using the StreamNative CLI snctl, run the following command.

snctl auth get-token [PULSAR_INSTANCE_NAME] -n <organization-namespace>

Activate a service account

This example shows how to activate a service account through a key file called bot.json.

snctl auth activate-service-account --key-file bot.json -a https://api.streamnative.cloud -i https://auth.streamnative.cloud/

Output

Logged in as [email protected].
Welcome to StreamNative Cloud!

Access the StreamNative Cloud API

This example shows how to access the StreamNative Cloud API through a service account.

  1. Log in to snctl.

    snctl auth login
    
  2. Create a service account.

    This example creates a service account named bot.

    snctl create serviceaccount bot.
    

    Output

    serviceaccount.cloud.streamnative.io/bot created
    
  3. Download the associated credentials of the service account to a JSON file。

    snctl auth export-service-account bot -f bot.json
    

    Output

    Wrote private key file 'bot.json'.
    
  4. Bind the service account with an "admin" role.

    snctl create rolebinding bot-cluster-admin --role admin --serviceaccount bot
    

    Output

    rolebinding.cloud.streamnative.io/bot-cluster-admin created
    
  5. Log out from snctl.

    snctl auth logout
    
  6. Log in to snctl with the service account.

    snctl auth activate-service-account --key-file bot.json
    

    Output

    Logged in as [email protected].
    Welcome to StreamNative Cloud!
    

Check service accounts

This example shows how to check the service accounts of an organization.

snctl get serviceaccount

Output

NAME   CREATED AT
bot   2020-08-11T16:25:10Z

From the output of this command, you can see all created service accounts and the time when these service accounts were created.

Check service account details

Before checking the details about a service account, you should use the following command to confirm whether the service account is available.

snctl get serviceaccount

Then, you can use the following command to check details about a service account.

snctl describe serviceaccount SERVICE_ACCOUNT_NAME

The following example checks the details about the service account bot.

snctl describe serviceaccount bot

Output

Name:         bot
Namespace:    matrix
Labels:       <none>
Annotations:  <none>
API Version:  cloud.streamnative.io/v1alpha1
Kind:         ServiceAccount
Metadata:
  Creation Timestamp:  2020-08-11T16:25:10Z
  Finalizers:
    serviceaccount.finalizers.cloud.streamnative.io
  Generation:        1
  Resource Version:  396516
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/serviceaccounts/bot
  UID:               874b226b-ea01-41c2-9a7b-059fdcc0d5c1
Spec:
Status:
  Conditions:
    Last Transition Time:  2020-08-14T06:25:52Z
    Reason:                Provisioned
    Status:                True
    Type:                  Ready
  Private Key Data:
  Private Key Type:        TYPE_SN_CREDENTIALS_FILE
Events:                    <none>

Delete a service account

You can use the following command to delete a service account based on the service account name.

snctl delete serviceaccount SERVICE_ACCOUNT_NAME

In addition, you can use the following command to delete the service account based on the name specified in the manifest file.

snctl delete -f ./sa-bot.yaml

Work with users

Note

In this document, we named the organization matrix for an example

Create a user

To create a user, follow these steps.

  1. Define a user named ironman by using a manifest file and save the manifest file user-ironman.yaml.

    apiVersion: cloud.streamnative.io/v1alpha1
    kind: User
    metadata:
      namespace: matrix
      name: ironman
    spec:
      email: [email protected]
    

    The following table lists fields in the manifest file.

    FieldDescription
    apiVersionSpecify the version of Pulsar API server.
    kindSpecify the component to be created.
    metadataConfigure the metadata information about the user.
    - namespace: specify the name of the organization.
    - name: specify the name of the user.
    specSpecify the email address for the user.
  2. Apply the manifest file to create the user.

    snctl apply -f /path/to/user-ironman.yaml
    
  3. Check whether the user is created successfully.

    snctl describe users [email protected]
    

    Output

    Name:         [email protected]
    Namespace:    matrix
    Labels:       <none>
    Annotations:  <none>
    API Version:  cloud.streamnative.io/v1alpha1
    Kind:         User
    Metadata:
      Creation Timestamp:  2020-08-10T14:31:12Z
      Finalizers:
        user.finalizers.cloud.streamnative.io
      Generation:        1
      Resource Version:  104126
      Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/users/[email protected]
      UID:               5f6297c6-8e64-4175-b866-bb6362437f29
    Spec:
      Email:  [email protected]
    Status:
    Events:  <none>
    

Check users

You can use the following command to check users created for an organization.

snctl get users

Output

NAME                     CREATED AT
[email protected]   2020-08-10T14:31:12Z

Check user details

Before checking details about a user, you should use the following command to confirm whether the target user is available.

snctl get users

Then, you can use the following command to check details about a specific user.

snctl describe users USER_NAME

The following example checks details about the user [email protected].

snctl describe users [email protected]

Output

Name:         [email protected]
Namespace:    matrix
Labels:       <none>
Annotations:  <none>
API Version:  cloud.streamnative.io/v1alpha1
Kind:         User
Metadata:
  Creation Timestamp:  2020-08-10T14:31:12Z
  Finalizers:
    user.finalizers.cloud.streamnative.io
  Generation:        1
  Resource Version:  104126
  Self Link:         /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/users/[email protected]
  UID:               5f6297c6-8e64-4175-b866-bb6362437f29
Spec:
  Email:  [email protected]
Status:
Events:  <none>

Delete a user

You can use the following command to delete a specific user based on the user name.

snctl delete users USER_NAME

In addition, you can use the following command to delete the cluster based on the type and name specified in the manifest file.

snctl delete -f ./user-ironman.yaml
Previous
Usage Conventions