- Cloud CLI (snctl)
Tutorials: Manage StreamNative Cloud resources using snctl
Work with organizations
An organization is a team account configured perfectly for your use case on the cloud provider of your choice.
Currently, snctl does not support creating organizations. You can create an organization through StreamNative Cloud Console. For details about how to create an organization, see create an organization.
After you have created an organization through StreamNative Cloud Console, you can use the snctl config set --namespace
command to set your default organization.
Note
With the namespace
option, you need to use the random string for the organization name, not the descriptive name. You can find the random string on the Dashboard, next to the organization's descriptive name. For an example, see organizations.
After setting the default organization, you can use snctl commands to perform other operations on StreamNative Cloud resources in the default organization without specifying the organization name.
Work with instances
Note
In this section, we named the organization matrix
for an example.
Create an instance
A Pulsar instance is a group of clusters that acts together as a single unit.
To create an instance through snctl, follow these steps.
Define an instance named
neo
by using a manifest file and save the manifest fileinstance-neo.yaml
.This example shows how to create an instance on Google Cloud.
apiVersion: cloud.streamnative.io/v1alpha1 kind: PulsarInstance metadata: namespace: matrix name: neo spec: availabilityMode: zonal poolRef: namespace: streamnative name: shared
This example shows how to create an instance on AWS.
apiVersion: cloud.streamnative.io/v1alpha1 kind: PulsarInstance metadata: namespace: matrix name: neo spec: availabilityMode: zonal poolRef: namespace: streamnative name: shared-aws
The following table lists fields in the manifest file.
Field Description apiVersion Specify the version of Pulsar API server. kind Specify the component to be created. metadata Configure the metadata information about the Pulsar instance.
-namespace: specify the name of the organization.
-name: specify the name of the Pulsar instance.spec Configure the specifications about the Pulsar instance.
-availabilityMode
: specify the availability mode of the Pulsar instance. You can set the parameter tozonal
orregional
. By default, it is set tozonal
.
-poolRef
: an abstract definition of the compute, storage, and networking needed to host Pulsar instances. It is set toshared
for instances hosted on Google Cloud and is set toshared-aws
for instances hosted on AWS.Apply the manifest file to create the instance.
snctl apply -f /path/to/instance-neo.yaml
Check whether the instance is created successfully. This example shows whether the instance is successfully created on AWS.
snctl describe pulsarinstance neo
Output
Name: neo Namespace: matrix Labels: <none> Annotations: <none> API Version: cloud.streamnative.io/v1alpha1 Kind: PulsarInstance Metadata: Creation Timestamp: 2020-08-11T07:37:49Z Finalizers: pulsarinstance.finalizers.cloud.streamnative.io Generation: 1 Resource Version: 367947 Self Link: /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarinstances/neo UID: 53f2958d-c7c9-4628-9290-a34e36cdca3b Spec: Availability Mode: zonal poolRef: namespace: streamnative name: shared-aws Status: Auth: oauth2: Audience: urn:sn:pulsar:test:test Issuer URL: https://auth.streamnative.cloud Type: oauth2 Conditions: Last Transition Time: 2020-08-11T07:38:00Z Status: True Type: SubscriptionReady Last Transition Time: 2020-08-11T07:37:51Z Reason: Created Status: True Type: ResourceServerReady Last Transition Time: 2020-08-11T07:37:51Z Reason: Created Status: True Type: ServiceAccountReady Last Transition Time: 2020-08-11T07:38:01Z Reason: AllConditionStatusTrue Status: True Type: Ready Events: <none>
From the outputs, you can see that the
status
andtype
parameters for items underConditions
are set totrue
andready
. This means that the instanceneo
is created successfully.
In addition, you can use the snctl create pulsarinstance PULSAR_INSTANCE_NAME
command to create an instance. For details, see snctl reference.
Check the instance
You can use the following command to list all created instances.
snctl get pulsarinstance
Output
NAME CREATED AT
instance-test 2020-08-13T14:14:43Z
neo 2020-08-11T07:37:49Z
test-orga-a 2020-08-12T05:32:33Z
From the output of this command, you can see all created instances and the time when these instances were created.
Check instance details
Before checking details about an instance, you should use the following command to confirm whether the target instance is available.
snctl get pulsarinstance
Then, you can use the following command to check details about a specific instance.
snctl describe pulsarinstance PULSAR_INSTANCE_NAME
The following example checks details about the instance neo
.
snctl describe pulsarinstance neo
Output
Name: neo
Namespace: matrix
Labels: <none>
Annotations: <none>
API Version: cloud.streamnative.io/v1alpha1
Kind: PulsarInstance
Metadata:
Creation Timestamp: 2020-08-11T07:37:49Z
Finalizers:
pulsarinstance.finalizers.cloud.streamnative.io
Generation: 1
Resource Version: 367947
Self Link: /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarinstances/neo
UID: 53f2958d-c7c9-4628-9290-a34e36cdca3b
Spec:
Availability Mode: zonal
Status:
Auth:
oauth2:
Audience: urn:sn:pulsar:test:test
Issuer URL: https://auth.streamnative.cloud
Type: oauth2
Conditions:
Last Transition Time: 2020-08-11T07:38:00Z
Status: True
Type: SubscriptionReady
Last Transition Time: 2020-08-11T07:37:51Z
Reason: Created
Status: True
Type: ResourceServerReady
Last Transition Time: 2020-08-11T07:37:51Z
Reason: Created
Status: True
Type: ServiceAccountReady
Last Transition Time: 2020-08-11T07:38:01Z
Reason: AllConditionStatusTrue
Status: True
Type: Ready
Events: <none>
Delete an instance
You can use the following command to delete a specific instance based on the instance name.
snctl delete pulsarinstance PULSAR_INSTANCE_NAME
In addition, you can use the following command to delete a instance based on the type and name specified in the manifest file.
snctl delete -f ./instance-neo.yaml
Work with clusters
A cluster is a secure messaging environment within Pulsar. Each Pulsar cluster consists a set of 3 components in a geographical location.
- Pulsar brokers - set of brokers handling all the data going in and out of Pulsar (or client requests)
- Metadata storage - providing coordination and service discovery between services
- Bookie ensemble - set of bookies that retain copies of the messages
A cluster has two layers: a stateless serving layer (made up of brokers) and a stateful storage layer (made up of bookies). See also Pulsar Architecture and Design.
Note
In StreamNative Console, you can create one and only one cluster for an instance.
The code examples in this section use an organization called, matrix
.
Create a cluster
Note
Before you can create a cluster through snctl
, you need to create a billing subscription. For more information, see subscribe to StreamNative Cloud.
To create a cluster, follow these steps.
Define a cluster named
neo-1
by using a manifest file and save the manifest fileclusterneo1.yaml
.apiVersion: cloud.streamnative.io/v1alpha1 kind: PulsarCluster metadata: namespace: matrix name: neo-1 spec: instanceName: neo location: us-east4 bookkeeper: replicas: 3 resourceSpec: nodeType: tiny-1 broker: replicas: 1 resourceSpec: nodeType: tiny-1 config: custom: backlogQuotaDefaultLimitBytes: '1000000000' websocketEnabled: true
The following table lists fields in the manifest file.
Field Description apiVersion Specify the version of Pulsar API server. kind Specify the component to be created. metadata Configure the metadata information about the cluster.
-namespace
: specify the name of the organization.
-name
: specify the name of the cluster.spec Configure the specifications about the cluster.
-instanceName
: specify the name of the instance where the cluster belongs to.
-location
: specify the location of the cluster.
-broker
: configure the broker, including the number of brokers and the resource type (node type).
-bookkeeper
: configure the bookie, including the number of bookies and the resource type (node type).
-config
: specify customized cluster configurations and configure cluster features.Apply the manifest file to create the cluster.
snctl apply -f /path/to/clusterneo1.yaml
Check whether the cluster is created successfully.
snctl describe pulsarcluster neo-1
Output
Name: neo-1 Namespace: matrix Labels: <none> Annotations: kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"cloud.streamnative.io/v1alpha1","kind":"PulsarCluster","metadata":{"annotations":{},"name":"neo-1","namespace":"matrix"},... API Version: cloud.streamnative.io/v1alpha1 Kind: PulsarCluster Metadata: Creation Timestamp: 2021-10-12T01:49:30Z Finalizers: pulsarcluster.finalizers.cloud.streamnative.io Generation: 1 Managed Fields: API Version: cloud.streamnative.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: f:bookkeeper: .: f:replicas: f:resourceSpec: .: f:nodeType: f:broker: f:replicas: f:resourceSpec: .: f:nodeType: f:config: .: f:custom: .: f:backlogQuotaDefaultLimitBytes: f:websocketEnabled: f:instanceName: f:location: Manager: Go-http-client Operation: Update Time: 2021-10-12T01:49:29Z API Version: cloud.streamnative.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: v:"pulsarcluster.finalizers.cloud.streamnative.io": Manager: controller-manager Operation: Update Time: 2021-10-12T01:49:30Z Resource Version: 11761383 Self Link: /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarclusters/neo-1 UID: b1409658-e9f5-4056-9dae-0d484b230607 Spec: Bookkeeper: Image: gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9 Replicas: 3 Resource Spec: Node Type: tiny-1 Broker: Image: gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9 Replicas: 1 Resource Spec: Node Type: tiny-1 Config: Custom: Backlog Quota Default Limit Bytes: 1000000000 Websocket Enabled: true Instance Name: ins-2 Location: us-east4 Pool Member Ref: Name: us-east4 Namespace: streamnative Service Endpoints: Dns Name: neo-1.matrix.sn2.dev Type: service Dns Name: neo-1.matrix.us-east4.streamnative.g.sn2.dev Status: Events: <none>
Check the cluster
You can use the following command to list all created clusters.
snctl get pulsarcluster
Output
NAME CREATED AT
cluster-test 2020-08-14T01:48:46Z
neo-1 2021-10-12T01:49:29Z
test11 2020-08-12T13:24:19Z
From the output of this command, you can see all created clusters and the time when these clusters were created.
Check cluster details
Before checking the details about a cluster, you should use the following command to confirm whether the target cluster is available.
snctl get pulsarcluster
Then, you can use the following command to check details about a specific cluster.
snctl describe pulsarcluster <cluster_name>
The following example checks details about the instance neo-1
.
snctl describe pulsarcluster neo-1
Output
Name: neo-1
Namespace: matrix
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"cloud.streamnative.io/v1alpha1","kind":"PulsarCluster","metadata":{"annotations":{},"name":"neo-1","namespace":"matrix"},...
API Version: cloud.streamnative.io/v1alpha1
Kind: PulsarCluster
Metadata:
Creation Timestamp: 2021-10-12T01:49:30Z
Finalizers:
pulsarcluster.finalizers.cloud.streamnative.io
Generation: 1
Managed Fields:
API Version: cloud.streamnative.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
f:bookkeeper:
.:
f:replicas:
f:resourceSpec:
.:
f:nodeType:
f:broker:
f:replicas:
f:resourceSpec:
.:
f:nodeType:
f:config:
.:
f:custom:
.:
f:backlogQuotaDefaultLimitBytes:
f:websocketEnabled:
f:instanceName:
f:location:
Manager: Go-http-client
Operation: Update
Time: 2021-10-12T01:49:29Z
API Version: cloud.streamnative.io/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"pulsarcluster.finalizers.cloud.streamnative.io":
Manager: controller-manager
Operation: Update
Time: 2021-10-12T01:49:30Z
Resource Version: 11761383
Self Link: /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/pulsarclusters/neo-1
UID: b1409658-e9f5-4056-9dae-0d484b230607
Spec:
Bookkeeper:
Image: gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
Replicas: 3
Resource Spec:
Node Type: tiny-1
Broker:
Image: gcr.io/affable-ray-226821/pulsar-cloud:2.8.0.9
Replicas: 1
Resource Spec:
Node Type: tiny-1
Config:
Custom:
Backlog Quota Default Limit Bytes: 1000000000
Websocket Enabled: true
Instance Name: ins-2
Location: us-east4
Pool Member Ref:
Name: us-east4
Namespace: streamnative
Service Endpoints:
Dns Name: neo-1.matrix.sn2.dev
Type: service
Dns Name: neo-1.matrix.us-east4.streamnative.g.sn2.dev
Status:
Events: <none>
Add clusters to the Pulsar configuration file
This section describes how to add a cluster to the Pulsar client configuration file by defining a context
for that cluster. Then, you can access the cluster using the pulsarctl
CLI tool.
Add a cluster to the Pulsar client configuration file.
This example adds a
neo-1
cluster to the Pulsar client configuration file.Input
snctl x update-pulsar-config --cluster-name <neo-1>
Output
Updated Pulsar client configuration for context 'neo-1'.
Verify that the current context has been changed.
Input
pulsarctl context current
Output
neo-1
Then, you can use the pulsarctl
CLI tool to interact with the target cluster. For details, see Connect to cluster through pulsarctl and perform other operations.
Delete a cluster
Note
You cannot delete a cluster if there are resources associated with the cluster.
You can use the following command to delete a specific cluster based on the cluster name.
snctl delete pulsarcluster <cluster_name>
In addition, you can use the following command to delete the cluster based on the cluster name specified in the manifest file.
snctl delete -f ./clusterneo1.yaml
Work with service accounts
Note
In this section, the organization has the name matrix
as an example name.
Create a service account
To create a service account through snctl, follow these steps.
Define a service account resource named
bot
by using a manifest file and save the manifest filesa-bot.yaml
.apiVersion: cloud.streamnative.io/v1alpha1 kind: ServiceAccount metadata: namespace: matrix name: bot
The following table lists fields in the manifest file.
Field Description apiVersion Specify the version of Pulsar API server. kind Specify the component to be created. metadata Configure the metadata information about the service account.
- namespace: specify the name of the organization.
- name: specify the name of the service account.Apply the manifest file to create the service account.
snctl apply -f /path/to/sa-bot.yaml
Output
serviceaccount.cloud.streamnative.io/bot created
Check whether the service account was created successfully.
snctl describe serviceaccount bot
Output
Name: bot Namespace: matrix Labels: <none> Annotations: <none> API Version: cloud.streamnative.io/v1alpha1 Kind: ServiceAccount Metadata: Creation Timestamp: 2020-08-11T16:25:10Z Finalizers: serviceaccount.finalizers.cloud.streamnative.io Generation: 1 Resource Version: 396516 Self Link: /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/serviceaccounts/bot UID: 874b226b-ea01-41c2-9a7b-059fdcc0d5c1 Spec: Status: Conditions: Last Transition Time: 2020-08-14T06:25:52Z Reason: Provisioned Status: True Type: Ready Private Key Data: Private Key Type: TYPE_SN_CREDENTIALS_FILE Events: <none>
From the output, you can see that the
status
andtype
parameters for items underConditions
are set totrue
andready
. This means that the service accountbot
was created successfully.
In addition, you can use the snctl create serviceaccount SERVICE_ACCOUNT_NAME
command to create a service account. For details, see snctl reference.
Download service account credentials
To use a service account, you first download its associated credentials to a JSON file. The information is made available through the status
block of the ServiceAccount
resource.
The following example shows how to download the service account credentials to a file called bot.json
.
snctl auth export-service-account bot --key-file bot.json
The file contents will be similar to the following:
{
"type": "SN_SERVICE_ACCOUNT",
"client_id": "CLIENT_ID",
"client_secret": "CLIENT_SECRET",
"client_email": "[email protected]"
}
The following table lists two fields in the JSON file.
Field | Description |
---|---|
client_id | It is an Auth0 Application that has been created. |
client_secret | It is used to authenticate to Auth0 for accessing snctl. |
Note
The file contains credentials information and should be well protected.
Download service account tokens
To get a token using the StreamNative CLI snctl
, run the following command.
snctl auth get-token [PULSAR_INSTANCE_NAME] -n <organization-namespace>
Activate a service account
This example shows how to activate a service account through a key file called bot.json
.
snctl auth activate-service-account --key-file bot.json -a https://api.streamnative.cloud -i https://auth.streamnative.cloud/
Output
Logged in as [email protected].
Welcome to StreamNative Cloud!
Access the StreamNative Cloud API
This example shows how to access the StreamNative Cloud API through a service account.
Log in to snctl.
snctl auth login
Create a service account.
This example creates a service account named bot.
snctl create serviceaccount bot.
Output
serviceaccount.cloud.streamnative.io/bot created
Download the associated credentials of the service account to a JSON file。
snctl auth export-service-account bot -f bot.json
Output
Wrote private key file 'bot.json'.
Bind the service account with an "admin" role.
snctl create rolebinding bot-cluster-admin --role admin --serviceaccount bot
Output
rolebinding.cloud.streamnative.io/bot-cluster-admin created
Log out from snctl.
snctl auth logout
Log in to snctl with the service account.
snctl auth activate-service-account --key-file bot.json
Output
Logged in as [email protected]. Welcome to StreamNative Cloud!
Check service accounts
This example shows how to check the service accounts of an organization.
snctl get serviceaccount
Output
NAME CREATED AT
bot 2020-08-11T16:25:10Z
From the output of this command, you can see all created service accounts and the time when these service accounts were created.
Check service account details
Before checking the details about a service account, you should use the following command to confirm whether the service account is available.
snctl get serviceaccount
Then, you can use the following command to check details about a service account.
snctl describe serviceaccount SERVICE_ACCOUNT_NAME
The following example checks the details about the service account bot
.
snctl describe serviceaccount bot
Output
Name: bot
Namespace: matrix
Labels: <none>
Annotations: <none>
API Version: cloud.streamnative.io/v1alpha1
Kind: ServiceAccount
Metadata:
Creation Timestamp: 2020-08-11T16:25:10Z
Finalizers:
serviceaccount.finalizers.cloud.streamnative.io
Generation: 1
Resource Version: 396516
Self Link: /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/serviceaccounts/bot
UID: 874b226b-ea01-41c2-9a7b-059fdcc0d5c1
Spec:
Status:
Conditions:
Last Transition Time: 2020-08-14T06:25:52Z
Reason: Provisioned
Status: True
Type: Ready
Private Key Data:
Private Key Type: TYPE_SN_CREDENTIALS_FILE
Events: <none>
Delete a service account
You can use the following command to delete a service account based on the service account name.
snctl delete serviceaccount SERVICE_ACCOUNT_NAME
In addition, you can use the following command to delete the service account based on the name specified in the manifest file.
snctl delete -f ./sa-bot.yaml
Work with API Keys
Note
To work with API keys using snctl
, you need to snctl version 0.16.0 or later.
Create an API Key
To create an API key that doesn't have an expiration date, do the following:
snctl create apikey test-apikey --instance-name test-key --service-account-name test-api-key --expiration-time 0 --description 'this is test' -n sndev
To create an API key that expires in 30 days, do the following:
snctl create apikey test-apikey2 --instance-name test-key --service-account-name test-api-key --expiration-time 30d --description 'this is test' -n sndev
To create an API key that expires in at a specific time, use the following:
snctl create apikey test-apikey3 --instance-name test-key --service-account-name test-api-key --expiration-time "2025-02-08T15:38:40Z" --description 'this is test' -n your-org-name
View API Keys
To view API keys, use the following command:
snctl get apikey -n your-org-name
Revoke an API Key
To revoke an API key, use the following:
snctl revoke apikey test-apikey3 -n your-org-name
Delete an API Key
To delete an API key, use the following:
snctl delete apikey test-apikey-name -n your-org-name
Deleting an API key will also revoke it.
Work with users
Note
In this document, we named the organization matrix
for an example
Create a user
To create a user, follow these steps.
Define a user named
ironman
by using a manifest file and save the manifest fileuser-ironman.yaml
.apiVersion: cloud.streamnative.io/v1alpha1 kind: User metadata: namespace: matrix name: ironman spec: email: [email protected]
The following table lists fields in the manifest file.
Field Description apiVersion Specify the version of Pulsar API server. kind Specify the component to be created. metadata Configure the metadata information about the user.
- namespace: specify the name of the organization.
- name: specify the name of the user.spec Specify the email address for the user. Apply the manifest file to create the user.
snctl apply -f /path/to/user-ironman.yaml
Check whether the user is created successfully.
snctl describe users [email protected]
Output
Name: [email protected] Namespace: matrix Labels: <none> Annotations: <none> API Version: cloud.streamnative.io/v1alpha1 Kind: User Metadata: Creation Timestamp: 2020-08-10T14:31:12Z Finalizers: user.finalizers.cloud.streamnative.io Generation: 1 Resource Version: 104126 Self Link: /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/users/[email protected] UID: 5f6297c6-8e64-4175-b866-bb6362437f29 Spec: Email: [email protected] Status: Events: <none>
Check users
You can use the following command to check users created for an organization.
snctl get users
Output
NAME CREATED AT
[email protected] 2020-08-10T14:31:12Z
Check user details
Before checking details about a user, you should use the following command to confirm whether the target user is available.
snctl get users
Then, you can use the following command to check details about a specific user.
snctl describe users USER_NAME
The following example checks details about the user [email protected]
.
snctl describe users [email protected]
Output
Name: [email protected]
Namespace: matrix
Labels: <none>
Annotations: <none>
API Version: cloud.streamnative.io/v1alpha1
Kind: User
Metadata:
Creation Timestamp: 2020-08-10T14:31:12Z
Finalizers:
user.finalizers.cloud.streamnative.io
Generation: 1
Resource Version: 104126
Self Link: /apis/cloud.streamnative.io/v1alpha1/namespaces/matrix/users/[email protected]
UID: 5f6297c6-8e64-4175-b866-bb6362437f29
Spec:
Email: [email protected]
Status:
Events: <none>
Delete a user
You can use the following command to delete a specific user based on the user name.
snctl delete users USER_NAME
In addition, you can use the following command to delete the cluster based on the type and name specified in the manifest file.
snctl delete -f ./user-ironman.yaml