StreamNative CLI Tutorial

This tutorial demonstrates how to use the snctl command to deploy a Serverless cluster and use the pulsarctl command to provision Pulsar resources in the Serverless cluster.

This tutorial provisions the following resources (assuming we name the application as sl-app):

1. Provision a Serverless Instance <your-instance-name>.
2. Provision a Serverless Cluster <your-cluster-name>.
3. Provision a Service Account named <your-service-account-name>.
4. Provision an API Key for the service account <your-api-key>.
5. Create a Pulsar Tenant <your-tenant-name>.
6. Create a Pulsar Namespace <your-namespace-name>.
7. Create a Pulsar Topic <your-topic-name> with 4 partitions.
8. Configure pulsarctl to use the API Key <your-api-key> to verify the API Key is valid and all the permissions are correct.

0. Prerequisites

Install snctl and pulsarctl

Make sure you have the following installed:

• snctl
• pulsarctl

Create a new directory for the tutorial

Create a new directory for the tutorial.

mkdir snctl-getting-started && cd snctl-getting-started

Create a Super-Admin Service Account

You need to create a service account with Super Admin access. Let's name it as snctl-super-admin. After creating the service account, download and save the OAuth2 credentials file. You will use these credentials to configure snctl to authenticate as snctl-super-admin when provisioning resources. Make sure to replace /path/to/oauth2-credentials-file.json with the actual path to your downloaded OAuth2 credentials file.

snctl auth activate-service-account snctl-super-admin --key-file=/path/to/oauth2-credentials-file.json

After the service account is activated, you should see the following message:

Logged in as snctl-super-admin@<your-org-id>.auth.streamnative.cloud.
Welcome to StreamNative Cloud!

Set the target organization as the default organization for snctl.

snctl config set --namespace <your-org-id>

1. Provision a Serverless Instance

Edit a file named 001-instance.yaml with the following content:

apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarInstance
metadata:
  name: <your-instance-name>
  namespace: <your-org-id>
spec:
  availabilityMode: regional
  poolRef:
    name: shared-gcp
    namespace: streamnative
  type: serverless

This yaml file defines a Serverless Instance running in GCP with regional availability mode. Replace the following placeholders with your actual values:

• <your-instance-name>: The name of the Serverless Instance.
• <your-org-id>: The organization ID.

Run the following command to provision the Serverless Instance:

snctl create -f 001-instance.yaml

You should see the following message:

pulsarinstance.cloud.streamnative.io/<your-instance-name> created

Query the instance to verify the instance is created.

snctl get PulsarInstance <your-instance-name> -o yaml

You will be able to see a similar status block of this instance in the output:

status:
  auth:
    oauth2:
      audience: urn:sn:pulsar:<your-org-id>:<your-instance-name>
      issuerURL: https://auth.streamnative.cloud/
    type: oauth2
  conditions:
    - lastTransitionTime: '...'
      message: a payment method is not required because discount is active
      reason: HasActiveDiscount
      status: 'True'
      type: SubscriptionReady
    - lastTransitionTime: '...'
      reason: Created
      status: 'True'
      type: ResourceServerReady
    - lastTransitionTime: '...'
      reason: Created
      status: 'True'
      type: ServiceAccountReady
    - lastTransitionTime: '...'
      reason: AllConditionStatusTrue
      status: 'True'
      type: Ready

When all the conditions are True, the instance is ready.

2. Provision a Serverless Cluster

Edit a file named 002-cluster.yaml with the following content:

apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarCluster
metadata:
  namespace: <your-org-id>
spec:
  # currently the `broker` section is still required despite the fact that
  # settings are not used by serverless
  broker:
    replicas: 2
    resources:
      cpu: '1'
      memory: 4Gi
  displayName: serverless-cluster
  instanceName: <your-instance-name>
  location: us-central1

This yaml file defines a Serverless Cluster in us-central1 region. Replace the following placeholders with your actual values:

• <your-instance-name>: The name of the Serverless Instance.
• <your-org-id>: The organization ID.

Run the following command to provision the Serverless Instance:

snctl create -f 002-cluster.yaml

You should see the following message:

pulsarcluster.cloud.streamnative.io/<your-cluster-name> created

Query the cluster to verify the cluster is created.

snctl get PulsarCluster <your-cluster-name> -o yaml

You will be able to see a similar status block of this cluster in the output:

status:
  broker:
    readyReplicas: 2
    replicas: 2
    updatedReplicas: 2
  conditions:
    - lastTransitionTime: '...'
      reason: Deploy
      status: 'True'
      type: PulsarBrokerReady
    - lastTransitionTime: '...'
      reason: AllConditionStatusTrue
      status: 'True'
      type: Ready
    - lastTransitionTime: '...'
      reason: Ready
      status: 'True'
      type: PulsarInstanceReady
  deploymentType: hosted
  instanceType: serverless

Wait for all the conditions to be True, then the cluster is ready. A Serverless Cluster is usually ready within 1~2 minutes.

3. Provision a Service Account

Edit a file named 003-sa.yaml with the following content:

apiVersion: cloud.streamnative.io/v1alpha1
kind: ServiceAccount
metadata:
  name: <your-service-account-name>
  namespace: <your-org-id>
spec: {}

This yaml file defines a Service Account with a name <your-service-account-name>. Replace the following placeholders with your actual values:

• <your-service-account-name>: The name of the Service Account.
• <your-org-id>: The organization ID.

Run the following command to provision the Service Account:

snctl create -f 003-sa.yaml

You should see the following message:

serviceaccount.cloud.streamnative.io/<your-service-account-name> created

Query the service account to verify the service account is created.

snctl get ServiceAccount <your-service-account-name> -o yaml

You will be able to see a similar status block of this service account in the output:

status:
  conditions:
    - lastTransitionTime: '...'
      reason: Provisioned
      status: 'True'
      type: Ready
  privateKeyData: ...
  privateKeyType: TYPE_SN_CREDENTIALS_FILE

Wait until the Ready condition is True, then the service account is ready.

4. Create an API Key for the Service Account

Edit a file named 004-api-key.yaml with the following content:

apiVersion: cloud.streamnative.io/v1alpha1
kind: APIKey
metadata:
  name: <your-api-key-name>
  namespace: <your-org-id>
spec:
  description: This is a test api key for <your-service-account-name> in running the snctl tutorial
  instanceName: <your-instance-name>
  serviceAccountName: <your-service-account-name>

This yaml file defines an API Key for the Service Account <your-service-account-name>. Replace the following placeholders with your actual values:

• <your-api-key-name>: The name of the API Key.
• <your-service-account-name>: The name of the Service Account.
• <your-instance-name>: The name of the Serverless Instance.
• <your-org-id>: The organization ID.

Run the following command to create the API Key:

snctl create -f 004-api-key.yaml

You should see the following message:

apikey.cloud.streamnative.io/<your-api-key-name> created

Query the API Key to verify the API Key is created.

snctl get apikey <your-api-key-name> -o yaml

You will be able to see a similar status block of this API Key in the output:

status:
  conditions:
    - lastTransitionTime: '...'
      message: ''
      reason: API Key has been provisioned
      status: 'True'
      type: Issued
    - lastTransitionTime: '...'
      message: ''
      reason: API Key is not revoked
      status: 'False'
      type: Revoked
    - lastTransitionTime: '...'
      message: ''
      reason: API Key will never expire
      status: 'False'
      type: Expired
  expiresAt: '1970-01-01T00:00:00Z'
  issuedAt: '...'
  keyId: <key-id>
  token: <token>

Wait until the Issued condition is True, then the API Key is issued and ready to use. You can obtain the token from the token field in the status block.

You can use the following command to obtain the token and export it as an environment variable API_KEY_TOKEN:

export API_KEY_TOKEN=$(snctl get apikey <your-api-key-name> -o jsonpath='{.status.token}')

5. Configure pulsarctl to provision Pulsar resources

First, get the admin service URL of the cluster by running the following command:

export ADMIN_SERVICE_URL="https://$(snctl get PulsarCluster <your-cluster-name> -o jsonpath='{.spec.serviceEndpoints[0].dnsName}')"
export BROKER_SERVICE_URL="pulsar+ssl://$(snctl get PulsarCluster <your-cluster-name> -o jsonpath='{.spec.serviceEndpoints[1].dnsName}'):6651"

Once you get the ADMIN_SERVICE_URL, you can use the following command to configure pulsarctl to access the cluster we created in the previous steps:

pulsarctl context set -s ${ADMIN_SERVICE_URL} --key-file /path/to/oauth2-credentials-file.json --audience urn:sn:pulsar:<your-org-id>:<your-instance-name> <your-cluster-name>-admin

This command will create a new context named <your-cluster-name>-admin and update the pulsarctl configuration to use the oauth2 credentials of snctl-super-admin to authenticate to the cluster.

You should see the following message:

Context "<your-cluster-name>-admin" created.

You can verify the pulsarctl has been configured properly by running the following command:

pulsarctl context current

You should see the following message:

<your-cluster-name>-admin

Then you can run pulsarctl tenants list to verify if you configured the pulsarctl properly.

pulsarctl tenants list -o yaml

You should be able to see the tenants in the cluster.

- public
- pulsar
- sn

6. Create Pulsar Resources

Assume you want to build a sample application sl-app that produces messages to a topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic and consumes messages from the same topic.

First, create a tenant named sl-app-tenant.

pulsarctl tenants create sl-app-tenant --allowed-clusters <your-cluster-name>

Output:

Create tenant sl-app-tenant successfully

Second, create a namespace named sl-app-ns under the tenant sl-app-tenant.

pulsarctl namespaces create sl-app-tenant/sl-app-ns --clusters <your-cluster-name>

Output:

Created sl-app-tenant/sl-app-ns successfully

Next, create a topic named sl-app-topic with 4 partitions under the namespace sl-app-tenant/sl-app-ns.

pulsarctl topics create sl-app-tenant/sl-app-ns/sl-app-topic 4

Output:

Create topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic with 4 partitions successfully

Finally, grant the Service Account <your-service-account-name> the permission to produce and consume messages from the namespace sl-app-tenant/sl-app-ns.

pulsarctl namespaces grant-permission --role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud --actions produce,consume sl-app-tenant/sl-app-ns

Output:

Grant permissions [produce consume] to the client role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud to access the namespace sl-app-tenant/sl-app-ns successfully

7. Product and consume messages using pulsar-client

1. Download the Pulsar distribution from Pulsar Downloads. Assume you have downloaded the Pulsar distribution and extracted the tarball to /path/to/pulsar-dist.

2. Enter the root directory of the Pulsar distribution:

   cd /path/to/pulsar-dist
   

3. Configure the conf/client.conf file:

   • webServiceUrl: Set the webServiceUrl to the ADMIN_SERVICE_URL you obtained in the previous steps.
   • brokerServiceUrl: Set the brokerServiceUrl to the BROKER_SERVICE_URL you obtained in the previous steps.
   • authPlugin: Set the authPlugin to org.apache.pulsar.client.impl.auth.AuthenticationToken.
   • authParams: Set the authParams to be token:<your-api-key>. <your-api-key> is the API Key you obtained in the previous steps.

4. Produce 10 messages.

   bin/pulsar-client produce -m "hello sl-app" -n 10 sl-app-tenant/sl-app-ns/sl-app-topic
   

   You should see a similar message in the output:

   10 messages successfully produced
   

5. Consume the messages.

   bin/pulsar-client consume -n 10 -p Earliest -s sl-app-sub sl-app-tenant/sl-app-ns/sl-app-topic
   

   You should see a similar message in the output:

   ----- got message -----
   publishTime:[1732951012862], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951012952], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013162], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013095], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013299], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013368], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013436], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013510], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013025], eventTime:[0], key:[null], properties:[], content:hello sl-app
   ----- got message -----
   publishTime:[1732951013229], eventTime:[0], key:[null], properties:[], content:hello sl-app
   

   You should see a final message in the output:

   10 messages successfully consumed
   

8. Next Steps

Once you have verified the application works as expected, you can try out more guided tutorials:

• Kafka Client Guides
• Pulsar Client Guides
• Run Pulsar I/O Connectors
• Run Kafka Connect Connectors
• Deploy Pulsar Functions

9. Clean up

After you finish the tutorial, you can clean up the resources you created in this tutorial by running the following command:

snctl delete -f 004-api-key.yaml
snctl delete -f 003-sa.yaml
snctl delete PulsarCluster <your-cluster-name>
snctl delete -f 001-instance.yaml