StreamNative CLI Tutorial
This tutorial demonstrates how to use the snctl
command to deploy a Serverless cluster and use the pulsarctl
command to provision Pulsar resources in the Serverless cluster.
This tutorial provisions the following resources (assuming we name the application as sl-app
):
- Provision a Serverless Instance
<your-instance-name>
. - Provision a Serverless Cluster
<your-cluster-name>
. - Provision a Service Account named
<your-service-account-name>
. - Provision an API Key for the service account
<your-api-key>
. - Create a Pulsar Tenant
<your-tenant-name>
. - Create a Pulsar Namespace
<your-namespace-name>
. - Create a Pulsar Topic
<your-topic-name>
with 4 partitions. - Configure
pulsarctl
to use the API Key<your-api-key>
to verify the API Key is valid and all the permissions are correct.
0. Prerequisites
Install snctl and pulsarctl
Make sure you have the following installed:
Create a new directory for the tutorial
Create a new directory for the tutorial.
mkdir snctl-getting-started && cd snctl-getting-started
Create a Super-Admin Service Account
You need to create a service account with Super Admin access. Let's name it as snctl-super-admin
. After creating the service account, download and save the OAuth2 credentials file. You will use these credentials to configure snctl
to authenticate as snctl-super-admin
when provisioning resources. Make sure to replace /path/to/oauth2-credentials-file.json
with the actual path to your downloaded OAuth2 credentials file.
snctl auth activate-service-account snctl-super-admin --key-file=/path/to/oauth2-credentials-file.json
After the service account is activated, you should see the following message:
Logged in as snctl-super-admin@<your-org-id>.auth.streamnative.cloud.
Welcome to StreamNative Cloud!
Set the target organization as the default organization for snctl
.
snctl config set --namespace <your-org-id>
1. Provision a Serverless Instance
Edit a file named 001-instance.yaml
with the following content:
apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarInstance
metadata:
name: <your-instance-name>
namespace: <your-org-id>
spec:
availabilityMode: regional
poolRef:
name: shared-gcp
namespace: streamnative
type: serverless
This yaml file defines a Serverless Instance running in GCP with regional availability mode. Replace the following placeholders with your actual values:
<your-instance-name>
: The name of the Serverless Instance.<your-org-id>
: The organization ID.
Run the following command to provision the Serverless Instance:
snctl create -f 001-instance.yaml
You should see the following message:
pulsarinstance.cloud.streamnative.io/<your-instance-name> created
Query the instance to verify the instance is created.
snctl get PulsarInstance <your-instance-name> -o yaml
You will be able to see a similar status block of this instance in the output:
status:
auth:
oauth2:
audience: urn:sn:pulsar:<your-org-id>:<your-instance-name>
issuerURL: https://auth.streamnative.cloud/
type: oauth2
conditions:
- lastTransitionTime: '...'
message: a payment method is not required because discount is active
reason: HasActiveDiscount
status: 'True'
type: SubscriptionReady
- lastTransitionTime: '...'
reason: Created
status: 'True'
type: ResourceServerReady
- lastTransitionTime: '...'
reason: Created
status: 'True'
type: ServiceAccountReady
- lastTransitionTime: '...'
reason: AllConditionStatusTrue
status: 'True'
type: Ready
When all the conditions are True
, the instance is ready.
2. Provision a Serverless Cluster
Edit a file named 002-cluster.yaml
with the following content:
apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarCluster
metadata:
namespace: <your-org-id>
spec:
# currently the `broker` section is still required despite the fact that
# settings are not used by serverless
broker:
replicas: 2
resources:
cpu: '1'
memory: 4Gi
displayName: serverless-cluster
instanceName: <your-instance-name>
location: us-central1
This yaml file defines a Serverless Cluster in us-central1
region. Replace the following placeholders with your actual values:
<your-instance-name>
: The name of the Serverless Instance.<your-org-id>
: The organization ID.
Run the following command to provision the Serverless Instance:
snctl create -f 002-cluster.yaml
You should see the following message:
pulsarcluster.cloud.streamnative.io/<your-cluster-name> created
Note
Please note the <your-cluster-name>
in the output message because the cluster name of a Serverless Cluster is generated by StreamNative Cloud. You will need this cluster name in the future steps.
Query the cluster to verify the cluster is created.
snctl get PulsarCluster <your-cluster-name> -o yaml
You will be able to see a similar status block of this cluster in the output:
status:
broker:
readyReplicas: 2
replicas: 2
updatedReplicas: 2
conditions:
- lastTransitionTime: '...'
reason: Deploy
status: 'True'
type: PulsarBrokerReady
- lastTransitionTime: '...'
reason: AllConditionStatusTrue
status: 'True'
type: Ready
- lastTransitionTime: '...'
reason: Ready
status: 'True'
type: PulsarInstanceReady
deploymentType: hosted
instanceType: serverless
Wait for all the conditions to be True
, then the cluster is ready. A Serverless Cluster is usually ready within 1~2 minutes.
3. Provision a Service Account
Edit a file named 003-sa.yaml
with the following content:
apiVersion: cloud.streamnative.io/v1alpha1
kind: ServiceAccount
metadata:
name: <your-service-account-name>
namespace: <your-org-id>
spec: {}
This yaml file defines a Service Account with a name <your-service-account-name>
. Replace the following placeholders with your actual values:
<your-service-account-name>
: The name of the Service Account.<your-org-id>
: The organization ID.
Run the following command to provision the Service Account:
snctl create -f 003-sa.yaml
You should see the following message:
serviceaccount.cloud.streamnative.io/<your-service-account-name> created
Query the service account to verify the service account is created.
snctl get ServiceAccount <your-service-account-name> -o yaml
You will be able to see a similar status block of this service account in the output:
status:
conditions:
- lastTransitionTime: '...'
reason: Provisioned
status: 'True'
type: Ready
privateKeyData: ...
privateKeyType: TYPE_SN_CREDENTIALS_FILE
Wait until the Ready
condition is True
, then the service account is ready.
4. Create an API Key for the Service Account
Edit a file named 004-api-key.yaml
with the following content:
apiVersion: cloud.streamnative.io/v1alpha1
kind: APIKey
metadata:
name: <your-api-key-name>
namespace: <your-org-id>
spec:
description: This is a test api key for <your-service-account-name> in running the snctl tutorial
instanceName: <your-instance-name>
serviceAccountName: <your-service-account-name>
This yaml file defines an API Key for the Service Account <your-service-account-name>
. Replace the following placeholders with your actual values:
<your-api-key-name>
: The name of the API Key.<your-service-account-name>
: The name of the Service Account.<your-instance-name>
: The name of the Serverless Instance.<your-org-id>
: The organization ID.
Run the following command to create the API Key:
snctl create -f 004-api-key.yaml
You should see the following message:
apikey.cloud.streamnative.io/<your-api-key-name> created
Query the API Key to verify the API Key is created.
snctl get apikey <your-api-key-name> -o yaml
You will be able to see a similar status block of this API Key in the output:
status:
conditions:
- lastTransitionTime: '...'
message: ''
reason: API Key has been provisioned
status: 'True'
type: Issued
- lastTransitionTime: '...'
message: ''
reason: API Key is not revoked
status: 'False'
type: Revoked
- lastTransitionTime: '...'
message: ''
reason: API Key will never expire
status: 'False'
type: Expired
expiresAt: '1970-01-01T00:00:00Z'
issuedAt: '...'
keyId: <key-id>
token: <token>
Wait until the Issued
condition is True
, then the API Key is issued and ready to use. You can obtain the token from the token
field in the status block.
You can use the following command to obtain the token and export it as an environment variable API_KEY_TOKEN
:
export API_KEY_TOKEN=$(snctl get apikey <your-api-key-name> -o jsonpath='{.status.token}')
5. Configure pulsarctl
to provision Pulsar resources
First, get the admin service URL of the cluster by running the following command:
export ADMIN_SERVICE_URL="https://$(snctl get PulsarCluster <your-cluster-name> -o jsonpath='{.spec.serviceEndpoints[0].dnsName}')"
export BROKER_SERVICE_URL="pulsar+ssl://$(snctl get PulsarCluster <your-cluster-name> -o jsonpath='{.spec.serviceEndpoints[1].dnsName}'):6651"
Once you get the ADMIN_SERVICE_URL
, you can use the following command to configure pulsarctl
to access the cluster we created in the previous steps:
pulsarctl context set -s ${ADMIN_SERVICE_URL} --key-file /path/to/oauth2-credentials-file.json --audience urn:sn:pulsar:<your-org-id>:<your-instance-name> <your-cluster-name>-admin
This command will create a new context named <your-cluster-name>-admin
and update the pulsarctl
configuration to use the oauth2 credentials of snctl-super-admin
to authenticate to the cluster.
You should see the following message:
Context "<your-cluster-name>-admin" created.
You can verify the pulsarctl
has been configured properly by running the following command:
pulsarctl context current
You should see the following message:
<your-cluster-name>-admin
Then you can run pulsarctl tenants list
to verify if you configured the pulsarctl
properly.
pulsarctl tenants list -o yaml
You should be able to see the tenants in the cluster.
- public
- pulsar
- sn
6. Create Pulsar Resources
Assume you want to build a sample application sl-app
that produces messages to a topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic
and consumes messages from the same topic.
First, create a tenant named sl-app-tenant
.
pulsarctl tenants create sl-app-tenant --allowed-clusters <your-cluster-name>
Output:
Create tenant sl-app-tenant successfully
Second, create a namespace named sl-app-ns
under the tenant sl-app-tenant
.
pulsarctl namespaces create sl-app-tenant/sl-app-ns --clusters <your-cluster-name>
Output:
Created sl-app-tenant/sl-app-ns successfully
Next, create a topic named sl-app-topic
with 4 partitions under the namespace sl-app-tenant/sl-app-ns
.
pulsarctl topics create sl-app-tenant/sl-app-ns/sl-app-topic 4
Output:
Create topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic with 4 partitions successfully
Finally, grant the Service Account <your-service-account-name>
the permission to produce and consume messages from the namespace sl-app-tenant/sl-app-ns
.
pulsarctl namespaces grant-permission --role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud --actions produce,consume sl-app-tenant/sl-app-ns
Output:
Grant permissions [produce consume] to the client role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud to access the namespace sl-app-tenant/sl-app-ns successfully
7. Product and consume messages using pulsar-client
Download the Pulsar distribution from Pulsar Downloads. Assume you have downloaded the Pulsar distribution and extracted the tarball to
/path/to/pulsar-dist
.Enter the root directory of the Pulsar distribution:
cd /path/to/pulsar-dist
Configure the
conf/client.conf
file:- webServiceUrl: Set the
webServiceUrl
to theADMIN_SERVICE_URL
you obtained in the previous steps. - brokerServiceUrl: Set the
brokerServiceUrl
to theBROKER_SERVICE_URL
you obtained in the previous steps. - authPlugin: Set the
authPlugin
toorg.apache.pulsar.client.impl.auth.AuthenticationToken
. - authParams: Set the
authParams
to betoken:<your-api-key>
.<your-api-key>
is the API Key you obtained in the previous steps.
- webServiceUrl: Set the
Produce 10 messages.
bin/pulsar-client produce -m "hello sl-app" -n 10 sl-app-tenant/sl-app-ns/sl-app-topic
You should see a similar message in the output:
10 messages successfully produced
Consume the messages.
bin/pulsar-client consume -n 10 -p Earliest -s sl-app-sub sl-app-tenant/sl-app-ns/sl-app-topic
You should see a similar message in the output:
----- got message ----- publishTime:[1732951012862], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951012952], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013162], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013095], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013299], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013368], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013436], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013510], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013025], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013229], eventTime:[0], key:[null], properties:[], content:hello sl-app
You should see a final message in the output:
10 messages successfully consumed
8. Next Steps
Once you have verified the application works as expected, you can try out more guided tutorials:
- Kafka Client Guides
- Pulsar Client Guides
- Run Pulsar I/O Connectors
- Run Kafka Connect Connectors
- Deploy Pulsar Functions
9. Clean up
After you finish the tutorial, you can clean up the resources you created in this tutorial by running the following command:
Note
Please note that you can't use snctl delete -f 002-cluster.yaml
to delete the cluster because the cluster name is generated by StreamNative Cloud. So you need to delete the cluster using the snctl delete PulsarCluster <your-cluster-name>
command.
snctl delete -f 004-api-key.yaml
snctl delete -f 003-sa.yaml
snctl delete PulsarCluster <your-cluster-name>
snctl delete -f 001-instance.yaml