StreamNative CLI Tutorial
This tutorial demonstrates how to use the StreamNative command-line tools to deploy a Serverless cluster and manage Pulsar resources within it. We will primarily focus on the modern, unified approach using StreamNative Cloud CLI (snctl
) for both cloud infrastructure and Pulsar resource management/interaction. We will also show alternative methods using the traditional pulsarctl
and pulsar-client
tools.
This tutorial covers:
- Provisioning a Serverless Instance (
snctl
). - Provisioning a Serverless Cluster (
snctl
). - Provisioning an Application Service Account (
snctl
). - Creating an API Key for the Service Account (
snctl
). - Method 1 (Unified
snctl
):- Configuring
snctl
Service Context. - Creating Pulsar resources (tenant, namespace, topic) using
snctl pulsar admin
. - Granting permissions using
snctl pulsar admin
. - Producing/Consuming messages using
snctl pulsar client
.
- Configuring
- Method 2 (Traditional Tools - Alternatives):
- Configuring
pulsarctl
context using the API Key. - Creating Pulsar resources using
pulsarctl
. - Granting permissions using
pulsarctl
. - Producing/Consuming messages using
pulsar-client
configured with the API Key.
- Configuring
0. Prerequisites
Install Required CLIs
Make sure you have the following installed:
- snctl (v1.0.0+ recommended)
- pulsarctl (Needed for the alternative method)
- An Apache Pulsar distribution (e.g., from Pulsar Downloads) for the
pulsar-client
alternative method.
Create a new directory for the tutorial
Create a new directory for the tutorial.
mkdir snctl-getting-started && cd snctl-getting-started
Create and Activate a Super-Admin Service Account
You need to create a service account with Super Admin access in StreamNative Cloud Console. Let's name it snctl-super-admin
. After creating the service account, download and save its OAuth2 credentials file (e.g., snctl-super-admin-credentials.json
).
Activate this service account for snctl
. This identity will be used for provisioning cloud resources (instances, clusters, other service accounts). Make sure to replace /path/to/snctl-super-admin-credentials.json
with the actual path.
# Activate the super-admin service account
snctl auth activate-service-account --key-file=/path/to/snctl-super-admin-credentials.json
After the service account is activated, you should see a similar message:
Logged in as snctl-super-admin@<your-org-id>.auth.streamnative.cloud.
Welcome to StreamNative Cloud!
Set the target organization as the default organization for snctl
to avoid specifying -n
or -O
repeatedly. Replace <your-org-id>
with your actual organization ID.
snctl config set --organization <your-org-id>
1. Provision a Serverless Instance
Edit a file named 001-instance.yaml
with the following content:
apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarInstance
metadata:
name: <your-instance-name>
namespace: <your-org-id>
spec:
availabilityMode: regional
poolRef:
name: shared-gcp
namespace: streamnative
type: serverless
This yaml file defines a Serverless Instance running in GCP with regional availability mode. Replace the following placeholders with your actual values:
<your-instance-name>
: The name of the Serverless Instance.<your-org-id>
: The organization ID.
Run the following command to provision the Serverless Instance:
snctl create -f 001-instance.yaml
You should see the following message:
pulsarinstance.cloud.streamnative.io/<your-instance-name> created
Query the instance to verify the instance is created.
snctl get PulsarInstance <your-instance-name> -o yaml
You will be able to see a similar status block of this instance in the output:
status:
auth:
oauth2:
audience: urn:sn:pulsar:<your-org-id>:<your-instance-name>
issuerURL: https://auth.streamnative.cloud/
type: oauth2
conditions:
- lastTransitionTime: '...'
message: a payment method is not required because discount is active
reason: HasActiveDiscount
status: 'True'
type: SubscriptionReady
- lastTransitionTime: '...'
reason: Created
status: 'True'
type: ResourceServerReady
- lastTransitionTime: '...'
reason: Created
status: 'True'
type: ServiceAccountReady
- lastTransitionTime: '...'
reason: AllConditionStatusTrue
status: 'True'
type: Ready
When all the conditions are True
, the instance is ready.
2. Provision a Serverless Cluster
Edit a file named 002-cluster.yaml
with the following content:
apiVersion: cloud.streamnative.io/v1alpha1
kind: PulsarCluster
metadata:
namespace: <your-org-id>
spec:
# currently the `broker` section is still required despite the fact that
# settings are not used by serverless
broker:
replicas: 2
resources:
cpu: '1'
memory: 4Gi
displayName: serverless-cluster
instanceName: <your-instance-name>
location: us-central1
This yaml file defines a Serverless Cluster in us-central1
region. Replace the following placeholders with your actual values:
<your-instance-name>
: The name of the Serverless Instance.<your-org-id>
: The organization ID.
Run the following command to provision the Serverless Instance:
snctl create -f 002-cluster.yaml
You should see the following message:
pulsarcluster.cloud.streamnative.io/<your-cluster-name> created
Note
Please note the <your-cluster-name>
in the output message because the cluster name of a Serverless Cluster is generated by StreamNative Cloud. You will need this cluster name in the future steps.
Query the cluster to verify the cluster is created.
snctl get PulsarCluster <your-cluster-name> -o yaml
You will be able to see a similar status block of this cluster in the output:
status:
broker:
readyReplicas: 2
replicas: 2
updatedReplicas: 2
conditions:
- lastTransitionTime: '...'
reason: Deploy
status: 'True'
type: PulsarBrokerReady
- lastTransitionTime: '...'
reason: AllConditionStatusTrue
status: 'True'
type: Ready
- lastTransitionTime: '...'
reason: Ready
status: 'True'
type: PulsarInstanceReady
deploymentType: hosted
instanceType: serverless
Wait for all the conditions to be True
, then the cluster is ready. A Serverless Cluster is usually ready within 1~2 minutes.
3. Provision a Service Account
Edit a file named 003-sa.yaml
with the following content:
apiVersion: cloud.streamnative.io/v1alpha1
kind: ServiceAccount
metadata:
name: <your-service-account-name>
namespace: <your-org-id>
spec: {}
This yaml file defines a Service Account with a name <your-service-account-name>
. Replace the following placeholders with your actual values:
<your-service-account-name>
: The name of the Service Account.<your-org-id>
: The organization ID.
Run the following command to provision the Service Account:
snctl create -f 003-sa.yaml
You should see the following message:
serviceaccount.cloud.streamnative.io/<your-service-account-name> created
Query the service account to verify the service account is created.
snctl get ServiceAccount <your-service-account-name> -o yaml
You will be able to see a similar status block of this service account in the output:
status:
conditions:
- lastTransitionTime: '...'
reason: Provisioned
status: 'True'
type: Ready
privateKeyData: ...
privateKeyType: TYPE_SN_CREDENTIALS_FILE
Wait until the Ready
condition is True
, then the service account is ready.
4. Create an API Key for the Service Account (Optional but shown for completeness)
While snctl
typically uses OAuth2 via auth activate-service-account
or context impersonation (--as-service-account
), you might need an API Key for external clients or tools that only support token authentication. This API key will be used in the alternative pulsarctl
and pulsar-client
method later.
Edit a file named 004-api-key.yaml
:
apiVersion: cloud.streamnative.io/v1alpha1
kind: APIKey
metadata:
name: <your-api-key-name>
namespace: <your-org-id>
spec:
description: This is a test api key for <your-service-account-name> in running the snctl tutorial
instanceName: <your-instance-name>
serviceAccountName: <your-service-account-name>
This yaml file defines an API Key for the Service Account <your-service-account-name>
. Replace the following placeholders with your actual values:
<your-api-key-name>
: The name of the API Key.<your-service-account-name>
: The name of the Service Account.<your-instance-name>
: The name of the Serverless Instance.<your-org-id>
: The organization ID.
Run the following command to create the API Key:
snctl create -f 004-api-key.yaml
You should see the following message:
apikey.cloud.streamnative.io/<your-api-key-name> created
Query the API Key to verify the API Key is created and optionally retrieve the token.
snctl get apikey <your-api-key-name> -o yaml
You will be able to see a similar status block of this API Key in the output:
status:
conditions:
- lastTransitionTime: '...'
message: ''
reason: API Key has been provisioned
status: 'True'
type: Issued
- lastTransitionTime: '...'
message: ''
reason: API Key is not revoked
status: 'False'
type: Revoked
- lastTransitionTime: '...'
message: ''
reason: API Key will never expire
status: 'False'
type: Expired
expiresAt: '1970-01-01T00:00:00Z'
issuedAt: '...'
keyId: <key-id>
token: <token>
Wait until the Issued
condition is True
, then the API Key is issued and ready to use. You can obtain the token from the token
field in the status block.
You can use the following command to obtain the token and export it as an environment variable API_KEY_TOKEN
:
export API_KEY_TOKEN=$(snctl get apikey <your-api-key-name> -o jsonpath='{.status.token}')
Method 1: Unified Management with snctl
This section demonstrates using snctl
for configuring access, managing Pulsar resources, and interacting with the data plane.
5. Configure snctl
Service Context for Pulsar Interaction
snctl
uses Service Contexts to manage connections to Pulsar/Kafka clusters. After creating a cluster, snctl
usually discovers it automatically. Let's explicitly set the context for the cluster we created to ensure subsequent commands target it correctly.
Set the active context to your newly created cluster. Replace <your-instance-name>
and <your-cluster-name>
with the actual name from step 2.
snctl context use --pulsar-instance <your-instance-name> --pulsar-cluster <your-cluster-name>
Verify the current context:
snctl context current
Now, verify connectivity by listing tenants. Since we activated the snctl-super-admin
service account (in step 0), snctl
commands will run as that identity by default.
snctl pulsar admin tenants list
You should see the default tenants:
public
pulsar
sn
This confirms snctl
can communicate with the Pulsar cluster's admin endpoint using the super-admin credentials via the active context.
6. Create Pulsar Resources using snctl
Now, let's create the tenant, namespace, and topic for our application sl-app
using snctl pulsar admin
commands. These commands will use the active context (<your-cluster-name>
) and run as the activated super-admin user (snctl-super-admin
).
First, create a tenant named sl-app-tenant
.
snctl pulsar admin tenants create sl-app-tenant --allowed-clusters <your-cluster-name>
Output:
Tenant "sl-app-tenant" created successfully.
Second, create a namespace named sl-app-ns
under the tenant sl-app-tenant
.
snctl pulsar admin namespaces create sl-app-tenant/sl-app-ns --clusters <your-cluster-name>
Output:
Namespace "sl-app-tenant/sl-app-ns" created successfully.
Next, create a partitioned topic named sl-app-topic
with 4 partitions under the namespace sl-app-tenant/sl-app-ns
.
snctl pulsar admin topics create sl-app-tenant/sl-app-ns/sl-app-topic 4
Output:
Create topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic with 4 partitions successfully
Finally, grant the application Service Account <your-service-account-name>
(created in step 3) the permission to produce and consume messages within the sl-app-tenant/sl-app-ns
namespace. We are still running as snctl-super-admin
to grant these permissions. Replace <your-service-account-name>
with the name from step 3.
snctl pulsar admin namespaces grant-permission --role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud --actions produce,consume sl-app-tenant/sl-app-ns
Output:
Grant permissions [produce consume] to the client role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud to access the namespace sl-app-tenant/sl-app-ns successfully
7. Produce and consume messages using snctl pulsar client
Now we'll use snctl pulsar client
commands to produce and consume messages. Crucially, these actions should be performed as the application service account (<your-service-account-name>
) because we granted it the produce/consume permissions, not the super-admin. We use the --as-service-account
flag for this, leveraging snctl
's ability to impersonate the specified service account (assuming the logged-in super-admin has permission to do so, which is typical).
Produce 10 messages to the topic, acting as the application service account. Replace <your-service-account-name>
with the name from step 3.
snctl pulsar client produce --topic sl-app-tenant/sl-app-ns/sl-app-topic \
--messages "hello sl-app from snctl" \
--num-times 10 \
--as-service-account <your-service-account-name>
You should see output indicating successful production, similar to:
Successfully produced 10 message(s) to topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic
(Exact output message may vary)
Consume the 10 messages from the topic, again acting as the application service account. Replace <your-service-account-name>
with the name from step 3.
snctl pulsar client consume --topic sl-app-tenant/sl-app-ns/sl-app-topic \
--subscription-name sl-app-sub \
--num-messages 10 \
--initial-position earliest \
--as-service-account <your-service-account-name>
You should see the 10 messages printed to your console, similar to:
----- got message -----
# ... message details ... content:hello sl-app from snctl
----- got message -----
# ... message details ... content:hello sl-app from snctl
... (10 messages total) ...
Followed by a confirmation like:
Consumed 10 message(s) from topic sl-app-tenant/sl-app-ns/sl-app-topic
(Exact output format may vary)
This demonstrates using snctl
for the entire lifecycle using the unified approach.
Method 2: Traditional Management with pulsarctl
and pulsar-client
(Alternative)
This section demonstrates the alternative approach using the separate pulsarctl
tool for admin tasks and the pulsar-client
tool for producing/consuming. This method often relies on API Key authentication for simplicity when interacting with StreamNative Cloud clusters via these tools.
Alt 5. Configure pulsarctl
Context (Using API Key)
First, get the admin service URL of the cluster by running the following command:
export ADMIN_SERVICE_URL="https://$(snctl get PulsarCluster <your-cluster-name> -o jsonpath='{.spec.serviceEndpoints[0].dnsName}')"
export BROKER_SERVICE_URL="pulsar+ssl://$(snctl get PulsarCluster <your-cluster-name> -o jsonpath='{.spec.serviceEndpoints[1].dnsName}'):6651"
Once you get the ADMIN_SERVICE_URL
, you can use the following command to configure pulsarctl
to access the cluster we created in the previous steps:
pulsarctl context set -s ${ADMIN_SERVICE_URL} --key-file /path/to/oauth2-credentials-file.json --audience urn:sn:pulsar:<your-org-id>:<your-instance-name> <your-cluster-name>-admin
This command will create a new context named <your-cluster-name>-admin
and update the pulsarctl
configuration to use the oauth2 credentials of snctl-super-admin
to authenticate to the cluster.
You should see the following message:
Context "<your-cluster-name>-admin" created.
You can verify the pulsarctl
has been configured properly by running the following command:
pulsarctl context current
You should see the following message:
<your-cluster-name>-admin
Then you can run pulsarctl tenants list
to verify if you configured the pulsarctl
properly.
pulsarctl tenants list -o yaml
You should be able to see the tenants in the cluster.
- public
- pulsar
- sn
Alt 6. Create Pulsar Resources using pulsarctl
Assume you want to build a sample application sl-app
that produces messages to a topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic
and consumes messages from the same topic.
First, create a tenant named sl-app-tenant
.
pulsarctl tenants create sl-app-tenant --allowed-clusters <your-cluster-name>
Output:
Create tenant sl-app-tenant successfully
Second, create a namespace named sl-app-ns
under the tenant sl-app-tenant
.
pulsarctl namespaces create sl-app-tenant/sl-app-ns --clusters <your-cluster-name>
Output:
Created sl-app-tenant/sl-app-ns successfully
Next, create a topic named sl-app-topic
with 4 partitions under the namespace sl-app-tenant/sl-app-ns
.
pulsarctl topics create sl-app-tenant/sl-app-ns/sl-app-topic 4
Output:
Create topic persistent://sl-app-tenant/sl-app-ns/sl-app-topic with 4 partitions successfully
Finally, grant the Service Account <your-service-account-name>
the permission to produce and consume messages from the namespace sl-app-tenant/sl-app-ns
.
pulsarctl namespaces grant-permission --role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud --actions produce,consume sl-app-tenant/sl-app-ns
Output:
Grant permissions [produce consume] to the client role <your-service-account-name>@<your-org-id>.auth.streamnative.cloud to access the namespace sl-app-tenant/sl-app-ns successfully
Alt 7. Produce and consume messages using pulsar-client
Download the Pulsar distribution from Pulsar Downloads. Assume you have downloaded the Pulsar distribution and extracted the tarball to
/path/to/pulsar-dist
.Enter the root directory of the Pulsar distribution:
cd /path/to/pulsar-dist
Configure the
conf/client.conf
file:- webServiceUrl: Set the
webServiceUrl
to theADMIN_SERVICE_URL
you obtained in the previous steps. - brokerServiceUrl: Set the
brokerServiceUrl
to theBROKER_SERVICE_URL
you obtained in the previous steps. - authPlugin: Set the
authPlugin
toorg.apache.pulsar.client.impl.auth.AuthenticationToken
. - authParams: Set the
authParams
to betoken:<your-api-key>
.<your-api-key>
is the API Key you obtained in the previous steps.
- webServiceUrl: Set the
Produce 10 messages.
bin/pulsar-client produce -m "hello sl-app" -n 10 sl-app-tenant/sl-app-ns/sl-app-topic
You should see a similar message in the output:
10 messages successfully produced
Consume the messages.
bin/pulsar-client consume -n 10 -p Earliest -s sl-app-sub sl-app-tenant/sl-app-ns/sl-app-topic
You should see a similar message in the output:
----- got message ----- publishTime:[1732951012862], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951012952], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013162], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013095], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013299], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013368], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013436], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013510], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013025], eventTime:[0], key:[null], properties:[], content:hello sl-app ----- got message ----- publishTime:[1732951013229], eventTime:[0], key:[null], properties:[], content:hello sl-app
You should see a final message in the output:
10 messages successfully consumed
8. Next Steps
Once you have verified the application works as expected using either method, you can try out more guided tutorials:
- Kafka Client Guides
- Pulsar Client Guides
- Run Pulsar I/O Connectors
- Run Kafka Connect Connectors
- Deploy Pulsar Functions
9. Clean up
After you finish the tutorial, you can clean up the resources you created in this tutorial by running the following command:
Note
Please note that you can't use snctl delete -f 002-cluster.yaml
to delete the cluster because the cluster name is generated by StreamNative Cloud. So you need to delete the cluster using the snctl delete PulsarCluster <your-cluster-name>
command.
snctl delete -f 004-api-key.yaml
snctl delete -f 003-sa.yaml
snctl delete PulsarCluster <your-cluster-name>
snctl delete -f 001-instance.yaml