StreamNative Cloud is a resilient, scalable, data streaming service powered by the URSA engine, delivered as a fully managed Pulsar and Kafka service.
StreamNative Cloud provides multiple interfaces for management and interaction:
StreamNative Cloud Console: A user-friendly web-based interface for managing cluster resources, configuring settings, and handling billing.
Command-Line Interface (CLI):
snctl
): The unified command-line interface for deploying and managing StreamNative Cloud infrastructure and interacting directly with your Pulsar clusters and Kafka-protocol endpoints.pulsarctl
): For managing cluster-specific resources, such as tenants, namespaces, topics, functions, connectors, and more.REST APIs: For programmatic access and integration with other systems.
Terraform Providers:
These tools provide flexibility in how you interact with and manage your StreamNative Cloud environment, catering to different user preferences and use cases.
Sign up for StreamNative Cloud and get $200 of free credits. No credit card required.
This quick start guides you through getting started with StreamNative BYOC. It demonstrates how to use a StreamNative BYOC cluster to create topics, produce data to the cluster, and consume data from it.
This QuickStart assumes you are familiar with the basics concepts of StreamNative Cloud Clusters.
Access to StreamNative Cloud.
Internet connectivity.
Access to Your AWS Account for provisioning the BYOC infrastructure.
Ensure you have installed Python 3.0 or higher versions and the Pulsar Python client.
If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
To sign up, navigate to the StreamNative Cloud Console signup page. Follow the prompts to create an account. After you click Finish, you might have to wait briefly for your first organization to be created. After your new organization is created, continue on to creating your first instance and cluster.
Before deploying a StreamNative cluster within your cloud account, you must first grant StreamNative vendor access. Follow the instructions in Account Access for BYOC on AWS to provision AWS access for StreamNative Cloud. This step ensures StreamNative has the necessary permissions to manage resources in your AWS account.
Once completed, please note the account ID of the AWS account you have granted access to StreamNative Cloud. You will use this account ID to create a Cloud Connection.
In the upper-right corner of the StreamNative Cloud Console, click your Profile and select Cloud Environments.
Select Cloud Connections tab and click New Cloud Connection.
In the Create connection dialog, enter the following information:
my-aws-connection
.Click Submit.
The cloud connection creation process will start. Once completed, you can see the status of the cloud connection turned to Connected
in the Cloud Connections tab.
In the upper-right corner of the StreamNative Cloud Console, click your Profile and select Cloud Environments.
Select Cloud Connections tab and click New Cloud Connection.
In the Create connection dialog, enter the following information:
my-aws-connection
.Click Submit.
The cloud connection creation process will start. Once completed, you can see the status of the cloud connection turned to Connected
in the Cloud Connections tab.
Once you have created a cloud connection, you can create a cloud environment and provision a BYOC instance.
Navigate to Organization Dashboard.
Select Instances at the left navigation pane.
On the Instances page, click + New Instance.
On the Choose the deployment type for your instance page, click Deploy BYOC.
You will see a dialog “Cloud Environment required”. Click Create button to create a cloud environment.
On the Cloud Connection page, select the cloud connection you created in the previous step. In this case, it is my-aws-connection
. Then click Environment setup.
Then fill out the information for the cloud environment.
us-west-2
.poc
.10.0.0.0/16
. If you need to specify a different CIDR, you can enter it here.Click Create.
Navigate to Organization Dashboard.
Select Instances at the left navigation pane.
On the Instances page, click + New Instance.
On the Choose the deployment type for your instance page, click Deploy BYOC.
You will see a dialog “Cloud Environment required”. Click Create button to create a cloud environment.
On the Cloud Connection page, select the cloud connection you created in the previous step. In this case, it is my-aws-connection
. Then click Environment setup.
Then fill out the information for the cloud environment.
us-west-2
.poc
.10.0.0.0/16
. If you need to specify a different CIDR, you can enter it here.Click Create.
The provisioning process of a cloud environment usually takes about 40 minutes to complete. You can safely close the page and come back later. You will also receive an email notification when the cloud environment is ready.
For more information about provisioning BYOC infrastructure, see Provision BYOC Infrastructure.
Once the cloud environment is ready, you can create a Pulsar instance.
Navigate to Organization Dashboard.
Select Instances at the left navigation pane.
On the Instances page, click + New Instance.
On the Choose the deployment type for your instance page, click Deploy BYOC again. You will not see the dialog “Cloud Environment required” this time.
On the Instance Configuration page, fill out the information for the Pulsar instance. Then, click Cluster Location to start the cluster creation process.
my-pulsar-instance
.my-aws-connection
.Classic Engine
.On the Cluster Location page, fill out the information for the cluster.
my-byoc-cluster
.aws-usw2-poc-<random-suffix>
.Click Cluster Operations to choose the release channel for the cluster and configure the cluster.
LTS
.Click Cluster Size to configure the cluster size.
At the right navigation pane, you can see the estimated cost for the cluster.
Click Finish to start the cluster creation process.
The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take several minutes to provision the cluster. Once the cluster has been provisioned, the page will show “Cluster Provisioned successfully” and you can click Go To The Dashboard to access the Cluster Dashboard page.
Navigate to Organization Dashboard.
Select Instances at the left navigation pane.
On the Instances page, click + New Instance.
On the Choose the deployment type for your instance page, click Deploy BYOC again. You will not see the dialog “Cloud Environment required” this time.
On the Instance Configuration page, fill out the information for the Pulsar instance. Then, click Cluster Location to start the cluster creation process.
my-pulsar-instance
.my-aws-connection
.Classic Engine
.On the Cluster Location page, fill out the information for the cluster.
my-byoc-cluster
.aws-usw2-poc-<random-suffix>
.Click Cluster Operations to choose the release channel for the cluster and configure the cluster.
LTS
.Click Cluster Size to configure the cluster size.
At the right navigation pane, you can see the estimated cost for the cluster.
Click Finish to start the cluster creation process.
The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take several minutes to provision the cluster. Once the cluster has been provisioned, the page will show “Cluster Provisioned successfully” and you can click Go To The Dashboard to access the Cluster Dashboard page.
Now you can get started configuring apps and data on your new cluster.
To interact with your cluster by producing and consuming messages, you need to set up authentication and authorization. This is done by creating a Service Account, which serves as an identity for authenticating and authorizing access to the cluster. The service account will provide the necessary credentials for your applications to securely connect and perform operations on the Pulsar cluster.
In the upper-right corner of the StreamNative Cloud Console, click your Profile and select Accounts & Accesses.
On the left navigation pane, click Service Accounts.
On the Service Account page, click + New.
On the Create Service Account dialog, enter a name for the service account, and then click Confirm.
On the Service Account page, in the row of the service account you just created, click the … icon, and select Create API Key in the dropdown menu.
On the New API Key dialog:
An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see Use API Keys to Authenticate to StreamNative Cloud.
In the upper-right corner of the StreamNative Cloud Console, click your Profile and select Accounts & Accesses.
On the left navigation pane, click Service Accounts.
On the Service Account page, click + New.
On the Create Service Account dialog, enter a name for the service account, and then click Confirm.
On the Service Account page, in the row of the service account you just created, click the … icon, and select Create API Key in the dropdown menu.
On the New API Key dialog:
An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see Use API Keys to Authenticate to StreamNative Cloud.
After creating the service account and obtaining the API key, the next crucial step is to authorize the service account. This process grants the necessary permissions for the service account to interact with your StreamNative Cloud cluster.
Authorization involves setting up Access Control Lists (ACLs) that define what actions the service account can perform. Typically, you’ll want to grant permissions for producing and consuming messages on specific topics or namespaces.
For more information about authorization, see Authorization and ACLs.
Go back to the Cluster Dashboard page.
On the left navigation pane, click Instances.
On the Instances page, click the name of the instance you created in Step 2.
On the Cluster Dashboard page, click Tenants on the left navigation pane.
On the Tenants page, click + New Tenant.
On the New Tenant dialog:
On the Tenants page, click the name of the tenant you just created. You will be directed to the Tenant Dashboard page.
On the Tenant Dashboard page, click Namespaces on the left navigation pane.
On the Namespaces page, click New Namespace.
On the New Namespace dialog:
On the Namespaces page, click the name of the namespace you just created. You will be directed to the Namespace Dashboard page.
On the Namespace Dashboard page, click Configuration on the left navigation pane.
On the Namespace configuration page, click ADD ROLE. Select the name of the service account you just created, and choose the consume and produce permissions. This grants your service account the produce
and consume
permissions for this namespace.
Go back to the Cluster Dashboard page.
On the left navigation pane, click Instances.
On the Instances page, click the name of the instance you created in Step 2.
On the Cluster Dashboard page, click Tenants on the left navigation pane.
On the Tenants page, click + New Tenant.
On the New Tenant dialog:
On the Tenants page, click the name of the tenant you just created. You will be directed to the Tenant Dashboard page.
On the Tenant Dashboard page, click Namespaces on the left navigation pane.
On the Namespaces page, click New Namespace.
On the New Namespace dialog:
On the Namespaces page, click the name of the namespace you just created. You will be directed to the Namespace Dashboard page.
On the Namespace Dashboard page, click Configuration on the left navigation pane.
On the Namespace configuration page, click ADD ROLE. Select the name of the service account you just created, and choose the consume and produce permissions. This grants your service account the produce
and consume
permissions for this namespace.
Now you have created a tenant, namespace, and granted the service account the produce
and consume
permissions for this namespace. You can now continue on to building a Python app, connecting to the cluster, and producing and consuming messages.
This QuickStart provides you with an example Python app to get you up and running with consuming and producing messages. This is a simple example and is not intended for production environments.
Return to the StreamNative Cloud Console and go to the “Cluster Dashboard” page.
On the left navigation pane, click Pulsar Clients.
On the Pulsar client setup page, click the Code libraries tab, follow the setup wizard to get the sample codes for your producer and consumer.
a. Select Python as the client library and click Next.
b. Select the service account you created and click Next.
c. Select API Key as the authentication type and click Next.
d. Install pulsar-client
python client library.
e. Select the target tenant, namespace, topic, and subscription.
f. You are now ready to copy the auto-generated sample codes.
Return to your text editor and create two new files: producer.py
and consumer.py
. Copy and paste the sample code for the producer into producer.py
and the sample code for the consumer into consumer.py
. In both files, replace <JWT Token>
with the API key you copied from the Service Account page.
Return to the StreamNative Cloud Console and go to the “Cluster Dashboard” page.
On the left navigation pane, click Pulsar Clients.
On the Pulsar client setup page, click the Code libraries tab, follow the setup wizard to get the sample codes for your producer and consumer.
a. Select Python as the client library and click Next.
b. Select the service account you created and click Next.
c. Select API Key as the authentication type and click Next.
d. Install pulsar-client
python client library.
e. Select the target tenant, namespace, topic, and subscription.
f. You are now ready to copy the auto-generated sample codes.
Return to your text editor and create two new files: producer.py
and consumer.py
. Copy and paste the sample code for the producer into producer.py
and the sample code for the consumer into consumer.py
. In both files, replace <JWT Token>
with the API key you copied from the Service Account page.
Open a terminal window, navigate to the folder containing the consumer.py
file, and run the following command:
Open a second terminal window, navigate to the folder containing the producer.py
file, and run the following command:
Return to the first terminal window. You should see the following:
You have now produced and consumed your first 10 messages.
After you have successfully provisioned a BYOC cluster and connected to the cluster, you can learn more about working with StreamNative Cloud by reading through Cloud Console basics.
If you want to learn more about Pulsar, Kafka, and StreamNative Cloud, take our developer courses at the StreamNative Developer Portal.
StreamNative Cloud is a resilient, scalable, data streaming service powered by the URSA engine, delivered as a fully managed Pulsar and Kafka service.
StreamNative Cloud provides multiple interfaces for management and interaction:
StreamNative Cloud Console: A user-friendly web-based interface for managing cluster resources, configuring settings, and handling billing.
Command-Line Interface (CLI):
snctl
): The unified command-line interface for deploying and managing StreamNative Cloud infrastructure and interacting directly with your Pulsar clusters and Kafka-protocol endpoints.pulsarctl
): For managing cluster-specific resources, such as tenants, namespaces, topics, functions, connectors, and more.REST APIs: For programmatic access and integration with other systems.
Terraform Providers:
These tools provide flexibility in how you interact with and manage your StreamNative Cloud environment, catering to different user preferences and use cases.
Sign up for StreamNative Cloud and get $200 of free credits. No credit card required.
This quick start guides you through getting started with StreamNative BYOC. It demonstrates how to use a StreamNative BYOC cluster to create topics, produce data to the cluster, and consume data from it.
This QuickStart assumes you are familiar with the basics concepts of StreamNative Cloud Clusters.
Access to StreamNative Cloud.
Internet connectivity.
Access to Your AWS Account for provisioning the BYOC infrastructure.
Ensure you have installed Python 3.0 or higher versions and the Pulsar Python client.
If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.
To sign up, navigate to the StreamNative Cloud Console signup page. Follow the prompts to create an account. After you click Finish, you might have to wait briefly for your first organization to be created. After your new organization is created, continue on to creating your first instance and cluster.
Before deploying a StreamNative cluster within your cloud account, you must first grant StreamNative vendor access. Follow the instructions in Account Access for BYOC on AWS to provision AWS access for StreamNative Cloud. This step ensures StreamNative has the necessary permissions to manage resources in your AWS account.
Once completed, please note the account ID of the AWS account you have granted access to StreamNative Cloud. You will use this account ID to create a Cloud Connection.
In the upper-right corner of the StreamNative Cloud Console, click your Profile and select Cloud Environments.
Select Cloud Connections tab and click New Cloud Connection.
In the Create connection dialog, enter the following information:
my-aws-connection
.Click Submit.
The cloud connection creation process will start. Once completed, you can see the status of the cloud connection turned to Connected
in the Cloud Connections tab.
In the upper-right corner of the StreamNative Cloud Console, click your Profile and select Cloud Environments.
Select Cloud Connections tab and click New Cloud Connection.
In the Create connection dialog, enter the following information:
my-aws-connection
.Click Submit.
The cloud connection creation process will start. Once completed, you can see the status of the cloud connection turned to Connected
in the Cloud Connections tab.
Once you have created a cloud connection, you can create a cloud environment and provision a BYOC instance.
Navigate to Organization Dashboard.
Select Instances at the left navigation pane.
On the Instances page, click + New Instance.
On the Choose the deployment type for your instance page, click Deploy BYOC.
You will see a dialog “Cloud Environment required”. Click Create button to create a cloud environment.
On the Cloud Connection page, select the cloud connection you created in the previous step. In this case, it is my-aws-connection
. Then click Environment setup.
Then fill out the information for the cloud environment.
us-west-2
.poc
.10.0.0.0/16
. If you need to specify a different CIDR, you can enter it here.Click Create.
Navigate to Organization Dashboard.
Select Instances at the left navigation pane.
On the Instances page, click + New Instance.
On the Choose the deployment type for your instance page, click Deploy BYOC.
You will see a dialog “Cloud Environment required”. Click Create button to create a cloud environment.
On the Cloud Connection page, select the cloud connection you created in the previous step. In this case, it is my-aws-connection
. Then click Environment setup.
Then fill out the information for the cloud environment.
us-west-2
.poc
.10.0.0.0/16
. If you need to specify a different CIDR, you can enter it here.Click Create.
The provisioning process of a cloud environment usually takes about 40 minutes to complete. You can safely close the page and come back later. You will also receive an email notification when the cloud environment is ready.
For more information about provisioning BYOC infrastructure, see Provision BYOC Infrastructure.
Once the cloud environment is ready, you can create a Pulsar instance.
Navigate to Organization Dashboard.
Select Instances at the left navigation pane.
On the Instances page, click + New Instance.
On the Choose the deployment type for your instance page, click Deploy BYOC again. You will not see the dialog “Cloud Environment required” this time.
On the Instance Configuration page, fill out the information for the Pulsar instance. Then, click Cluster Location to start the cluster creation process.
my-pulsar-instance
.my-aws-connection
.Classic Engine
.On the Cluster Location page, fill out the information for the cluster.
my-byoc-cluster
.aws-usw2-poc-<random-suffix>
.Click Cluster Operations to choose the release channel for the cluster and configure the cluster.
LTS
.Click Cluster Size to configure the cluster size.
At the right navigation pane, you can see the estimated cost for the cluster.
Click Finish to start the cluster creation process.
The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take several minutes to provision the cluster. Once the cluster has been provisioned, the page will show “Cluster Provisioned successfully” and you can click Go To The Dashboard to access the Cluster Dashboard page.
Navigate to Organization Dashboard.
Select Instances at the left navigation pane.
On the Instances page, click + New Instance.
On the Choose the deployment type for your instance page, click Deploy BYOC again. You will not see the dialog “Cloud Environment required” this time.
On the Instance Configuration page, fill out the information for the Pulsar instance. Then, click Cluster Location to start the cluster creation process.
my-pulsar-instance
.my-aws-connection
.Classic Engine
.On the Cluster Location page, fill out the information for the cluster.
my-byoc-cluster
.aws-usw2-poc-<random-suffix>
.Click Cluster Operations to choose the release channel for the cluster and configure the cluster.
LTS
.Click Cluster Size to configure the cluster size.
At the right navigation pane, you can see the estimated cost for the cluster.
Click Finish to start the cluster creation process.
The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take several minutes to provision the cluster. Once the cluster has been provisioned, the page will show “Cluster Provisioned successfully” and you can click Go To The Dashboard to access the Cluster Dashboard page.
Now you can get started configuring apps and data on your new cluster.
To interact with your cluster by producing and consuming messages, you need to set up authentication and authorization. This is done by creating a Service Account, which serves as an identity for authenticating and authorizing access to the cluster. The service account will provide the necessary credentials for your applications to securely connect and perform operations on the Pulsar cluster.
In the upper-right corner of the StreamNative Cloud Console, click your Profile and select Accounts & Accesses.
On the left navigation pane, click Service Accounts.
On the Service Account page, click + New.
On the Create Service Account dialog, enter a name for the service account, and then click Confirm.
On the Service Account page, in the row of the service account you just created, click the … icon, and select Create API Key in the dropdown menu.
On the New API Key dialog:
An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see Use API Keys to Authenticate to StreamNative Cloud.
In the upper-right corner of the StreamNative Cloud Console, click your Profile and select Accounts & Accesses.
On the left navigation pane, click Service Accounts.
On the Service Account page, click + New.
On the Create Service Account dialog, enter a name for the service account, and then click Confirm.
On the Service Account page, in the row of the service account you just created, click the … icon, and select Create API Key in the dropdown menu.
On the New API Key dialog:
An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see Use API Keys to Authenticate to StreamNative Cloud.
After creating the service account and obtaining the API key, the next crucial step is to authorize the service account. This process grants the necessary permissions for the service account to interact with your StreamNative Cloud cluster.
Authorization involves setting up Access Control Lists (ACLs) that define what actions the service account can perform. Typically, you’ll want to grant permissions for producing and consuming messages on specific topics or namespaces.
For more information about authorization, see Authorization and ACLs.
Go back to the Cluster Dashboard page.
On the left navigation pane, click Instances.
On the Instances page, click the name of the instance you created in Step 2.
On the Cluster Dashboard page, click Tenants on the left navigation pane.
On the Tenants page, click + New Tenant.
On the New Tenant dialog:
On the Tenants page, click the name of the tenant you just created. You will be directed to the Tenant Dashboard page.
On the Tenant Dashboard page, click Namespaces on the left navigation pane.
On the Namespaces page, click New Namespace.
On the New Namespace dialog:
On the Namespaces page, click the name of the namespace you just created. You will be directed to the Namespace Dashboard page.
On the Namespace Dashboard page, click Configuration on the left navigation pane.
On the Namespace configuration page, click ADD ROLE. Select the name of the service account you just created, and choose the consume and produce permissions. This grants your service account the produce
and consume
permissions for this namespace.
Go back to the Cluster Dashboard page.
On the left navigation pane, click Instances.
On the Instances page, click the name of the instance you created in Step 2.
On the Cluster Dashboard page, click Tenants on the left navigation pane.
On the Tenants page, click + New Tenant.
On the New Tenant dialog:
On the Tenants page, click the name of the tenant you just created. You will be directed to the Tenant Dashboard page.
On the Tenant Dashboard page, click Namespaces on the left navigation pane.
On the Namespaces page, click New Namespace.
On the New Namespace dialog:
On the Namespaces page, click the name of the namespace you just created. You will be directed to the Namespace Dashboard page.
On the Namespace Dashboard page, click Configuration on the left navigation pane.
On the Namespace configuration page, click ADD ROLE. Select the name of the service account you just created, and choose the consume and produce permissions. This grants your service account the produce
and consume
permissions for this namespace.
Now you have created a tenant, namespace, and granted the service account the produce
and consume
permissions for this namespace. You can now continue on to building a Python app, connecting to the cluster, and producing and consuming messages.
This QuickStart provides you with an example Python app to get you up and running with consuming and producing messages. This is a simple example and is not intended for production environments.
Return to the StreamNative Cloud Console and go to the “Cluster Dashboard” page.
On the left navigation pane, click Pulsar Clients.
On the Pulsar client setup page, click the Code libraries tab, follow the setup wizard to get the sample codes for your producer and consumer.
a. Select Python as the client library and click Next.
b. Select the service account you created and click Next.
c. Select API Key as the authentication type and click Next.
d. Install pulsar-client
python client library.
e. Select the target tenant, namespace, topic, and subscription.
f. You are now ready to copy the auto-generated sample codes.
Return to your text editor and create two new files: producer.py
and consumer.py
. Copy and paste the sample code for the producer into producer.py
and the sample code for the consumer into consumer.py
. In both files, replace <JWT Token>
with the API key you copied from the Service Account page.
Return to the StreamNative Cloud Console and go to the “Cluster Dashboard” page.
On the left navigation pane, click Pulsar Clients.
On the Pulsar client setup page, click the Code libraries tab, follow the setup wizard to get the sample codes for your producer and consumer.
a. Select Python as the client library and click Next.
b. Select the service account you created and click Next.
c. Select API Key as the authentication type and click Next.
d. Install pulsar-client
python client library.
e. Select the target tenant, namespace, topic, and subscription.
f. You are now ready to copy the auto-generated sample codes.
Return to your text editor and create two new files: producer.py
and consumer.py
. Copy and paste the sample code for the producer into producer.py
and the sample code for the consumer into consumer.py
. In both files, replace <JWT Token>
with the API key you copied from the Service Account page.
Open a terminal window, navigate to the folder containing the consumer.py
file, and run the following command:
Open a second terminal window, navigate to the folder containing the producer.py
file, and run the following command:
Return to the first terminal window. You should see the following:
You have now produced and consumed your first 10 messages.
After you have successfully provisioned a BYOC cluster and connected to the cluster, you can learn more about working with StreamNative Cloud by reading through Cloud Console basics.
If you want to learn more about Pulsar, Kafka, and StreamNative Cloud, take our developer courses at the StreamNative Developer Portal.