streamnative.io
Help Center
Login
  1. StreamNative Cloud
  2. Get Started

Quick Start for StreamNative BYOC

StreamNative Cloud is a resilient, scalable, data streaming service powered by the URSA engine, delivered as a fully managed Pulsar and Kafka service.

StreamNative Cloud provides multiple interfaces for management and interaction:

  1. StreamNative Cloud Console: A user-friendly web-based interface for managing cluster resources, configuring settings, and handling billing.

  2. Command-Line Interface (CLI):

    • StreamNative CLI (snctl): The unified command-line interface for deploying and managing StreamNative Cloud infrastructure and interacting directly with your Pulsar clusters and Kafka-protocol endpoints.
    • Pulsar CLI (pulsarctl): For managing cluster-specific resources, such as tenants, namespaces, topics, functions, connectors, and more.

  3. REST APIs: For programmatic access and integration with other systems.

  4. Terraform Providers:

These tools provide flexibility in how you interact with and manage your StreamNative Cloud environment, catering to different user preferences and use cases.

Get started for Free

Sign up for StreamNative Cloud and get $200 of free credits. No credit card required.

This quick start guides you through getting started with StreamNative BYOC. It demonstrates how to use a StreamNative BYOC cluster to create topics, produce data to the cluster, and consume data from it.

Note

This QuickStart assumes you are familiar with the basics concepts of StreamNative Cloud Clusters.

Prerequisites

  • Access to StreamNative Cloud.

  • Internet connectivity.

  • Access to Your AWS Account for provisioning the BYOC infrastructure.

  • Ensure you have installed Python 3.0 or higher versions and the Pulsar Python client.

    python -m pip install pulsar-client

Step 1: Sign up

Note

If you have an email account configured for using Single Sign-On (SSO) with StreamNative Cloud, use that email address and password when signing up.

To sign up, navigate to the StreamNative Cloud Console signup page. Follow the prompts to create an account. After you click Finish, you might have to wait briefly for your first organization to be created. After your new organization is created, continue on to creating your first instance and cluster.

Step 2: Grant StreamNative Vendor Access

Before deploying a StreamNative cluster within your cloud account, you must first grant StreamNative vendor access. Follow the instructions in Account Access for BYOC on AWS to provision AWS access for StreamNative Cloud. This step ensures StreamNative has the necessary permissions to manage resources in your AWS account.

Once completed, please note the account ID of the AWS account you have granted access to StreamNative Cloud. You will use this account ID to create a Cloud Connection.

Step 3: Create a Cloud Connection

  1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select Cloud Environments.

  2. Select Cloud Connections tab and click New Cloud Connection.

  3. In the Create connection dialog, enter the following information:

    • Name: Enter a name for the cloud connection. For example, my-aws-connection.
    • Connection provider: Select AWS as the connection provider.
    • AWS Account ID: Enter the account ID of the AWS account you noted in the previous step.
    • Check Confirm if vendor access Terraform module is executed.

  4. Click Submit.

  5. The cloud connection creation process will start. Once completed, you can see the status of the cloud connection turned to Connected in the Cloud Connections tab.

Step 4: Create a Cloud Environment

Once you have created a cloud connection, you can create a cloud environment and provision a BYOC instance.

  1. Navigate to Organization Dashboard.

  2. Select Instances at the left navigation pane.

  3. On the Instances page, click + New Instance.

  4. On the Choose the deployment type for your instance page, click Deploy BYOC.

  5. You will see a dialog "Cloud Environment required". Click Create button to create a cloud environment.

  6. On the Cloud Connection page, select the cloud connection you created in the previous step. In this case, it is my-aws-connection. Then click Environment setup.

  7. Then fill out the information for the cloud environment.

    • Region: Select the region where you want to deploy the BYOC cluster. In this example, it is us-west-2.
    • Environment tag: Enter a tag for the cloud environment. For example, poc.
    • Configure StreamNative Managed VPC Network:
      • Network CIDR: By default, it creates a StreamNative Managed VPC with a CIDR of 10.0.0.0/16. If you need to specify a different CIDR, you can enter it here.
    • Default Gateway: You can configure how do you want to expose your BYOC cluster, whether it is public or private. By default, it is public.

  8. Click Create.

The provisioning process of a cloud environment usually takes about 40 minutes to complete. You can safely close the page and come back later. You will also receive an email notification when the cloud environment is ready.

For more information about provisioning BYOC infrastructure, see Provision BYOC Infrastructure.

Step 5: Create a StreamNative Instance & Cluster

Once the cloud environment is ready, you can create a Pulsar instance.

  1. Navigate to Organization Dashboard.

  2. Select Instances at the left navigation pane.

  3. On the Instances page, click + New Instance.

  4. On the Choose the deployment type for your instance page, click Deploy BYOC again. You will not see the dialog "Cloud Environment required" this time.

  5. On the Instance Configuration page, fill out the information for the Pulsar instance. Then, click Cluster Location to start the cluster creation process.

    • Instance Name: Enter a name for the Pulsar instance. For example, my-pulsar-instance.
    • Cloud Connection: Select the cloud connection you created in the previous step. In this example, it is my-aws-connection.
    • Engine: Select the engine for the Pulsar instance. In this example, it is Classic Engine.

  6. On the Cluster Location page, fill out the information for the cluster.

    • Cluster Name: Enter a name for the cluster. For example, my-byoc-cluster.
    • Cloud Environment: Select the cloud environment you created in the previous step. In this example, it is aws-usw2-poc-<random-suffix>.

  7. Click Cluster Operations to choose the release channel for the cluster and configure the cluster.

    • Release Channel: Select the release channel for the cluster. In this example, it is LTS.
    • Features: Select the features for the cluster. You can keep the default features or customize it.
    • Maintenance Window: If you have a Enterprise or Production support plan, you can customize the maintenance window for the cluster. Otherwise, you can skip this step.
    • Custom Configuration: If you need to customize the cluster configuration, you can expand the Add optional custom configuration section.

  8. Click Cluster Size to configure the cluster size.

    • You can use the slider to adjust the throughput accordingly.
    • You can also manually configure the number of brokers, bookies, and their corresponding resources in the Advanced section.

    At the right navigation pane, you can see the estimated cost for the cluster.

  9. Click Finish to start the cluster creation process.

The cluster page appears, displaying the cluster creation process. Depending on the chosen cloud provider and other settings, it may take several minutes to provision the cluster. Once the cluster has been provisioned, the page will show "Cluster Provisioned successfully" and you can click Go To The Dashboard to access the Cluster Dashboard page.

Now you can get started configuring apps and data on your new cluster.

Step 6: Create a service account

To interact with your cluster by producing and consuming messages, you need to set up authentication and authorization. This is done by creating a Service Account, which serves as an identity for authenticating and authorizing access to the cluster. The service account will provide the necessary credentials for your applications to securely connect and perform operations on the Pulsar cluster.

  1. In the upper-right corner of the StreamNative Cloud Console, click your Profile and select Accounts & Accesses.

  2. On the left navigation pane, click Service Accounts.

  3. On the Service Account page, click + New.

  4. On the Create Service Account dialog, enter a name for the service account, and then click Confirm.

  5. On the Service Account page, in the row of the service account you just created, click the ... icon, and select Create API Key in the dropdown menu.

  6. On the New API Key dialog:

    • Enter a name for the API key
    • Set the expiration date
    • Select the instance you created in previous step
    • Write a description for the API key
    • Click Confirm

    Note

    An API key and associated secret apply to the active StreamNative instance. If you add a new instance, you must create a new API key for producers and consumers on the new Pulsar instance. For more information, see Use API Keys to Authenticate to StreamNative Cloud.

  7. After the API key is created, you can see the API key shown in the New API Key dialog. Click the Copy and close button to copy the API key to your clipboard. Please note that you cannot retrieve the API key later after closing the dialog.

Step 7: Create Tenant and Namespace, and Authorize the Service Account

After creating the service account and obtaining the API key, the next crucial step is to authorize the service account. This process grants the necessary permissions for the service account to interact with your StreamNative Cloud cluster.

Authorization involves setting up Access Control Lists (ACLs) that define what actions the service account can perform. Typically, you'll want to grant permissions for producing and consuming messages on specific topics or namespaces.

For more information about authorization, see Authorization and ACLs.

  1. Go back to the Cluster Dashboard page.

  2. On the left navigation pane, click Instances.

  3. On the Instances page, click the name of the instance you created in Step 2.

  4. On the Cluster Dashboard page, click Tenants on the left navigation pane.

  5. On the Tenants page, click + New Tenant.

  6. On the New Tenant dialog:

    • Enter a name for the tenant
    • Select your user account as the Admin role
    • Select the cluster created in Step 2 as Allowed clusters
    • Click Confirm

  7. On the Tenants page, click the name of the tenant you just created. You will be directed to the Tenant Dashboard page.

  8. On the Tenant Dashboard page, click Namespaces on the left navigation pane.

  9. On the Namespaces page, click New Namespace.

  10. On the New Namespace dialog:

    • Enter a name for the namespace
    • Select the cluster created in Step 2 as Allowed clusters and Replication clusters
    • Click Confirm

  11. On the Namespaces page, click the name of the namespace you just created. You will be directed to the Namespace Dashboard page.

  12. On the Namespace Dashboard page, click Configuration on the left navigation pane.

  13. On the Namespace configuration page, click ADD ROLE. Select the name of the service account you just created, and choose the consume and produce permissions. This grants your service account the produce and consume permissions for this namespace.

Now you have created a tenant, namespace, and granted the service account the produce and consume permissions for this namespace. You can now continue on to building a Python app, connecting to the cluster, and producing and consuming messages.

Step 8: Produce and consume messages

Note

This QuickStart provides you with an example Python app to get you up and running with consuming and producing messages. This is a simple example and is not intended for production environments.

Create a producer/consumer

  1. Return to the StreamNative Cloud Console and go to the "Cluster Dashboard" page.

  2. On the left navigation pane, click Pulsar Clients.

  3. On the Pulsar client setup page, click the Code libraries tab, follow the setup wizard to get the sample codes for your producer and consumer.

    a. Select Python as the client library and click Next.

    b. Select the service account you created and click Next.

    c. Select API Key as the authentication type and click Next.

    d. Install pulsar-client python client library.

    pip3 install pulsar-client

    e. Select the target tenant, namespace, topic, and subscription.

    f. You are now ready to copy the auto-generated sample codes.

  4. Return to your text editor and create two new files: producer.py and consumer.py. Copy and paste the sample code for the producer into producer.py and the sample code for the consumer into consumer.py. In both files, replace <JWT Token> with the API key you copied from the Service Account page.

Run the clients to produce and consume your first message

  1. Open a terminal window, navigate to the folder containing the consumer.py file, and run the following command:

    python3 consumer.py

  2. Open a second terminal window, navigate to the folder containing the producer.py file, and run the following command:

    python3 producer.py

  3. Return to the first terminal window. You should see the following:

    Received message 'Hello-0' id='<message-id>'
Received message 'Hello-1' id='<message-id>'
Received message 'Hello-2' id='<message-id>'
Received message 'Hello-3' id='<message-id>'
Received message 'Hello-4' id='<message-id>'
Received message 'Hello-5' id='<message-id>'
Received message 'Hello-6' id='<message-id>'
Received message 'Hello-7' id='<message-id>'
Received message 'Hello-8' id='<message-id>'
Received message 'Hello-9' id='<message-id>'

You have now produced and consumed your first 10 messages.

Next steps

  • After you have successfully provisioned a BYOC cluster and connected to the cluster, you can learn more about working with StreamNative Cloud by reading through Cloud Console basics.

  • If you want to learn more about Pulsar, Kafka, and StreamNative Cloud, take our developer courses at the StreamNative Developer Portal.

Previous
Cloud Console
Next
Ursa Engine Quick Start