streamnative.io
Help Center
Login
  1. StreamNative Cloud
  2. Manage StreamNative Clusters

StreamNative Cloud Clusters Overview

Use this topic to understand the cluster concepts in StreamNative Cloud.

Management Concepts

StreamNative clusters in StreamNative Cloud differ from general Pulsar or Kafka clusters in that StreamNative offers a fully managed service. Use the following concepts to learn more.

Basics

You can choose from different types of clusters for use in development, production, and high-traffic workloads. StreamNative Cloud is available from all major cloud providers in regions around the world, either fully hosted by StreamNative or as BYOC (Bring-Your-Own-Cloud). This provides access to StreamNative cluster and configuration settings for resources such as tenants, namespaces, and topics. For more information, see Cluster Types and Regions.

Fault Tolerance

StreamNative Cloud provides a centralized global control plane that manages a distributed collection of servers running in multiple regions, either fully hosted by StreamNative or as BYOC (Bring-Your-Own-Cloud). Clusters can span cloud provider availability zones (AZs) and provide highly scalable and fault-tolerant systems that support mission-critical applications.

Hosted vs BYOC

StreamNative Cloud offers two primary deployment models: Hosted and Bring-Your-Own-Cloud (BYOC). Hosted clusters, which include Serverless and Dedicated options, are deployed and managed entirely on StreamNative's cloud infrastructure. These are ideal for teams seeking a turnkey solution with minimal operational overhead. On the other hand, BYOC clusters allow customers to leverage StreamNative's management expertise while maintaining the clusters within their own cloud environment. This option is particularly suited for organizations with existing cloud infrastructure, specific compliance requirements, or those desiring greater control over their cloud resources. Both models benefit from StreamNative's full management and expertise, ensuring optimal performance and reliability. The choice between Hosted and BYOC depends on factors such as operational preferences, compliance needs, and existing cloud investments. For a detailed comparison of cluster types and available regions, refer to Cluster Types and Regions.

Instances & Clusters

StreamNative Cloud organizes its clusters into Instances. Within any StreamNative Cloud organization, it is possible to host one or more Pulsar Instances. Each Pulsar Instance serves as a dedicated environment within a cloud provider, capable of hosting multiple Pulsar clusters along with various deployed components such as Connectors and Functions. This setup allows different departments or teams to operate in separate instances to prevent overlap and interference.

A Pulsar Instance can be either fully Hosted on StreamNative's cloud account or on your own public cloud account through the Bring-Your-Own-Cloud (BYOC) deployment option.

Within each Pulsar Instance, multiple Pulsar clusters can be deployed across various regions. Each cluster within a Pulsar Instance is deployed to a specific cloud region as configured in its cloud provider settings. Furthermore, each Pulsar cluster provides various service endpoints to enable client libraries to connect, produce and consume messages, execute functions, run connectors, and perform SQL queries.

Clusters within a Pulsar instance can also replicate data among themselves using geo-replication.

StreamNative Cloud utilizes Public Cloud infrastructure to host Pulsar instances and clusters. Below is a diagram illustrating the relationship between Pulsar instances, clusters, and the cloud infrastructure:

image of Pulsar Instances

Infrastructure Pools

As illustrated above, StreamNative Cloud infrastructure is organized into infrastructure pools. Each pool encompasses a collection of infrastructure environments, known as pool members, distributed across multiple regions within a cloud provider. These members can be equated to Kubernetes clusters dedicated to deploying Pulsar clusters.

A Pulsar Instance can be created within and deployed to a specific infrastructure pool. Each infrastructure pool can support multiple Pulsar Instances. When setting up a Pulsar Instance, you select the appropriate pool using its namespace and name. Within each Pulsar Instance, you can deploy clusters to various regions by specifying the desired region. StreamNative Cloud then deploys the Pulsar Cluster to the corresponding pool member in that region.

The type of Pulsar Instance, whether Hosted or BYOC, is determined by the location and configuration of its Infrastructure Pool. Infrastructure Pools can either be fully Hosted by StreamNative or set up in your own public cloud account through the BYOC deployment option.

Hosted Infrastructure Pools

Hosted Infrastructure Pools are fully managed and maintained on StreamNative's cloud account. These pools are pre-provisioned, allowing you to utilize them when setting up your Pulsar Instances and Clusters.

If you provision a Pulsar instance or cluster via the StreamNative Cloud console, you can simply select the cloud provider and the respective location. The console will automatically select the appropriate Hosted Pool for you.

If you prefer to provision a Pulsar instance or cluster using the snctl command-line tool, you can specify the pool using the --pool flag. For example:

snctl create pulsarinstance --name <instance-name> --pool <pool-name> --namespace <organization-id>

Finding Available Hosted Pools

You can use the snctl get pooloptions command to list all available Hosted Infrastructure Pools.

$ snctl get pooloptions
NAME                        CREATED AT
streamnative-shared-aws     2022-03-04T00:44:51Z
streamnative-shared-azure   2024-03-25T20:32:15Z
streamnative-shared-gcp     2022-04-26T16:48:31Z

Each PoolOption contains the following key information of a Hosted pool:

  1. CloudType: Indicates the cloud provider (e.g., AWS, GCP, Azure) for the pool.
  2. DeploymentType: The type of deployment (e.g., Hosted, BYOC).
  3. PoolRef: A reference to the specific infrastructure pool.
  4. Locations: A list of available locations (regions) for the pool.
  5. Features: A map of enabled features for the pool.

To view the details of a specific PoolOption, you can use the snctl get pooloption <pool-name> command. It will display detailed information about the pool, including its configuration, features, and other relevant settings.

An example of the output of the snctl get pooloption <pool-name> command is as follows:

$ snctl get pooloption streamnative-shared-aws -o yaml
apiVersion: cloud.streamnative.io/v1alpha1
kind: PoolOption
metadata:
  creationTimestamp: "2022-03-04T00:44:51Z"
  generation: 28
  name: streamnative-shared-aws
  namespace: sndev
  resourceVersion: "34349179"
  uid: bf56b5c8-e99e-4376-96ab-aecf728343f7
spec:
  cloudType: aws
  deploymentType: ""
  features:
    AOP: true
    ApiKey: true
    AutoScaling: true
    Function: true
    Istio: true
    KOP: true
    MOP: true
    SnRBAC: true
    Transaction: true
    WebSocket: true
  locations:
  - location: ap-southeast-2
  - location: eu-central-1
  - location: eu-west-1
  - location: us-east-2
  poolRef:
    name: shared-aws
    namespace: streamnative
status: {}

When selecting a pool for your Pulsar Instance or Cluster, consider the following:

  • Choose a pool with a CloudType that matches your preferred cloud provider.
  • Ensure the pool has Locations (regions) that meet your geographical requirements.
  • Check that the Features align with your needs.
  • Verify that the DeploymentType is "Hosted" for fully managed StreamNative infrastructure.

By carefully reviewing the PoolOptions, you can select the most suitable Hosted Infrastructure Pool for your StreamNative Cloud deployment.

BYOC Infrastructure Pools

Unlike Hosted Infrastructure Pools, BYOC Infrastructure Pools must be provisioned prior to deploying any BYOC Pulsar Instances or Clusters. The setup process for a BYOC Pool includes establishing a Cloud Connection—this allows the StreamNative Cloud control plane to interact with your cloud account. It also involves defining and provisioning Pool Members within the Pool to deploy Pulsar Clusters across different regions of your cloud account. Provisioning Pool Members within a Pool is done by creating Cloud Environments in different regions. Each Cloud Environment defines the essential infrastructure resources such as compute, storage, and networking that are necessary for a Pool Member to deploy Pulsar clusters.

For more information about how to provision and manage BYOC infrastructure pools, see Manage BYOC Infrastructure.

Related Content

Previous
Ursa Engine Quick Start
Next
Cluster Types and Regions