Create, manage, and delete clusters
Cluster Overview
A cluster is a secure messaging environment within Pulsar. Each Pulsar cluster consists a set of 3 components in a geographical location.
- Pulsar brokers - set of brokers handling all the data going in and out of Pulsar (or client requests)
- Metadata storage - providing coordination and service discovery between services
- Bookie ensemble - set of bookies that retain copies of the messages
A cluster has two layers: a stateless serving layer (made up of brokers) and a stateful storage layer (made up of bookies). See also Pulsar Architecture and Design.
Note
In StreamNative Console, you can create one and only one cluster for an instance.
A Pulsar instance consists of one or more Pulsar clusters working in unison. Clusters can be distributed across geographical locations and can replicate amongst themselves using geo-replication.
Cluster Features and Usage limits
The following table lists features and usage limits supported by the cluster.
| Type | Features | Capability |
|---|---|---|
| Service | Uptime SLA | 99.95% |
| Multi AZ | Yes | |
| Scale | Throughput limit per topic | Max 100 MBps |
| Storage limit per topic | Max 1000 TB | |
| Tenant limit | Max 128 | |
| Namespace limit | Max 1024 | |
| Topic limit | Max 10240 | |
| Pulsar components | Pub/Sub | Yes |
| Cloud providers | Google Cloud | Yes |
| AWS | Yes | |
| Monitoring & visibility | Cloud Console | Yes |
| Security | At-rest & In-transit data encryption | Yes |
| OAuth2 authentication | Yes |
For details about how to work with clusters, such as creating, editing, checking, and deleting clusters, see work with clusters.
Cluster location
Each Pulsar cluster is associated with a geographical location. The following locations are available:
| Identifier | Location |
|---|---|
| ap-south-1 | Asia Pacific (Mumbai) |
| ap-southeast-2 | Asia Pacific (Sydney) |
| asia-south1 | Mumbai, India, APAC |
| eu-central-1 | Europe (Frankfurt) |
| eu-west-1 | Europe (Ireland) |
| europe-west1 | St. Ghislain, Belgium, Europe |
| europe-west3 | Frankfurt, Germany, Europe |
| us-central1 | Council Bluffs, Iowa, North America |
| us-east-1 | US East (N. Virginia) |
| us-east-2 | US East (Ohio) |
| us-east4 | Ashburn, Virginia, North America |
| us-west1 | The Dalles, Oregon, North America |
Available Cluster features
You can enable the following features for a Pulsar cluster.
WebSocket
The Pulsar WebSocket API provides a simple way to interact with Pulsar using languages that do not have an official client library. Through the WebSocket API, you can use the WebSocket producer and consumer to produce and consume messages to and from a topic. For details, see connect to cluster through WebSocket API.
Kafka protocol
You can add the Kafka protocol to your Pulsar cluster. With Kafka protocol enabled, you can migrate your existing Kafka applications and services to Pulsar without modifying the code. Currently, this is a beta feature. If you want to try out this feature, submit a ticket with this request to the support team.
Audit log
Audit logs track and store authorization activities in Pulsar clusters, tenants, namespaces, and topics. For details about how to enable and work with audit logs, see audit log.
Cluster configurations
When you provision a Pulsar cluster, StreamNative Cloud automatically configures the security, networking, compute, and storage aspects of the cluster. Various scaling parameters are customizable based on your workload and your budget. These parameters stem from the Pulsar architecture and are specified on a per-component basis.
Broker parameters
| Parameter | Default value | Description |
|---|---|---|
| Replicas | 1 | Specify the number of brokers. |
| Node type | tiny-1 | Specify the compute characteristics (CPU, memory) per broker. |
BookKeeper parameters
| Parameter | Default value | Description |
|---|---|---|
| Replicas | 3 | Specify the number of bookies. |
| Node type | tiny-1 | Specify the compute characteristics (CPU, memory) per bookie. |
Backlog parameters
| Parameter | Default value | Description |
|---|---|---|
| Backlog Quota Size | -1 | Specify the backlog quota size. |
| Backlog Retention Policy | N/A | Specify the retention policy when the backlog quota threshold is reached. - producer_request_hold: the broker holds producers' sent requests until the resource becomes available or until the holding times out. - producer_exception: the broker rejects producers' send requests. - consumer_backlog_eviction: the broker evicts the oldest messages from the slowest consumer's backlog. |
Throttling parameters
| Parameter | Default value | Description |
|---|---|---|
| Max Producers Per Topic | N/A | Specify the maximum number of producers allowed to connect to a topic. |
| Max Consumers Per Topic | N/A | Specify the maximum number of consumers allowed to connect to a topic. |
| Max Consumers Per Subscription | N/A | Specify the maximum number of consumers allowed to connect to a subscription. |
Dispatch rate parameters
| Parameter | Default value | Description |
|---|---|---|
| Dispatch Rate Per Topic | N/A | Specify the topic-based dispatch rate. - Throughput (bytes/second): specify the topic-based dispatch rate (in bytes/second). - Rate (messages/second): specify the topic-based dispatch rate (in messages/second). |
| Dispatch Rate Per Subscription | N/A | Specify the subscription-based dispatch rate. - Throughput (bytes/second): specify the subscription-based dispatch rate (in bytes/second). - Rate (messages/second): specify the subscription-based dispatch rate (in messages/second). |
| Subscribe Rate Per Consumer | N/A | Specify the subscribe rate for each consumer. Rate (subscribes/second): specify the subscribe rate for each consumer (in subscribes/second). |
Cluster Service URLs
Once provisioned, a Pulsar cluster is accessible through service URLs. A service URL is an HTTPS endpoint that is exposed to the Internet and protected by OAuth 2.0. The Fully Qualified Domain Name (FQDN) of a service endpoint is based on the name of the Pulsar cluster.
StreamNative Cloud Console provides four kinds of service URLs for a Pulsar cluster:
- HTTP Service URL: the HTTP URL of a Pulsar cluster. After being granted the Admin access right, you can use the
pulsar-adminorpulsarctlCLI tool to connect to and manage the Pulsar cluster. - Broker Service URL: the broker URL of a Pulsar cluster. After being granted the
produceandconsumepermissions, you can use thepulsar-clientCLI tool or Pulsar clients to connect to and communicate with the Pulsar cluster. - WebSocket Service URL: the URL for a WebSocket-enabled Pulsar cluster. After being granted the Admin access right, you can use the WebSocket API to connect to and communicate with the Pulsar cluster.
- Kafka Service URL: the URL for a Pulsar cluster with the Kafka protocol enabled. After being granted the Admin access right, you can use the Kafka CLI tool or Kafka clients to connect to and communicate with the Pulsar cluster.
Create a cluster
To create a cluster, follow these steps.
On the left navigation pane, click Dashboard.
On the Instances card, perform one of the following operations:
- If you have deplopyed one or more instances, click the Add icon create a new instances.
- If you have not deployed any instance, click New to create a new instance.
Click Deploy Hosted to start the instance creation process.
On the Instance Configuration page, enter a name for your instance, select an infrastructure pool, and select the multi Availability Zone (AZ).
Click Cluster Location to start the cluster creation process.
Enter a name for your cluster, select the cluster location, and then click Cluster Size.
Use the slider to select your cluster size. There are 5 pre-selected sizes you can choose from. The Approximate Cost section updates as you move the slider.
(Optional) Click the Advanced tab to select custom sizing options.
(Optional) In the Features section, enable the features you want on your cluster. You can enable or disable these features at any time by editing your cluster after it has been created.
(Optional) Expand the Add optional custom configurations section to specify optional custom configuration parameters.
Item Description Backlog Quota Size Enter the backlog quota size. Backlog Retention Policy Enter the retention policy when the backlog quota threshold is reached.
-producer_request_hold: the broker holds producers' sent requests until the resource becomes available or until the holding times out.
-producer_exception: the broker rejects producers' send requests.
-consumer_backlog_eviction: the broker evicts the oldest messages from the slowest consumer's backlog.Max Producers Per Topic Enter the maximum number of producers allowed to connect to a topic. Max Consumers Per Topic Enter the maximum number of consumers allowed to connect to a topic. Max Consumers Per Subscription Enter the maximum number of consumers allowed to connect to a subscription. Dispatch Rate Per Topic Enter the topic-based dispatch rate.
- Throughput (bytes/second): specify the topic-based dispatch rate (in bytes/second).
- Rate (messages/second): specify the topic-based dispatch rate (in messages/second).Dispatch Rate Per Subscription Enter the subscription-based dispatch rate.
- Throughput (bytes/second): specify the subscription-based dispatch rate (in bytes/second).
- Rate (messages/second): specify the subscription-based dispatch rate (in messages/second).Subscribe Rate Per Consumer Enter the subscribe rate for each consumer.
Rate (subscribes/second): specify the subscribe rate for each consumer (in subscribes/second).Click Finish. The cluster page displays, showing the cluster creation process. The cluster is ready for use after all components have been successfully deployed.
View cluster details
You can view the cluster for a specific instance, including the cluster name and service URLs. Currently, only one cluster is available for each instance.
On the left navigation pane, in the Admin area, click Pulsar Clusters.
- On the Dashboard tab, you can view some metrics about the cluster, including the number of topics, subscriptions, consumers, producers, throughput, storage size, and backlog size.
- On the Details tab, you can view the details about the cluster, including the cluster name, location, availability mode, cloud provider, cluster type, service URLs, the features enabled on the cluster, and your Pulsar, BookKeeper, and ZooKeeper versions.
Update a cluster
On the left navigation pane, in the Admin area, click Pulsar Clusters.
On the Dashboard tab, click Edit Cluster.
Update the cluster.
- Select the Basic tab, and then select a different cluster size.
- Select the Advanced tab to change advanced cluster settings.
Click Finish when you are done updating your cluster.
Delete a cluster
Note
You cannot delete a cluster if there are resources associated with the cluster.
To delete a cluster, follow these steps.
On the left navigation pane, in the Admin area, click Pulsar Clusters.
Select the Details tab.
In the Warning section, click Delete Cluster. A dialog box displays asking, Are you sure you want to delete this?
Enter the cluster name and then click Confirm.