Skip to main content
This document describes how to configure asynchronous geo-replication on StreamNative Platform.

Prerequisites

These instructions assume that you have installed StreamNative Platform in two Kubernetes namespaces k8s-01 and k8s-02. If you are new to StreamNative Platform, work through the StreamNative Platform deployment guide first, and then return to this tutorial.

Enable asynchronous geo-replication

This section guides you through the steps to enable asynchronous geo-replication. You can enable asynchronous geo-replication at either the namespace or topic level.
  • Namespace-level asynchronous geo-replication: when namespace-level asynchronous geo-replication is enabled, the data that is written to the topics in the configured namespace can be replicated to other Pulsar clusters.
  • Topic-level asynchronous geo-replication: when topic-level asynchronous geo-replication is enabled, only the data that is written to the configured topic can be replicated to other Pulsar clusters.
  • The instructions in this section assume that you work on two Pulsar clusters (pulsar-a and pulsar-b).
  • If you only want to implement the unidirectional asynchronous geo-replication between these two Pulsar clusters, such as from pulsar-a to pulsar-b, you can enable asynchronous geo-replication on Pulsar cluster pulsar-a, and vice versa.
  • If you want to implement the bidirectional asynchronous geo-replication between these two Pulsar clusters (pulsar-a and pulsar-b), you need to enable asynchronous geo-replication on both Pulsar clusters.
  • It is recommended to disable Vault service to save resources.

Enable namespace-level asynchronous geo-replication

This section describes how to enable namespace-level asynchronous geo-replication.

Step 1: Create cluster connection

Run the commands below to create the connection between these two Pulsar clusters.
Run the command below on Pulsar cluster pulsar-a to create the connection from Pulsar cluster pulsar-a to Pulsar cluster pulsar-b. Then, when a client writes data to the Pulsar cluster pulsar-a, the data is replicated from Pulsar cluster pulsar-a to Pulsar cluster pulsar-b.

Step 2: Create tenants and grant permissions

Create a tenant (rep-tenant-1) in both Pulsar clusters, grant the Admin role to the tenant, and allow the tenant to access both Pulsar clusters.

Step 3: Create namespaces

Create a namespace (rep-tenant-1/rep-ns) in both Pulsar clusters, grant the Admin role as well as the produce and consume permissions to the namespace, and set the replication clusters for the namespace.

Step 4: Create topics

Create a partitioned topic (rep-tenant-1/rep-ns/rep-topic) in both Pulsar clusters.

Enable topic-level asynchronous geo-replication

This section describes how to enable namespace-level asynchronous geo-replication.

Step 1: Create cluster connection

Run the commands below to create the connection between these two Pulsar clusters.
Run the command below on Pulsar cluster pulsar-a to create the connection from Pulsar cluster pulsar-a to Pulsar cluster pulsar-b. Then, when a client writes data to the Pulsar cluster pulsar-a, the data is replicated from Pulsar cluster pulsar-a to Pulsar cluster pulsar-b.

Step 2: Create tenants and grant permissions

Create a tenant (rep-tenant-1) in both Pulsar clusters, grant the Admin role to the tenant, and allow the tenant to access both Pulsar clusters.

Step 3: Create namespaces

Create a namespace (rep-tenant-1/rep-ns) in both Pulsar clusters.

Step 4: Create topics

Create a partitioned topic (rep-tenant-1/rep-ns/rep-topic) in both Pulsar clusters and set the replication clusters for the topic.

Verify asynchronous geo-replication

This example shows that when you produce messages to Pulsar cluster pulsar-b, the consumer can read it from Pulsar cluster pulsar-a.
  1. Consume the message from Pulsar cluster pulsar-a.
  2. Produce a message (hello-world) to Pulsar cluster pulsar-b.
On the consumer terminal, the output should be similar to the following:
This example shows that when you produce messages to Pulsar cluster pulsar-a, the consumer can read it from Pulsar cluster pulsar-b.
  1. Consume the message from Pulsar cluster pulsar-b.
  2. Produce a message (hello-world) to Pulsar cluster pulsar-a.
On the consumer terminal, the output should be similar to the following:

Examples

This section provides some examples on how to enable asynchronous geo-replication when different authentication methods are enabled on StreamNative Platform.
These instructions assume that you have installed StreamNative Platform in two Kubernetes namespaces (k8s-01 and k8s-02) and that you work on two Pulsar clusters (pulsar-a and pulsar-b).

Enable JWT authentication and asynchronous geo-replication

If you want to enable JWT authentication and geo-replication for your Pulsar clusters, follow the instructions in this section.

Create JWT secrets for your Pulsar clusters

Before enabling JWT authentication on your Pulsar clusters, you need to create JWT secrets for your Pulsar cluster. You can run the prepare_helm_release.sh script to automatically generate JWT secrets for one Pulsar cluster. Then, copy these secrets from one Kubernetes namespace to another and create new secrets for another Pulsar cluster based on the copied secrets.
  1. Clone this repository and switch to the target directory.
  2. Run the prepare_helm_release.sh script to create JWT secrets.
    • -n,--namespace: the Kubernetes namespace to install the helm chart
    • -k,--release: the helm release name
    By default, the command generates the asymmetric public/private key pair. You can choose to generate a symmetric secret key by specifying --symmetric in the command.
  3. Check the created secrets.
    • pulsar-a-token-admin: the role that is used to access the admin tools.
    • pulsar-a-token-asymmetric-key: a public/private key pair that is used to generate and validate tokens in an asymmetric algorithm. You can use the private key to generate tokens. You can use the public key to validate tokens.
    • pulsar-a-token-broker-admin: the role that is used for inter-broker communications.
    • pulsar-a-token-proxy-admin: the role that is used for Pulsar proxies to communicate to Pulsar brokers.
    • pulsar-a-token-pulsar-manager-admin: the superuser role that is used to access the StreamNative Console.
Now, these two Pulsar clusters share the same private key.

Configure JWT authentication for your Pulsar clusters

For details, see configure JWT authentication.

Enable asynchronous geo-replication

For details, see enable asynchronous geo-replication.

Enable OAuth2 authentication and asynchronous geo-replication

If you want to enable OAuth2 authentication and asynchronous geo-replication for your Pulsar clusters, follow these steps:
  1. Configure OAuth2 authentication for your Pulsar clusters. For details, see Configure OAuth2 authentication.
  2. Enable asynchronous geo-replication. For details, see enable asynchronous geo-replication.