StreamNative Terraform Tutorial

This tutorial demonstrates how to use StreamNative Terraform Provider to deploy a Serverless cluster and use the Pulsar Terraform Provider to provision Pulsar resources in the Serverless cluster.

This tutorial provisions the following resources (assuming we name the application as sl-app):

Provisions a Serverless Instance (i.e., sl-instance).
Provisions a Serverless Cluster (i.e., sl-clu).
Provisions a Service Account named sl-app-sa.
Provisions an API Key for the Service Account sl-app-apikey.
Provisions a Pulsar Tenant named sl-app-tenant
Provisions a Pulsar Namespace sl-app-ns and grants produce and consume permissions to the Service Account on the namespace.
Provisions a Pulsar Topic with 4 partitions sl-app-topic.
After provisioning all the resources, we will verify the resources by using the pulsarctl command.

The code examples is available in the examples/terraform/serverless folder.

0. Prerequisites

Create a new directory anywhere you'd like for this project.

mkdir terraform-getting-started && cd terraform-getting-started

1. Create a Super-Admin Service Account

First, you need to create a service account called tf-runner with Super Admin access. Please refer to Create a Service Account for details.

After you have created the service account, download the OAuth2 credentials file and save it as tf-runner.json in the terraform-getting-started folder that you created earlier.

2. Create Terraform Configuration Files to provision StreamNative Cloud resources

2.1 Create a Module Folder

Create a module folder streamnative_cloud.

mkdir streamnative_cloud && cd streamnative_cloud

2.2 Create Variables File

Create a variables.tf file inside the streamnative_cloud folder and add the following code snippet to prepare the variables. Remember to replace <your-organization-id> with your StreamNative Cloud organization id.

org_id: Get your StreamNative Cloud organization id from here and replace <your-organization-id> with it.
instance_name: The name of the Pulsar instance. Default value is sl-instance.
cluster_name: The name of the Pulsar cluster. Default value is sl-clu.
app_name: The name of the application. Default value is sl-app.

variable "org_id" {
  type = string
}

variable "instance_name" {
  type = string
}

variable "cluster_name" {
  type = string
}

variable "app_name" {
  type = string
}

2.3 Create Terraform Configuration File

Create a main.tf file inside the streamnative_cloud folder and add the following code snippet to create the resources.

Note

Use the StreamNative Terraform Provider version 0.7.0 or later.


terraform {
  required_providers {
    streamnative = {
      source  = "streamnative/streamnative"
      version = ">= 0.7.0"
    }
  }
}

provider "streamnative" {
  key_file_path = "./tf-runner.json"
}

data "streamnative_service_account" "tf-runner" {
  organization = var.org_id
  name = "tf-runner"
}

resource "streamnative_pulsar_instance" "serverless-instance" {
  organization = var.org_id
  name = var.instance_name
  availability_mode = "regional"
  pool_name = "shared-gcp"
  pool_namespace = "streamnative"
  type = "serverless"
}

data "streamnative_pulsar_instance" "serverless-instance" {
  depends_on = [streamnative_pulsar_instance.serverless-instance]
  name = streamnative_pulsar_instance.serverless-instance.name
  organization = streamnative_pulsar_instance.serverless-instance.organization
}

resource "streamnative_pulsar_cluster" "serverless-cluster" {
  depends_on = [streamnative_pulsar_instance.serverless-instance]
  organization    = streamnative_pulsar_instance.serverless-instance.organization
  name            = var.cluster_name
  display_name    = "serverless-cluster"
  instance_name   = streamnative_pulsar_instance.serverless-instance.name
  location        = "us-central1"
}

data "streamnative_pulsar_cluster" "serverless-cluster" {
  depends_on   = [streamnative_pulsar_cluster.serverless-cluster]
  organization = streamnative_pulsar_cluster.serverless-cluster.organization
  name         = split("/", streamnative_pulsar_cluster.serverless-cluster.id)[1]
}

resource "streamnative_service_account" "app-sa" {
  organization = var.org_id
  name = "${var.app_name}-sa"
  admin = false
}

data "streamnative_service_account" "app-sa" {
  depends_on = [streamnative_service_account.app-sa]
  organization = streamnative_service_account.app-sa.organization
  name = streamnative_service_account.app-sa.name
}

resource "streamnative_apikey" "app-apikey" {
  depends_on = [streamnative_pulsar_cluster.serverless-cluster, streamnative_service_account.app-sa]
  organization = var.org_id
  name = "${var.app_name}-apikey2"
  instance_name = streamnative_pulsar_instance.serverless-instance.name
  service_account_name = streamnative_service_account.app-sa.name
  description = "This is a test api key for ${var.app_name} in running the terraform tutorial"
  # If you don't want to set expiration time, you can set expiration_time to "0"
  # expiration_time = "2025-01-01T10:00:00Z"
  expiration_time = "0"
}

data "streamnative_apikey" "app-apikey" {
  depends_on = [streamnative_apikey.app-apikey]
  organization = streamnative_apikey.app-apikey.organization
  name = streamnative_apikey.app-apikey.name
  private_key = streamnative_apikey.app-apikey.private_key
}

output "apikey_token" {
  value = data.streamnative_apikey.app-apikey.token
}

output "pulsar_web_service_url" {
  value = data.streamnative_pulsar_cluster.serverless-cluster.http_tls_service_url
}

output "pulsar_cluster_name" {
  value = data.streamnative_pulsar_cluster.serverless-cluster.name
}

output "pulsar_instance_audience" {
  value = data.streamnative_pulsar_instance.serverless-instance.oauth2_audience
}

output "app_service_account_principal" {
  value = "${data.streamnative_service_account.app-sa.name}@${data.streamnative_service_account.app-sa.organization}.auth.streamnative.cloud"
}

output "service_urls" {
  value = {
    pulsar_web_service_url = data.streamnative_pulsar_cluster.serverless-cluster.http_tls_service_url
    pulsar_broker_service_url = data.streamnative_pulsar_cluster.serverless-cluster.pulsar_tls_service_url
    kafka_bootstrap_url = data.streamnative_pulsar_cluster.serverless-cluster.kafka_service_url
  }
}

3. Create Terraform Configuration Files to provision Pulsar resources

Go back to the root folder terraform-getting-started.

cd ..

3.1 Create Variables File

You can copy the variables.tf file from the streamnative_cloud module folder.

cp streamnative_cloud/variables.tf .

3.2 Create Terraform Configuration File

Create a main.tf file in the root folder and add the following code snippet to create the resources.

module "streamnative_cloud" {
  source = "./streamnative_cloud"
  org_id = var.org_id
  instance_name = var.instance_name
  cluster_name = var.cluster_name
  app_name = var.app_name
}

terraform {
  required_providers {
    pulsar = {
      source = "streamnative/pulsar"
    }
  }
}

provider "pulsar" {
  web_service_url = module.streamnative_cloud.pulsar_web_service_url
  key_file_path = "./tf-runner.json"
  audience = module.streamnative_cloud.pulsar_instance_audience
}

resource "pulsar_tenant" "app_tenant" {
  depends_on = [module.streamnative_cloud.depends_on]
  tenant = "${var.app_name}-tenant"
  allowed_clusters = [
    module.streamnative_cloud.pulsar_cluster_name,
  ]
}

resource "pulsar_namespace" "app_namespace" {
  depends_on = [pulsar_tenant.app_tenant, module.streamnative_cloud.depends_on]
  tenant    = pulsar_tenant.app_tenant.tenant
  namespace = "${var.app_name}-ns"

  namespace_config {
    replication_clusters = [
      module.streamnative_cloud.pulsar_cluster_name
    ]

  }
  permission_grant {
    role = module.streamnative_cloud.app_service_account_principal
    actions = ["produce", "consume"]
  }
}

resource "pulsar_topic" "app_topic" {
  depends_on = [pulsar_namespace.app_namespace, pulsar_tenant.app_tenant]
  tenant     = pulsar_tenant.app_tenant.tenant
  namespace  = pulsar_namespace.app_namespace.namespace
  topic_type = "persistent"
  topic_name = "${var.app_name}-topic"
  partitions =  4
}

output "apikey" {
  value = module.streamnative_cloud.apikey_token
}

output "pulsar_oauth2_audience" {
  value = module.streamnative_cloud.pulsar_instance_audience
}

output "service_urls" {
  value = module.streamnative_cloud.service_urls
}

output "pulsar_cluster_name" {
  value = module.streamnative_cloud.pulsar_cluster_name
}

output "pulsarctl_command" {
  value = "pulsarctl context set -s ${module.streamnative_cloud.pulsar_web_service_url} --token ${module.streamnative_cloud.apikey_token} ${module.streamnative_cloud.pulsar_cluster_name} && pulsarctl topics get ${var.app_name}-tenant/${var.app_name}-ns/${var.app_name}-topic"
}

4. Run Terraform Commands

Before running the Terraform commands, you need to expose the following variables:

export TF_VAR_org_id=<your-org-id>
export TF_VAR_instance_name=<your-instance-name>
export TF_VAR_cluster_name=<your-cluster-name>
export TF_VAR_app_name=<your-app-name>

Please replace the above placeholders with your actual values:

<your-org-id>: Your StreamNative Cloud organization ID
<your-instance-name>: A unique name for your Pulsar instance (e.g., "sl-instance")
<your-cluster-name>: A unique name for your Pulsar cluster (e.g., "sl-clu")
<your-app-name>: A unique name for your application (e.g., "sl-app")

An example of exposing the variables is as follows:

export TF_VAR_org_id=<your-org-id>
export TF_VAR_instance_name=sl-instance
export TF_VAR_cluster_name=sl-clu
export TF_VAR_app_name=sl-app

After exposing the variables, you can run the following Terraform commands to provision the resources.

First, initialize the Terraform working directory.

terraform init

Secondly, validate the Terraform configuration files.

terraform validate

Since we use two providers in this example (the StreamNative Provider and the Pulsar Provider), we need to provision the resources in two steps. The Pulsar Provider resources depend on the StreamNative Provider resources being created first.

4.1 Provision the Cloud Resources

Run a targeted plan to see the changes and preview the resources that will be created.

terraform plan --target=module.streamnative_cloud.streamnative_apikey.app-apikey

After that, run a targeted apply to create the resources.

terraform apply --target=module.streamnative_cloud.streamnative_apikey.app-apikey

4.2 Provision all the Resources

Run terraform plan to see the changes and preview the resources that will be created.

terraform plan

After that, run terraform apply to create the resources.

terraform apply

You should see a similar output as follows:

apikey = "<...>"
pulsarctl_command = "pulsarctl context set -s <pulsar-web-service-url> --token <apikey> sl-clu && pulsarctl topics get sl-app-tenant/sl-app-ns/sl-app-topic"
service_urls = {
  "kafka_bootstrap_url" = "..."
  "pulsar_broker_service_url" = "..."
  "pulsar_web_service_url" = "..."
}

5. Verify the Resources

Use the pulsarctl command to verify all the resources created. Make sure you have installed pulsarctl before running the command.

Copy the pulsarctl_command output and run it in your terminal.

pulsarctl context set -s <pulsar-web-service-url> --token <apikey> sl-clu && pulsarctl topics get sl-app-tenant/sl-app-ns/sl-app-topic

This command will set the context and get the topic details. It will verify the following resources are created:

A Pulsar cluster named sl-clu
A Pulsar tenant named sl-app-tenant
A Pulsar namespace named sl-app-ns
A Pulsar topic named sl-app-topic
produce and consume permissions are granted to sl-app-sa on the namespace sl-app-tenant/sl-app-ns

You should see the output as follows:

{
  "partitions": 4
}