Deploy Pulsar with Istio Service Mesh

This guide explains how to deploy StreamNative Private Cloud on Kubernetes with Istio service mesh integration. Istio provides enhanced security, observability, and traffic management for your Pulsar cluster.

Why Use Istio with Pulsar

Deploying Pulsar with Istio provides:

Enhanced Security: Automatic mTLS encryption for service-to-service communication
Traffic Management: Advanced routing, load balancing, and traffic control
Observability: Built-in metrics, logging, and distributed tracing
External Access: Unified ingress gateway for all protocols (Pulsar, Kafka, MQTT, AMQP)
Policy Enforcement: Fine-grained authorization policies

Prerequisites

Istio Requirements

Istio Installation: Istio installed in your Kubernetes cluster
- Tested with Istio 1.24.2 and earlier versions
- Use the default revision or adjust according to your Istio setup
- Default trust domain is cluster.local, adjust if needed
Istio Ingress Gateway: Must be installed and configured
- Service type must be LoadBalancer
- Required port mappings:
  - 443->8443: HTTP/HTTPS
  - 6651->6651: Pulsar with TLS
  - 9093->9093: Kafka with TLS
  - 5671->5671: AMQP with TLS
  - 8883->8883: MQTT with TLS

TLS Certificate

A TLS certificate secret exists in the Istio Ingress Gateway namespace
Covers your endpoint domain (e.g., pulsar.example.com)
Covers per-broker domains (e.g., pb0-pulsar.example.com, pb1-pulsar.example.com, etc.)
Alternative: A wildcard certificate is used (e.g., *.example.com)

DNS Configuration

DNS records point to the Ingress Gateway LoadBalancer IP
Main endpoint: pulsar.example.com
Per-broker endpoints: pb0-pulsar.example.com, pb1-pulsar.example.com, up to pb${replicas-1}-pulsar.example.com
Alternative: Wildcard DNS *.example.com is configured

If the StreamNative Operator is not Istio sidecar injected, configure the cloud.streamnative.io/ignore-leader-check: "true" annotation on the ZooKeeperCluster resource.

Deploy Pulsar Cluster

For a complete working example, see the Pulsar with Istio example in the StreamNative Private Cloud repository.

Istio-Specific Configuration

Add the following Istio configurations to your Pulsar components:

PulsarCoordinator

Configure the Istio revision and trust domain for the cluster:

spec:
  istio:
    revision: default
    trustDomain: cluster.local

ZooKeeperCluster

Enable Istio with mTLS:

spec:
  istio:
    enabled: true
    mtls:
      mode: strict
    revision: default

If the StreamNative Operator is not Istio sidecar injected, add this annotation:

metadata:
  annotations:
    cloud.streamnative.io/ignore-leader-check: "true"

BookKeeperCluster

Enable Istio with mTLS:

spec:
  istio:
    enabled: true
    mtls:
      mode: strict
    revision: default

PulsarBroker

Enable Istio with mTLS and Gateway configuration for external access:

spec:
  config:
    advertisedDomain: pulsar.example.com
    serviceURLGenerationPolicy: OrdinalPrefix
  istio:
    enabled: true
    mtls:
      mode: strict
    revision: default
    gateway:
      selector:
        istio: ingressgateway
      tls:
        certSecretName: "ingressgateway-tls"
        mode: "simple"

Key Istio Configuration Fields:

spec.istio.enabled: Enables Istio integration
spec.istio.mtls.mode: mTLS mode (strict, permissive, or none)
spec.istio.revision: Istio revision to use
spec.istio.gateway.selector: Label selector for the Istio Ingress Gateway
spec.istio.gateway.tls.certSecretName: Name of the TLS certificate secret
spec.istio.gateway.tls.mode: TLS mode (simple for standard TLS)
spec.config.advertisedDomain: Domain for external access
spec.config.serviceURLGenerationPolicy: Set to OrdinalPrefix for per-broker routing

Istio Resources Created

When you enable Istio for the PulsarBroker, the StreamNative Operator automatically creates several Istio resources to secure and expose your cluster: Security Resources:

PeerAuthentication: Enforces mutual TLS (mTLS) between services within the mesh. Configurable in strict mode (reject non-mTLS connections) or permissive mode (accept both mTLS and plaintext during migration).
DestinationRule: Configures client-side TLS settings for service-to-service communication, automatically enabling ISTIO_MUTUAL mode for broker internal services.
AuthorizationPolicy: Defines fine-grained access control rules, allowing traffic from specific service accounts and the Istio ingress gateway.

External Access Resources:

Gateway: Exposes brokers externally through the Istio Ingress Gateway, with support for both cluster-level and per-pod access. Handles TLS termination or passthrough based on your configuration.
VirtualService: Routes incoming traffic from the Gateway to the appropriate broker pods based on hostname and protocol (Pulsar, Kafka, MQTT, AMQP, HTTP).
ServiceEntry: Creates DNS entries for individual broker pods (e.g., pb0-pulsar.example.com, pb1-pulsar.example.com), enabling Pulsar clients to discover and connect to specific brokers.

All Istio resources are managed automatically by the operator. You only need to configure the spec.istio section in your custom resources. The operator handles creation, updates, and deletion of all related Istio resources.

Verify Deployment

Check Pod Status

Verify that all pods are running with Istio sidecars (2/2 containers):

kubectl get pods -n pulsar

Expected output:

NAME                          READY   STATUS    RESTARTS   AGE
private-cloud-bk-0            2/2     Running   0          5m
private-cloud-bk-1            2/2     Running   0          5m
private-cloud-bk-2            2/2     Running   0          5m
private-cloud-broker-0        2/2     Running   0          3m
private-cloud-broker-1        2/2     Running   0          3m
private-cloud-zk-0            2/2     Running   0          8m
private-cloud-zk-1            2/2     Running   0          8m
private-cloud-zk-2            2/2     Running   0          8m

Get Service Endpoints

Get the service endpoints from the PulsarBroker status:

kubectl get pulsarbroker private-cloud -n pulsar -o=jsonpath='{.status.serviceEndpoints.cluster}{"\n"}{.status.serviceEndpoints.external}'

Expected output:

{"pulsarServiceURL":"pulsar://private-cloud-broker.pulsar.svc.cluster.local:6650","webServiceURL":"http://private-cloud-broker.pulsar.svc.cluster.local:8080"}
{"pulsarServiceURL":"pulsar+ssl://pulsar.example.com:6651","webServiceURL":"https://pulsar.example.com"}

Internal Endpoints (within Kubernetes cluster):

Admin: http://private-cloud-broker.pulsar.svc.cluster.local:8080
Pulsar: pulsar://private-cloud-broker.pulsar.svc.cluster.local:6650

External Endpoints (outside Kubernetes cluster):

Admin: https://pulsar.example.com
Pulsar TLS: pulsar+ssl://pulsar.example.com:6651

Test Connectivity

Test Within Kubernetes Cluster

Test Pulsar using the toolset created by the StreamNative Operator or any Istio-injected pod:

kubectl exec -it -n pulsar private-cloud-toolset-0 -- bash

# List tenants
pulsar-admin --admin-url http://private-cloud-broker.pulsar.svc.cluster.local:8080 tenants list

# Consume messages
pulsar-client --url pulsar://private-cloud-broker.pulsar.svc.cluster.local:6650 \
  consume public/default/test -s sub1 -n 10

# Produce messages
pulsar-client --url pulsar://private-cloud-broker.pulsar.svc.cluster.local:6650 \
  produce public/default/test -m "hello" -n 10

Test Outside Kubernetes Cluster

Test Pulsar using TLS from outside the cluster:

# List tenants
pulsar-admin --admin-url https://pulsar.example.com tenants list

# Consume messages
pulsar-client --url pulsar+ssl://pulsar.example.com:6651 \
  consume public/default/test -s sub1 -n 10

# Produce messages
pulsar-client --url pulsar+ssl://pulsar.example.com:6651 \
  produce public/default/test -m "hello" -n 10

Istio Ambient Mode

Starting with sn-operator v0.13, you can deploy Pulsar with Istio ambient mode — a sidecarless data plane architecture that replaces per-pod Envoy sidecars with a shared node-level ztunnel DaemonSet.

Benefits

Reduced mesh overhead: No per-pod sidecar containers (saves ~100MB memory + 0.1 CPU per pod)
Eliminates startup delays: No sidecar injection or init container waits before the application starts
Simpler operations: New pods join the mesh via labels without sidecar injection
Full feature parity: All Istio resources (Gateway, VirtualService, PeerAuthentication, AuthorizationPolicy) continue to work

Ambient mode uses ztunnel for L4 mTLS only. L7 features such as HTTP routing and TLS termination are handled by the existing Istio Gateway, which has its own Envoy proxy. No waypoint proxy is needed for Pulsar workloads.

Prerequisites

Istio 1.24+ with ambient mode enabled and the ztunnel DaemonSet running on your cluster. Refer to the Istio Ambient Mode documentation for installation and configuration instructions specific to your environment.

Verify the ztunnel DaemonSet is running:

kubectl get daemonset -n istio-system ztunnel

Configure Ambient Mode

To configure ambient mode, set dataplaneMode: ambient on the PulsarCoordinator. The operator propagates this setting to all managed components (PulsarBroker, PulsarProxy, BookKeeperCluster, ZooKeeperCluster) automatically via server-side apply.

PulsarCoordinator

Add the following dataplaneMode field to the existing spec.istio section of your PulsarCoordinator resource:

spec:
  istio:
    dataplaneMode: ambient
    # ... other existing istio settings (revision, trustDomain, etc.)

The operator propagates dataplaneMode to all managed component CRDs. If you manage components individually without a PulsarCoordinator, add the same field directly to each component’s spec.istio section. Configuration reference:

Field	Values	Description
`spec.istio.dataplaneMode`	`ambient` or `sidecar`	Data plane mode. Defaults to `sidecar` if not set

Verify Ambient Mode Deployment

Verify that pods are running without sidecars (1/1 containers instead of 2/2):

kubectl get pods -n pulsar

Expected output:

NAME                          READY   STATUS    RESTARTS   AGE
private-cloud-bk-0            1/1     Running   0          5m
private-cloud-bk-1            1/1     Running   0          5m
private-cloud-bk-2            1/1     Running   0          5m
private-cloud-broker-0        1/1     Running   0          3m
private-cloud-broker-1        1/1     Running   0          3m
private-cloud-zk-0            1/1     Running   0          8m
private-cloud-zk-1            1/1     Running   0          8m
private-cloud-zk-2            1/1     Running   0          8m

Verify pods have the ambient mode label:

kubectl get pods -n pulsar -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.istio\.io/dataplane-mode}{"\n"}{end}'

Expected output shows ambient for each pod:

private-cloud-bk-0       ambient
private-cloud-bk-1       ambient
private-cloud-bk-2       ambient
private-cloud-broker-0   ambient
private-cloud-broker-1   ambient
private-cloud-zk-0       ambient
private-cloud-zk-1       ambient
private-cloud-zk-2       ambient

Migrate from Sidecar to Ambient Mode

To migrate an existing sidecar-mode cluster to ambient mode:

Ensure Istio 1.24+ with ambient mode is running in your cluster.

Update the PulsarCoordinator (replace <name> with your PulsarCoordinator resource name and <namespace> with the Pulsar namespace):

kubectl patch pulsarcoordinator <name> -n <namespace> --type merge \
  -p '{"spec":{"istio":{"dataplaneMode":"ambient"}}}'

Monitor the rolling update by checking that each component CR reaches ready state:
```
kubectl get pulsarbroker,bookkeepercluster,zookeepercluster -n <namespace>
```
Wait until the READY REPLICAS column matches REPLICAS for each component.
Verify ambient enrollment (see Verify Ambient Mode Deployment above).

During the rolling update, mTLS is maintained. Istio supports interoperability between ztunnel (ambient) and sidecar proxies, so old and new pods can communicate throughout the migration.

Roll back to sidecar mode: To revert, set dataplaneMode back to sidecar and restore any namespace-level sidecar injection labels (e.g., istio-injection=enabled or istio.io/rev=<revision>) that were previously removed.

kubectl patch pulsarcoordinator <name> -n <namespace> --type merge \
  -p '{"spec":{"istio":{"dataplaneMode":"sidecar"}}}'

Ambient Mode Troubleshooting

Symptom	Cause	Solution
Pods still show 2/2 containers	Sidecar still injected	Verify `dataplaneMode: ambient` is set and pods have restarted
Pods missing `istio.io/dataplane-mode` label	Setting not propagated	Check PulsarCoordinator spec and component CRD specs
mTLS not working	ztunnel not running	Verify `kubectl get daemonset -n istio-system ztunnel` shows pods on all nodes
Gateway routing not working	Gateway resource misconfigured	Check Gateway and VirtualService resources — these work the same in ambient mode

Enable Protocol Handlers

After deploying the base Pulsar cluster with Istio, you can enable additional protocol handlers:

Configure Kafka Protocol - Enable Kafka protocol support
Configure MQTT Protocol - Enable MQTT protocol support
Configure AMQP Protocol - Enable AMQP 0-9-1 protocol support

Overview

Quick Start

Configure Private Cloud

Operate Private Cloud

References

Why Use Istio with Pulsar

Prerequisites

Istio Requirements

TLS Certificate

DNS Configuration

Deploy Pulsar Cluster

Istio-Specific Configuration

PulsarCoordinator

ZooKeeperCluster

BookKeeperCluster

PulsarBroker

Istio Resources Created

Verify Deployment

Check Pod Status

Get Service Endpoints

Test Connectivity

Test Within Kubernetes Cluster

Test Outside Kubernetes Cluster

Istio Ambient Mode

Benefits

Prerequisites

Configure Ambient Mode

PulsarCoordinator

Verify Ambient Mode Deployment

Migrate from Sidecar to Ambient Mode

Ambient Mode Troubleshooting

Enable Protocol Handlers

Overview

Quick Start

Configure Private Cloud

Operate Private Cloud

References

​Why Use Istio with Pulsar

​Prerequisites

​Istio Requirements

​TLS Certificate

​DNS Configuration

​Deploy Pulsar Cluster

​Istio-Specific Configuration

​PulsarCoordinator

​ZooKeeperCluster

​BookKeeperCluster

​PulsarBroker

​Istio Resources Created

​Verify Deployment

​Check Pod Status

​Get Service Endpoints

​Test Connectivity

​Test Within Kubernetes Cluster

​Test Outside Kubernetes Cluster

​Istio Ambient Mode

​Benefits

​Prerequisites

​Configure Ambient Mode

​PulsarCoordinator

​Verify Ambient Mode Deployment

​Migrate from Sidecar to Ambient Mode

​Ambient Mode Troubleshooting

​Enable Protocol Handlers

Why Use Istio with Pulsar

Prerequisites

Istio Requirements

TLS Certificate

DNS Configuration

Deploy Pulsar Cluster

Istio-Specific Configuration

PulsarCoordinator

ZooKeeperCluster

BookKeeperCluster

PulsarBroker

Istio Resources Created

Verify Deployment

Check Pod Status

Get Service Endpoints

Test Connectivity

Test Within Kubernetes Cluster

Test Outside Kubernetes Cluster

Istio Ambient Mode

Benefits

Prerequisites

Configure Ambient Mode

PulsarCoordinator

Verify Ambient Mode Deployment

Migrate from Sidecar to Ambient Mode

Ambient Mode Troubleshooting

Enable Protocol Handlers