> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploy Pulsar with Istio Service Mesh

This guide explains how to deploy StreamNative Private Cloud on Kubernetes with Istio service mesh integration. Istio provides enhanced security, observability, and traffic management for your Pulsar cluster.

## Why Use Istio with Pulsar

Deploying Pulsar with Istio provides:

* **Enhanced Security**: Automatic mTLS encryption for service-to-service communication
* **Traffic Management**: Advanced routing, load balancing, and traffic control
* **Observability**: Built-in metrics, logging, and distributed tracing
* **External Access**: Unified ingress gateway for all protocols (Pulsar, Kafka, MQTT, AMQP)
* **Policy Enforcement**: Fine-grained authorization policies

## Prerequisites

### Istio Requirements

1. **Istio Installation**: Istio installed in your Kubernetes cluster
   * Tested with Istio 1.24.2 and earlier versions
   * Use the default revision or adjust according to your Istio setup
   * Default trust domain is `cluster.local`, adjust if needed

2. **Istio Ingress Gateway**: Must be installed and configured
   * Service type must be `LoadBalancer`
   * Required port mappings:
     * `443->8443`: HTTP/HTTPS
     * `6651->6651`: Pulsar with TLS
     * `9093->9093`: Kafka with TLS
     * `5671->5671`: AMQP with TLS
     * `8883->8883`: MQTT with TLS

### TLS Certificate

* A TLS certificate secret exists in the Istio Ingress Gateway namespace
* Covers your endpoint domain (e.g., `pulsar.example.com`)
* Covers per-broker domains (e.g., `pb0-pulsar.example.com`, `pb1-pulsar.example.com`, etc.)
* Alternative: A wildcard certificate is used (e.g., `*.example.com`)

### DNS Configuration

* DNS records point to the Ingress Gateway LoadBalancer IP
* Main endpoint: `pulsar.example.com`
* Per-broker endpoints: `pb0-pulsar.example.com`, `pb1-pulsar.example.com`, up to `pb${replicas-1}-pulsar.example.com`
* Alternative: Wildcard DNS `*.example.com` is configured

<Note title="Note">
  If the StreamNative Operator is not Istio sidecar injected, configure the `cloud.streamnative.io/ignore-leader-check: "true"` annotation on the ZooKeeperCluster resource.
</Note>

## Deploy Pulsar Cluster

For a complete working example, see the [Pulsar with Istio example](https://github.com/streamnative/private-cloud/tree/main/quick-start/istio/pulsar-cluster-istio.yaml) in the StreamNative Private Cloud repository.

### Istio-Specific Configuration

Add the following Istio configurations to your Pulsar components:

#### PulsarCoordinator

Configure the Istio revision and trust domain for the cluster:

```yaml theme={null}
spec:
  istio:
    revision: default
    trustDomain: cluster.local
```

#### ZooKeeperCluster

Enable Istio with mTLS:

```yaml theme={null}
spec:
  istio:
    enabled: true
    mtls:
      mode: strict
    revision: default
```

<Note title="Note">
  If the StreamNative Operator is not Istio sidecar injected, add this annotation:

  ```yaml theme={null}
  metadata:
    annotations:
      cloud.streamnative.io/ignore-leader-check: "true"
  ```
</Note>

#### BookKeeperCluster

Enable Istio with mTLS:

```yaml theme={null}
spec:
  istio:
    enabled: true
    mtls:
      mode: strict
    revision: default
```

#### PulsarBroker

Enable Istio with mTLS and Gateway configuration for external access:

```yaml theme={null}
spec:
  config:
    advertisedDomain: pulsar.example.com
    serviceURLGenerationPolicy: OrdinalPrefix
  istio:
    enabled: true
    mtls:
      mode: strict
    revision: default
    gateway:
      selector:
        istio: ingressgateway
      tls:
        certSecretName: "ingressgateway-tls"
        mode: "simple"
```

**Key Istio Configuration Fields:**

* `spec.istio.enabled`: Enables Istio integration
* `spec.istio.mtls.mode`: mTLS mode (`strict`, `permissive`, or `none`)
* `spec.istio.revision`: Istio revision to use
* `spec.istio.gateway.selector`: Label selector for the Istio Ingress Gateway
* `spec.istio.gateway.tls.certSecretName`: Name of the TLS certificate secret
* `spec.istio.gateway.tls.mode`: TLS mode (`simple` for standard TLS)
* `spec.config.advertisedDomain`: Domain for external access
* `spec.config.serviceURLGenerationPolicy`: Set to `OrdinalPrefix` for per-broker routing

### Istio Resources Created

When you enable Istio for the PulsarBroker, the StreamNative Operator automatically creates several Istio resources to secure and expose your cluster:

**Security Resources:**

* **PeerAuthentication**: Enforces mutual TLS (mTLS) between services within the mesh. Configurable in `strict` mode (reject non-mTLS connections) or `permissive` mode (accept both mTLS and plaintext during migration).
* **DestinationRule**: Configures client-side TLS settings for service-to-service communication, automatically enabling `ISTIO_MUTUAL` mode for broker internal services.
* **AuthorizationPolicy**: Defines fine-grained access control rules, allowing traffic from specific service accounts and the Istio ingress gateway.

**External Access Resources:**

* **Gateway**: Exposes brokers externally through the Istio Ingress Gateway, with support for both cluster-level and per-pod access. Handles TLS termination or passthrough based on your configuration.
* **VirtualService**: Routes incoming traffic from the Gateway to the appropriate broker pods based on hostname and protocol (Pulsar, Kafka, MQTT, AMQP, HTTP).
* **ServiceEntry**: Creates DNS entries for individual broker pods (e.g., `pb0-pulsar.example.com`, `pb1-pulsar.example.com`), enabling Pulsar clients to discover and connect to specific brokers.

<Note>
  All Istio resources are managed automatically by the operator. You only need to configure the `spec.istio` section in your custom resources. The operator handles creation, updates, and deletion of all related Istio resources.
</Note>

## Verify Deployment

### Check Pod Status

Verify that all pods are running with Istio sidecars (2/2 containers):

```bash theme={null}
kubectl get pods -n pulsar
```

Expected output:

```
NAME                          READY   STATUS    RESTARTS   AGE
private-cloud-bk-0            2/2     Running   0          5m
private-cloud-bk-1            2/2     Running   0          5m
private-cloud-bk-2            2/2     Running   0          5m
private-cloud-broker-0        2/2     Running   0          3m
private-cloud-broker-1        2/2     Running   0          3m
private-cloud-zk-0            2/2     Running   0          8m
private-cloud-zk-1            2/2     Running   0          8m
private-cloud-zk-2            2/2     Running   0          8m
```

### Get Service Endpoints

Get the service endpoints from the PulsarBroker status:

```bash theme={null}
kubectl get pulsarbroker private-cloud -n pulsar -o=jsonpath='{.status.serviceEndpoints.cluster}{"\n"}{.status.serviceEndpoints.external}'
```

Expected output:

```json theme={null}
{"pulsarServiceURL":"pulsar://private-cloud-broker.pulsar.svc.cluster.local:6650","webServiceURL":"http://private-cloud-broker.pulsar.svc.cluster.local:8080"}
{"pulsarServiceURL":"pulsar+ssl://pulsar.example.com:6651","webServiceURL":"https://pulsar.example.com"}
```

**Internal Endpoints (within Kubernetes cluster):**

* Admin: `http://private-cloud-broker.pulsar.svc.cluster.local:8080`
* Pulsar: `pulsar://private-cloud-broker.pulsar.svc.cluster.local:6650`

**External Endpoints (outside Kubernetes cluster):**

* Admin: `https://pulsar.example.com`
* Pulsar TLS: `pulsar+ssl://pulsar.example.com:6651`

## Test Connectivity

### Test Within Kubernetes Cluster

Test Pulsar using the toolset created by the StreamNative Operator or any Istio-injected pod:

```bash theme={null}
kubectl exec -it -n pulsar private-cloud-toolset-0 -- bash

# List tenants
pulsar-admin --admin-url http://private-cloud-broker.pulsar.svc.cluster.local:8080 tenants list

# Consume messages
pulsar-client --url pulsar://private-cloud-broker.pulsar.svc.cluster.local:6650 \
  consume public/default/test -s sub1 -n 10

# Produce messages
pulsar-client --url pulsar://private-cloud-broker.pulsar.svc.cluster.local:6650 \
  produce public/default/test -m "hello" -n 10
```

### Test Outside Kubernetes Cluster

Test Pulsar using TLS from outside the cluster:

```bash theme={null}
# List tenants
pulsar-admin --admin-url https://pulsar.example.com tenants list

# Consume messages
pulsar-client --url pulsar+ssl://pulsar.example.com:6651 \
  consume public/default/test -s sub1 -n 10

# Produce messages
pulsar-client --url pulsar+ssl://pulsar.example.com:6651 \
  produce public/default/test -m "hello" -n 10
```

## Istio Ambient Mode

Starting with sn-operator v0.13, you can deploy Pulsar with Istio **ambient mode** — a sidecarless data plane architecture that replaces per-pod Envoy sidecars with a shared node-level **ztunnel** DaemonSet.

### Benefits

* **Reduced mesh overhead**: No per-pod sidecar containers (saves \~100MB memory + 0.1 CPU per pod)
* **Eliminates startup delays**: No sidecar injection or init container waits before the application starts
* **Simpler operations**: New pods join the mesh via labels without sidecar injection
* **Full feature parity**: All Istio resources (Gateway, VirtualService, PeerAuthentication, AuthorizationPolicy) continue to work

<Note title="Note">
  Ambient mode uses ztunnel for L4 mTLS only. L7 features such as HTTP routing and TLS termination are handled by the existing Istio Gateway, which has its own Envoy proxy. No waypoint proxy is needed for Pulsar workloads.
</Note>

### Prerequisites

* **Istio 1.24+** with ambient mode enabled and the ztunnel DaemonSet running on your cluster. Refer to the [Istio Ambient Mode documentation](https://istio.io/latest/docs/ambient/) for installation and configuration instructions specific to your environment.

Verify the ztunnel DaemonSet is running:

```bash theme={null}
kubectl get daemonset -n istio-system ztunnel
```

### Configure Ambient Mode

To configure ambient mode, set `dataplaneMode: ambient` on the PulsarCoordinator. The operator propagates this setting to all managed components (PulsarBroker, PulsarProxy, BookKeeperCluster, ZooKeeperCluster) automatically via server-side apply.

#### PulsarCoordinator

Add the following `dataplaneMode` field to the existing `spec.istio` section of your PulsarCoordinator resource:

```yaml theme={null}
spec:
  istio:
    dataplaneMode: ambient
    # ... other existing istio settings (revision, trustDomain, etc.)
```

The operator propagates `dataplaneMode` to all managed component CRDs. If you manage components individually without a PulsarCoordinator, add the same field directly to each component's `spec.istio` section.

**Configuration reference:**

| Field                      | Values                 | Description                                       |
| -------------------------- | ---------------------- | ------------------------------------------------- |
| `spec.istio.dataplaneMode` | `ambient` or `sidecar` | Data plane mode. Defaults to `sidecar` if not set |

### Verify Ambient Mode Deployment

Verify that pods are running **without sidecars** (1/1 containers instead of 2/2):

```bash theme={null}
kubectl get pods -n pulsar
```

Expected output:

```
NAME                          READY   STATUS    RESTARTS   AGE
private-cloud-bk-0            1/1     Running   0          5m
private-cloud-bk-1            1/1     Running   0          5m
private-cloud-bk-2            1/1     Running   0          5m
private-cloud-broker-0        1/1     Running   0          3m
private-cloud-broker-1        1/1     Running   0          3m
private-cloud-zk-0            1/1     Running   0          8m
private-cloud-zk-1            1/1     Running   0          8m
private-cloud-zk-2            1/1     Running   0          8m
```

Verify pods have the ambient mode label:

```bash theme={null}
kubectl get pods -n pulsar -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.labels.istio\.io/dataplane-mode}{"\n"}{end}'
```

Expected output shows `ambient` for each pod:

```
private-cloud-bk-0       ambient
private-cloud-bk-1       ambient
private-cloud-bk-2       ambient
private-cloud-broker-0   ambient
private-cloud-broker-1   ambient
private-cloud-zk-0       ambient
private-cloud-zk-1       ambient
private-cloud-zk-2       ambient
```

### Migrate from Sidecar to Ambient Mode

To migrate an existing sidecar-mode cluster to ambient mode:

1. Ensure Istio 1.24+ with ambient mode is running in your cluster.

2. Update the PulsarCoordinator (replace `<name>` with your PulsarCoordinator resource name and `<namespace>` with the Pulsar namespace):

   ```bash theme={null}
   kubectl patch pulsarcoordinator <name> -n <namespace> --type merge \
     -p '{"spec":{"istio":{"dataplaneMode":"ambient"}}}'
   ```

3. Monitor the rolling update by checking that each component CR reaches ready state:

   ```bash theme={null}
   kubectl get pulsarbroker,bookkeepercluster,zookeepercluster -n <namespace>
   ```

   Wait until the `READY REPLICAS` column matches `REPLICAS` for each component.

4. Verify ambient enrollment (see [Verify Ambient Mode Deployment](#verify-ambient-mode-deployment) above).

<Note title="Note">
  During the rolling update, mTLS is maintained. Istio supports interoperability between ztunnel (ambient) and sidecar proxies, so old and new pods can communicate throughout the migration.
</Note>

**Roll back to sidecar mode**: To revert, set `dataplaneMode` back to `sidecar` and restore any namespace-level sidecar injection labels (e.g., `istio-injection=enabled` or `istio.io/rev=<revision>`) that were previously removed.

```bash theme={null}
kubectl patch pulsarcoordinator <name> -n <namespace> --type merge \
  -p '{"spec":{"istio":{"dataplaneMode":"sidecar"}}}'
```

### Ambient Mode Troubleshooting

| Symptom                                      | Cause                          | Solution                                                                         |
| -------------------------------------------- | ------------------------------ | -------------------------------------------------------------------------------- |
| Pods still show 2/2 containers               | Sidecar still injected         | Verify `dataplaneMode: ambient` is set and pods have restarted                   |
| Pods missing `istio.io/dataplane-mode` label | Setting not propagated         | Check PulsarCoordinator spec and component CRD specs                             |
| mTLS not working                             | ztunnel not running            | Verify `kubectl get daemonset -n istio-system ztunnel` shows pods on all nodes   |
| Gateway routing not working                  | Gateway resource misconfigured | Check Gateway and VirtualService resources — these work the same in ambient mode |

## Enable Protocol Handlers

After deploying the base Pulsar cluster with Istio, you can enable additional protocol handlers:

* [Configure Kafka Protocol](/private-cloud/v2/configure-private-cloud/protocols/configure-kafka-protocol) - Enable Kafka protocol support
* [Configure MQTT Protocol](/private-cloud/v2/configure-private-cloud/protocols/configure-mqtt-protocol) - Enable MQTT protocol support
* [Configure AMQP Protocol](/private-cloud/v2/configure-private-cloud/protocols/configure-amqp-091-protocol) - Enable AMQP 0-9-1 protocol support