Broker Ingress Architecture

External clients cannot reach a Pulsar cluster through a single shared address. After a lookup, a client connects directly to the one broker that owns the topic, so the ingress in front of your cluster must be able to reach each broker individually. This page explains the architecture that makes that possible, and what each part of it must do.

How a client reaches a broker

Every Pulsar connection happens in two steps. The first connection can go to any broker, because any broker can answer a lookup. The second connection must reach one specific broker. If every broker advertises the same address, the second connection lands on whichever broker the load balancer chooses, and the client receives ServiceNotReady ... Please redo the lookup until it happens to reach the owner. That failure is intermittent rather than total. Topics owned by the broker the shared address reaches will work, which makes partial success a symptom of the problem rather than a sign of progress.

Reference architecture

To satisfy the second step, each broker needs its own externally resolvable address, and the ingress needs a way to tell those addresses apart. The only field in a Pulsar connection that carries the intended broker is the TLS server name, so the architecture is built around preserving it. Four properties make this work:

The load balancer forwards TCP without terminating TLS, so the server name survives to the gateway.
The gateway holds the certificate, terminates TLS, and selects a route by server name.
Each route forwards to one broker, not to a service that fronts all of them.
Each broker advertises its own external hostname, so the address a client receives from a lookup matches a route.

Whichever component terminates TLS consumes the server name. Every hop behind it sees an undifferentiated TCP stream and can no longer tell brokers apart, which is why termination belongs at the gateway rather than at the load balancer.

Ports

Each broker exposes several listeners, and the port a client connects to determines which address the broker returns.

Port	Listener	Address returned	Reachable from
6650	internal	in-cluster DNS name of the broker Pod	inside the Kubernetes cluster
6651	gateway	external hostname of the broker	outside the cluster, through the ingress
6652	gateway bind	not advertised	the gateway only

Clients connect on 6651. The gateway terminates TLS and forwards plaintext to 6652, which is why the port a client dials and the port a broker listens on are different. Brokers need no certificates of their own in this arrangement.

Configure the brokers

Enable the gateway listener and set the advertised domain on the PulsarBroker resource. Each broker then advertises <pod-name>-<advertisedDomain>, for example pulsar-broker-0-broker.example.com.

apiVersion: pulsar.streamnative.io/v1alpha1
kind: PulsarBroker
metadata:
  name: pulsar
  namespace: pulsar
spec:
  # Expose each broker through the ingress gateway
  istio:
    enabled: true
    gateway:
      tls:
        mode: SIMPLE
        certSecretName: "<gateway-certificate-secret>"
  config:
    advertisedDomain: broker.example.com
    serviceURLGenerationPolicy: FQDN

Routing generated for an Istio gateway targets each broker Pod through the headless Service, so no additional Service is required. If your ingress instead routes to Services, as Kubernetes Gateway API routes do, enable per-Pod Services to give each broker its own backend:

spec:
  networking:
    podService:
      enabled: true
      type: ClusterIP

See Configure Broker Per-Pod Networking for the full set of per-Pod networking options.

Configure the ingress

Create one route per broker, plus one for lookups. Match on the TLS server name and forward to that broker’s backend.

TLS server name	Port	Backend
`broker.example.com`	6651	the broker Service, any broker
`pulsar-broker-0-broker.example.com`	6651	`pulsar-broker-0` on 6652
`pulsar-broker-1-broker.example.com`	6651	`pulsar-broker-1` on 6652
`pulsar-broker-2-broker.example.com`	6651	`pulsar-broker-2` on 6652

The lookup route may target the shared broker Service, because any broker can answer a lookup. The per-broker routes must each reach exactly one broker. How you express a single broker as a backend depends on the ingress: routing generated for an Istio gateway addresses the Pod through the headless Service, while Kubernetes Gateway API routes need a per-Pod Service. The rule set grows with the broker count, so adding brokers means adding routes.

Requirements

Before clients can connect, all of the following must be true:

Port 6651 reaches the ingress gateway as plain TCP, without TLS termination at the edge.
DNS resolves the cluster hostname and every per-broker hostname to the ingress. A wildcard record for the external domain covers both.
The certificate presented by the gateway covers the cluster hostname and every per-broker hostname.
One routing rule exists per broker, matching on TLS server name.
Clients connect with pulsar+ssl:// on 6651 and trust the certificate chain.

Verify the setup

First, confirm the gateway distinguishes the hostnames. Each server name must complete a TLS handshake and keep the connection open. A handshake that completes and then closes immediately means the name matched no route.

openssl s_client -connect broker.example.com:6651 \
  -servername pulsar-broker-0-broker.example.com -brief </dev/null

Then confirm the routes reach the right brokers. Consume a partitioned topic whose partitions are spread across brokers. Every partition must subscribe:

bin/pulsar-client --url pulsar+ssl://broker.example.com:6651 \
  consume persistent://public/default/<partitioned-topic> \
  -s ingress-check -p Earliest -n 6

If only some partitions subscribe, at least one route still points at a destination shared by several brokers.

Overview

Quick Start

Configure Private Cloud

Operate Private Cloud

References

How a client reaches a broker

Reference architecture

Ports

Configure the brokers

Configure the ingress

Requirements

Verify the setup

​How a client reaches a broker

​Reference architecture

​Ports

​Configure the brokers

​Configure the ingress

​Requirements

​Verify the setup

How a client reaches a broker

Reference architecture

Ports

Configure the brokers

Configure the ingress

Requirements

Verify the setup