Configure Istio

Istio is an open source service mesh that layers transparently onto existing distributed applications. It provides a uniform and more efficient way to secure, connect, and monitor services. This document describes how to configure Istio on StreamNative Platform to expose KoP, MoP, AoP, the Pulsar broker, StreamNative Console, and Grafana services.

Prerequisites

sn-platform chart: 1.6.2 or higher
pulsar-operator chart: 0.12.3 or higher
cert-manager operator: v1.0.0 or higher
Install kubectl v1.16 or higher.
Install Helm 3.0 or higher.
Install the istioctl CLI tool.

Limitations

In the sn-platform chart versions prior to 1.6.2, it is not supported to expose HTTP requests through Pulsar Proxy when Istio is enabled. Therefore, it is recommended to disable Pulsar Proxy when you want to enable Istio.

Install Istio

This section describes how to install Istio with the Istio operator. For other Istio installation methods, see installation guides.

Go to the Istio release page to download the installation file for your Operating System (OS), or download and extract the latest release automatically (Linux or macOS):
```
curl -L https://istio.io/downloadIstio | sh -
```

The command above downloads the latest release (numerically) of Istio. You can pass variables on the command line to download a specific version or to override the processor architecture. For example, to download Istio 1.13.1 for the x86_64 architecture, execute the following command:

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.13.1 TARGET_ARCH=x86_64 sh -

Move to the Istio package directory. For example, if the package is istio-1.13.1:
```
cd istio-1.13.1
```
Install the Istio operator.
```
bin/istioctl operator init
```

Install Istio Ingress Gateway

Istio Ingress Gateway describes a load balancer operating at the edge of the Pulsar that receives incoming HTTP/TCP connections. It configures exposed ports, protocols, etc.

Define a YAML file of the Istio Ingress Gateway.

This example defines an Istio Ingress Gateway YAML file (ingressgateway.yaml), which configures gateways on different ports for KoP, MoP, AoP, HTTPS, TLS, and HTTP services.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: example-istiocontrolplane
  namespace: istio-system
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          service:
            ports:
              - port: 9093
                targetPort: 9093
                name: tcp-kop
              - port: 443
                targetPort: 8443
                name: https
              - name: tls-pulsar
                port: 6651
                targetPort: 6651
              - port: 80
                targetPort: 8080
                name: http
              - port: 8883
                targetPort: 8883
                name: tcp-mop
              - port: 5671
                targetPort: 5671
                name: tcp-aop

Apply the YAML file to install the Istio Ingress Gateway.
```
kubectl apply -f <path/to/ingressgateway.yaml>
```

View whether configurations are applied successfully.

kubectl get svc -n istio-system

Example output

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                AGE
istio-ingressgateway   LoadBalancer   10.99.165.241   <pending>     9093:9093/TCP,443:31897/TCP,80:31509/TCP,6651:6651/TCP,8883:8883/TCP,5671:5671/TCP   30h
istiod                 ClusterIP      10.96.121.192   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP                                  30h

From the above output you can see that the Istio Ingress Gateway is installed successfully.

If the EXTERNAL-IP value is set, your environment has an external load balancer that you can use for the Istio Ingress Gateway. If the EXTERNAL-IP value is <none> (or perpetually <pending>), your environment does not provide an external load balancer for the Istio Ingress Gateway. In this case, you can access the Istio Ingress Gateway using the service’s node port.
If you have an external load balancer for your Kubernetes cluster, you need to specify the port 9093, port 8883, port 5671, and port 6651 for KoP, MoP, AoP, and the Pulsar Broker respectively on the load balancer.
If you do not have an external load balancer for your Kubernetes cluster, you need to specify the node port 9093 for KoP, the node port 8883 for MoP, the node port 5671 for AoP, and the node port 6651 for the Pulsar Broker. By default, Kubernetes does not permit port 9093, port 8883, or port 6651. Therefore, you need to add the --service-node-port-range=6000-40000 field at the /etc/kubernetes/manifests/kube-apiserver.yaml Kubernetes API server configuration file and then use the systemctl restart kubelet command to restart the Kubernetes API server to reload new configurations.

Enable Istio Gateway

This section describes how to enable the Istio Gateway.

Configure TLS Ingress Gateway

Certificates can be issued in multiple methods. This section describes two methods. You can choose either of them to issue certificates for the Istio Ingress Gateway.

`openssl`

This section describes how to manually configure the TLS Ingress Gateway to expose a secure service using TLS.

Generate client and server certificates and keys.

a. Create a root certificate and private key to sign the certificates for your services.

openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=<example Inc./CN=example.com>' -keyout <example.com.key> -out <example.com.crt>

b. Create a certificate and a private key.

openssl req -out <certs/example.com.csr> -newkey rsa:2048 -nodes -keyout <certs/example.com.key> -subj "/CN=<*.example.com/O=httpbin organization>"
openssl x509 -req -sha256 -days 365 -CA <example.com.crt> -CAkey <example.com.key> -set_serial 0 -in <certs/example.com.csr> -out <certs/example.com.crt>

Create a secret for the Istio Ingress Gateway.

kubectl create secret tls <example-credential> -n <k8s_namespace> \
--key=<key/example.com.key> \
--cert=<certs/example.com.crt>

cert-manager

cert-manager supports issuing certificates through different issuers. You can use either an internal or a public issuer to issue the certificates.

To generate certificates with an internal issuer, you can set certs.istio_internal_issuer.enabled to true in the values.yaml YAML file and then use the helm upgrade command to update the resource. The certificates generated by the internal issuer are self-signed certificates.

certs:
  istio_internal_issuer:
    enabled: true # --- [1]
    component: istio-internal-cert-issuer
    type: selfsigning # --- [2]
  issuers: # --- [3]
    selfsigning:

[1] enabled: use an internal issuer.
[2] type: specify the type of the internal issuer.
- selfsigning: the selfsigning issuer generates a Certificate Authority (CA) based on an automatically-generated secret. You can use the private key of the certificate to sign the certificate itself.
- secret: the secret issuer represents a CA whose certificate and private key are stored inside the cluster as a Kubernetes Secret. Then, you can use the Kubernetes Secret to sign a certificate.
  certs: internal_issuer: enabled: true type: secret issuers: secret: secretName: ca-keypair ... # other configs
- custom: you can specify an internal issuer through this option. For example, you can specify using Vault to sign certificates for your PKI, as shown below.
  certs: internal_issuer: enabled: true type: custom issuers: custom: path: pki_int/sign/example-dot-com server: https://vault.local caBundle: <base64 encoded CA Bundle PEM file> ... # other configs
[3] issuers and its sub fields: the configuration of the internal issuer.
- selfsigning: the selfsigning issuer has no dependency on any other resource. Therefore, you do not need to configure any item for this issuer.
- secret: secretName is the name of the Secret resource that is automatically created and managed by the CA. It is populated with a private key and certificate, signed by the issuer.
- custom: you can configure items for custom certificate issuers. For more detailed configurations, see the cert-manager documentation.

certs:
  istio_internal_issuer:
    enabled: true # --- [1]
    component: istio-internal-cert-issuer
    type: selfsigning # --- [2]
  issuers: # --- [3]
    selfsigning:

[1] enabled: use an internal issuer.
[2] type: specify the type of the internal issuer.
- selfsigning: the selfsigning issuer generates a Certificate Authority (CA) based on an automatically-generated secret. You can use the private key of the certificate to sign the certificate itself.
- secret: the secret issuer represents a CA whose certificate and private key are stored inside the cluster as a Kubernetes Secret. Then, you can use the Kubernetes Secret to sign a certificate.
  certs: internal_issuer: enabled: true type: secret issuers: secret: secretName: ca-keypair ... # other configs
- custom: you can specify an internal issuer through this option. For example, you can specify using Vault to sign certificates for your PKI, as shown below.
  certs: internal_issuer: enabled: true type: custom issuers: custom: path: pki_int/sign/example-dot-com server: https://vault.local caBundle: <base64 encoded CA Bundle PEM file> ... # other configs
[3] issuers and its sub fields: the configuration of the internal issuer.
- selfsigning: the selfsigning issuer has no dependency on any other resource. Therefore, you do not need to configure any item for this issuer.
- secret: secretName is the name of the Secret resource that is automatically created and managed by the CA. It is populated with a private key and certificate, signed by the issuer.
- custom: you can configure items for custom certificate issuers. For more detailed configurations, see the cert-manager documentation.

To generate certificates with the public issuer, you can specify the certs.istio_public_issuer.enabled to true in the values.yaml YAML file and configure the information about the public issuer. Then, you can use the helm upgrade command to update the resource.

This example shows how to use the ACME public issuer to issue certificates. For a list of available issuers, see the cert-Manager official documentation.

certs:
  istio_public_issuer:
    enabled: false # --- [1]
    component: istio-public-cert-issuer
    type: acme # --- [2]

  issuers:
    acme: # --- [3]
      # You must replace this email address with your own.
      # Let's Encrypt will use this to contact you about expiring
      # certificates, and issues related to your account.
      email: contact@example.local
      # change this to production endpoint once you successfully test it
      # server: https://acme-v02.api.letsencrypt.org/directory
      server: https://acme-staging-v02.api.letsencrypt.org/directory
      solver: clouddns
      solvers:
        clouddns:
          # TODO: add a link about how to configure this section
          project: '[YOUR GCP PROJECT ID]'
          serviceAccountSecretRef:
            name: '[NAME OF SECRET]'
            key: '[KEY OF SECRET]'
        # route53:
        #   region: "[ROUTE53 REGION]"
        #   secretAccessKeySecretRef:
        #     name: "[NAME OF SECRET]"
        #     key: "[KEY OF SECRET]"
        #   role: "[ASSUME A ROLE]"

[1] enabled: use a public issuer.
[2] type: specify the type of the certificate issuer.
[3] acme and its sub fields: the configuration of the ACME public issuer.

Enable Istio Gateway and expose services through the Pulsar broker

This example enables the Istio Gateway, specifies values for certSecretName (the name of the Kubernetes secret that was created in the previous step), and defines the URLs for the Pulsar broker and StreamNative Console.

Configure the Istio Gateway for StreamNative Platform in the values.yaml YAML file as follows:

istio:
  enabled: true
  gateway:
    namespace: '' # --- [1]
    tls:
      mode: SIMPLE # --- [2]
      certSecretName: '[CERTIFICATE SECRET NAME IN ISTIO ROOT NAMESPACE]' # --- [3]

ingress:
  broker:
    enabled: true
    external_domain: 'httpbin.example.com' # --- [4]
  control_center:
    enabled: true
    external_domain: 'console-httpbin.example.com' # --- [5]
    external_domain_scheme: https://

[1] namespace: the default namespace for the Istio gateway. By default, it is set to istio-system.
[2] mode: the TLS mode for the Istio gateway. Available options are SIMPLE and PASSTHROUGH. By default, it is set to SIMPLE.
- SIMPLE: terminate TLS traffic at the Istio gateway. In this case, the gateway certificate is used.
- PASSTHROUGH: do not terminate TLS traffic at the Istio gateway. Instead, terminate TLS traffic at the component. In this case, the certificate that is mount to the component is used. Therefore, you need to configure the gateway TLS of the Pulsar broker in the values.yaml YAML file, as shown below.
  tls: broker: gateway: # name of chart generated certificate cert_name: tls-broker-gateway # specify name of secret contain certificate if using pre-generated certificate certSecretName: trustCertsEnabled: false

Currently, the PASSTHROUGH TLS mode is not available for the StreamNative Console. You still need to configure the certSecretName option for the StreamNative Console.

[3] certSecretName: the name of the Kubernetes Secret. It can also either be a certificate issued by the internal or the external issuer. If neither the internal nor the external issuer is configured, there should be a Secret named certSecretName that contains certificates under the Istio Gateway namespace.
[4][5] external_domain: Pulsar Broker generates the Pod domain names based on the advertisedDomain field (the domain of the Pulsar Broker). Therefore, you must include the suffix of the external domains for Pulsar Broker and control center in the domain.suffix. As shown below, if the external domains for Pulsar Broker and control center are set to broker.example.com and control_center.example.com respectively, thedomain.suffix should be set to example.com.
```
ingress:
  broker:
    external_domain: broker.example.com
  control_center:
    external_domain: control_center.example.com
domain:
  suffix: example.com
```

Enable Istio Gateway and expose service through the Pulsar Proxy

Configure the Istio Gateway for StreamNative Platform in the values.yaml YAML file as follows:
```
istio:
  enabled: true
  gateway:
    namespace: '' # --- [1]
    tls:
      certSecretName: '[CERTIFICATE SECRET NAME IN ISTIO ROOT NAMESPACE]' # --- [2]

ingress:
  proxy:
    enabled: true
    type: IstioGateway # --- [3]
    external_domain: 'proxy-httpbin.example.com' # --- [4]
  control_center:
    enabled: true
    external_domain: 'console-httpbin.example.com' # --- [5]
    external_domain_scheme: https://
```
- [1] namespace: the default namespace for the Istio gateway. By default, it is set to istio-system.
- [2] certSecretName: the name of the Kubernetes Secret. It can also either be a certificate issued by the internal or the external issuer. If neither the internal nor the external issuer is configured, there should be a Secret named certSecretName that contains certificates under the Istio Gateway namespace.
- [3] type: the method of exposing the Pulsar Proxy service. It should be set to IstioGateway when Istio is enabled.
- [4][5] external_domain: if the external domains for the Pulsar Proxy and the control center are set to proxy.example.com and control_center.example.com respectively, thedomain.suffix should be set to example.com.

Apply the new configuration.

helm upgrade -f /path/to/your/values.yaml <release_name> -n <k8s_namespace>

Access ingress services

This section describes how to access KoP, MoP, the Pulsar broker, StreamNative Console, and Grafana after installing Istio and Istio Ingress Gateway, and enabling Istio Gateway.

Access KoP

To access KoP, follow these steps.

Configure the Kafka client.

a. Create a certificate in the JKS format. This example creates a self-signed JKS certificate.

If you use a certificate signed by a public issuer, you do not need to configure the certificate on the Kafka client.

keytool -keystore truststore.jks -alias CARoot -import -file <example.com.crt> -storepass <password> -keypass <password>

b. Create the Kafka client configuration file (client.properties).

security.protocol=SSL
# You need to use the full path of client.truststore.jks
ssl.truststore.location=<./truststore.jks>
ssl.truststore.password=<password>
# The identification algorithm must be empty
ssl.endpoint.identification.algorithm=

Configure the DNS address.

a. Get the related host domains.

kubectl get vs -n <k8s_namespace>

The output looks similar to:

NAME                             GATEWAYS                             HOSTS                                              AGE
snp-sn-platform-broker           ["snp-sn-platform-broker"]           ["httpbin.example.com"]                            3h25m
snp-sn-platform-broker-0         ["snp-sn-platform-broker-0"]         ["snp-sn-platform-broker-0-httpbin.example.com"]   3h24m
snp-sn-platform-broker-1         ["snp-sn-platform-broker-1"]         ["snp-sn-platform-broker-1-httpbin.example.com"]   3h24m
snp-sn-platform-broker-2         ["snp-sn-platform-broker-2"]         ["snp-sn-platform-broker-2-httpbin.example.com"]   3h24m
snp-sn-platform-control-center   ["snp-sn-platform-control-center"]   ["console-httpbin.example.com"]                    3h24m

b. Configure the DNS address in /etc/hosts.

<your_k8s_node_IP> httpbin.example.com console-httpbin.example.com snp-sn-platform-broker-0-httpbin.example.com snp-sn-platform-broker-1-httpbin.example.com snp-sn-platform-broker-2-httpbin.example.com

Connect to KoP via the Kafka client.

a. Start a Kafka producer and send a message from the Kafka producer.

bin/kafka-console-producer.sh \
--broker-list <httpbin.example.com:9093> \
--topic <topic_name> \
--producer.config client.properties

b. Open a new terminal window and enter the Kafka Pod.

kubectl exec -it -n <k8s_namespace> kafka -- bash

c. Start a Kafka consumer.

bin/kafka-console-consumer.sh \
--bootstrap-server <httpbin.example.com:9093> \
--topic <topic_name> \
--consumer.config client.properties

Access MoP

To access MoP using the Eclipse Mosquitto MQTT CLI tool, follow these steps.

Create a Kubernetes Secret (mqtt-cert) using the CA certificate that is generated in configure TLS Ingress Gateway.

If you use a certificate signed by a public issuer, you do not need to configure the certificate on MoP.

Deploy an MQTT client Pod.

a. Define an MQTT client Pod as below and save the YAML file (pod.yaml).

apiVersion: v1
kind: Pod
metadata:
  name: mqtt
  namespace: <k8s_namespace>
  labels:
    app: mqtt
spec:
  containers:
    - name: test-mqtt
      image: eclipse-mosquitto
      imagePullPolicy: IfNotPresent
      command:
        - sleep
        - 3600s
      volumeMounts:
        - mountPath: /tmp
          name: cert
  volumes:
    - name: cert
      secret:
        defaultMode: 420
        optional: true
        secretName: mqtt-cert

b. Apply your configurations.

kubectl apply -f pod.yaml

Enter the MQTT client Pod and configure the DNS address in /etc/hosts.

# enter the pod
kubectl -n <k8s_namespace> exec -it mqtt -- sh
# config /etc/hosts
<istio_ingressgateway_server_ip> httpbin.example.com

Connect to MoP using the MQTT client.

a. Publish messages to the Pulsar cluster.

i. Open a new terminal window and enter the MQTT client Pod.
```
kubectl -n <k8s_namespace> exec -it mqtt -- sh
```
ii. Publish messages to the Pulsar cluster.
```
mosquitto_pub -h httpbin.example.com -p 8883 --cafile /tmp/ca.crt  -t test -u mqtt -P <token> -m "hello"
```
b. Subscribe to messages from the Pulsar cluster.

i. Open a new terminal window and enter the MQTT client Pod.
```
kubectl -n <k8s_namespace> exec -it mqtt -- sh
```
ii. Subscribe to messages from the Pulsar cluster.
```
mosquitto_sub -h httpbin.example.com -p 8883 --cafile /tmp/ca.crt  -t test -u mqtt -P <token>
```

Access AoP

To access AoP using the RabbitMQ client, follow these steps.

Create a Pulsar namespace and set the retention policy for the namespace.

This example shows how to use the pulsar-admin CLI tool to create a Pulsar namespace named vhost1 and set the retention size and retention time to 100M and 2 days, respectively.
```
bin/pulsar-admin namespaces create -b 1 public/vhost1
bin/pulsar-admin namespaces set-retention -s 100M -t 2d public/vhost1
```

Create a Maven project and add a dependency to the RabbitMQ client.

# add the RabbitMQ client dependency in your project
<dependency>
  <groupId>com.rabbitmq</groupId>
  <artifactId>amqp-client</artifactId>
  <version>5.8.0</version>
</dependency>

Connect to AoP using the RabbitMQ client.

import com.rabbitmq.client.AMQP;
import com.rabbitmq.client.BuiltinExchangeType;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.DefaultConsumer;
import com.rabbitmq.client.Envelope;

import java.io.IOException;
import java.time.Duration;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.CountDownLatch;

public class main {
    public static void main(String[] args) throws Exception
    {
        // create connection
        ConnectionFactory connectionFactory = new ConnectionFactory();
        connectionFactory.setVirtualHost("vhost1");                         # --- [1]
        connectionFactory.setHost("istio_ingressgateway_server_ip");        # --- [2]
        connectionFactory.setPort(5671);                                    # --- [3]
        connectionFactory.useSslProtocol();

        Connection connection = connectionFactory.newConnection();
        Channel channel = connection.createChannel();

        String exchange = "ex";
        String queue = "qu";

        // declare exchange
        channel.exchangeDeclare(exchange, BuiltinExchangeType.FANOUT, true, false, false, null);

        // queue declare and bind
        channel.queueDeclare(queue, true, false, false, null);
        channel.queueBind(queue, exchange, queue);

        // publish some messages
        for (int i = 0; i < 100; i++) {
            channel.basicPublish(exchange, "", null, ("hello - " + i).getBytes());
        }

        // consume messages
        CountDownLatch countDownLatch = new CountDownLatch(100);
        channel.basicConsume(queue, true, new DefaultConsumer(channel) {
            @Override
            public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body) throws IOException {
                System.out.println("receive msg: " + new String(body));
                countDownLatch.countDown();
            }
        });
        countDownLatch.await();

        // release resource
        channel.close();
        connection.close();
    }
}

[1] connectionFactory.setVirtualHost: represent the Pulsar namespace for AoP.
[2] connectionFactory.setHost: represent the endpoint for Pulsar service.
[3] connectionFactory.setPort: represent the port ID for AoP. It is set to port 5671.

Access a Pulsar broker

To access a Pulsar broker, follow these steps:

Create a certificate. For details, see configure TLS Ingress Gateway.

If you use a certificate signed by a public issuer, you do not need to configure the certificate on the Pulsar broker.

Update the Pulsar broker configuration file (conf/client.conf).

webServiceUrl=https://[broker_ingress_address]/
brokerServiceUrl=pulsar+ssl://[broker_ingress_address]:6651/
useTls=true
tlsTrustCertsFilePath=<./example.com.crt>

Validate the configuration by producing and consuming data with the pulsar-client CLI tool.

Access the StreamNative Console

Create a certificate. For details, see configure TLS Ingress Gateway.

If you use a certificate signed by a public issuer, you do not need to configure the certificate on the StreamNative Console.

Verify whether you can connect to the StreamNative Console using the self-signed certificate.
```
curl --cacert <example.com.crt> https://console-httpbin.example.com:31897/
```

Access Grafana

Create a certificate. For details, see configure TLS Ingress Gateway.

If you use a certificate signed by a public issuer, you do not need to configure the certificate on Grafana.

Verify whether you can connect to the Grafana service using the self-signed certificate.
```
curl --cacert <example.com.crt> https://console-httpbin.example.com:31897/grafana
```

Overview

Quick Start

Streamnative Console

Clients

Connectors

Monitor

References

Operating Streamnative Platform

Tools

Prerequisites

Limitations

Install Istio

Install Istio Ingress Gateway

Enable Istio Gateway

Configure TLS Ingress Gateway

`openssl`

cert-manager

Enable Istio Gateway and expose services through the Pulsar broker

Enable Istio Gateway and expose service through the Pulsar Proxy

Access ingress services

Access KoP

Access MoP

Access AoP

Access a Pulsar broker

Access the StreamNative Console

Access Grafana

Overview

Quick Start

Streamnative Console

Clients

Connectors

Monitor

References

Operating Streamnative Platform

Tools

​Prerequisites

​Limitations

​Install Istio

​Install Istio Ingress Gateway

​Enable Istio Gateway

​Configure TLS Ingress Gateway

​openssl

​cert-manager

​Enable Istio Gateway and expose services through the Pulsar broker

​Enable Istio Gateway and expose service through the Pulsar Proxy

​Access ingress services

​Access KoP

​Access MoP

​Access AoP

​Access a Pulsar broker

​Access the StreamNative Console

​Access Grafana

Prerequisites

Limitations

Install Istio

Install Istio Ingress Gateway

Enable Istio Gateway

Configure TLS Ingress Gateway

`openssl`

cert-manager

Enable Istio Gateway and expose services through the Pulsar broker

Enable Istio Gateway and expose service through the Pulsar Proxy

Access ingress services

Access KoP

Access MoP

Access AoP

Access a Pulsar broker

Access the StreamNative Console

Access Grafana