Enable TLS on StreamNative Platform components

StreamNative Platform supports Transport Layer Security (TLS), an industry-standard encryption protocol, to protect network communications of StreamNative Platform components. TLS relies on keys and certificates to establish trusted connections.

This document describes how to enable TLS encryption on StreamNative Platform components (such as the Pulsar proxy and the Pulsar broker) and the Kafka on Pulsar (KoP) protocol handler.

StreamNative Platform supports enabling TLS encryption through either auto-generated certificates or manually-generated certificates.

Auto-generated certificates: cert-manager automatically generates the certificates and attaches them to Kubernetes Pods.
Manually-generated certificates: you generate the private key, public key, and Certificate Authority (CA).

For scenarios where you do not need to use your own server certificates, it is recommended to use the auto-generated certificates.

Enable TLS encryption with cert-manager

cert-manager adds certificates and certificate issuers as resource types in Kubernetes clusters, and simplifies the process of obtaining, renewing, and using those certificates.

In StreamNative Platform, cert-manager supports issuing certificates from an internal issuer or a public issuer. This section describes how to generate certificates with an internal issuer and a public issuer, and then enable TLS encryption.

To generate TLS certificates and enable TLS encryption with an internal issuer, complete the following steps.

Create a Kubernetes Secret for the JKS certificate.

kubectl create secret generic cert-jks-passwd --from-literal=password=passwd -n <k8s_namespace>

Enable TLS encryption on StreamNative Platform components.
```
tls:
  enabled: true # --- [1]
  <components>: # --- [2]
    enabled: true # --- [3]

certs:
  internal_issuer:
    enabled: true # --- [4]
    component: internal-cert-issuer
    type: selfsigning # --- [5]

  issuers: # --- [6]
    selfsigning:
```
- [1] tls.enabled: enable TLS encryption to protect network communications of StreamNative Platform components.
- [2] tls.<components>: specify the StreamNative Platform components (such as the Pulsar proxy and Pulsar broker) where you can enable TLS encryption.
- [3] tls.<components>.enabled: enable TLS encryption on StreamNative Platform components.
  Note
  When both the broker TLS and gateway TLS (Istio PASSTHROUGH mode) are enabled, only the gateway TLS configuration (tls.broker.gateway) takes effect. If you want these two TLS modes to work together, you need to manually specify the internal and external domain names (dnsNames) in one certificate.
- [4] certs.internal_issuer.enabled: use an internal issuer to sign certificates.
- [5] certs.internal_issuer.type: specify the type of the internal issuer.
  - selfsigning: the selfsigning issuer generates a CA based on an automatically-generated secret. You can use the private key of the certificate to sign the certificate itself.
  - secret: the secret issuer represents a CA whose certificate and private key are stored inside the cluster as a Kubernetes Secret. Then, you can use the Kubernetes Secret to sign a certificate.
```
certs:
  internal_issuer:
    enabled: true
    type: secret

  issuers:
    secret:
      secretName: ca-keypair
  ...
  # other configs
```
  - custom: you can specify an internal issuer through this option. For example, you can specify using Vault to sign certificates for your PKI, as shown below.
```
certs:
  internal_issuer:
    enabled: true
    type: custom
  issuers:
    custom:
      path: pki_int/sign/example-dot-com
      server: https://vault.local
      caBundle: <base64 encoded CA Bundle PEM file>
  ...
  # other configs
```
- [6] issuers and its sub fields: the configuration of the internal issuer.
  - selfsigning: the selfsigning issuer has no dependency on any other resource. Therefore, you do not need to configure any item for this issuer.
  - secret: secretName is the name of the Kubernetes Secret that is automatically created and managed by the CA. It is populated with a private key and certificate, signed by the issuer.
  - custom: you can configure items for custom certificate issuers. For more detailed configurations, see the cert-manager documentation.

Apply the new configuration.

helm upgrade -f /path/to/your/file.yaml <release_name> streamnative/sn-platform -n <k8s_namespace>

Download TLS certificates to your local directory. The ca.crt is used to trust the Pulsar proxy server, and the truststore.jks is used for the Kafka client to trust the KoP server.

kubectl get secret <release_name>-tls-proxy -o=jsonpath='{.data.ca\.crt}' -n <k8s_namespace> | base64 --decode -o ca.crt
kubectl get secret <release_name>-tls-proxy -o=jsonpath='{.data.truststore\.jks}' -n <k8s_namespace> | base64 --decode -o truststore.jks

Enable TLS encryption with manually-generated certificates

This section describes how to enable TLS encryption on StreamNative components and KoP with manually-generated certificates.

Enable TLS encryption on StreamNative Platform components

To enable TLS encryption on StreamNative components with manually-generated certificates, you need to create a CA certificate, generate a server certificate, and then enable TLS encryption.

Create a CA certificate

You can use a CA to sign certificates for both the Pulsar broker and Pulsar clients. The CA certificate consists of the root key (ca.key.pem) and root certificate (ca.cert.pem). This ensures that each party trusts the other. You should store the CA certificate in a secure location (ideally completely disconnected from networks, air gapped, and fully encrypted).

Create a directory for the CA, and place the OpenSSL configuration file in the directory.
Tip
If you need to modify the default values for the company name and department in the configuration file, you can export the location of the CA directory to the environment variable CA_HOME. The configuration file uses this environment variable to find the files and directories for the CA.
```
mkdir my-ca
cd my-ca
wget https://raw.githubusercontent.com/apache/pulsar/master/site2/website/static/examples/openssl.cnf
export CA_HOME=$(pwd)
```

Create the necessary directories and generate the CA private key.

mkdir certs crl newcerts private
chmod 700 private/
touch index.txt
echo 1000 > serial
# Generate the certificate authority private key.
openssl genrsa -aes256 -out private/ca.key.pem 4096
# Restrict Access to the certificate authority private key.
chmod 400 private/ca.key.pem

Create the X.509 certificate.

openssl req -config openssl.cnf -key private/ca.key.pem \
  -new -x509 -days 7300 -sha256 -extensions v3_ca \
  -out certs/ca.cert.pem
# Restrict Access to the X.509 certificate.
chmod 444 certs/ca.cert.pem

Answer the prompts.
CA-related files are stored in the ./my-ca directory. At this point, the script has generated a private key (ca.key.pem) and root certificate (ca.cert.pem) for our internal CA.
- private/ca.key.pem: the private key. You only need the key when you are signing a new certificate for either a broker or client. You must safely guard this private key.
- certs/ca.cert.pem: the public certificate that is distributed to all parties involved.

Generate a server certificate

Now that you have created a CA certificate, you can generate a certificate for the Pulsar broker and sign it with our internal CA.

Tip

The common name should match the broker. You can also use a wildcard to match a group of broker hostnames, for example, *.broker.usw.example.com. This ensures that multiple machines can reuse the same certificate. However, when it is impossible to match the hostname, you can disable TLS hostname verification for the client.

To create a server certificate, follow these steps.

Generate the server certificate private key.

openssl genrsa -out broker.key.pem 2048

The key is in PKCS 8 format. You can convert it to PEM format.

openssl pkcs8 -topk8 -inform PEM -outform PEM \
  -in broker.key.pem -out broker.key-pk8.pem -nocrypt

Generate the server certificate request.

openssl req -config openssl.cnf \
  -key broker.key.pem -new -sha256 -out broker.csr.pem

Sign the server certificate with the CA.

openssl ca -config openssl.cnf -extensions server_cert \
  -days 1000 -notext -md sha256 \
  -in broker.csr.pem -out broker.cert.pem

At this point, you have a broker certificate (broker.cert.pem) and its associated private key (broker.key-pk8.pem) that you can use along with the root certificate ca.cert.pem to configure TLS encryption for your broker and proxy nodes.

Enable TLS encryption

To enable TLS encryption on StreamNative components, follow these steps.

Create TLS certification files. For details, see create a CA certificate and create a server certificate.

Create a Kubernetes Secret.

kubectl create secret generic <release_name>-sn-platform-proxy-tls -n <k8s_namespace> \
  --from-file=tls.crt=$(PWD)/cert/broker.cert.pem \
  --from-file=tls.key=$(PWD)/cert/broker.key-pk8.pem \
  --from-file=ca.crt=$(PWD)/cert/ca.cert.pem

Enable TLS encryption on StreamNative Platform components.
```
tls:
  enabled: true # --- [1]
  <components>: # --- [2]
    enabled: true # --- [3]
    certSecretName: '<release_name>-sn-platform-proxy-tls' # --- [4]
```
- [1] tls.enabled: enable TLS encryption to protect network communications of StreamNative Platform components.
- [2] tls.<components>: specify the StreamNative Platform components (such as the Pulsar proxy and Pulsar broker) where you can enable TLS encryption.
- [3] tls.<components>.enabled: enable TLS encryption on StreamNative Platform components.
- [4] tls.<components>.certSecretName: specify the name of the secret that contains the certificates. This is obtained from the previous step.

Apply the new configuration.

helm upgrade -f /path/to/your/file.yaml <release_name> streamnative/sn-platform -n <k8s_namespace>

Enable TLS encryption on KoP

To enable TLS encryption on KoP with a manually-generated certificate, you need to first generate the SSL key and certificate, and then create a Kubernetes Secret for the KoP certificate and password.

Generate a SSL key and certificate

To generate a SSL key and certificate, follow these steps.

Generate the SSL key and certificate.
```
keytool -keystore server.keystore.jks -alias localhost -validity <validity> -keyalg RSA -genkey
```
- keystore: the file that stores the certificate. The keystore contains the private key of the certificate and should be kept in a secure place.
- validity: Days before the certificate expires.
Generate the Certificate Authority (CA).
a. Generate a CA.
A CA is a public-private key pair and certificate used to sign other certificates.
```
openssl req -new -x509 -keyout ca-key -out ca-cert -days <validity>
```
b. Add the generated CA to the broker's truststore so that the brokers can trust this CA.
```
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
```
c. Add the generated CA to the client's truststore.
```
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
```

Sign a certificate.

a. Export the certificate from the keystore.

keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file

b. Sign the certificate with the CA.

openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days <validity> -CAcreateserial -passin pass:<ca-password>

c. Import both the certificate of the CA and the signed certificate into the broker keystore.

keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed

At this point, you have the following files:

keystore: the file that stores the certificate
ca-cert: the certificate of the CA
ca-key: the private key of the CA
ca-password: the passphrase of the CA
cert-file: the exported, unsigned certificate of the server
cert-signed: the signed certificate of the server

Create Kubernetes Secrets for the KoP certificate and password

Kubernetes Secrets store and manage sensitive information, such as passwords, tokens, and keys. Storing confidential information in a Secret is safer and more flexible than putting it verbatim in a Pod definition or in a container image.

Create a Kubernetes Secret for the KoP certificate.

kubectl create secret generic kop-secret -n <k8s_namespace> --from-file=keystore.jks=$(PWD)/cert/server.keystore.jks --from-file=truststore.jks=$(PWD)/cert/server.truststore.jks

Create a Kubernetes Secret for the KoP certification password.

kubectl create secret generic kop-keystore-password --from-literal=password=<ca-password> -n <k8s_namespace>

Enable TLS encryption

To enable KoP and TLS encryption, follow these steps.

Set the KoP parameters in the Pulsar configuration YAML file.

broker:
  # The domain name for accessing KoP outside from the Kubernetes cluster.
  advertisedDomain: 'kop.sn.dev'
  kop:
    enabled: true # --- [1]
    tls:
      enabled: true # --- [2]
      # create a secret with keystore.jks and truststore.jks for KoP TLS security
      certSecretName: 'kop-secret' # --- [3]
      # create a secret for keystore and truststore cert
      passwordSecretRef:
        key: password
        name: kop-keystore-password # --- [4]

[1] broker.kop.enabled: enable KoP on StreamNative Platform.
[2] broker.kop.tls.enabled: enable TLS encryption on KoP.
[3] broker.kop.tls.certSecretName: specify the Kubernetes Secret that contains the KoP certificate.
[4] broker.kop.tls.passwordSecretRef.name: specify the Kubernetes Secret that contains the KoP certificate password.

Apply the new configuration.

helm upgrade -f /path/to/your/file.yaml <release_name> streamnative/sn-platform -n <k8s_namespace>

Enable mTLS with Istio

Mutual TLS (mTLS) authentication is a way to encrypt services traffic using certificates.

With Istio, you can enforce mTLS automatically. This works because Istio control plane mounts client certificates into the sidecar proxies for you. Therefore, Pods can authenticate with each other. For details about how to enable mTLS with Istio, see configure Istio.