Multiple private keys support for JWT token authentication

StreamNative Pulsar extends the JWT token authentication to support multiple private keys AuthenticationProvider AuthenticationProviderMultipleSignKeyToken, and implement automatic reloading of configuration files into the cache without restarting the broker. The multiple multiple private keys will be maintained in a JSON file and issue new JWT token needs to include a kid field in the header to match the private key for signature verification.

Before you begin

Install the following tools.
- Install kubectl v1.16 or higher.
- Install pulsarctl.

Create multiple secret keys

Create the default secret key

pulsarctl token create-secret-key -a HS256 --output-file default-secret.key

cat default-secret.key | base64
GRgrvdFdNijm0xPompgfJGwE77ifM+U31exWERDEiB0=

Create the second secret key

pulsarctl token create-secret-key -a HS256 --output-file kid-secret.key

cat kid-secret.key| base64
IeNNHJmTM6icg71/D6aFQ/buslNV+FZWT/TiJRDZpYo=

Create a JSON to include the multiple secret keys, key is kid, value is the corresponding secret key encoded by base64

{
  "KID1": "Cysl6yGZb6xFygc0hB5Pt+ha1ZXpt7jmFU3CR1AS2os=",
  "DEFAULT_KID": "nVXSRdlQm4wbgxiS4/MHjxgLqyL9iF2oEXdq9E6iIeo="
}

Create the K8s Secret for the multi secret key JSON

kubectl create secret generic multi-secret-key --from-file=secret-keys.json

Create broker and proxy tokens

Use the default secret key to generate broker admin token

pulsarctl token create -a HS256 --secret-key-file default-secret.key --subject broker-admin
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJicm9rZXItYWRtaW4ifQ.bi1Bx9Wg-4HOA-xvAPJQQbf-J70u359TL_8_GqSdC6E

Create the K8s Secret for the tokens

kubectl create secret generic broker-admin --from-literal=broker-admin=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJicm9rZXItYWRtaW4ifQ.bi1Bx9Wg-4HOA-xvAPJQQbf-J70u359TL_8_GqSdC6E

kubectl create secret generic proxy-admin --from-literal=proxy-admin=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJwcm94eS1hZG1pbiJ9.ELqIKJlb0ISShCTBmhV0MenPbBLG0xkRk1mrfbnWT8s

Configure PulsarBroker and PulsarProxy

PulsarBroker

spec:
  image: streamnative/private-cloud:3.3.0.7
  replicas: 1
  zkServers: private-cloud-zk:2181
  config:
    clusterName: private-cloud
    custom:
      managedLedgerDefaultAckQuorum: '1'
      managedLedgerDefaultEnsembleSize: '1'
      managedLedgerDefaultWriteQuorum: '1'
      authenticationEnabled: 'true'
      authenticateOriginalAuthData: 'true'
      authenticationProviders: 'io.streamnative.pulsar.broker.authentication.AuthenticationProviderMultipleSignKeyToken'
      brokerClientAuthenticationPlugin: 'org.apache.pulsar.client.impl.auth.AuthenticationToken'
      superUserRoles: 'admin,proxy-admin,broker-admin,admin-approle'
      proxyRoles: 'proxy-admin'
      authorizationEnabled: 'true'
      authorizationProvider: 'org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider'
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
    securityContext:
      runAsNonRoot: true
    secretRefs:
      - mountPath: /mnt/secrets
        secretName: multi-secret-key
      - mountPath: /mnt/tokens
        secretName: broker-admin
    vars:
      - name: brokerClientAuthenticationParameters
        value: 'file:///mnt/tokens/broker-admin'
      - name: tokenSecretKey
        value: 'file:///mnt/secrets/secret-keys.json'

[1] spec.config.cusom: authenticationProviders should use the io.streamnative.pulsar.broker.authentication.AuthenticationProviderMultipleSignKeyToken

PulsarProxy

spec:
  image: streamnative/private-cloud:3.3.0.7
  replicas: 1
  config:
    custom:
      authenticationEnabled: 'true'
      authenticateOriginalAuthData: 'true'
      forwardAuthorizationCredentials: 'true'
      authenticationProviders: 'io.streamnative.pulsar.broker.authentication.AuthenticationProviderMultipleSignKeyToken'
      brokerClientAuthenticationPlugin: 'org.apache.pulsar.client.impl.auth.AuthenticationToken'
      superUserRoles: 'proxy-admin'
  brokerAddress: private-cloud-broker
  pod:
    annotations:
      prometheus.io/port: '8080'
      prometheus.io/scrape: 'true'
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
    securityContext:
      runAsNonRoot: true
    secretRefs:
      - mountPath: /mnt/secrets
        secretName: multi-secret-key
      - mountPath: /mnt/tokens
        secretName: proxy-admin
    vars:
      - name: brokerClientAuthenticationParameters
        value: 'file:///mnt/tokens/proxy-admin'
      - name: tokenSecretKey
        value: 'file:///mnt/secrets/secret-keys.json'

Create client tokens

Create a token by the default secret key

pulsarctl token create -a HS256 --secret-key-file default-secret.key --subject test-client
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.BC8hq47exYyCirBCP5AlIDzFQtnxhCPrv94BHeQ08to

Create a token by the kid secret key

The kid=KID1 in the headers should be consistent with the secret-keys.json defination.

pulsarctl token create -a HS256 --secret-key-file kid-secret.key --subject test-client --headers kid=KID1
eyJhbGciOiJIUzI1NiIsImtpZCI6ImtpZDEiLCJ0eXAiOiJKV1QifQ.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.y5qXOgb_ibTQD09nVaS6sr4bGI8caByY_FnJXuhq4_4

Grant permission to the role

pulsarctl --token "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJicm9rZXItYWRtaW4ifQ.bi1Bx9Wg-4HOA-xvAPJQQbf-J70u359TL_8_GqSdC6E" \
namespaces grant-permission --role test-client --actions produce --actions consume public/default

Use the token generated by the default secret key to produce data

bin/pulsar-client --auth-params token:eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.BC8hq47exYyCirBCP5AlIDzFQtnxhCPrv94BHeQ08to --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken produce default-secret-token -m "test-with-multisecretKey" -n 15

2024-05-29T21:46:20,293+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650] Connected through proxy to target broker at private-cloud-broker-0.private-cloud-broker-headless.default.svc.cluster.local:6650
2024-05-29T21:46:20,854+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [null] Creating producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650]
2024-05-29T21:46:21,263+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Created producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650]
2024-05-29T21:46:25,671+0800 [main] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [default-secret-token] [private-cloud-0-3] --- Publish throughput: 2.66 msg/s --- 0.00 Mbit/s --- Latency: med: 267.000 ms - 95pct: 441.000 ms - 99pct: 441.000 ms - 99.9pct: 441.000 ms - max: 441.000 ms --- BatchSize: med: 1.000 - 95pct: 1.000 - 99pct: 1.000 - 99.9pct: 1.000 - max: 1.000 --- MsgSize: med: 24.000 bytes - 95pct: 24.000 bytes - 99pct: 24.000 bytes - 99.9pct: 24.000 bytes - max: 24.000 bytes --- Ack received rate: 2.66 ack/s --- Failed messages: 0 --- Pending messages: 0
2024-05-29T21:46:25,928+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Closed Producer
2024-05-29T21:46:25,930+0800 [main] INFO  org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://172.171.186.162:6650/
2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 ! R:/172.171.186.162:6650] Disconnected
2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-3] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xe6914d5b, L:/198.18.0.1:62075 ! R:/172.171.186.162:6650] Disconnected
2024-05-29T21:46:27,989+0800 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 15 messages successfully produced

Use the token generated by the KID1 secret key to produce data

bin/pulsar-client --auth-params token:eyJhbGciOiJIUzI1NiIsImtpZCI6IktJRDEiLCJ0eXAiOiJKV1QifQ.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.YbsyE54ulm9ioMx2xZBqLgCNrYmjk2o6wxR5WOwaPSE --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken produce test -m "test-with-multisecretKey" -n 15

2024-05-29T21:46:20,293+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650] Connected through proxy to target broker at private-cloud-broker-0.private-cloud-broker-headless.default.svc.cluster.local:6650
2024-05-29T21:46:20,854+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [null] Creating producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650]
2024-05-29T21:46:21,263+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Created producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650]
2024-05-29T21:46:25,671+0800 [main] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [default-secret-token] [private-cloud-0-3] --- Publish throughput: 2.66 msg/s --- 0.00 Mbit/s --- Latency: med: 267.000 ms - 95pct: 441.000 ms - 99pct: 441.000 ms - 99.9pct: 441.000 ms - max: 441.000 ms --- BatchSize: med: 1.000 - 95pct: 1.000 - 99pct: 1.000 - 99.9pct: 1.000 - max: 1.000 --- MsgSize: med: 24.000 bytes - 95pct: 24.000 bytes - 99pct: 24.000 bytes - 99.9pct: 24.000 bytes - max: 24.000 bytes --- Ack received rate: 2.66 ack/s --- Failed messages: 0 --- Pending messages: 0
2024-05-29T21:46:25,928+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Closed Producer
2024-05-29T21:46:25,930+0800 [main] INFO  org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://172.171.186.162:6650/
2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 ! R:/172.171.186.162:6650] Disconnected
2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-3] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xe6914d5b, L:/198.18.0.1:62075 ! R:/172.171.186.162:6650] Disconnected
2024-05-29T21:46:27,989+0800 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 15 messages successfully produced

Overview

Quick Start

Configure Private Cloud

Operate Private Cloud

References

Multiple private keys support for JWT token authentication

Before you begin

Create multiple secret keys

Create broker and proxy tokens

Configure PulsarBroker and PulsarProxy

PulsarBroker

PulsarProxy

Create client tokens

​Before you begin

​Create multiple secret keys

​Create broker and proxy tokens

​Configure PulsarBroker and PulsarProxy

​PulsarBroker

​PulsarProxy

​Create client tokens

Before you begin

Create multiple secret keys

Create broker and proxy tokens

Configure PulsarBroker and PulsarProxy

PulsarBroker

PulsarProxy

Create client tokens