StreamNative Pulsar extends the JWT token authentication to support multiple private keys AuthenticationProvider AuthenticationProviderMultipleSignKeyToken, and implement automatic reloading of configuration files into the cache without restarting the broker. The multiple multiple private keys will be maintained in a JSON file and issue new JWT token needs to include a kid field in the header to match the private key for signature verification.

​
Before you begin

  • Install the following tools.

​
Create multiple secret keys

  • Create the default secret key 
    pulsarctl token create-secret-key -a HS256 --output-file default-secret.key

cat default-secret.key | base64
GRgrvdFdNijm0xPompgfJGwE77ifM+U31exWERDEiB0=
  • Create the second secret key 
    pulsarctl token create-secret-key -a HS256 --output-file kid-secret.key

cat kid-secret.key| base64
IeNNHJmTM6icg71/D6aFQ/buslNV+FZWT/TiJRDZpYo=
  • Create a JSON to include the multiple secret keys, key is kid, value is the corresponding secret key encoded by base64 
    {
  "KID1": "Cysl6yGZb6xFygc0hB5Pt+ha1ZXpt7jmFU3CR1AS2os=",
  "DEFAULT_KID": "nVXSRdlQm4wbgxiS4/MHjxgLqyL9iF2oEXdq9E6iIeo="
}
  • Create the K8s Secret for the multi secret key JSON 
    kubectl create secret generic multi-secret-key --from-file=secret-keys.json

​
Create broker and proxy tokens

  • Use the default secret key to generate broker admin token 
    pulsarctl token create -a HS256 --secret-key-file default-secret.key --subject broker-admin
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJicm9rZXItYWRtaW4ifQ.bi1Bx9Wg-4HOA-xvAPJQQbf-J70u359TL_8_GqSdC6E
  • Create the K8s Secret for the tokens 
    kubectl create secret generic broker-admin --from-literal=broker-admin=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJicm9rZXItYWRtaW4ifQ.bi1Bx9Wg-4HOA-xvAPJQQbf-J70u359TL_8_GqSdC6E

    kubectl create secret generic proxy-admin --from-literal=proxy-admin=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJwcm94eS1hZG1pbiJ9.ELqIKJlb0ISShCTBmhV0MenPbBLG0xkRk1mrfbnWT8s

​
Configure PulsarBroker and PulsarProxy

​
PulsarBroker

spec:
  image: streamnative/private-cloud:3.3.0.7
  replicas: 1
  zkServers: private-cloud-zk:2181
  config:
    clusterName: private-cloud
    custom:
      managedLedgerDefaultAckQuorum: '1'
      managedLedgerDefaultEnsembleSize: '1'
      managedLedgerDefaultWriteQuorum: '1'
      authenticationEnabled: 'true'
      authenticateOriginalAuthData: 'true'
      authenticationProviders: 'io.streamnative.pulsar.broker.authentication.AuthenticationProviderMultipleSignKeyToken'
      brokerClientAuthenticationPlugin: 'org.apache.pulsar.client.impl.auth.AuthenticationToken'
      superUserRoles: 'admin,proxy-admin,broker-admin,admin-approle'
      proxyRoles: 'proxy-admin'
      authorizationEnabled: 'true'
      authorizationProvider: 'org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider'
  pod:
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
    securityContext:
      runAsNonRoot: true
    secretRefs:
      - mountPath: /mnt/secrets
        secretName: multi-secret-key
      - mountPath: /mnt/tokens
        secretName: broker-admin
    vars:
      - name: brokerClientAuthenticationParameters
        value: 'file:///mnt/tokens/broker-admin'
      - name: tokenSecretKey
        value: 'file:///mnt/secrets/secret-keys.json'
  • [1] spec.config.cusom: authenticationProviders should use the io.streamnative.pulsar.broker.authentication.AuthenticationProviderMultipleSignKeyToken

​
PulsarProxy

spec:
  image: streamnative/private-cloud:3.3.0.7
  replicas: 1
  config:
    custom:
      authenticationEnabled: 'true'
      authenticateOriginalAuthData: 'true'
      forwardAuthorizationCredentials: 'true'
      authenticationProviders: 'io.streamnative.pulsar.broker.authentication.AuthenticationProviderMultipleSignKeyToken'
      brokerClientAuthenticationPlugin: 'org.apache.pulsar.client.impl.auth.AuthenticationToken'
      superUserRoles: 'proxy-admin'
  brokerAddress: private-cloud-broker
  pod:
    annotations:
      prometheus.io/port: '8080'
      prometheus.io/scrape: 'true'
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
    securityContext:
      runAsNonRoot: true
    secretRefs:
      - mountPath: /mnt/secrets
        secretName: multi-secret-key
      - mountPath: /mnt/tokens
        secretName: proxy-admin
    vars:
      - name: brokerClientAuthenticationParameters
        value: 'file:///mnt/tokens/proxy-admin'
      - name: tokenSecretKey
        value: 'file:///mnt/secrets/secret-keys.json'

​
Create client tokens

  • Create a token by the default secret key 
    pulsarctl token create -a HS256 --secret-key-file default-secret.key --subject test-client
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.BC8hq47exYyCirBCP5AlIDzFQtnxhCPrv94BHeQ08to
  • Create a token by the kid secret key
The kid=KID1 in the headers should be consistent with the secret-keys.json defination.
pulsarctl token create -a HS256 --secret-key-file kid-secret.key --subject test-client --headers kid=KID1
eyJhbGciOiJIUzI1NiIsImtpZCI6ImtpZDEiLCJ0eXAiOiJKV1QifQ.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.y5qXOgb_ibTQD09nVaS6sr4bGI8caByY_FnJXuhq4_4
  • Grant permission to the role 
    pulsarctl --token "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJicm9rZXItYWRtaW4ifQ.bi1Bx9Wg-4HOA-xvAPJQQbf-J70u359TL_8_GqSdC6E" \
namespaces grant-permission --role test-client --actions produce --actions consume public/default
  • Use the token generated by the default secret key to produce data 
    bin/pulsar-client --auth-params token:eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.BC8hq47exYyCirBCP5AlIDzFQtnxhCPrv94BHeQ08to --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken produce default-secret-token -m "test-with-multisecretKey" -n 15

2024-05-29T21:46:20,293+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650] Connected through proxy to target broker at private-cloud-broker-0.private-cloud-broker-headless.default.svc.cluster.local:6650
2024-05-29T21:46:20,854+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [null] Creating producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650]
2024-05-29T21:46:21,263+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Created producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650]
2024-05-29T21:46:25,671+0800 [main] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [default-secret-token] [private-cloud-0-3] --- Publish throughput: 2.66 msg/s --- 0.00 Mbit/s --- Latency: med: 267.000 ms - 95pct: 441.000 ms - 99pct: 441.000 ms - 99.9pct: 441.000 ms - max: 441.000 ms --- BatchSize: med: 1.000 - 95pct: 1.000 - 99pct: 1.000 - 99.9pct: 1.000 - max: 1.000 --- MsgSize: med: 24.000 bytes - 95pct: 24.000 bytes - 99pct: 24.000 bytes - 99.9pct: 24.000 bytes - max: 24.000 bytes --- Ack received rate: 2.66 ack/s --- Failed messages: 0 --- Pending messages: 0
2024-05-29T21:46:25,928+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Closed Producer
2024-05-29T21:46:25,930+0800 [main] INFO  org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://172.171.186.162:6650/
2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 ! R:/172.171.186.162:6650] Disconnected
2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-3] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xe6914d5b, L:/198.18.0.1:62075 ! R:/172.171.186.162:6650] Disconnected
2024-05-29T21:46:27,989+0800 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 15 messages successfully produced
  • Use the token generated by the KID1 secret key to produce data 
    bin/pulsar-client --auth-params token:eyJhbGciOiJIUzI1NiIsImtpZCI6IktJRDEiLCJ0eXAiOiJKV1QifQ.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.YbsyE54ulm9ioMx2xZBqLgCNrYmjk2o6wxR5WOwaPSE --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken produce test -m "test-with-multisecretKey" -n 15

2024-05-29T21:46:20,293+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650] Connected through proxy to target broker at private-cloud-broker-0.private-cloud-broker-headless.default.svc.cluster.local:6650
2024-05-29T21:46:20,854+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [null] Creating producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650]
2024-05-29T21:46:21,263+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Created producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650]
2024-05-29T21:46:25,671+0800 [main] INFO  org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [default-secret-token] [private-cloud-0-3] --- Publish throughput: 2.66 msg/s --- 0.00 Mbit/s --- Latency: med: 267.000 ms - 95pct: 441.000 ms - 99pct: 441.000 ms - 99.9pct: 441.000 ms - max: 441.000 ms --- BatchSize: med: 1.000 - 95pct: 1.000 - 99pct: 1.000 - 99.9pct: 1.000 - max: 1.000 --- MsgSize: med: 24.000 bytes - 95pct: 24.000 bytes - 99pct: 24.000 bytes - 99.9pct: 24.000 bytes - max: 24.000 bytes --- Ack received rate: 2.66 ack/s --- Failed messages: 0 --- Pending messages: 0
2024-05-29T21:46:25,928+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Closed Producer
2024-05-29T21:46:25,930+0800 [main] INFO  org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://172.171.186.162:6650/
2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-5] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 ! R:/172.171.186.162:6650] Disconnected
2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-3] INFO  org.apache.pulsar.client.impl.ClientCnx - [id: 0xe6914d5b, L:/198.18.0.1:62075 ! R:/172.171.186.162:6650] Disconnected
2024-05-29T21:46:27,989+0800 [main] INFO  org.apache.pulsar.client.cli.PulsarClientTool - 15 messages successfully produced