> ## Documentation Index > Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt > Use this file to discover all available pages before exploring further. # Multiple private keys support for JWT token authentication StreamNative Pulsar extends the JWT token authentication to support multiple private keys AuthenticationProvider `AuthenticationProviderMultipleSignKeyToken`, and implement automatic reloading of configuration files into the cache without restarting the broker. The multiple multiple private keys will be maintained in a JSON file and issue new JWT token needs to include a kid field in the header to match the private key for signature verification. ## Before you begin * Install the following tools. * Install [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) v1.16 or higher. * Install [pulsarctl](https://github.com/streamnative/pulsarctl). ## Create multiple secret keys * Create the default secret key ```bash theme={null} pulsarctl token create-secret-key -a HS256 --output-file default-secret.key cat default-secret.key | base64 GRgrvdFdNijm0xPompgfJGwE77ifM+U31exWERDEiB0= ``` * Create the second secret key ```bash theme={null} pulsarctl token create-secret-key -a HS256 --output-file kid-secret.key cat kid-secret.key| base64 IeNNHJmTM6icg71/D6aFQ/buslNV+FZWT/TiJRDZpYo= ``` * Create a JSON to include the multiple secret keys, key is kid, value is the corresponding secret key encoded by base64 ```json theme={null} { "KID1": "Cysl6yGZb6xFygc0hB5Pt+ha1ZXpt7jmFU3CR1AS2os=", "DEFAULT_KID": "nVXSRdlQm4wbgxiS4/MHjxgLqyL9iF2oEXdq9E6iIeo=" } ``` * Create the K8s Secret for the multi secret key JSON ```bash theme={null} kubectl create secret generic multi-secret-key --from-file=secret-keys.json ``` ## Create broker and proxy tokens * Use the default secret key to generate broker admin token ```bash theme={null} pulsarctl token create -a HS256 --secret-key-file default-secret.key --subject broker-admin eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJicm9rZXItYWRtaW4ifQ.bi1Bx9Wg-4HOA-xvAPJQQbf-J70u359TL_8_GqSdC6E ``` * Create the K8s Secret for the tokens ```bash theme={null} kubectl create secret generic broker-admin --from-literal=broker-admin=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJicm9rZXItYWRtaW4ifQ.bi1Bx9Wg-4HOA-xvAPJQQbf-J70u359TL_8_GqSdC6E ``` ```bash theme={null} kubectl create secret generic proxy-admin --from-literal=proxy-admin=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJwcm94eS1hZG1pbiJ9.ELqIKJlb0ISShCTBmhV0MenPbBLG0xkRk1mrfbnWT8s ``` ## Configure PulsarBroker and PulsarProxy ### PulsarBroker ```yaml theme={null} spec: image: streamnative/private-cloud:3.3.0.7 replicas: 1 zkServers: private-cloud-zk:2181 config: clusterName: private-cloud custom: managedLedgerDefaultAckQuorum: '1' managedLedgerDefaultEnsembleSize: '1' managedLedgerDefaultWriteQuorum: '1' authenticationEnabled: 'true' authenticateOriginalAuthData: 'true' authenticationProviders: 'io.streamnative.pulsar.broker.authentication.AuthenticationProviderMultipleSignKeyToken' brokerClientAuthenticationPlugin: 'org.apache.pulsar.client.impl.auth.AuthenticationToken' superUserRoles: 'admin,proxy-admin,broker-admin,admin-approle' proxyRoles: 'proxy-admin' authorizationEnabled: 'true' authorizationProvider: 'org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider' pod: resources: requests: cpu: 200m memory: 512Mi securityContext: runAsNonRoot: true secretRefs: - mountPath: /mnt/secrets secretName: multi-secret-key - mountPath: /mnt/tokens secretName: broker-admin vars: - name: brokerClientAuthenticationParameters value: 'file:///mnt/tokens/broker-admin' - name: tokenSecretKey value: 'file:///mnt/secrets/secret-keys.json' ``` * \[1] `spec.config.cusom`: `authenticationProviders` should use the `io.streamnative.pulsar.broker.authentication.AuthenticationProviderMultipleSignKeyToken` ### PulsarProxy ```yaml theme={null} spec: image: streamnative/private-cloud:3.3.0.7 replicas: 1 config: custom: authenticationEnabled: 'true' authenticateOriginalAuthData: 'true' forwardAuthorizationCredentials: 'true' authenticationProviders: 'io.streamnative.pulsar.broker.authentication.AuthenticationProviderMultipleSignKeyToken' brokerClientAuthenticationPlugin: 'org.apache.pulsar.client.impl.auth.AuthenticationToken' superUserRoles: 'proxy-admin' brokerAddress: private-cloud-broker pod: annotations: prometheus.io/port: '8080' prometheus.io/scrape: 'true' resources: requests: cpu: 200m memory: 512Mi securityContext: runAsNonRoot: true secretRefs: - mountPath: /mnt/secrets secretName: multi-secret-key - mountPath: /mnt/tokens secretName: proxy-admin vars: - name: brokerClientAuthenticationParameters value: 'file:///mnt/tokens/proxy-admin' - name: tokenSecretKey value: 'file:///mnt/secrets/secret-keys.json' ``` ### Create client tokens * Create a token by the default secret key ```bash theme={null} pulsarctl token create -a HS256 --secret-key-file default-secret.key --subject test-client eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.BC8hq47exYyCirBCP5AlIDzFQtnxhCPrv94BHeQ08to ``` * Create a token by the kid secret key The `kid=KID1` in the headers should be consistent with the `secret-keys.json` defination. ```bash theme={null} pulsarctl token create -a HS256 --secret-key-file kid-secret.key --subject test-client --headers kid=KID1 eyJhbGciOiJIUzI1NiIsImtpZCI6ImtpZDEiLCJ0eXAiOiJKV1QifQ.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.y5qXOgb_ibTQD09nVaS6sr4bGI8caByY_FnJXuhq4_4 ``` * Grant permission to the role ```bash theme={null} pulsarctl --token "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJicm9rZXItYWRtaW4ifQ.bi1Bx9Wg-4HOA-xvAPJQQbf-J70u359TL_8_GqSdC6E" \ namespaces grant-permission --role test-client --actions produce --actions consume public/default ``` * Use the token generated by the default secret key to produce data ```bash theme={null} bin/pulsar-client --auth-params token:eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.BC8hq47exYyCirBCP5AlIDzFQtnxhCPrv94BHeQ08to --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken produce default-secret-token -m "test-with-multisecretKey" -n 15 2024-05-29T21:46:20,293+0800 [pulsar-client-io-1-5] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650] Connected through proxy to target broker at private-cloud-broker-0.private-cloud-broker-headless.default.svc.cluster.local:6650 2024-05-29T21:46:20,854+0800 [pulsar-client-io-1-5] INFO org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [null] Creating producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650] 2024-05-29T21:46:21,263+0800 [pulsar-client-io-1-5] INFO org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Created producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650] 2024-05-29T21:46:25,671+0800 [main] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [default-secret-token] [private-cloud-0-3] --- Publish throughput: 2.66 msg/s --- 0.00 Mbit/s --- Latency: med: 267.000 ms - 95pct: 441.000 ms - 99pct: 441.000 ms - 99.9pct: 441.000 ms - max: 441.000 ms --- BatchSize: med: 1.000 - 95pct: 1.000 - 99pct: 1.000 - 99.9pct: 1.000 - max: 1.000 --- MsgSize: med: 24.000 bytes - 95pct: 24.000 bytes - 99pct: 24.000 bytes - 99.9pct: 24.000 bytes - max: 24.000 bytes --- Ack received rate: 2.66 ack/s --- Failed messages: 0 --- Pending messages: 0 2024-05-29T21:46:25,928+0800 [pulsar-client-io-1-5] INFO org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Closed Producer 2024-05-29T21:46:25,930+0800 [main] INFO org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://172.171.186.162:6650/ 2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-5] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 ! R:/172.171.186.162:6650] Disconnected 2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-3] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xe6914d5b, L:/198.18.0.1:62075 ! R:/172.171.186.162:6650] Disconnected 2024-05-29T21:46:27,989+0800 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 15 messages successfully produced ``` * Use the token generated by the KID1 secret key to produce data ```bash theme={null} bin/pulsar-client --auth-params token:eyJhbGciOiJIUzI1NiIsImtpZCI6IktJRDEiLCJ0eXAiOiJKV1QifQ.eyJzdWIiOiJ0ZXN0LWNsaWVudCJ9.YbsyE54ulm9ioMx2xZBqLgCNrYmjk2o6wxR5WOwaPSE --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken produce test -m "test-with-multisecretKey" -n 15 2024-05-29T21:46:20,293+0800 [pulsar-client-io-1-5] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650] Connected through proxy to target broker at private-cloud-broker-0.private-cloud-broker-headless.default.svc.cluster.local:6650 2024-05-29T21:46:20,854+0800 [pulsar-client-io-1-5] INFO org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [null] Creating producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650] 2024-05-29T21:46:21,263+0800 [pulsar-client-io-1-5] INFO org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Created producer on cnx [id: 0xc1116ccc, L:/198.18.0.1:62077 - R:/172.171.186.162:6650] 2024-05-29T21:46:25,671+0800 [main] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - [default-secret-token] [private-cloud-0-3] --- Publish throughput: 2.66 msg/s --- 0.00 Mbit/s --- Latency: med: 267.000 ms - 95pct: 441.000 ms - 99pct: 441.000 ms - 99.9pct: 441.000 ms - max: 441.000 ms --- BatchSize: med: 1.000 - 95pct: 1.000 - 99pct: 1.000 - 99.9pct: 1.000 - max: 1.000 --- MsgSize: med: 24.000 bytes - 95pct: 24.000 bytes - 99pct: 24.000 bytes - 99.9pct: 24.000 bytes - max: 24.000 bytes --- Ack received rate: 2.66 ack/s --- Failed messages: 0 --- Pending messages: 0 2024-05-29T21:46:25,928+0800 [pulsar-client-io-1-5] INFO org.apache.pulsar.client.impl.ProducerImpl - [default-secret-token] [private-cloud-0-3] Closed Producer 2024-05-29T21:46:25,930+0800 [main] INFO org.apache.pulsar.client.impl.PulsarClientImpl - Client closing. URL: pulsar://172.171.186.162:6650/ 2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-5] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xc1116ccc, L:/198.18.0.1:62077 ! R:/172.171.186.162:6650] Disconnected 2024-05-29T21:46:25,935+0800 [pulsar-client-io-1-3] INFO org.apache.pulsar.client.impl.ClientCnx - [id: 0xe6914d5b, L:/198.18.0.1:62075 ! R:/172.171.186.162:6650] Disconnected 2024-05-29T21:46:27,989+0800 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 15 messages successfully produced ```