> ## Documentation Index > Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt > Use this file to discover all available pages before exploring further. # Configure custom OIDC issuers for API key authentication Extend [API key authentication](/private-cloud/v2/configure-private-cloud/security/authentication/private-cloud-api-key-auth) to tokens from a user-supplied OIDC issuer. Use the API key authentication provider, `io.streamnative.pulsar.broker.authentication.AuthenticationProviderApiKeys`, with an OpenID Connect (OIDC) issuer to authenticate signed JWT tokens, including Kubernetes ServiceAccount tokens. Use `io.streamnative.pulsar.broker.authorization.AuthorizationProviderApiKeys` for authorization. Configure the issuer, expected audience, role claim, and permissions separately. ## Before you begin * Upgrade to StreamNative Operator Helm chart **0.20.6** (Operator **v0.20.18**) and update its CRDs. This release preserves `clientAuth` when `authentication` is omitted. See [Upgrade StreamNative Operator](/private-cloud/v2/operate-private-cloud/upgrade/upgrade-sn-operator). * Use a Pulsar image that includes the API key authentication and authorization providers. The Kubernetes discovery limitation described below was observed with Private Cloud Pulsar **4.2.1.17**. * Obtain your issuer's HTTPS URL. For EKS, this is the cluster's OIDC issuer, not its Kubernetes API URL. * Choose an audience for the target Pulsar cluster. The examples use `pulsar-cluster-a`. * Decide which ServiceAccounts can administer the cluster and which receive application permissions. * Use TLS endpoints when sending credentials across a network you do not control. Enabling OIDC does not enable server TLS. ## Configure the API key provider The typed `spec.config.authentication.apiKey.oidcIssuers` field selects `AuthenticationProviderApiKeys`. A Kubernetes projected token is validated as a signed JWT; it does not need to be created by an API key service. Choose the revocation configuration based on whether you deploy the API key service: * **With the API key service:** No additional revocation setting is required. The Operator configures the service's revocation-list URL and permits authentication if the list has not initialized. Once loaded, the list applies across the provider: tokens without a `jti` skip the lookup, and tokens with a `jti` are rejected if that ID is revoked. Add the custom issuer to `oidcIssuers` on the broker and proxy; entries merge by `issuerUrl` with the service's issuer. Preserve the service's internal client credentials and authorization settings. Do not set `PULSAR_PREFIX_skipRevocationList: "true"`, because it disables revocation checks for the service's API keys too. * **Without the API key service or another revocation list:** Set `spec.config.custom.PULSAR_PREFIX_skipRevocationList: "true"`, as shown below. This disables revocation-list loading and checks for the provider, while preserving JWT signature, issuer, audience, and expiration validation. Use short-lived tokens. Remove this override before adding an API key service whose keys require revocation checks. The typed setting `spec.config.authentication.apiKey.skipRevocationListValidationOnRevocationListInitFailure: true` is an alternative when a revocation list is expected but might be temporarily unavailable. It permits authentication before the list initializes and enforces revocation once a list loads. It does not stop the loader: without a configured endpoint, the provider keeps trying `http://localhost:8081/v1/revocation-list` and logging `Failed to load revocation list`. Use `skipRevocationList` for a deployment that has no revocation list. The following example configures a deployment **without the API key service**. Merge it into an existing `PulsarBroker`. Replace the issuer and administrative ServiceAccount names. Keep the cluster's existing storage, networking, and Pod configuration. For EKS, also configure the local JWKS Secret described in [Use an EKS issuer](#use-an-eks-issuer) before starting the broker. ```yaml theme={null} spec: config: authentication: apiKey: oidcIssuers: - issuerUrl: https:/// audience: pulsar-cluster-a subjectClaim: sub authzRoleClaim: sub authzAdminRole: system:serviceaccount:pulsar:pulsar-admin authorization: superUserRoles: system:serviceaccount:pulsar:pulsar-broker,system:serviceaccount:pulsar:pulsar-admin custom: # Only when no API key service or other revocation list is configured. PULSAR_PREFIX_skipRevocationList: "true" PULSAR_PREFIX_authorizationProvider: io.streamnative.pulsar.broker.authorization.AuthorizationProviderApiKeys clientAuth: generic: authPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken authParams: file:///mnt/oidc/token apiObjects: statefulSet: volumeMounts: - name: oidc-token mountPath: /mnt/oidc readOnly: true pod: serviceAccountName: pulsar-broker volumes: - name: oidc-token projected: sources: - serviceAccountToken: audience: pulsar-cluster-a expirationSeconds: 3600 path: token ``` The example uses `sub` for the Pulsar role and the authorization role claim. `authzAdminRole` restricts issuer-based administrator access to the `pulsar-admin` subject. The explicit `superUserRoles` entries also grant administrative access to the broker and administrator. For another identity provider, select the claims and administrator value that match its tokens. Configure these authorization claim settings for each trusted issuer. Create the `pulsar-broker` and `pulsar-admin` ServiceAccounts in the Pulsar namespace if they do not exist. The broker's projected token authenticates its internal client as `system:serviceaccount:pulsar:pulsar-broker`. For a `PulsarProxy` without the API key service or another revocation list, configure the same API key provider, issuer, audience, role claim, and `PULSAR_PREFIX_skipRevocationList: "true"`. Use its own ServiceAccount and projected token for `clientAuth`. Add `PULSAR_PREFIX_forwardAuthorizationCredentials: "true"` to the proxy's custom configuration. Add the proxy ServiceAccount role to the broker's `spec.config.authorization.proxyRoles`, and grant the administrative permissions required by your proxy setup. Configure ordinary application permissions through [Grant permissions](/private-cloud/v2/configure-private-cloud/security/access-control/grant-permissions). ## Use an EKS issuer In 4.2.1.17, the API key provider rejects Kubernetes discovery metadata containing: ```text theme={null} authorization_endpoint: urn:kubernetes:programmatic_authorization ``` Startup fails with `Attribute 'authorization_endpoint' has invalid value`. To use this provider with EKS, supply the issuer's public JSON Web Key Set (JWKS) locally. This discovery workaround is separate from revocation configuration. EKS issuers can coexist with the API key service while its revocation checks remain enabled. Upgrading only the Operator does not fix the discovery parser. For an AWS EKS OIDC provider, directly mount the issuer's public keys as a `jwks.json` file. Reference a Secret containing this file through `publicKeySecretRef`. 1. Download the public JWKS from the HTTPS `jwks_uri` in the issuer's `/.well-known/openid-configuration` document and save it as `jwks.json`. Verify that the document's `issuer` matches your expected issuer URL, and retain TLS certificate verification when downloading both files. Create a Secret containing the public JWKS: ```bash theme={null} kubectl -n pulsar create secret generic eks-oidc-jwks \ --from-file=jwks.json ``` 2. Merge these settings into the broker. The example assumes no API key service. With the service deployed, add only the custom `oidcIssuers` entry and preserve the existing revocation, client authentication, and authorization settings. Without the service, configure `clientAuth` and the projected token as shown in the preceding example. ```yaml theme={null} spec: config: authentication: apiKey: oidcIssuers: - issuerUrl: https://oidc.eks..amazonaws.com/id/ audience: pulsar-cluster-a subjectClaim: sub authzRoleClaim: sub authzAdminRole: system:serviceaccount:pulsar:pulsar-admin publicKeySecretRef: name: eks-oidc-jwks key: jwks.json authorization: superUserRoles: system:serviceaccount:pulsar:pulsar-broker,system:serviceaccount:pulsar:pulsar-admin custom: # Only when no API key service or other revocation list is configured. PULSAR_PREFIX_skipRevocationList: "true" PULSAR_PREFIX_authorizationProvider: io.streamnative.pulsar.broker.authorization.AuthorizationProviderApiKeys ``` The Operator mounts the JWKS and configures its file path. Apply the equivalent issuer settings to any proxy that authenticates these tokens. Refresh the Secret when the issuer rotates its signing keys, and verify that the provider has loaded the new keys before retiring old ones. A local JWKS does not automatically track the issuer's remote key rotation. When the API key service is deployed, keep its revocation checks enabled for both EKS and other custom issuers. A custom issuer does not need its own revocation-list endpoint. Remove `PULSAR_PREFIX_skipRevocationList` from the standalone configuration before using it with the service. ## Generate a token and connect a client The following commands use Kubernetes TokenRequest. Your Kubernetes identity must have permission to create tokens for the selected ServiceAccount. 1. Create a short-lived administrative token in a private temporary file: ```bash theme={null} umask 077 TOKEN_FILE="$(mktemp)" kubectl -n pulsar create token pulsar-admin \ --audience=pulsar-cluster-a --duration=10m > "$TOKEN_FILE" ``` The JWT contains `iss`, `aud`, `sub`, and expiration claims. For this ServiceAccount, `sub` is `system:serviceaccount:pulsar:pulsar-admin`. Inspect claims locally without publishing the bearer token. Use a projected token volume for a long-running Kubernetes workload; a TokenRequest file created once does not refresh itself. 2. Check an authenticated HTTP endpoint. Set `WEB_URL` to your broker or proxy URL, and run the command from a host that can reach it: ```bash theme={null} WEB_URL="http://:8080" curl --fail --show-error \ -H "Authorization: Bearer $(cat "$TOKEN_FILE")" \ "$WEB_URL/admin/v2/clusters" ``` Use `https://...:8443` and the appropriate CA file when server TLS is enabled. HTTP 200 confirms this request succeeded. Also check a protected administrative operation and application permissions; listing clusters alone does not prove a role is a superuser. 3. Produce messages with the Java CLI. Use an existing tenant and namespace for which the ServiceAccount role has permissions: ```bash theme={null} bin/pulsar-client \ --url pulsar://:6650 \ --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \ --auth-params "file://$TOKEN_FILE" \ produce persistent:////oidc-check -m hello -n 3 ``` In a second terminal, use the same connection options and replace the `produce` command with `consume persistent:////oidc-check -s oidc-check -n 3`. Start the consumer before producing. For TLS, use the TLS service URL and CA settings described in [mTLS authentication](/private-cloud/v2/configure-private-cloud/security/authentication/private-cloud-mtls-auth#connect-a-client), while retaining the token authentication plugin. 4. Verify rejected requests with no token, an invalid signature, and a token requested for a different audience. Expect HTTP 401 from the broker and proxy for these cases. Also test a valid application token whose subject is absent from `superUserRoles` and does not match `authzAdminRole`: a protected administrative operation must be denied. Check that valid credentials produce and consume the expected messages. Remove the temporary token file when you finish the test. Its contents are bearer credentials until the token expires. ## Trust multiple issuers The API key provider returns the configured role claim as a role string. The issuer is not automatically included in that role. If two trusted issuers assert the same `sub`, the resulting role can match the same global `superUserRoles` or role permissions. `AuthorizationProviderApiKeys` also supports issuer-specific authorization claims, such as `authzRoleClaim` and `authzAdminRole`; configure these for each issuer. Audience validation rejects an existing token whose audience belongs to another cluster. It does not stop another trusted issuer from signing a new token with the target audience and the same privileged subject. Trust an additional issuer only if its issuance policies can enforce the required audience and role boundaries, or use a role mapping and authorization design that separates issuers. See [OIDC issuer boundaries](/private-cloud/v2/configure-private-cloud/security/access-control/grant-permissions#oidc-issuer-boundaries). ## Troubleshoot authentication | Symptom | Check | | - | - | | Broker fails to start with an invalid `authorization_endpoint` URN | The API key provider cannot parse this Kubernetes discovery document; use the local JWKS workaround. | | All API key tokens fail with provider-not-ready errors | With the API key service, check its revocation-list URL and availability. Without the service or another revocation list, set `spec.config.custom.PULSAR_PREFIX_skipRevocationList: "true"`. | | Authentication succeeds, but logs repeatedly report `Failed to load revocation list` | The initialization-failure setting allows authentication but keeps retrying the list. For a deployment with no revocation list, set `PULSAR_PREFIX_skipRevocationList: "true"`. With the API key service, fix the revocation-list URL or service availability and keep revocation checks enabled. | | Certificate or token volume exists, but internal clients have no authentication configuration | Upgrade to Helm chart 0.20.6 (Operator v0.20.18) and its CRDs; inspect generated `brokerClientAuthenticationPlugin` and parameters. | | Validly signed token is rejected | Verify the exact issuer, audience, expiry, and configured role claim. | | Token is rejected with `Unable to resolve key with id` | Check whether the token's `kid` is present in the mounted JWKS. Download the issuer's current public keys, update the Secret referenced by `publicKeySecretRef`, and verify that the provider loads them. A local JWKS does not automatically follow signing-key rotation. | | Another issuer's identity receives an existing role's permissions | Check shared role strings and issuer issuance policies; audience validation does not namespace roles. | Configure detector authentication separately. Its JSON file parameter is different from the Java client's `file:///...` parameter, and requires a compatible detector image. See [Pulsar Detector](/private-cloud/v2/operate-private-cloud/observability/private-cloud-detector). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.