Scrape Metrics from the Local Metrics Endpoint

The Local Metrics Endpoint is a per-cluster Prometheus endpoint that serves every metric your cluster collects, including the system-level broker, bookie, ZooKeeper, proxy, and Pulsar Functions metrics that the Metrics API does not expose. You scrape it from your own observability stack: Prometheus, the OpenTelemetry Collector, Grafana Agent, or any Prometheus-compatible collector.

The Local Metrics Endpoint is only available for BYOC Pro clusters. To enable it, see Enable the Local Metrics Endpoint.

The Metrics API accepts OAuth2 client credentials. The Local Metrics Endpoint accepts API keys only, in a custom x-jwt-Authorization header. If you copy an existing OAuth2 scrape configuration to this endpoint, every request returns 403 Forbidden.

Prerequisites

A BYOC Pro cluster with the Local Metrics Endpoint enabled, and its endpoint URL. The URL has the form https://metric-<id>.<your-domain>, and there is one per cluster.
A Super Admin service account that is authorized for the endpoint. To authorize an additional service account, submit a request through the support portal.
An API key for that service account.

Authenticate

Send the API key in the x-jwt-Authorization header, prefixed with ApiKey and a single space:

x-jwt-Authorization: ApiKey <your-api-key>

The standard Authorization: Bearer header is not used by this endpoint. Verify your credentials with an instant query:

curl --fail-with-body --show-error --get \
  -H "x-jwt-Authorization: ApiKey ${PULSAR_API_KEY}" \
  --data-urlencode 'query=up' \
  "https://metric-<id>.<your-domain>/api/v1/query"

Choose an endpoint path

Path	Use
`/federate`	Scrape metrics in bulk into your own Prometheus or collector. Use this for continuous collection.
`/api/v1/query`, `/api/v1/query_range`	Run ad-hoc queries and build dashboards.
`/metrics`	Not available.

The Local Metrics Endpoint blocks the /metrics path. Requests to it return 403 Forbidden no matter which credentials you send. Collect metrics from /federate instead.

The Local Metrics Endpoint serves the standard Prometheus federation API. Federation requires at least one match[] selector, so to collect everything the endpoint exposes, select every series:

GET /federate?match[]={__name__=~".+"}

Narrowing the selector to a single name prefix, such as {__name__=~"pulsar_.*"}, also drops the BookKeeper, ZooKeeper, JVM, and node metrics, because those series do not share that prefix. Select every series unless you have a specific reason to collect less.

Configure your collector

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: "pulsar-local-metrics"
          scrape_interval: 30s
          honor_labels: true
          metrics_path: /federate
          params:
            'match[]':
              - '{__name__=~".+"}'
          scheme: https
          http_headers:
            x-jwt-Authorization:
              secrets:
                - "ApiKey ${PULSAR_API_KEY}"
          static_configs:
            - targets: ["metric-<id-us-east-1>.<your-domain>"]
              labels:
                region: "us-east-1"
            - targets: ["metric-<id-us-west-2>.<your-domain>"]
              labels:
                region: "us-west-2"

scrape_configs:
  - job_name: pulsar-local-metrics
    scrape_interval: 30s
    honor_labels: true
    metrics_path: /federate
    params:
      'match[]':
        - '{__name__=~".+"}'
    scheme: https
    http_headers:
      x-jwt-Authorization:
        files:
          - /etc/prometheus/secrets/local-metrics-header
    static_configs:
      - targets: ["metric-<id>.<your-domain>"]

Set honor_labels: true so that federation preserves the original instance, job, and Pulsar labels instead of replacing them with the labels of the federation target. Your collector must be able to set a custom request header:

In the OpenTelemetry Collector, the Prometheus receiver reads the value from the environment, so ${PULSAR_API_KEY} resolves at startup.
In Prometheus, http_headers takes the header value from a file (files) or inline (values); it does not expand environment variables. The file must contain the complete header value, ApiKey <your-api-key>.
The authorization setting does not work with this endpoint, because it writes the standard Authorization header.

Each cluster has its own endpoint URL. List every cluster as a target in the same scrape job and add a label, such as region, to keep the sources distinguishable in your backend.

Set your scrape interval

The cluster Prometheus retains data for 30 minutes. It is a federation source, not long-term storage, so:

Scrape every 15 to 60 seconds and retain the data in your own backend.
If your collector stops for longer than 30 minutes, the metrics for that period are no longer available from the endpoint.

Metric names

The Local Metrics Endpoint serves Apache Pulsar’s own metric names. For the complete list of broker, BookKeeper, ZooKeeper, and Pulsar Functions metrics, see the Pulsar metrics reference. The endpoint also serves Kubernetes-level metrics for the nodes that run your cluster, which is why a query such as up returns Kubernetes jobs alongside Pulsar jobs.

Troubleshoot a 403 Forbidden response

Check each of the following:

Check	Expected
Header name	`x-jwt-Authorization`, not `Authorization`
Header value	The `ApiKey` prefix, one space, then the key
Credential type	An API key, not an OAuth2 access token
Key state	Not expired, and issued in the correct organization
Service account	Authorized for the Local Metrics Endpoint on this cluster
Hostname	The endpoint URL of the cluster you intend to scrape
Path	`/federate` or `/api/v1/...`. The `/metrics` path is blocked and always returns `403`.

If every check passes and the endpoint still returns 403, submit a request through the support portal with the request time, the endpoint hostname, and the service account name.

BYOC

Manage Instances

Security

Networking

Logging & Monitoring

Billing

Tools

Terraform

References

Scrape Metrics from the Local Metrics Endpoint

Prerequisites

Authenticate

Choose an endpoint path

Configure your collector

Set your scrape interval

Metric names

Troubleshoot a 403 Forbidden response

​Prerequisites

​Authenticate

​Choose an endpoint path

​Configure your collector

​Set your scrape interval

​Metric names

​Troubleshoot a 403 Forbidden response

​Related topics

Prerequisites

Authenticate

Choose an endpoint path

Configure your collector

Set your scrape interval

Metric names

Troubleshoot a 403 Forbidden response

Related topics