> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape Metrics from the Local Metrics Endpoint

The Local Metrics Endpoint is a per-cluster Prometheus endpoint that serves every metric your
cluster collects, including the system-level broker, bookie, ZooKeeper, proxy, and Pulsar Functions
metrics that the [Metrics API](/cloud/log-and-monitor/cloud-metrics-api) does not expose. You scrape
it from your own observability stack: Prometheus, the OpenTelemetry Collector, Grafana Agent, or any
Prometheus-compatible collector.

<Note title="Note">
  The Local Metrics Endpoint is only available for [**BYOC Pro**](/cloud/clusters/cluster-types#byoc-pro-clusters) clusters. To enable it, see [Enable the Local Metrics Endpoint](/cloud/log-and-monitor/advanced-observability#local-metrics-endpoint).
</Note>

<Warning title="The Local Metrics Endpoint and the Metrics API authenticate differently">
  The [Metrics API](/cloud/log-and-monitor/cloud-metrics-api) accepts OAuth2 client credentials. The
  Local Metrics Endpoint accepts **API keys only**, in a custom `x-jwt-Authorization` header. If you
  copy an existing OAuth2 scrape configuration to this endpoint, every request returns `403 Forbidden`.
</Warning>

## Prerequisites

* A BYOC Pro cluster with the Local Metrics Endpoint enabled, and its endpoint URL. The URL has the
  form `https://metric-<id>.<your-domain>`, and there is one per cluster.
* A [Super Admin service account](/cloud/security/authentication/service-accounts/service-accounts#create-a-service-account)
  that is authorized for the endpoint. To authorize an additional service account, submit a request
  through the [support portal](https://support.streamnative.cloud/).
* An [API key](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview) for
  that service account.

## Authenticate

Send the API key in the `x-jwt-Authorization` header, prefixed with `ApiKey` and a single space:

```
x-jwt-Authorization: ApiKey <your-api-key>
```

The standard `Authorization: Bearer` header is not used by this endpoint.

Verify your credentials with an instant query:

```bash theme={null}
curl --fail-with-body --show-error --get \
  -H "x-jwt-Authorization: ApiKey ${PULSAR_API_KEY}" \
  --data-urlencode 'query=up' \
  "https://metric-<id>.<your-domain>/api/v1/query"
```

## Choose an endpoint path

| Path                                   | Use                                                                                               |
| -------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `/federate`                            | Scrape metrics in bulk into your own Prometheus or collector. Use this for continuous collection. |
| `/api/v1/query`, `/api/v1/query_range` | Run ad-hoc queries and build dashboards.                                                          |
| `/metrics`                             | Not available.                                                                                    |

<Warning title="/metrics is not available">
  The Local Metrics Endpoint blocks the `/metrics` path. Requests to it return `403 Forbidden` no matter
  which credentials you send. Collect metrics from `/federate` instead.
</Warning>

The Local Metrics Endpoint serves the standard
[Prometheus federation](https://prometheus.io/docs/prometheus/latest/federation/) API. Federation
requires at least one `match[]` selector, so to collect everything the endpoint exposes, select every
series:

```
GET /federate?match[]={__name__=~".+"}
```

<Note title="Note">
  Narrowing the selector to a single name prefix, such as `{__name__=~"pulsar_.*"}`, also drops the
  BookKeeper, ZooKeeper, JVM, and node metrics, because those series do not share that prefix. Select
  every series unless you have a specific reason to collect less.
</Note>

## Configure your collector

<CodeGroup>
  ```yaml OpenTelemetry Collector theme={null}
  receivers:
    prometheus:
      config:
        scrape_configs:
          - job_name: "pulsar-local-metrics"
            scrape_interval: 30s
            honor_labels: true
            metrics_path: /federate
            params:
              'match[]':
                - '{__name__=~".+"}'
            scheme: https
            http_headers:
              x-jwt-Authorization:
                secrets:
                  - "ApiKey ${PULSAR_API_KEY}"
            static_configs:
              - targets: ["metric-<id-us-east-1>.<your-domain>"]
                labels:
                  region: "us-east-1"
              - targets: ["metric-<id-us-west-2>.<your-domain>"]
                labels:
                  region: "us-west-2"
  ```

  ```yaml Prometheus theme={null}
  scrape_configs:
    - job_name: pulsar-local-metrics
      scrape_interval: 30s
      honor_labels: true
      metrics_path: /federate
      params:
        'match[]':
          - '{__name__=~".+"}'
      scheme: https
      http_headers:
        x-jwt-Authorization:
          files:
            - /etc/prometheus/secrets/local-metrics-header
      static_configs:
        - targets: ["metric-<id>.<your-domain>"]
  ```
</CodeGroup>

Set `honor_labels: true` so that federation preserves the original `instance`, `job`, and Pulsar
labels instead of replacing them with the labels of the federation target.

Your collector must be able to set a custom request header:

* In the OpenTelemetry Collector, the Prometheus receiver reads the value from the environment, so
  `${PULSAR_API_KEY}` resolves at startup.
* In Prometheus, `http_headers` takes the header value from a file (`files`) or inline (`values`);
  it does not expand environment variables. The file must contain the complete header value,
  `ApiKey <your-api-key>`.
* The `authorization` setting does not work with this endpoint, because it writes the standard
  `Authorization` header.

<Tip title="Scrape more than one cluster">
  Each cluster has its own endpoint URL. List every cluster as a target in the same scrape job and add
  a label, such as `region`, to keep the sources distinguishable in your backend.
</Tip>

## Set your scrape interval

The cluster Prometheus retains data for **30 minutes**. It is a federation source, not long-term
storage, so:

* Scrape every 15 to 60 seconds and retain the data in your own backend.
* If your collector stops for longer than 30 minutes, the metrics for that period are no longer
  available from the endpoint.

## Metric names

The Local Metrics Endpoint serves Apache Pulsar's own metric names. For the complete list of broker,
BookKeeper, ZooKeeper, and Pulsar Functions metrics, see the
[Pulsar metrics reference](https://pulsar.apache.org/docs/next/reference-metrics/).

The endpoint also serves Kubernetes-level metrics for the nodes that run your cluster, which is why
a query such as `up` returns Kubernetes jobs alongside Pulsar jobs.

## Troubleshoot a 403 Forbidden response

Check each of the following:

| Check           | Expected                                                                               |
| --------------- | -------------------------------------------------------------------------------------- |
| Header name     | `x-jwt-Authorization`, not `Authorization`                                             |
| Header value    | The `ApiKey` prefix, one space, then the key                                           |
| Credential type | An API key, not an OAuth2 access token                                                 |
| Key state       | Not expired, and issued in the correct organization                                    |
| Service account | Authorized for the Local Metrics Endpoint on this cluster                              |
| Hostname        | The endpoint URL of the cluster you intend to scrape                                   |
| Path            | `/federate` or `/api/v1/...`. The `/metrics` path is blocked and always returns `403`. |

If every check passes and the endpoint still returns `403`, submit a request through the
[support portal](https://support.streamnative.cloud/) with the request time, the endpoint hostname,
and the service account name.

## Related topics

* [Advanced Observability Integration](/cloud/log-and-monitor/advanced-observability)
* [Cluster Metrics (Metrics API)](/cloud/log-and-monitor/cloud-metrics-api)
* [Use API Keys to Authenticate to StreamNative Cloud](/cloud/security/authentication/service-accounts/use-api-keys/api-keys-overview)