Advanced Observability Integration

Note

The advanced observability features, including the Local Metrics Endpoint and Remote Write Integration, are only available for BYOC Pro clusters. Use the Metrics API for other clusters, including Serverless, Dedicated, and BYOC.

In addition to the Metrics API, StreamNative Cloud offers advanced observability features to help you gain deeper insights into your Pulsar clusters and applications. This document covers two key aspects of advanced observability:

Local Metrics Endpoint: A local Prometheus endpoint that is deployed with your BYOC Pro cluster. It provides access to all available metrics from your Pulsar clusters including both resource-related metrics and system-level metrics.
Remote Write Integration: Capabilities for seamlessly forwarding metrics to external monitoring and analytics platforms.

Local Metrics Endpoint

The Metrics API can be used for monitoring and troubleshooting your business applications. You can get the metrics from your Pulsar clusters, including tenants/namespaces/topics, connectors, functions, and so on. These metrics focus on the Pulsar resources inside your cluster. Metrics API excludes system-level metrics, such as the broker, bookie, and Pulsar Functions worker metrics. StreamNative manages these system-level metrics for you to ensure the stability and reliability of your Pulsar clusters.

If you need to access these system-level metrics, you can use the Local Metrics Endpoint. The Local Metrics Endpoint provides access to all available metrics from your Pulsar clusters, including the system-level metrics.

Note

The Local Metrics Endpoint is only available for BYOC Pro clusters.

Enable the Local Metrics Endpoint

The Local Metrics Endpoint can be enabled on a per-cluster basis. Submit a Change Request in our support portal to enable the Local Metrics Endpoint for your cluster.

After the request is approved and the Local Metrics Endpoint is enabled, you will be provided with a Local Metrics Endpoint URL. Use this URL to access the metrics data, similar to the existing Metrics API.

Remote Write Integration

Note

If you do not see your Observability system in the list below, please reach out to support and we can evaluate your request

In addition to pulling metrics data from the metrics endpoints using Metrics API, you can also push metrics data to your observability platform using Remote Write.

StreamNative Cloud supports Remote Write with Prometheus-compatible systems. This includes, but is not limited to:

Datadog
Grafana Cloud (Grafana Labs)
New Relic
Victoria Metrics
Prometheus
Thanos
Cortex

For authentication, StreamNative Cloud currently supports the following methods for Remote Write:

Basic Authentication
OAuth2

How to enable the Remote Write Integration

To enable the Remote Write Integration:

Submit a Change Request in our support portal.
Specify your desired Prometheus-compatible system and authentication method.
Provide any necessary credentials or configuration details for the chosen system.

Our team will review your request and assist you in setting up the integration.