This feature is currently in private preview. If you want to try it out or have any questions, submit a ticket to the support team.
Introduction
Monitoring Lakehouse Storage in a private cloud setup is crucial for ensuring the efficiency and performance of your data offloading processes. By enabling the embedded stat serviceembeddedStatServiceEnabled and configuring the metric port statServicePort in the offload.conf file, you can access detailed insights into the offload framework’s operations.
Components of Tiered Storage Metrics
-
Service Stats:
- Provides resource usage statistics at the offload service level.
-
Offload Stats:
- Focuses on the offload framework’s performance for data offloading, including namespace, topic, and partition labels.
-
Container Stats:
- Offers insights into message container activity, aiding in optimizing data flushing to the Lakehouse storage.
-
Write Stats:
- Monitors the encoding and writing processes to the Lakehouse storage, measuring latency and throughput.
-
Read Stats:
- Tracks the reading process from the Lakehouse storage, including cache hits, misses, and entry retrieval metrics.
Service Stats Metrics
| Metric Name | Type | Description |
|---|---|---|
| offload_service_IN_USAGE_CONTAINERS_COUNT | Gauge | Number of allocated message containers |
| offload_service_IDLE_CONTAINERS_COUNT | Gauge | Number of idle message containers |
| offload_service_LEDGER_DELETION_LATENCY | Summary | Latency for ledger deletions |
| offload_service_PARALLEL_READ_CONTAINERS | Gauge | Number of parallel read containers |
| offload_service_OFFLOADING_TOPICS_COUNT | Gauge | Number of parallel offloading topics |
| offload_service_READ_CACHE_SIZE | Gauge | Size of the read cache in bytes |
| offload_service_READ_CACHE_COUNT | Gauge | Entry count in the read cache |
Offload Stats Metrics
| Metric Name | Type | Description |
|---|---|---|
| offload_framework_WAITING_PROMISE | Gauge | Number of waiting offload promises |
| offload_framework_MESSAGE_OUT_OF_ORDER_COUNT | Counter | Count of out-of-order messages |
| offload_framework_MESSAGE_PUT_INTO_CONTAINER_FAILED_COUNT | Counter | Count of failed message placements into containers |
| offload_framework_MESSAGE_PUT_INTO_CONTAINER_RETRY_COUNT | Counter | Count of retried message placements into containers |
| offload_framework_FETCH_LEDGER_METADATA_LATENCY | Summary | Latency for fetching ledger metadata |
| offload_framework_FETCH_TOPIC_SCHEMA_LATENCY | Summary | Latency for fetching topic schemas |
| offload_framework_ACKNOWLEDGE_FAILED_COUNT | Counter | Count of failed message acknowledgments |
Container Stats Metrics
| Metric Name | Type | Description |
|---|---|---|
| offload_container_SWITCH_COUNT | Counter | Container switch count |
| offload_container_CURRENT_LEDGERID | Gauge | Current container’s ledger ID |
| offload_container_BYTES_IN | Counter | Bytes put into message containers |
| offload_container_MESSAGE_IN | Counter | Number of messages put into containers |
| offload_container_MESSAGE_COUNT_IN_LAST_CONTAINER | Gauge | Message count in the last container |
| offload_container_BYTES_IN_LAST_CONTAINER | Gauge | Bytes in the last container |
| offload_container_ALLOCATED_COUNT | Gauge | Number of allocated containers per topic |
| offload_container_LIFECYCLE_TIME | Gauge | Lifecycle time of a container |
| offload_container_PROCESS_FAILED_COUNT | Counter | Count of container processing failures |
Write Stats Metrics
| Metric Name | Type | Description |
|---|---|---|
| storage_write_MESSAGE_ENCODE_LATENCY | Summary | Latency for message encoding |
| storage_write_CONTAINER_WRITE_LATENCY | Summary | Latency for writing container data to Lakehouse storage |
| storage_write_CONTAINER_WRITE_MESSAGE_LATENCY | Summary | Latency for writing messages in a container |
| storage_write_CONTAINER_WRITE_METADATA_LATENCY | Summary | Latency for writing metadata in a container |
| storage_write_CONTAINER_WRITE_FLUSH_LATENCY | Summary | Latency for flushing a container |
| storage_write_CONTAINER_WRITE_COMMIT_RETRY_COUNT | Counter | Retry count for container write commits |
| storage_write_DATA_DELETION_LATENCY | Summary | Latency for data deletion in the Lakehouse storage |
| storage_write_BYTES_OUT | Counter | Bytes written to the Lakehouse storage |
| storage_write_MESSAGE_OUT | Counter | Message count written to the Lakehouse storage |
Read Stats Metrics
| Metric Name | Type | Description |
|---|---|---|
| storage_read_READ_ADDITIONAL_DATA_LATENCY | Summary | Latency for reading ledger metadata |
| storage_read_READ_ENTRIES_LATENCY | Summary | Latency for reading entries from Lakehouse storage |
| storage_read_READ_CACHE_HITS | Counter | Count of read cache hits |
| storage_read_READ_CACHE_MISSES | Counter | Count of read cache misses |
| storage_read_READ_CACHE_HIT_ENTRIES | Counter | Number of entries hit in the read cache |
| storage_read_READ_CACHE_HIT_BYTES | Counter | Number of bytes hit in the read cache |
| storage_read_READ_CACHE_PREFETCHED_ENTRIES | Counter | Number of prefetched entries in the read cache |
| storage_read_READ_CACHE_PREFETCHED_BYTES | Counter | Number of prefetched bytes in the read cache |
| storage_read_READ_ENTRIES_FROM_STORAGE_COUNT | Counter | Number of entries read from Lakehouse storage |
| storage_read_READ_BYTES_FROM_STORAGE_COUNT | Counter | Number of bytes read from Lakehouse storage |
| storage_read_READ_FROM_STORAGE_LATENCY | Summary | Latency for reading entries from Lakehouse storage |
| storage_read_MESSAGE_DECODE_LATENCY | Summary | Latency for decoding fetched messages into Pulsar entries |