Lakehouse tiered storage metrics

Note

This feature is currently in private preview. If you want to try it out or have any questions, submit a ticket to the support team.

Introduction

Monitoring tiered storage metrics in a private cloud setup is crucial for ensuring the efficiency and performance of your data offloading processes. By enabling the embedded stat service embeddedStatServiceEnabled and configuring the metric port statServicePort in the offload.conf file, you can access detailed insights into the offload framework's operations.

Components of Tiered Storage Metrics

Service Stats:
- Provides resource usage statistics at the offload service level.
Offload Stats:
- Focuses on the offload framework's performance for data offloading, including namespace, topic, and partition labels.
Container Stats:
- Offers insights into message container activity, aiding in optimizing data flushing to the Lakehouse storage.
Write Stats:
- Monitors the encoding and writing processes to the Lakehouse storage, measuring latency and throughput.
Read Stats:
- Tracks the reading process from the Lakehouse storage, including cache hits, misses, and entry retrieval metrics.

Lakehouse tiered storage architecture

Service Stats Metrics

Metric Name	Type	Description
offload_service_IN_USAGE_CONTAINERS_COUNT	Gauge	Number of allocated message containers
offload_service_IDLE_CONTAINERS_COUNT	Gauge	Number of idle message containers
offload_service_LEDGER_DELETION_LATENCY	Summary	Latency for ledger deletions
offload_service_PARALLEL_READ_CONTAINERS	Gauge	Number of parallel read containers
offload_service_OFFLOADING_TOPICS_COUNT	Gauge	Number of parallel offloading topics
offload_service_READ_CACHE_SIZE	Gauge	Size of the read cache in bytes
offload_service_READ_CACHE_COUNT	Gauge	Entry count in the read cache

Offload Stats Metrics

Metric Name	Type	Description
offload_framework_WAITING_PROMISE	Gauge	Number of waiting offload promises
offload_framework_MESSAGE_OUT_OF_ORDER_COUNT	Counter	Count of out-of-order messages
offload_framework_MESSAGE_PUT_INTO_CONTAINER_FAILED_COUNT	Counter	Count of failed message placements into containers
offload_framework_MESSAGE_PUT_INTO_CONTAINER_RETRY_COUNT	Counter	Count of retried message placements into containers
offload_framework_FETCH_LEDGER_METADATA_LATENCY	Summary	Latency for fetching ledger metadata
offload_framework_FETCH_TOPIC_SCHEMA_LATENCY	Summary	Latency for fetching topic schemas
offload_framework_ACKNOWLEDGE_FAILED_COUNT	Counter	Count of failed message acknowledgments

Container Stats Metrics

Metric Name	Type	Description
offload_container_SWITCH_COUNT	Counter	Container switch count
offload_container_CURRENT_LEDGERID	Gauge	Current container's ledger ID
offload_container_BYTES_IN	Counter	Bytes put into message containers
offload_container_MESSAGE_IN	Counter	Number of messages put into containers
offload_container_MESSAGE_COUNT_IN_LAST_CONTAINER	Gauge	Message count in the last container
offload_container_BYTES_IN_LAST_CONTAINER	Gauge	Bytes in the last container
offload_container_ALLOCATED_COUNT	Gauge	Number of allocated containers per topic
offload_container_LIFECYCLE_TIME	Gauge	Lifecycle time of a container
offload_container_PROCESS_FAILED_COUNT	Counter	Count of container processing failures

Write Stats Metrics

Metric Name	Type	Description
storage_write_MESSAGE_ENCODE_LATENCY	Summary	Latency for message encoding
storage_write_CONTAINER_WRITE_LATENCY	Summary	Latency for writing container data to Lakehouse storage
storage_write_CONTAINER_WRITE_MESSAGE_LATENCY	Summary	Latency for writing messages in a container
storage_write_CONTAINER_WRITE_METADATA_LATENCY	Summary	Latency for writing metadata in a container
storage_write_CONTAINER_WRITE_FLUSH_LATENCY	Summary	Latency for flushing a container
storage_write_CONTAINER_WRITE_COMMIT_RETRY_COUNT	Counter	Retry count for container write commits
storage_write_DATA_DELETION_LATENCY	Summary	Latency for data deletion in the Lakehouse storage
storage_write_BYTES_OUT	Counter	Bytes written to the Lakehouse storage
storage_write_MESSAGE_OUT	Counter	Message count written to the Lakehouse storage

Read Stats Metrics

Metric Name	Type	Description
storage_read_READ_ADDITIONAL_DATA_LATENCY	Summary	Latency for reading ledger metadata
storage_read_READ_ENTRIES_LATENCY	Summary	Latency for reading entries from Lakehouse storage
storage_read_READ_CACHE_HITS	Counter	Count of read cache hits
storage_read_READ_CACHE_MISSES	Counter	Count of read cache misses
storage_read_READ_CACHE_HIT_ENTRIES	Counter	Number of entries hit in the read cache
storage_read_READ_CACHE_HIT_BYTES	Counter	Number of bytes hit in the read cache
storage_read_READ_CACHE_PREFETCHED_ENTRIES	Counter	Number of prefetched entries in the read cache
storage_read_READ_CACHE_PREFETCHED_BYTES	Counter	Number of prefetched bytes in the read cache
storage_read_READ_ENTRIES_FROM_STORAGE_COUNT	Counter	Number of entries read from Lakehouse storage
storage_read_READ_BYTES_FROM_STORAGE_COUNT	Counter	Number of bytes read from Lakehouse storage
storage_read_READ_FROM_STORAGE_LATENCY	Summary	Latency for reading entries from Lakehouse storage
storage_read_MESSAGE_DECODE_LATENCY	Summary	Latency for decoding fetched messages into Pulsar entries

Additional Metrics for Delta Offloader

For the Delta offloader, specific metrics are available for the Delta writer, reader, and index design, enhancing monitoring capabilities for Delta-based operations.

By leveraging these comprehensive metrics, you can effectively monitor and optimize the performance of your Lakehouse tiered storage in a private cloud environment.