Private Cloud Usage Report

Private Cloud supports Pulsar cluster usage data collection and the data will be stored using the Avro schema inside the Pulsar topic.

Usage data

The usage data for the Pulsar cluster includes the CPU and memory usage for the PulsarBroker, BookkeeperCluster, PulsarProxy, ZookeeperCluster, Sink, Source, and Function components.

+---------------------+---------------+-----------+---------------------------+--------------------------+---------------------------+---------------------------+----------------------+----------------------+----------------------+
|        TIME         | CLUSTER NAME  | NAMESPACE |          BROKER           |          PROXY           |          BOOKIE           |         ZOOKEEPER         |         SINK         |        SOURCE        |       FUNCTION       |
+---------------------+---------------+-----------+---------------------------+--------------------------+---------------------------+---------------------------+----------------------+----------------------+----------------------+
| 2024-06_16 00:11:07 | private-cloud | pulsar    | cpu: 600m, memory: 1536Mi | cpu: 200m, memory: 512Mi | cpu: 600m, memory: 1536Mi | cpu: 600m, memory: 1536Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi |
| 2024-06_16 01:11:07 | private-cloud | pulsar    | cpu: 600m, memory: 1536Mi | cpu: 200m, memory: 512Mi | cpu: 600m, memory: 1536Mi | cpu: 600m, memory: 1536Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi |
| 2024-06_16 02:11:07 | private-cloud | pulsar    | cpu: 600m, memory: 1536Mi | cpu: 200m, memory: 512Mi | cpu: 600m, memory: 1536Mi | cpu: 600m, memory: 1536Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi |
| 2024-06_16 03:11:07 | private-cloud | pulsar    | cpu: 600m, memory: 1536Mi | cpu: 200m, memory: 512Mi | cpu: 600m, memory: 1536Mi | cpu: 600m, memory: 1536Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi |
+---------------------+---------------+-----------+---------------------------+--------------------------+---------------------------+---------------------------+----------------------+----------------------+----------------------+

Usage data access

Private Cloud users can access to the Pulsar cluster usage data through the cli tool snctl which support to display the cluster usage data in terminal or export the usage data in CSV format.

Prerequisites for Pulsar

Install the latest snctl v0.19.1 or above. Install snctl
Upgrade to the sn-operator helm chart v0.1.6 or above.
Enable the PulsarCoordinator object for your Pulsar cluster.

View the usage data

Get the Private Cloud Proxy external service endpoint:

kubectl get services -o jsonpath='{.items[?(@.spec.type=="LoadBalancer")].status.loadBalancer.ingress[0].ip}' -n pulsar

4.153.37.80

View the usage data through the usage-data get local commands:

snctl usage-data get local -u pulsar://4.153.37.80:6650
INFO[0000] Connecting to broker                          remote_addr="pulsar://4.153.37.80:6650"
INFO[0000] TCP connection established                    local_addr="10.6.100.20:51750" remote_addr="pulsar://4.153.37.80:6650"
INFO[0000] Connection is ready                           local_addr="10.6.100.20:51750" remote_addr="pulsar://4.153.37.80:6650"
INFO[0001] Connecting to broker                          remote_addr="pulsar://4.153.37.80:6650"
INFO[0001] TCP connection established                    local_addr="10.6.100.20:51751" remote_addr="pulsar://4.153.37.80:6650"
INFO[0001] Connection is ready                           local_addr="10.6.100.20:51751" remote_addr="pulsar://4.153.37.80:6650"
INFO[0002] Connected consumer                            consumerID=1 name= subscription=reader-hutim topic="persistent://sn/system/__usage"
INFO[0002] Created consumer                              consumerID=1 name= subscription=reader-hutim topic="persistent://sn/system/__usage"
+---------------------+---------------+-----------+---------------------------+--------------------------+---------------------------+---------------------------+----------------------+----------------------+----------------------+
|        TIME         | CLUSTER NAME  | NAMESPACE |          BROKER           |          PROXY           |          BOOKIE           |         ZOOKEEPER         |         SINK         |        SOURCE        |       FUNCTION       |
+---------------------+---------------+-----------+---------------------------+--------------------------+---------------------------+---------------------------+----------------------+----------------------+----------------------+
| 2024-08_29 12:49:29 | private-cloud | pulsar    | cpu: 600m, memory: 1536Mi | cpu: 200m, memory: 512Mi | cpu: 600m, memory: 1536Mi | cpu: 600m, memory: 1536Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi |
| 2024-08_29 13:49:29 | private-cloud | pulsar    | cpu: 600m, memory: 1536Mi | cpu: 200m, memory: 512Mi | cpu: 600m, memory: 1536Mi | cpu: 600m, memory: 1536Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi |
| 2024-08_29 14:49:29 | private-cloud | pulsar    | cpu: 600m, memory: 1536Mi | cpu: 200m, memory: 512Mi | cpu: 600m, memory: 1536Mi | cpu: 600m, memory: 1536Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi | cpu: 0m, memory: 0Mi |
+---------------------+---------------+-----------+---------------------------+--------------------------+---------------------------+---------------------------+----------------------+----------------------+----------------------+
INFO[0002] Closing consumer=1                            consumerID=1 name= subscription=reader-hutim topic="persistent://sn/system/__usage"
INFO[0003] Closed consumer                               consumerID=1 name= subscription=reader-hutim topic="persistent://sn/system/__usage"

Export the usage data

Export the usage data through the extra -o, --csv-file-path flag with CSV format:

snctl usage-data get local -u pulsar://4.153.37.80:6650 -o usage-data.csv