Audit log - Docs

Audit logs track and store authorization activities in Pulsar clusters, tenants, namespaces, and topics. After a Pulsar cluster is up and running within a large team, it's critical to keep an eye on who is touching data and what they're doing with it. Structured audit logs provide an easy way to track user/application access, so you can identify potential anomalies and bad actors.

Structured audit logs enable you to capture audit logs in a set of dedicated Pulsar topics, either on a local or a remote cluster, including:

low-volume, management-related activities, such as creating or deleting tenants, namespaces or topics
high-volume activities, such as produce, consume, and acknowledge events

Enable Audit Log

To enable the Audit Log on StreamNative Private Cloud, you need to configure them on the PulsarCoordinator CR:

spec:
  auditLog:
    categories:
      - Management
      - Describe
      - Produce
      - Consume

For Pulsar 3.0 and above, please add this configuration on your PulsarBroker CR, otherwiese the Audit Log can't work. This issue will be fixed in Pulsar 3.2.

spec:
  config:
    custom:
      PULSAR_PREFIX_disableBrokerInterceptors: "false"

After enabling the Audit Log, audit data will be capture and sent to persistent://sn/system/audit_log_all topic.

Note

Because the audit logs are stored in a Pulsar topic, you should configure the cleanup and backlog policy for the namespace to avoid having the Pulsar retention policies inadvertently clean up the audit logs.

Work with audit log

You can use Pulsar clients, Pulsar CLI, Rest API, and sink connectors to process and analyze the audit events stored in the Pulsar topics.

Consume the audit log topic with Pulsar clients

You can get data from the audit log topic with Pulsar clients.

Consume the audit log topic with Pulsar CLI

The pulsar-client is also a helpful tool to get data from the audit log topic. You can use pulsar-client for StreamNative Platform cluster and consume data from the audit log topic.

Consume the audit log topic with Rest API

StreamNative Cloud supports Rest API, which provides a RESTful interface to a Pulsar cluster. For more information, see Rest API to learn how to leverage the Rest API to consume data from the audit log topic.

Sink audit log topic with connectors

If you want to integrate audit log data with your other data systems like Google BigQuery, AWS SQS, and AWS Kinesis, use a sink connector.

Audit log example

----- got message -----
key:[null], properties:[], content:{"id":"4fb444e1-8457-4032-a2d3-4124ee904d47","specVersion":"0.1","category":"Management","time":"2023-11-01T07:48:19.706Z","eventType":"CreateTenant","resourceInfo":{"resourceType":"Tenant","cluster":"brokers","tenant":"audit"},"authenticationInfo":{},"authorizationInfo":{"granted":true,"superUserAuthorization":false},"requestInfo":{"metadata":{"clientAddress":"10.0.2.156","uri":"/admin/v2/tenants/audit","method":"PUT"}},"responseInfo":{"responseType":"SUCCESS","responseCode":204}}
----- got message -----
key:[null], properties:[], content:{"id":"068e5ead-a2e5-48f7-979a-be1614205166","specVersion":"0.1","category":"Describe","time":"2023-11-01T07:48:42.626Z","eventType":"ListTopics","resourceInfo":{"resourceType":"NonPersistentTopic","cluster":"brokers","tenant":"sn","namespace":"system"},"authenticationInfo":{},"authorizationInfo":{"granted":true,"superUserAuthorization":false},"requestInfo":{"metadata":{"clientAddress":"10.0.2.156","uri":"/admin/v2/non-persistent/sn/system","method":"GET"}},"responseInfo":{"responseType":"SUCCESS","responseCode":200}}
----- got message -----
key:[null], properties:[], content:{"id":"8188c64e-70cb-4e53-ab41-4c03283a90d4","specVersion":"0.1","category":"Describe","time":"2023-11-01T07:49:18.120Z","eventType":"GetPartitions","resourceInfo":{"resourceType":"PersistentTopic","cluster":"brokers","tenant":"sn","namespace":"system","topic":"audit_log_all"},"authenticationInfo":{},"authorizationInfo":{"granted":true,"superUserAuthorization":false},"requestInfo":{"metadata":{"clientAddress":"10.0.2.156","uri":"/admin/v2/persistent/sn/system/audit_log_all/partitions","method":"GET"}},"responseInfo":{"responseType":"SUCCESS","responseCode":200}}
----- got message -----
key:[null], properties:[], content:{"id":"21efcf3b-baa8-41db-9eb1-7796f12e823e","specVersion":"0.1","category":"Management","time":"2023-11-01T07:50:40.242Z","eventType":"DeleteTenant","resourceInfo":{"resourceType":"Tenant","cluster":"brokers","tenant":"audit"},"authenticationInfo":{},"authorizationInfo":{"granted":true,"superUserAuthorization":false},"requestInfo":{"metadata":{"clientAddress":"10.0.2.156","uri":"/admin/v2/tenants/audit","method":"DELETE"}},"responseInfo":{"responseType":"SUCCESS","responseCode":204}}

Event type

Each audit log includes information about the event, event time, and permission status. The supported audit event types include:

Category	Type	Event type	Description	Default value
Management	Cluster	`CreateCluster`	Create a Pulsar cluster.	true
	Cluster	`UpdateCluster`	Update Pulsar cluster information.	true
	Cluster	`DeleteCluster`	Delete Pulsar cluster information.	true
	Tenant	`CreateTenant`	Create a Pulsar tenant.	true
	Tenant	`UpdateTenant`	Update tenant information.	true
	Tenant	`DeleteTenant`	Delete a Pulsar tenant.	true
	Namespace	`CreateNamespace`	Create a namespace.	true
	Namespace	`DeleteNamespace`	Delete a namespace.	true
	Topic	`CreatePartitionedTopic`	Create a partitioned topic.	true
	Topic	`UpdatePartitions`	Update partitions for a partitioned topic.	true
	Topic	`DeletePartitionedTopic`	Delete a partitioned topic.	true
	Subscription	`CreateSubscription`	Create a subscription.	true
	Subscription	`DeleteSubscription`	Delete a subscription.	true
Describe	Cluster	`ListClusters`	List Pulsar clusters.	false
	Cluster	`GetCluster`	Get cluster information.	false
	Tenant	`ListTenants`	List Pulsar tenants.	false
	Tenant	`GetTenant`	Get tenant information.	false
	Namespace	`ListNamespaces`	List Pulsar namespaces.	false
	Namespace	`GetNamespace`	Get namespace information.	false
	Topic	`ListTopics`	List Pulsar topics.	false
	Topic	`ListPartitionedTopics`	List partitioned Pulsar topics.	false
	Topic	`GetPartitions`	Get partitions of a partitioned topic.	false
	Subscription	`ListSubscriptions`	List subscriptions of a topic.	false
Produce	Producer	`NewProducer`	Create a producer to produce messages to the topic.	false
	Producer	`CloseProducer`	Close a producer.	false
Consume	Consumer	`NewConsumer`	Create a consumer to subscribe to the topic.	false
	Consumer	`CloseConsumer`	Close a consumer.	false