Audit logs track and store authorization activities in Pulsar clusters, tenants, namespaces, and topics. After a Pulsar cluster is up and running within a large team, it’s critical to keep an eye on who is touching data and what they’re doing with it. Structured audit logs provide an easy way to track user/application access, so you can identify potential anomalies and bad actors.

Structured audit logs enable you to capture audit logs in a set of dedicated Pulsar topics, either on a local or a remote cluster, including:

  • low-volume, management-related activities, such as creating or deleting tenants, namespaces or topics
  • high-volume activities, such as produce, consume, and acknowledge events

​
Enable Audit Log

To enable the Audit Log on StreamNative Private Cloud, you need to configure them on the PulsarCoordinator CR:

spec:
  auditLog:
    categories:
      - Management
      - Describe
      - Produce
      - Consume

For Pulsar 3.0 and above, please add this configuration on your PulsarBroker CR, otherwiese the Audit Log can’t work. This issue will be fixed in Pulsar 3.2.

spec:
  config:
    custom:
      PULSAR_PREFIX_disableBrokerInterceptors: "false"

After enabling the Audit Log, audit data will be capture and sent to persistent://sn/system/audit_log_all topic.

Because the audit logs are stored in a Pulsar topic, you should configure the cleanup and backlog policy for the namespace to avoid having the Pulsar retention policies inadvertently clean up the audit logs.

​
Work with audit log

You can use Pulsar clients, Pulsar CLI, Rest API, and sink connectors to process and analyze the audit events stored in the Pulsar topics.

​
Consume the audit log topic with Pulsar clients

You can get data from the audit log topic with Pulsar clients.

​
Consume the audit log topic with Pulsar CLI

The pulsar-client is also a helpful tool to get data from the audit log topic. You can use pulsar-client for StreamNative Platform cluster and consume data from the audit log topic.

​
Consume the audit log topic with Rest API

StreamNative Cloud supports Rest API, which provides a RESTful interface to a Pulsar cluster. For more information, see Rest API to learn how to leverage the Rest API to consume data from the audit log topic.

​
Sink audit log topic with connectors

If you want to integrate audit log data with your other data systems like Google BigQuery, AWS SQS, and AWS Kinesis, use a sink connector.

​
Audit log example

----- got message -----
key:[null], properties:[], content:{"id":"4fb444e1-8457-4032-a2d3-4124ee904d47","specVersion":"0.1","category":"Management","time":"2023-11-01T07:48:19.706Z","eventType":"CreateTenant","resourceInfo":{"resourceType":"Tenant","cluster":"brokers","tenant":"audit"},"authenticationInfo":{},"authorizationInfo":{"granted":true,"superUserAuthorization":false},"requestInfo":{"metadata":{"clientAddress":"10.0.2.156","uri":"/admin/v2/tenants/audit","method":"PUT"}},"responseInfo":{"responseType":"SUCCESS","responseCode":204}}
sidebarTitle: Audit Log
----- got message -----
key:[null], properties:[], content:{"id":"068e5ead-a2e5-48f7-979a-be1614205166","specVersion":"0.1","category":"Describe","time":"2023-11-01T07:48:42.626Z","eventType":"ListTopics","resourceInfo":{"resourceType":"NonPersistentTopic","cluster":"brokers","tenant":"sn","namespace":"system"},"authenticationInfo":{},"authorizationInfo":{"granted":true,"superUserAuthorization":false},"requestInfo":{"metadata":{"clientAddress":"10.0.2.156","uri":"/admin/v2/non-persistent/sn/system","method":"GET"}},"responseInfo":{"responseType":"SUCCESS","responseCode":200}}
----- got message -----
key:[null], properties:[], content:{"id":"8188c64e-70cb-4e53-ab41-4c03283a90d4","specVersion":"0.1","category":"Describe","time":"2023-11-01T07:49:18.120Z","eventType":"GetPartitions","resourceInfo":{"resourceType":"PersistentTopic","cluster":"brokers","tenant":"sn","namespace":"system","topic":"audit_log_all"},"authenticationInfo":{},"authorizationInfo":{"granted":true,"superUserAuthorization":false},"requestInfo":{"metadata":{"clientAddress":"10.0.2.156","uri":"/admin/v2/persistent/sn/system/audit_log_all/partitions","method":"GET"}},"responseInfo":{"responseType":"SUCCESS","responseCode":200}}
sidebarTitle: Audit Log
----- got message -----
key:[null], properties:[], content:{"id":"21efcf3b-baa8-41db-9eb1-7796f12e823e","specVersion":"0.1","category":"Management","time":"2023-11-01T07:50:40.242Z","eventType":"DeleteTenant","resourceInfo":{"resourceType":"Tenant","cluster":"brokers","tenant":"audit"},"authenticationInfo":{},"authorizationInfo":{"granted":true,"superUserAuthorization":false},"requestInfo":{"metadata":{"clientAddress":"10.0.2.156","uri":"/admin/v2/tenants/audit","method":"DELETE"}},"responseInfo":{"responseType":"SUCCESS","responseCode":204}}

​
Event type

Each audit log includes information about the event, event time, and permission status. The supported audit event types include:

CategoryTypeEvent typeDescriptionDefault value
ManagementClusterCreateClusterCreate a Pulsar cluster.true
ClusterUpdateClusterUpdate Pulsar cluster information.true
ClusterDeleteClusterDelete Pulsar cluster information.true
TenantCreateTenantCreate a Pulsar tenant.true
TenantUpdateTenantUpdate tenant information.true
TenantDeleteTenantDelete a Pulsar tenant.true
NamespaceCreateNamespaceCreate a namespace.true
NamespaceDeleteNamespaceDelete a namespace.true
TopicCreatePartitionedTopicCreate a partitioned topic.true
TopicUpdatePartitionsUpdate partitions for a partitioned topic.true
TopicDeletePartitionedTopicDelete a partitioned topic.true
SubscriptionCreateSubscriptionCreate a subscription.true
SubscriptionDeleteSubscriptionDelete a subscription.true
DescribeClusterListClustersList Pulsar clusters.false
ClusterGetClusterGet cluster information.false
TenantListTenantsList Pulsar tenants.false
TenantGetTenantGet tenant information.false
NamespaceListNamespacesList Pulsar namespaces.false
NamespaceGetNamespaceGet namespace information.false
TopicListTopicsList Pulsar topics.false
TopicListPartitionedTopicsList partitioned Pulsar topics.false
TopicGetPartitionsGet partitions of a partitioned topic.false
SubscriptionListSubscriptionsList subscriptions of a topic.false
ProduceProducerNewProducerCreate a producer to produce messages to the topic.false
ProducerCloseProducerClose a producer.false
ConsumeConsumerNewConsumerCreate a consumer to subscribe to the topic.false
ConsumerCloseConsumerClose a consumer.false