1. Operating StreamNative Platform

Pulsar Audit log

Audit logs capture, protect, and preserve authorization activity in Pulsar clusters, tenants, namespaces, and topics. Once Pulsar is up and running within a large team, it's critical to keep an eye on who is touching data and what they're doing with it. Structured audit logs provide an easy way to track user/application access, so you can identify potential anomalies and bad actors.

Structured audit logs enable you to capture audit logs in a set of dedicated Pulsar topics, either on a local or a remote cluster, including:

  • Capture low-volume, management-related activities, such as creating or deleting tenants, namespaces or topics (enabled by default).
  • Capture high-volume activities, such as produce, consume, and acknowledge events (you can enable when needed).

You can use Pulsar integrated tools, like Pulsar Functions and Pulsar SQL, to process and analyze the audit events stored in the Pulsar topics. Additionally, you can offload audit events to external data lakes or data warehouses (like Snowflake or Databricks) for analysis using Pulsar IO connectors.

Configure audit log

The audit logger writes the audit log into a Pulsar topic using the following default settings:

  • The topic name of the audit log is persistent://sn/system/audit_log_all.
  • The audit log captures only events in the Management category.

To customize your Pulsar audit log, you can set configurations in the PULSAR_PREFIX_snAuditLogConfig section in the values.yaml file.

For example, to write the audit logs into different Pulsar topics based on permission settings, you can add the following configurations in the PULSAR_PREFIX_snAuditLogConfig section. And then events are written to persistent://sn/system/audit_log_allowed if they succeed in permission, and written to persistent://sn/system/audit_log_denied if they fail in permission.

PULSAR_PREFIX_brokerInterceptors: 'audit-log'
PULSAR_PREFIX_brokerInterceptorsDirectory: './interceptors'
PULSAR_PREFIX_snAuditLogConfig: >

To capture more audit log events, add the captured setting in the PULSAR_PREFIX_snAuditLogConfig section.

To write events of different categories to separate topics, add the routes setting in the PULSAR_PREFIX_snAuditLogConfig section.

The following example captures audit log events in the Management and Produce categories. Events in the Produce category are written to persistent://sn/system/audit_log_produce_allowed if they succeed in permission, and written to persistent://sn-system/audit/audit_log_produce_denied if they fail in permission.

PULSAR_PREFIX_brokerInterceptors: 'audit-log'
PULSAR_PREFIX_brokerInterceptorsDirectory: './interceptors'
PULSAR_PREFIX_snAuditLogConfig: >
  {"captured":{"principal://User:bob":{"srn://cluster=.*/tenant=.*/namespace=.*/topic=.*": {"category":"Management|Produce","eventType":".*"}}},”routes”:{"srn://cluster=.*/tenant=.*/namespace=.*/topic=.*":{"Produce":{"allowed":"persistent://sn/system/audit_log_produce_allowed","denied":"persistent://sn/system/audit_log_produce_denied"}}},defaultTopics":{"allowed":"persistent://sn/system/audit_log_allowed","denied":"persistent://sn/system/audit_log_denied"}}

When you finish configuration, upgrade your cluster with the following command:

helm upgrade -f /path/to/your/file.yaml <release_name> streamnative/sn-platform -n <k8s_namespace>

Your Pulsar broker restarts automatically after the setting, and then you can create a new namespace and check the audit log in the topic created for the audit log.

Verify audit log

The audit log topic stores the audit log messages. To verify the audit log configuration, complete the following steps:

  1. Create a new namespace in the public tenant.

    bin/pulsar-admin namespaces create public/audit_log
  2. Check the audit log in the persistent://sn/system/audit_log_all topic.

    bin/pulsar-admin topics peek-messages -n 1 -s audit persistent://sn/system/audit_log_all

    If you see similar audit logs below, your audit log configuration works.

      "id": "11c5296d-bf17-431a-80be-79ba66ba8a35",
      "specVersion": "0.1",
      "category": "Management",
      "time": "2021-06-15T04:58:41.710Z",
      "eventType": "CreateNamespace",
      "resourceInfo": {
        "resourceType": "Namespace",
        "cluster": "<release_name>-sn-platform",
        "tenant": "public",
        "namespace": "audit_log"
      "authenticationInfo": { "role": "admin" },
      "authorizationInfo": { "granted": true, "superUserAuthorization": true },
      "requestInfo": {
        "metadata": {
          "clientAddress": "",
          "uri": "/admin/v2/namespaces/public/audit_log",
          "method": "PUT"
      "responseInfo": { "responseType": "SUCCESS", "responseCode": 204 }

Event type

Each audit log event includes information about the event, event time, and permission status. The supported audit event types include:

TypeEvent typeDescriptionCategoryDefault value
ListClustersList Pulsar clusters.Describefalse
GetClusterGet cluster information.Describefalse
CreateClusterCreate a Pulsar cluster.Managementtrue
UpdateClusterUpdate Pulsar cluster information.Managementtrue
DeleteClusterDelete a Pulsar cluster.Managementtrue
ListTenantsList Pulsar tenants.Describefalse
GetTenantGet tenant information.Describefalse
CreateTenantCreate a Pulsar tenant.Describetrue
UpdateTenantUpdate tenant information.Describetrue
DeleteTenantDelete a Pulsar tenant.Describetrue
CreateNamespaceCreate a namespace.Managementtrue
DeleteNamespaceDelete a namespace.Managementtrue
ListNamespacesList Pulsar namespaces.Describefalse
GetNamespaceGet namespace information.Describefalse
CreatePartitionedTopicCreate a partitioned topic.Managementtrue
UpdatePartitionsUpdate partitions for a partitioned topic.Managementtrue
DeletePartitionedTopicDelete a partitioned topic.Managementtrue
ListTopicsList Pulsar topics.Describefalse
ListPartitionedTopicsList partitioned Pulsar topics.Describefalse
GetPartitionsGet partitions of a partitioned topic.Describefalse
CreateSubscriptionCreate a subscription.Managementtrue
DeleteSubscriptionDelete a subscription.Managementtrue
ListSubscriptionsList subscriptions of a topic.Describefalse
NewProducerCreate a producer attached to the topic.Producefalse
CloseProducerClose a producer.Producefalse
ProduceProduce messages to the broker.Producefalse
NewConsumerCreate a consumer attached to the topic.Consumefalse
CloseConsumerClose a consumer.Consumefalse
ConsumeConsume messages from a topic.Consumefalse
AcknowledgeAcknowledge messages by the consumer.Consumefalse
TLS on Load Balancer