Pulsar Audit log

Audit logs track and store authorization activities in Pulsar clusters, tenants, namespaces, and topics. After a Pulsar cluster is up and running within a large team, it's critical to keep an eye on who is touching data and what they're doing with it. Structured audit logs provide an easy way to track user/application access, so you can identify potential anomalies and bad actors.

Structured audit logs enable you to capture audit logs in a set of dedicated Pulsar topics, either on a local or a remote cluster, including:

• low-volume, management-related activities, such as creating or deleting tenants, namespaces or topics
• high-volume activities, such as produce, consume, and acknowledge events

Configure audit log

The audit logger writes the audit log into a Pulsar topic using the following default settings:

• The topic name of the audit log is persistent://sn/system/audit_log_all.
• The audit log captures only events in the Management category.

To enable the audit log, you need to add the below configutions under the broker.configData in Helm values:

broker:
  configData:
    PULSAR_PREFIX_disableBrokerInterceptors: 'false'
    PULSAR_PREFIX_brokerInterceptors: 'audit-log'
    PULSAR_PREFIX_brokerInterceptorsDirectory: './interceptors'
    PULSAR_PREFIX_snAuditLogConfig: >
      {"captured":{".*":{".*":{"category":"Management","eventType":".*"}}},"defaultTopics":{"allowed":"persistent://sn/system/audit_log_all","denied":"persistent://sn/system/audit_log_all"}}

To capture more audit log events, add the captured setting in the PULSAR_PREFIX_snAuditLogConfig section.

To write events of different categories to separate topics, add the routes setting in the PULSAR_PREFIX_snAuditLogConfig section.

The following example captures audit log events in the Management and Produce categories. Events in the Produce category are written to persistent://sn/system/audit_log_produce_allowed if they succeed in permission, and written to persistent://sn-system/audit/audit_log_produce_denied if they fail in permission.

broker:
  configData:
    PULSAR_PREFIX_disableBrokerInterceptors: 'false'
    PULSAR_PREFIX_brokerInterceptors: 'audit-log'
    PULSAR_PREFIX_brokerInterceptorsDirectory: './interceptors'
    PULSAR_PREFIX_snAuditLogConfig: >
      {"captured":{"principal://User:bob":{"srn://cluster=.*/tenant=.*/namespace=.*/topic=.*": {"category":"Management|Produce","eventType":".*"}}},”routes”:{"srn://cluster=.*/tenant=.*/namespace=.*/topic=.*":{"Produce":{"allowed":"persistent://sn/system/audit_log_produce_allowed","denied":"persistent://sn/system/audit_log_produce_denied"}}},defaultTopics":{"allowed":"persistent://sn/system/audit_log_allowed","denied":"persistent://sn/system/audit_log_denied"}}

Work with audit log

You can use Pulsar clients, Pulsar CLI, Rest API, and sink connectors to process and analyze the audit events stored in the Pulsar topics.

Consume the audit log topic with Pulsar clients

You can get data from the audit log topic with Pulsar clients. Refer to the Connect section to learn how to configure Pulsar clients for StreamNative Platform cluster and consume data from the audit log topic.

Consume the audit log topic with Pulsar CLI

The pulsar-client is also a helpful tool to get data from the audit log topic. You can use pulsar-client for StreamNative Platform cluster and consume data from the audit log topic.

Consume the audit log topic with Rest API

StreamNative Cloud supports Rest API, which provides a RESTful interface to a Pulsar cluster. For more information, see Rest API to learn how to leverage the Rest API to consume data from the audit log topic.

Sink audit log topic with connectors

If you want to integrate audit log data with your other data systems like Google BigQuery, AWS SQS, and AWS Kinesis, use a sink connector. For more information, see sink connectors to learn how to leverage sink connectors to integrate and synchronize audit log data.

Audit log example

{
  "id": "11c5296d-bf17-431a-80be-79ba66ba8a35",
  "specVersion": "0.1",
  "category": "Management",
  "time": "2021-06-15T04:58:41.710Z",
  "eventType": "CreateNamespace",
  "resourceInfo": {
    "resourceType": "Namespace",
    "cluster": "<release_name>-sn-platform",
    "tenant": "public",
    "namespace": "audit_log"
  },
  "authenticationInfo": { "role": "admin" },
  "authorizationInfo": { "granted": true, "superUserAuthorization": true },
  "requestInfo": {
    "metadata": {
      "clientAddress": "10.225.14.43",
      "uri": "/admin/v2/namespaces/public/audit_log",
      "method": "PUT"
    }
  },
  "responseInfo": { "responseType": "SUCCESS", "responseCode": 204 }
}

Event type

Each audit log includes information about the event, event time, and permission status. The supported audit event types include: