Enable Lakehouse Integration

Lakehouse integration can be enabled or disabled at the cluster, namespace, and topic levels using dynamic configuration. This allows fine-grained control over which topics produce lakehouse tables.

For the full set of dynamic configuration keys (including topic-level feature settings such as partitionKey and upsertModeEnabled), see the Dynamic Configuration Guide.

Table Modes

Mode	Property	Description
External Table (SDT)	`sdt.enabled`	Delivers data to external lakehouse catalogs. Supports upsert, partition key, and schema evolution.
Internal Table (SBT)	`sbt.enabled`	Coming Soon. Managed tables with streaming reads + analytics on the same data.

Compaction task publishing is enabled when either sdt.enabled or sbt.enabled is true.

Configuration Key Format

All dynamic configuration keys must be prefixed with the cluster name. The cluster name is the value of clusterName in the Pulsar broker’s conf/broker.conf. See Finding the Cluster Name for how to retrieve it. For example, if the cluster name is private-cloud:

Scope	Key format	Example
Cluster	`<cluster>.cluster.<key>`	`private-cloud.cluster.sdt.enabled`
Namespace/Topic	`<cluster>.<key>`	`private-cloud.sdt.enabled`

Supported Dynamic Configuration Keys

Cluster Level

Applied via the sn/system namespace:

Key	Description
`<cluster>.cluster.sdt.enabled`	Enable/disable External Table (SDT) for the cluster
`<cluster>.cluster.sbt.enabled`	Enable/disable Internal Table (SBT) — Coming Soon
`<cluster>.cluster.sdt.catalog.name`	Default catalog name for SDT
`<cluster>.cluster.tail.compact.data.visibility.interval.in.seconds`	Data visibility delay override

Namespace and Topic Level

Key	Description
`<cluster>.sdt.enabled`	Enable/disable SDT for a namespace or topic
`<cluster>.sbt.enabled`	Enable/disable SBT — Coming Soon
`<cluster>.sdt.catalog.name`	Catalog name for a namespace or topic
`<cluster>.tail.compact.data.visibility.interval.in.seconds`	Data visibility delay override

Cluster-Level Configuration

Cluster-level properties are stored in the sn/system namespace.

Enable SDT for the cluster

bin/pulsar-admin namespaces set-properties \
  -p private-cloud.cluster.sdt.enabled=true \
  sn/system

Set the default catalog

bin/pulsar-admin namespaces set-properties \
  -p private-cloud.cluster.sdt.catalog.name=my-catalog \
  sn/system

Namespace-Level Configuration

Enable SDT for a namespace

bin/pulsar-admin namespaces set-properties \
  -p private-cloud.sdt.enabled=true \
  <tenant>/<namespace>

Assign a catalog to a namespace

bin/pulsar-admin namespaces set-property \
  -k catalog.name -v <catalog-name> \
  <tenant>/<namespace>

Example:

bin/pulsar-admin namespaces set-property \
  -k catalog.name -v polaris-prod \
  public/iceberg

Topic-Level Configuration

Topic-level properties override namespace-level settings.

Enable SDT for a topic

bin/pulsar-admin topics update-properties \
  -p private-cloud.sdt.enabled=true \
  persistent://<tenant>/<namespace>/<topic>

Assign a catalog to a topic

bin/pulsar-admin topics update-properties \
  -p catalog.name=<catalog-name> \
  persistent://<tenant>/<namespace>/<topic>

Example:

bin/pulsar-admin topics update-properties \
  -p catalog.name=s3table-analytics \
  persistent://public/iceberg/events-topic

Configuration Override Priority

Settings at a more specific level override broader settings:

Topic properties
    ↓ (overrides)
Namespace properties
    ↓ (overrides)
Cluster properties (sn/system)

Extending Valid Configuration Keys

To introduce additional dynamic configuration keys, set the following environment variable when starting the Pulsar broker:

LAKEHOUSE_DYNAMIC_EXTRA_VALID_KEYS_IN_CONF_FILE=key1,key2,key3

Example:

LAKEHOUSE_DYNAMIC_EXTRA_VALID_KEYS_IN_CONF_FILE=clusterSdtEnabled,clusterSbtEnabled,clusterSdtCatalogName

Next Steps

Explore features:

Overview

Quick Start

Configure Private Cloud

Operate Private Cloud

References

Table Modes

Configuration Key Format

Supported Dynamic Configuration Keys

Cluster Level

Namespace and Topic Level

Cluster-Level Configuration

Enable SDT for the cluster

Set the default catalog

Namespace-Level Configuration

Enable SDT for a namespace

Assign a catalog to a namespace

Topic-Level Configuration

Enable SDT for a topic

Assign a catalog to a topic

Configuration Override Priority

Extending Valid Configuration Keys

Next Steps

​Table Modes

​Configuration Key Format

​Supported Dynamic Configuration Keys

​Cluster Level

​Namespace and Topic Level

​Cluster-Level Configuration

​Enable SDT for the cluster

​Set the default catalog

​Namespace-Level Configuration

​Enable SDT for a namespace

​Assign a catalog to a namespace

​Topic-Level Configuration

​Enable SDT for a topic

​Assign a catalog to a topic

​Configuration Override Priority

​Extending Valid Configuration Keys

​Next Steps

Table Modes

Configuration Key Format

Supported Dynamic Configuration Keys

Cluster Level

Namespace and Topic Level

Cluster-Level Configuration

Enable SDT for the cluster

Set the default catalog

Namespace-Level Configuration

Enable SDT for a namespace

Assign a catalog to a namespace

Topic-Level Configuration

Enable SDT for a topic

Assign a catalog to a topic

Configuration Override Priority

Extending Valid Configuration Keys

Next Steps