Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt

Use this file to discover all available pages before exploring further.

Lakehouse integration can be enabled or disabled at the cluster, namespace, and topic levels using dynamic configuration. This allows fine-grained control over which topics produce lakehouse tables.
For the full set of dynamic configuration keys (including topic-level feature settings such as partitionKey and upsertModeEnabled), see the Dynamic Configuration Guide.

​
Table Modes

ModePropertyDescription
External Table (SDT)sdt.enabledDelivers data to external lakehouse catalogs. Supports upsert, partition key, and schema evolution.
Internal Table (SBT)sbt.enabledComing Soon. Managed tables with streaming reads + analytics on the same data.
Compaction task publishing is enabled when either sdt.enabled or sbt.enabled is true.

​
Configuration Key Format

All dynamic configuration keys must be prefixed with the cluster name. The cluster name is the value of clusterName in the Pulsar broker’s conf/broker.conf. See Finding the Cluster Name for how to retrieve it. For example, if the cluster name is private-cloud:
ScopeKey formatExample
Cluster<cluster>.cluster.<key>private-cloud.cluster.sdt.enabled
Namespace/Topic<cluster>.<key>private-cloud.sdt.enabled

​
Supported Dynamic Configuration Keys

​
Cluster Level

Applied via the sn/system namespace:
KeyDescription
<cluster>.cluster.sdt.enabledEnable/disable External Table (SDT) for the cluster
<cluster>.cluster.sbt.enabledEnable/disable Internal Table (SBT) — Coming Soon
<cluster>.cluster.sdt.catalog.nameDefault catalog name for SDT
<cluster>.cluster.tail.compact.data.visibility.interval.in.secondsData visibility delay override

​
Namespace and Topic Level

KeyDescription
<cluster>.sdt.enabledEnable/disable SDT for a namespace or topic
<cluster>.sbt.enabledEnable/disable SBT — Coming Soon
<cluster>.sdt.catalog.nameCatalog name for a namespace or topic
<cluster>.tail.compact.data.visibility.interval.in.secondsData visibility delay override

​
Cluster-Level Configuration

Cluster-level properties are stored in the sn/system namespace.

​
Enable SDT for the cluster

bin/pulsar-admin namespaces set-properties \
  -p private-cloud.cluster.sdt.enabled=true \
  sn/system

​
Set the default catalog

bin/pulsar-admin namespaces set-properties \
  -p private-cloud.cluster.sdt.catalog.name=my-catalog \
  sn/system

​
Namespace-Level Configuration

​
Enable SDT for a namespace

bin/pulsar-admin namespaces set-properties \
  -p private-cloud.sdt.enabled=true \
  <tenant>/<namespace>

​
Assign a catalog to a namespace

bin/pulsar-admin namespaces set-property \
  -k catalog.name -v <catalog-name> \
  <tenant>/<namespace>
Example: 
bin/pulsar-admin namespaces set-property \
  -k catalog.name -v polaris-prod \
  public/iceberg

​
Topic-Level Configuration

Topic-level properties override namespace-level settings.

​
Enable SDT for a topic

bin/pulsar-admin topics update-properties \
  -p private-cloud.sdt.enabled=true \
  persistent://<tenant>/<namespace>/<topic>

​
Assign a catalog to a topic

bin/pulsar-admin topics update-properties \
  -p catalog.name=<catalog-name> \
  persistent://<tenant>/<namespace>/<topic>
Example: 
bin/pulsar-admin topics update-properties \
  -p catalog.name=s3table-analytics \
  persistent://public/iceberg/events-topic

​
Configuration Override Priority

Settings at a more specific level override broader settings: 
Topic properties
    ↓ (overrides)
Namespace properties
    ↓ (overrides)
Cluster properties (sn/system)

​
Extending Valid Configuration Keys

To introduce additional dynamic configuration keys, set the following environment variable when starting the Pulsar broker: 
LAKEHOUSE_DYNAMIC_EXTRA_VALID_KEYS_IN_CONF_FILE=key1,key2,key3
Example: 
LAKEHOUSE_DYNAMIC_EXTRA_VALID_KEYS_IN_CONF_FILE=clusterSdtEnabled,clusterSbtEnabled,clusterSdtCatalogName

​
Next Steps