> ## Documentation Index > Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt > Use this file to discover all available pages before exploring further. # Configure Lakehouse Catalogs After [preparing your external catalog](/private-cloud/v2/configure-private-cloud/private-preview/ursa-lakehouse/prepare-lakehouse-catalogs), configure the compaction service to connect to it. Catalog configuration is added to the `compactionScheduler.config.custom` section of the PulsarBroker YAML. ## Multi-Catalog Architecture StreamNative supports configuring multiple catalogs simultaneously. Different namespaces or topics can route data to different catalogs. ### Configuration Pattern * **Iceberg catalogs:** `iceberg.catalog..` * **Delta catalogs:** `delta.catalog..` ### Default Catalog Set the default catalog used when no topic or namespace override is specified: ```yaml theme={null} custom: catalog.default: ``` ### Catalog Resolution Priority The catalog used for a topic is resolved in this order: ``` Topic property (catalog.name) ↓ (if not set) Namespace property (catalog.name) ↓ (if not set) Default catalog (catalog.default) ``` See [Enable Lakehouse Integration](/private-cloud/v2/configure-private-cloud/private-preview/ursa-lakehouse/enable-lakehouse-integration) for how to assign catalogs at namespace and topic level. *** ## Iceberg Catalogs ### Unity Catalog (Managed Iceberg Table) ```yaml theme={null} compactionScheduler: config: custom: catalog.default: iceberg.catalog..catalog-backend: "UNITYCATALOG" iceberg.catalog..type: "rest" iceberg.catalog..uri: "https:///api/2.1/unity-catalog/iceberg-rest" iceberg.catalog..warehouse: "" iceberg.catalog..credential: "" iceberg.catalog..oauth2-server-uri: "https:///oidc/v1/token" iceberg.catalog..scope: "all-apis" iceberg.catalog..security: "OAUTH2" iceberg.catalog..vended-credentials-enabled: "true" iceberg.catalog..token-refresh-enabled: "true" ``` | Property | Description | | ---------------------------- | ---------------------------------- | | `catalog-backend` | `UNITYCATALOG` | | `type` | `rest` | | `uri` | Databricks workspace URL | | `warehouse` | Catalog name created in Databricks | | `credential` | Databricks access token | | `oauth2-server-uri` | Databricks oauth2 service uri | | `scope` | `all-apis` | | `security` | `OAUTH2` | | `vended-credentials-enabled` | `true` | | `token-refresh-enabled` | `true` | ### Snowflake Horizon Catalog ```yaml theme={null} compactionScheduler: config: custom: catalog.default: iceberg.catalog..catalog-backend: "HORIZON" iceberg.catalog..type: "rest" iceberg.catalog..uri: "https://-.snowflakecomputing.com/polaris/api/catalog" iceberg.catalog..credential: "" iceberg.catalog..scope: "session:role:" iceberg.catalog..warehouse: "" iceberg.catalog..header.X-Iceberg-Access-Delegation: "vended-credentials" iceberg.catalog..token-refresh-enabled: "true" ``` | Property | Description | | ------------------------------------ | -------------------------------------------------- | | `catalog-backend` | `HORIZON` | | `uri` | Snowflake Horizon REST API endpoint | | `credential` | PAT token | | `scope` | Snowflake role scope (e.g., `session:role:PUBLIC`) | | `warehouse` | Snowflake database name | | `header.X-Iceberg-Access-Delegation` | `vended-credentials` (required) | | `token-refresh-enabled` | `true` (recommended) | ### Snowflake Open Catalog (Polaris) ```yaml theme={null} compactionScheduler: config: custom: catalog.default: iceberg.catalog..catalog-backend: "POLARIS" iceberg.catalog..type: "rest" iceberg.catalog..uri: "https://.snowflakecomputing.com/polaris/api/catalog" iceberg.catalog..credential: ":" iceberg.catalog..warehouse: "" iceberg.catalog..header.X-Iceberg-Access-Delegation: "vended-credentials" iceberg.catalog..scope: "PRINCIPAL_ROLE:ALL" iceberg.catalog..token-refresh-enabled: "true" ``` | Property | Description | | ------------------------------------ | ---------------------------------------------- | | `catalog-backend` | `POLARIS` | | `credential` | Client ID and secret in `:` format | | `warehouse` | Polaris catalog name | | `header.X-Iceberg-Access-Delegation` | `vended-credentials` | | `scope` | `PRINCIPAL_ROLE:ALL` | | `token-refresh-enabled` | `true` | ### AWS S3Table ```yaml theme={null} compactionScheduler: config: custom: catalog.default: iceberg.catalog..catalog-backend: "S3TABLE" iceberg.catalog..type: "rest" iceberg.catalog..rest.sigv4-enabled: "true" iceberg.catalog..rest.signing-name: "s3tables" iceberg.catalog..rest.signing-region: "" iceberg.catalog..uri: "https://s3tables..amazonaws.com/iceberg" iceberg.catalog..warehouse: "arn:aws:s3tables:::bucket/" iceberg.catalog..rest-metrics-reporting-enabled: "false" ``` | Property | Description | | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | `catalog-backend` | `S3TABLE` | | `rest.sigv4-enabled` | `true` (required for AWS SigV4 auth) | | `rest.signing-name` | `s3tables` | | `rest.signing-region` | AWS region of the S3Table bucket | | `uri` | S3Tables REST endpoint (varies by [region](https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-regions-quotas.html#s3-tables-regions)) | | `warehouse` | S3Table bucket ARN | | `rest-metrics-reporting-enabled` | `false` (S3Table does not support metric reporting) | > **Important:** The Ursa cluster must run in the same region as the S3Table bucket. ### Google BigLake ```yaml theme={null} compactionScheduler: config: custom: catalog.default: iceberg.catalog..catalog-backend: "BIGLAKE" iceberg.catalog..type: "rest" iceberg.catalog..uri: "https://biglake.googleapis.com/iceberg/v1/restcatalog" iceberg.catalog..warehouse: "gs://" iceberg.catalog..header.x-goog-user-project: "" iceberg.catalog..rest.auth.type: "org.apache.iceberg.gcp.auth.GoogleAuthManager" iceberg.catalog..io-impl: "org.apache.iceberg.gcp.gcs.GCSFileIO" iceberg.catalog..rest-metrics-reporting-enabled: "false" iceberg.catalog..header.X-Iceberg-Access-Delegation: "vended-credentials" ``` | Property | Description | | ------------------------------------ | ------------------------------------------------------- | | `catalog-backend` | `BIGLAKE` | | `warehouse` | GCS bucket path from BigLake catalog properties | | `header.x-goog-user-project` | GCP project ID from BigLake catalog properties | | `rest.auth.type` | `org.apache.iceberg.gcp.auth.GoogleAuthManager` (fixed) | | `io-impl` | `org.apache.iceberg.gcp.gcs.GCSFileIO` (fixed) | | `header.X-Iceberg-Access-Delegation` | `vended-credentials` (fixed) | *** ## Delta Lake Catalogs ### Unity Catalog (Delta) ```yaml theme={null} compactionScheduler: config: custom: catalog.default: delta.catalog..unityCatalogUri: "https://" delta.catalog..unityCatalogName: "" delta.catalog..unityCatalogToken: "" ``` #### Authentication Options **Token-based (recommended):** ```yaml theme={null} delta.catalog..unityCatalogToken: "" # OR from file: delta.catalog..unityCatalogTokenFile: "/path/to/token/file" ``` **OAuth2 (machine-to-machine):** ```yaml theme={null} delta.catalog..unityCatalogClientId: "" delta.catalog..unityCatalogClientSecret: "" ``` #### BYOL (Bring Your Own Lakehouse) Enable managed commit support for Unity Catalog: ```yaml theme={null} # Delta Lake delta.catalog..unityCatalogByolEnabled: "true" # Iceberg iceberg.catalog..unityCatalogByolEnabled: "true" ``` *** ## Without Catalog (Direct Bucket) If you do not need an external catalog service, data can be written directly to the object storage bucket. > **Required permissions:** When no external catalog is used, the compaction-scheduler pod's IAM role (AWS), service account (GCP), or workload identity (Azure) must have **read**, **write**, **create**, and **list** permissions on the target bucket. Without an external catalog, the compaction service interacts with object storage directly to create namespaces, write metadata, list existing files, and read prior snapshots. > > Examples of the required permissions per cloud: > > | Cloud | Permissions | > | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | > | AWS S3 | `s3:GetObject`, `s3:PutObject`, `s3:DeleteObject`, `s3:ListBucket`, `s3:GetBucketLocation` on the warehouse bucket and prefix | > | GCS | `storage.buckets.get`, `storage.objects.get`, `storage.objects.list`, `storage.objects.create`, `storage.objects.delete` (or the `Storage Object Admin` role) | > | Azure Blob / ADLS | `Storage Blob Data Contributor` on the container | ### Iceberg (Hadoop Catalog) The default Hadoop catalog writes Iceberg metadata and data files directly to the configured storage path. No external catalog service is required. ```yaml theme={null} compactionScheduler: config: lakehouseType: iceberg catalog.default: iceberg.catalog..type: "hadoop" iceberg.catalog..warehouse: "/suffix" streamTableMode: "EXTERNAL" ``` #### Table maintenance Snowflake Open Catalog (Polaris) and the Hadoop catalog do **not** run table maintenance on your behalf. Streaming writes from the StreamNative Ursa compaction service produce many small Parquet files and accumulate snapshot history over time, which degrades query performance and inflates storage costs. You are responsible for scheduling and running maintenance against every Iceberg table written by Ursa. Run the maintenance operations below on a regular schedule. They are provided as [Apache Iceberg Spark stored procedures](https://iceberg.apache.org/docs/latest/spark-procedures/) and can be triggered from any Spark cluster (Databricks, AWS EMR, AWS Glue, GCP Dataproc, or self-managed Spark) that has the Iceberg Spark runtime, catalog credentials, and IAM access to the warehouse bucket. **Maintenance operations** | Operation | Purpose | Suggested cadence | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | `rewrite_data_files` | Compact small Parquet files into fewer, larger files. Reduces file-listing overhead and improves scan performance. | Hourly to daily, depending on ingestion rate | | `expire_snapshots` | Drop snapshots older than the retention window so their data and manifest files can be cleaned up. | Daily; retain at least 1–7 days so in-flight readers and time-travel queries keep working | | `remove_orphan_files` | Delete files in the table location that are no longer referenced by any snapshot (typically left behind by failed or partial writes). | Weekly | | `rewrite_manifests` | Rewrite manifest files so they align with the current partition layout. Improves query planning time. | Weekly, or after large schema or partition changes | **Example: run maintenance from Spark** The following examples assume the catalog has been registered in Spark as ``. Replace ``, ``, and `` with your values. ```sql theme={null} -- Compact small files. Iceberg targets files smaller than the default 512 MB. CALL .system.rewrite_data_files(table => '.
'); -- Expire snapshots older than 3 days; keep the 5 most recent snapshots. CALL .system.expire_snapshots( table => '.
', older_than => TIMESTAMP '2026-05-20 00:00:00', retain_last => 5 ); -- Remove orphan files older than 7 days. CALL .system.remove_orphan_files( table => '.
', older_than => TIMESTAMP '2026-05-20 00:00:00' ); -- Rewrite manifests to match the current partition layout. CALL .system.rewrite_manifests(table => '.
'); ``` **Operational guidance** * **Credentials.** The principal that runs maintenance must have catalog privileges to read and write the target table (for example, the same `TABLE_READ_DATA`, `TABLE_WRITE_DATA`, `TABLE_READ_PROPERTIES`, and `TABLE_WRITE_PROPERTIES` privileges configured for the Ursa compaction service) and IAM access to the warehouse bucket so it can read and rewrite the underlying data files. With the Hadoop catalog there is no catalog service to authenticate against — only the bucket IAM access is required. * **Concurrency.** Iceberg uses optimistic concurrency control. If maintenance commits race with the Ursa compaction writer, one of them retries. Schedule heavy operations (`rewrite_data_files`, `rewrite_manifests`) during low-write windows when possible. * **Retention vs. time travel.** `expire_snapshots` and `remove_orphan_files` permanently delete files. Choose a retention window that exceeds the longest expected read query and your time-travel SLA. * **Schedule the workload.** Most teams orchestrate these procedures from Databricks Jobs, AWS EMR steps, Airflow, Dagster, or a Kubernetes `CronJob`. Pick a scheduler that fits your existing operational stack. * **Reference.** See the [Iceberg Spark procedures](https://iceberg.apache.org/docs/latest/spark-procedures/#metadata-management) documentation for the full parameter list, including options for partial rewrites (`where`), file-size targets, and merge-on-read delete file compaction. ### Delta (No Unity Catalog) Delta tables are written directly to the configured storage path without Unity Catalog integration. ```yaml theme={null} compactionScheduler: config: catalog.default: lakehouseType: delta delta.catalog..directExternalStoragePath: "/suffix" streamTableMode: "EXTERNAL" ``` #### Table maintenance When Delta tables are written directly to object storage without a managed catalog, no service runs maintenance on your behalf. Streaming writes from the StreamNative Ursa compaction service produce many small Parquet files and accumulate Delta transaction-log history over time, which degrades query performance and inflates storage costs. You are responsible for scheduling and running maintenance against every Delta table written by Ursa. Run the maintenance operations below on a regular schedule. They can be executed from any Spark cluster (Databricks, AWS EMR, AWS Glue, GCP Dataproc, or self-managed Spark) that has the [`delta-spark`](https://docs.delta.io/latest/quick-start.html#set-up-apache-spark-with-delta-lake) runtime and IAM access to the warehouse bucket. For background and tuning recommendations, see the [Databricks Delta Lake best practices](https://docs.databricks.com/aws/en/delta/best-practices) guide. **Maintenance operations** | Operation | Purpose | Suggested cadence | | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | | `OPTIMIZE` | Compact small Parquet files into fewer, larger files (bin-packing). Reduces file-listing overhead and improves scan performance. | Hourly to daily, depending on ingestion rate | | `OPTIMIZE … ZORDER BY` | Co-locate data by frequently filtered columns so query engines can skip more files. | Weekly, or after large inserts | | `VACUUM` | Delete data files that are no longer referenced by the Delta log and are older than the retention threshold. | Daily or weekly; retain at least 7 days so in-flight readers and time-travel queries keep working | **Example: run maintenance from Spark** Replace `` with the table location (`s3://...`, `gs://...`, or `abfss://...`). ```sql theme={null} -- Compact small files. Default target file size is 1 GB. OPTIMIZE delta.``; -- Compact and co-locate by frequently filtered columns. OPTIMIZE delta.`` ZORDER BY (user_id, event_time); -- Remove data files older than 7 days that are not referenced by the log. VACUUM delta.`` RETAIN 168 HOURS; ``` **Operational guidance** * **Credentials.** The principal that runs maintenance must have IAM read, write, list, and delete permissions on the warehouse bucket so it can rewrite and remove data files. * **`VACUUM` retention.** Do not reduce the retention window below 7 days without explicitly disabling the safety check (`spark.databricks.delta.retentionDurationCheck.enabled = false`). Shorter windows risk breaking concurrent readers and time-travel queries. * **Concurrency.** Delta uses optimistic concurrency control. Bin-packing `OPTIMIZE` runs do not generally conflict with append-only writes from Ursa, but `OPTIMIZE … ZORDER BY` rewrites larger portions of the table and is best scheduled in lower-write windows. * **Schedule the workload.** Most teams orchestrate maintenance from Databricks Jobs, AWS EMR steps, Airflow, Dagster, or a Kubernetes `CronJob`. Pick a scheduler that fits your existing operational stack. * **Reference.** See the [Databricks Delta best practices](https://docs.databricks.com/aws/en/delta/best-practices) and the [Delta Lake utility commands](https://docs.delta.io/latest/delta-utility.html) for full syntax, retention options, and tuning guidance. *** ## Multi-Catalog Example Configure two catalogs (one Polaris, one S3Table) and set a default: ```yaml theme={null} compactionScheduler: config: custom: # Default catalog catalog.default: polaris-prod # Catalog 1: Snowflake Open Catalog (Polaris) iceberg.catalog.polaris-prod.catalog-backend: "POLARIS" iceberg.catalog.polaris-prod.type: "rest" iceberg.catalog.polaris-prod.uri: "https://xyz.snowflakecomputing.com/polaris/api/catalog" iceberg.catalog.polaris-prod.credential: ":" iceberg.catalog.polaris-prod.warehouse: "prod-catalog" # Catalog 2: AWS S3Table iceberg.catalog.s3table-analytics.catalog-backend: "S3TABLE" iceberg.catalog.s3table-analytics.type: "rest" iceberg.catalog.s3table-analytics.rest.sigv4-enabled: "true" iceberg.catalog.s3table-analytics.rest.signing-name: "s3tables" iceberg.catalog.s3table-analytics.rest.signing-region: "us-east-2" iceberg.catalog.s3table-analytics.uri: "https://s3tables.us-east-2.amazonaws.com/iceberg" iceberg.catalog.s3table-analytics.warehouse: "arn:aws:s3tables:us-east-2:123456789:bucket/analytics" iceberg.catalog.s3table-analytics.rest-metrics-reporting-enabled: "false" # Configure SDT streamTableMode: "EXTERNAL" # Configure to use Iceberg lakehouseType: "ICEBERG" ``` Then assign catalogs per namespace or topic: ```bash theme={null} # Use default (polaris-prod) for all topics in the namespace pulsar-admin namespaces set-property -k catalog.name -v polaris-prod public/default # Override for a specific topic to use S3Table pulsar-admin topics update-properties \ -p catalog.name=s3table-analytics \ persistent://public/default/analytics-topic ``` ### Limitations * A namespace or topic can reference only **one catalog** at a time * You can assign **different catalogs** to different topics or namespaces * You **cannot** assign multiple catalogs to a single topic or namespace ## Next Steps * [Enable Lakehouse Integration](/private-cloud/v2/configure-private-cloud/private-preview/ursa-lakehouse/enable-lakehouse-integration) -- Enable SDT at cluster, namespace, or topic level