Skip to main content
If an Iceberg reader returns unexpected duplicate rows from a Unity Catalog table, check whether the current snapshot references the same Parquet file more than once. A snapshot describes the files that make up a version of the table; duplicate file references can cause a reader to read the same records multiple times. This guide helps you identify duplicate file references and work with support to repair the affected metadata. It does not cover duplicate records stored in different data files.

​
Diagnose duplicate references

Use an Iceberg client connected to the affected Unity Catalog REST endpoint to inspect the current snapshot. Record:
  • The fully qualified table name and current snapshot ID.
  • The number of live data-file references and distinct file paths.
  • Any file paths that have more than one live reference.
  • The total row count and occurrence counts of affected records.
For example, 51 live file references and 50 distinct paths indicate an extra reference to an existing file. Count entries within the same current snapshot. A file appearing in multiple historical snapshots does not by itself indicate this issue. If you use the Iceberg Java API, inspect table.newScan().useSnapshot(snapshotId).planFiles() before splitting scan tasks. Use the same snapshot for file inspection and row verification so that concurrent writes do not change the data being compared. Verify through the Iceberg REST catalog and reader that exposed the issue. Results from Databricks SQL alone might not reveal duplicate references in the Iceberg metadata.

​
Confirm the repair procedure with support

Share the diagnostic information with StreamNative or Databricks support before applying a metadata repair. Include the Iceberg client version and your Databricks compute type and version.
The procedure below uses spark.databricks.delta.managedIceberg.repair.forceRegenerateMetadata, a setting provided by Databricks support for this issue. Ask support to confirm that the setting and procedure apply to your table and runtime before using them. They are not a general-purpose deduplication procedure.

​
Repair metadata with support guidance

​
1. Create Runtime 19 compute and prepare the Notebook

Use all-purpose compute running Databricks Runtime 19, as requested for this procedure. The screenshots use example workspace, compute, and table names; select your own resources. Create or select compute. If you already have compatible compute, use it. Otherwise, open Compute > All-purpose compute and select Create compute.
Databricks Compute page with All-purpose compute and Create compute highlighted
Select 19 in Databricks runtime. Choose a policy and node configuration appropriate for your workspace, then select Create. The node sizes and worker counts in the screenshot are examples; adjust them to your available quota.
Create new compute form with Databricks Runtime 19 highlighted
Create a Notebook. Open Workspace, then select Create > Notebook.
Databricks Workspace with Create and Notebook highlighted
Attach the Notebook and confirm the runtime. In the Notebook, use the compute selector in the upper-right corner to connect to your Runtime 19 compute. Once it is running, execute:
Check that dbr_version identifies Runtime 19. A SQL warehouse version is not the runtime version of an all-purpose cluster.
Notebook compute selector showing Runtime 19 and the current_version query result
Run the following steps in the same Notebook session. Replace catalog_name, schema_name, and table_name with the affected table’s identifiers. Each example includes %sql so you can paste it into a separate Notebook cell and run it as SQL.

​
2. Enable metadata regeneration

After support confirms the procedure, run:

​
3. Preview the repair

Save the output and review it with support. DRY RUN reports issues without applying repairs. For command semantics, see the Databricks FSCK command reference.
Notebook running the metadata regeneration setting and FSCK METADATA ONLY DRY RUN with a result table

​
4. Apply the repair

When support confirms that you should proceed, run the command without DRY RUN:
Notebook running FSCK METADATA ONLY without DRY RUN
The screenshot shows the command completing. Verify the result through Iceberg as described below.

​
Verify the repair

Reload and refresh the table through the Iceberg REST catalog. Record the current snapshot ID, inspect that snapshot’s live file references, and read the affected records again. Confirm that:
  • No file path has duplicate live references in the current snapshot.
  • Records affected by the duplicate references appear the expected number of times.
  • Expected file paths and records remain available.
For example, if one three-row file was referenced twice and no other writes occurred, removing the extra reference reduces the reference count by one and the row count by three. The number of distinct file paths remains unchanged. A successful SQL command or a changed snapshot ID alone does not establish that the duplicates are gone. Verify the file references and records through the affected Iceberg reader.

​
If duplicate references remain

Confirm that the repair targeted the same table inspected by the Iceberg client and that you ran the repair command without DRY RUN. Share these details with support:
  • The fully qualified table name.
  • The Databricks compute type, runtime version, and Iceberg client version.
  • The exact SQL statements, outputs, and any error request IDs.
  • The before-and-after snapshot IDs, duplicate file paths, and row counts.
Do not delete the referenced Parquet file to remove a duplicate reference. The remaining valid reference still needs that file.