> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streamnative.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Troubleshoot duplicate file references in Unity Catalog
> Identify duplicate Iceberg file references in Databricks Unity Catalog and verify a repair with support guidance.
If an Iceberg reader returns unexpected duplicate rows from a Unity Catalog table, check whether the current snapshot references the same Parquet file more than once. A snapshot describes the files that make up a version of the table; duplicate file references can cause a reader to read the same records multiple times.
This guide helps you identify duplicate file references and work with support to repair the affected metadata. It does not cover duplicate records stored in different data files.
## Diagnose duplicate references
Use an Iceberg client connected to the affected Unity Catalog REST endpoint to inspect the current snapshot. Record:
* The fully qualified table name and current snapshot ID.
* The number of live data-file references and distinct file paths.
* Any file paths that have more than one live reference.
* The total row count and occurrence counts of affected records.
For example, 51 live file references and 50 distinct paths indicate an extra reference to an existing file. Count entries within the same current snapshot. A file appearing in multiple historical snapshots does not by itself indicate this issue.
If you use the Iceberg Java API, inspect `table.newScan().useSnapshot(snapshotId).planFiles()` before splitting scan tasks. Use the same snapshot for file inspection and row verification so that concurrent writes do not change the data being compared.
Verify through the Iceberg REST catalog and reader that exposed the issue. Results from Databricks SQL alone might not reveal duplicate references in the Iceberg metadata.
## Confirm the repair procedure with support
Share the diagnostic information with StreamNative or Databricks support before applying a metadata repair. Include the Iceberg client version and your Databricks compute type and version.
The procedure below uses `spark.databricks.delta.managedIceberg.repair.forceRegenerateMetadata`, a setting provided by Databricks support for this issue. Ask support to confirm that the setting and procedure apply to your table and runtime before using them. They are not a general-purpose deduplication procedure.
## Repair metadata with support guidance
### 1. Create Runtime 19 compute and prepare the Notebook
Use **all-purpose compute running Databricks Runtime 19**, as requested for this procedure. The screenshots use example workspace, compute, and table names; select your own resources.
**Create or select compute.** If you already have compatible compute, use it. Otherwise, open **Compute > All-purpose compute** and select **Create compute**.
Select **19** in **Databricks runtime**. Choose a policy and node configuration appropriate for your workspace, then select **Create**. The node sizes and worker counts in the screenshot are examples; adjust them to your available quota.
**Create a Notebook.** Open **Workspace**, then select **Create > Notebook**.
**Attach the Notebook and confirm the runtime.** In the Notebook, use the compute selector in the upper-right corner to connect to your Runtime 19 compute. Once it is running, execute:
```sql theme={null}
%sql
SELECT current_version();
```
Check that `dbr_version` identifies Runtime 19. A SQL warehouse version is not the runtime version of an all-purpose cluster.
Run the following steps in the same Notebook session. Replace `catalog_name`, `schema_name`, and `table_name` with the affected table's identifiers. Each example includes `%sql` so you can paste it into a separate Notebook cell and run it as SQL.
### 2. Enable metadata regeneration
After support confirms the procedure, run:
```sql theme={null}
%sql
SET spark.databricks.delta.managedIceberg.repair.forceRegenerateMetadata = true;
```
### 3. Preview the repair
```sql theme={null}
%sql
FSCK REPAIR TABLE `catalog_name`.`schema_name`.`table_name`
METADATA ONLY DRY RUN;
```
Save the output and review it with support. `DRY RUN` reports issues without applying repairs. For command semantics, see the [Databricks FSCK command reference](https://docs.databricks.com/aws/en/sql/language-manual/delta-fsck).
### 4. Apply the repair
When support confirms that you should proceed, run the command without `DRY RUN`:
```sql theme={null}
%sql
FSCK REPAIR TABLE `catalog_name`.`schema_name`.`table_name`
METADATA ONLY;
```
The screenshot shows the command completing. Verify the result through Iceberg as described below.
## Verify the repair
Reload and refresh the table through the Iceberg REST catalog. Record the current snapshot ID, inspect that snapshot's live file references, and read the affected records again.
Confirm that:
* No file path has duplicate live references in the current snapshot.
* Records affected by the duplicate references appear the expected number of times.
* Expected file paths and records remain available.
For example, if one three-row file was referenced twice and no other writes occurred, removing the extra reference reduces the reference count by one and the row count by three. The number of distinct file paths remains unchanged.
A successful SQL command or a changed snapshot ID alone does not establish that the duplicates are gone. Verify the file references and records through the affected Iceberg reader.
## If duplicate references remain
Confirm that the repair targeted the same table inspected by the Iceberg client and that you ran the repair command without `DRY RUN`. Share these details with support:
* The fully qualified table name.
* The Databricks compute type, runtime version, and Iceberg client version.
* The exact SQL statements, outputs, and any error request IDs.
* The before-and-after snapshot IDs, duplicate file paths, and row counts.
Do not delete the referenced Parquet file to remove a duplicate reference. The remaining valid reference still needs that file.