Skip to content

Validate Iceberg name mapping for imported Parquet files in native scans #2702

Open

Description

@Sigma-Ma

Is your feature request related to a problem? Please describe.

Imported Parquet files may not contain field IDs. Iceberg uses schema.name-mapping.default to resolve these files, including historical column names.

Auron currently falls back to case-insensitive name matching when a file field has no ID. The native Iceberg scan does not consume the table's name mapping, so existing field-ID coverage does not establish compatibility for imported files.

This is based on source inspection; an incorrect query result has not yet been reproduced.

Describe the solution you'd like

Add compatibility tests using Parquet files whose footers are verified to contain no field IDs. Cover:

  • A normal import through add_files.
  • Reading historical names after a column rename.
  • Drop/add with the same name and an explicit mapping that preserves the imported column's original identity.
  • Projections and filters over mixed imported files and newer files written with field IDs.

Validate explicit expected results with Auron disabled before checking Auron's results and scan plan. Inspect the mapping after schema changes, as Iceberg updates it automatically.

Where native scans cannot safely honor the mapping, fall back to the Iceberg reader with a clear auron.never.convert.reason. Apply the check consistently to regular scans and supported changelog scans.

A conservative initial option is to fall back when the table has a name mapping. This would also disable native scans for some files that already have field IDs, so the performance tradeoff should be agreed before implementation.

Full native name mapping support is outside this proposal.

Describe alternatives you've considered

Keep the current name-based fallback. This does not account for historical aliases or imported column identity.

Implement native name mapping resolution. This requires broader reader changes and should be considered separately.

Additional context

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions