You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Validate Iceberg name mapping for imported Parquet files in native scans #2702
Is your feature request related to a problem? Please describe.
Imported Parquet files may not contain field IDs. Iceberg uses schema.name-mapping.default to resolve these files, including historical column names.
Auron currently falls back to case-insensitive name matching when a file field has no ID. The native Iceberg scan does not consume the table's name mapping, so existing field-ID coverage does not establish compatibility for imported files.
This is based on source inspection; an incorrect query result has not yet been reproduced.
Describe the solution you'd like
Add compatibility tests using Parquet files whose footers are verified to contain no field IDs. Cover:
A normal import through add_files.
Reading historical names after a column rename.
Drop/add with the same name and an explicit mapping that preserves the imported column's original identity.
Projections and filters over mixed imported files and newer files written with field IDs.
Validate explicit expected results with Auron disabled before checking Auron's results and scan plan. Inspect the mapping after schema changes, as Iceberg updates it automatically.
Where native scans cannot safely honor the mapping, fall back to the Iceberg reader with a clear auron.never.convert.reason. Apply the check consistently to regular scans and supported changelog scans.
A conservative initial option is to fall back when the table has a name mapping. This would also disable native scans for some files that already have field IDs, so the performance tradeoff should be agreed before implementation.
Full native name mapping support is outside this proposal.
Describe alternatives you've considered
Keep the current name-based fallback. This does not account for historical aliases or imported column identity.
Implement native name mapping resolution. This requires broader reader changes and should be considered separately.
Activity