Skip to content

[Feature] Read Iceberg Parquet deletes with independent file encryption keys #70316

Open

Description

@Gabriel39

Description

Continue the native Iceberg encryption reader from #70246 and #70248 by carrying and using each Parquet delete file's independent key and AAD. A plaintext data file can reference encrypted position/equality deletes, and an encrypted data file can reference plaintext deletes.

Expected scope and verification

  • Preserve immutable delete encryption metadata through FE scan planning and fresh Thrift descriptors, including the native position_deletes metadata table.
  • Pass data and delete encryption descriptors separately into native Parquet readers.
  • Keep decrypted position/equality objects and type-converted equality keys out of shared split caches so a warmed file path cannot bypass another split's key validation.
  • Explicitly reject encrypted ORC and Puffin deletion-vector descriptors and retain the metadata COUNT fence.
  • Cover all eight plaintext/encrypted data/position/equality combinations, wrong and missing keys, wrong AAD, reused splits, converted equality keys and native metadata-table reads in UT.
  • Add independently generated Java reference fixtures and SQL regressions for V2 and V2 snapshots upgraded to V3, including real plaintext delete application and continued rejection at encrypted SQL dispatch/COUNT boundaries.

Encrypted SQL admission, protected key transport and shared decrypted caches remain separate integration work. This unit should be independently mergeable from authenticated index/Bloom support in #70314.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions