Skip to content

[Feature] Add native reading primitives for Iceberg encrypted Parquet files #70245

Open

Description

@Gabriel39

Iceberg encrypted metadata planning and the file-key wire contract are available after #70192 and #70213, but native encrypted Parquet data-file reading is not implemented yet.

The next independently reviewable units are:

  • Extract each data file's StandardKeyMetadata key and AAD into every scan split, preserving plaintext files and rejecting malformed metadata without exposing key material.
  • Decrypt Parquet encrypted footers, page headers, dictionary pages and data pages through parquet-cpp crypto primitives, while keeping the native Doris decoder.

Keep encrypted SQL admission disabled during these stages. Encrypted indexes, delete files and final scan integration need their own follow-up validation before execution is enabled.

Also fix the encrypted metadata fixture reset so that retained HMS tables can be recreated after the disposable KMS restarts with fresh keys. Validate the first reset after restart with the SDK and Spark reference readers.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions