Repository navigation
[Bug] (iceberg) branch-4.1/4.2: AncestorNullDefaultExpr never opens its children, so a cast default under a nested equality-delete key can abort the BE #69782
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
branch-4.2 (checked at 9f63be85cea) and branch-4.1 (checked at 5fc51d21256), FileScannerV2 Iceberg reader.
What's Wrong?
AncestorNullDefaultExpr in be/src/format_v2/table/iceberg_reader.cpp never opens its children. Its open() is
RETURN_IF_ERROR_OR_PREPARED(VExpr::open(state, context, scope));and RETURN_IF_ERROR_OR_PREPARED (be/src/exprs/vexpr.h) returns Status::OK() as soon as _prepared is set, which prepare() has already done with the same macro. So VExpr::open is never called and none of the children is opened.
The equality-delete key of a missing nested field is built in _build_missing_equality_delete_key_expr. When the table leaf type differs from the key type, the initial default is wrapped in a cast (EvolutionCast on branch-4.2, Cast on branch-4.1). When the data file has the key's parent struct, that cast becomes the second child of AncestorNullDefaultExpr, which returns NULL for a NULL parent and the default otherwise. The cast is therefore executed without having been opened, and it fails Check failed: _open_finished || block == nullptr in Cast, which aborts the BE.
This follows from the code. It was not reproduced on branch-4.1 or branch-4.2. On master the same code was hit by a unit test, where the BE aborted exactly this way, and it is fixed in #69711 (commit 3c2a8e9cb87) by opening the children the way EqualityDeletePredicate does.
What You Expected?
The scan applies the equality delete: rows under a NULL parent get a NULL key, other rows get the converted default, and the BE does not abort.
How to Reproduce?
Not reproduced on 4.x. From the code, the conditions are:
- An Iceberg table with an equality delete keyed on a nested field (
s.k). - A data file that has the parent struct
sbut not the fieldk, sokis resolved through its initial default (Iceberg v3 initial defaults). - The table leaf type of
kdiffers from the key type the reader compares the key in (filter.key_types), so the default takes the!table_leaf->type->equals(*filter.key_types[key_idx])branch and gets the cast. - A non-NULL initial default, so the cast actually runs.
The master unit test IcebergMissingNestedPlaceholderKeyParsesTextDefaultInTombstoneType in be/test/format_v2/table/iceberg_reader_test.cpp (in #69711) builds this shape.
Anything Else?
A second possible issue on branch-4.2, also not verified: the branch-4.2 FE may send an Iceberg TIME or timestamptz_ns field as TPrimitiveType.UNSUPPORTED. thrift_to_type (be/src/core/data_type/primitive_type.cpp) has no case for it, and its default: is CHECK(false) << ", meet unknown type ". If such a field reaches the BE as an equality-delete key type, the BE would abort while building the key. Master instead sends these types as a STRING placeholder; #69711 also handles that placeholder for TIME keys and their defaults.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Doris-Breakwater commentedon Oct 28, 2026
Breakwater-GitHub-Analysis-Slot: slot_e6d6cdcbb651
Assessment: the expression lifecycle defect is confirmed by source inspection at both reported revisions (
9f63be85cea/ branch-4.2 and5fc51d21256/ branch-4.1). The specific BE abort on those branches remains unverified: the missing piece is a reachable 4.x plan that puts the cast under this expression. Treat this as an actionable Iceberg/FileScannerV2 lifecycle issue with a potentially serious crash consequence, while retaining that qualification.Verified mechanism
prepare()sets_prepared. The guard inopen()then returns before evaluatingVExpr::open(...), so neither the children nor the assignment to_open_finishedis reached. This is present in 4.2AncestorNullDefaultExprand 4.1; see the guard definition. The surroundingEqualityDeletePredicatecorrectly opens its children, but cannot compensate for this child returning early.EvolutionCaston 4.2, orCaston 4.1, belowAncestorNullDefaultExpr.EvolutionCastinheritsCast::open(), which initializes the function context. Execution checks_open_finished || block == nullptrwithDCHECK, so the exact assertion abort depends on the build's DCHECK configuration; an unopened function context remains a lifecycle violation regardless. See the builder and Cast lifecycle.What is not yet established on 4.x
A table/delete-file type difference alone does not prove the required cast is constructed. In these revisions,
_resolve_equality_delete_fieldsalready restores the table type intofilter.key_types. The later table-path lookup also restores primitive types from the external schema throughrestore_current_primitive_type. With consistent metadata, these two types ordinarily agree. Please provide the concrete differing types and schema carriers that make condition 3 true on each branch; do not assume that the new per-file comparison behavior in #69711 already exists in 4.x.I inspected the named test in PR commit
3c2a8e9cb87: it supplies a STRING schema placeholder and text default, then exercises the PR's new tombstone-type comparison path. That is useful regression coverage for the lifecycle fix, but is not itself a reproduction against unchanged 4.x. Also, #69711 is currently open and unmerged; the cited commit is its head, not evidence of a released fix. Its current backport label isbackport/4.2-master.Separate TIME / timestamptz_ns concern
The 4.2 source supports the reported transport hazard: Iceberg type mapping returns UNSUPPORTED for TIME and zoned nanosecond timestamps; the schema carrier serializes the column type, and BE type restoration calls
thrift_to_type, whose default is an unconditionalCHECK(false). If that carrier reaches equality-key resolution, it can fail before default evaluation. This is a separate failure path; an end-to-end FE/BE reproduction is still needed to establish whether planning rejects the input earlier. Opening expression children alone does not resolve it.Recommended next steps / missing evidence
VExpr::openwithout the prepare-state guard. Keep it accompanied by a focused test proving that a cast child is actually opened. AuditNestedStructFieldExpr::open()too: it has the same guard pattern, although no separate crash is established here.spresent ands.kabsent, a non-NULL initial default, and a concrete equality-delete key type that reaches the cast branch. Recordtable_leaf->type,filter.key_types[key_idx], and the constructed expression tree. Cover NULL/non-NULL parents, matching/nonmatching/NULL tombstones, and Parquet/ORC as supported. Check row IDs as well as survival of the BE.timestamptz_ns; inspect the query-wide and split-specific serialized schema types. Handle unsupported input with defined semantics or a controlled error, and verify any placeholder change together with default parsing and comparison behavior.Castorthrift_to_type, plus query ID. These distinguish the two failure paths.Validation scope: read-only inspection of the exact branch snapshots and PR head, plus live issue/PR metadata. No Doris build or unit test was run, no 4.x crash was reproduced, and no code was changed. The issue currently has no labels; suggested triage is bug / Iceberg / FileScannerV2 with branch-4.1 and branch-4.2 tracking, prioritizing a confirmed reachable BE abort once reproduced.
[test](be) Fix nullable value checks in Iceberg expression test
[fix](be) Open children of Iceberg nested struct expressions (#69832)
[fix](be) Open children of Iceberg nested struct expressions (#69832)