Repository navigation
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
Local pipeline review — ✅ PASSschema: doris-repo-review/v1
status: PASS
pr: apache/doris#70214
commit: 216a30a6219b7fac42d991c958baad9b8c58337a
base: b1720bdaee9c26c2b68c6955837257dd901bd736
reviewed_at: 2026-11-04T19:13+08:00
reviewer: hubgeter
model: gpt-6.1-sol
effort: max
findings: {blocker: 0, major: 0, minor: 0, nit: 0}
rounds: 1
converged: trueNotes for maintainers
Reviewed locally with the |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 28941 ms |
Contributor
TPC-DS: Total hot run time: 152007 ms |
Contributor
ClickBench: Total hot run time: 24.41 s |
hubgeter
force-pushed
the
analyze/doris-29979-master
branch
from
November 5, 2026 07:01
216a30a to
d332f85
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28902 ms |
Contributor
TPC-DS: Total hot run time: 153829 ms |
Contributor
ClickBench: Total hot run time: 25.09 s |
hubgeter
marked this pull request as ready for review
November 9, 2026 09:52
### What problem does this PR solve? Issue Number: N/A Related PR: apache#68786 Problem Summary: Iceberg STRING partition values equal to the literal 'null' and SQL NULL share the same legacy commit token. Even with distinct partition writers, the FE can commit the literal value as a null partition, so partition pruning misses matching rows. Preserve nullable values through dynamic, static, and mixed BE writers, transmit an optional NULL bitmap, and share decoding across data-file metadata, delete-file metadata, and identity-partition conflict filters. Retain legacy sender behavior and typed JSON nulls, and reject inconsistent commit partition metadata. ### Release note Fix Iceberg writes incorrectly treating literal STRING partition values equal to 'null' as SQL NULL. The complete correction requires upgraded FE and BE processes and applies to newly committed files. ### Check List (For Author) - Test: Unit Test / Regression test / Manual test - 150 FE writer-helper and connector-transaction unit tests passed. - BE focused unit tests are building; not yet executed. - Parquet/ORC regression output generation and comparison passed (28 query results). - ASAN BE build, FE build, formatting, header hygiene, and FE Checkstyle passed. - Both modified BE test files passed full-file clang-tidy; the final changed-line scan is in progress. - Behavior changed: Yes. New literal 'null' partitions retain non-null manifest values and conflict filters; malformed partition metadata fails before commit. - Does this need documentation: No
hubgeter
force-pushed
the
analyze/doris-29979-master
branch
from
November 9, 2026 09:53
d332f85 to
62104bf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: #68786
Problem Summary:
Writing an Iceberg STRING partition whose value is the literal
'null'can commit it as SQL NULL. For example, after inserting(1, 'null')and(2, NULL), partition pruning forp = 'null'can omit row 1. The writer routing fix in #68786 separates the files, but commit fragments still use the same textual token for both partition values.Preserve nullable partition values through dynamic, static, and mixed partition writers, and add an optional NULL bitmap to the commit protocol. Decode it consistently when constructing data files, delete files, and identity partition conflict filters. Keep the legacy token fallback for senders without the bitmap, and preserve typed NULLs in existing JSON fragments. Reject inconsistent bitmap or partition-field counts before committing metadata.
The complete correction requires upgraded FE and BE processes. It applies to new writes and does not rewrite existing incorrect manifests. Older receivers ignore the optional field and retain their previous behavior.
Release note
Fix Iceberg writes incorrectly treating the literal STRING partition value
'null'as SQL NULL, which could cause partition-filtered queries to miss rows.Check List (For Author)
Test
bash build.sh --be -j 6(ASAN) andbash build.sh --fe -j 4passed; formatting, header hygiene, and FE Checkstyle passed. All six changed BE source/header/test files passed clang-tidy on changed lines; both modified BE test files also passed full-file clang-tidy.Behavior changed:
'null'and SQL NULL partitions retain distinct manifest values and conflict filters; malformed commit partition metadata is rejected.Does this need documentation?
Check List (For Reviewer who merge this PR)