[Gabriel39]

What problem does this PR solve?

Related issue: #67366
Related PR: #67449

On branch-4.1, Arrow Flight SQL returns 0000-01-02 for a stored DATE of 0000-01-01. Doris follows the MySQL calendar, which excludes year zero's leap day, while Arrow DATE uses the proleptic Gregorian calendar. Subtracting the Unix epoch day number directly shifts dates in 0000-01-01 through 0000-02-28 by one day, including elements of ARRAY<DATE>.

Adapt the DATE portion of #67449 to branch-4.1, with credit to its author. Convert between the calendars at Arrow and ORC write boundaries and Arrow/default Parquet/ORC read boundaries. Use the same conversion for ORC statistics so pushed-down MIN/MAX agrees with row decoding. Reject unrepresentable external dates and ignore undefined payloads in null Arrow slots.

The two DATE regression suites and their expected results are copied unchanged from #67449. Add its DATE unit coverage plus an explicit null-payload test and an exhaustive comparison against the civil calendar for all 3,652,424 representable dates.

Also align Iceberg DATE day/bucket partition transforms with file ordinals. For ORC/Parquet, metadata is usable only when its raw DATE range proves that every ordinal is representable, including the interior of the range. Unsafe ranges fall back to decoding: strict reads retain their errors, non-strict Parquet reads retain conversion-generated NULLs, and COUNT uses materialized values when physical null counts are insufficient. ORC uses a metadata-only SARG to check projected DATE leaves per stripe: safe excluded stripes retain SARG pruning, while unsafe stripes remain readable with SDK row-group pruning disabled. ORC COUNT validates every DATE leaf of its argument in each selected stripe, and strict Parquet COUNT validates projected DATE descendants before reading only levels.

Scope: DATE, including its Iceberg partition transforms. This does not include DATETIME changes or the independent legacy file scanner conversions used when enable_file_scanner_v2=false. V1 reader changes and mixed-version/rolling-upgrade compatibility gates are explicitly outside this PR's scope.

Validation

• Baseline: 9 of the original 10 DATE unit tests fail; the unchanged Flight regression reproduces the one-day shift.

• Fixed: 250 unit tests pass under ASAN, including the complete ORC reader suite, DATE SerDe exhaustive calendar check, Iceberg partition transforms, Parquet statistics, DATE scans and COUNT tests. These compile the modified branch-4.1 sources against cached test dependencies.

• The unchanged test_date_year_zero Flight regression passes locally, including nullable and array values.

• test_outfile_date_year_zero passes for Parquet and ORC using the original rows and golden assertions, with S3 transport replaced by local outfile/TVF only for the local run. The committed suite retains S3 transport for CI.

• Local integration runs use the same fix rebuilt into an existing ASAN/UBSAN BE build at 5bcb996a5ff, with an existing FE. A full branch-4.1 build and the original S3 run remain for CI.

• clang-format 16 passes for all affected C++ files. Source whitespace checks pass; generated golden files retain their conventional trailing blank line from [fix](arrow) encode DATE in the proleptic Gregorian calendar at format boundaries #67449.

• Review coverage: new tests reproduce partition mismatches, invalid-interior MIN/MAX bypass, ORC SARG/Parquet footer/page pruning inconsistencies, and Parquet IS NULL/COUNT discrepancies. All five reproducing tests fail before the review fixes.

• Added a DATE-only Spark/Iceberg partitioned write/read regression, including selective day/bucket predicates and partition metadata comparison. Its Groovy compilation passes; execution requires the external CI environment.

• Follow-up review: scalar/nested ORC COUNT and strict nested Parquet COUNT reproductions fail before the guards and pass afterward. ARRAY, STRUCT, MAP, all-null values, safe ranges, non-strict Parquet, and unprojected DATE siblings are covered.

• External ORC regression: retain the original LZ4 file as a BE fixture and verify that ordinal -719530 raises a data-quality error. Replace the obsolete 1900-01-01 golden result with an explicit error assertion, restoring the scanner setting afterward. The fixture test and Groovy compilation pass locally. The complete group6 suite is also validated locally with HDFS transport replaced by LOCAL TVF; the committed suite retains HDFS for CI.

• Pruning review: both new tests fail before the fixes and pass afterward. Partial ORC STRUCT projection retains SARG when the invalid DATE sibling is not read; selecting that DATE still preserves its error. Multi-page Parquet tests retain the unsafe DATE page while pruning safe pages for IS NULL and range predicates, through both single-slot and multi-slot OR evaluators.

• Mixed-stripe ORC coverage: the original file-wide guard prunes zero safe stripes; the fix prunes both safe excluded stripes and still reports the invalid DATE in the retained stripe. The active SDK options outlive their reader, and the cached SARG evaluator is discarded before opening an unsafe range.

• External group6 coverage: the original non_vec_orc_scanner.orc fixture encodes 20191111 as an invalid epoch-day ordinal. Replace its obsolete 1900-01-01 golden result with an explicit data-quality error assertion and keep the original bytes as a small BE fixture; projecting other columns still succeeds.

• Final local integration after the stripe fix: all three suites pass with no failures or skips (complete group6: 41 fixtures, 39 golden-result assertions and two expected errors; Flight year zero; Parquet/ORC outfile year zero). These use the incremental ASAN/UBSAN build and local transport substitutions described above.

• Metadata safety review: the malformed pruned-stripe index and truncated nested DATE statistics reproduce SDK bounds assertions before the fix. Both regressions pass afterward; the DATE preflight now adds no file read calls or bytes compared with the same excluded stripe without DATE projection.

• Historical-file migration: an explicit offline tool converts only confirmed legacy Doris DATE ordinals, including nested values. It publishes a separate output atomically and refuses overwrites. Tests cover year-zero boundaries, NULLs, hidden null-container payloads, nested ORC/Parquet round trips, schema metadata, timestamp precision, and failed-write cleanup.

• Final follow-up validation: all 250 ASAN tests and 10 migration-tool tests pass. The final incremental production BE passes the three existing local suites plus six legacy migration/read/export/read checks, covering both ORC and Parquet. Original HDFS/S3 transport and the full branch build remain for CI as described above.

Release note

Fix year-zero DATE values being returned one day late through Arrow Flight SQL, align DATE encoding and decoding in the default Parquet/ORC paths, and keep Iceberg DATE partition values consistent with file contents.

Existing files written with the former year-zero encoding cannot be distinguished automatically from correctly encoded files. Before switching their consumers to the corrected reader, use the explicit legacy DATE migration tool and procedure on files confirmed to use the old Doris encoding, or re-export from authoritative original data. The tool writes new ORC/Parquet files without modifying the source. Existing Iceberg files must be migrated through a new table or a supported table-format transaction that recomputes partition metadata; do not replace files in existing snapshots. The proleptic-only 0000-02-29 remains unrepresentable as a Doris DATE.

Check List (For Author)

• Test
  • Regression test
  • Unit Test
• Behavior changed:
  • Yes, external DATE ordinals use the correct calendar; internal DATE arithmetic is unchanged.
• Does this need documentation?
  • No.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[Gabriel39]

run buildall

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

/review

[github-actions]

Static review of PR #68649 at 5945bab. Reviewed all nine files in the authoritative merge-base diff and relevant Arrow, ORC, Parquet, Iceberg, scanner, and aggregation paths. No additional user focus was provided. No existing inline threads or live comments apply.

Findings: four new inline issues (three P1, one P2): Iceberg DATE partition/file ordinal mismatch; ORC/Parquet MIN/MAX metadata bypass of an invalid interior DATE; ORC SARG/strict Parquet pruning that suppresses the new DATE error; and non-strict nullable Parquet IS NULL/COUNT(d) wrong results from physical versus logical nulls.

Critical checkpoint conclusions:

• Goal and proof: The intended proleptic Gregorian year-zero DATE correction works in direct SerDe conversions by static inspection. Added calendar/ORC unit tests and Arrow Flight/OUTFILE regression cases cover representative boundary values, but do not cover the four downstream failures. Tests and builds were prohibited and were not run; checked expected output files and author claims are not independent runtime proof.
• Scope: The changed DATE conversion code and tests are focused. The explicitly excluded legacy FileScanner v1 path remains a known scope limitation, not a new inline finding.
• Concurrency and lifecycle: The edited conversions and statistics are scan/write-local or constexpr; no new threads, locks, lock order, ownership cycle, or cross-translation-unit static initialization dependency was found. Existing reader close/pushdown lifecycle is material to the metadata bypass findings.
• Configuration: No new option is introduced. Existing enable_strict_cast reaches the strict Parquet error path; no dynamic configuration propagation issue was found.
• Compatibility and storage: The corrected external DATE ordinal changes year-zero file interpretation and old incorrectly encoded exports remain ambiguous, as described in the PR. New Iceberg file payload and partition metadata disagree for year-zero values (M1).
• Parallel paths and conditions: Arrow/Parquet/ORC normal DATE decoding, ORC statistics/SARG, Parquet footer/page indexes, aggregate pushdown, and Iceberg partition transforms were traced. Year-zero leap-day rejection is not consistently preserved by metadata shortcuts (M2-M4).
• Coverage/results: The added tests cover direct calendar conversion, ORC nested/null cases, Flight SQL, and export output. They lack partitioned Iceberg round trips and invalid-interior metadata/predicate/nullable aggregate cases. No test was run under the review restrictions.
• Observability: New conversion errors include the encoded bad DATE value; no separate logging or metric defect was substantiated. Metadata paths can suppress that error as described inline.
• Transaction, persistence, and writes: No EditLog or transaction protocol change is present. External file writes retain the existing lifecycle, but an Iceberg committed partition value can contradict its corrected DATE payload (M1); no separate crash or atomicity issue was substantiated.
• FE/BE variables: No new transmitted variable or thrift field is added. Existing strict-cast and aggregate request paths were checked.
• Performance and remaining issues: The conversion is constant time per value; no distinct regression was substantiated. The four inline issues are the remaining correctness findings. BE null handling aligns selected ORC rows and skips null payload decoding, but nullable Parquet conversion failure creates a logical NULL that physical metadata cannot currently represent (M4).

Review execution reached the permitted three rounds and all discovered candidates were resolved. Convergence is incomplete under the review prompt because the third round produced valuable finding M4; no fourth round is permitted. This is a static review only.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Static review complete for head 0842fe5d56e88da5faf91e27fe84f35c3e559394 (base c6536cfde23bf8ad39d87a4498e4ea7eb59111a8). I reviewed all 16 changed files and the related reader, planner, and writer paths. Two new aggregate bypasses remain: ORC COUNT of DATE, including DATE descendants, and Parquet COUNT of a complex root containing DATE. Both skip a strict DATE conversion error; see the inline findings. The four earlier inline concerns (Parquet scalar NULL/COUNT, unsafe DATE pruning, invalid-interior MIN/MAX, and Iceberg day/bucket partitions) are fixed on this head, so none is repeated or listed as an existing blocker. The review converged after three rounds, with no unresolved candidate. The focus file supplied no additional focus points.

Part 1.3 checkpoint conclusions:

1. Goal and proof: The Gregorian external DATE ordinal and Iceberg partition alignment are implemented across Arrow, ORC, and Parquet. Boundary, invalid-value, pruning, aggregate, Flight, outfile, and Iceberg cases are present; the two COUNT paths above remain gaps.
2. Scope: Changes are focused on DATE conversion, metadata guards, partition transforms, and related tests.
3. Concurrency: No new thread, lock, shared mutable state, or changed lock order is introduced.
4. Lifecycle and initialization: Conversion scratch is local and constants are compile-time; no new ownership cycle or cross-translation-unit static initialization dependency was found.
5. Configuration: No configuration item or dynamic-change requirement is added.
6. Compatibility: Doris-authored old physical files containing year-zero DATE values are ambiguous under the corrected encoding and require re-export as described in the PR context. Standard-compliant files now decode consistently; no protocol or symbol change was found.
7. Parallel paths: DATE32/DATE64, nullable and nested values, ORC and Parquet readers, metadata shortcuts, and Iceberg day/bucket were traced. The remaining parallel-path gaps are the two inline COUNT findings.
8. Conditions: Null payload, int32 range, invalid 0000-02-29, and raw statistics interval checks follow the external encoding. The Parquet root-only COUNT condition misses DATE descendants.
9. Tests: Unit and regression coverage includes boundaries, invalid ordinals, pruning on/off, metadata, and external readback; scalar and nested aggregate error regressions are still needed as described inline. No tests were run in this static review.
10. Expected results: The changed Flight and outfile results match their queries and deterministic ordering; Iceberg assertions compare rows and partition values. External Iceberg execution was not independently verified here.
11. Observability: New conversion errors carry the bad ordinal or row/column context; no additional metric or long-running operation was identified.
12. Transactions and persistence: No EditLog or transaction-protocol change is introduced. Existing Iceberg commit metadata uses the corrected dynamic partition value.
13. Writes and crashes: The Iceberg write path's payload and partition ordinal agree for valid DATE values; no new write-lifecycle or crash-recovery mechanism was found.
14. FE/BE variables: None added or transmitted.
15. Performance: Added arithmetic and range checks are constant-time; the ORC file-statistics walk occurs during setup. No separate material regression was substantiated.
16. Other issues: Null carriers, selected stripes and Row Groups, page/footer pruning, Arrow failure propagation, all-NULL metadata, and output ordering were rechecked. No further distinct issue remained.

This is a static code review only. The review contract prohibited builds and tests; author and CI test claims were not treated as independent runtime validation.

[hello-stephen]

BE UT Coverage Report

Increment line coverage 86.76% (118/136) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	61.07% (26697/43713)
Line Coverage	45.71% (277071/606132)
Region Coverage	41.36% (219179/529993)
Branch Coverage	42.86% (101555/236957)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 87.50% (119/136) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.88% (31840/42519)
Line Coverage	59.22% (356106/601361)
Region Coverage	55.90% (296946/531246)
Branch Coverage	56.78% (134425/236740)

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Review result

Complete static review at 15768f335c1981efb3786e1457d1257610936a80. I found one P1 compatibility issue and two P2 pruning regressions; the three inline comments give the triggering cases and fixes. All three review rounds converged with no new valuable findings in the final round. No builds or tests were run under the review contract.

I rechecked all six existing inline issues against this head. The prior P1 Iceberg partition, DATE MIN/MAX, Parquet NULL/COUNT, and ORC COUNT concerns are addressed by the current code and tests; the P2 invalid-DATE pruning and nested Parquet COUNT concerns are also addressed. None remains a blocking P0/P1 issue. The new comments cover distinct V1/mixed-BE decoding, unprojected ORC DATE SARG gating, and Parquet safe-page pruning.

Critical checkpoints

• Goal and proof: The change maps Doris DATE values to proleptic-Gregorian external ordinals across Arrow, Parquet, ORC, and Iceberg, including year zero and the invalid Gregorian-only day. Calendar, format-reader, Flight, export, and Iceberg tests cover same-version paths. The goal is incomplete for the V1 and mixed-version reads in the P1 comment.
• Scope and clarity: The production change is focused on DATE conversion, partition transforms, and statistics safety. Shared conversion helpers are constant-time and make the boundary explicit. The persisted writer change has a wider rollout effect than the focused code suggests.
• Concurrency and locks: The changed conversion and reader paths execute in existing scan/write contexts; they add no thread, lock, shared mutable state, or lock-order dependency. I found no new concurrency issue.
• Lifecycle and initialization: The new helpers are inline/constexpr and the reader checks use local state. No cross-translation-unit initializer, retained allocation, ownership change, or lifecycle leak is introduced.
• Configuration: No new configuration item or dynamic-update requirement is introduced. The existing enable_file_scanner_v2 choice exposes the V1 decoder mismatch in the P1 finding.
• Compatibility and rolling upgrades: There is no new FE/BE protocol or storage schema, but the persisted DATE ordinal changes. Newly written year-zero files can be misread by selectable V1 readers in the same build and rejected by older BEs during smooth upgrades. The PR discloses re-export of historical incorrectly encoded files; that separate migration limitation is not a new inline finding.
• Parallel paths and conditions: V2 Parquet/ORC, Arrow, Iceberg day/bucket, NULL handling, selected-stripe COUNT, projected Parquet COUNT, and invalid-DATE MIN/MAX were traced. V1 DATE readers remain incompatible; the ORC SARG guard checks omitted nested DATE leaves; one unsafe Parquet DATE page disables pruning of safe pages. The inline comments specify each affected condition.
• Tests and results: I inspected all changed unit tests, the ORC fixture, all four regression suites, and their three result files. The result ordering and expected values match the test code as static evidence. The added tests do not cover V1/mixed-version reads, an omitted ORC DATE sibling, or a multi-page Parquet index with one unsafe page. Author-reported ASAN results are not independent validation; this review ran no tests.
• Observability and errors: New DATE conversion failures report the raw ordinal; existing scan errors and logs remain available. I found no new metrics or log requirement. The two pruning findings cause avoidable I/O rather than a new failure message.
• Transactions, persistence, and writes: No EditLog, visible-version, delete-bitmap, or transaction commit path changes. Parquet/ORC/Iceberg writes do persist the changed ordinal, which is the P1 compatibility issue. The reviewed write code introduces no separate atomicity or crash-recovery defect.
• FE/BE variables: No new variable crosses FE/BE paths; existing scanner selection and mixed-BE planning are sufficient to expose the compatibility issue.
• Performance and remaining issues: Per-value conversion is O(1). The ORC and Parquet findings can read many otherwise prunable stripes or pages. After checking all 19 changed paths and parallel reader paths, I found no other substantiated issue.

User focus: No additional review focus was provided.

[github-actions]

[P1] Keep new year-zero DATE files readable by the other scanner and older BEs. This writes 0000-01-01 as -719528 to Parquet (and the ORC writer now does likewise), but the v1 Parquet/ORC readers still decode through date_day_offset_dict when enable_file_scanner_v2=false: its failed daynr-zero fallback leaves 1900-01-01 as the result. A pre-upgrade v2 Parquet reader rejects the same ordinal. A newly committed Iceberg or exported file can therefore return a wrong DATE in a same-build v1 scan or fail on an older BE during a rolling upgrade. Update the legacy readers and gate the persisted encoding until older BEs are gone, with cross-reader and mixed-version coverage.

[Gabriel39]

V1 reader changes and mixed-version/rolling-upgrade compatibility are explicitly outside the requested scope of this PR. No changes are being made for this finding. The PR description now states this scope, and this thread remains open rather than being marked fixed.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Review complete for head 768c42200314322850ceb90d8f1472db8716c289 (static review only).

Findings: One new P2 inline finding: an unsafe projected ORC DATE disables SARG pruning for safe stripes in the same file. The existing P1 V1/rolling-upgrade compatibility thread still applies: the legacy Parquet/ORC readers use the old DATE offset dictionary, and older BEs cannot safely read new year-zero ordinals. I confirmed it independently and list its ID as an existing blocker instead of repeating it. Earlier DATE MIN/MAX, NULL-statistics, COUNT, unread-sibling, and Parquet-page concerns are addressed in the current code; no other new actionable finding survived two review rounds and the final 19-file sweep.

Critical checkpoints

• Goal and tests: the shared epoch-day helpers align year-zero DATE values across Arrow, V2 Parquet/ORC, and Iceberg day/bucket partitions. The added unit and regression cases cover boundaries, NULLs, invalid ordinals, nested DATEs, COUNT, and pruning. They were inspected, not run in this invocation; the external Iceberg regression has no independent runtime result here.
• Scope and parallel paths: changes are focused on DATE conversion, metadata safety, and corresponding tests. V2 read/write and partition paths were traced. The unchanged V1 readers and older BE compatibility remain the existing P1.
• Concurrency and lifecycle: no new threads, shared mutable state, locks, lock ordering, or cross-translation-unit initialization dependency was introduced. New reader scratch/statistics checks are per-call or per-reader and follow existing cleanup and error propagation.
• Configuration and protocol: no new dynamic configuration, FE/BE field, function symbol, or wire protocol was added. Existing strict-mode and scanner settings were considered in the read-path review.
• Conditions and errors: representability checks handle bounds and the interior Gregorian-only leap day; unsafe metadata falls back to materialization. Projected ORC leaves and Parquet nested COUNT/page paths were checked against their actual readers. The new P2 is the remaining overly broad ORC condition.
• Results, observability, and performance: expected output rows and golden-file contents were inspected statically. Existing stripe/page counters can show pruning behavior; the P2 case needs a mixed-stripe regression to prove safe-stripe pruning. No other concrete hot-path regression was substantiated.
• Persistence, writes, and atomicity: external DATE files and Iceberg partition values change their persisted ordinals. No new transaction or EditLog path, commit step, or crash window was introduced; the existing V1/mixed-version P1 is the material persistence risk.

User focus: none was provided. No builds, tests, submodule initialization, or repository source changes were performed.

Existing P0/P1 findings confirmed for this head: #68649 (comment)

[hello-stephen]

BE UT Coverage Report

Increment line coverage 91.62% (175/191) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	61.14% (26729/43716)
Line Coverage	45.78% (277574/606296)
Region Coverage	41.44% (219694/530151)
Branch Coverage	42.97% (101871/237066)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 92.15% (176/191) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.86% (31830/42521)
Line Coverage	59.16% (355805/601441)
Region Coverage	55.68% (295828/531298)
Branch Coverage	56.69% (134241/236785)

[Gabriel39]

run buildall

[Gabriel39]

/review

[Gabriel39]

Updated in 7c53e83 after local validation.

The external group6 failure comes from non_vec_orc_scanner.orc: its DATE payload is the invalid epoch-day ordinal 20191111. The old 1900-01-01 expected result hid a conversion failure. The regression now asserts the data-quality error and restores the scanner setting; the original file is retained as a BE fixture, including coverage that projecting other columns still succeeds.

Validation before this push:

• 247 ASAN unit tests passed, including the complete ORC reader suite and mixed-stripe pruning coverage.
• All three local integration suites passed, with no failures or skips: full group6 (41 fixtures, 39 golden-result assertions and two expected errors), Flight year zero, and Parquet/ORC outfile year zero. The original group6 failure was reproduced first.
• clang-format 16 and source whitespace checks passed.

Integration used the final changes rebuilt into an existing ASAN/UBSAN BE with an existing FE. HDFS and S3 transport were replaced by local file access for the local run; committed CI suites retain their original transport. A full branch-4.1 build and those external transports remain for CI. The separately muted S3 connection timeout also occurs on unrelated PRs and has no evidence tying it to this DATE change.

Scope remains unchanged: legacy v1 scanner changes and rolling-upgrade compatibility are excluded; that existing thread remains open.

[github-actions]

Static review of PR #68649 at head 7c53e83. I reviewed all 20 changed paths and the affected scanner, writer, aggregate, partition, and SDK paths. Three new P1 findings are inline: historical Doris DATE files read incorrectly after upgrade (M2), ORC DATE preflight exposing an unchecked row-index access and avoidable index I/O (M1), and nested ORC COUNT accessing short stripe statistics out of bounds (M4). Existing P1 comment 4140089872 still applies to newly written files read by the v1 scanner or older BEs; it is not duplicated. The other earlier P1 threads are addressed by this head.

Review status: INCOMPLETE convergence. The third and final permitted review round produced M4. I independently validated it and included it here, but the workflow prohibits a fourth convergence round. All current candidates have a clear disposition.

Critical checkpoints:

• Goal and proof: The intended year-zero DATE correction maps Doris day numbers to external proleptic Gregorian ordinals and protects metadata shortcuts. New calendar, ORC/Parquet, Flight, outfile, and Iceberg tests cover the target boundaries and read/write paths. Historical writer fixtures are absent, so M2 is untested by this PR.
• Scope and clarity: The production changes are focused on DATE conversion, external partition transforms, and metadata pruning; the additional files are targeted tests and expected output.
• Concurrency and lifecycle: No new thread, lock, shared static initializer, or cross-thread state is introduced. I traced per-reader ORC range state, active RowReaderOptions lifetime, and the pinned SDK's cached SARG evaluator through temporary and surviving readers; no separate lifecycle bug was found.
• Configuration: No new configuration item or dynamic setting is added. The existing v1 scanner switch remains a compatibility path affected by open P1 4140089872.
• Compatibility: Parquet/ORC DATE ordinals and Iceberg partitions are persisted externally. M2 affects already committed old files; the existing P1 affects new files on v1/older BEs. No FE/BE symbol, Thrift field, or transmitted variable changes are present.
• Parallel paths and conditions: Arrow, ORC, Parquet, and Iceberg writes/reads were traced, including nested DATE projection, strict/non-strict NULL conversion, unknown statistics, page/stripe pruning, and COUNT/MINMAX. M1 and M4 identify two distinct unsafe ORC metadata paths; other checked fallback conditions preserve row validation.
• Test coverage and expected results: The added boundary, nested aggregate, pruning, selected-row, Flight, outfile, Iceberg, and group6 assertions and golden rows are consistent with their inputs. Old-writer file fixtures, a malformed pruned row index, and truncated nested stripe statistics are missing. This review was static only; I ran no build or test and do not treat author/CI statements as independent validation.
• Observability: Conversion failures have ordinal context and ORC fallback paths include file context. No new metric measures the pre-pruning index I/O in M1; an I/O-count regression is warranted.
• Persistence, transactions, and writes: Existing Parquet/ORC sinks persist the changed ordinal, and Iceberg persists matching partition values. There is no new EditLog, visibility-version, commit, or failover path. The persisted-file compatibility gaps above are the material write/read risk; M1 and M4 add malformed-file memory-safety risk.
• Performance and remaining concerns: M1 adds footer/index reads and parsing for SARG-pruned stripes, potentially repeated by physical child readers. The final sweep found no other substantiated new issue. No extra user review focus was supplied.

Request changes for the three new P1 findings and the still-open existing P1.

Existing P0/P1 findings confirmed for this head: #68649 (comment)

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Static review of PR #68649 at 082429b00f2392f8e8eabcee6e6a5a94a403e7aa. The review is complete: the main scan, two full-coverage subagent reviews, two focused risk scans, candidate convergence, and final sweep found no new valuable inline finding. Existing P1 comment 4140089872 still applies and is carried below; it is not reposted. The other existing DATE correctness and safety threads were checked against this head and their fixes or migration procedure are present.

Critical checkpoints:

• Goal and proof: The patch aligns year-zero DATE conversions across Arrow, default Parquet/ORC paths and Iceberg day/bucket partitions, while rejecting external ordinals Doris cannot represent. Added unit and regression cases inspect the year-zero boundaries, NULL/nested values, aggregate and pruning paths, malformed ORC metadata, and readback. This invocation was static-only: no build, test, or runtime validation was performed; author-reported ASAN/integration results and the pending external Iceberg execution are not independent validation.
• Scope and clarity: Production edits are limited to the DATE SerDe/calendar helper, Iceberg partition transforms, and ORC/Parquet reader metadata guards. The standalone migration utility handles confirmed historical files. No unrelated data format or generic scanner contract was changed.
• Concurrency and lifecycle: The new ORC options and DATE stripe state are per reader. The pinned SDK's cached SARG evaluator is consumed while its options live; replacing active options follows RowReader reset, and range advancement rebuilds the reader. No new thread, lock, lock order, cross-TU initializer, or persistent ownership cycle was found. Migration closes/fsyncs its temporary output before exclusive publication and removes it on errors.
• Configuration and compatibility: No configuration item or FE-to-BE variable was added. Existing strict-cast and pruning settings drive these paths. The persisted DATE ordinal change is incompatible with the unchanged v1 Parquet/ORC readers and older BEs during mixed-version writes: the still-open P1 4140089872 remains blocking. Historical files written with the old encoding require provenance-based migration before switching consumers; the new offline utility and procedure address existing thread 4146799308, but do not infer origin automatically.
• Parallel paths and conditions: ORC projected DATE collection matches the physical projection; metadata-only stripe preflight retains unsafe/unknown stripes and disables row-index SARG only for those ranges. Parquet validates requested DATE leaves before COUNT shortcuts; unsafe footer/page ranges invalidate physical NULL proofs and retain page candidates. TableReader permits metadata aggregates only with the required no-filter/no-delete and mapping conditions. The existing v1 path remains the compatibility exception noted above.
• Errors, memory, and observability: Invalid DATE values return checked errors or nullable non-strict conversion NULLs. ORC errors include the column and raw ordinal. Changed metadata failures decline shortcuts or retain rows; no ignored Status or new unsafe nullable-column cast was found. Temporary buffers/options are reader-scoped. Existing stripe and I/O counters remain in use; no new critical metric was identified.
• Persistence, writes, and performance: Iceberg day and bucket values now use the same external epoch day as file payloads. No Doris EditLog, transaction, delete bitmap, or visible-version behavior changes. The migration publishes separate files and documents table-format transactional rewrite requirements. DATE preflight uses file-level stripe metadata; safe stripes/pages remain eligible for pruning. No substantiated new hot-loop or I/O amplification was found.
• Test results and focus: New .out files carry generated-file headers and ordering is specified in the changed suites. The user focus file contains no additional focus points. Review claims here are limited to source and test inspection.

Disposition: no new inline comments; carry existing blocking P1 4140089872. Review execution is complete, while the PR retains that unresolved compatibility finding.

Existing P0/P1 findings confirmed for this head: #68649 (comment)

[hello-stephen]

BE UT Coverage Report

Increment line coverage 92.98% (212/228) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	61.18% (26791/43787)
Line Coverage	45.83% (278513/607659)
Region Coverage	41.52% (220588/531344)
Branch Coverage	43.02% (102327/237870)

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 93.42% (213/228) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.92% (31903/42582)
Line Coverage	59.26% (357199/602763)
Region Coverage	55.95% (297948/532478)
Branch Coverage	56.80% (134957/237587)

1