Repository navigation
[refactor](external) Remove Hive ACID, Avro and LakeSoul readers on master - #69601
Merged
Merged
Conversation
Gabriel39
requested review from
924060929,
CalvinKirs,
englefly,
morrySnow and
starocean999
as code owners
October 26, 2026 03:00
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Reviewed PR #69601 at efc0ccc8d70d7997316cf94357dc7bcaae93726f. One P2 compatibility issue is reported inline: retain the removed ACID session-variable name as a no-op so existing session initialization continues to work. No P0/P1 findings or previously reported blocking comments were identified.
The removal otherwise rejects Hive full-ACID and insert-only reads, standalone Avro, and deprecated LakeSoul markers before ordinary reader/count execution. Supported Hive, Iceberg, Paimon and Hudi paths retain their relevant read/delete behavior.
| Required checkpoint | Conclusion |
|---|---|
| 1. Goal and evidence | Removal guards match the stated goal; new FE/BE negative tests cover removed formats and transactional modes. Tests were inspected statically. |
| 2. Scope and clarity | Reader/API deletions, fixture/sample cleanup and release-script adjustment are related to the removal. |
| 3. Concurrency and locks | No new threads, shared state, lock order or blocking critical section; supported streaming/cache paths retain existing synchronization. |
| 4. Lifecycle and static initialization | Obsolete read-transaction/reader ownership is removed with its consumers; no new static initializer, reference cycle or cleanup path. |
| 5. Configuration | No configuration added; obsolete LakeSoul test settings have no surviving consumers. The removed session variable needs the compatibility entry described inline. |
| 6. Compatibility and upgrades | Removed reads fail explicitly. One unintended SET/SELECT session-variable compatibility regression remains; native storage formats and supported function symbols are unchanged. |
| 7. Thrift/protobuf identities | All nine touched definitions were read on HEAD and branch-3.1/4.0/4.1/4.2. Existing field IDs, optional qualifiers and enum values are retained; PR edits add only comments. No protobuf changes. |
| 8. Parallel paths | Checked synchronous/streaming/batch planning, v1/v2 scanners, native/JNI dispatch, metadata counts, schema setup and row-ID fetch; no additional removal-boundary defect confirmed. |
| 9. Conditional checks | Guards have concrete removed-format failure cases and explain the transaction/merge/count invariant; ordinary ranges preserve their dispatch. |
| 10. Test coverage | New rejection cases and retained ORC/delete/cache/path/streaming tests were inspected. The removed-variable compatibility case is missing; no end-to-end removal execution was performed. |
| 11. Expected results | The obsolete acid5k query and its test_32 output were removed together; all 42 remaining regression tags align, with no new expected data. |
| 12. Observability | Rejections identify the unsupported format and propagate exceptions/non-OK Status. No new distributed operation requires metrics or logging. |
| 13. Transactions and persistence | Hive read locks/snapshots are removed behind explicit rejection; supported write-lock APIs remain. Native visible versions, MoW bitmaps and EditLog transitions are untouched; unknown removed global variables are skipped during replay. |
| 14. Writes, atomicity and crashes | No new write/commit protocol, durable state or crash-cleanup responsibility; supported load paths keep their partition/null handling. |
| 15. FE/BE propagation | No wire fields added or removed. Obsolete IDs remain reserved, with checks at scan and range boundaries. |
| 16. Performance and memory | Added validation is bounded per scanner/range without I/O or material allocations; retained ORC hot loops match the previous non-ACID paths. No new memory-tracking issue found. |
| 17. Other concerns and coverage | All 91 changed paths covered, including Docker, packaging and removed samples. No additional actionable finding. |
No additional user focus was supplied. Validation was static only: no builds, unit/regression tests, cluster execution, binary fixture decoding or final ELF/glibc runtime audit. The pure-text BE build-hygiene check passed. These execution limits are separate from completion of the required static coverage.
Review converged after 2 round(s).
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Gabriel39
added a commit
to Gabriel39/incubator-doris
that referenced
this pull request
Oct 26, 2026
### What problem does this PR solve? Related PR: apache#69601 Problem Summary: Clients that still set skip_checking_acid_version_file when initializing a connection encounter an unknown-variable error after the Hive ACID reader removal. Register the retired name in the existing removed-variable compatibility set so SET remains a no-op and both variable-read paths use the empty-string fallback. Do not restore the session field or the removed Hive read support. ### Release note Continue accepting the retired skip_checking_acid_version_file session variable as a compatibility no-op. ### Check List (For Author) - Test: Targeted FE unit tests and FE Checkstyle. - Behavior changed: Yes, preserve legacy client session initialization. - Does this need documentation: No.
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
Static review of all 93 changed files at e0ab4f990d38fd8b735eee6468be2102abae897f: no new substantiated findings. The intentional removal of Hive transactional/insert-only and standalone Avro reads is enforced by explicit errors; supported Hive scanning and Iceberg/Paimon ORC delete handling remain intact in the inspected control flow.
Existing thread 4373671401 is fixed on this head: skip_checking_acid_version_file is in the removed-variable compatibility set, both SET paths accept it as a no-op, variable reads retain the empty-string fallback, and exports omit the deleted field. No duplicate comment is posted. No independently confirmed existing P0/P1 finding remains.
| Critical checkpoint | Conclusion |
|---|---|
| Goal and test evidence | Full ACID and insert-only reads reject before scan planning; standalone Avro rejects during common format parsing. New negative/helper/compatibility unit tests exercise these contracts. Tests were inspected, not executed. |
| Scope and clarity | Removal, compatibility reservations, caller updates, and obsolete fixture cleanup are focused; one shared BE validator avoids divergent gates. |
| Concurrency and locks | No new threads, shared mutable state, lock nesting, or waits. Retained batch streaming, listing-cache ownership, and Hive writer locking are unchanged. |
| Lifecycle and static initialization | Removed read transactions are no longer opened; deleted manager/readers/constants have no remaining Doris consumers. Supported reader/delete-vector ownership is unchanged; no new static initialization dependency. |
| Configuration | No new production setting. The retired session variable uses the established compatibility mechanism; obsolete LakeSoul regression settings have no remaining consumers. |
| Symbols, formats, and rolling compatibility | Deprecated read support is intentionally removed with explicit failures. Supported storage serialization and reader interfaces remain unchanged; retired wire definitions stay reserved. |
| Thrift/protobuf IDs | Read all nine touched definitions at HEAD and branch-3.1/4.0/4.1/4.2; live branch refs matched the inspected local refs. Deprecated values/field IDs and optionality retain their meanings. All three Thrift files have unchanged non-comment tokens. No protobuf edit. |
| Parallel paths | Traced synchronous, reused, batched, streaming, metadata-count, schema-fetch, and two-phase row-fetch paths; V1/V2 validation and supported connector routing remain consistent. Legacy/Nereids load callers and primary/follower variable handling are covered. |
| Conditional checks and errors | Concrete removed-format checks have explanatory comments and propagate exceptions/non-OK Status before data access or metadata-count handling; no supported read falls through an ACID-only branch. |
| Test coverage | Negative FE/BE helper tests and session compatibility tests cover the changed contracts; retained ordinary Hive streaming/partition and ORC delete/cache tests remain. Runtime and end-to-end proof are outside this static invocation. |
| Changed expected results | The ORC regression output removes exactly the block for the deleted active query; remaining expected values are unchanged. No new golden values are introduced. |
| Observability | Format-specific unsupported errors identify the rejected feature; retained scan profiles/logs are preserved. No new operational execution path requires extra metrics. |
| Transactions and persistence | Hive read snapshots/shared-lock lifecycle is removed behind the rejection gate. Nontransactional Hive write exclusive-lock, heartbeat, and finally-release APIs remain. No Doris EditLog, rowset, visible-version, or delete-bitmap persistence change. |
| Writes, atomicity, and crashes | No retained publication or crash-recovery ordering is changed; ordinary load/write paths retain their existing behavior apart from rejection of the removed format. No new resource leak identified. |
| FE/BE variable transport | No new transported variable. The removed ACID flag is absent from exports/forwarding; retained marker IDs/fields are not reused. |
| Performance and memory | Removed ACID RPCs, directory resolution, row filters, and allocations disappear. Added checks are constant-time; supported lazy/eager ORC logic, split sizes, caches, and streaming bounds are preserved. No new untracked large allocation. |
| Other issues and cleanup | Deleted-symbol/build-list/fixture references, release extension enumeration, Hive HQL discovery, nested-column/null/row-position behavior, position deletes, deletion vectors, and condition-cache exclusion were rechecked. No additional issue found. |
User focus: no additional focus was supplied. Initial risks about rejection shortcuts, shared ORC semantics, and compatibility/dependency closure were dismissed with concrete control-flow evidence. No unresolved candidate or uncovered static scope remains.
Convergence: converged after 2 review rounds, with both complete reviewers and the independent risk reviewer returning NO_NEW_VALUABLE_FINDINGS in the final round. Validation limit: static inspection only; no builds, tests, runtime checks, or source changes were performed. The author's reported test results were not independently executed by this review.
Review converged after 2 round(s).
Contributor
|
run buildall |
Contributor
TPC-H: Total hot run time: 28832 ms |
Gabriel39
added a commit
to Gabriel39/incubator-doris
that referenced
this pull request
Oct 26, 2026
### What problem does this PR solve? Related PR: apache#69601 Remove the Elasticsearch HTTP scan implementation and its FE connector, ES-only functions, HTTP proxy endpoints, test services, and regression fixtures. Reject retired formats before reader creation or metadata-only counts. Keep the old catalog metadata readable during image loading and journal replay, but reject catalog creation and access even when an obsolete plugin remains. Preserve all Thrift definitions and numeric identities. Accept the retired ES scroll session variable as a compatibility no-op for old clients. ### Release note Elasticsearch HTTP scans and the esquery function are no longer supported. Existing Elasticsearch metadata can still load during an upgrade. ### Check List (For Author) - Test: Focused FE unit and upgrade tests, FE Checkstyle, clang-format 16, BE syntax-only compilation, build hygiene and shell syntax checks. - Behavior changed: Yes, remove Elasticsearch reads. - Does this need documentation: No; removal is described in the release note.
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Gabriel39
changed the title
Contributor
There was a problem hiding this comment.
Static review of PR #69601 at 457bc77 found no new verified defects. All 237 paths in the supplied aggregate diff were covered by the main review and the assigned BE/infrastructure, FE/protocol, and independent risk reviews.
Existing thread disposition: P2 comment 4373671401 is fixed. skip_checking_acid_version_file now uses the removed-variable compatibility set, so both SET entry points accept it as a no-op and legacy/Nereids variable reads keep the empty-string fallback. The ES setting receives the same treatment. No duplicate comment is proposed; the supplied context contains no existing P0/P1 finding to carry forward.
Core invariants: the change does not modify partition-version visibility or MoW bitmap protocols. Traced shared ORC paths retain Iceberg/Paimon position-delete and deletion-vector filtering. Retired reads propagate explicit failures, and the added validation introduces no long-lived memory owner.
Critical checkpoint conclusions:
- Goal and proving tests: the traced FE planning/property and BE scanner/range paths reject the retired external reads. Negative gate, catalog replay, session compatibility, and surviving Hive/ORC tests were inspected; they were not executed in this review.
- Scope and clarity: implementation, build, service, fixture, and suite removals follow the retired features, with a shared BE rejection helper. No unrelated functional change was identified.
- Concurrency and locks: new validation reads existing parameters without adding threads, shared mutable state, locks, or lock-order edges. Supported Hive write locking/heartbeat/release and existing split/cache synchronization remain.
- Lifecycle and static initialization: removed read-transaction/client owners have no surviving consumers. Legacy ES metadata stays lazy; rejected directory providers are discarded. No new ownership cycle or cross-translation-unit initializer dependency is introduced.
- Configuration: two ES-only BE settings are removed, and both retired FE session names remain compatibility no-ops. No new dynamic configuration or propagation requirement is added.
- Compatibility and rolling upgrades: the external-read removals are deliberate and release-noted. ES image aliases and journal placeholders remain loadable, while first access rejects ES even with an obsolete provider installed. Legacy scan markers fail explicitly.
- Thrift/protobuf: all three touched Thrift files change comments only. Touched definitions were compared with live-matching branch-3.1/4.0/4.1/4.2 refs; field/enum meanings, IDs, types, and optionality remain compatible. No protobuf change.
- Parallel paths: synchronous, batch, streaming, scan-property, V1/V2, metadata-count, schema-discovery, and row-id fetch routes were traced. Full ACID and insert-only are both gated. Supported Paimon JNI reads and Iceberg/Hudi Avro metadata dependencies remain available.
- Conditional checks: retirement guards document concrete visibility/merge/count failure scenarios. Scan/range marker checks and effective format precedence match the traced producers and readers.
- Test coverage: focused negative tests cover removed formats at scan/range levels, both transactional modes, ES catalog creation/replay, and removed settings; supported scan/partition/write tests retain their core behavior. Runtime and assembled-artifact validation remain for CI.
- Test results: the ORC golden-output deletion matches the removed test_32 query; removed ES outputs accompany removed suites. No new expected result values were added. Deleted binary Avro fixtures have no textual payload in the supplied diff; their removed consumer and provisioning dependencies were checked.
- Observability: retirement errors identify the unsupported source/format; supported reader counters and logging remain. No new hot logging or critical runtime operation requiring additional metrics was identified.
- Transactions and persistence: retired Hive read transactions cannot open through the guarded planning paths. Nontransactional HMS commit locking remains, ES image/journal loading is retained, and no new EditLog mutation or publish protocol is added.
- Writes and atomicity: supported load, file-output, and storage-write logic is preserved. No new partial-write or crash-cleanup path is introduced.
- FE/BE variables: no new transmitted variable or numeric protocol value is introduced. Retired carriers remain reserved, and both scan-level and per-range representations are checked.
- Performance and memory: added checks are bounded string/enum comparisons per scanner/range, with no per-row work, large allocation, or additional locked I/O. Surviving plain-Hive listing/splitting and non-ACID ORC filtering retain their prior structure.
- Further-issue sweep: removed-symbol/build/service/fixture dependencies, Hive provisioning, empty/pruned scans, stale providers, schema RPCs, row-id markers, and delete-aware condition-cache behavior were rechecked without a substantiated additional defect.
User focus: no additional focus points were supplied. R1-R4 are dismissed with recorded evidence; there are no unresolved code candidates or unassigned changed paths. The review converged in one round: all three assigned reviewers returned NO_NEW_VALUABLE_FINDINGS and the final main sweep confirmed the dispositions. This was static-only: no builds, tests, Docker startup, release artifact/ELF audit, or live upgrade validation was performed.
Review converged after 1 round(s).
Gabriel39
force-pushed
the
dev/pick-69312-master
branch
from
October 26, 2026 05:58
457bc77 to
e0ab4f9
Compare
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Gabriel39
changed the title
Contributor
There was a problem hiding this comment.
Reviewed all 93 changed paths at e0ab4f9. No new actionable findings.
The existing P2 session-variable compatibility comment is fixed at this head: the retired name is accepted by both SET paths, both variable-read paths preserve the empty-string fallback, and the added test covers case variants and export exclusion. It is not reposted. No existing P0/P1 finding remains to carry forward from the supplied threads. No additional user focus was supplied.
| Critical checkpoint | Conclusion |
|---|---|
| Goal and correctness | Hive full-ACID and insert-only reads reject before reuse, scan properties and file planning. Standalone Avro and retired LakeSoul markers fail explicitly. Negative unit assertions are present. |
| Scope and reuse | Deletions, shared rejection helper, compatibility handling and associated fixture cleanup form a focused removal. |
| Concurrency and locks | No new threads, locks or shared mutable state. Retained streaming/listing synchronization and standalone Hive write locks are preserved. |
| Lifecycle and static initialization | Removed read-transaction owners and callers disappear together; retained filesystem/reader ownership remains intact. No new static-initialization dependency. |
| Configuration | No new setting. The retired ACID session name remains a compatibility no-op; obsolete LakeSoul test settings are removed. |
| Behavioral and format compatibility | Read-support retirement is intentional. Ordinary Hive, Iceberg/Paimon ORC deletion handling and Iceberg/Hudi Avro metadata consumers retain their independent paths. |
| Thrift/protobuf | All three Thrift changes are comments only. All nine touched definitions were compared at HEAD and current branch-3.1/4.0/4.1/4.2 refs: IDs retain their meanings; no new required field or reuse. No protobuf change. |
| Parallel execution paths | Checked V1/V2, initial/later ranges, metadata counts, row-ID fetch, synchronous/reused/batched/streamed Hive planning, and both legacy/Nereids load callers. |
| Conditional checks and errors | New checks address concrete retired payloads and propagate explicit failures before reader creation; retained non-ACID partition/NULL parsing is equivalent to the old false-ACID branch. |
| Test coverage | Inspected added full-ACID/insert-only, Avro/LakeSoul and session-compatibility cases, adapted supported-path tests, and retained ORC position-delete/cache tests. No tests executed. |
| Expected results | The removed ORC test_32 query matches its removed output block. No new expected result values are introduced. |
| Observability | Errors identify the retired format. Removed ACID counters belong to the removed feature; no new distributed work needs metrics. |
| Transactions and persistence | Read-side HMS snapshots/locks are removed after FE rejection. Retained Hive write lock/heartbeat/release operations and Doris EditLog/visibility/MoW paths are unchanged. |
| Writes, atomicity and crash cleanup | No production data-write behavior is added or changed; no new crash-cleanup ownership. |
| FE/BE state passing | No new transmitted variable. Deprecated definitions remain available; BE validates both scan-level and per-range markers. |
| Performance and memory | Checks are bounded per scan/range, outside row loops. Removed ACID state/filter work adds no retained-path allocation or memory owner. |
| Further recheck | Rechecked stale class/header/build/fixture references, shared ORC lazy/eager/no-predicate deletion and condition-cache handling, late materialization, mixed-version requests and metadata Avro dependencies. No remaining concrete candidate. |
Static review only, as required: no builds, tests, source edits or runtime validation. Compilation, full linking, final-artifact glibc compatibility and end-to-end/mixed-binary execution were not verified. These are validation limits; there is no unreviewed changed path or unresolved code candidate.
Review converged in 2 rounds after the main review, two complete-scope reviewers and a separate risk reviewer, including recheck of this frozen summary and empty new-comment set.
Review converged after 2 round(s).
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
4 of 9 tasks
Contributor
TPC-H: Total hot run time: 28767 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-DS: Total hot run time: 151954 ms |
morningman
previously approved these changes
Oct 26, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
924060929
previously approved these changes
Oct 26, 2026
Contributor
FE Regression Coverage ReportIncrement line coverage |
Gabriel39
added a commit
that referenced
this pull request
Oct 26, 2026
…#69614) ### What problem does this PR solve? Related PR: #69601 Problem Summary: Elasticsearch HTTP scans always use FileScanner v1, even when `enable_file_scanner_v2` is enabled. Add an ES TableReader/FileReader pair and route `FORMAT_ES_HTTP` queries through FileScannerV2 when that session option is enabled. The reader exposes a file-local schema, uses MaterializedColumnMapper for projection and predicate localization, and applies residual filters with v2 row accounting. It reuses the existing ES query builder, HTTP scroll transport, authentication, SSL settings, catalog-provided endpoints, and type conversions. Shared host selection and materialized complex-schema helpers keep the two scanner paths consistent. Parse the initial cached response during open so early LIMIT/cancellation can clear its scroll context. Close the stream on EOF, cancellation, open/read errors, and split changes, and report close failures once. The RLE header reach baseline increases by two for the new ES reader and test translation units, which use the existing table/expr header closure. The shared parser retains its existing type dispatch, with a documented complexity suppression on the new projection overload. ### Release note Support Elasticsearch HTTP queries in FileScannerV2 when `enable_file_scanner_v2` is enabled. Disabling the option continues to select FileScanner v1. ### Check List (For Author) - Test - [x] Unit Test: 61 ASAN BE tests covering ES v2 projection, doc values, `_id`, nullable values/arrays, multi-page batching, zero-column row counting, residual filtering, split advancement, cancellation, error cleanup, and a real local HTTP scroll server; existing scanner dispatch, legacy ES, and Remote Doris tests also pass. - [ ] Regression test: added ES 5/6/7/8 v1/v2 differential coverage with source/docvalue scans and small batches. Groovy syntax checked; live external ES regression not run locally. - [x] clang-format 16, clang-tidy on changed C++ lines, and BE build-hygiene checks. - Behavior changed: - [x] Yes. ES queries honor the v2 session option. - Does this need documentation? - [x] No. Uses the existing scanner session option and ES catalog properties. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
Contributor
FE UT Coverage ReportIncrement line coverage |
…aster ### What problem does this PR solve? Related PR: apache#69312 Problem Summary: Port the removal of unsupported external readers to master. Adapt Hive transaction cleanup to the connector modules and preserve the unified reader initialization interfaces. Reject deprecated reads before file access or metadata-only count shortcuts, and remove obsolete fixtures and examples. Preserve Iceberg and Hudi Avro metadata dependencies. ### Release note Remove Hive transactional table reads, including insert-only tables, and standalone Avro file reads. LakeSoul remains unsupported. Iceberg and Hudi Avro metadata parsing is preserved. All Thrift definitions and IDs remain unchanged. ### Check List (For Author) - Test: Hive connector and HMS unit tests; targeted FE core unit tests; FE Checkstyle; clang-format 16; BE syntax checks; build hygiene; shell syntax and Thrift definition checks. - Behavior changed: Yes, reject unsupported external reads explicitly. - Does this need documentation: No, this ports the existing removal. (cherry picked from commit 518be1a, adapted to master)
### What problem does this PR solve? Related PR: apache#69601 Problem Summary: Clients that still set skip_checking_acid_version_file when initializing a connection encounter an unknown-variable error after the Hive ACID reader removal. Register the retired name in the existing removed-variable compatibility set so SET remains a no-op and both variable-read paths use the empty-string fallback. Do not restore the session field or the removed Hive read support. ### Release note Continue accepting the retired skip_checking_acid_version_file session variable as a compatibility no-op. ### Check List (For Author) - Test: Targeted FE unit tests and FE Checkstyle. - Behavior changed: Yes, preserve legacy client session initialization. - Does this need documentation: No.
Gabriel39
force-pushed
the
dev/pick-69312-master
branch
from
October 26, 2026 13:45
e0ab4f9 to
31d7e91
Compare
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
There was a problem hiding this comment.
Static review of #69601 at 31d7e91: no new actionable findings.
The existing P2 thread 4373671401 is fixed: the retired skip_checking_acid_version_file name now uses VariableMgr's compatibility no-op for master/follower SET and empty-string variable reads, and stays out of session export. No existing P0/P1 finding remains to carry forward. No additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and proof: full-ACID and insert-only Hive reads are rejected at connector planning entrances; standalone Avro is rejected by the file-format factory and BE support checks; deprecated Hive/LakeSoul markers cannot become ordinary data or metadata-count reads. New unit tests encode rejection and legacy-session compatibility. Runtime behavior was not tested in this review.
- Scope and clarity: implementation changes remove retired readers/transactions and update their callers; obsolete LakeSoul services, fixtures and samples are removed consistently.
- Concurrency and locking: no new workers, shared mutable state or lock ordering; ordinary listing/cache/streaming synchronization is preserved. Surviving Hive write acquisition, heartbeat and lock release are unchanged.
- Lifecycle and initialization: read-transaction ownership disappears with its callers; filesystem borrowing and remaining reader/cache lifetimes are preserved. No new static-initialization dependency or ownership cycle found.
- Configuration: no new configuration. The removed ACID session setting retains compatibility; removed LakeSoul regression settings have no remaining consumer.
- Compatibility: existing session initialization remains accepted. Deleted internal read APIs have no remaining repository callers; retained Avro metadata dependencies for Iceberg/Hudi are unaffected. Unsupported reads are an intentional documented behavior change.
- Thrift/protobuf: all three touched Thrift files differ only in comments. Touched definitions were compared on HEAD and branch-3.1/4.0/4.1/4.2; field IDs, types, enum meanings and optionality remain compatible. No protobuf changes.
- Parallel paths: synchronous, reused, partition-batch and streaming Hive planning; BE v1/v2; row-id fetch; both load planners; eager/lazy ORC predicates; Iceberg position deletes and Paimon deletion vectors were checked. Retained ordinary read paths are preserved.
- Conditions: the transactional guard explains why both Hive layouts need visibility handling; BE guards precede range pruning/reader/count setup. Removing ACID branches preserves the former non-ACID behavior.
- Test coverage: added JUnit/BE unit cases cover unsupported inputs and session compatibility; retained scan/cache/partition tests cover ordinary paths. End-to-end rejection and runtime delete behavior remain CI validation responsibilities under this static-only contract.
- Test results: the removed ORC golden-output block matches the removed query; no replacement expected-result values are introduced. Test code and results were inspected, not executed.
- Observability: removed formats fail explicitly with actionable unsupported-read messages; retained scan profiles/logs remain. No new distributed operation requiring metrics was added.
- Persistence and failover: no EditLog, storage publication, visible-version or failover state machine changes; the removed-variable compatibility path covers old names.
- Writes and atomicity: no data-write/commit algorithm changes; nontransactional Hive writes retain their independent standalone HMS locks, and transactional writes remain rejected. No introduced write/crash atomicity issue found.
- FE-BE propagation: no new transmitted variable or wire field; deprecated definitions remain reserved. Scan-level/range checks reject retired markers while supported table formats retain their payload handling.
- Performance and other invariants: no new large allocation, untracked buffer, hot-loop copy or repeated listing introduced. Existing ORC row-position/delete/cache behavior is retained; build.sh no longer enumerates avro-scanner, so removing its obsolete release-build ignore argument is consistent.
Coverage: all 93 changed paths and relevant upstream/downstream mechanisms were inspected. No unresolved candidate or uncovered review scope remains. Validation was static only: no builds, tests, service startup or source modifications were performed.
Convergence: true after 1 review round; both complete reviewers and the additional rejection/ORC risk reviewer returned NO_NEW_VALUABLE_FINDINGS on the final review set.
Review converged after 1 round(s).
Contributor
TPC-H: Total hot run time: 28615 ms |
Contributor
TPC-DS: Total hot run time: 151149 ms |
Contributor
ClickBench: Total hot run time: 25.11 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
924060929
approved these changes
Oct 27, 2026
This was referenced Oct 27, 2026
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 31, 2026
… in the suite comment ### What problem does this PR solve? Issue Number: None Related PR: apache#61141, apache#69601 Problem Summary: The header comment of test_tvf_avro_format_rejected said that apache#69601 removed the BE Avro reader. On master the reader was removed by apache#61141; apache#69601 made FE reject format=avro during analysis and added a BE-side guard. Correct the PR number in the comment. Comment only. ### Release note None ### Check List (For Author) - Test: No need to test (comment-only change); the suite still passes the offline Groovy parse check - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
morningman
added a commit
that referenced
this pull request
Oct 31, 2026
…69849) ### What problem does this PR solve? Issue Number: None Related PR: #69601, #64699 Problem Summary: **In short.** #69601 made FE reject `format=avro` while it parses file format properties. This PR adds end-to-end coverage for that: a regression suite that runs `s3()` and `hdfs()` with `format=avro` and checks that they fail during analysis, and a unit test for the two table-function constructors. Test only, no code change. **Background.** Master no longer reads standalone Avro files: #61141 removed the BE reader, #65733 the scanner module, and #69601 made `FileFormatProperties.createFileFormatProperties` throw `Avro file reads are no longer supported`. Before #69601, FE accepted the format and the query failed later in BE with `Not supported file format in fetch table schema: 12`. branch-4.2 still reads Avro through the JNI scanner (#64699 fixed a crash in it), so users who upgrade from 4.x with Avro table functions or loads get this error. **What was missing.** #69601's unit test calls `createFileFormatProperties` directly. Nothing checks that a table function reaches that check before it validates storage properties or lists files. If a constructor validated the location first, `format=avro` against an unreachable location would report a storage error instead, and no test would notice. **What this PR adds.** - `external_table_p0/tvf/test_tvf_avro_format_rejected`: `s3()` and `hdfs()` with `format` set to `avro` and to `AVRO`, against a made-up bucket, made-up credentials and an HDFS address nothing listens on. Each call must fail with `Avro file reads are no longer supported`. - `UnsupportedExternalReadsTest#testAvroTableFunctionsAreRejectedBeforeStorageAccess`: constructs `S3TableValuedFunction` and `HdfsTableValuedFunction` with `format=avro` / `AVRO` and checks the same message. **Results.** On master `2a4191c1dc9` with this change, all four statements of the suite fail during analysis with `Can not build s3(): Avro file reads are no longer supported` / `Can not build hdfs(): ...`, and the unit test passes. Before #69601 the same `s3()` call would have gone on to validate and list the made-up location, so the suite pins the order as well as the message.
yiguolei
pushed a commit
that referenced
this pull request
Nov 6, 2026
### What problem does this PR solve? Issue Number: None Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110 #67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690 #68813 #68815 #68707 #68649 Problem Summary: **In short.** This ports the Lance BE read path of branch-4.1 / branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus the BE side of three 4.x changes merged after that snapshot: the search filter `prefilter` flag and profile layout (#68813), distance bounds for `vector_search` (#68815), and the split-level index rule of #68707 that #68815 relies on. It gathers the BE parts of the PRs listed above. Master links `liblance_c` but has no Lance reader, so every `FORMAT_LANCE` scan fails today. With this PR the BE reads Lance datasets through FileScannerV2: plain scans, vector search and full-text search, predicate and runtime-filter pushdown, the schema fetch of a local Lance TVF, and the second phase of TopN lazy materialization. The PR is BE only. The FE connector plugin and the regression suites come in separate PRs, so until then nothing on master plans a Lance scan, and the BE unit tests carry the proof. Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju @FANNG1 **Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` / `s3()` with `format=lance`, and the `vector_search` / `full_text_search` TVFs. On master the FE side is being rebuilt as a connector plugin (`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5` (`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added after the snapshot, so a BE built from master also reads the Lance scan ranges and search requests that a 4.x FE sends during a rolling upgrade. The one exception is `TVectorSearchOptions` 6, branch-4.2's `approx_mode` (#69852, merged into branch-4.2 after the snapshot and not ported): it is reserved in a comment, and a master BE ignores it, so during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate` gets Lance's default (`normal`). **The problem, and what it cost.** | | master today | with this PR | |---|---|---| | A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan fails with "FileScannerV2 does not support file format" | read by `LanceTableReader` in FileScannerV2 | | `fetch_table_schema` for a local Lance TVF | not handled | returns the Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` | | The row location of a Lance row in TopN lazy materialization | `GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a 64-bit id within the snapshot | a version 1 layout carries the 64-bit row id; phase two reads the rows with the dataset's `take_rows` | **How this PR fixes it.** - Reader: `be/src/format_v2/lance/*` and `be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one local renamed for master's `-Wshadow`. The unit tests construct `RuntimeState(TQueryOptions(), globals)`, because master removed the globals-only constructor. - FileScannerV2: the reader dispatch and `_to_file_format` handle `FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two exhaustive switches. Since #69723 every file scan uses FileScannerV2, so nothing else routes Lance scans (on branch-4.2 a Lance query fails with NotSupported when `enable_file_scanner_v2` is off). - Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0, LANCE_DATASET_ROW_ID = 1}` and a union layout for `GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2 varint, so the wire format is compatible). The materialization operator decodes row locations by version, rejects an unknown version or a wrong size, and stamps `row_location_version`. `RowIdStorageReader` validates the counts and the version, range-checks internal row ids against `uint32`, and groups Lance rows by plan node, snapshot version and dataset URI, so each dataset is read with one `take_rows`. Master's segment batching, scan-slot dedup and fetch concurrency stay as they are. - Thrift (`PlanNodes.thrift`): the external search request body (`TVectorElementType`, `TVectorMetric`, `TSearchVector`, `TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`, `TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`, `TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`), `TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in comments and added 6 (`paimon_options`); this PR replaces those comments with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is reserved in a comment for branch-4.2's `approx_mode` (#69852). - After the snapshot (one commit each): - #68813: `prefilter` decides whether the TVF search filter runs before (true) or after (unset / false) candidate selection, for vector and full-text search. The profile groups the Lance counters under `LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan` and `LanceDataCache`, uses the 4.x counter names, and registers every scan metric before the statistics callback is installed, so the callback, which Lance invokes from its execution streams, never adds counters. #68813's FE tweak that hides `TUnit.NONE` labels in `AggCounter` belongs to the FE port. - #68707 (part): a vector split searches exactly the index segments FE planned for it, and a split without segments is searched flat whatever `use_index` says; a split with segments under `use_index = false` is rejected. 4.x FEs plan segments only when `use_index` allows them. - #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) are validated (finite FLOAT values, lower < upper at FLOAT precision, single-vector only) and installed with `lance_scanner_set_distance_range()`, which master's pinned lance-c provides. - Review fixes: - A nested array goes to the Doris SerDes without a copy only when it and every descendant start at offset 0 and reference exactly their child ranges. Lance's legacy (v1) decoder can return a struct or list that is tight at its own level over a Boolean child with a bit offset (a batch that starts mid-byte in the bitmap), and the SerDes reject that under Arrow input validation. - A zero-field `STRUCT<>`, which Lance decodes, is reported as an unsupported column at any nesting level, like the other unsupported Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot hold (its row count comes from its first child). - A Lance Decimal256 field with precision <= 38 is narrowed to Decimal128 by a safe Arrow cast before decoding. Doris types it DECIMAL32/64/128I by precision, and those SerDes read only Decimal128 arrays; the BE used to crash on the first value, also for nested leaves. Only values a reader can see are cast: before a nested leaf is narrowed, every child slot that no visible row references (under a NULL list, fixed-size-list, map or struct row, or outside a sliced parent) is marked NULL, so a hidden value does not fail the cast and a visible value outside the declared precision still does; map keys, which cannot be NULL, stay checked. - FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows that Doris keeps. - Runtime filters that mention INT64_MIN stay residual-only. The pinned Lance SQL parser reads `-9223372036854775808` as the negation of a magnitude that it parses as i64 first; the overflow turns the literal into a Float64, which the planner cannot coerce to Int64 or Decimal, so a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal column failed planning and the query failed. Every other i64 is still pushed. - The shared Arrow DATE32 decoding (`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside 0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`, skips null slots and adds the epoch in 64 bits: the DATE32 range part of branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and paimon-rust reads. - Child values hidden by a NULL list, fixed-size-list, map or struct row are not decoded: the array, map and struct SerDes give the element SerDe a validity bitmap that also clears those slots (`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden 10000-01-01 under a NULL fixed-size list used to fail the DATE32 range check above. Visible values are unchanged; this applies to ADBC, Arrow stream load and paimon-rust as well. The masks come from ExecEnv's tracked Arrow pool. A nested Lance array that is compacted before decoding is masked the same way, at every level, before its `ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision decimal, Date64 or time no longer fails the batch while a visible one still does. Since #69697 the same masks keep hidden DATETIMEV2 / TIMESTAMPTZ values out of its range checks. A mask costs at most one pass per batch: a NULL row that owns no child slots, or only NULL ones, allocates nothing, and Arrow stream load masks each retained IPC batch once (`mask_arrow_descendants_not_visible()`) before it hands row ranges to the SerDes, so its per-block reads allocate nothing. The one per-block mask left is for map keys under a NULL map row that still owns entries, since a MapArray cannot hold NULL keys; Arrow's map builder writes NULL map rows without entries. - Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The nullable serde reads an Arrow Null array as NULLs without touching buffers that the array does not have. - Configs, caches and metrics, with the 4.x names and defaults: | Config | Default | |---|---| | `lance_index_cache_size_bytes` | 10 GiB | | `lance_metadata_cache_size_bytes` | 1 GiB | | `enable_lance_data_cache` | true | | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` | | `lance_data_cache_disk_capacity_bytes` | 100 GiB | | `lance_data_cache_read_block_size_bytes` | 1 MiB | | `lance_io_buffer_size_bytes` (mutable) | 2 GiB | | `lance_batch_readahead` (mutable) | 5 | | `lance_fragment_readahead` (mutable) | 5 | The BE-wide Lance session (index cache, metadata cache and the optional Foyer disk cache) is created by the first Lance scan, so a BE that never reads Lance creates no cache directory. The first six configs take effect after a restart. A negative cache size (and, while the data cache is enabled, a negative disk capacity or read block size) fails the first Lance scan with a message that names the setting, as does any data-cache value lance-c rejects; the rest of the BE keeps serving. Ten `lance_session_*` metrics report the two in-memory caches. With the defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from the first Lance scan on (100 GiB in blocks of twice the read block size), the same as on branch-4.1 / 4.2. Size `lance_data_cache_disk_capacity_bytes` and `lance_data_cache_read_block_size_bytes` against the BE's open-file limit, or set `enable_lance_data_cache = false`. The index and metadata caches live in lance-c memory: they are not charged to a MemTracker or registered with CacheManager, so memory GC, the cache capacity adjustment under memory pressure and `/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache = false` does not turn them off. lance-c has no call to clear, evict or resize them, so size them with the two configs. The disk tier keys blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS, without the endpoint) and the object path (`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve the same bucket name and data-file path with different bytes (data files registered under chosen names or replaced in place; Lance's writers use random file names) share entries, also across a BE restart; such a deployment should set `enable_lance_data_cache = false`. Both are as on branch-4.1 / 4.2 and tracked as follow-ups. A Lance scan's native working set, that is the I/O buffered up to `lance_io_buffer_size_bytes` per scanner, the decode readahead and the operators of a search plan such as the top-k of a flat vector search, is allocated inside lance-c and is charged to neither the query nor its workload group. It counts toward the BE process memory only, so memory GC, which ranks queries by tracked memory, does not attribute it to the Lance query. lance-c accepts no memory budget, so on BEs with little memory headroom size `lance_io_buffer_size_bytes`, `lance_batch_readahead` and `lance_fragment_readahead` for the number of concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a follow-up. - Build: macOS links `-framework IOKit`. This is the first time objects of `liblance_c.a` are pulled into `doris_be`, so the binary and the link time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and `util/threadpool.h` 589, the same as master `315c2e00331` alone (limits 275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562 (master alone 559, limit 660); the reach budgets are unchanged. Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`); the FE side, including the rolling-upgrade guard that keeps Lance lazy materialization off while an old BE could decode a version 1 row location as version 0; the regression suites; the rest of #68707 (searching historical versions, tags and branches, rejecting fragment ids missing from the snapshot, naming the snapshot in errors and the profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that #67005 carried goes in a separate PR. **Results.** | | master | with this PR | |---|---|---| | A `FORMAT_LANCE` query | fails before reading anything | read by `LanceTableReader` in FileScannerV2 | | A Lance row in the second phase of TopN lazy materialization | cannot be represented | a version 1 row location, read back with `take_rows` | | A row location of an unknown version or size in phase two | decoded as version 0 | rejected with an error | | An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read fails with `InvalidArgument` | BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run, 1027 pass and 2 Parquet microbenchmarks skip by design. They cover the Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10, `LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the row-location protocol (`MaterializationSharedStateTest`, `RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`: count mismatch, unknown version, a Lance version on an internal mapping, an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`, `ParquetScanTest` and the other `Parquet*` suites), whose row ids now go through `file_local`; and the Arrow / protobuf / DATE serde suites with the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader suites that decode Arrow dates. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest` 10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310 pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The ninth review round adds `LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and `LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`: both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto `339e087b11f`, and restoring `parse_int64()` in either check makes exactly these two fail. On the head rebased onto master `315c2e00331`, with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest` (15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`, `AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest` and the Lance suites. Allocating at the first NULL row of each call again fails the new `ArrowParentNullsTest` case and the four new `LoadArrowPipeTest` block cases, dropping the per-batch mask fails the LIST and STRUCT block cases, and returning the child unmasked fails the new timestamp case, four `ArrowParentNullsTest` cases and `LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`. ### Release note Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They take effect once the FE can plan Lance scans. Reading an Arrow DATE32 value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or paimon-rust now fails instead of returning 0000-00-00. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - `run-be-ut.sh --run --filter=...` with the suites listed under Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the rebases, and the Lance and Arrow SerDe suites pass again with the nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the scanner selection and in `fetch_table_schema`, which now hands every file format to `format::fetch_file_schema()`, so the Lance branch runs ahead of that call; #69646 added a TopN row-group test that reads the version 0 row location fields, which now go through `file_local` (the earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the scanner selection, the `FileFormat` enum, its test and the reach budget table). The 95 TUs that this PR changes or that master changed since `5caaeb3e3d4` were syntax-checked against the rebased tree with the UT build flags, and CI runs the suites again. Rebased again onto master `8d88f1cc6f7`: the only conflict was the budget table, where #65858 moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that this PR changes or that master changed since `43f9ceb48d7` pass the same syntax check. Rebased again onto master `339e087b11f`: the only conflict was the budget table again, where #67292 moved the `util/rle_encoding.h` baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent masking in the map and struct readers and merged cleanly. Of the 114 sources that this PR changes or that master changed since `8d88f1cc6f7`, 111 pass the same syntax check. The other three are master's: two new #67292 files under `storage/segment` fail the same way on master alone, since they compile only inside their unity batch, and this PR adds no source to that target; the #69818 benchmark is not in the UT build and includes only `util/simd` headers that this PR does not touch. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. Rebased again onto master `378404eeb12`, which removed the legacy file scanner (#69723): the conflicts were in the scanner-selection code and tests that #69723 deleted, so this PR's Lance lines there went with them (including its `LanceQueriesAlwaysUseScannerV2` test), in the profile keys of `materialization_opertor.cpp`, and in the row fetch of `rowid_fetcher.cpp`, whose non-Lance path is now only `FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests took this PR's `file_local` rename with them, and the `util/rle_encoding.h` budget commit is dropped because master is no longer near that limit. Of the 100 sources that this PR changes or that master changed since `339e087b11f`, 99 pass the same syntax check; the other, `format_v2/orc/orc_reader.cpp`, fails the same way on master alone. On the rebased head the suites listed under Results run 1029 cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that `DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master. Rebased again onto master `315c2e00331`: the only conflict was `TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had reserved fields 2-5 in comments; the two commits that add the fields replace those comments, and field 6 stays #70075's `paimon_options`. #69717 cut transitive includes from shared headers: of the 160 sources that this PR changes or that master changed since `378404eeb12`, 156 pass the same syntax check; the two `jemalloc_hook` tests have no compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc aliases that macOS cannot build (#69819), and `format_v2/table_reader.cpp` alone fails the same way on master alone while its unity batch compiles. The full `doris_be_test` builds and links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases) pass before the mask fix, and the 328 cases above with it. - `build-support/check-build-hygiene.sh` (on this branch rebased onto master `315c2e00331`), clang-format 16 and `build-support/run-clang-tidy.sh` on the changed files: all pass; on the changed lines clang-tidy (llvm 22) reports only `readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of the masking functions, the macro's expansion that master's existing `DORIS_CHECK`s report too. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [ ] No. - [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN lazy materialization rejects row locations of an unknown version or size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day outside the Doris DATE range fails the read instead of becoming `0000-00-00`. - Does this need documentation? - [x] No. Lance is documented with the FE connector. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: zy-kkk <zhongyk10@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #69312
Port #69312 from
branch-4.2to currentmaster.skip_checking_acid_version_filename as a compatibility no-op for client session initialization, using the existing empty-string fallback for variable reads.Release note
Remove support for Hive transactional reads, including insert-only tables, and standalone Avro file reads. LakeSoul remains unsupported. Iceberg and Hudi Avro metadata parsing remains supported. The retired
skip_checking_acid_version_filesession variable remains accepted as a compatibility no-op.Validation
git diff --checkpassed.Check List (For Author)