Repository navigation
[feature](external) Support Elasticsearch HTTP scans in FileScannerV2 - #69614
Merged
Merged
Conversation
### What problem does this PR solve? Related PR: apache#69601 Problem Summary: ES HTTP queries always select FileScanner v1. Add a v2 TableReader/FileReader that honors the scanner session option, projects file-local columns, applies localized filters, and reuses existing ES HTTP scroll transport and conversions. Capture the initial scroll ID during open and clear the context on early close, cancellation, EOF, or read errors. Share host selection and recursive materialized schema helpers. Increase the RLE header reach baseline from 823 to 825 for the new reader and test translation units using the existing table/expr header closure. Preserve the shared parser's existing type dispatch with a documented complexity exemption. ### Release note Support Elasticsearch HTTP queries through FileScannerV2 when enabled. ### Check List (For Author) - Test: 61 ASAN BE unit tests, including real local HTTP scroll requests; clang-format 16, clang-tidy on changed lines, BE build-hygiene checks, and Groovy syntax verification. Add ES 5/6/7/8 differential regression; live external ES regression was not run locally. - Behavior changed: Yes, ES queries honor enable_file_scanner_v2. - Does this need documentation: No, existing scanner and catalog options.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
No new actionable findings were verified for head 9d1075efad881efa44152c4e90d8847e8e312897. The static review converged in 1 round: two coverage reviewers and a separate scroll-lifecycle risk reviewer all returned NO_NEW_VALUABLE_FINDINGS, and the main reviewer independently checked the risk dispositions and all 19 changed files. There are no existing inline findings to carry forward. No additional user focus was supplied.
Reviewed execution path: EsScanPlanProvider -> FileScanOperator -> FileScannerV2 -> EsReader/TableReader -> MaterializedColumnMapper -> EsFileReader -> ESScanReader/ScrollParser -> residual filters -> table finalize.
| Required checkpoint | Conclusion |
|---|---|
| Goal and evidence | ES HTTP query scans now honor the existing v2 option through a format-local reader and generic mapping/finalization. Unit and differential regression sources cover the intended behavior; execution was not performed. |
| Scope and reuse | Changes are focused on dispatch, ES readers, shared host/schema helpers, tests, and a justified header-reach baseline. The existing HTTP transport and type conversions are reused. |
| Concurrency and locks | Scanner workers retain their delegate during synchronous reads; each scanner owns its reader state. Stop requests use existing scanner signaling. No new thread entry, shared reader state, or lock ordering was introduced. |
| Lifecycle and static initialization | The cached initial response is parsed during open to retain its scroll ID. EOF/error/stop/early close paths release the stream; moving ownership before DELETE prevents repeated cleanup after failure. Descriptor and parameter lifetimes follow scanner ownership. No new cross-TU initialization dependency was found. |
| Configuration | No new configuration item. The existing session option controls query dispatch; loads retain their prior selection. |
| Compatibility | V1 remains selectable and delegates to the same host/parser behavior. The appended FileFormat value is internal C++; no wire or stored-format change was introduced. |
| Thrift/protobuf IDs | No changes under gensrc/thrift or gensrc/proto. |
| Parallel/special paths | Shard-specific and whole-index shard=-1 planning, source/docvalue reads, _id, COUNT carriers, and LIMIT paths were traced. Remote Doris retains equivalent recursive child-schema construction. |
| Conditional checks | ES range/property validation and row-ID rejection fail explicitly; source-only fields disable docvalues. Physical row counts determine EOF before residual filtering, so empty filtered batches continue scanning. |
| Test coverage | Inspected 13 new ES unit cases, scanner dispatch tests, and ES 5/6/7/8 v1/v2 differential regression. They cover projection/order, nullable arrays, docvalue aliases, pages/batches/splits, counts, filtering, cleanup errors, and a local HTTP scroll server. |
| Expected test results | No handwritten result file was added. Differential tests compare ordered results with v1; unit expectations were checked against the source paths. No runtime pass is claimed. |
| Observability | Read/materialize/filter/IO timers and batch/physical-row counters are present; localized predicate filtering updates row accounting. HTTP wire-byte accounting remains limited by the existing transport. |
| Transactions and persistence | No transaction, EditLog, replay, visible-version, or delete-bitmap change. |
| Data writes and atomicity | No data-write path changed. ES search-context creation/cleanup was reviewed as resource lifecycle. |
| FE/BE variable propagation | No new transmitted field or variable; existing ES properties, range overrides, authentication, SSL, and field contexts are consumed. |
| Performance | Requested fields are projected; host selection and recursive schema setup remain outside row loops. No substantiated new allocation/copy/read-amplification regression was found. The RLE reach increase is explained in the commit. |
| Other correctness | Explicit slot-to-block positions preserve sparse/reordered column identity, full materialized complex shapes match parser destinations, late runtime filters retain scanner fallback, and ES does not publish condition-cache miss bitmaps. No unresolved static candidate remains. |
Validation limits: this invocation performed static inspection only, with no builds, test execution, sanitizer run, or performance measurement. Live ES interoperability is unverified. The added differential suite does not explicitly test both parallel-scroll settings; its batch_size=1 / LIMIT 2 cases do not exercise the exactly-once source-limit optimization. Actual RuntimeState cancellation, a controlled stop during HTTP, and real-HTTP failed DELETE are not directly exercised by the new tests; their code paths were traced. Blocking HTTP cancellation still relies on the inherited timeout, and malformed responses can only clear a previously known scroll ID. These limits do not leave an unresolved defect candidate in the reviewed changes.
Review converged after 1 round(s).
Contributor
TPC-H: Total hot run time: 29172 ms |
Contributor
TPC-DS: Total hot run time: 150788 ms |
Contributor
ClickBench: Total hot run time: 24.32 s |
yiguolei
approved these changes
Oct 26, 2026
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Gabriel39
added a commit
that referenced
this pull request
Oct 27, 2026
… on 4.2 (#69644) ### What problem does this PR solve? Backport #69614 to branch-4.2. ES HTTP queries do not use FileScannerV2 when `enable_file_scanner_v2` is enabled. Add the ES TableReader/FileReader, file-local projection and residual filtering, and scroll cleanup from the master implementation. Branch 4.2 still plans ES through `EsScanNode` and a dedicated ES scanner. Adapt that planner to emit FILE_SCAN plans and per-shard ES parameters when v2 is enabled; disabling the option retains the existing ES plan and scanner. Preserve the query DSL, authentication, SSL, docvalue settings, and backend assignments. Capture the scanner choice during planning so range assignment uses the same protocol without relying on a thread-local context. Reuse master's optional Thrift field IDs for the shared and per-shard ES parameters. Retain branch-4.2's existing Lance v2 dispatch and physical split scheduling. ### Release note Support Elasticsearch HTTP queries through FileScannerV2 when `enable_file_scanner_v2` is enabled. ### Check List (For Author) - Test: 18 FE unit tests, 72 ASAN BE unit tests, clang-format 16, FE Checkstyle, and clang-tidy on changed C++ lines. - Added ES 5/6/7/8 v1/v2 differential regression coverage; Groovy syntax verified. Live external ES regression was not run locally. - clang-tidy checks changed C++ lines using a comment-only VFS overlay for an existing unmatched NOLINTEND in an unchanged header. - Behavior changed: Yes, ES queries honor the scanner session option. - Does this need documentation: No, uses the existing scanner option and ES catalog properties.
6 of 11 tasks
airborne12
pushed a commit
to airborne12/apache-doris
that referenced
this pull request
Oct 27, 2026
…ssion setup (apache#69650) ### What problem does this PR solve? Related PR: apache#69614 `external_table_p2/es/test_es_file_scanner_v2.groovy` fails on its first legacy-scanner baseline query with `Table [test1] does not exist in database [default_db]`. Its interpolated port key is a Groovy `GString`, while `otherConfigs` is a `Properties` containing `String` keys. `Properties.get(Object)` returns null even when the port is configured, producing `http://<es-host>:null`. Convert the key to `String` and fail explicitly when the configuration is absent. The failure happens before any V2 result comparison or BE scan. The coverage review also found a query correctness issue shared by both scanners: a nested AND can return a partial ES DSL and record an unsupported child, but the single-expression path ignored that residual tracking. For `(a = 1 AND local_predicate) OR a = 2`, the engine could drop the local predicate; `NOT(a = 1 AND local_predicate)` could also become an unsafe `NOT(a = 1)` at the source. Retain the complete expression locally whenever conversion records a residual. Fully pushable expressions and independent top-level AND conjuncts retain their existing pushdown behavior. Expand coverage in the same change: - ES 5/6/7/8 V1/V2 comparisons across source/docvalue scans, parallel/whole-index scrolling, and batch sizes 1/1024. Add scalar/date/array/JSON, IN/NOT IN, null, raw ES query, residual filtering, aggregation and LIMIT cases; restore every changed session variable. - An independent equivalent-query oracle for nested residual OR, since a V1/V2 comparison alone cannot detect a shared FE error. Compare cardinality for unordered LIMIT rather than document identity across BEs. - FE tests for residual tracking, docvalue eligibility and field-count boundaries, alias/context serialization, shard planning and safe early-stop parameters. - BE tests for source-only fallback, missing/null docvalues and empty arrays, real HTTP empty results, early close and scroll 404/500 cleanup. Add an opt-in authenticated live-reader smoke test. ### Release note Preserve local filtering for Elasticsearch compound predicates that cannot be completely pushed down. ### Check List (For Author) - Test - [x] FE: all 127 ES connector tests pass. The two new nested OR/NOT tests fail before the residual-tracking fix. - [x] BE: all 27 selected ES ASAN tests pass against authenticated Elasticsearch 9.0.0 and 9.5.5, including source/docvalue reads with batch size 1. - [x] FE Checkstyle for the ES module and its reactor dependencies; clang-format 16, clang-tidy on changed C++ lines, and BE build-hygiene checks. - [x] Groovy 4.0.19: reproduce the null port before the fix; verify all four port lookups, suite setup/matrix execution, missing configuration, disabled execution and restoration after an injected query failure. The setup probe substitutes SQL execution and does not validate SQL results. - [ ] Full external ES 5/6/7/8 SQL regression run; requires the external Docker fixtures and a Doris cluster. - Behavior changed: - [x] Yes: residual ES predicates remain enforced; regression setup reads configured ports correctly. - Does this need documentation? - [x] No: existing catalog properties and scanner session variables are unchanged. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Oct 27, 2026
…r-mode block iterators ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The header gate rejects util/rle_encoding.h once it reaches more than its baseline plus 10% (825 -> 908 translation units). Master already reaches 907. This branch adds eleven translation units and removes four, so the header reaches 914: - added: the per-read-mode TabletBlockIterator sources (agg_key, changelog, direct, sequence_map, tablet, topn and unique_key block iterators) and the tests block_reader_changelog_merge_test, block_reader_overlapping_segments_test, block_reader_table_models_test and tablet_reader_decide_merge_test; - removed: vcollect_iterator.cpp and the tests block_reader_agg_flush_test, block_reader_changelog_vcollect_merge_test and vcollect_iterator_collected_enough_test. The new sources include storage/iterators.h or the rowset reader headers, which already reach the RLE header through storage/index/ann/ann_topn_runtime.h -> exprs/vexpr_context.h -> storage/segment/column_reader.h -> storage/segment/parsed_page.h; no header gains an include edge. The baseline is set to the measured reach, 914, the way apache#69614 raised it for the new Elasticsearch reader sources. ### Release note None ### Check List (For Author) - Test: build-support/check-build-hygiene.sh passes. - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 of 12 tasks
4 tasks done
Gabriel39
added a commit
to Gabriel39/incubator-doris
that referenced
this pull request
Oct 29, 2026
Apply the outstanding changes in master merge order: apache#69545 7d16729 apache#69551 7224a2d apache#69612 61d1aef apache#69615 8d2bd1e apache#69650 3dc7341 apache#69651 c00ecb1 apache#69646 e493c61 apache#69668 7865646 apache#69614 already exists on branch-4.2 through apache#69644. Adapt datasource APIs and tests to the release branch, retaining its existing scanner dispatch and statement-bound metadata contracts.
yiguolei
pushed a commit
that referenced
this pull request
Oct 30, 2026
### What problem does this PR solve? Backport the requested external-data and FileScannerV2 changes to `branch-4.2` in their actual master merge order: | Order | Master PR | Change | Backport | | --- | --- | --- | --- | | 1 | #69545 | Preserve Paimon schema options after field renames | Included | | 2 | #69551 | Expand Iceberg capability, transaction and snapshot tests | Included with 4.2 API adaptations | | 3 | #69614 | Elasticsearch HTTP scans in FileScannerV2 | Already present through #69644 (`2104b7509cda`); not applied twice | | 4 | #69612 | Check catalog visibility before SHOW DATABASES accesses metadata | Included | | 5 | #69615 | Publish Iceberg Parquet split offsets | Included | | 6 | #69650 | Preserve ES compound residuals and expand scanner coverage | Included | | 7 | #69651 | Accept Elasticsearch 9.x during node discovery | Included | | 8 | #69646 | Prioritize Parquet V2 row groups using TopN bounds | Included | | 9 | #69668 | Arrow loads and schema inference in FileScannerV2 | Included | The eight outstanding changes are combined in one commit. This branch retains the 4.2 FE datasource APIs instead of importing master's connector-module refactor. ### Backport adaptations - Port Paimon schema restoration, catalog-option provenance, fallback handling and tests to the existing table-copy/serialization lifecycle. Keep statement-bound fields when reader options are applied after a schema-only rename. - Exercise Iceberg's existing metadata and transaction entry points with real in-memory tables and retained-generation fixtures. Preserve the nine disabled known-bug reproductions; omit two connector-only lifecycle tests. Expiration checks verify live data-file sets and exact delete-file counters; 4.2's separate manifest-name/data-counter issue remains unchanged. The delete/history SQL suite checks visible rows and eager/late-materialization parity because 4.2 does not expose master's `_file`/`_pos` columns. The inventory labels upstream-only evidence explicitly. - Port ES residual tracking to the legacy planner and add compound OR/NOT, docvalue eligibility and Thrift-context tests there. Use Java 8-compatible HTTP APIs for opt-in ES 9 tests and the branch's supported node-address fixtures. - Keep the existing full-schema Iceberg metadata-table scan, which already supplies the file-size dependency for split offsets. Adapt the external write suite to verify offsets without requiring master-only distribution plans or golden output. - Use the existing Parquet statistics interface and physical row-location representation. Keep the TopN scheduling and pruning invariants. - Include the shared Arrow array normalizer required by the new reader, taken from the same master revision. Preserve 4.2 Lance/Avro dispatch; omit the schema-inference test for the unavailable Hive OpenCSV parser. ### Validation - FE production and test sources compile with the branch's Java 8 target. Across the targeted runs and focused reruns, 20 test classes report 181 discovered, 171 passed, zero failures/errors, and 10 skipped: nine intentionally disabled known-bug reproductions plus the opt-in live ES 9 class. - FE Checkstyle: passed with zero violations. - Clang compiler syntax checks: all 23 affected C++ translation units passed against the current sources and freshly generated Thrift, using cached third-party/generated build dependencies. Focused BE runtime validation is listed below. - clang-format 16: all 30 affected C++ source/header files passed. - Groovy 4.0.19 syntax and AST conversion: all seven changed suites passed. All 105 local inventory links resolve. - Complete staged-diff self-review, whitespace/conflict checks, and sensitive-information checks passed. A full BE rebuild and SQL/external-service regressions were not run locally. Live ES 9 tests require their opt-in endpoint. CI is requested for the combined backport. ### Release note Fix Paimon reads after field renames, external-catalog SHOW DATABASES authorization, and ES compound residual filtering. Support ES 9.x node discovery, publish Iceberg Parquet split offsets, improve Parquet V2 TopN row-group scheduling, and use FileScannerV2 for Arrow loads and text/JSON schema inference. ### Check List (For Author) - Test - [x] Unit Test - [x] Regression test additions and syntax checks; SQL execution pending CI - Behavior changed: - [x] Yes. Backport the changes listed above. - Does this need documentation? - [x] No. Existing interfaces and settings are retained; the test coverage inventory is included. ### Follow-up fixes - Set the Thrift `enable_file_scanner_v2` default to `true`, matching the session default. Preserve explicit `false` for legacy query scans and extend the existing load-selection test. - Keep the selected Paimon snapshot and schema together during relation projection. Historical sequence/aggregation options must not be combined with the latest field names. Read-only principals can resolve historical selectors without INSERT permission. - Read legacy append schemas with missing or empty bucket keys without bucket pruning, preserving every historical file. The regression covers both unaware and multiple fixed buckets. Follow-up validation: 169 targeted Paimon FE tests passed with zero failures/errors/skips. FE production compilation and Checkstyle passed. Both changed C++ test translation units passed compiler syntax checks and clang-format 16. Regenerated Java/C++ defaults are enabled; a Java 8-targeted protocol check verifies constructor/clear defaults, an omitted wire field, and an explicit-false round trip. Complete BE CI and live external SQL suites remain pending. The two new reproductions first failed with the same missing-bucket-key and renamed-sequence-field errors as CI. A separate read-only-principal reproduction also failed before the permission correction. Complete external regression execution remains pending CI. ### BE unit-test compatibility - Complete the native Arrow Variant input dependency for `branch-4.2`: accept binary storage and the Parquet Variant extension, validate metadata and payloads before appending, and retain owned data after Arrow buffers are released. Allow that ingestion path through the load reader while preserving query-format restrictions and source-default validation. - Preserve Variant V2 markers in the Arrow test fixtures, including nested types. Verify timestamp input against this branch's SerDe for positive, negative, and second-boundary values. Match the existing DATE range diagnostic in the TopN pruning test. - Add tests for sliced and empty reads, invalid ranges and carriers, malformed metadata/payloads, NULL children, atomic failure, buffer ownership, and the query capability boundary. Validation: all 13 previously failing BE tests passed. The expanded run passed all 309 tests across nine suites, including Arrow/JSON loads, load validation, Iceberg, table mapping, and Variant input/output. The focused executable rebuilds the affected readers and SerDe against cached 4.2 build dependencies; this is not a complete BE rebuild. All seven changed C++ translation units passed compiler checks with current headers/generated Thrift, and all 33 affected C/C++ files passed clang-format 16. The complete staged diff, whitespace, and sensitive-information checks passed. Full CI is requested with `run buildall` and `/review`.
yiguolei
pushed a commit
that referenced
this pull request
Nov 6, 2026
### What problem does this PR solve? Issue Number: None Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110 #67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690 #68813 #68815 #68707 #68649 Problem Summary: **In short.** This ports the Lance BE read path of branch-4.1 / branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus the BE side of three 4.x changes merged after that snapshot: the search filter `prefilter` flag and profile layout (#68813), distance bounds for `vector_search` (#68815), and the split-level index rule of #68707 that #68815 relies on. It gathers the BE parts of the PRs listed above. Master links `liblance_c` but has no Lance reader, so every `FORMAT_LANCE` scan fails today. With this PR the BE reads Lance datasets through FileScannerV2: plain scans, vector search and full-text search, predicate and runtime-filter pushdown, the schema fetch of a local Lance TVF, and the second phase of TopN lazy materialization. The PR is BE only. The FE connector plugin and the regression suites come in separate PRs, so until then nothing on master plans a Lance scan, and the BE unit tests carry the proof. Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju @FANNG1 **Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` / `s3()` with `format=lance`, and the `vector_search` / `full_text_search` TVFs. On master the FE side is being rebuilt as a connector plugin (`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5` (`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added after the snapshot, so a BE built from master also reads the Lance scan ranges and search requests that a 4.x FE sends during a rolling upgrade. The one exception is `TVectorSearchOptions` 6, branch-4.2's `approx_mode` (#69852, merged into branch-4.2 after the snapshot and not ported): it is reserved in a comment, and a master BE ignores it, so during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate` gets Lance's default (`normal`). **The problem, and what it cost.** | | master today | with this PR | |---|---|---| | A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan fails with "FileScannerV2 does not support file format" | read by `LanceTableReader` in FileScannerV2 | | `fetch_table_schema` for a local Lance TVF | not handled | returns the Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` | | The row location of a Lance row in TopN lazy materialization | `GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a 64-bit id within the snapshot | a version 1 layout carries the 64-bit row id; phase two reads the rows with the dataset's `take_rows` | **How this PR fixes it.** - Reader: `be/src/format_v2/lance/*` and `be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one local renamed for master's `-Wshadow`. The unit tests construct `RuntimeState(TQueryOptions(), globals)`, because master removed the globals-only constructor. - FileScannerV2: the reader dispatch and `_to_file_format` handle `FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two exhaustive switches. Since #69723 every file scan uses FileScannerV2, so nothing else routes Lance scans (on branch-4.2 a Lance query fails with NotSupported when `enable_file_scanner_v2` is off). - Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0, LANCE_DATASET_ROW_ID = 1}` and a union layout for `GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2 varint, so the wire format is compatible). The materialization operator decodes row locations by version, rejects an unknown version or a wrong size, and stamps `row_location_version`. `RowIdStorageReader` validates the counts and the version, range-checks internal row ids against `uint32`, and groups Lance rows by plan node, snapshot version and dataset URI, so each dataset is read with one `take_rows`. Master's segment batching, scan-slot dedup and fetch concurrency stay as they are. - Thrift (`PlanNodes.thrift`): the external search request body (`TVectorElementType`, `TVectorMetric`, `TSearchVector`, `TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`, `TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`, `TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`), `TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in comments and added 6 (`paimon_options`); this PR replaces those comments with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is reserved in a comment for branch-4.2's `approx_mode` (#69852). - After the snapshot (one commit each): - #68813: `prefilter` decides whether the TVF search filter runs before (true) or after (unset / false) candidate selection, for vector and full-text search. The profile groups the Lance counters under `LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan` and `LanceDataCache`, uses the 4.x counter names, and registers every scan metric before the statistics callback is installed, so the callback, which Lance invokes from its execution streams, never adds counters. #68813's FE tweak that hides `TUnit.NONE` labels in `AggCounter` belongs to the FE port. - #68707 (part): a vector split searches exactly the index segments FE planned for it, and a split without segments is searched flat whatever `use_index` says; a split with segments under `use_index = false` is rejected. 4.x FEs plan segments only when `use_index` allows them. - #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) are validated (finite FLOAT values, lower < upper at FLOAT precision, single-vector only) and installed with `lance_scanner_set_distance_range()`, which master's pinned lance-c provides. - Review fixes: - A nested array goes to the Doris SerDes without a copy only when it and every descendant start at offset 0 and reference exactly their child ranges. Lance's legacy (v1) decoder can return a struct or list that is tight at its own level over a Boolean child with a bit offset (a batch that starts mid-byte in the bitmap), and the SerDes reject that under Arrow input validation. - A zero-field `STRUCT<>`, which Lance decodes, is reported as an unsupported column at any nesting level, like the other unsupported Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot hold (its row count comes from its first child). - A Lance Decimal256 field with precision <= 38 is narrowed to Decimal128 by a safe Arrow cast before decoding. Doris types it DECIMAL32/64/128I by precision, and those SerDes read only Decimal128 arrays; the BE used to crash on the first value, also for nested leaves. Only values a reader can see are cast: before a nested leaf is narrowed, every child slot that no visible row references (under a NULL list, fixed-size-list, map or struct row, or outside a sliced parent) is marked NULL, so a hidden value does not fail the cast and a visible value outside the declared precision still does; map keys, which cannot be NULL, stay checked. - FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows that Doris keeps. - Runtime filters that mention INT64_MIN stay residual-only. The pinned Lance SQL parser reads `-9223372036854775808` as the negation of a magnitude that it parses as i64 first; the overflow turns the literal into a Float64, which the planner cannot coerce to Int64 or Decimal, so a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal column failed planning and the query failed. Every other i64 is still pushed. - The shared Arrow DATE32 decoding (`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside 0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`, skips null slots and adds the epoch in 64 bits: the DATE32 range part of branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and paimon-rust reads. - Child values hidden by a NULL list, fixed-size-list, map or struct row are not decoded: the array, map and struct SerDes give the element SerDe a validity bitmap that also clears those slots (`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden 10000-01-01 under a NULL fixed-size list used to fail the DATE32 range check above. Visible values are unchanged; this applies to ADBC, Arrow stream load and paimon-rust as well. The masks come from ExecEnv's tracked Arrow pool. A nested Lance array that is compacted before decoding is masked the same way, at every level, before its `ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision decimal, Date64 or time no longer fails the batch while a visible one still does. Since #69697 the same masks keep hidden DATETIMEV2 / TIMESTAMPTZ values out of its range checks. A mask costs at most one pass per batch: a NULL row that owns no child slots, or only NULL ones, allocates nothing, and Arrow stream load masks each retained IPC batch once (`mask_arrow_descendants_not_visible()`) before it hands row ranges to the SerDes, so its per-block reads allocate nothing. The one per-block mask left is for map keys under a NULL map row that still owns entries, since a MapArray cannot hold NULL keys; Arrow's map builder writes NULL map rows without entries. - Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The nullable serde reads an Arrow Null array as NULLs without touching buffers that the array does not have. - Configs, caches and metrics, with the 4.x names and defaults: | Config | Default | |---|---| | `lance_index_cache_size_bytes` | 10 GiB | | `lance_metadata_cache_size_bytes` | 1 GiB | | `enable_lance_data_cache` | true | | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` | | `lance_data_cache_disk_capacity_bytes` | 100 GiB | | `lance_data_cache_read_block_size_bytes` | 1 MiB | | `lance_io_buffer_size_bytes` (mutable) | 2 GiB | | `lance_batch_readahead` (mutable) | 5 | | `lance_fragment_readahead` (mutable) | 5 | The BE-wide Lance session (index cache, metadata cache and the optional Foyer disk cache) is created by the first Lance scan, so a BE that never reads Lance creates no cache directory. The first six configs take effect after a restart. A negative cache size (and, while the data cache is enabled, a negative disk capacity or read block size) fails the first Lance scan with a message that names the setting, as does any data-cache value lance-c rejects; the rest of the BE keeps serving. Ten `lance_session_*` metrics report the two in-memory caches. With the defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from the first Lance scan on (100 GiB in blocks of twice the read block size), the same as on branch-4.1 / 4.2. Size `lance_data_cache_disk_capacity_bytes` and `lance_data_cache_read_block_size_bytes` against the BE's open-file limit, or set `enable_lance_data_cache = false`. The index and metadata caches live in lance-c memory: they are not charged to a MemTracker or registered with CacheManager, so memory GC, the cache capacity adjustment under memory pressure and `/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache = false` does not turn them off. lance-c has no call to clear, evict or resize them, so size them with the two configs. The disk tier keys blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS, without the endpoint) and the object path (`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve the same bucket name and data-file path with different bytes (data files registered under chosen names or replaced in place; Lance's writers use random file names) share entries, also across a BE restart; such a deployment should set `enable_lance_data_cache = false`. Both are as on branch-4.1 / 4.2 and tracked as follow-ups. A Lance scan's native working set, that is the I/O buffered up to `lance_io_buffer_size_bytes` per scanner, the decode readahead and the operators of a search plan such as the top-k of a flat vector search, is allocated inside lance-c and is charged to neither the query nor its workload group. It counts toward the BE process memory only, so memory GC, which ranks queries by tracked memory, does not attribute it to the Lance query. lance-c accepts no memory budget, so on BEs with little memory headroom size `lance_io_buffer_size_bytes`, `lance_batch_readahead` and `lance_fragment_readahead` for the number of concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a follow-up. - Build: macOS links `-framework IOKit`. This is the first time objects of `liblance_c.a` are pulled into `doris_be`, so the binary and the link time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and `util/threadpool.h` 589, the same as master `315c2e00331` alone (limits 275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562 (master alone 559, limit 660); the reach budgets are unchanged. Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`); the FE side, including the rolling-upgrade guard that keeps Lance lazy materialization off while an old BE could decode a version 1 row location as version 0; the regression suites; the rest of #68707 (searching historical versions, tags and branches, rejecting fragment ids missing from the snapshot, naming the snapshot in errors and the profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that #67005 carried goes in a separate PR. **Results.** | | master | with this PR | |---|---|---| | A `FORMAT_LANCE` query | fails before reading anything | read by `LanceTableReader` in FileScannerV2 | | A Lance row in the second phase of TopN lazy materialization | cannot be represented | a version 1 row location, read back with `take_rows` | | A row location of an unknown version or size in phase two | decoded as version 0 | rejected with an error | | An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read fails with `InvalidArgument` | BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run, 1027 pass and 2 Parquet microbenchmarks skip by design. They cover the Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10, `LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the row-location protocol (`MaterializationSharedStateTest`, `RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`: count mismatch, unknown version, a Lance version on an internal mapping, an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`, `ParquetScanTest` and the other `Parquet*` suites), whose row ids now go through `file_local`; and the Arrow / protobuf / DATE serde suites with the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader suites that decode Arrow dates. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest` 10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310 pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The ninth review round adds `LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and `LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`: both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto `339e087b11f`, and restoring `parse_int64()` in either check makes exactly these two fail. On the head rebased onto master `315c2e00331`, with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest` (15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`, `AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest` and the Lance suites. Allocating at the first NULL row of each call again fails the new `ArrowParentNullsTest` case and the four new `LoadArrowPipeTest` block cases, dropping the per-batch mask fails the LIST and STRUCT block cases, and returning the child unmasked fails the new timestamp case, four `ArrowParentNullsTest` cases and `LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`. ### Release note Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They take effect once the FE can plan Lance scans. Reading an Arrow DATE32 value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or paimon-rust now fails instead of returning 0000-00-00. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - `run-be-ut.sh --run --filter=...` with the suites listed under Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the rebases, and the Lance and Arrow SerDe suites pass again with the nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the scanner selection and in `fetch_table_schema`, which now hands every file format to `format::fetch_file_schema()`, so the Lance branch runs ahead of that call; #69646 added a TopN row-group test that reads the version 0 row location fields, which now go through `file_local` (the earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the scanner selection, the `FileFormat` enum, its test and the reach budget table). The 95 TUs that this PR changes or that master changed since `5caaeb3e3d4` were syntax-checked against the rebased tree with the UT build flags, and CI runs the suites again. Rebased again onto master `8d88f1cc6f7`: the only conflict was the budget table, where #65858 moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that this PR changes or that master changed since `43f9ceb48d7` pass the same syntax check. Rebased again onto master `339e087b11f`: the only conflict was the budget table again, where #67292 moved the `util/rle_encoding.h` baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent masking in the map and struct readers and merged cleanly. Of the 114 sources that this PR changes or that master changed since `8d88f1cc6f7`, 111 pass the same syntax check. The other three are master's: two new #67292 files under `storage/segment` fail the same way on master alone, since they compile only inside their unity batch, and this PR adds no source to that target; the #69818 benchmark is not in the UT build and includes only `util/simd` headers that this PR does not touch. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. Rebased again onto master `378404eeb12`, which removed the legacy file scanner (#69723): the conflicts were in the scanner-selection code and tests that #69723 deleted, so this PR's Lance lines there went with them (including its `LanceQueriesAlwaysUseScannerV2` test), in the profile keys of `materialization_opertor.cpp`, and in the row fetch of `rowid_fetcher.cpp`, whose non-Lance path is now only `FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests took this PR's `file_local` rename with them, and the `util/rle_encoding.h` budget commit is dropped because master is no longer near that limit. Of the 100 sources that this PR changes or that master changed since `339e087b11f`, 99 pass the same syntax check; the other, `format_v2/orc/orc_reader.cpp`, fails the same way on master alone. On the rebased head the suites listed under Results run 1029 cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that `DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master. Rebased again onto master `315c2e00331`: the only conflict was `TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had reserved fields 2-5 in comments; the two commits that add the fields replace those comments, and field 6 stays #70075's `paimon_options`. #69717 cut transitive includes from shared headers: of the 160 sources that this PR changes or that master changed since `378404eeb12`, 156 pass the same syntax check; the two `jemalloc_hook` tests have no compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc aliases that macOS cannot build (#69819), and `format_v2/table_reader.cpp` alone fails the same way on master alone while its unity batch compiles. The full `doris_be_test` builds and links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases) pass before the mask fix, and the 328 cases above with it. - `build-support/check-build-hygiene.sh` (on this branch rebased onto master `315c2e00331`), clang-format 16 and `build-support/run-clang-tidy.sh` on the changed files: all pass; on the changed lines clang-tidy (llvm 22) reports only `readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of the masking functions, the macro's expansion that master's existing `DORIS_CHECK`s report too. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [ ] No. - [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN lazy materialization rejects row locations of an unknown version or size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day outside the Doris DATE range fails the read instead of becoming `0000-00-00`. - Does this need documentation? - [x] No. Lance is documented with the FE connector. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: zy-kkk <zhongyk10@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #69601
Problem Summary:
Elasticsearch HTTP scans always use FileScanner v1, even when
enable_file_scanner_v2is enabled. Add an ES TableReader/FileReader pair and routeFORMAT_ES_HTTPqueries through FileScannerV2 when that session option is enabled.The reader exposes a file-local schema, uses MaterializedColumnMapper for projection and predicate localization, and applies residual filters with v2 row accounting. It reuses the existing ES query builder, HTTP scroll transport, authentication, SSL settings, catalog-provided endpoints, and type conversions. Shared host selection and materialized complex-schema helpers keep the two scanner paths consistent.
Parse the initial cached response during open so early LIMIT/cancellation can clear its scroll context. Close the stream on EOF, cancellation, open/read errors, and split changes, and report close failures once.
The RLE header reach baseline increases by two for the new ES reader and test translation units, which use the existing table/expr header closure. The shared parser retains its existing type dispatch, with a documented complexity suppression on the new projection overload.
Release note
Support Elasticsearch HTTP queries in FileScannerV2 when
enable_file_scanner_v2is enabled. Disabling the option continues to select FileScanner v1.Check List (For Author)
_id, nullable values/arrays, multi-page batching, zero-column row counting, residual filtering, split advancement, cancellation, error cleanup, and a real local HTTP scroll server; existing scanner dispatch, legacy ES, and Remote Doris tests also pass.Check List (For Reviewer who merge this PR)