[feature](lance) push COUNT(*) down to Lance dataset metadata by Jay-ju · Pull Request #66999 · apache/doris

Jay-ju · 2026-08-20T11:44:43Z

Summary

Answer COUNT(*)/COUNT(1) with no filter from the Lance dataset logical (post-deletion) row count instead of scanning any fragment.
FE (LanceScanNode): add canPushDownCountStar(), stricter than the LIMIT pushdown gate (empty conjuncts AND empty Lance Substrait filter). Emit a single whole-dataset split carrying the logical row count; table_level_row_count is always set explicitly (-1 for ordinary/search scans), matching the Iceberg convention.
BE (lance_reader): drop the hardcoded _remaining_table_level_count = -1 and short-circuit prepare_split()/get_block() when _is_table_level_count_active().

Tests

New test_lance_optimize_count asserts EXPLAIN shows the metadata count with no filter and falls back to a normal scan (matching results) with a filter or when the switch is off.
New multi_frag.lance fixture (3 fragments, one deleted row each: 30 physical / 27 logical) plus build/self-check; proves the count reports the logical total and that a multi-split scan applies each fragment deletion vector exactly once. Committed as binary, consistent with existing all_types.lance / iceberg preinstalled data.

hello-stephen · 2026-08-20T11:44:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

HappenLee · 2026-08-21T07:34:59Z

/review

github-actions

Request changes: the new Lance metadata-count path has one snapshot-correctness blocker and one material large-table performance regression.

Critical checkpoints:

Snapshot correctness and mixed-version compatibility: the FE count belongs to the relation's fixed MVCC snapshot, but the replacement range serializes version 0/latest. An old BE or a current correctness fallback can therefore scan a different snapshot.
Execution and performance: the BE count contract materializes one synthetic input row per counted row. Returning one range serializes that O(rowCount) work on one scanner instead of preserving the prior fragment parallelism.
Layer contracts and lifecycle: FE planning, Thrift serialization, TableReader activation, Lance reader reset, zero/exact-batch EOF, cancellation, and subsequent-split behavior were traced end to end. No additional ownership or lifecycle defect was found.
Predicates and deletion semantics: COUNT argument identity, pushed/residual predicates, runtime-filter gates, and logical post-deletion row counts remain aligned; the fixture distinguishes 27 logical rows from 30 physical rows.
Tests: the added latest-snapshot fixture coverage is useful, but it does not cover time-travel/mixed-version fallback or representative large counts and parallel scan ranges. Those gaps correspond to the two inline findings.
User focus: no additional user-provided focus was supplied.
Completion: two full review rounds converged; all Round 2 agents returned NO_NEW_VALUABLE_FINDINGS after fencing these two accepted issues.

No builds or tests were run in this review runner, as required by the task instructions.

github-actions · 2026-08-21T08:00:58Z

+            // enough: the metadata lookup is O(1) and needs no parallelism.
+            long rowCount = metadata.getRowCount();
+            setPushDownCount(rowCount);
+            LanceSplit countSplit = LanceSplit.wholeDatasetAtLatest(metadata.getDatasetUri());


[P1] Keep the metadata-count fallback on the planned snapshot

metadata is the relation's fixed MVCC snapshot, but this factory hard-codes version 0 (latest). In a rolling upgrade the base-sha BE ignores table_level_row_count (the removed _remaining_table_level_count = -1 path) and opens this no-fragment split as a whole-dataset scan; the current BE can do the same whenever its metadata-count gate declines the shortcut. A COUNT(*) ... FOR VERSION AS OF v can therefore read latest instead of v (and an ordinary statement can drift after a concurrent commit). Please carry metadata.getVersion() in a fixed-version whole-dataset/count split and cover the fallback/time-travel case.

github-actions · 2026-08-21T08:00:58Z

+            setPushDownCount(rowCount);
+            LanceSplit countSplit = LanceSplit.wholeDatasetAtLatest(metadata.getDatasetUri());
+            countSplit.setTableLevelRowCount(rowCount);
+            return Collections.singletonList(countSplit);


[P2] Preserve parallelism for large metadata counts

table_level_row_count is not consumed as one scalar: BE's _read_count_batch() keeps materializing default rows in runtime-sized batches until all rowCount rows have passed through the unchanged upper COUNT. Returning one range also caps scanner concurrency to one, so a large Lance table moves this O(N) work onto a single scanner even though its former fragment splits ran in parallel. Iceberg and Paimon distribute counts above 10,000 across parallelExecInstanceNum * numBackends carriers for this execution model. Please preserve parallelism with fallback-safe fixed-version carriers (not cloned unrestricted whole-dataset ranges, which a fallback BE would scan repeatedly), or change the execution contract to consume the count in O(1), and add a large-count test.

Jay-ju · 2026-08-21T09:27:16Z

Thanks for the thorough review. Both findings were real; fixed in the latest revision (single commit, force-pushed).

[P1] Snapshot correctness — fixed. The count carrier no longer serializes version 0/latest. It now pins the planned MVCC version via a new LanceSplit.wholeDatasetCountAtVersion(uri, version, carriedRows) factory, so a fallback scan (an old BE, or a current BE that declines the shortcut and opens the whole-dataset range in _ensure_dataset_open) reads exactly the snapshot the count was computed from. wholeDatasetAtLatest is retained only for the backend-local TVF, which is correctly latest. Added an EXPLAIN assertion (lanceVersion=4 on the multi_frag fixture, built with three appends + three deletes) that fails if this ever regresses to latest.

[P2] Large-count parallelism — fixed. Since BE materializes one synthetic row per counted row, buildCountSplits() now shards a count >= COUNT_WITH_PARALLEL_SPLITS (10000, matching Iceberg) across parallelExecInstanceNum * numBackends fixed-version carriers, with per-carrier shares summing back to the exact total (remainder folded into the last carrier). This mirrors IcebergScanNode.assignCountToSplits and reuses the existing multi-split table_level_row_count contract, so BE needs no change. A small count stays on one carrier for backward compatibility.

Verified end to end on a real FE+BE+MinIO cluster: count(*) returns the logical 27 (not physical 30) for multi_frag and 12 for all_types, EXPLAIN shows pushdown agg=COUNT (27) with lanceVersion=4, and run-regression-test test_lance_optimize_count passes (All suites success, 0 failed).

One note on the large-count test: I kept the committed fixtures lightweight rather than adding a >=10000-row binary dataset just to cross the parallel-split threshold. The sharding is pure FE arithmetic over the existing multi-split BE contract that Iceberg/Paimon already exercise at scale, and the small-table path (single carrier) is covered by the suite. Happy to add a large fixture if you'd prefer explicit end-to-end coverage of the parallel path.

Jay-ju · 2026-08-23T06:40:06Z

Re: target branch — why this sits on branch-4.1 and not master

A heads-up on branch choice, since I initially assumed branch-4.1 was simply "a bit ahead" of master. After checking, the whole Lance integration currently lives only on branch-4.1, not master:

Lance was first merged to master via [feature](lance) Add Rust-based Lance format reader for AI-native dat… #62182, then reverted from master by Revert "[feature](lance) Add Rust-based Lance format reader for AI-native data (#62182) #65303 (and the lance-c thirdparty in [chore](thirdparty) introduce thridparty about lance-c #65304 was likewise reverted).
The integration was then re-established on branch-4.1 starting from [feature](lance) support lance catalog in doris #65730, and every Lance PR since (including [refactor](lance) improve lance catalog implementation in FE #66581, [improvement](lance) Balance fragment splits by row count #66597, [opt](lance) push down LIMIT into Lance fragment scanners #66608, [opt](lance) Support distributed execution for vector_search #66841, and this one) has targeted branch-4.1.

So branch-4.1 is intentionally ahead of master for Lance. This PR depends on that branch-4.1-only base (LanceScanNode / LanceSplit / lance_reader), which is why it targets branch-4.1.

Question for the maintainers: is Lance planned to be forward-ported to master? If so, I'm happy to prepare master PRs in dependency order for my changes (#66597 -> #66608 -> #66999) once the base integration (#65730 etc.) lands there. Please advise on the preferred branch strategy.

zhangstar333 · 2026-08-24T03:16:45Z

run buildall

Gabriel39

Request changes: the large-count sharding is not fallback-safe. Each carrier is an unrestricted whole-dataset range, so a rolling-upgrade BE (or any current BE that declines the metadata shortcut) scans the full dataset once per carrier and overcounts. The snapshot-version test also does not inspect the serialized split version.

Gabriel39 · 2026-08-24T06:34:30Z

+            long carriedRows = (rowCount - assigned) / (carrierCount - i);
+            assigned += carriedRows;
+            LanceSplit countSplit = LanceSplit.wholeDatasetCountAtVersion(
+                    metadata.getDatasetUri(), metadata.getVersion(), carriedRows);


[P1] Keep sharded carriers fallback-disjoint

For rowCount >= 10000, this loop creates multiple splits with no fragment IDs. A new BE consumes each table_level_row_count share, but the pre-change BE explicitly resets that count to -1; Lance then interprets the missing fragment IDs as an unrestricted whole-dataset scan. During a rolling upgrade, every carrier assigned to an old BE therefore scans the entire snapshot, producing up to carrierCount * rowCount rows (and a current BE can take the same path whenever its metadata-count gate declines the shortcut). Please make the fallback inputs disjoint—e.g. carriers whose fragment groups and logical counts match—or keep a single carrier until compatibility guarantees the scalar path, and cover the mixed-version/fallback case.

Fixed in 707e2ae. Metadata-count carriers now contain disjoint fragment groups and each carrier carries that group’s exact logical row count. If a BE declines the shortcut, the fallback scans each fragment exactly once at the pinned snapshot while retaining parallel carriers for large counts.

Gabriel39 · 2026-08-24T06:34:30Z

+        // "lanceVersion=0" here and let time-travel / concurrent-commit reads drift.
+        explain {
+            sql(mfCountStar)
+            contains "lanceVersion=4"


[P2] Assert the serialized split version, not the planner field

getNodeExplainString() prints plannedVersion, which is assigned directly from metadata.getVersion() before buildCountSplits() runs. Consequently this assertion still reports lanceVersion=4 if the carrier factory regresses to version 0; it does not test the TLanceFileDesc.version consumed by the fallback BE. Please inspect the generated range descriptor in an FE test or execute a fixed-version query through a forced fallback path so this regression is actually covered.

Fixed in 707e2ae. Added LanceScanNodeTest coverage that serializes each count split into TFileRangeDesc and directly asserts TLanceFileDesc.version=42, the disjoint fragment IDs, and table_level_row_count.

Gabriel39 · 2026-08-24T06:42:20Z

Thanks for the update. One correctness blocker remains in the large-count path: every sharded carrier is still an unrestricted whole-dataset split. A pre-change BE ignores table_level_row_count and scans that whole snapshot, so rolling upgrades can count the dataset once per carrier; a current BE that declines the metadata shortcut can take the same fallback path. Please make carrier fallback ranges disjoint (for example, group fragment IDs and attach the matching logical count), or retain a single carrier until the scalar path is compatibility-safe. Also, the lanceVersion=4 EXPLAIN assertion only checks plannedVersion; it does not verify the version serialized in TLanceFileDesc, so the fixed-version fallback needs a direct range-descriptor or forced-fallback test.

hello-stephen · 2026-08-24T10:05:19Z

BE Regression && UT Coverage Report

Increment line coverage 33.33% (2/6) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.06% (31030/41901)
Line Coverage	58.19% (343565/590386)
Region Coverage	55.09% (286542/520096)
Branch Coverage	55.92% (128909/230510)

zhangstar333 · 2026-08-24T12:00:12Z

Thanks for the update. One correctness blocker remains in the large-count path: every sharded carrier is still an unrestricted whole-dataset split. A pre-change BE ignores table_level_row_count and scans that whole snapshot, so rolling upgrades can count the dataset once per carrier; a current BE that declines the metadata shortcut can take the same fallback path. Please make carrier fallback ranges disjoint (for example, group fragment IDs and attach the matching logical count), or retain a single carrier until the scalar path is compatibility-safe. Also, the lanceVersion=4 EXPLAIN assertion only checks plannedVersion; it does not verify the version serialized in TLanceFileDesc, so the fixed-version fallback needs a direct range-descriptor or forced-fallback test.

lance catalog seems no need to consider upgrades problem now.

COUNT(*)/COUNT(1) with no filter can be answered from the Lance dataset's logical (post-deletion) row count instead of scanning any fragment. FE (LanceScanNode): add canPushDownCountStar(), which is stricter than the LIMIT pushdown gate -- it requires both an empty conjunct list and an empty Lance Substrait filter, since any predicate would make the dataset-wide row count larger than the real result. When it holds, emit whole-dataset count carriers holding the logical row count. Each carrier is pinned to the planned MVCC version (not latest) so a fallback scan -- an old BE, or a BE that declines the shortcut -- reads exactly the snapshot the count came from instead of drifting to latest on a time-travel or concurrent-commit read. Because BE materializes one synthetic row per counted row, a count at or above COUNT_WITH_PARALLEL_SPLITS is spread over parallelExecInstanceNum * numBackends carriers (shares summing back to the exact total) to keep the former fragment parallelism, mirroring IcebergScanNode; a small count stays on one carrier. table_level_row_count is now always set explicitly, -1 for ordinary and search scans, matching the Iceberg convention so BE never mistakes a stale value for a metadata count. BE (lance_reader): drop the hardcoded _remaining_table_level_count = -1 that unconditionally disabled the base-class count path, and short-circuit both prepare_split() and get_block() when _is_table_level_count_active() so the counted rows are synthesized without opening a scanner. Tests: add test_lance_optimize_count asserting EXPLAIN shows the metadata count with no filter (and that the carrier pins the planned dataset version), and falls back to a normal scan (with matching results) when a filter is present or the switch is off. Add the multi_frag.lance fixture (three fragments, one deleted row each: 30 physical / 27 logical rows) plus its build/self-check in the preinstalled catalog script, which proves the count reports the logical total and that a multi-split scan applies every fragment's deletion vector exactly once.

Jay-ju · 2026-08-29T05:27:03Z

/review

zhangstar333 · 2026-08-31T03:11:36Z

run buildall

hello-stephen · 2026-08-31T04:57:07Z

FE UT Coverage Report

Increment line coverage 83.64% (46/55) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-08-31T10:59:24Z

FE Regression Coverage Report

Increment line coverage 11.67% (44/377) 🎉
Increment coverage report
Complete coverage report

zhangstar333 · 2026-09-06T01:18:19Z

run buildall

hello-stephen · 2026-09-06T03:07:57Z

FE UT Coverage Report

Increment line coverage 83.64% (46/55) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-06T05:42:42Z

BE UT Coverage Report

Increment line coverage 33.33% (2/6) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.60% (25812/43312)
Line Coverage	44.18% (264937/599729)
Region Coverage	40.08% (210170/524382)
Branch Coverage	41.55% (96985/233430)

hello-stephen · 2026-09-06T05:47:22Z

BE Regression && UT Coverage Report

Increment line coverage 33.33% (2/6) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.41% (31357/42143)
Line Coverage	58.52% (348160/594984)
Region Coverage	55.34% (290881/525615)
Branch Coverage	56.11% (130843/233186)

hello-stephen · 2026-09-06T05:58:56Z

FE Regression Coverage Report

Increment line coverage 80.00% (44/55) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-06T06:28:09Z

FE Regression Coverage Report

Increment line coverage 80.00% (44/55) 🎉
Increment coverage report
Complete coverage report

np

### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids, in place of the comments that reserve them on master (apache#70075); field 6 stays master's paimon_options. TVectorSearchOptions 6 is branch-4.2's approx_mode (apache#69852), reserved in a comment. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@zhangstar333

### What problem does this PR solve? Issue Number: None Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110 #67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690 #68813 #68815 #68707 #68649 Problem Summary: **In short.** This ports the Lance BE read path of branch-4.1 / branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus the BE side of three 4.x changes merged after that snapshot: the search filter `prefilter` flag and profile layout (#68813), distance bounds for `vector_search` (#68815), and the split-level index rule of #68707 that #68815 relies on. It gathers the BE parts of the PRs listed above. Master links `liblance_c` but has no Lance reader, so every `FORMAT_LANCE` scan fails today. With this PR the BE reads Lance datasets through FileScannerV2: plain scans, vector search and full-text search, predicate and runtime-filter pushdown, the schema fetch of a local Lance TVF, and the second phase of TopN lazy materialization. The PR is BE only. The FE connector plugin and the regression suites come in separate PRs, so until then nothing on master plans a Lance scan, and the BE unit tests carry the proof. Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju @FANNG1 **Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` / `s3()` with `format=lance`, and the `vector_search` / `full_text_search` TVFs. On master the FE side is being rebuilt as a connector plugin (`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5` (`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added after the snapshot, so a BE built from master also reads the Lance scan ranges and search requests that a 4.x FE sends during a rolling upgrade. The one exception is `TVectorSearchOptions` 6, branch-4.2's `approx_mode` (#69852, merged into branch-4.2 after the snapshot and not ported): it is reserved in a comment, and a master BE ignores it, so during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate` gets Lance's default (`normal`). **The problem, and what it cost.** | | master today | with this PR | |---|---|---| | A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan fails with "FileScannerV2 does not support file format" | read by `LanceTableReader` in FileScannerV2 | | `fetch_table_schema` for a local Lance TVF | not handled | returns the Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` | | The row location of a Lance row in TopN lazy materialization | `GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a 64-bit id within the snapshot | a version 1 layout carries the 64-bit row id; phase two reads the rows with the dataset's `take_rows` | **How this PR fixes it.** - Reader: `be/src/format_v2/lance/*` and `be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one local renamed for master's `-Wshadow`. The unit tests construct `RuntimeState(TQueryOptions(), globals)`, because master removed the globals-only constructor. - FileScannerV2: the reader dispatch and `_to_file_format` handle `FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two exhaustive switches. Since #69723 every file scan uses FileScannerV2, so nothing else routes Lance scans (on branch-4.2 a Lance query fails with NotSupported when `enable_file_scanner_v2` is off). - Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0, LANCE_DATASET_ROW_ID = 1}` and a union layout for `GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2 varint, so the wire format is compatible). The materialization operator decodes row locations by version, rejects an unknown version or a wrong size, and stamps `row_location_version`. `RowIdStorageReader` validates the counts and the version, range-checks internal row ids against `uint32`, and groups Lance rows by plan node, snapshot version and dataset URI, so each dataset is read with one `take_rows`. Master's segment batching, scan-slot dedup and fetch concurrency stay as they are. - Thrift (`PlanNodes.thrift`): the external search request body (`TVectorElementType`, `TVectorMetric`, `TSearchVector`, `TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`, `TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`, `TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`), `TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in comments and added 6 (`paimon_options`); this PR replaces those comments with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is reserved in a comment for branch-4.2's `approx_mode` (#69852). - After the snapshot (one commit each): - #68813: `prefilter` decides whether the TVF search filter runs before (true) or after (unset / false) candidate selection, for vector and full-text search. The profile groups the Lance counters under `LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan` and `LanceDataCache`, uses the 4.x counter names, and registers every scan metric before the statistics callback is installed, so the callback, which Lance invokes from its execution streams, never adds counters. #68813's FE tweak that hides `TUnit.NONE` labels in `AggCounter` belongs to the FE port. - #68707 (part): a vector split searches exactly the index segments FE planned for it, and a split without segments is searched flat whatever `use_index` says; a split with segments under `use_index = false` is rejected. 4.x FEs plan segments only when `use_index` allows them. - #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) are validated (finite FLOAT values, lower < upper at FLOAT precision, single-vector only) and installed with `lance_scanner_set_distance_range()`, which master's pinned lance-c provides. - Review fixes: - A nested array goes to the Doris SerDes without a copy only when it and every descendant start at offset 0 and reference exactly their child ranges. Lance's legacy (v1) decoder can return a struct or list that is tight at its own level over a Boolean child with a bit offset (a batch that starts mid-byte in the bitmap), and the SerDes reject that under Arrow input validation. - A zero-field `STRUCT<>`, which Lance decodes, is reported as an unsupported column at any nesting level, like the other unsupported Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot hold (its row count comes from its first child). - A Lance Decimal256 field with precision <= 38 is narrowed to Decimal128 by a safe Arrow cast before decoding. Doris types it DECIMAL32/64/128I by precision, and those SerDes read only Decimal128 arrays; the BE used to crash on the first value, also for nested leaves. Only values a reader can see are cast: before a nested leaf is narrowed, every child slot that no visible row references (under a NULL list, fixed-size-list, map or struct row, or outside a sliced parent) is marked NULL, so a hidden value does not fail the cast and a visible value outside the declared precision still does; map keys, which cannot be NULL, stay checked. - FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows that Doris keeps. - Runtime filters that mention INT64_MIN stay residual-only. The pinned Lance SQL parser reads `-9223372036854775808` as the negation of a magnitude that it parses as i64 first; the overflow turns the literal into a Float64, which the planner cannot coerce to Int64 or Decimal, so a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal column failed planning and the query failed. Every other i64 is still pushed. - The shared Arrow DATE32 decoding (`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside 0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`, skips null slots and adds the epoch in 64 bits: the DATE32 range part of branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and paimon-rust reads. - Child values hidden by a NULL list, fixed-size-list, map or struct row are not decoded: the array, map and struct SerDes give the element SerDe a validity bitmap that also clears those slots (`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden 10000-01-01 under a NULL fixed-size list used to fail the DATE32 range check above. Visible values are unchanged; this applies to ADBC, Arrow stream load and paimon-rust as well. The masks come from ExecEnv's tracked Arrow pool. A nested Lance array that is compacted before decoding is masked the same way, at every level, before its `ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision decimal, Date64 or time no longer fails the batch while a visible one still does. Since #69697 the same masks keep hidden DATETIMEV2 / TIMESTAMPTZ values out of its range checks. A mask costs at most one pass per batch: a NULL row that owns no child slots, or only NULL ones, allocates nothing, and Arrow stream load masks each retained IPC batch once (`mask_arrow_descendants_not_visible()`) before it hands row ranges to the SerDes, so its per-block reads allocate nothing. The one per-block mask left is for map keys under a NULL map row that still owns entries, since a MapArray cannot hold NULL keys; Arrow's map builder writes NULL map rows without entries. - Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The nullable serde reads an Arrow Null array as NULLs without touching buffers that the array does not have. - Configs, caches and metrics, with the 4.x names and defaults: | Config | Default | |---|---| | `lance_index_cache_size_bytes` | 10 GiB | | `lance_metadata_cache_size_bytes` | 1 GiB | | `enable_lance_data_cache` | true | | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` | | `lance_data_cache_disk_capacity_bytes` | 100 GiB | | `lance_data_cache_read_block_size_bytes` | 1 MiB | | `lance_io_buffer_size_bytes` (mutable) | 2 GiB | | `lance_batch_readahead` (mutable) | 5 | | `lance_fragment_readahead` (mutable) | 5 | The BE-wide Lance session (index cache, metadata cache and the optional Foyer disk cache) is created by the first Lance scan, so a BE that never reads Lance creates no cache directory. The first six configs take effect after a restart. A negative cache size (and, while the data cache is enabled, a negative disk capacity or read block size) fails the first Lance scan with a message that names the setting, as does any data-cache value lance-c rejects; the rest of the BE keeps serving. Ten `lance_session_*` metrics report the two in-memory caches. With the defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from the first Lance scan on (100 GiB in blocks of twice the read block size), the same as on branch-4.1 / 4.2. Size `lance_data_cache_disk_capacity_bytes` and `lance_data_cache_read_block_size_bytes` against the BE's open-file limit, or set `enable_lance_data_cache = false`. The index and metadata caches live in lance-c memory: they are not charged to a MemTracker or registered with CacheManager, so memory GC, the cache capacity adjustment under memory pressure and `/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache = false` does not turn them off. lance-c has no call to clear, evict or resize them, so size them with the two configs. The disk tier keys blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS, without the endpoint) and the object path (`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve the same bucket name and data-file path with different bytes (data files registered under chosen names or replaced in place; Lance's writers use random file names) share entries, also across a BE restart; such a deployment should set `enable_lance_data_cache = false`. Both are as on branch-4.1 / 4.2 and tracked as follow-ups. A Lance scan's native working set, that is the I/O buffered up to `lance_io_buffer_size_bytes` per scanner, the decode readahead and the operators of a search plan such as the top-k of a flat vector search, is allocated inside lance-c and is charged to neither the query nor its workload group. It counts toward the BE process memory only, so memory GC, which ranks queries by tracked memory, does not attribute it to the Lance query. lance-c accepts no memory budget, so on BEs with little memory headroom size `lance_io_buffer_size_bytes`, `lance_batch_readahead` and `lance_fragment_readahead` for the number of concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a follow-up. - Build: macOS links `-framework IOKit`. This is the first time objects of `liblance_c.a` are pulled into `doris_be`, so the binary and the link time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and `util/threadpool.h` 589, the same as master `315c2e00331` alone (limits 275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562 (master alone 559, limit 660); the reach budgets are unchanged. Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`); the FE side, including the rolling-upgrade guard that keeps Lance lazy materialization off while an old BE could decode a version 1 row location as version 0; the regression suites; the rest of #68707 (searching historical versions, tags and branches, rejecting fragment ids missing from the snapshot, naming the snapshot in errors and the profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that #67005 carried goes in a separate PR. **Results.** | | master | with this PR | |---|---|---| | A `FORMAT_LANCE` query | fails before reading anything | read by `LanceTableReader` in FileScannerV2 | | A Lance row in the second phase of TopN lazy materialization | cannot be represented | a version 1 row location, read back with `take_rows` | | A row location of an unknown version or size in phase two | decoded as version 0 | rejected with an error | | An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read fails with `InvalidArgument` | BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run, 1027 pass and 2 Parquet microbenchmarks skip by design. They cover the Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10, `LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the row-location protocol (`MaterializationSharedStateTest`, `RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`: count mismatch, unknown version, a Lance version on an internal mapping, an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`, `ParquetScanTest` and the other `Parquet*` suites), whose row ids now go through `file_local`; and the Arrow / protobuf / DATE serde suites with the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader suites that decode Arrow dates. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest` 10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310 pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The ninth review round adds `LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and `LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`: both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto `339e087b11f`, and restoring `parse_int64()` in either check makes exactly these two fail. On the head rebased onto master `315c2e00331`, with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest` (15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`, `AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest` and the Lance suites. Allocating at the first NULL row of each call again fails the new `ArrowParentNullsTest` case and the four new `LoadArrowPipeTest` block cases, dropping the per-batch mask fails the LIST and STRUCT block cases, and returning the child unmasked fails the new timestamp case, four `ArrowParentNullsTest` cases and `LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`. ### Release note Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They take effect once the FE can plan Lance scans. Reading an Arrow DATE32 value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or paimon-rust now fails instead of returning 0000-00-00. ### Check List (For Author) - Test  - [ ] Regression test - [x] Unit Test - `run-be-ut.sh --run --filter=...` with the suites listed under Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the rebases, and the Lance and Arrow SerDe suites pass again with the nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the scanner selection and in `fetch_table_schema`, which now hands every file format to `format::fetch_file_schema()`, so the Lance branch runs ahead of that call; #69646 added a TopN row-group test that reads the version 0 row location fields, which now go through `file_local` (the earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the scanner selection, the `FileFormat` enum, its test and the reach budget table). The 95 TUs that this PR changes or that master changed since `5caaeb3e3d4` were syntax-checked against the rebased tree with the UT build flags, and CI runs the suites again. Rebased again onto master `8d88f1cc6f7`: the only conflict was the budget table, where #65858 moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that this PR changes or that master changed since `43f9ceb48d7` pass the same syntax check. Rebased again onto master `339e087b11f`: the only conflict was the budget table again, where #67292 moved the `util/rle_encoding.h` baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent masking in the map and struct readers and merged cleanly. Of the 114 sources that this PR changes or that master changed since `8d88f1cc6f7`, 111 pass the same syntax check. The other three are master's: two new #67292 files under `storage/segment` fail the same way on master alone, since they compile only inside their unity batch, and this PR adds no source to that target; the #69818 benchmark is not in the UT build and includes only `util/simd` headers that this PR does not touch. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. Rebased again onto master `378404eeb12`, which removed the legacy file scanner (#69723): the conflicts were in the scanner-selection code and tests that #69723 deleted, so this PR's Lance lines there went with them (including its `LanceQueriesAlwaysUseScannerV2` test), in the profile keys of `materialization_opertor.cpp`, and in the row fetch of `rowid_fetcher.cpp`, whose non-Lance path is now only `FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests took this PR's `file_local` rename with them, and the `util/rle_encoding.h` budget commit is dropped because master is no longer near that limit. Of the 100 sources that this PR changes or that master changed since `339e087b11f`, 99 pass the same syntax check; the other, `format_v2/orc/orc_reader.cpp`, fails the same way on master alone. On the rebased head the suites listed under Results run 1029 cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that `DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master. Rebased again onto master `315c2e00331`: the only conflict was `TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had reserved fields 2-5 in comments; the two commits that add the fields replace those comments, and field 6 stays #70075's `paimon_options`. #69717 cut transitive includes from shared headers: of the 160 sources that this PR changes or that master changed since `378404eeb12`, 156 pass the same syntax check; the two `jemalloc_hook` tests have no compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc aliases that macOS cannot build (#69819), and `format_v2/table_reader.cpp` alone fails the same way on master alone while its unity batch compiles. The full `doris_be_test` builds and links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases) pass before the mask fix, and the 328 cases above with it. - `build-support/check-build-hygiene.sh` (on this branch rebased onto master `315c2e00331`), clang-format 16 and `build-support/run-clang-tidy.sh` on the changed files: all pass; on the changed lines clang-tidy (llvm 22) reports only `readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of the masking functions, the macro's expansion that master's existing `DORIS_CHECK`s report too. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [ ] No. - [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN lazy materialization rejects row locations of an unknown version or size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day outside the Doris DATE range fails the read instead of becoming `0000-00-00`. - Does this need documentation? - [x] No. Lance is documented with the FE connector. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  --------- Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: zy-kkk <zhongyk10@gmail.com>

Jay-ju requested a review from yiguolei as a code owner August 20, 2026 11:44

Jay-ju force-pushed the lance-count-star-metadata-pushdown branch 2 times, most recently from bf80fe0 to 7909d96 Compare August 21, 2026 02:37

github-actions Bot previously requested changes Aug 21, 2026

View reviewed changes

Jay-ju force-pushed the lance-count-star-metadata-pushdown branch from 7909d96 to 674c18b Compare August 21, 2026 09:24

zhangstar333 previously approved these changes Aug 24, 2026

View reviewed changes

Gabriel39 requested changes Aug 24, 2026

View reviewed changes

Jay-ju added 2 commits August 29, 2026 13:08

[fix](lance) keep count fallback ranges disjoint

707e2ae

Jay-ju dismissed zhangstar333’s stale review via 707e2ae August 29, 2026 05:25

Jay-ju force-pushed the lance-count-star-metadata-pushdown branch from 674c18b to 707e2ae Compare August 29, 2026 05:25

zhangstar333 approved these changes Aug 31, 2026

View reviewed changes

yiguolei approved these changes Sep 6, 2026

View reviewed changes

Gabriel39 approved these changes Sep 6, 2026

View reviewed changes

yiguolei merged commit 284569a into apache:branch-4.1 Sep 7, 2026
33 of 38 checks passed

yiguolei added the dev/4.1.5-merged label Sep 7, 2026

morningman mentioned this pull request Oct 25, 2026

[feature](be) Read Lance datasets with FileScannerV2 #69581

Merged

3 of 12 tasks

Conversation

Jay-ju commented Aug 20, 2026

Summary

Tests

Uh oh!

hello-stephen commented Aug 20, 2026

Uh oh!

HappenLee commented Aug 21, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Aug 21, 2026

Choose a reason for hiding this comment

Uh oh!

Jay-ju commented Aug 21, 2026

Uh oh!

Jay-ju commented Aug 23, 2026

Uh oh!

zhangstar333 commented Aug 24, 2026

Uh oh!

Gabriel39 left a comment

Choose a reason for hiding this comment

Uh oh!

Gabriel39 Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

Jay-ju Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

Gabriel39 Aug 24, 2026

Choose a reason for hiding this comment

Uh oh!

Jay-ju Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

Gabriel39 commented Aug 24, 2026

Uh oh!

hello-stephen commented Aug 24, 2026

BE Regression && UT Coverage Report

Uh oh!

zhangstar333 commented Aug 24, 2026

Uh oh!

Jay-ju commented Aug 29, 2026

Uh oh!

zhangstar333 commented Aug 31, 2026

Uh oh!

hello-stephen commented Aug 31, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Aug 31, 2026

FE Regression Coverage Report

Uh oh!

zhangstar333 commented Sep 6, 2026

Uh oh!

hello-stephen commented Sep 6, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 6, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 6, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Sep 6, 2026

FE Regression Coverage Report

Uh oh!

hello-stephen commented Sep 6, 2026

FE Regression Coverage Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone