Skip to content

[feature](lance) Support distance bounds in vector_search - #68815

Merged
yiguolei merged 2 commits into
apache:branch-4.1from
Gabriel39:dev/lance-distance-range-4.1
Oct 10, 2026
Merged

yiguolei merged 2 commits into
apache:branch-4.1from
Gabriel39:dev/lance-distance-range-4.1

Conversation

@Gabriel39

@Gabriel39 Gabriel39 commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: Closes #66493

vector_search() cannot currently restrict results by distance. This adds optional distance_lower_bound (inclusive) and distance_upper_bound (exclusive) parameters for single-vector Radius/Range Search on Lance tables.

The FE validates finite FLOAT bounds and their order, includes the range in EXPLAIN, and carries it through the external-search request. The BE applies the bounds through the existing pinned lance-c API after configuring the query vector, covering indexed and Flat Search splits, including appended data outside index coverage. The existing external-search schema version remains unchanged at 1.

Results retain the existing table columns and FLOAT _distance, prefilter semantics, offset, and Top-K cap. Bounds can be bound through prepared-statement parameters and are revalidated on each execution. Multi-vector distance bounds are rejected.

Bounded index search is limited to explicitly identified IVF_FLAT segments with both bounds present. Quantized or unknown index types and single-sided ranges fall back to exact fragment scans, reported as DISTANCE_RANGE_FALLBACK in EXPLAIN. This avoids pre-refinement quantization errors and the pinned index implementation’s finite sentinels for absent bounds, at the cost of additional scanning. Unbounded queries retain their index selection; indexed IVF_FLAT recall still depends on probed partitions.

Standalone BM25 is already delivered; the distributed BM25 statistics follow-up remains tracked separately in #67435.

Release note

Lance vector_search() supports optional lower and upper distance bounds for single-vector radius and range queries.

Validation

  • BE: rebuilt the current branch and passed all 68 LanceTableReader* tests under ASAN, including exact SQ threshold crossing, the finite FLT_MAX DOT endpoint, prefilter variants, and existing reader coverage.
  • FE: 76 tests passed across VectorSearchTableValuedFunctionTest, LancePreparedSearchTest, LancePreparedStatementTest, and LanceScanNodeTest, including repeated bound binding, invalid parameters, all index subtypes, and indexed/flat split selection.
  • Native reproduction: the pinned indexed paths drop both the SQ threshold-crossing row and the finite FLT_MAX DOT endpoint; exact Flat Search returns both. Added self-contained fixtures and reader coverage for these cases.
  • Native execution: 10 cases passed against the pinned lance-c revision, covering indexed/Flat Search, appended unindexed data, inclusive/exclusive bounds, empty results, prefiltering, and offset.
  • FE Checkstyle and clang-format 16 checks passed.
  • Added SQL regression coverage for range results, projected stored columns, filtering, offset, invalid bounds, and EXPLAIN. Its Groovy syntax compilation passed; the SQL suite has not yet run against a live Doris cluster and requires External Regression CI.

Check List (For Author)

  • Test
    • Regression test added (execution pending CI)
    • Unit Test
    • Manual native execution test
  • Behavior changed:
    • Yes: optional distance bounds on single-vector searches; existing unbounded queries retain their behavior.
  • Does this need documentation?

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Gabriel39
Gabriel39 requested a review from yiguolei as a code owner October 9, 2026 06:20
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete at head 30fface8a2d5953d24db816300a1c323593891ec after three bounded static-review rounds and a final sweep of all 20 changed paths. I found two new issues: prepared SQL rejects placeholders for the new bounds (P2), and a lower-only maximal FLOAT bound loses an eligible row in a pinned Lance indexed path (P3). No user-specific review focus or existing inline blocker was present.

Goal, scope, and behavior: The FE parses optional finite FLOAT lower/upper bounds, compares them after FLOAT rounding, and sends optional Thrift fields with schema version 2; the BE validates and installs the range after the vector query. The lower endpoint is inclusive and the upper endpoint exclusive. Literal-bound, single-vector requests keep their values through split copies, indexed and flat scans, filters, projected rows, and the global TopK/OFFSET merge. The change is focused on query behavior, a fixture, and tests. The new matrix rejection matches the supported multi-vector shape. The two findings are the remaining functional gaps.

Concurrency and lifecycle: The FE builds fresh requests and copies them per split; the BE builds a scanner per reader. This adds no shared mutable state, lock order, static initialization, or new cleanup path. The pinned C API copies the local bound values before they leave scope. Explicit existing query_parallelism is the path that exposes the indexed endpoint issue. Scanner setter failures propagate as errors. Configuration and compatibility: no new runtime setting was added; version-2 range requests are rejected by older readers rather than silently losing bounds, while unbounded requests remain version 1. There is no production data write, transaction, EditLog, or persistent-state change.

Tests and output: The added FE unit tests inspect parsing, FLOAT rounding, Thrift presence/version, and multi-vector rejection. BE tests and a V3 indexed/appended fixture inspect validation, split coverage, filtering, and offset; the SQL suite checks exact endpoints, projected values, and Explain. Their expected values are consistent under static inspection. Coverage misses actual SQL PREPARE bound placeholders and maximal-finite lower-only indexed/flat parity, corresponding to the inline findings. I did not build or run tests under this review contract. The PR author reports BE, FE, and native runs; those are author claims, not independent validation, and the author says the live SQL suite is pending CI.

Observability and cost: Explain displays bound presence; BE returns explicit validation/native errors and existing scanner profiles remain available. No additional production metric appears necessary for this per-query option. Bounds add per-request parsing/copying and native range filtering before candidate truncation, with no separate persistent or broad allocation path identified. No other substantiated issue remains after checking numeric conditions, parallel indexed/flat paths, lifecycle, compatibility, and tests.

Comment thread be/src/format_v2/table/lance_reader.cpp Resolved
@Gabriel39
Gabriel39 force-pushed the dev/lance-distance-range-4.1 branch from 30fface to aaf2621 Compare October 9, 2026 07:02
@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Complete static review of PR #68815 at aaf2621. Two coverage rounds converged and the final sweep covered all 20 changed paths. One new P1 finding is inline: bounded IVF_SQ search can discard a true in-range row using its quantized score before exact refinement.

Existing comments 4227385313 (P2 prepared bound placeholders) and 4227385317 (P3 missing-bound indexed endpoint) still apply and are not reposted. The symmetric absent-lower endpoint case shares the latter's native sentinel root cause. No previously reported P0/P1 inline blocker applies to this head.

Critical checkpoints: The goal is single-vector Lance distance-range search. FE FLOAT parsing, multi-vector rejection, Thrift schema version 2, BE validation, scanner setup after nearest(), and split request copies are consistent; old BE rejects version 2 during rolling upgrade. Changes are focused on TVF, wire, reader, EXPLAIN, and tests. Scanner-local bounds introduce no new concurrency, lock, static-initialization, or ownership issue; Lance-C copies the bound values. There is no new configuration, transaction, production write, or persistence path. Indexed and flat execution were compared across bound presence, metrics, top-k/offset, refinement, and appended fragments; the lossy-index threshold issue is the remaining correctness defect. EXPLAIN exposes the configured range, and no separate observability or performance regression was substantiated. FE/BE unit and regression cases cover ordinary L2 IVF_FLAT, appended fragments, and invalid bounds, but do not cover the SQ threshold crossing. No result file was added; the new row and distance assertions are consistent with the fixture recipe on static inspection. Tests were inspected only; builds and tests were prohibited by the review prompt and were not run. The change does not touch committed-data visibility or delete bitmap behavior.

User focus: no additional focus points were provided. Review completion: complete, with the new P1 issue reported inline.

Comment thread be/src/format_v2/table/lance_reader.cpp Resolved
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 74.07% (20/27) 🎉
Increment coverage report
Complete coverage report

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete at bc67e3b. One new P1 remains: a bounded vector_search request can run without its range on an older BE during rolling upgrade, returning rows outside the requested interval. The new comment is inline on the Thrift fields. Existing P1 comment 4227661849 (lossy-index range filtering) is resolved by the IVF_FLAT-only two-sided index gate; P3 comment 4227385317 (absent-upper FLT_MAX sentinel) is resolved by single-sided fragment fallback; P2 comment 4227385313 (prepared bounds) is resolved by the parser allow-list and repeated-binding path. No existing P0/P1 comment still applies to this head. There was no additional user-provided focus.

Critical checkpoints:

  • Goal, scope, and tests: FE parses FLOAT bounds, carries them through Thrift and split planning, and BE installs them after nearest(); this implements same-version bounded search but leaves the rolling-upgrade defect above. The production change is focused on the TVF, parser, planner, protocol, and Lance reader. FE/BE unit cases and the external regression cover parsing, binding, indexed and fragment paths, filters, offsets, and boundary fixtures. The external regression does not assert that a two-sided query actually selected the real IVF_FLAT index, so that integration check is less discriminating than the component checks. Tests and author-reported results were inspected, not executed here.
  • Concurrency and lifecycle: No new threads, shared mutable range state, locks, lock ordering, or cross-translation-unit static initialization were introduced. Each scanner receives its range after nearest() and uses the existing split and close lifecycle; no new resource-lifetime defect was substantiated.
  • Configuration and compatibility: No new configuration or dynamic-setting path was added. The new optional FE-to-BE Thrift fields need a version or capability gate for mixed-version execution (P1). No on-disk format, function-symbol, or persisted metadata change was introduced.
  • Parallel paths and conditions: Two-sided IVF_FLAT ranges may use indexed splits; single-sided and lossy/unknown-index ranges use exact fragment splits. BE disables index use for splits without segment UUIDs. Prepared execution rebinds bounds; finite FLOAT ordering, multi-vector rejection, filter, offset, and metric paths were checked. No additional correctness issue was substantiated.
  • Error handling, memory, and observability: FE and BE validate finite endpoints and return checked errors; the new bounds add no significant owned allocation. EXPLAIN reports range and index-fallback status; existing scan counters remain in use. No separate logging or metric defect was established.
  • Persistence, writes, and performance: This is a read-only search change, with no transaction, EditLog, write atomicity, or crash-recovery path. Exact fragment fallback can cost more than index search for unsafe ranges; that cost is documented, and no other performance regression was substantiated.

This was a static review under the no-build/no-test instruction. All 40 changed paths and every candidate were swept after two convergence rounds; all round-2 reviewers returned NO_NEW_VALUABLE_FINDINGS.

5: optional TVectorMetric metric
// FLOAT bounds: inclusive lower and exclusive upper.
6: optional double distance_lower_bound
7: optional double distance_upper_bound

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Guard bounded searches across BE rolling upgrades. These new optional fields are the only distance filter, but vector_search() still sends schema version 1. A pre-change BE accepts that version, skips fields 6/7, and runs an unbounded nearest search; the FE's old-BE check only covers additional projected types. For example, distance_upper_bound=16 can return rows at distance 16 or above. Use a range-specific request version that old BEs reject, or require range-capable BEs before dispatch.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (35/35) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 80.00% (28/35) 🎉
Increment coverage report
Complete coverage report

@yiguolei
yiguolei merged commit cd673b8 into apache:branch-4.1 Oct 10, 2026
38 of 41 checks passed
Gabriel39 added a commit to apache/doris-website that referenced this pull request Oct 10, 2026
Companion to apache/doris#68815. Merge the
documentation with the implementation.

Document `distance_lower_bound` and `distance_upper_bound` in the
English and Chinese 4.x Lance Catalog guides, including FLOAT and
squared-L2 semantics, expected results, filtering and offset order,
Top-K limits, and prepared-statement bound parameters.

Explain that only two-sided ranges on explicitly identified IVF_FLAT
indexes use indexed search. Single-sided ranges and quantized or unknown
index types fall back to exact Flat Search, with
`DISTANCE_RANGE_FALLBACK` in EXPLAIN and potentially higher scan cost.
Remove the schema-version upgrade requirement; the implementation
retains the existing protocol version.

Validation: the modified range sections compile as MDX in both locales,
and `git diff --check` passes. A full website build was not run.
yiguolei pushed a commit that referenced this pull request Oct 10, 2026
### What problem does this PR solve?

Issue Number: Closes #66493

`vector_search()` cannot currently restrict results by distance. This
adds optional `distance_lower_bound` (inclusive) and
`distance_upper_bound` (exclusive) parameters for single-vector
Radius/Range Search on Lance tables.

The FE validates finite FLOAT bounds and their order, includes the range
in EXPLAIN, and carries it through the external-search request. The BE
applies the bounds through the existing pinned lance-c API after
configuring the query vector, covering indexed and Flat Search splits,
including appended data outside index coverage. The existing
external-search schema version remains unchanged at 1.

Results retain the existing table columns and FLOAT `_distance`,
prefilter semantics, offset, and Top-K cap. Bounds can be bound through
prepared-statement parameters and are revalidated on each execution.
Multi-vector distance bounds are rejected.

Bounded index search is limited to explicitly identified IVF_FLAT
segments with both bounds present. Quantized or unknown index types and
single-sided ranges fall back to exact fragment scans, reported as
`DISTANCE_RANGE_FALLBACK` in EXPLAIN. This avoids pre-refinement
quantization errors and the pinned index implementation’s finite
sentinels for absent bounds, at the cost of additional scanning.
Unbounded queries retain their index selection; indexed IVF_FLAT recall
still depends on probed partitions.

Standalone BM25 is already delivered; the distributed BM25 statistics
follow-up remains tracked separately in #67435.

### Release note

Lance `vector_search()` supports optional lower and upper distance
bounds for single-vector radius and range queries.

### Validation

- BE: rebuilt the current branch and passed all 68 `LanceTableReader*`
tests under ASAN, including exact SQ threshold crossing, the finite
FLT_MAX DOT endpoint, prefilter variants, and existing reader coverage.
- FE: 76 tests passed across `VectorSearchTableValuedFunctionTest`,
`LancePreparedSearchTest`, `LancePreparedStatementTest`, and
`LanceScanNodeTest`, including repeated bound binding, invalid
parameters, all index subtypes, and indexed/flat split selection.
- Native reproduction: the pinned indexed paths drop both the SQ
threshold-crossing row and the finite FLT_MAX DOT endpoint; exact Flat
Search returns both. Added self-contained fixtures and reader coverage
for these cases.
- Native execution: 10 cases passed against the pinned lance-c revision,
covering indexed/Flat Search, appended unindexed data,
inclusive/exclusive bounds, empty results, prefiltering, and offset.
- FE Checkstyle and clang-format 16 checks passed.
- Added SQL regression coverage for range results, projected stored
columns, filtering, offset, invalid bounds, and EXPLAIN. Its Groovy
syntax compilation passed; the SQL suite has not yet run against a live
Doris cluster and requires External Regression CI.

### Check List (For Author)

- Test
  - [x] Regression test added (execution pending CI)
  - [x] Unit Test
  - [x] Manual native execution test
- Behavior changed:
- [x] Yes: optional distance bounds on single-vector searches; existing
unbounded queries retain their behavior.
- Does this need documentation?
- [x] Yes: bilingual 4.x Lance Catalog documentation:
apache/doris-website#4200.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
morningman added a commit to morningman/doris that referenced this pull request Oct 25, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 25, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, apache#68815 added `TVectorSearchParams.distance_lower_bound` (6) and
`distance_upper_bound` (7) to branch-4.1 and branch-4.2: FLOAT bounds,
inclusive lower and exclusive upper, which the 4.x FE fills for a
single-vector `vector_search()` with distance bounds. apache#68815 kept the
external-search `schema_version` at 1, so a BE cannot tell a bounded request
from an unbounded one by version. A master BE without the fields ignored them
and returned the unbounded top-k rows, for example rows with `_distance`
below the lower bound.

Fix: port the BE side of apache#68815.
- `gensrc/thrift/PlanNodes.thrift`: add `6: optional double
  distance_lower_bound` and `7: optional double distance_upper_bound` to
  `TVectorSearchParams` with the 4.x comment.
- `LanceTableReader::_validate_external_search_request()` rejects bounds on a
  multi-vector query (NotSupported), a bound that is not a finite FLOAT value
  (NaN, infinity, or beyond FLT_MAX), and a lower bound that is not below the
  upper bound at FLOAT precision (InvalidArgument), before the dataset is
  opened.
- `_configure_vector_search()` narrows the bounds to FLOAT and installs them
  with `lance_scanner_set_distance_range()` after `lance_scanner_nearest()`,
  which clears any earlier range. A missing bound is passed as NULL
  (unbounded on that side). Requests without bounds are unchanged.

Before: distance bounds sent by a 4.x FE were silently ignored. After: the
reader returns only rows with lower <= _distance < upper, still capped by
top_k and offset, and rejects invalid bounds.

Compatibility (gensrc/AGENTS.md): fields 6 and 7 of `TVectorSearchParams` are
`optional double distance_lower_bound` and `optional double
distance_upper_bound` on the branch-4.2 (a7ae6f4) and branch-4.1
(3f911d0) heads, the same ids and types as here. branch-4.0 and
branch-3.1 have no `TVectorSearchParams`, and on master the struct is new in
apache#69581 with fields 1-5. No id is reused, and both fields are optional.

`lance_scanner_set_distance_range()` is in master's lance-c pin
(4fafcd64525), so only the BE changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New (copied unchanged from apache#68815 except for master's
      `RuntimeState(TQueryOptions(), globals)` constructor):
      `LanceTableReaderVectorSearchTest.ValidatesDistanceRangeBeforeDatasetAccess`,
      `LanceTableReaderVectorSearchTest.AppliesDistanceBoundsAndOffset`,
      `LanceTableReaderVectorSearchTest.DistanceRangeFallbackPreservesExactScores`,
      `LanceTableReaderVectorSearchTest.DistanceRangeCoversIndexedAndAppendedFragments`.
    - Fixtures copied from d306323 (identical on the branch-4.2 head):
      `be/test/format_v2/table/lance/data/distance_range.lance` (11 files),
      `distance_range_dot.lance` (8 files), `distance_range_sq.lance` (8
      files) and their description
      `be/test/format_v2/table/lance/distance_range_fixture.md`.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the
      Lance suites. `DistanceRangeFallbackPreservesExactScores` needs the
      previous commit: without it the IVF_SQ fallback split is searched with
      the quantized index and loses its only row. With the range not installed,
      the three cases that read rows fail.
- Behavior changed: Yes, on the Lance external-search path added by apache#69581:
  distance bounds are applied instead of ignored, and invalid bounds are
  rejected. Master has no FE that sends Lance search requests yet.
- Does this need documentation: No

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 26, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 26, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 26, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, apache#68815 added `TVectorSearchParams.distance_lower_bound` (6) and
`distance_upper_bound` (7) to branch-4.1 and branch-4.2: FLOAT bounds,
inclusive lower and exclusive upper, which the 4.x FE fills for a
single-vector `vector_search()` with distance bounds. apache#68815 kept the
external-search `schema_version` at 1, so a BE cannot tell a bounded request
from an unbounded one by version. A master BE without the fields ignored them
and returned the unbounded top-k rows, for example rows with `_distance`
below the lower bound.

Fix: port the BE side of apache#68815.
- `gensrc/thrift/PlanNodes.thrift`: add `6: optional double
  distance_lower_bound` and `7: optional double distance_upper_bound` to
  `TVectorSearchParams` with the 4.x comment.
- `LanceTableReader::_validate_external_search_request()` rejects bounds on a
  multi-vector query (NotSupported), a bound that is not a finite FLOAT value
  (NaN, infinity, or beyond FLT_MAX), and a lower bound that is not below the
  upper bound at FLOAT precision (InvalidArgument), before the dataset is
  opened.
- `_configure_vector_search()` narrows the bounds to FLOAT and installs them
  with `lance_scanner_set_distance_range()` after `lance_scanner_nearest()`,
  which clears any earlier range. A missing bound is passed as NULL
  (unbounded on that side). Requests without bounds are unchanged.

Before: distance bounds sent by a 4.x FE were silently ignored. After: the
reader returns only rows with lower <= _distance < upper, still capped by
top_k and offset, and rejects invalid bounds.

Compatibility (gensrc/AGENTS.md): fields 6 and 7 of `TVectorSearchParams` are
`optional double distance_lower_bound` and `optional double
distance_upper_bound` on the branch-4.2 (a7ae6f4) and branch-4.1
(3f911d0) heads, the same ids and types as here. branch-4.0 and
branch-3.1 have no `TVectorSearchParams`, and on master the struct is new in
apache#69581 with fields 1-5. No id is reused, and both fields are optional.

`lance_scanner_set_distance_range()` is in master's lance-c pin
(4fafcd64525), so only the BE changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New (copied unchanged from apache#68815 except for master's
      `RuntimeState(TQueryOptions(), globals)` constructor):
      `LanceTableReaderVectorSearchTest.ValidatesDistanceRangeBeforeDatasetAccess`,
      `LanceTableReaderVectorSearchTest.AppliesDistanceBoundsAndOffset`,
      `LanceTableReaderVectorSearchTest.DistanceRangeFallbackPreservesExactScores`,
      `LanceTableReaderVectorSearchTest.DistanceRangeCoversIndexedAndAppendedFragments`.
    - Fixtures copied from d306323 (identical on the branch-4.2 head):
      `be/test/format_v2/table/lance/data/distance_range.lance` (11 files),
      `distance_range_dot.lance` (8 files), `distance_range_sq.lance` (8
      files) and their description
      `be/test/format_v2/table/lance/distance_range_fixture.md`.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the
      Lance suites. `DistanceRangeFallbackPreservesExactScores` needs the
      previous commit: without it the IVF_SQ fallback split is searched with
      the quantized index and loses its only row. With the range not installed,
      the three cases that read rows fail.
- Behavior changed: Yes, on the Lance external-search path added by apache#69581:
  distance bounds are applied instead of ignored, and invalid bounds are
  rejected. Master has no FE that sends Lance search requests yet.
- Does this need documentation: No

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 27, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 27, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 27, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, apache#68815 added `TVectorSearchParams.distance_lower_bound` (6) and
`distance_upper_bound` (7) to branch-4.1 and branch-4.2: FLOAT bounds,
inclusive lower and exclusive upper, which the 4.x FE fills for a
single-vector `vector_search()` with distance bounds. apache#68815 kept the
external-search `schema_version` at 1, so a BE cannot tell a bounded request
from an unbounded one by version. A master BE without the fields ignored them
and returned the unbounded top-k rows, for example rows with `_distance`
below the lower bound.

Fix: port the BE side of apache#68815.
- `gensrc/thrift/PlanNodes.thrift`: add `6: optional double
  distance_lower_bound` and `7: optional double distance_upper_bound` to
  `TVectorSearchParams` with the 4.x comment.
- `LanceTableReader::_validate_external_search_request()` rejects bounds on a
  multi-vector query (NotSupported), a bound that is not a finite FLOAT value
  (NaN, infinity, or beyond FLT_MAX), and a lower bound that is not below the
  upper bound at FLOAT precision (InvalidArgument), before the dataset is
  opened.
- `_configure_vector_search()` narrows the bounds to FLOAT and installs them
  with `lance_scanner_set_distance_range()` after `lance_scanner_nearest()`,
  which clears any earlier range. A missing bound is passed as NULL
  (unbounded on that side). Requests without bounds are unchanged.

Before: distance bounds sent by a 4.x FE were silently ignored. After: the
reader returns only rows with lower <= _distance < upper, still capped by
top_k and offset, and rejects invalid bounds.

Compatibility (gensrc/AGENTS.md): fields 6 and 7 of `TVectorSearchParams` are
`optional double distance_lower_bound` and `optional double
distance_upper_bound` on the branch-4.2 (a7ae6f4) and branch-4.1
(3f911d0) heads, the same ids and types as here. branch-4.0 and
branch-3.1 have no `TVectorSearchParams`, and on master the struct is new in
apache#69581 with fields 1-5. No id is reused, and both fields are optional.

`lance_scanner_set_distance_range()` is in master's lance-c pin
(4fafcd64525), so only the BE changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New (copied unchanged from apache#68815 except for master's
      `RuntimeState(TQueryOptions(), globals)` constructor):
      `LanceTableReaderVectorSearchTest.ValidatesDistanceRangeBeforeDatasetAccess`,
      `LanceTableReaderVectorSearchTest.AppliesDistanceBoundsAndOffset`,
      `LanceTableReaderVectorSearchTest.DistanceRangeFallbackPreservesExactScores`,
      `LanceTableReaderVectorSearchTest.DistanceRangeCoversIndexedAndAppendedFragments`.
    - Fixtures copied from d306323 (identical on the branch-4.2 head):
      `be/test/format_v2/table/lance/data/distance_range.lance` (11 files),
      `distance_range_dot.lance` (8 files), `distance_range_sq.lance` (8
      files) and their description
      `be/test/format_v2/table/lance/distance_range_fixture.md`.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the
      Lance suites. `DistanceRangeFallbackPreservesExactScores` needs the
      previous commit: without it the IVF_SQ fallback split is searched with
      the quantized index and loses its only row. With the range not installed,
      the three cases that read rows fail.
- Behavior changed: Yes, on the Lance external-search path added by apache#69581:
  distance bounds are applied instead of ignored, and invalid bounds are
  rejected. Master has no FE that sends Lance search requests yet.
- Does this need documentation: No

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 28, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 28, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 28, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, apache#68815 added `TVectorSearchParams.distance_lower_bound` (6) and
`distance_upper_bound` (7) to branch-4.1 and branch-4.2: FLOAT bounds,
inclusive lower and exclusive upper, which the 4.x FE fills for a
single-vector `vector_search()` with distance bounds. apache#68815 kept the
external-search `schema_version` at 1, so a BE cannot tell a bounded request
from an unbounded one by version. A master BE without the fields ignored them
and returned the unbounded top-k rows, for example rows with `_distance`
below the lower bound.

Fix: port the BE side of apache#68815.
- `gensrc/thrift/PlanNodes.thrift`: add `6: optional double
  distance_lower_bound` and `7: optional double distance_upper_bound` to
  `TVectorSearchParams` with the 4.x comment.
- `LanceTableReader::_validate_external_search_request()` rejects bounds on a
  multi-vector query (NotSupported), a bound that is not a finite FLOAT value
  (NaN, infinity, or beyond FLT_MAX), and a lower bound that is not below the
  upper bound at FLOAT precision (InvalidArgument), before the dataset is
  opened.
- `_configure_vector_search()` narrows the bounds to FLOAT and installs them
  with `lance_scanner_set_distance_range()` after `lance_scanner_nearest()`,
  which clears any earlier range. A missing bound is passed as NULL
  (unbounded on that side). Requests without bounds are unchanged.

Before: distance bounds sent by a 4.x FE were silently ignored. After: the
reader returns only rows with lower <= _distance < upper, still capped by
top_k and offset, and rejects invalid bounds.

Compatibility (gensrc/AGENTS.md): fields 6 and 7 of `TVectorSearchParams` are
`optional double distance_lower_bound` and `optional double
distance_upper_bound` on the branch-4.2 (a7ae6f4) and branch-4.1
(3f911d0) heads, the same ids and types as here. branch-4.0 and
branch-3.1 have no `TVectorSearchParams`, and on master the struct is new in
apache#69581 with fields 1-5. No id is reused, and both fields are optional.

`lance_scanner_set_distance_range()` is in master's lance-c pin
(4fafcd64525), so only the BE changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New (copied unchanged from apache#68815 except for master's
      `RuntimeState(TQueryOptions(), globals)` constructor):
      `LanceTableReaderVectorSearchTest.ValidatesDistanceRangeBeforeDatasetAccess`,
      `LanceTableReaderVectorSearchTest.AppliesDistanceBoundsAndOffset`,
      `LanceTableReaderVectorSearchTest.DistanceRangeFallbackPreservesExactScores`,
      `LanceTableReaderVectorSearchTest.DistanceRangeCoversIndexedAndAppendedFragments`.
    - Fixtures copied from d306323 (identical on the branch-4.2 head):
      `be/test/format_v2/table/lance/data/distance_range.lance` (11 files),
      `distance_range_dot.lance` (8 files), `distance_range_sq.lance` (8
      files) and their description
      `be/test/format_v2/table/lance/distance_range_fixture.md`.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the
      Lance suites. `DistanceRangeFallbackPreservesExactScores` needs the
      previous commit: without it the IVF_SQ fallback split is searched with
      the quantized index and loses its only row. With the range not installed,
      the three cases that read rows fail.
- Behavior changed: Yes, on the Lance external-search path added by apache#69581:
  distance bounds are applied instead of ignored, and invalid bounds are
  rejected. Master has no FE that sends Lance search requests yet.
- Does this need documentation: No

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 29, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 29, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 29, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, apache#68815 added `TVectorSearchParams.distance_lower_bound` (6) and
`distance_upper_bound` (7) to branch-4.1 and branch-4.2: FLOAT bounds,
inclusive lower and exclusive upper, which the 4.x FE fills for a
single-vector `vector_search()` with distance bounds. apache#68815 kept the
external-search `schema_version` at 1, so a BE cannot tell a bounded request
from an unbounded one by version. A master BE without the fields ignored them
and returned the unbounded top-k rows, for example rows with `_distance`
below the lower bound.

Fix: port the BE side of apache#68815.
- `gensrc/thrift/PlanNodes.thrift`: add `6: optional double
  distance_lower_bound` and `7: optional double distance_upper_bound` to
  `TVectorSearchParams` with the 4.x comment.
- `LanceTableReader::_validate_external_search_request()` rejects bounds on a
  multi-vector query (NotSupported), a bound that is not a finite FLOAT value
  (NaN, infinity, or beyond FLT_MAX), and a lower bound that is not below the
  upper bound at FLOAT precision (InvalidArgument), before the dataset is
  opened.
- `_configure_vector_search()` narrows the bounds to FLOAT and installs them
  with `lance_scanner_set_distance_range()` after `lance_scanner_nearest()`,
  which clears any earlier range. A missing bound is passed as NULL
  (unbounded on that side). Requests without bounds are unchanged.

Before: distance bounds sent by a 4.x FE were silently ignored. After: the
reader returns only rows with lower <= _distance < upper, still capped by
top_k and offset, and rejects invalid bounds.

Compatibility (gensrc/AGENTS.md): fields 6 and 7 of `TVectorSearchParams` are
`optional double distance_lower_bound` and `optional double
distance_upper_bound` on the branch-4.2 (a7ae6f4) and branch-4.1
(3f911d0) heads, the same ids and types as here. branch-4.0 and
branch-3.1 have no `TVectorSearchParams`, and on master the struct is new in
apache#69581 with fields 1-5. No id is reused, and both fields are optional.

`lance_scanner_set_distance_range()` is in master's lance-c pin
(4fafcd64525), so only the BE changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New (copied unchanged from apache#68815 except for master's
      `RuntimeState(TQueryOptions(), globals)` constructor):
      `LanceTableReaderVectorSearchTest.ValidatesDistanceRangeBeforeDatasetAccess`,
      `LanceTableReaderVectorSearchTest.AppliesDistanceBoundsAndOffset`,
      `LanceTableReaderVectorSearchTest.DistanceRangeFallbackPreservesExactScores`,
      `LanceTableReaderVectorSearchTest.DistanceRangeCoversIndexedAndAppendedFragments`.
    - Fixtures copied from d306323 (identical on the branch-4.2 head):
      `be/test/format_v2/table/lance/data/distance_range.lance` (11 files),
      `distance_range_dot.lance` (8 files), `distance_range_sq.lance` (8
      files) and their description
      `be/test/format_v2/table/lance/distance_range_fixture.md`.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the
      Lance suites. `DistanceRangeFallbackPreservesExactScores` needs the
      previous commit: without it the IVF_SQ fallback split is searched with
      the quantized index and loses its only row. With the range not installed,
      the three cases that read rows fail.
- Behavior changed: Yes, on the Lance external-search path added by apache#69581:
  distance bounds are applied instead of ignored, and invalid bounds are
  rejected. Master has no FE that sends Lance search requests yet.
- Does this need documentation: No

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 31, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 31, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 31, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, apache#68815 added `TVectorSearchParams.distance_lower_bound` (6) and
`distance_upper_bound` (7) to branch-4.1 and branch-4.2: FLOAT bounds,
inclusive lower and exclusive upper, which the 4.x FE fills for a
single-vector `vector_search()` with distance bounds. apache#68815 kept the
external-search `schema_version` at 1, so a BE cannot tell a bounded request
from an unbounded one by version. A master BE without the fields ignored them
and returned the unbounded top-k rows, for example rows with `_distance`
below the lower bound.

Fix: port the BE side of apache#68815.
- `gensrc/thrift/PlanNodes.thrift`: add `6: optional double
  distance_lower_bound` and `7: optional double distance_upper_bound` to
  `TVectorSearchParams` with the 4.x comment.
- `LanceTableReader::_validate_external_search_request()` rejects bounds on a
  multi-vector query (NotSupported), a bound that is not a finite FLOAT value
  (NaN, infinity, or beyond FLT_MAX), and a lower bound that is not below the
  upper bound at FLOAT precision (InvalidArgument), before the dataset is
  opened.
- `_configure_vector_search()` narrows the bounds to FLOAT and installs them
  with `lance_scanner_set_distance_range()` after `lance_scanner_nearest()`,
  which clears any earlier range. A missing bound is passed as NULL
  (unbounded on that side). Requests without bounds are unchanged.

Before: distance bounds sent by a 4.x FE were silently ignored. After: the
reader returns only rows with lower <= _distance < upper, still capped by
top_k and offset, and rejects invalid bounds.

Compatibility (gensrc/AGENTS.md): fields 6 and 7 of `TVectorSearchParams` are
`optional double distance_lower_bound` and `optional double
distance_upper_bound` on the branch-4.2 (a7ae6f4) and branch-4.1
(3f911d0) heads, the same ids and types as here. branch-4.0 and
branch-3.1 have no `TVectorSearchParams`, and on master the struct is new in
apache#69581 with fields 1-5. No id is reused, and both fields are optional.

`lance_scanner_set_distance_range()` is in master's lance-c pin
(4fafcd64525), so only the BE changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New (copied unchanged from apache#68815 except for master's
      `RuntimeState(TQueryOptions(), globals)` constructor):
      `LanceTableReaderVectorSearchTest.ValidatesDistanceRangeBeforeDatasetAccess`,
      `LanceTableReaderVectorSearchTest.AppliesDistanceBoundsAndOffset`,
      `LanceTableReaderVectorSearchTest.DistanceRangeFallbackPreservesExactScores`,
      `LanceTableReaderVectorSearchTest.DistanceRangeCoversIndexedAndAppendedFragments`.
    - Fixtures copied from d306323 (identical on the branch-4.2 head):
      `be/test/format_v2/table/lance/data/distance_range.lance` (11 files),
      `distance_range_dot.lance` (8 files), `distance_range_sq.lance` (8
      files) and their description
      `be/test/format_v2/table/lance/distance_range_fixture.md`.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the
      Lance suites. `DistanceRangeFallbackPreservesExactScores` needs the
      previous commit: without it the IVF_SQ fallback split is searched with
      the quantized index and loses its only row. With the range not installed,
      the three cases that read rows fail.
- Behavior changed: Yes, on the Lance external-search path added by apache#69581:
  distance bounds are applied instead of ignored, and invalid bounds are
  rejected. Master has no FE that sends Lance search requests yet.
- Does this need documentation: No

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Nov 2, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Nov 2, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Nov 2, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, apache#68815 added `TVectorSearchParams.distance_lower_bound` (6) and
`distance_upper_bound` (7) to branch-4.1 and branch-4.2: FLOAT bounds,
inclusive lower and exclusive upper, which the 4.x FE fills for a
single-vector `vector_search()` with distance bounds. apache#68815 kept the
external-search `schema_version` at 1, so a BE cannot tell a bounded request
from an unbounded one by version. A master BE without the fields ignored them
and returned the unbounded top-k rows, for example rows with `_distance`
below the lower bound.

Fix: port the BE side of apache#68815.
- `gensrc/thrift/PlanNodes.thrift`: add `6: optional double
  distance_lower_bound` and `7: optional double distance_upper_bound` to
  `TVectorSearchParams` with the 4.x comment.
- `LanceTableReader::_validate_external_search_request()` rejects bounds on a
  multi-vector query (NotSupported), a bound that is not a finite FLOAT value
  (NaN, infinity, or beyond FLT_MAX), and a lower bound that is not below the
  upper bound at FLOAT precision (InvalidArgument), before the dataset is
  opened.
- `_configure_vector_search()` narrows the bounds to FLOAT and installs them
  with `lance_scanner_set_distance_range()` after `lance_scanner_nearest()`,
  which clears any earlier range. A missing bound is passed as NULL
  (unbounded on that side). Requests without bounds are unchanged.

Before: distance bounds sent by a 4.x FE were silently ignored. After: the
reader returns only rows with lower <= _distance < upper, still capped by
top_k and offset, and rejects invalid bounds.

Compatibility (gensrc/AGENTS.md): fields 6 and 7 of `TVectorSearchParams` are
`optional double distance_lower_bound` and `optional double
distance_upper_bound` on the branch-4.2 (a7ae6f4) and branch-4.1
(3f911d0) heads, the same ids and types as here. branch-4.0 and
branch-3.1 have no `TVectorSearchParams`, and on master the struct is new in
apache#69581 with fields 1-5. No id is reused, and both fields are optional.

`lance_scanner_set_distance_range()` is in master's lance-c pin
(4fafcd64525), so only the BE changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New (copied unchanged from apache#68815 except for master's
      `RuntimeState(TQueryOptions(), globals)` constructor):
      `LanceTableReaderVectorSearchTest.ValidatesDistanceRangeBeforeDatasetAccess`,
      `LanceTableReaderVectorSearchTest.AppliesDistanceBoundsAndOffset`,
      `LanceTableReaderVectorSearchTest.DistanceRangeFallbackPreservesExactScores`,
      `LanceTableReaderVectorSearchTest.DistanceRangeCoversIndexedAndAppendedFragments`.
    - Fixtures copied from d306323 (identical on the branch-4.2 head):
      `be/test/format_v2/table/lance/data/distance_range.lance` (11 files),
      `distance_range_dot.lance` (8 files), `distance_range_sq.lance` (8
      files) and their description
      `be/test/format_v2/table/lance/distance_range_fixture.md`.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the
      Lance suites. `DistanceRangeFallbackPreservesExactScores` needs the
      previous commit: without it the IVF_SQ fallback split is searched with
      the quantized index and loses its only row. With the range not installed,
      the three cases that read rows fail.
- Behavior changed: Yes, on the Lance external-search path added by apache#69581:
  distance bounds are applied instead of ignored, and invalid bounds are
  rejected. Master has no FE that sends Lance search requests yet.
- Does this need documentation: No

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Nov 5, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids, in place of the
  comments that reserve them on master (apache#70075); field 6 stays master's
  paimon_options.
  TVectorSearchOptions 6 is branch-4.2's approx_mode (apache#69852), reserved
  in a comment.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Nov 5, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Nov 5, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, apache#68815 added `TVectorSearchParams.distance_lower_bound` (6) and
`distance_upper_bound` (7) to branch-4.1 and branch-4.2: FLOAT bounds,
inclusive lower and exclusive upper, which the 4.x FE fills for a
single-vector `vector_search()` with distance bounds. apache#68815 kept the
external-search `schema_version` at 1, so a BE cannot tell a bounded request
from an unbounded one by version. A master BE without the fields ignored them
and returned the unbounded top-k rows, for example rows with `_distance`
below the lower bound.

Fix: port the BE side of apache#68815.
- `gensrc/thrift/PlanNodes.thrift`: add `6: optional double
  distance_lower_bound` and `7: optional double distance_upper_bound` to
  `TVectorSearchParams` with the 4.x comment.
- `LanceTableReader::_validate_external_search_request()` rejects bounds on a
  multi-vector query (NotSupported), a bound that is not a finite FLOAT value
  (NaN, infinity, or beyond FLT_MAX), and a lower bound that is not below the
  upper bound at FLOAT precision (InvalidArgument), before the dataset is
  opened.
- `_configure_vector_search()` narrows the bounds to FLOAT and installs them
  with `lance_scanner_set_distance_range()` after `lance_scanner_nearest()`,
  which clears any earlier range. A missing bound is passed as NULL
  (unbounded on that side). Requests without bounds are unchanged.

Before: distance bounds sent by a 4.x FE were silently ignored. After: the
reader returns only rows with lower <= _distance < upper, still capped by
top_k and offset, and rejects invalid bounds.

Compatibility (gensrc/AGENTS.md): fields 6 and 7 of `TVectorSearchParams` are
`optional double distance_lower_bound` and `optional double
distance_upper_bound` on the branch-4.2 (a7ae6f4) and branch-4.1
(3f911d0) heads, the same ids and types as here. branch-4.0 and
branch-3.1 have no `TVectorSearchParams`, and on master the struct is new in
apache#69581 with fields 1-5. No id is reused, and both fields are optional.

`lance_scanner_set_distance_range()` is in master's lance-c pin
(4fafcd64525), so only the BE changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New (copied unchanged from apache#68815 except for master's
      `RuntimeState(TQueryOptions(), globals)` constructor):
      `LanceTableReaderVectorSearchTest.ValidatesDistanceRangeBeforeDatasetAccess`,
      `LanceTableReaderVectorSearchTest.AppliesDistanceBoundsAndOffset`,
      `LanceTableReaderVectorSearchTest.DistanceRangeFallbackPreservesExactScores`,
      `LanceTableReaderVectorSearchTest.DistanceRangeCoversIndexedAndAppendedFragments`.
    - Fixtures copied from d306323 (identical on the branch-4.2 head):
      `be/test/format_v2/table/lance/data/distance_range.lance` (11 files),
      `distance_range_dot.lance` (8 files), `distance_range_sq.lance` (8
      files) and their description
      `be/test/format_v2/table/lance/distance_range_fixture.md`.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the
      Lance suites. `DistanceRangeFallbackPreservesExactScores` needs the
      previous commit: without it the IVF_SQ fallback split is searched with
      the quantized index and loses its only row. With the range not installed,
      the three cases that read rows fail.
- Behavior changed: Yes, on the Lance external-search path added by apache#69581:
  distance bounds are applied instead of ignored, and invalid bounds are
  rejected. Master has no FE that sends Lance search requests yet.
- Does this need documentation: No

Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
yiguolei pushed a commit that referenced this pull request Nov 6, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110
#67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690
#68813 #68815 #68707 #68649

Problem Summary:

**In short.** This ports the Lance BE read path of branch-4.1 /
branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus
the BE side of three 4.x changes merged after that snapshot: the search
filter `prefilter` flag and profile layout (#68813), distance bounds for
`vector_search` (#68815), and the split-level index rule of #68707 that
#68815 relies on. It gathers the BE parts of the PRs listed above.
Master links `liblance_c` but has no Lance reader, so every
`FORMAT_LANCE` scan fails today. With this PR the BE reads Lance
datasets through FileScannerV2: plain scans, vector search and full-text
search, predicate and runtime-filter pushdown, the schema fetch of a
local Lance TVF, and the second phase of TopN lazy materialization. The
PR is BE only. The FE connector plugin and the regression suites come in
separate PRs, so until then nothing on master plans a Lance scan, and
the BE unit tests carry the proof.

Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju
@FANNG1

**Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` /
`s3()` with `format=lance`, and the `vector_search` / `full_text_search`
TVFs. On master the FE side is being rebuilt as a connector plugin
(`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift
ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5`
(`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added
after the snapshot, so a BE built from master also reads the Lance scan
ranges and search requests that a 4.x FE sends during a rolling upgrade.
The one exception is `TVectorSearchOptions` 6, branch-4.2's
`approx_mode` (#69852, merged into branch-4.2 after the snapshot and not
ported): it is reserved in a comment, and a master BE ignores it, so
during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate`
gets Lance's default (`normal`).

**The problem, and what it cost.**

| | master today | with this PR |
|---|---|---|
| A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan
fails with "FileScannerV2 does not support file format" | read by
`LanceTableReader` in FileScannerV2 |
| `fetch_table_schema` for a local Lance TVF | not handled | returns the
Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` |
| The row location of a Lance row in TopN lazy materialization |
`GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a
64-bit id within the snapshot | a version 1 layout carries the 64-bit
row id; phase two reads the rows with the dataset's `take_rows` |

**How this PR fixes it.**

- Reader: `be/src/format_v2/lance/*` and
`be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one
local renamed for master's `-Wshadow`. The unit tests construct
`RuntimeState(TQueryOptions(), globals)`, because master removed the
globals-only constructor.
- FileScannerV2: the reader dispatch and `_to_file_format` handle
`FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two
exhaustive switches. Since #69723 every file scan uses FileScannerV2, so
nothing else routes Lance scans (on branch-4.2 a Lance query fails with
NotSupported when `enable_file_scanner_v2` is off).
- Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0,
LANCE_DATASET_ROW_ID = 1}` and a union layout for
`GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static
asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2
varint, so the wire format is compatible). The materialization operator
decodes row locations by version, rejects an unknown version or a wrong
size, and stamps `row_location_version`. `RowIdStorageReader` validates
the counts and the version, range-checks internal row ids against
`uint32`, and groups Lance rows by plan node, snapshot version and
dataset URI, so each dataset is read with one `take_rows`. Master's
segment batching, scan-slot dedup and fetch concurrency stay as they
are.
- Thrift (`PlanNodes.thrift`): the external search request body
(`TVectorElementType`, `TVectorMetric`, `TSearchVector`,
`TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`,
`TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`,
`TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`),
`TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the
branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is
reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in
comments and added 6 (`paimon_options`); this PR replaces those comments
with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is
reserved in a comment for branch-4.2's `approx_mode` (#69852).
- After the snapshot (one commit each):
- #68813: `prefilter` decides whether the TVF search filter runs before
(true) or after (unset / false) candidate selection, for vector and
full-text search. The profile groups the Lance counters under
`LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`,
`LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan`
and `LanceDataCache`, uses the 4.x counter names, and registers every
scan metric before the statistics callback is installed, so the
callback, which Lance invokes from its execution streams, never adds
counters. #68813's FE tweak that hides `TUnit.NONE` labels in
`AggCounter` belongs to the FE port.
- #68707 (part): a vector split searches exactly the index segments FE
planned for it, and a split without segments is searched flat whatever
`use_index` says; a split with segments under `use_index = false` is
rejected. 4.x FEs plan segments only when `use_index` allows them.
- #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound`
(exclusive) are validated (finite FLOAT values, lower < upper at FLOAT
precision, single-vector only) and installed with
`lance_scanner_set_distance_range()`, which master's pinned lance-c
provides.
- Review fixes:
- A nested array goes to the Doris SerDes without a copy only when it
and every descendant start at offset 0 and reference exactly their child
ranges. Lance's legacy (v1) decoder can return a struct or list that is
tight at its own level over a Boolean child with a bit offset (a batch
that starts mid-byte in the bitmap), and the SerDes reject that under
Arrow input validation.
- A zero-field `STRUCT<>`, which Lance decodes, is reported as an
unsupported column at any nesting level, like the other unsupported
Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot
hold (its row count comes from its first child).
- A Lance Decimal256 field with precision <= 38 is narrowed to
Decimal128 by a safe Arrow cast before decoding. Doris types it
DECIMAL32/64/128I by precision, and those SerDes read only Decimal128
arrays; the BE used to crash on the first value, also for nested leaves.
Only values a reader can see are cast: before a nested leaf is narrowed,
every child slot that no visible row references (under a NULL list,
fixed-size-list, map or struct row, or outside a sliced parent) is
marked NULL, so a hidden value does not fail the cast and a visible
value outside the declared precision still does; map keys, which cannot
be NULL, stay checked.
- FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares
floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows
that Doris keeps.
- Runtime filters that mention INT64_MIN stay residual-only. The pinned
Lance SQL parser reads `-9223372036854775808` as the negation of a
magnitude that it parses as i64 first; the overflow turns the literal
into a Float64, which the planner cannot coerce to Int64 or Decimal, so
a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal
column failed planning and the query failed. Every other i64 is still
pushed.
- The shared Arrow DATE32 decoding
(`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside
0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`,
skips null slots and adds the epoch in 64 bits: the DATE32 range part of
branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and
paimon-rust reads.
- Child values hidden by a NULL list, fixed-size-list, map or struct row
are not decoded: the array, map and struct SerDes give the element SerDe
a validity bitmap that also clears those slots
(`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them
unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden
10000-01-01 under a NULL fixed-size list used to fail the DATE32 range
check above. Visible values are unchanged; this applies to ADBC, Arrow
stream load and paimon-rust as well. The masks come from ExecEnv's
tracked Arrow pool. A nested Lance array that is compacted before
decoding is masked the same way, at every level, before its
`ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision
decimal, Date64 or time no longer fails the batch while a visible one
still does. Since #69697 the same masks keep hidden DATETIMEV2 /
TIMESTAMPTZ values out of its range checks. A mask costs at most one
pass per batch: a NULL row that owns no child slots, or only NULL ones,
allocates nothing, and Arrow stream load masks each retained IPC batch
once (`mask_arrow_descendants_not_visible()`) before it hands row ranges
to the SerDes, so its per-block reads allocate nothing. The one
per-block mask left is for map keys under a NULL map row that still owns
entries, since a MapArray cannot hold NULL keys; Arrow's map builder
writes NULL map rows without entries.
- Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a
Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The
nullable serde reads an Arrow Null array as NULLs without touching
buffers that the array does not have.
- Configs, caches and metrics, with the 4.x names and defaults:

  | Config | Default |
  |---|---|
  | `lance_index_cache_size_bytes` | 10 GiB |
  | `lance_metadata_cache_size_bytes` | 1 GiB |
  | `enable_lance_data_cache` | true |
  | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` |
  | `lance_data_cache_disk_capacity_bytes` | 100 GiB |
  | `lance_data_cache_read_block_size_bytes` | 1 MiB |
  | `lance_io_buffer_size_bytes` (mutable) | 2 GiB |
  | `lance_batch_readahead` (mutable) | 5 |
  | `lance_fragment_readahead` (mutable) | 5 |

The BE-wide Lance session (index cache, metadata cache and the optional
Foyer disk cache) is created by the first Lance scan, so a BE that never
reads Lance creates no cache directory. The first six configs take
effect after a restart. A negative cache size (and, while the data cache
is enabled, a negative disk capacity or read block size) fails the first
Lance scan with a message that names the setting, as does any data-cache
value lance-c rejects; the rest of the BE keeps serving. Ten
`lance_session_*` metrics report the two in-memory caches. With the
defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from
the first Lance scan on (100 GiB in blocks of twice the read block
size), the same as on branch-4.1 / 4.2. Size
`lance_data_cache_disk_capacity_bytes` and
`lance_data_cache_read_block_size_bytes` against the BE's open-file
limit, or set `enable_lance_data_cache = false`.

The index and metadata caches live in lance-c memory: they are not
charged to a MemTracker or registered with CacheManager, so memory GC,
the cache capacity adjustment under memory pressure and
`/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache
= false` does not turn them off. lance-c has no call to clear, evict or
resize them, so size them with the two configs. The disk tier keys
blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS,
without the endpoint) and the object path
(`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve
the same bucket name and data-file path with different bytes (data files
registered under chosen names or replaced in place; Lance's writers use
random file names) share entries, also across a BE restart; such a
deployment should set `enable_lance_data_cache = false`. Both are as on
branch-4.1 / 4.2 and tracked as follow-ups.

A Lance scan's native working set, that is the I/O buffered up to
`lance_io_buffer_size_bytes` per scanner, the decode readahead and the
operators of a search plan such as the top-k of a flat vector search, is
allocated inside lance-c and is charged to neither the query nor its
workload group. It counts toward the BE process memory only, so memory
GC, which ranks queries by tracked memory, does not attribute it to the
Lance query. lance-c accepts no memory budget, so on BEs with little
memory headroom size `lance_io_buffer_size_bytes`,
`lance_batch_readahead` and `lance_fragment_readahead` for the number of
concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a
follow-up.
- Build: macOS links `-framework IOKit`. This is the first time objects
of `liblance_c.a` are pulled into `doris_be`, so the binary and the link
time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and
`util/threadpool.h` 589, the same as master `315c2e00331` alone (limits
275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562
(master alone 559, limit 660); the reach budgets are unchanged.

Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`);
the FE side, including the rolling-upgrade guard that keeps Lance lazy
materialization off while an old BE could decode a version 1 row
location as version 0; the regression suites; the rest of #68707
(searching historical versions, tags and branches, rejecting fragment
ids missing from the snapshot, naming the snapshot in errors and the
profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that
#67005 carried goes in a separate PR.

**Results.**

| | master | with this PR |
|---|---|---|
| A `FORMAT_LANCE` query | fails before reading anything | read by
`LanceTableReader` in FileScannerV2 |
| A Lance row in the second phase of TopN lazy materialization | cannot
be represented | a version 1 row location, read back with `take_rows` |
| A row location of an unknown version or size in phase two | decoded as
version 0 | rejected with an error |
| An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream
load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read
fails with `InvalidArgument` |

BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on
the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run,
1027 pass and 2 Parquet microbenchmarks skip by design. They cover the
Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10,
`LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the
row-location protocol (`MaterializationSharedStateTest`,
`RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`:
count mismatch, unknown version, a Lance version on an internal mapping,
an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and
Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`,
`ParquetScanTest` and the other `Parquet*` suites), whose row ids now go
through `file_local`; and the Arrow / protobuf / DATE serde suites with
the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader
suites that decode Arrow dates. Reverting any of the review fixes, the
#68813 flag, the #68707 split rule or the #68815 bounds makes its new
tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest`
10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow
SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310
pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The
ninth review round adds
`LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and
`LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`:
both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto
`339e087b11f`, and restoring `parse_int64()` in either check makes
exactly these two fail. On the head rebased onto master `315c2e00331`,
with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites
pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest`
(15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe
suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`,
`AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest`
and the Lance suites. Allocating at the first NULL row of each call
again fails the new `ArrowParentNullsTest` case and the four new
`LoadArrowPipeTest` block cases, dropping the per-batch mask fails the
LIST and STRUCT block cases, and returning the child unmasked fails the
new timestamp case, four `ArrowParentNullsTest` cases and
`LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`.

### Release note

Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They
take effect once the FE can plan Lance scans. Reading an Arrow DATE32
value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or
paimon-rust now fails instead of returning 0000-00-00.

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
- `run-be-ut.sh --run --filter=...` with the suites listed under
Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the
rebases, and the Lance and Arrow SerDe suites pass again with the
nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and
compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite
driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the
scanner selection and in `fetch_table_schema`, which now hands every
file format to `format::fetch_file_schema()`, so the Lance branch runs
ahead of that call; #69646 added a TopN row-group test that reads the
version 0 row location fields, which now go through `file_local` (the
earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the
scanner selection, the `FileFormat` enum, its test and the reach budget
table). The 95 TUs that this PR changes or that master changed since
`5caaeb3e3d4` were syntax-checked against the rebased tree with the UT
build flags, and CI runs the suites again. Rebased again onto master
`8d88f1cc6f7`: the only conflict was the budget table, where #65858
moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that
this PR changes or that master changed since `43f9ceb48d7` pass the same
syntax check. Rebased again onto master `339e087b11f`: the only conflict
was the budget table again, where #67292 moved the `util/rle_encoding.h`
baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent
masking in the map and struct readers and merged cleanly. Of the 114
sources that this PR changes or that master changed since `8d88f1cc6f7`,
111 pass the same syntax check. The other three are master's: two new
#67292 files under `storage/segment` fail the same way on master alone,
since they compile only inside their unity batch, and this PR adds no
source to that target; the #69818 benchmark is not in the UT build and
includes only `util/simd` headers that this PR does not touch. Reverting
any of the review fixes, the #68813 flag, the #68707 split rule or the
#68815 bounds makes its new tests fail. Rebased again onto master
`378404eeb12`, which removed the legacy file scanner (#69723): the
conflicts were in the scanner-selection code and tests that #69723
deleted, so this PR's Lance lines there went with them (including its
`LanceQueriesAlwaysUseScannerV2` test), in the profile keys of
`materialization_opertor.cpp`, and in the row fetch of
`rowid_fetcher.cpp`, whose non-Lance path is now only
`FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests
took this PR's `file_local` rename with them, and the
`util/rle_encoding.h` budget commit is dropped because master is no
longer near that limit. Of the 100 sources that this PR changes or that
master changed since `339e087b11f`, 99 pass the same syntax check; the
other, `format_v2/orc/orc_reader.cpp`, fails the same way on master
alone. On the rebased head the suites listed under Results run 1029
cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that
`DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master.
Rebased again onto master `315c2e00331`: the only conflict was
`TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had
reserved fields 2-5 in comments; the two commits that add the fields
replace those comments, and field 6 stays #70075's `paimon_options`.
#69717 cut transitive includes from shared headers: of the 160 sources
that this PR changes or that master changed since `378404eeb12`, 156
pass the same syntax check; the two `jemalloc_hook` tests have no
compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc
aliases that macOS cannot build (#69819), and
`format_v2/table_reader.cpp` alone fails the same way on master alone
while its unity batch compiles. The full `doris_be_test` builds and
links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases)
pass before the mask fix, and the 328 cases above with it.
- `build-support/check-build-hygiene.sh` (on this branch rebased onto
master `315c2e00331`), clang-format 16 and
`build-support/run-clang-tidy.sh` on the changed files: all pass; on the
changed lines clang-tidy (llvm 22) reports only
`readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of
the masking functions, the macro's expansion that master's existing
`DORIS_CHECK`s report too.
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- Behavior changed:
    - [ ] No.
- [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN
lazy materialization rejects row locations of an unknown version or
size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day
outside the Doris DATE range fails the read instead of becoming
`0000-00-00`.
- Does this need documentation?
    - [x] No. Lance is documented with the FE connector.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants