Repository navigation
[docs](lance) Document radius and range vector search - #4200
Merged
Gabriel39 merged 2 commits intoOct 10, 2026
Merged
Conversation
5 of 9 tasks
yiguolei
pushed a commit
to apache/doris
that referenced
this pull request
Oct 10, 2026
### What problem does this PR solve? Issue Number: Closes #66493 `vector_search()` cannot currently restrict results by distance. This adds optional `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) parameters for single-vector Radius/Range Search on Lance tables. The FE validates finite FLOAT bounds and their order, includes the range in EXPLAIN, and carries it through the external-search request. The BE applies the bounds through the existing pinned lance-c API after configuring the query vector, covering indexed and Flat Search splits, including appended data outside index coverage. The existing external-search schema version remains unchanged at 1. Results retain the existing table columns and FLOAT `_distance`, prefilter semantics, offset, and Top-K cap. Bounds can be bound through prepared-statement parameters and are revalidated on each execution. Multi-vector distance bounds are rejected. Bounded index search is limited to explicitly identified IVF_FLAT segments with both bounds present. Quantized or unknown index types and single-sided ranges fall back to exact fragment scans, reported as `DISTANCE_RANGE_FALLBACK` in EXPLAIN. This avoids pre-refinement quantization errors and the pinned index implementation’s finite sentinels for absent bounds, at the cost of additional scanning. Unbounded queries retain their index selection; indexed IVF_FLAT recall still depends on probed partitions. Standalone BM25 is already delivered; the distributed BM25 statistics follow-up remains tracked separately in #67435. ### Release note Lance `vector_search()` supports optional lower and upper distance bounds for single-vector radius and range queries. ### Validation - BE: rebuilt the current branch and passed all 68 `LanceTableReader*` tests under ASAN, including exact SQ threshold crossing, the finite FLT_MAX DOT endpoint, prefilter variants, and existing reader coverage. - FE: 76 tests passed across `VectorSearchTableValuedFunctionTest`, `LancePreparedSearchTest`, `LancePreparedStatementTest`, and `LanceScanNodeTest`, including repeated bound binding, invalid parameters, all index subtypes, and indexed/flat split selection. - Native reproduction: the pinned indexed paths drop both the SQ threshold-crossing row and the finite FLT_MAX DOT endpoint; exact Flat Search returns both. Added self-contained fixtures and reader coverage for these cases. - Native execution: 10 cases passed against the pinned lance-c revision, covering indexed/Flat Search, appended unindexed data, inclusive/exclusive bounds, empty results, prefiltering, and offset. - FE Checkstyle and clang-format 16 checks passed. - Added SQL regression coverage for range results, projected stored columns, filtering, offset, invalid bounds, and EXPLAIN. Its Groovy syntax compilation passed; the SQL suite has not yet run against a live Doris cluster and requires External Regression CI. ### Check List (For Author) - Test - [x] Regression test added (execution pending CI) - [x] Unit Test - [x] Manual native execution test - Behavior changed: - [x] Yes: optional distance bounds on single-vector searches; existing unbounded queries retain their behavior. - Does this need documentation? - [x] Yes: bilingual 4.x Lance Catalog documentation: apache/doris-website#4200. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
yiguolei
pushed a commit
to apache/doris
that referenced
this pull request
Oct 10, 2026
### What problem does this PR solve? Issue Number: Closes #66493 `vector_search()` cannot currently restrict results by distance. This adds optional `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) parameters for single-vector Radius/Range Search on Lance tables. The FE validates finite FLOAT bounds and their order, includes the range in EXPLAIN, and carries it through the external-search request. The BE applies the bounds through the existing pinned lance-c API after configuring the query vector, covering indexed and Flat Search splits, including appended data outside index coverage. The existing external-search schema version remains unchanged at 1. Results retain the existing table columns and FLOAT `_distance`, prefilter semantics, offset, and Top-K cap. Bounds can be bound through prepared-statement parameters and are revalidated on each execution. Multi-vector distance bounds are rejected. Bounded index search is limited to explicitly identified IVF_FLAT segments with both bounds present. Quantized or unknown index types and single-sided ranges fall back to exact fragment scans, reported as `DISTANCE_RANGE_FALLBACK` in EXPLAIN. This avoids pre-refinement quantization errors and the pinned index implementation’s finite sentinels for absent bounds, at the cost of additional scanning. Unbounded queries retain their index selection; indexed IVF_FLAT recall still depends on probed partitions. Standalone BM25 is already delivered; the distributed BM25 statistics follow-up remains tracked separately in #67435. ### Release note Lance `vector_search()` supports optional lower and upper distance bounds for single-vector radius and range queries. ### Validation - BE: rebuilt the current branch and passed all 68 `LanceTableReader*` tests under ASAN, including exact SQ threshold crossing, the finite FLT_MAX DOT endpoint, prefilter variants, and existing reader coverage. - FE: 76 tests passed across `VectorSearchTableValuedFunctionTest`, `LancePreparedSearchTest`, `LancePreparedStatementTest`, and `LanceScanNodeTest`, including repeated bound binding, invalid parameters, all index subtypes, and indexed/flat split selection. - Native reproduction: the pinned indexed paths drop both the SQ threshold-crossing row and the finite FLT_MAX DOT endpoint; exact Flat Search returns both. Added self-contained fixtures and reader coverage for these cases. - Native execution: 10 cases passed against the pinned lance-c revision, covering indexed/Flat Search, appended unindexed data, inclusive/exclusive bounds, empty results, prefiltering, and offset. - FE Checkstyle and clang-format 16 checks passed. - Added SQL regression coverage for range results, projected stored columns, filtering, offset, invalid bounds, and EXPLAIN. Its Groovy syntax compilation passed; the SQL suite has not yet run against a live Doris cluster and requires External Regression CI. ### Check List (For Author) - Test - [x] Regression test added (execution pending CI) - [x] Unit Test - [x] Manual native execution test - Behavior changed: - [x] Yes: optional distance bounds on single-vector searches; existing unbounded queries retain their behavior. - Does this need documentation? - [x] Yes: bilingual 4.x Lance Catalog documentation: apache/doris-website#4200. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to apache/doris#68815. Merge the documentation with the implementation.
Document
distance_lower_boundanddistance_upper_boundin the English and Chinese 4.x Lance Catalog guides, including FLOAT and squared-L2 semantics, expected results, filtering and offset order, Top-K limits, and prepared-statement bound parameters.Explain that only two-sided ranges on explicitly identified IVF_FLAT indexes use indexed search. Single-sided ranges and quantized or unknown index types fall back to exact Flat Search, with
DISTANCE_RANGE_FALLBACKin EXPLAIN and potentially higher scan cost. Remove the schema-version upgrade requirement; the implementation retains the existing protocol version.Validation: the modified range sections compile as MDX in both locales, and
git diff --checkpasses. A full website build was not run.