Repository navigation
[doc](lance) document multi-vector search usage and limits - #4148
Merged
Merged
Conversation
Gabriel39
temporarily deployed
to
Production
September 17, 2026 01:45 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
5 of 9 tasks
Gabriel39
force-pushed
the
dev/lance-multivector-docs
branch
from
September 18, 2026 06:03
29874cd to
b89ac44
Compare
Gabriel39
temporarily deployed
to
Production
September 18, 2026 06:03 — with
GitHub Actions
Inactive
GitHub Actions
Inactive
Gabriel39
added a commit
to apache/doris
that referenced
this pull request
Sep 18, 2026
### What problem does this PR solve? Issue Number: close #66494 `vector_search` accepts one query matrix for a Lance `List<FixedSizeList<T, D>>` column on branch-4.1, supporting Float16, Float32 and Float64. A matrix such as `[[1,0],[0,1]]` ranks logical rows containing variable numbers of subvectors. The row score is `sum(q in query, min(v in row, distance(q, v)))`; the TVF filter precedes candidate selection, and TopK/offset count rows. The Lance-C adapter computes scores before candidate truncation, preserving small L2 differences and aggregating each ANN query child across all output batches. Indexed candidates undergo exact refinement, and fragment-scoped results regain distance order before limit/offset. ANN candidate selection remains approximate. Omitted metrics resolve consistently to L2 on every split, including partially indexed datasets. The pinned Lance version supports cosine multi-vector indexes; L2/dot use exact search when no compatible index exists. FE, BE and C API checks bound query width to 128 subvectors and candidate work to 100,000 for both `num_vectors * (top_k + offset)` and `refine_factor * (top_k + offset)`. Actual null/non-finite stored elements encountered by scoring fail execution; empty/null outer rows do not rank. Undefined cosine pairs from zero-norm vectors are ignored, and rows without a defined match for any query subvector are filtered. The adapter resolves nested vector paths and backtick-quoted child names consistently with Lance. The shared result-window code also preserves Lance's strict batching stage after sorting and slicing; the older Doris C binding does not expose a strict-batching setter, so that path is covered by a native execution-plan test without adding an API. Requests retain schema version 1; the optional `num_vectors` field distinguishes matrices from ordinary single-vector payloads. The vendored patch adds the matrix API without changing existing C signatures. Following #67741, the third-party build now applies the community patch chain on Lance-C 0.1.9, including the Lance v11 upgrade and merged [lance-format/lance-c#83](lance-format/lance-c#83). PR #83's patch records its upstream commit and adapts overlapping context to retain PR #79's scalar-segment execution path; its multi-vector implementation and integration tests match upstream. **Rebuild Lance-C: previously installed libraries lack the required API.** Re-extract sources carrying the older patch chain; the build rejects stale patch markers instead of silently retaining an old library. ### Validation Completed a full local self-review and corrected the issues found before committing. - 53 Lance BE reader tests passed with the reader/test sources and generated Thrift linked to the newly patched Lance-C and existing BE test dependencies. Segment-scoped TopK is compared with an independent global scoring oracle across two index segments and an unindexed fragment. - The integrated Lance v11 patch chain passed 410 Rust tests, including 14 multi-vector integration tests. All three opt-in C/C++ compile/run and static OSS transport tests also passed. - 88 additional native query combinations passed independent score, ID, window and full nested-payload checks against the same integrated library. - 31 focused FE tests and FE Checkstyle passed after retaining schema version 1 and removing the multi-vector upgrade fence. - clang-format 16, Groovy syntax compilation, standalone Groovy scoring-oracle checks, fixture self-checks, shell syntax and zero-fuzz patch application passed. The actual patch driver was checked for clean application, repeat execution and rejection of stale source caches. - Added SQL regression coverage with 768-row, 128-dimensional IVF_FLAT and IVF_PQ fixtures: four partitions, two index segments and one unindexed fragment. EXPLAIN assertions verify index/split planning; independent expected results cover global TopK/offset, filtering, empty windows, and full vector/nullable payload retrieval with lazy materialization enabled and disabled. - Expanded type/dimension/scoring coverage to Float16/Float32/Float64 at dimensions 1, 3, 8 and 128, all three metrics, repeated query subvectors, empty/null outer rows, unsupported schemas, invalid query elements, Float16 overflow, and actual null/NaN/infinite stored elements for every element type. Existing regressions retain nested-path, precision, batch-size, default-metric and work-budget checks. The complete `test_lance_multivector_coverage` and `test_lance_vector_search_index_types` suites passed locally (2 suites, 0 failures, 0 skips) against the CI build of d38fba4. The schema-version change was compiled and tested separately in the focused FE/BE harnesses; those results are not a full Doris build of the final revision. The complete external regression set remains for CI. Exact equality in the representative index fixtures uses all partitions and sufficient candidate overfetch; it is not a general ANN recall guarantee. ### Release note Support multi-vector Lance columns and query matrices in `vector_search`, with documented row-level distance, filtering and TopK semantics. ### Check List (For Author) - Test - [x] Regression test added - [x] Unit Test - Behavior changed - [x] Multi-vector columns accept matrix queries; omitted metrics consistently use L2. - Documentation - [x] Added `docs/lance-multivector-search.md`. - [x] Website documentation: [apache/doris-website#4148](apache/doris-website#4148), with English and Chinese usage examples, parameter defaults and limits, scoring, null handling, and index/refinement guidance. Merge the documentation with or after this PR. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
This was referenced Sep 30, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Document how to query a Lance multi-vector column with
vector_search(), including data preparation, expected results, scoring, index usage, and parameter limits in the English and Chinese 4.x Lance Catalog guides.Depends on apache/doris#68028; merge the documentation with or after that implementation. Related feature request: apache/doris#66494.
The guide explains supported Arrow types and nullability, row-level L2/cosine/dot scoring, matrix dimensions, the 128-subvector limit, both 100,000-candidate budgets, mandatory multi-vector refinement, and cosine index usage. It also corrects the metric/refinement defaults and reader versions to match the implementation.
Versions
The Dev Lance Catalog pages were deliberately removed in #4086. This change updates the existing 4.x pages without restoring those pages or advertising support in older releases. This explains the existing missing-current-counterpart warning from the locale/version check.
Languages
Validation
git diff --checkpassed.Docs Checklist