Repository navigation
[feature](lance) Search historical versions, tags and branches with vector_search and full_text_search - #68707
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Review status: COMPLETE for head ebe7d6f. Static review covered all 99 changed paths, the prepared diff, regression expectations, and the relevant FE/BE call chains. One P2 test-coverage finding is inline. The prepared threads and live PR comments contain no existing P0/P1 finding to carry. No additional user focus was supplied.
Critical checkpoints:
- Goal and scope: Snapshot-aware Lance vector and FTS search is wired from TVF selector parsing through catalog metadata, FE split planning, BE scan, and row-id fetch. The change is focused on this feature and its fixtures/tests; the active regression proof for historical branch selectors is incomplete as noted inline.
- Correctness and conditionals: Reviewed conflicting/invalid selectors, missing versions, strict versus index-only FTS coverage, mixed indexed/flat vector splits, historical deletes, schema evolution, and fixed positive versions. The same selected URI/version flows to search and two-phase fetch; no additional concrete wrong-row path survived review.
- Concurrency and lifecycle: Per-query metadata pins URI/version across planning and BE reads, including when a tag or branch head later moves. Reader close/reopen resets snapshot-related state; no new lock ordering or shared mutable state was found. Resource and allocator lifetimes were traced across metadata reads and scans.
- Configuration and compatibility: No new dynamic configuration or Doris transaction/EditLog path is introduced. FE/BE reuse the existing scan transport for URI, version, fragments, and index segments. Filesystem and managed REST catalogs, main/branch/tag selection, and vector/FTS paths were checked separately; rolling-upgrade behavior was assessed statically only.
- Parallel paths and observability: Reviewed normal scans, search scans, prepared TVFs, and lazy row fetch, plus branch/tag and managed table resolution. Existing errors, EXPLAIN fields, and metadata/scan profiling expose the selected version and coverage; no separate observability defect was substantiated.
- Tests and results: Inspected FE/BE unit tests, SQL regressions and expected output, binary fixture inventory, and compose/REST setup. The new JNI snapshot test class is disabled; active tests leave the historical branch selector gap described inline. Builds and tests were prohibited and were not run, so no runtime result is claimed.
- Performance and writes: The fixture builder stages and promotes test data; this PR does not add a Doris data-write or persistence path. Vector indexes without usable segment details now take flat search, which can be costly for older large datasets; available metadata did not establish a safe indexed fallback, so this remains a review limitation rather than a separate inline finding.
| contains "lanceVectorIndexStatus=DISABLED" | ||
| } | ||
| qt_branch_fork """ | ||
| SELECT row_id, _distance FROM ${vectorSearch(', "branch"="dev", "version"="4"', "[100,101,102,103]")} |
There was a problem hiding this comment.
[P2] Exercise a historical snapshot on the dev branch. This branch=dev,version=4 query selects the fork point, whose rows and indexes match main version 4, so a second checkout on main in readFromLatest would still pass. The branch timestamp assertion asks for year 2999, so ignoring that timestamp also selects dev head. Add discriminating assertions: explicit dev version 6 with STRICT FTS (dev rebuilt coverage over fragment 2, main v6 did not), and an as-of time between dev commits 5 and 6 with lanceVersion=5. The JNI snapshot test class is disabled, leaving these search paths without an active historical check.
There was a problem hiding this comment.
Agreed: the fork-point version and the future timestamp could not tell the branch from main, or the time from the head. Added in e55bd49:
branch=dev, version=5returns the branch's rows 101/102/103 (main version 5 would return 24/23/22), with EXPLAINlanceVersion=5andlanceBranch=dev.- STRICT
full_text_searchsucceeds onbranch=dev, version=6, whosebody_idxwas rebuilt over fragment 2, and fails on main version 6. branch=dev, timestamp=2026-09-29 14:52:18, between dev versions 5 and 6, gives EXPLAINlanceVersion=5, and STRICT full-text search fails namingdataset version 5 of branch 'dev'. Ignoring the time would select dev version 6, and resolving it on main would select main version 7.
All 29 suites in external_table_p0/lance pass locally.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Review complete at head e55bd497a453399971f3ae57d5b993261eedd006 after two convergence rounds and a final sweep of all 99 changed paths. Two distinct inline findings follow: a P2 legacy vector-index performance regression and a P3 historical two-phase test gap. No P0/P1 finding was substantiated.
Critical checkpoints
- Goal and scope: The search TVFs now select version, timestamp, tag, and branch snapshots; FE binds schema/index metadata from the selected snapshot and sends its URI and positive version through existing split fields. The production changes are focused on that path and its BE checks.
- Concurrency and lifecycle: No new threads or locks were added. The BE fragment list is reader-local, bound to the opened dataset key, and reset on close; a changed URI/version/options key fails. Search and row-id fetch reject an unpinned version, and missing fragment IDs fail rather than silently dropping rows. No new cross-translation-unit initializer or ownership cycle was found.
- Compatibility and parallel paths: Reviewed filesystem and managed REST resolution, main and dev histories, vector indexed/flat/disabled paths, FTS coverage, normal scans, prepared re-execution, SQL cache exclusion, and two-phase row fetch. Existing Thrift URI/version/options fields carry the selection. The compatible legacy-index path has the P2 regression reported inline; no other mixed-path defect was substantiated.
- Conditions, persistence, and writes: Selector combinations, timestamp parsing in the session time zone, branch-name validation, and deleted/pruned histories have explicit errors or conservative fallback. Production changes do not add configuration, transaction writes, or persistent Doris metadata. The fixture generator writes test datasets only.
- Tests and observability: Inspected FE/BE tests, generated regression outputs, fixture histories, and active filesystem/REST suites, including negative and schema-evolution cases. The JNI snapshot test class is disabled; active assertions cover its main historical paths, with the P3 two-phase gap noted inline. EXPLAIN shows the selected version/branch, and BE profile/errors identify the opened snapshot. No builds or tests were run in this review, as instructed; the author's local test claim was not treated as independent validation.
- Performance and remaining questions: The P2 flat-scan regression is the remaining performance concern. All other main risk items and candidate points have a recorded conclusion; no unresolved suspicious point remains.
Existing inline discussion 4172612906 is a P2 test concern already addressed on this head by discriminating dev version-5 rows, dev version-6 STRICT FTS, and a between-commit timestamp. It is not reposted or carried as a P0/P1 blocker. No additional user review focus was supplied.
| // it as one. Lance would otherwise search it with the first index on the column in manifest | ||
| // order, which need not be the index FE selected. A split with segments searches exactly | ||
| // those; one that also carries use_index=false was rejected above. | ||
| if (lance_scanner_set_use_index(scanner, segment_count > 0) != 0) { |
There was a problem hiding this comment.
[P2] Preserve ANN use for compatible legacy indexes. A dataset with one legacy L2 vector index whose physical entries have no index details is skipped by describeUserIndexes; FE then sends fragment splits without a segment UUID, so this line overrides even use_index=true and scans every vector flat. Previously Lance could select that index for the same query, so large existing collections lose indexed-search performance. Recover enough metadata to plan a compatible legacy index or provide another safe indexed path, and add a legacy-index execution regression.
There was a problem hiding this comment.
I checked this against real legacy indexes before changing anything. Indexes without index details written by pylance 0.10.18 and 0.18.2 (IVF_PQ, L2) and 0.24.1 (IVF_PQ, cosine, multi-vector) are not skipped: Lance infers the missing details from the index files when it loads the index (infer_missing_vector_details), so describeUserIndexes describes them, the FE plans one segment (lanceVectorIndexStatus=USED), and the BE profile shows LanceIVFPartitionsSearched: 1 with no flat-search fragments.
The skip in describeUserIndexes is now reached only when that inference fails, for example when an index file cannot be read at planning time. Doris then cannot verify the index's coverage or metric, so a flat search is the conservative result, and a persistent failure would break the BE's use of the index too.
Added in a7b9979: legacy_vector_index.lance, built by pylance 0.18.2 without index details, and test_lance_vector_search_legacy_index, which checks that the index is planned with one segment and that the indexed search returns the flat search's rows. The docs now say an older index is skipped only when Lance cannot infer its details.
| contains "VMaterializeNode" | ||
| } | ||
| qt_branch_two_phase "${twoPhase}" | ||
| qt_tag_two_phase """ |
There was a problem hiding this comment.
[P3] Make this historical tag test exercise and distinguish phase-two fetch. rel selects version 5, but rows 24/23/22 and their body values are identical at latest version 7; this result would still pass if row-id fetch reopened v7. This query also lacks the adjacent VMaterializeNode EXPLAIN assertion. Assert that plan shape and select historical payload that changes or disappears in a later version (for example, row 3, deleted in v6) so the test proves the fetched rows came from the selected snapshot.
There was a problem hiding this comment.
Done in a7b9979: the tag case now searches next to row 3, which rel (version 5) still has and version 6 deleted. A fetch from a later version would return fewer rows and fail the row-count check. It also asserts VMaterializeNode.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Complete static review of PR #68707 at a7b9979. No new actionable inline findings. The user supplied no additional focus points.
The change enables vector_search() and full_text_search() to select a Lance version, timestamp, tag, or branch. FE derives schema, field IDs, fragments, and indexes from one selected Dataset, then sends its URI and positive version in each split. BE uses that fixed snapshot for search and row-id fetch, validates FE fragment IDs against the opened dataset, and follows FE's segment plan for indexed versus flat vector splits. I inspected the active filesystem and REST regression cases, BE and FE unit tests, fixture generators and committed history, and expected outputs. The JNI LanceSearchSnapshotTest class is disabled, so the active regression suites are the relevant end-to-end coverage. Builds and tests were prohibited and were not run; author-reported passing suites are not independent validation here.
Previously reported inline concerns: #4172612906 [P2] is addressed by active dev-v5 rows, dev-v6 STRICT FTS, and an intervening dev timestamp assertion; #4172821003 [P3] is addressed by the VMaterializeNode assertion and a tagged row deleted in a later version; #4172821000 [P2] remains the existing legacy-index discussion, with an inferable legacy fixture and index-planning regression added on this head. I did not repost these threads. No existing P0/P1 inline finding was independently confirmed as still applicable.
Critical checkpoints:
- Goal and tests: historical selectors are propagated through metadata, split planning, BE scanning, and phase-two fetch. Active SQL cases cover main and dev with equal numeric versions, tags, timestamps, index coverage, deletes, missing versions, schema evolution, and row fetch. This is inspected coverage, not an executed result.
- Scope and clarity: production edits are confined to Lance search/snapshot handling and the related prepared-statement parser check. Added binary fixtures support version, branch, pruning, evolution, and legacy-index scenarios.
- Concurrency and locks: metadata is resolved per read and the BE fragment-ID cache is reader-owned; split reuse is sequential for that reader. No new thread entry, shared lock, nested-lock order, or heavy operation under a new lock was introduced.
- Lifecycle and initialization: opened URI/version/options remain fixed for reader reuse; scanner and FTS context cleanup and fragment-cache reset were traced. No new cross-translation-unit static initializer dependency was found.
- Configuration: no product configuration item was added, so dynamic-change handling is inapplicable. Regression suites restore their session settings.
- Compatibility: selector omission retains latest-main behavior. Existing Thrift version and index-segment fields carry the selected snapshot; no new storage format or transmitted variable was introduced. Legacy-index and older-BE behavior was checked against the existing inline discussion; no distinct regression on this head was substantiated.
- Parallel paths and conditions: normal scan, vector search, FTS, and take_rows were traced. Positive search/fetch versions, fragment membership, and contradictory use_index/index-segment checks fail explicitly; indexed and uncovered vector splits take their intended paths.
- Test results and negative cases: active suites and expected .out files include missing selector targets, invalid combinations, pruned history, managed namespace gaps, branch differences, and lazy materialization. No test result was generated or verified by running it in this review.
- Observability: BE profiles now name the opened version and sanitized URI, and errors name the snapshot; existing index and flat-search counters remain available.
- Transactions, persistence, and writes: production changes are external read paths, with no Doris transaction, EditLog, or product data-write change. Fixture construction stages and checks histories before catalog promotion; that construction was inspected only.
- Performance and other issues: BE caches the fragment list for an opened reader and uses FE segment selection to avoid unintended ANN work on flat splits. No further substantiated performance or correctness issue survived the final sweep.
All 109 changed paths in the authoritative diff were covered. Three normal and two risk-focused subagent passes returned NO_NEW_VALUABLE_FINDINGS. One final-sweep mixed-version hypothesis was checked and dismissed as a pre-existing legacy-index behavior at an already-commented location without a distinct verified failure on this head. No candidate remains unresolved. Review complete.
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
|
Reviewed commit Review conclusion: request changes — one managed-branch compatibility regression. The new For example, This also breaks the existing ordinary managed-table query path, not just the new search TVFs: SELECT * FROM catalog.db.items@branch('name'='dev');At the merge base, Suggested fix: retain the traversal and unsafe-URI checks, but align character acceptance with the pinned Lance contract, through native validation or matching versioned Unicode data. Add coverage for an existing managed branch containing a post-Unicode-13 letter, for both ordinary scans and search TVFs. References: JDK 17 Character, Lance v12 branch validation, Lance v12 Rust requirement, Rust 1.91 Unicode tables. Validation scope: two static review rounds covering all 109 changed files and relevant dependency contracts. No builds, tests, or JNI reproduction were run. No additional substantiated blocking findings were identified. |
|
Thanks, confirmed: a managed branch named I did not align the guard with Lance's character set exactly, because neither route stays aligned:
So the guard no longer decides whether a name is valid. It only keeps the joined URI inside the table's
Coverage, as you suggested, on an existing managed branch with a post-Unicode-13 letter:
|
|
run buildall |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
|
/review |
There was a problem hiding this comment.
Complete static review of PR #68707 at head 0eac821. No new actionable findings. All three Round 1 review passes returned NO_NEW_VALUABLE_FINDINGS; the final changed-file and unresolved-point sweep found none. There was no additional user-provided focus.
Critical checkpoints:
- Goal and evidence: Both Lance search TVFs accept version, timestamp, tag, and branch selectors. The selected URI and positive version flow from catalog metadata into FE splits, BE search, and phase-two row-id fetch. Active external regression suites inspect distinct main/branch rows, historical deletion, index coverage, schema evolution, managed namespace gaps, and lazy materialization. The new LanceSearchSnapshotTest class is disabled by its stated JNI ABI limitation, so its assertions are not executed coverage.
- Scope and parallel paths: Changes stay within selector plumbing, snapshot diagnostics, BE plan enforcement, and fixtures. Vector and full-text searches, indexed and basic metadata loading, managed and filesystem catalogs, and table scans share the relevant resolver behavior. Invalid combinations and unavailable snapshots fail explicitly.
- Concurrency and lifecycle: No new threads, locks, or shared mutable state were added. One catalog client lease reads a selected Dataset snapshot; the metadata retains its URI/version after the handle closes. A BE reader retains one dataset key for its splits; its fragment-ID cache resets on close. No new static initialization dependency was found.
- Configuration and compatibility: No configuration item or Thrift field was added. Existing URI/version fields carry selectors across FE and BE, and the smooth-upgrade check for additional reader types remains in place. The older BE's optional automatic index choice for segmentless splits predates this PR; no distinct new selector-specific wrong-result path was substantiated.
- Conditional checks, data correctness, and performance: BE rejects unpinned search/fetch versions and fragments absent from the opened snapshot. Indexed vector splits use FE-selected segment UUIDs; uncovered splits search flat. FTS retains strict/index-only coverage behavior. The extra fragment listing is once per opened BE dataset, not per batch. No reachable new correctness, memory, or performance defect was substantiated.
- Tests and results: I inspected FE and BE unit cases, fixture builders/artifacts, regression SQL, and expected output; I did not run builds or tests, as required by the review prompt. The author-reported local suite results are not independent validation. Negative cases cover invalid selectors, missing versions/tags/branches, prepared placeholders, timestamp zones, and missing fragment IDs.
- Observability, persistence, and writes: EXPLAIN now names catalog type, version, managed status, and branch; BE errors/profile identify the opened snapshot. FE changes add no EditLog, transaction, or data-write operation. No new FE-to-BE variable requires separate sending paths.
Existing inline P2/P3 threads on historical branch assertions (4172612906), legacy ANN planning (4172821000), and historical tag phase-two coverage (4172821003) were read and checked against the current diff. Their current-head follow-ups add discriminating assertions/fixtures; I found no distinct new issue to repost. No existing P0/P1 inline finding remains to carry.
Gabriel39
left a comment
There was a problem hiding this comment.
The updated managed-branch guard addresses the Unicode compatibility issue from my earlier review: it checks URI/path safety without relying on the JDK to recognize every character accepted by Lance. The added coverage includes the existing managed branch scan and search paths. Approved based on static review; tests were not run locally.
…ector_search and full_text_search vector_search() and full_text_search() take optional version, timestamp, tag and branch properties with the semantics of FOR VERSION AS OF, FOR TIME AS OF, @tag and @Branch on a Lance table. A tag stands alone, version and timestamp exclude each other, branch may be combined with either, and branch=main is the main chain. version takes only a number, since tag names a tag without ambiguity, and an empty value is an error rather than an absent selector. The selector goes through the same catalog snapshot resolution as table scans, including REST Namespace managed tables, and the schema, search column, fragments and index metadata come from the selected snapshot. Every execution resolves the selector again; nothing falls back to the latest version. EXPLAIN of a search node shows the catalog type, the managed flag and the branch, and planner errors name the branch. full_text_search rejects prepared-statement placeholders, which used to reach it as the literal text "?". A managed table's branch is opened by a URI Doris joins, so Doris now checks the branch name against Lance's rules before joining it. The time-resolution error no longer names FOR TIME AS OF, since the timestamp property reaches it too. The BE rejects search splits and row-id fetches without a positive version, rejects FE fragment ids the opened snapshot lacks (lance-c would skip them silently), names the dataset version and URI in execution errors, and records the snapshot it read in the profile. New regression fixtures search_snapshot.lance, search_snapshot_pruned.lance and search_snapshot_evolved.lance cover index history, deletes, an index rebuild, tags, a branch overlapping main's versions, cleanup, and a column added and renamed after the index was built.
The FE plans a vector-search split without index segments as a flat search, and EXPLAIN and the profile count it as one. The BE forwarded use_index only when the user set it, so Lance searched such a split with the first index on the column in manifest order, which need not be the index the FE selected, for example while an index is rebuilt under a new name, or an index the FE could not plan with at all. The BE now sets use_index to whether the split carries index segments. A split that carries index segments while the request says use_index=false is rejected. The FE plans segments only when use_index allows them, so such a split contradicts itself, and the BE would otherwise have to override either the plan or the request.
…h_snapshot branch=dev with version 4 selects the fork point, whose rows and indexes match main version 4, and the branch timestamp case asked for a future time, which selects the branch head even if the time were ignored. Add assertions that tell the branch apart from main and the time from the head: dev version 5 returns the branch's rows, STRICT full-text search succeeds on dev version 6 (its index was rebuilt over fragment 2) and fails on main version 6, and a time between dev versions 5 and 6 selects dev version 5.
…fetch A vector index written before Lance 0.20 carries no index details. Lance infers them from the index files when it loads the index, so the FE plans it and the BE searches it through the planned segment rather than flat. legacy_vector_index.lance, built by pylance 0.18.2 and carried over as-is by the catalog build, and test_lance_vector_search_legacy_index check that it is planned with one segment and returns the flat search's rows. The tag two-phase case now fetches row 3, which version 5 still has and version 6 deleted, so a fetch from a later version would return fewer rows, and it asserts the materialization node.
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
…search functions (#4185) Documents the `version`, `timestamp`, `tag` and `branch` parameters of `vector_search()` and `full_text_search()`, added by apache/doris#68707 (issue apache/doris#68625). Changes to `lakehouse/catalogs/lance-catalog.mdx` (4.x, English and Chinese): - Time Travel limitations: the search functions now select a snapshot with their own parameters. Index inspection (`SHOW INDEX`, `lance_index_entries()`) still uses the latest version of `main`. - `vector_search()` and `full_text_search()` parameter tables: the four selector parameters. - Vector index selection: when several compatible indexes exist, Doris uses one per query, the first by index name, and fragments it does not cover use Flat Search. Indexes Doris cannot plan with, including legacy indexes without index details, are not used. - Catalog refresh: a refresh with cache invalidation clears the FE's Lance Session only, not the BEs' caches. - New section "Search a Historical Version or Branch": combination rules, what is bound to the selected snapshot, no fallback to the latest version, selection per execution, per relation and in `CREATE TABLE AS SELECT`, the `EXPLAIN` and Profile fields, and known limitations (branches of branches, a branch or Dataset recreated at the same location, FTS indexes without token positions). - Execution flow: the snapshot step names the selectors. ## Versions - [ ] dev - [x] 4.x - [ ] 3.x - [ ] 2.1 or older (not covered by version/language sync gate) ## Languages - [x] Chinese - [x] English ## Docs Checklist - [x] Checked by AI - [ ] Test Cases Built - [x] Updated required version and language counterparts, or explained why not: the dev docs have no Lance catalog page yet, so only 4.x changes. - [x] If only one language changed, confirmed whether source/translation counterparts need sync
…ector_search and full_text_search (#68707) ### What problem does this PR solve? Issue Number: close #68625 Related PR: #68453 (time travel, tags, branches and managed versioning for Lance table scans) Problem Summary: `vector_search()` and `full_text_search()` always searched the latest version of `main`, and their `table` argument rejected `FOR VERSION AS OF`, `@tag` and `@branch`. A search could not be reproduced against a released tag, compared across versions, or run on an experimental branch. Both TVFs now take four optional properties, with the semantics of the table syntax from #68453: | Property | Value | Table syntax | Combines with | |-|-|-|-| | `version` | positive integer | `FOR VERSION AS OF N` | `branch` | | `timestamp` | `yyyy-MM-dd HH:mm:ss[.SSS]`, session time zone | `FOR TIME AS OF` | `branch` | | `tag` | tag name | `@tag(name)` | none | | `branch` | branch name, `main` is the main chain | `@branch(name)` | `version` or `timestamp` | ```sql SELECT id, _distance FROM vector_search( "table" = "lance_catalog.db.documents", "column" = "embedding", "query_vector" = "[0.1, 0.2, 0.3]", "version" = "3") ORDER BY _distance; SELECT id, _score FROM full_text_search( "table" = "lance_catalog.db.documents", "column" = "content", "query" = "database", "branch" = "dev", "timestamp" = "2026-09-20 12:00:00") ORDER BY _score DESC; ``` How it works: - **FE.** The TVF parses the properties into a `LanceRefSelector` and loads its metadata through the same `LanceCatalogClient.readTableSnapshot` as table scans, so storage-versioned and namespace-managed tables follow the #68453 rules unchanged. Schema, search column and field id, vector type and dimension, fragments and index segments all come from the selected snapshot, and planning never reopens the dataset. The splits carry the branch directory URI and the version, so no thrift change is needed. - **Each execution resolves again.** A bound TVF is never cached, prepared statements re-analyze on every `EXECUTE`, and TVF statements skip the SQL cache, so the latest version, a future time, a branch head and a moved tag are resolved per execution and stay fixed within its plan. `CREATE TABLE AS SELECT` plans its query twice (schema, then rows), so there a moving selector can resolve twice, as it already can for a table scan with an explicit selector; pinning `version` gives an exact copy. This is documented. - **No fallback.** A snapshot that cannot be selected, or whose index files are missing, is an error; nothing falls back to the latest version. - **EXPLAIN** of a search node now shows `lanceCatalogType`, `lanceManagedVersioning` and `lanceBranch` next to `lanceVersion`, as table scans do. Planner errors that name "dataset version N" also name the branch, since branch versions overlap main's. Decisions worth calling out: - `version` takes only a number. `FOR VERSION AS OF 'x'` names a tag because SQL has one clause for both; the TVF has a `tag` property, and Lance allows tag names made of digits, which `FOR VERSION AS OF` cannot reach. - An empty selector value is an error rather than "not set", so an empty client-side variable cannot silently search the latest version of `main`. - The selector properties must be constants. Accepting `?` would leave PREPARE without a snapshot to take the result schema from. `full_text_search` now rejects `?` in any property: until now a placeholder reached it as the literal text "?", so `'query'=?` searched for "?". - Each search relation resolves its own snapshot, as each explicit selector already does for table scans. `vector_search(t) JOIN t` can read two versions if a commit lands in between; pinning both with `version` / `FOR VERSION AS OF` avoids that. This is documented. - A managed table's branch is opened by a URI Doris joins (`<table>/tree/<branch>`), which Lance does not validate on that path, so Doris now rejects names that could leave the table's `tree/` directory or change that URI: empty segments, `..`, `\`, and ASCII characters other than letters, digits, `.`, `-`, `_` and `/`, so `../other` and a percent-encoded dot segment are rejected before the branch is opened. Outside ASCII only whitespace and control characters are rejected, so no name Lance accepts is rejected, whichever Unicode version the JDK and Lance's Rust use. Other tables check a branch out through Lance, which validates the name itself. BE changes: - A search split or a two-phase row fetch without a positive version is rejected instead of opening the latest version. - Errors from opening the dataset, reading batches, preparing the FTS query context and the two-phase row fetch name the snapshot: "... at Lance dataset version N at <uri>", with the query string and any password removed from the URI. Missing index files surface at execution because Lance loads indexes lazily. - Fragment ids from the FE must exist in the opened snapshot. lance-c silently skips unknown fragment ids, which would drop rows without an error. - The profile records `LanceDatasetVersion` and `LanceDatasetUri`, the snapshot an execution actually read. EXPLAIN cannot show this for moving selectors because it resolves again. - **Behavior change for every vector search:** a split without index segments now always runs flat (`use_index=false`). The FE plans such splits as flat search, and EXPLAIN and the profile count them as flat, but Lance used to search them with the first index on the column in manifest order, which need not be the index the FE selected (for example while an index is rebuilt under a new name). Indexes the FE cannot plan with (no fragment coverage metadata, no recorded metric, or a legacy index whose details Lance cannot infer from its files, which the FE skips; `lanceVectorIndexStatus=UNKNOWN_COVERAGE` / `METRIC_MISMATCH` / `NO_MATCH`) are therefore no longer used implicitly, and such searches become exact flat searches. Results can change only where the unplanned index was approximate; latency can grow on such datasets. Legacy indexes that lack details but whose details Lance infers, such as those written by Lance 0.10 to 0.24, are planned and searched as before (`test_lance_vector_search_legacy_index`). - A vector split that carries index segments while the request says `use_index=false` is rejected. The FE plans segments only when `use_index` allows them, so it never sends such a split; the BE now takes `use_index` from the segments and would otherwise have to override either the plan or the request. Known limitations (documented): - A branch of a branch, or a branch of a shallow clone, can fail to find the index files it inherited, because Lance writers before v13 record them under the intermediate branch (lance-format/lance#9176). The search fails with an error. - After a branch is deleted and recreated under the same name, or a dataset is replaced at the same URI, version numbers repeat. Since #68615 the BE's Lance keys the index list and the row-id index by the manifest e_tag (lance-format/lance#8904, #8930), but the BE's process-wide Lance session still keys two entries by version alone, until eviction or restart. On datasets with stable row ids, the live-row mask that vector and full-text search prefilter with can come from the old data, so rows deleted in the new data can reappear or live rows can be missing (lance-format/lance#9079, open). For a manifest without an e_tag (V1 naming, or a store that returns no ETag), the index list can come from the old data, on the BE and in the FE's Lance session until the catalog is refreshed. Neither is introduced by this PR. - `SHOW INDEX` and `lance_index_entries()` keep showing the latest version of `main`. ### Release note `vector_search()` and `full_text_search()` accept `version`, `timestamp`, `tag` and `branch` to search a historical version, the version at a time, a tag or a branch of a Lance table, including REST Namespace managed tables. Vector search no longer uses an index the planner did not select, and `full_text_search()` rejects prepared-statement placeholders. ### Check List (For Author) - Test: Regression test / Unit Test - Regression: `external_table_p0/lance/test_lance_search_snapshot` (filesystem catalog) and `test_lance_rest_search_snapshot` (REST, storage-versioned and managed) on a new fixture `search_snapshot.lance`. It has seven versions (appends, a vector index, an FTS index, an append no index covers, a delete, a vector index rebuild), a tag, and a branch `dev` whose versions and fragment ids overlap main's with different rows, plus a tag at a non-latest branch version. `search_snapshot_pruned.lance` lost a version to cleanup and the index files of a tagged version. `search_snapshot_evolved.lance` adds a column and renames the vector column after its index is built, then appends data the index does not cover. The suites check rows, distances, two-phase reads (on a branch, a tag, the columns of a version before and after a rename, and a managed table through REST), EXPLAIN, full-text search across a delete, `use_index=false` with a selector, every selector error, and a managed branch named with a Unicode 16 letter (`unicode_branch.lance`, built by pylance 12.0.0) through a table scan, a search and a two-phase read. `test_lance_rest_catalog.out` and `test_lance_rest_time_travel.out` list the new REST tables. `test_lance_vector_search_legacy_index` searches a vector index written by pylance 0.18.2 without index details and checks that it is planned (`USED`, one segment) and returns the flat search's rows. All 30 suites in `external_table_p0/lance` pass locally. - FE unit tests: `LanceSearchSelectorTest` (property rules, time zones, selector forwarding), `LancePreparedSearchTest` (both TVFs pass their selector to the table; each analysis resolves it exactly once, and a moved tag is resolved again on the next `EXECUTE`; search statements bypass the SQL cache; placeholders rejected), `LanceScanNodeTest` (EXPLAIN, errors naming the branch), `LanceCatalogClientTest` (branch names, including letters newer than JDK 17's Unicode tables), and, with real datasets and locally only like the other JNI tests, `LanceSearchSnapshotTest` (historical index coverage, branches, tags, schema evolution, cleanup, a snapshot planned before later commits) and `LanceManagedVersioningTest` (managed versions, tags, branches, unfinalized heads through the search entry points). - BE unit tests in `lance_reader_test.cpp`: missing fragment ids on normal, vector and FTS splits, unpinned versions, error and profile contents, a split without index segments searching flat although an index covers it, and a split with index segments under `use_index=false` being rejected. The indexed variants of the existing multi-vector tests now pass the index segments the FE would plan. - Behavior changed: Yes. New TVF properties; vector splits without index segments always search flat; `full_text_search` rejects `?`; managed branch names are validated; the error for a time in removed history now reads "Lance cannot select the version at or before '...'" instead of "Lance cannot resolve FOR TIME AS OF '...'", because the TVF reaches it too; BE errors name the dataset version and URI. - Does this need documentation: Yes (apache/doris-website#4185)
… at graceful exit (#68942) Problem Summary: **In short.** The NonConcurrent regression pipeline fails a few percent of its runs with `Lsan fail` while every case passes. LeakSanitizer finds, at the BE's graceful exit, one stream load's `ResourceContext` that was never released, but its report only shows allocation stacks, which are the same for every stream load, so nobody can tell which load it was. This PR makes the BE print, at graceful exit in memleak-check mode, every `ResourceContext` still alive with its task id, the references still held, its memory tracker and workload group, and whether its `QueryContext` is already gone. Nothing changes unless `enable_graceful_exit_check=true`. **Background.** - With `enable_graceful_exit_check=true` (memleak-check mode, used by the regression pipelines on the ASAN build), `doris_main` stops every service and runs `ExecEnv::destroy()` instead of `_exit()`, so that LeakSanitizer checks the heap when the process exits. The NonConcurrent pipeline greps `be.out` for `detected memory leaks` after `stop_be.sh --grace` and reports `Lsan fail`. - Every query or load has a `ResourceContext`: CPU/memory/IO contexts, a task controller, the query memory tracker and the workload group. The `QueryContext` owns it, and the components that work for the task (load channel, memtables, scanner context, `RuntimeQueryStatisticsMgr`, threads attached to the task, ...) hold references of their own while they run. **The problem, and what it cost.** - `Lsan fail` hits 1–5% of NonConcurrent runs depending on the month (448 of the last 14000, back to March 2026), on PRs that have nothing to do with it; authors rerun or skip. Every report TeamCity still keeps (10 runs from 2026-10-05 to 2026-10-12, on #68707, #68281, #68791, #68803, #68798, #68543, #68901, #68919, #68879) has the same shape, down to the byte counts, and all of its leaks are indirect: | Leaked | Size | Origin | |---|---|---| | one `ResourceContext` with its CPU/memory/IO contexts, `QueryTaskController` and both trackers (`Load#Id=…`, `Memtable#Load#Id=…`) | 6.5 KB, 94 objects | an HTTP stream load: `StreamLoadAction::_process_put` → `StreamLoadExecutor` → `FragmentMgr::_get_or_create_query_ctx` | | the `normal` workload group created by `WorkloadGroupMgr::create_internal_wg()` | 6.1 KB + 4.3 KB | kept alive by `ResourceContext::_workload_group` | | the `std::make_shared` block of the load's `QueryContext` | 2.5 KB | kept allocated by the `weak_ptr` in `QueryTaskController` | - The `QueryContext` itself was destroyed: in `be.INFO`, each of the 21894 `query_id: … use wg:` lines of the run has its `Query … deconstructed`. So one reference to the load's `ResourceContext` is never released, from a place LeakSanitizer does not scan (a bthread stack, whose memory brpc maps itself) or a count that is never decremented. LeakSanitizer cannot tell which of these two it is. - A NonConcurrent run issues about 430 HTTP stream loads (about 380 through the load channel, 150 of them audit-log loads), and the report cannot say which one leaked: the stacks are identical, and the one thing that would identify the load, the tracker label with its id, is inside the leaked memory, which LeakSanitizer does not print. Without the load there is nothing to follow, so the flake stays unexplained. **How this PR fixes it.** - In memleak-check mode each `ResourceContext` registers itself in a process-wide set when it is constructed and removes itself when it is destroyed. The set stores the addresses bitwise inverted, so LeakSanitizer does not take it for a reference and still reports a leaked context. - `ResourceContext::describe_live_contexts()` returns one line per context still alive: address, strong references still held, task id, finished flag, running time, memory tracker (label, consumption, peak), workload group and, for a query, whether its `QueryContext` is still alive. - `doris_main` calls `ResourceContext::log_live_contexts()` right after `ExecEnv::destroy()` on the graceful exit path. It writes the description to `be.WARNING` and to `be.out`, where the LeakSanitizer report follows once `main()` returns, or one INFO line when no context is alive. The printed address falls inside one of the `Objects leaked above` ranges of that report. - The description is built on a short-lived thread, and the main thread only handles the resulting text. Describing a context touches it; done on the main thread, it could leave a pointer to a leaked context in a dead slot of the main thread's stack, which LeakSanitizer scans at exit — it would then take the context for referenced and not report it, so the run that reproduces the leak could pass. The stack of a finished thread is not scanned. **Results.** What the next `Lsan fail` will tell, compared with today: | | Before | After | |---|---|---| | Which query or load leaked | unknown | `task_id=…`; its label, table and test are then in `be.INFO` | | How many references are left | unknown | `references=N` | | Whether the `QueryContext` is gone | only by pairing ~20k log lines | `query_context=destroyed` | | Where to look | the LeakSanitizer report in `be.out` | the same `be.out`, just before that report (the build log's `head -n 100 be.out` shows it) | | Cost with `enable_graceful_exit_check=false` (default) | — | one boolean check per `ResourceContext` construction and destruction | A line looks like this (values made up): ``` 1 resource contexts are still alive. LeakSanitizer reports a leaked context as the memory block that holds it, so match the addresses below against the ranges it prints under 'Objects leaked above'. 0x7c0000001010: references=1, task_id=1eac0c4ec4ec0000-0000000000000003, finished=true, running_ms=7569, mem_tracker=Load#Id=1eac0c4ec4ec0000-0000000000000003 (consumption=0, peak=17.18 MB), workload_group=normal(1), query_context=destroyed ``` **Classes, and how they call each other.** - `ResourceContext` (`resource_context.{h,cpp}`): registers in the constructor and unregisters in the destructor when memleak-check mode is on; `describe_live_contexts()` formats the live ones; `log_live_contexts()` runs that on its own thread and logs the result. - `QueryTaskController`: new `is_query_context_alive()`. - `doris_main`: one call to `log_live_contexts()` after `ExecEnv::destroy()`. ``` doris_main (graceful exit, enable_graceful_exit_check=true) ├─ ExecEnv::destroy() └─ ResourceContext::log_live_contexts() ──► be.WARNING + be.out (or one INFO line) └─ [short-lived thread] describe_live_contexts(100) ├─ registry of disguised addresses ◄── ResourceContext() / ~ResourceContext() └─ per live context: ├─ TaskController: task id, finished, running time ├─ MemoryContext → MemTrackerLimiter: label, consumption, peak ├─ WorkloadGroup: name, id └─ QueryTaskController::is_query_context_alive() ``` No holder of `ResourceContext` changes, and neither do the LeakSanitizer options or suppressions. Once the leaked load is known, the fix for the leak itself will be a separate PR. --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…69126) ### What problem does this PR solve? Related PR: #69124, #68707 Apply the fixture portability fix from #69124 to `branch-4.2`. The Unicode Lance branch fixture tracks five files below `tree/dev/`, causing checkout failures on filesystems that cannot represent that directory name. Store these files under `tree/dev_u1c89/` in Git and restore the original Unicode object prefix when uploading to MinIO. Fixture payloads and runtime Unicode branch coverage remain unchanged. Update the fixture checker and regeneration instructions to use the portable layout, and fail initialization if upload or relocation fails. ### Release note None ### Check List (For Author) - Test - [x] Manual test (details below) - Behavior changed: - [x] No. Runtime fixture paths and contents are unchanged. - Does this need documentation? - [x] No. Fixture regeneration instructions are updated inline. Validation on this branch: - All tracked paths are ASCII and contain no literal backslashes. - Fresh checkout of all 10 Unicode fixture files with `core.protectNTFS=true` and `core.protectHFS=true` on Linux. - `lance_build_unicode_branch.py --check` passes with pylance 12.0.0. - Python, YAML and shell syntax checks and `git diff --check` pass. - Verified the complete Iceberg fixture/setup subtree is identical to #69124. That change was validated against temporary MinIO with two initialization runs, matching all 515 original object keys and blob hashes, and successful reads of both Unicode branch versions and vector search. The MinIO integration test was not repeated for this identical port. Full Doris regression tests and a native Windows checkout were not run locally. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
### What problem does this PR solve? Related PR: #68707 The Unicode Lance branch fixture introduced by #68707 tracks five files below `tree/dev/`. This causes checkout failures on filesystems that cannot represent that directory name. Store these files under the ASCII directory `tree/dev_u1c89/` and restore the original Unicode object prefix when uploading fixtures to MinIO. The fixture payloads remain byte-for-byte identical, and the existing Unicode branch regression coverage is preserved. Update the fixture checker and regeneration instructions to use the portable Git layout. Initialization exits on upload or relocation failure instead of continuing with an incomplete fixture. ### Release note None ### Check List (For Author) - Test - [x] Manual test (details below) - Behavior changed: - [x] No. Runtime fixture paths and contents are unchanged. - Does this need documentation? - [x] No. Fixture regeneration instructions are updated inline. Validation: 1. Scanned all tracked paths: no non-ASCII names or literal backslashes remain. Checked out all 10 files in the Unicode fixture using a fresh Git index with `core.protectNTFS=true` and `core.protectHFS=true` on Linux. 2. Ran `lance_build_unicode_branch.py --check` on the renamed fixture with pylance 12.0.0. 3. Executed the upload commands extracted from `iceberg.yaml.tpl` against a temporary MinIO instance with the pinned mc image. Ran initialization twice, injecting a stale object before the second run. After each run, all 515 object keys and blob hashes matched the original fixtures exactly; neither the ASCII alias nor the stale object remained. 4. Read the main snapshot and both Unicode branch versions with pylance 12.0.0. Confirmed 8 rows in branch version 1, 12 rows in version 2, and vector search returning the expected branch-only rows. 5. Passed YAML parsing, shell syntax validation, and `git diff --check`. Full Doris regression tests and a native Windows checkout were not run locally. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
… planned ### What problem does this PR solve? Issue Number: None Related PR: apache#69581, apache#68707, apache#68815 Problem Summary: apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where `_configure_vector_search()` passes the request's `use_index` to `lance_scanner_set_use_index()`. FE decides per split whether to use an index: a split it plans with index segments carries their UUIDs, and a split without them is meant to be searched flat. With `use_index = true` and no segments, Lance instead searches the split with the first index on the column in manifest order. That index need not be the one FE chose: when FE finds no index whose metric matches the query, it plans every split flat, and the snapshot BE still searched them with the mismatched index. On branch-4.2, apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as flat fallback splits, because quantized distances can drop rows inside the range; the snapshot BE searches those with the index and returns too few rows. Fix: port the split-level rule of apache#68707. - `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with segments searches exactly those, a split without segments searches flat, and `LancePlannedFlatSearchFragmentCount` now matches what Lance does. - A split that carries index segments while the request sets `use_index = false` is rejected with InvalidArgument. FE plans segments only when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2 alike, so a 4.x FE never sends this combination. - Not ported: the rest of apache#68707 (searching historical versions, tags and branches; rejecting fragment ids missing from the snapshot; naming the snapshot in errors and the profile). That comes with the next 4.x catch-up. ### Release note None ### Check List (For Author) - Test: Unit Test - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)` constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`, `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`. - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`, `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and `MultiVectorCosineMasksUndefinedRows` pass the index segments for their indexed cases (new helper `make_indexed_lance_range`) and check `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments` checks that every split is searched flat. - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the Lance suites. With `use_index` taken from the request and the rejection removed, the two new cases fail. - Behavior changed: Yes, on the Lance path added by apache#69581: a vector split without index segments is searched flat even when `use_index` is true, and a split with segments under `use_index = false` is rejected. Master has no FE that plans Lance scans yet. - Does this need documentation: No Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… planned ### What problem does this PR solve? Issue Number: None Related PR: apache#69581, apache#68707, apache#68815 Problem Summary: apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where `_configure_vector_search()` passes the request's `use_index` to `lance_scanner_set_use_index()`. FE decides per split whether to use an index: a split it plans with index segments carries their UUIDs, and a split without them is meant to be searched flat. With `use_index = true` and no segments, Lance instead searches the split with the first index on the column in manifest order. That index need not be the one FE chose: when FE finds no index whose metric matches the query, it plans every split flat, and the snapshot BE still searched them with the mismatched index. On branch-4.2, apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as flat fallback splits, because quantized distances can drop rows inside the range; the snapshot BE searches those with the index and returns too few rows. Fix: port the split-level rule of apache#68707. - `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with segments searches exactly those, a split without segments searches flat, and `LancePlannedFlatSearchFragmentCount` now matches what Lance does. - A split that carries index segments while the request sets `use_index = false` is rejected with InvalidArgument. FE plans segments only when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2 alike, so a 4.x FE never sends this combination. - Not ported: the rest of apache#68707 (searching historical versions, tags and branches; rejecting fragment ids missing from the snapshot; naming the snapshot in errors and the profile). That comes with the next 4.x catch-up. ### Release note None ### Check List (For Author) - Test: Unit Test - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)` constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`, `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`. - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`, `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and `MultiVectorCosineMasksUndefinedRows` pass the index segments for their indexed cases (new helper `make_indexed_lance_range`) and check `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments` checks that every split is searched flat. - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the Lance suites. With `use_index` taken from the request and the rejection removed, the two new cases fail. - Behavior changed: Yes, on the Lance path added by apache#69581: a vector split without index segments is searched flat even when `use_index` is true, and a split with segments under `use_index = false` is rejected. Master has no FE that plans Lance scans yet. - Does this need documentation: No Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… planned ### What problem does this PR solve? Issue Number: None Related PR: apache#69581, apache#68707, apache#68815 Problem Summary: apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where `_configure_vector_search()` passes the request's `use_index` to `lance_scanner_set_use_index()`. FE decides per split whether to use an index: a split it plans with index segments carries their UUIDs, and a split without them is meant to be searched flat. With `use_index = true` and no segments, Lance instead searches the split with the first index on the column in manifest order. That index need not be the one FE chose: when FE finds no index whose metric matches the query, it plans every split flat, and the snapshot BE still searched them with the mismatched index. On branch-4.2, apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as flat fallback splits, because quantized distances can drop rows inside the range; the snapshot BE searches those with the index and returns too few rows. Fix: port the split-level rule of apache#68707. - `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with segments searches exactly those, a split without segments searches flat, and `LancePlannedFlatSearchFragmentCount` now matches what Lance does. - A split that carries index segments while the request sets `use_index = false` is rejected with InvalidArgument. FE plans segments only when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2 alike, so a 4.x FE never sends this combination. - Not ported: the rest of apache#68707 (searching historical versions, tags and branches; rejecting fragment ids missing from the snapshot; naming the snapshot in errors and the profile). That comes with the next 4.x catch-up. ### Release note None ### Check List (For Author) - Test: Unit Test - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)` constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`, `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`. - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`, `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and `MultiVectorCosineMasksUndefinedRows` pass the index segments for their indexed cases (new helper `make_indexed_lance_range`) and check `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments` checks that every split is searched flat. - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the Lance suites. With `use_index` taken from the request and the rejection removed, the two new cases fail. - Behavior changed: Yes, on the Lance path added by apache#69581: a vector split without index segments is searched flat even when `use_index` is true, and a split with segments under `use_index = false` is rejected. Master has no FE that plans Lance scans yet. - Does this need documentation: No Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… planned ### What problem does this PR solve? Issue Number: None Related PR: apache#69581, apache#68707, apache#68815 Problem Summary: apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where `_configure_vector_search()` passes the request's `use_index` to `lance_scanner_set_use_index()`. FE decides per split whether to use an index: a split it plans with index segments carries their UUIDs, and a split without them is meant to be searched flat. With `use_index = true` and no segments, Lance instead searches the split with the first index on the column in manifest order. That index need not be the one FE chose: when FE finds no index whose metric matches the query, it plans every split flat, and the snapshot BE still searched them with the mismatched index. On branch-4.2, apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as flat fallback splits, because quantized distances can drop rows inside the range; the snapshot BE searches those with the index and returns too few rows. Fix: port the split-level rule of apache#68707. - `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with segments searches exactly those, a split without segments searches flat, and `LancePlannedFlatSearchFragmentCount` now matches what Lance does. - A split that carries index segments while the request sets `use_index = false` is rejected with InvalidArgument. FE plans segments only when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2 alike, so a 4.x FE never sends this combination. - Not ported: the rest of apache#68707 (searching historical versions, tags and branches; rejecting fragment ids missing from the snapshot; naming the snapshot in errors and the profile). That comes with the next 4.x catch-up. ### Release note None ### Check List (For Author) - Test: Unit Test - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)` constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`, `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`. - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`, `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and `MultiVectorCosineMasksUndefinedRows` pass the index segments for their indexed cases (new helper `make_indexed_lance_range`) and check `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments` checks that every split is searched flat. - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the Lance suites. With `use_index` taken from the request and the rejection removed, the two new cases fail. - Behavior changed: Yes, on the Lance path added by apache#69581: a vector split without index segments is searched flat even when `use_index` is true, and a split with segments under `use_index = false` is rejected. Master has no FE that plans Lance scans yet. - Does this need documentation: No Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… planned ### What problem does this PR solve? Issue Number: None Related PR: apache#69581, apache#68707, apache#68815 Problem Summary: apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where `_configure_vector_search()` passes the request's `use_index` to `lance_scanner_set_use_index()`. FE decides per split whether to use an index: a split it plans with index segments carries their UUIDs, and a split without them is meant to be searched flat. With `use_index = true` and no segments, Lance instead searches the split with the first index on the column in manifest order. That index need not be the one FE chose: when FE finds no index whose metric matches the query, it plans every split flat, and the snapshot BE still searched them with the mismatched index. On branch-4.2, apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as flat fallback splits, because quantized distances can drop rows inside the range; the snapshot BE searches those with the index and returns too few rows. Fix: port the split-level rule of apache#68707. - `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with segments searches exactly those, a split without segments searches flat, and `LancePlannedFlatSearchFragmentCount` now matches what Lance does. - A split that carries index segments while the request sets `use_index = false` is rejected with InvalidArgument. FE plans segments only when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2 alike, so a 4.x FE never sends this combination. - Not ported: the rest of apache#68707 (searching historical versions, tags and branches; rejecting fragment ids missing from the snapshot; naming the snapshot in errors and the profile). That comes with the next 4.x catch-up. ### Release note None ### Check List (For Author) - Test: Unit Test - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)` constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`, `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`. - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`, `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and `MultiVectorCosineMasksUndefinedRows` pass the index segments for their indexed cases (new helper `make_indexed_lance_range`) and check `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments` checks that every split is searched flat. - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the Lance suites. With `use_index` taken from the request and the rejection removed, the two new cases fail. - Behavior changed: Yes, on the Lance path added by apache#69581: a vector split without index segments is searched flat even when `use_index` is true, and a split with segments under `use_index = false` is rejected. Master has no FE that plans Lance scans yet. - Does this need documentation: No Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… planned ### What problem does this PR solve? Issue Number: None Related PR: apache#69581, apache#68707, apache#68815 Problem Summary: apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where `_configure_vector_search()` passes the request's `use_index` to `lance_scanner_set_use_index()`. FE decides per split whether to use an index: a split it plans with index segments carries their UUIDs, and a split without them is meant to be searched flat. With `use_index = true` and no segments, Lance instead searches the split with the first index on the column in manifest order. That index need not be the one FE chose: when FE finds no index whose metric matches the query, it plans every split flat, and the snapshot BE still searched them with the mismatched index. On branch-4.2, apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as flat fallback splits, because quantized distances can drop rows inside the range; the snapshot BE searches those with the index and returns too few rows. Fix: port the split-level rule of apache#68707. - `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with segments searches exactly those, a split without segments searches flat, and `LancePlannedFlatSearchFragmentCount` now matches what Lance does. - A split that carries index segments while the request sets `use_index = false` is rejected with InvalidArgument. FE plans segments only when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2 alike, so a 4.x FE never sends this combination. - Not ported: the rest of apache#68707 (searching historical versions, tags and branches; rejecting fragment ids missing from the snapshot; naming the snapshot in errors and the profile). That comes with the next 4.x catch-up. ### Release note None ### Check List (For Author) - Test: Unit Test - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)` constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`, `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`. - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`, `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and `MultiVectorCosineMasksUndefinedRows` pass the index segments for their indexed cases (new helper `make_indexed_lance_range`) and check `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments` checks that every split is searched flat. - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the Lance suites. With `use_index` taken from the request and the rejection removed, the two new cases fail. - Behavior changed: Yes, on the Lance path added by apache#69581: a vector split without index segments is searched flat even when `use_index` is true, and a split with segments under `use_index = false` is rejected. Master has no FE that plans Lance scans yet. - Does this need documentation: No Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… planned ### What problem does this PR solve? Issue Number: None Related PR: apache#69581, apache#68707, apache#68815 Problem Summary: apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where `_configure_vector_search()` passes the request's `use_index` to `lance_scanner_set_use_index()`. FE decides per split whether to use an index: a split it plans with index segments carries their UUIDs, and a split without them is meant to be searched flat. With `use_index = true` and no segments, Lance instead searches the split with the first index on the column in manifest order. That index need not be the one FE chose: when FE finds no index whose metric matches the query, it plans every split flat, and the snapshot BE still searched them with the mismatched index. On branch-4.2, apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as flat fallback splits, because quantized distances can drop rows inside the range; the snapshot BE searches those with the index and returns too few rows. Fix: port the split-level rule of apache#68707. - `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with segments searches exactly those, a split without segments searches flat, and `LancePlannedFlatSearchFragmentCount` now matches what Lance does. - A split that carries index segments while the request sets `use_index = false` is rejected with InvalidArgument. FE plans segments only when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2 alike, so a 4.x FE never sends this combination. - Not ported: the rest of apache#68707 (searching historical versions, tags and branches; rejecting fragment ids missing from the snapshot; naming the snapshot in errors and the profile). That comes with the next 4.x catch-up. ### Release note None ### Check List (For Author) - Test: Unit Test - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)` constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`, `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`. - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`, `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and `MultiVectorCosineMasksUndefinedRows` pass the index segments for their indexed cases (new helper `make_indexed_lance_range`) and check `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments` checks that every split is searched flat. - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the Lance suites. With `use_index` taken from the request and the rejection removed, the two new cases fail. - Behavior changed: Yes, on the Lance path added by apache#69581: a vector split without index segments is searched flat even when `use_index` is true, and a split with segments under `use_index = false` is rejected. Master has no FE that plans Lance scans yet. - Does this need documentation: No Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids, in place of the comments that reserve them on master (apache#70075); field 6 stays master's paimon_options. TVectorSearchOptions 6 is branch-4.2's approx_mode (apache#69852), reserved in a comment. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… planned ### What problem does this PR solve? Issue Number: None Related PR: apache#69581, apache#68707, apache#68815 Problem Summary: apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where `_configure_vector_search()` passes the request's `use_index` to `lance_scanner_set_use_index()`. FE decides per split whether to use an index: a split it plans with index segments carries their UUIDs, and a split without them is meant to be searched flat. With `use_index = true` and no segments, Lance instead searches the split with the first index on the column in manifest order. That index need not be the one FE chose: when FE finds no index whose metric matches the query, it plans every split flat, and the snapshot BE still searched them with the mismatched index. On branch-4.2, apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as flat fallback splits, because quantized distances can drop rows inside the range; the snapshot BE searches those with the index and returns too few rows. Fix: port the split-level rule of apache#68707. - `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with segments searches exactly those, a split without segments searches flat, and `LancePlannedFlatSearchFragmentCount` now matches what Lance does. - A split that carries index segments while the request sets `use_index = false` is rejected with InvalidArgument. FE plans segments only when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2 alike, so a 4.x FE never sends this combination. - Not ported: the rest of apache#68707 (searching historical versions, tags and branches; rejecting fragment ids missing from the snapshot; naming the snapshot in errors and the profile). That comes with the next 4.x catch-up. ### Release note None ### Check List (For Author) - Test: Unit Test - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)` constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`, `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`. - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`, `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and `MultiVectorCosineMasksUndefinedRows` pass the index segments for their indexed cases (new helper `make_indexed_lance_range`) and check `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments` checks that every split is searched flat. - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the Lance suites. With `use_index` taken from the request and the rejection removed, the two new cases fail. - Behavior changed: Yes, on the Lance path added by apache#69581: a vector split without index segments is searched flat even when `use_index` is true, and a split with segments under `use_index = false` is rejected. Master has no FE that plans Lance scans yet. - Does this need documentation: No Co-authored-by: zy-kkk <zhongyk10@gmail.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
### What problem does this PR solve? Issue Number: None Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110 #67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690 #68813 #68815 #68707 #68649 Problem Summary: **In short.** This ports the Lance BE read path of branch-4.1 / branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus the BE side of three 4.x changes merged after that snapshot: the search filter `prefilter` flag and profile layout (#68813), distance bounds for `vector_search` (#68815), and the split-level index rule of #68707 that #68815 relies on. It gathers the BE parts of the PRs listed above. Master links `liblance_c` but has no Lance reader, so every `FORMAT_LANCE` scan fails today. With this PR the BE reads Lance datasets through FileScannerV2: plain scans, vector search and full-text search, predicate and runtime-filter pushdown, the schema fetch of a local Lance TVF, and the second phase of TopN lazy materialization. The PR is BE only. The FE connector plugin and the regression suites come in separate PRs, so until then nothing on master plans a Lance scan, and the BE unit tests carry the proof. Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju @FANNG1 **Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` / `s3()` with `format=lance`, and the `vector_search` / `full_text_search` TVFs. On master the FE side is being rebuilt as a connector plugin (`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5` (`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added after the snapshot, so a BE built from master also reads the Lance scan ranges and search requests that a 4.x FE sends during a rolling upgrade. The one exception is `TVectorSearchOptions` 6, branch-4.2's `approx_mode` (#69852, merged into branch-4.2 after the snapshot and not ported): it is reserved in a comment, and a master BE ignores it, so during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate` gets Lance's default (`normal`). **The problem, and what it cost.** | | master today | with this PR | |---|---|---| | A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan fails with "FileScannerV2 does not support file format" | read by `LanceTableReader` in FileScannerV2 | | `fetch_table_schema` for a local Lance TVF | not handled | returns the Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` | | The row location of a Lance row in TopN lazy materialization | `GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a 64-bit id within the snapshot | a version 1 layout carries the 64-bit row id; phase two reads the rows with the dataset's `take_rows` | **How this PR fixes it.** - Reader: `be/src/format_v2/lance/*` and `be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one local renamed for master's `-Wshadow`. The unit tests construct `RuntimeState(TQueryOptions(), globals)`, because master removed the globals-only constructor. - FileScannerV2: the reader dispatch and `_to_file_format` handle `FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two exhaustive switches. Since #69723 every file scan uses FileScannerV2, so nothing else routes Lance scans (on branch-4.2 a Lance query fails with NotSupported when `enable_file_scanner_v2` is off). - Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0, LANCE_DATASET_ROW_ID = 1}` and a union layout for `GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2 varint, so the wire format is compatible). The materialization operator decodes row locations by version, rejects an unknown version or a wrong size, and stamps `row_location_version`. `RowIdStorageReader` validates the counts and the version, range-checks internal row ids against `uint32`, and groups Lance rows by plan node, snapshot version and dataset URI, so each dataset is read with one `take_rows`. Master's segment batching, scan-slot dedup and fetch concurrency stay as they are. - Thrift (`PlanNodes.thrift`): the external search request body (`TVectorElementType`, `TVectorMetric`, `TSearchVector`, `TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`, `TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`, `TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`), `TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in comments and added 6 (`paimon_options`); this PR replaces those comments with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is reserved in a comment for branch-4.2's `approx_mode` (#69852). - After the snapshot (one commit each): - #68813: `prefilter` decides whether the TVF search filter runs before (true) or after (unset / false) candidate selection, for vector and full-text search. The profile groups the Lance counters under `LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan` and `LanceDataCache`, uses the 4.x counter names, and registers every scan metric before the statistics callback is installed, so the callback, which Lance invokes from its execution streams, never adds counters. #68813's FE tweak that hides `TUnit.NONE` labels in `AggCounter` belongs to the FE port. - #68707 (part): a vector split searches exactly the index segments FE planned for it, and a split without segments is searched flat whatever `use_index` says; a split with segments under `use_index = false` is rejected. 4.x FEs plan segments only when `use_index` allows them. - #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) are validated (finite FLOAT values, lower < upper at FLOAT precision, single-vector only) and installed with `lance_scanner_set_distance_range()`, which master's pinned lance-c provides. - Review fixes: - A nested array goes to the Doris SerDes without a copy only when it and every descendant start at offset 0 and reference exactly their child ranges. Lance's legacy (v1) decoder can return a struct or list that is tight at its own level over a Boolean child with a bit offset (a batch that starts mid-byte in the bitmap), and the SerDes reject that under Arrow input validation. - A zero-field `STRUCT<>`, which Lance decodes, is reported as an unsupported column at any nesting level, like the other unsupported Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot hold (its row count comes from its first child). - A Lance Decimal256 field with precision <= 38 is narrowed to Decimal128 by a safe Arrow cast before decoding. Doris types it DECIMAL32/64/128I by precision, and those SerDes read only Decimal128 arrays; the BE used to crash on the first value, also for nested leaves. Only values a reader can see are cast: before a nested leaf is narrowed, every child slot that no visible row references (under a NULL list, fixed-size-list, map or struct row, or outside a sliced parent) is marked NULL, so a hidden value does not fail the cast and a visible value outside the declared precision still does; map keys, which cannot be NULL, stay checked. - FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows that Doris keeps. - Runtime filters that mention INT64_MIN stay residual-only. The pinned Lance SQL parser reads `-9223372036854775808` as the negation of a magnitude that it parses as i64 first; the overflow turns the literal into a Float64, which the planner cannot coerce to Int64 or Decimal, so a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal column failed planning and the query failed. Every other i64 is still pushed. - The shared Arrow DATE32 decoding (`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside 0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`, skips null slots and adds the epoch in 64 bits: the DATE32 range part of branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and paimon-rust reads. - Child values hidden by a NULL list, fixed-size-list, map or struct row are not decoded: the array, map and struct SerDes give the element SerDe a validity bitmap that also clears those slots (`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden 10000-01-01 under a NULL fixed-size list used to fail the DATE32 range check above. Visible values are unchanged; this applies to ADBC, Arrow stream load and paimon-rust as well. The masks come from ExecEnv's tracked Arrow pool. A nested Lance array that is compacted before decoding is masked the same way, at every level, before its `ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision decimal, Date64 or time no longer fails the batch while a visible one still does. Since #69697 the same masks keep hidden DATETIMEV2 / TIMESTAMPTZ values out of its range checks. A mask costs at most one pass per batch: a NULL row that owns no child slots, or only NULL ones, allocates nothing, and Arrow stream load masks each retained IPC batch once (`mask_arrow_descendants_not_visible()`) before it hands row ranges to the SerDes, so its per-block reads allocate nothing. The one per-block mask left is for map keys under a NULL map row that still owns entries, since a MapArray cannot hold NULL keys; Arrow's map builder writes NULL map rows without entries. - Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The nullable serde reads an Arrow Null array as NULLs without touching buffers that the array does not have. - Configs, caches and metrics, with the 4.x names and defaults: | Config | Default | |---|---| | `lance_index_cache_size_bytes` | 10 GiB | | `lance_metadata_cache_size_bytes` | 1 GiB | | `enable_lance_data_cache` | true | | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` | | `lance_data_cache_disk_capacity_bytes` | 100 GiB | | `lance_data_cache_read_block_size_bytes` | 1 MiB | | `lance_io_buffer_size_bytes` (mutable) | 2 GiB | | `lance_batch_readahead` (mutable) | 5 | | `lance_fragment_readahead` (mutable) | 5 | The BE-wide Lance session (index cache, metadata cache and the optional Foyer disk cache) is created by the first Lance scan, so a BE that never reads Lance creates no cache directory. The first six configs take effect after a restart. A negative cache size (and, while the data cache is enabled, a negative disk capacity or read block size) fails the first Lance scan with a message that names the setting, as does any data-cache value lance-c rejects; the rest of the BE keeps serving. Ten `lance_session_*` metrics report the two in-memory caches. With the defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from the first Lance scan on (100 GiB in blocks of twice the read block size), the same as on branch-4.1 / 4.2. Size `lance_data_cache_disk_capacity_bytes` and `lance_data_cache_read_block_size_bytes` against the BE's open-file limit, or set `enable_lance_data_cache = false`. The index and metadata caches live in lance-c memory: they are not charged to a MemTracker or registered with CacheManager, so memory GC, the cache capacity adjustment under memory pressure and `/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache = false` does not turn them off. lance-c has no call to clear, evict or resize them, so size them with the two configs. The disk tier keys blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS, without the endpoint) and the object path (`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve the same bucket name and data-file path with different bytes (data files registered under chosen names or replaced in place; Lance's writers use random file names) share entries, also across a BE restart; such a deployment should set `enable_lance_data_cache = false`. Both are as on branch-4.1 / 4.2 and tracked as follow-ups. A Lance scan's native working set, that is the I/O buffered up to `lance_io_buffer_size_bytes` per scanner, the decode readahead and the operators of a search plan such as the top-k of a flat vector search, is allocated inside lance-c and is charged to neither the query nor its workload group. It counts toward the BE process memory only, so memory GC, which ranks queries by tracked memory, does not attribute it to the Lance query. lance-c accepts no memory budget, so on BEs with little memory headroom size `lance_io_buffer_size_bytes`, `lance_batch_readahead` and `lance_fragment_readahead` for the number of concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a follow-up. - Build: macOS links `-framework IOKit`. This is the first time objects of `liblance_c.a` are pulled into `doris_be`, so the binary and the link time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and `util/threadpool.h` 589, the same as master `315c2e00331` alone (limits 275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562 (master alone 559, limit 660); the reach budgets are unchanged. Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`); the FE side, including the rolling-upgrade guard that keeps Lance lazy materialization off while an old BE could decode a version 1 row location as version 0; the regression suites; the rest of #68707 (searching historical versions, tags and branches, rejecting fragment ids missing from the snapshot, naming the snapshot in errors and the profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that #67005 carried goes in a separate PR. **Results.** | | master | with this PR | |---|---|---| | A `FORMAT_LANCE` query | fails before reading anything | read by `LanceTableReader` in FileScannerV2 | | A Lance row in the second phase of TopN lazy materialization | cannot be represented | a version 1 row location, read back with `take_rows` | | A row location of an unknown version or size in phase two | decoded as version 0 | rejected with an error | | An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read fails with `InvalidArgument` | BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run, 1027 pass and 2 Parquet microbenchmarks skip by design. They cover the Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10, `LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the row-location protocol (`MaterializationSharedStateTest`, `RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`: count mismatch, unknown version, a Lance version on an internal mapping, an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`, `ParquetScanTest` and the other `Parquet*` suites), whose row ids now go through `file_local`; and the Arrow / protobuf / DATE serde suites with the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader suites that decode Arrow dates. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest` 10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310 pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The ninth review round adds `LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and `LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`: both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto `339e087b11f`, and restoring `parse_int64()` in either check makes exactly these two fail. On the head rebased onto master `315c2e00331`, with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest` (15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`, `AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest` and the Lance suites. Allocating at the first NULL row of each call again fails the new `ArrowParentNullsTest` case and the four new `LoadArrowPipeTest` block cases, dropping the per-batch mask fails the LIST and STRUCT block cases, and returning the child unmasked fails the new timestamp case, four `ArrowParentNullsTest` cases and `LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`. ### Release note Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They take effect once the FE can plan Lance scans. Reading an Arrow DATE32 value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or paimon-rust now fails instead of returning 0000-00-00. ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [x] Unit Test - `run-be-ut.sh --run --filter=...` with the suites listed under Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the rebases, and the Lance and Arrow SerDe suites pass again with the nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the scanner selection and in `fetch_table_schema`, which now hands every file format to `format::fetch_file_schema()`, so the Lance branch runs ahead of that call; #69646 added a TopN row-group test that reads the version 0 row location fields, which now go through `file_local` (the earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the scanner selection, the `FileFormat` enum, its test and the reach budget table). The 95 TUs that this PR changes or that master changed since `5caaeb3e3d4` were syntax-checked against the rebased tree with the UT build flags, and CI runs the suites again. Rebased again onto master `8d88f1cc6f7`: the only conflict was the budget table, where #65858 moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that this PR changes or that master changed since `43f9ceb48d7` pass the same syntax check. Rebased again onto master `339e087b11f`: the only conflict was the budget table again, where #67292 moved the `util/rle_encoding.h` baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent masking in the map and struct readers and merged cleanly. Of the 114 sources that this PR changes or that master changed since `8d88f1cc6f7`, 111 pass the same syntax check. The other three are master's: two new #67292 files under `storage/segment` fail the same way on master alone, since they compile only inside their unity batch, and this PR adds no source to that target; the #69818 benchmark is not in the UT build and includes only `util/simd` headers that this PR does not touch. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. Rebased again onto master `378404eeb12`, which removed the legacy file scanner (#69723): the conflicts were in the scanner-selection code and tests that #69723 deleted, so this PR's Lance lines there went with them (including its `LanceQueriesAlwaysUseScannerV2` test), in the profile keys of `materialization_opertor.cpp`, and in the row fetch of `rowid_fetcher.cpp`, whose non-Lance path is now only `FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests took this PR's `file_local` rename with them, and the `util/rle_encoding.h` budget commit is dropped because master is no longer near that limit. Of the 100 sources that this PR changes or that master changed since `339e087b11f`, 99 pass the same syntax check; the other, `format_v2/orc/orc_reader.cpp`, fails the same way on master alone. On the rebased head the suites listed under Results run 1029 cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that `DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master. Rebased again onto master `315c2e00331`: the only conflict was `TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had reserved fields 2-5 in comments; the two commits that add the fields replace those comments, and field 6 stays #70075's `paimon_options`. #69717 cut transitive includes from shared headers: of the 160 sources that this PR changes or that master changed since `378404eeb12`, 156 pass the same syntax check; the two `jemalloc_hook` tests have no compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc aliases that macOS cannot build (#69819), and `format_v2/table_reader.cpp` alone fails the same way on master alone while its unity batch compiles. The full `doris_be_test` builds and links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases) pass before the mask fix, and the 328 cases above with it. - `build-support/check-build-hygiene.sh` (on this branch rebased onto master `315c2e00331`), clang-format 16 and `build-support/run-clang-tidy.sh` on the changed files: all pass; on the changed lines clang-tidy (llvm 22) reports only `readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of the masking functions, the macro's expansion that master's existing `DORIS_CHECK`s report too. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [ ] No. - [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN lazy materialization rejects row locations of an unknown version or size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day outside the Doris DATE range fails the read instead of becoming `0000-00-00`. - Does this need documentation? - [x] No. Lance is documented with the FE connector. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> --------- Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: zy-kkk <zhongyk10@gmail.com>
What problem does this PR solve?
Issue Number: close #68625
Related PR: #68453 (time travel, tags, branches and managed versioning for Lance table scans)
Problem Summary:
vector_search()andfull_text_search()always searched the latest version ofmain, and theirtableargument rejectedFOR VERSION AS OF,@tagand@branch. A search could not be reproduced against a released tag, compared across versions, or run on an experimental branch.Both TVFs now take four optional properties, with the semantics of the table syntax from #68453:
versionFOR VERSION AS OF Nbranchtimestampyyyy-MM-dd HH:mm:ss[.SSS], session time zoneFOR TIME AS OFbranchtag@tag(name)branchmainis the main chain@branch(name)versionortimestampHow it works:
LanceRefSelectorand loads its metadata through the sameLanceCatalogClient.readTableSnapshotas table scans, so storage-versioned and namespace-managed tables follow the [feature](lance) Support time travel, tags, branches and REST Namespace managed versioning #68453 rules unchanged. Schema, search column and field id, vector type and dimension, fragments and index segments all come from the selected snapshot, and planning never reopens the dataset. The splits carry the branch directory URI and the version, so no thrift change is needed.EXECUTE, and TVF statements skip the SQL cache, so the latest version, a future time, a branch head and a moved tag are resolved per execution and stay fixed within its plan.CREATE TABLE AS SELECTplans its query twice (schema, then rows), so there a moving selector can resolve twice, as it already can for a table scan with an explicit selector; pinningversiongives an exact copy. This is documented.lanceCatalogType,lanceManagedVersioningandlanceBranchnext tolanceVersion, as table scans do. Planner errors that name "dataset version N" also name the branch, since branch versions overlap main's.Decisions worth calling out:
versiontakes only a number.FOR VERSION AS OF 'x'names a tag because SQL has one clause for both; the TVF has atagproperty, and Lance allows tag names made of digits, whichFOR VERSION AS OFcannot reach.main.?would leave PREPARE without a snapshot to take the result schema from.full_text_searchnow rejects?in any property: until now a placeholder reached it as the literal text "?", so'query'=?searched for "?".vector_search(t) JOIN tcan read two versions if a commit lands in between; pinning both withversion/FOR VERSION AS OFavoids that. This is documented.<table>/tree/<branch>), which Lance does not validate on that path, so Doris now rejects names that could leave the table'stree/directory or change that URI: empty segments,..,\, and ASCII characters other than letters, digits,.,-,_and/, so../otherand a percent-encoded dot segment are rejected before the branch is opened. Outside ASCII only whitespace and control characters are rejected, so no name Lance accepts is rejected, whichever Unicode version the JDK and Lance's Rust use. Other tables check a branch out through Lance, which validates the name itself.BE changes:
LanceDatasetVersionandLanceDatasetUri, the snapshot an execution actually read. EXPLAIN cannot show this for moving selectors because it resolves again.use_index=false). The FE plans such splits as flat search, and EXPLAIN and the profile count them as flat, but Lance used to search them with the first index on the column in manifest order, which need not be the index the FE selected (for example while an index is rebuilt under a new name). Indexes the FE cannot plan with (no fragment coverage metadata, no recorded metric, or a legacy index whose details Lance cannot infer from its files, which the FE skips;lanceVectorIndexStatus=UNKNOWN_COVERAGE/METRIC_MISMATCH/NO_MATCH) are therefore no longer used implicitly, and such searches become exact flat searches. Results can change only where the unplanned index was approximate; latency can grow on such datasets. Legacy indexes that lack details but whose details Lance infers, such as those written by Lance 0.10 to 0.24, are planned and searched as before (test_lance_vector_search_legacy_index).use_index=falseis rejected. The FE plans segments only whenuse_indexallows them, so it never sends such a split; the BE now takesuse_indexfrom the segments and would otherwise have to override either the plan or the request.Known limitations (documented):
SHOW INDEXandlance_index_entries()keep showing the latest version ofmain.Release note
vector_search()andfull_text_search()acceptversion,timestamp,tagandbranchto search a historical version, the version at a time, a tag or a branch of a Lance table, including REST Namespace managed tables. Vector search no longer uses an index the planner did not select, andfull_text_search()rejects prepared-statement placeholders.Check List (For Author)
external_table_p0/lance/test_lance_search_snapshot(filesystem catalog) andtest_lance_rest_search_snapshot(REST, storage-versioned and managed) on a new fixturesearch_snapshot.lance. It has seven versions (appends, a vector index, an FTS index, an append no index covers, a delete, a vector index rebuild), a tag, and a branchdevwhose versions and fragment ids overlap main's with different rows, plus a tag at a non-latest branch version.search_snapshot_pruned.lancelost a version to cleanup and the index files of a tagged version.search_snapshot_evolved.lanceadds a column and renames the vector column after its index is built, then appends data the index does not cover. The suites check rows, distances, two-phase reads (on a branch, a tag, the columns of a version before and after a rename, and a managed table through REST), EXPLAIN, full-text search across a delete,use_index=falsewith a selector, every selector error, and a managed branch named with a Unicode 16 letter (unicode_branch.lance, built by pylance 12.0.0) through a table scan, a search and a two-phase read.test_lance_rest_catalog.outandtest_lance_rest_time_travel.outlist the new REST tables.test_lance_vector_search_legacy_indexsearches a vector index written by pylance 0.18.2 without index details and checks that it is planned (USED, one segment) and returns the flat search's rows. All 30 suites inexternal_table_p0/lancepass locally.LanceSearchSelectorTest(property rules, time zones, selector forwarding),LancePreparedSearchTest(both TVFs pass their selector to the table; each analysis resolves it exactly once, and a moved tag is resolved again on the nextEXECUTE; search statements bypass the SQL cache; placeholders rejected),LanceScanNodeTest(EXPLAIN, errors naming the branch),LanceCatalogClientTest(branch names, including letters newer than JDK 17's Unicode tables), and, with real datasets and locally only like the other JNI tests,LanceSearchSnapshotTest(historical index coverage, branches, tags, schema evolution, cleanup, a snapshot planned before later commits) andLanceManagedVersioningTest(managed versions, tags, branches, unfinalized heads through the search entry points).lance_reader_test.cpp: missing fragment ids on normal, vector and FTS splits, unpinned versions, error and profile contents, a split without index segments searching flat although an index covers it, and a split with index segments underuse_index=falsebeing rejected. The indexed variants of the existing multi-vector tests now pass the index segments the FE would plan.full_text_searchrejects?; managed branch names are validated; the error for a time in removed history now reads "Lance cannot select the version at or before '...'" instead of "Lance cannot resolve FOR TIME AS OF '...'", because the TVF reaches it too; BE errors name the dataset version and URI.