Skip to content

[feature](lance) Search historical versions, tags and branches with vector_search and full_text_search - #68707

Merged
Gabriel39 merged 7 commits into
apache:branch-4.1from
zy-kkk:lance-search-snapshot
Oct 9, 2026
Merged

Gabriel39 merged 7 commits into
apache:branch-4.1from
zy-kkk:lance-search-snapshot

Conversation

@zy-kkk

@zy-kkk zy-kkk commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close #68625

Related PR: #68453 (time travel, tags, branches and managed versioning for Lance table scans)

Problem Summary:

vector_search() and full_text_search() always searched the latest version of main, and their table argument rejected FOR VERSION AS OF, @tag and @branch. A search could not be reproduced against a released tag, compared across versions, or run on an experimental branch.

Both TVFs now take four optional properties, with the semantics of the table syntax from #68453:

Property Value Table syntax Combines with
version positive integer FOR VERSION AS OF N branch
timestamp yyyy-MM-dd HH:mm:ss[.SSS], session time zone FOR TIME AS OF branch
tag tag name @tag(name) none
branch branch name, main is the main chain @branch(name) version or timestamp
SELECT id, _distance FROM vector_search(
    "table" = "lance_catalog.db.documents", "column" = "embedding",
    "query_vector" = "[0.1, 0.2, 0.3]", "version" = "3")
ORDER BY _distance;

SELECT id, _score FROM full_text_search(
    "table" = "lance_catalog.db.documents", "column" = "content", "query" = "database",
    "branch" = "dev", "timestamp" = "2026-09-20 12:00:00")
ORDER BY _score DESC;

How it works:

  • FE. The TVF parses the properties into a LanceRefSelector and loads its metadata through the same LanceCatalogClient.readTableSnapshot as table scans, so storage-versioned and namespace-managed tables follow the [feature](lance) Support time travel, tags, branches and REST Namespace managed versioning #68453 rules unchanged. Schema, search column and field id, vector type and dimension, fragments and index segments all come from the selected snapshot, and planning never reopens the dataset. The splits carry the branch directory URI and the version, so no thrift change is needed.
  • Each execution resolves again. A bound TVF is never cached, prepared statements re-analyze on every EXECUTE, and TVF statements skip the SQL cache, so the latest version, a future time, a branch head and a moved tag are resolved per execution and stay fixed within its plan. CREATE TABLE AS SELECT plans its query twice (schema, then rows), so there a moving selector can resolve twice, as it already can for a table scan with an explicit selector; pinning version gives an exact copy. This is documented.
  • No fallback. A snapshot that cannot be selected, or whose index files are missing, is an error; nothing falls back to the latest version.
  • EXPLAIN of a search node now shows lanceCatalogType, lanceManagedVersioning and lanceBranch next to lanceVersion, as table scans do. Planner errors that name "dataset version N" also name the branch, since branch versions overlap main's.

Decisions worth calling out:

  • version takes only a number. FOR VERSION AS OF 'x' names a tag because SQL has one clause for both; the TVF has a tag property, and Lance allows tag names made of digits, which FOR VERSION AS OF cannot reach.
  • An empty selector value is an error rather than "not set", so an empty client-side variable cannot silently search the latest version of main.
  • The selector properties must be constants. Accepting ? would leave PREPARE without a snapshot to take the result schema from. full_text_search now rejects ? in any property: until now a placeholder reached it as the literal text "?", so 'query'=? searched for "?".
  • Each search relation resolves its own snapshot, as each explicit selector already does for table scans. vector_search(t) JOIN t can read two versions if a commit lands in between; pinning both with version / FOR VERSION AS OF avoids that. This is documented.
  • A managed table's branch is opened by a URI Doris joins (<table>/tree/<branch>), which Lance does not validate on that path, so Doris now rejects names that could leave the table's tree/ directory or change that URI: empty segments, .., \, and ASCII characters other than letters, digits, ., -, _ and /, so ../other and a percent-encoded dot segment are rejected before the branch is opened. Outside ASCII only whitespace and control characters are rejected, so no name Lance accepts is rejected, whichever Unicode version the JDK and Lance's Rust use. Other tables check a branch out through Lance, which validates the name itself.

BE changes:

  • A search split or a two-phase row fetch without a positive version is rejected instead of opening the latest version.
  • Errors from opening the dataset, reading batches, preparing the FTS query context and the two-phase row fetch name the snapshot: "... at Lance dataset version N at ", with the query string and any password removed from the URI. Missing index files surface at execution because Lance loads indexes lazily.
  • Fragment ids from the FE must exist in the opened snapshot. lance-c silently skips unknown fragment ids, which would drop rows without an error.
  • The profile records LanceDatasetVersion and LanceDatasetUri, the snapshot an execution actually read. EXPLAIN cannot show this for moving selectors because it resolves again.
  • Behavior change for every vector search: a split without index segments now always runs flat (use_index=false). The FE plans such splits as flat search, and EXPLAIN and the profile count them as flat, but Lance used to search them with the first index on the column in manifest order, which need not be the index the FE selected (for example while an index is rebuilt under a new name). Indexes the FE cannot plan with (no fragment coverage metadata, no recorded metric, or a legacy index whose details Lance cannot infer from its files, which the FE skips; lanceVectorIndexStatus=UNKNOWN_COVERAGE / METRIC_MISMATCH / NO_MATCH) are therefore no longer used implicitly, and such searches become exact flat searches. Results can change only where the unplanned index was approximate; latency can grow on such datasets. Legacy indexes that lack details but whose details Lance infers, such as those written by Lance 0.10 to 0.24, are planned and searched as before (test_lance_vector_search_legacy_index).
  • A vector split that carries index segments while the request says use_index=false is rejected. The FE plans segments only when use_index allows them, so it never sends such a split; the BE now takes use_index from the segments and would otherwise have to override either the plan or the request.

Known limitations (documented):

Release note

vector_search() and full_text_search() accept version, timestamp, tag and branch to search a historical version, the version at a time, a tag or a branch of a Lance table, including REST Namespace managed tables. Vector search no longer uses an index the planner did not select, and full_text_search() rejects prepared-statement placeholders.

Check List (For Author)

  • Test: Regression test / Unit Test
    • Regression: external_table_p0/lance/test_lance_search_snapshot (filesystem catalog) and test_lance_rest_search_snapshot (REST, storage-versioned and managed) on a new fixture search_snapshot.lance. It has seven versions (appends, a vector index, an FTS index, an append no index covers, a delete, a vector index rebuild), a tag, and a branch dev whose versions and fragment ids overlap main's with different rows, plus a tag at a non-latest branch version. search_snapshot_pruned.lance lost a version to cleanup and the index files of a tagged version. search_snapshot_evolved.lance adds a column and renames the vector column after its index is built, then appends data the index does not cover. The suites check rows, distances, two-phase reads (on a branch, a tag, the columns of a version before and after a rename, and a managed table through REST), EXPLAIN, full-text search across a delete, use_index=false with a selector, every selector error, and a managed branch named with a Unicode 16 letter (unicode_branch.lance, built by pylance 12.0.0) through a table scan, a search and a two-phase read. test_lance_rest_catalog.out and test_lance_rest_time_travel.out list the new REST tables. test_lance_vector_search_legacy_index searches a vector index written by pylance 0.18.2 without index details and checks that it is planned (USED, one segment) and returns the flat search's rows. All 30 suites in external_table_p0/lance pass locally.
    • FE unit tests: LanceSearchSelectorTest (property rules, time zones, selector forwarding), LancePreparedSearchTest (both TVFs pass their selector to the table; each analysis resolves it exactly once, and a moved tag is resolved again on the next EXECUTE; search statements bypass the SQL cache; placeholders rejected), LanceScanNodeTest (EXPLAIN, errors naming the branch), LanceCatalogClientTest (branch names, including letters newer than JDK 17's Unicode tables), and, with real datasets and locally only like the other JNI tests, LanceSearchSnapshotTest (historical index coverage, branches, tags, schema evolution, cleanup, a snapshot planned before later commits) and LanceManagedVersioningTest (managed versions, tags, branches, unfinalized heads through the search entry points).
    • BE unit tests in lance_reader_test.cpp: missing fragment ids on normal, vector and FTS splits, unpinned versions, error and profile contents, a split without index segments searching flat although an index covers it, and a split with index segments under use_index=false being rejected. The indexed variants of the existing multi-vector tests now pass the index segments the FE would plan.
  • Behavior changed: Yes. New TVF properties; vector splits without index segments always search flat; full_text_search rejects ?; managed branch names are validated; the error for a time in removed history now reads "Lance cannot select the version at or before '...'" instead of "Lance cannot resolve FOR TIME AS OF '...'", because the TVF reaches it too; BE errors name the dataset version and URI.
  • Does this need documentation: Yes ([docs](lance) Search historical versions, tags and branches with the search functions doris-website#4185)

@zy-kkk
zy-kkk requested a review from yiguolei as a code owner October 3, 2026 09:18
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk

zy-kkk commented Oct 3, 2026

Copy link
Copy Markdown
Member Author

run buildall

@zy-kkk

zy-kkk commented Oct 3, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review status: COMPLETE for head ebe7d6f. Static review covered all 99 changed paths, the prepared diff, regression expectations, and the relevant FE/BE call chains. One P2 test-coverage finding is inline. The prepared threads and live PR comments contain no existing P0/P1 finding to carry. No additional user focus was supplied.

Critical checkpoints:

  • Goal and scope: Snapshot-aware Lance vector and FTS search is wired from TVF selector parsing through catalog metadata, FE split planning, BE scan, and row-id fetch. The change is focused on this feature and its fixtures/tests; the active regression proof for historical branch selectors is incomplete as noted inline.
  • Correctness and conditionals: Reviewed conflicting/invalid selectors, missing versions, strict versus index-only FTS coverage, mixed indexed/flat vector splits, historical deletes, schema evolution, and fixed positive versions. The same selected URI/version flows to search and two-phase fetch; no additional concrete wrong-row path survived review.
  • Concurrency and lifecycle: Per-query metadata pins URI/version across planning and BE reads, including when a tag or branch head later moves. Reader close/reopen resets snapshot-related state; no new lock ordering or shared mutable state was found. Resource and allocator lifetimes were traced across metadata reads and scans.
  • Configuration and compatibility: No new dynamic configuration or Doris transaction/EditLog path is introduced. FE/BE reuse the existing scan transport for URI, version, fragments, and index segments. Filesystem and managed REST catalogs, main/branch/tag selection, and vector/FTS paths were checked separately; rolling-upgrade behavior was assessed statically only.
  • Parallel paths and observability: Reviewed normal scans, search scans, prepared TVFs, and lazy row fetch, plus branch/tag and managed table resolution. Existing errors, EXPLAIN fields, and metadata/scan profiling expose the selected version and coverage; no separate observability defect was substantiated.
  • Tests and results: Inspected FE/BE unit tests, SQL regressions and expected output, binary fixture inventory, and compose/REST setup. The new JNI snapshot test class is disabled; active tests leave the historical branch selector gap described inline. Builds and tests were prohibited and were not run, so no runtime result is claimed.
  • Performance and writes: The fixture builder stages and promotes test data; this PR does not add a Doris data-write or persistence path. Vector indexes without usable segment details now take flat search, which can be costly for older large datasets; available metadata did not establish a safe indexed fallback, so this remains a review limitation rather than a separate inline finding.

contains "lanceVectorIndexStatus=DISABLED"
}
qt_branch_fork """
SELECT row_id, _distance FROM ${vectorSearch(', "branch"="dev", "version"="4"', "[100,101,102,103]")}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Exercise a historical snapshot on the dev branch. This branch=dev,version=4 query selects the fork point, whose rows and indexes match main version 4, so a second checkout on main in readFromLatest would still pass. The branch timestamp assertion asks for year 2999, so ignoring that timestamp also selects dev head. Add discriminating assertions: explicit dev version 6 with STRICT FTS (dev rebuilt coverage over fragment 2, main v6 did not), and an as-of time between dev commits 5 and 6 with lanceVersion=5. The JNI snapshot test class is disabled, leaving these search paths without an active historical check.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: the fork-point version and the future timestamp could not tell the branch from main, or the time from the head. Added in e55bd49:

  • branch=dev, version=5 returns the branch's rows 101/102/103 (main version 5 would return 24/23/22), with EXPLAIN lanceVersion=5 and lanceBranch=dev.
  • STRICT full_text_search succeeds on branch=dev, version=6, whose body_idx was rebuilt over fragment 2, and fails on main version 6.
  • branch=dev, timestamp=2026-09-29 14:52:18, between dev versions 5 and 6, gives EXPLAIN lanceVersion=5, and STRICT full-text search fails naming dataset version 5 of branch 'dev'. Ignoring the time would select dev version 6, and resolving it on main would select main version 7.

All 29 suites in external_table_p0/lance pass locally.

@zy-kkk

zy-kkk commented Oct 3, 2026

Copy link
Copy Markdown
Member Author

run buildall

@zy-kkk

zy-kkk commented Oct 3, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review complete at head e55bd497a453399971f3ae57d5b993261eedd006 after two convergence rounds and a final sweep of all 99 changed paths. Two distinct inline findings follow: a P2 legacy vector-index performance regression and a P3 historical two-phase test gap. No P0/P1 finding was substantiated.

Critical checkpoints

  • Goal and scope: The search TVFs now select version, timestamp, tag, and branch snapshots; FE binds schema/index metadata from the selected snapshot and sends its URI and positive version through existing split fields. The production changes are focused on that path and its BE checks.
  • Concurrency and lifecycle: No new threads or locks were added. The BE fragment list is reader-local, bound to the opened dataset key, and reset on close; a changed URI/version/options key fails. Search and row-id fetch reject an unpinned version, and missing fragment IDs fail rather than silently dropping rows. No new cross-translation-unit initializer or ownership cycle was found.
  • Compatibility and parallel paths: Reviewed filesystem and managed REST resolution, main and dev histories, vector indexed/flat/disabled paths, FTS coverage, normal scans, prepared re-execution, SQL cache exclusion, and two-phase row fetch. Existing Thrift URI/version/options fields carry the selection. The compatible legacy-index path has the P2 regression reported inline; no other mixed-path defect was substantiated.
  • Conditions, persistence, and writes: Selector combinations, timestamp parsing in the session time zone, branch-name validation, and deleted/pruned histories have explicit errors or conservative fallback. Production changes do not add configuration, transaction writes, or persistent Doris metadata. The fixture generator writes test datasets only.
  • Tests and observability: Inspected FE/BE tests, generated regression outputs, fixture histories, and active filesystem/REST suites, including negative and schema-evolution cases. The JNI snapshot test class is disabled; active assertions cover its main historical paths, with the P3 two-phase gap noted inline. EXPLAIN shows the selected version/branch, and BE profile/errors identify the opened snapshot. No builds or tests were run in this review, as instructed; the author's local test claim was not treated as independent validation.
  • Performance and remaining questions: The P2 flat-scan regression is the remaining performance concern. All other main risk items and candidate points have a recorded conclusion; no unresolved suspicious point remains.

Existing inline discussion 4172612906 is a P2 test concern already addressed on this head by discriminating dev version-5 rows, dev version-6 STRICT FTS, and a between-commit timestamp. It is not reposted or carried as a P0/P1 blocker. No additional user review focus was supplied.

// it as one. Lance would otherwise search it with the first index on the column in manifest
// order, which need not be the index FE selected. A split with segments searches exactly
// those; one that also carries use_index=false was rejected above.
if (lance_scanner_set_use_index(scanner, segment_count > 0) != 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve ANN use for compatible legacy indexes. A dataset with one legacy L2 vector index whose physical entries have no index details is skipped by describeUserIndexes; FE then sends fragment splits without a segment UUID, so this line overrides even use_index=true and scans every vector flat. Previously Lance could select that index for the same query, so large existing collections lose indexed-search performance. Recover enough metadata to plan a compatible legacy index or provide another safe indexed path, and add a legacy-index execution regression.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked this against real legacy indexes before changing anything. Indexes without index details written by pylance 0.10.18 and 0.18.2 (IVF_PQ, L2) and 0.24.1 (IVF_PQ, cosine, multi-vector) are not skipped: Lance infers the missing details from the index files when it loads the index (infer_missing_vector_details), so describeUserIndexes describes them, the FE plans one segment (lanceVectorIndexStatus=USED), and the BE profile shows LanceIVFPartitionsSearched: 1 with no flat-search fragments.

The skip in describeUserIndexes is now reached only when that inference fails, for example when an index file cannot be read at planning time. Doris then cannot verify the index's coverage or metric, so a flat search is the conservative result, and a persistent failure would break the BE's use of the index too.

Added in a7b9979: legacy_vector_index.lance, built by pylance 0.18.2 without index details, and test_lance_vector_search_legacy_index, which checks that the index is planned with one segment and that the indexed search returns the flat search's rows. The docs now say an older index is skipped only when Lance cannot infer its details.

contains "VMaterializeNode"
}
qt_branch_two_phase "${twoPhase}"
qt_tag_two_phase """

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] Make this historical tag test exercise and distinguish phase-two fetch. rel selects version 5, but rows 24/23/22 and their body values are identical at latest version 7; this result would still pass if row-id fetch reopened v7. This query also lacks the adjacent VMaterializeNode EXPLAIN assertion. Assert that plan shape and select historical payload that changes or disappears in a later version (for example, row 3, deleted in v6) so the test proves the fetched rows came from the selected snapshot.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a7b9979: the tag case now searches next to row 3, which rel (version 5) still has and version 6 deleted. A fetch from a later version would return fewer rows and fail the row-count check. It also asserts VMaterializeNode.

@zy-kkk

zy-kkk commented Oct 3, 2026

Copy link
Copy Markdown
Member Author

run buildall

@zy-kkk

zy-kkk commented Oct 3, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Complete static review of PR #68707 at a7b9979. No new actionable inline findings. The user supplied no additional focus points.

The change enables vector_search() and full_text_search() to select a Lance version, timestamp, tag, or branch. FE derives schema, field IDs, fragments, and indexes from one selected Dataset, then sends its URI and positive version in each split. BE uses that fixed snapshot for search and row-id fetch, validates FE fragment IDs against the opened dataset, and follows FE's segment plan for indexed versus flat vector splits. I inspected the active filesystem and REST regression cases, BE and FE unit tests, fixture generators and committed history, and expected outputs. The JNI LanceSearchSnapshotTest class is disabled, so the active regression suites are the relevant end-to-end coverage. Builds and tests were prohibited and were not run; author-reported passing suites are not independent validation here.

Previously reported inline concerns: #4172612906 [P2] is addressed by active dev-v5 rows, dev-v6 STRICT FTS, and an intervening dev timestamp assertion; #4172821003 [P3] is addressed by the VMaterializeNode assertion and a tagged row deleted in a later version; #4172821000 [P2] remains the existing legacy-index discussion, with an inferable legacy fixture and index-planning regression added on this head. I did not repost these threads. No existing P0/P1 inline finding was independently confirmed as still applicable.

Critical checkpoints:

  • Goal and tests: historical selectors are propagated through metadata, split planning, BE scanning, and phase-two fetch. Active SQL cases cover main and dev with equal numeric versions, tags, timestamps, index coverage, deletes, missing versions, schema evolution, and row fetch. This is inspected coverage, not an executed result.
  • Scope and clarity: production edits are confined to Lance search/snapshot handling and the related prepared-statement parser check. Added binary fixtures support version, branch, pruning, evolution, and legacy-index scenarios.
  • Concurrency and locks: metadata is resolved per read and the BE fragment-ID cache is reader-owned; split reuse is sequential for that reader. No new thread entry, shared lock, nested-lock order, or heavy operation under a new lock was introduced.
  • Lifecycle and initialization: opened URI/version/options remain fixed for reader reuse; scanner and FTS context cleanup and fragment-cache reset were traced. No new cross-translation-unit static initializer dependency was found.
  • Configuration: no product configuration item was added, so dynamic-change handling is inapplicable. Regression suites restore their session settings.
  • Compatibility: selector omission retains latest-main behavior. Existing Thrift version and index-segment fields carry the selected snapshot; no new storage format or transmitted variable was introduced. Legacy-index and older-BE behavior was checked against the existing inline discussion; no distinct regression on this head was substantiated.
  • Parallel paths and conditions: normal scan, vector search, FTS, and take_rows were traced. Positive search/fetch versions, fragment membership, and contradictory use_index/index-segment checks fail explicitly; indexed and uncovered vector splits take their intended paths.
  • Test results and negative cases: active suites and expected .out files include missing selector targets, invalid combinations, pruned history, managed namespace gaps, branch differences, and lazy materialization. No test result was generated or verified by running it in this review.
  • Observability: BE profiles now name the opened version and sanitized URI, and errors name the snapshot; existing index and flat-search counters remain available.
  • Transactions, persistence, and writes: production changes are external read paths, with no Doris transaction, EditLog, or product data-write change. Fixture construction stages and checks histories before catalog promotion; that construction was inspected only.
  • Performance and other issues: BE caches the fragment list for an opened reader and uses FE segment selection to avoid unintended ANN work on flat splits. No further substantiated performance or correctness issue survived the final sweep.

All 109 changed paths in the authoritative diff were covered. Three normal and two risk-focused subagent passes returned NO_NEW_VALUABLE_FINDINGS. One final-sweep mixed-version hypothesis was checked and dismissed as a pre-existing legacy-index behavior at an already-commented location without a distinct verified failure on this head. No candidate remains unresolved. Review complete.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 77.32% (75/97) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 61.18% (26788/43785)
Line Coverage 45.82% (278347/607511)
Region Coverage 41.53% (220629/531268)
Branch Coverage 43.00% (102250/237788)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 83.02% (88/106) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 77.32% (75/97) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.92% (31900/42580)
Line Coverage 59.25% (357024/602615)
Region Coverage 55.92% (297729/532402)
Branch Coverage 56.79% (134886/237505)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 71.70% (76/106) 🎉
Increment coverage report
Complete coverage report

@Gabriel39

Copy link
Copy Markdown
Contributor

Reviewed commit a7b99799c5235919fece14d978bae065b5a945bd.

Review conclusion: request changes — one managed-branch compatibility regression.

The new isLanceAlphanumeric check uses JDK 17 character classifications, which use Unicode 13. The pinned Lance 12.0.0 validates names with Rust is_alphanumeric; its minimum Rust version is 1.91, whose tables use Unicode 16. These predicates therefore accept different sets of characters.

For example, devᲉ contains U+1C89 (CYRILLIC CAPITAL LETTER TJE), introduced in Unicode 16. A Lance-created managed branch with this name and a valid namespace-recorded version is accepted by Lance but rejected by the new Java guard with Invalid Lance branch name.

This also breaks the existing ordinary managed-table query path, not just the new search TVFs:

SELECT * FROM catalog.db.items@branch('name'='devᲉ');

At the merge base, readManagedBranch opened the recorded version without this Java character filter. At this head, validation rejects the name before namespace lookup. This is a narrow functional compatibility regression (Major under the review rubric); no data corruption or resource-exhaustion claim is intended.

Suggested fix: retain the traversal and unsafe-URI checks, but align character acceptance with the pinned Lance contract, through native validation or matching versioned Unicode data. Add coverage for an existing managed branch containing a post-Unicode-13 letter, for both ordinary scans and search TVFs.

References: JDK 17 Character, Lance v12 branch validation, Lance v12 Rust requirement, Rust 1.91 Unicode tables.

Validation scope: two static review rounds covering all 109 changed files and relevant dependency contracts. No builds, tests, or JNI reproduction were run. No additional substantiated blocking findings were identified.

@zy-kkk

zy-kkk commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

Thanks, confirmed: a managed branch named devᲉ was rejected by the guard, including @branch(...) table scans. Fixed in 0eac821.

I did not align the guard with Lance's character set exactly, because neither route stays aligned:

  • Native validation: the Lance 12 Java SDK exposes no branch-name validator. check_valid_branch only runs inside Rust (checkout_by_ref and the dataset builder), and the managed path opens the joined URI without it.
  • Versioned Unicode data: the tables Lance uses come from the rustc that built the shipped lance-jni and lance-c, not from the minimum Rust version, so FE and BE can differ, and a table copied into Java would drift again on the next toolchain bump.

So the guard no longer decides whether a name is valid. It only keeps the joined URI inside the table's tree/ directory:

  • Rejected: empty, a leading or trailing /, empty segments, .., a trailing .lock, \, and any ASCII character other than letters, digits, ., -, _ and /. In ASCII this is exactly Lance's set, and no segment can climb out of tree/, percent-encoded or not.
  • Outside ASCII only whitespace and control characters are rejected. ASCII classification does not depend on the Unicode version, and no whitespace or control character has ever been alphabetic or numeric, so every name check_valid_branch accepts passes, whichever Unicode version the JDK or rustc uses.
  • A name that passes but that Lance would reject cannot have been created by Lance, so the namespace reports it as a branch it does not record.

Coverage, as you suggested, on an existing managed branch with a post-Unicode-13 letter:

  • unicode_branch.lance has a branch devᲉ (U+1C89, Unicode 16), served by the REST fixture as the managed table unicode_branch_managed. It is built with pylance 12.0.0, since Lance 7 stores a non-ASCII branch directory percent-encoded, which Lance 12 does not find.
  • test_lance_rest_search_snapshot reads that branch through an @branch table scan, vector_search(... "branch"="devᲉ") and a two-phase read.
  • LanceCatalogClientTest checks that devᲉ and other non-ASCII letters and digits are accepted, and that the unsafe names are still rejected.

@zy-kkk

zy-kkk commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 84.54% (82/97) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 61.18% (26787/43785)
Line Coverage 45.82% (278345/607511)
Region Coverage 41.52% (220567/531268)
Branch Coverage 43.00% (102250/237788)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 80.95% (85/105) 🎉
Increment coverage report
Complete coverage report

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 84.54% (82/97) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.91% (31895/42580)
Line Coverage 59.22% (356876/602615)
Region Coverage 55.86% (297422/532402)
Branch Coverage 56.76% (134813/237505)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 72.38% (76/105) 🎉
Increment coverage report
Complete coverage report

@zy-kkk

zy-kkk commented Oct 5, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Complete static review of PR #68707 at head 0eac821. No new actionable findings. All three Round 1 review passes returned NO_NEW_VALUABLE_FINDINGS; the final changed-file and unresolved-point sweep found none. There was no additional user-provided focus.

Critical checkpoints:

  • Goal and evidence: Both Lance search TVFs accept version, timestamp, tag, and branch selectors. The selected URI and positive version flow from catalog metadata into FE splits, BE search, and phase-two row-id fetch. Active external regression suites inspect distinct main/branch rows, historical deletion, index coverage, schema evolution, managed namespace gaps, and lazy materialization. The new LanceSearchSnapshotTest class is disabled by its stated JNI ABI limitation, so its assertions are not executed coverage.
  • Scope and parallel paths: Changes stay within selector plumbing, snapshot diagnostics, BE plan enforcement, and fixtures. Vector and full-text searches, indexed and basic metadata loading, managed and filesystem catalogs, and table scans share the relevant resolver behavior. Invalid combinations and unavailable snapshots fail explicitly.
  • Concurrency and lifecycle: No new threads, locks, or shared mutable state were added. One catalog client lease reads a selected Dataset snapshot; the metadata retains its URI/version after the handle closes. A BE reader retains one dataset key for its splits; its fragment-ID cache resets on close. No new static initialization dependency was found.
  • Configuration and compatibility: No configuration item or Thrift field was added. Existing URI/version fields carry selectors across FE and BE, and the smooth-upgrade check for additional reader types remains in place. The older BE's optional automatic index choice for segmentless splits predates this PR; no distinct new selector-specific wrong-result path was substantiated.
  • Conditional checks, data correctness, and performance: BE rejects unpinned search/fetch versions and fragments absent from the opened snapshot. Indexed vector splits use FE-selected segment UUIDs; uncovered splits search flat. FTS retains strict/index-only coverage behavior. The extra fragment listing is once per opened BE dataset, not per batch. No reachable new correctness, memory, or performance defect was substantiated.
  • Tests and results: I inspected FE and BE unit cases, fixture builders/artifacts, regression SQL, and expected output; I did not run builds or tests, as required by the review prompt. The author-reported local suite results are not independent validation. Negative cases cover invalid selectors, missing versions/tags/branches, prepared placeholders, timestamp zones, and missing fragment IDs.
  • Observability, persistence, and writes: EXPLAIN now names catalog type, version, managed status, and branch; BE errors/profile identify the opened snapshot. FE changes add no EditLog, transaction, or data-write operation. No new FE-to-BE variable requires separate sending paths.

Existing inline P2/P3 threads on historical branch assertions (4172612906), legacy ANN planning (4172821000), and historical tag phase-two coverage (4172821003) were read and checked against the current diff. Their current-head follow-ups add discriminating assertions/fixtures; I found no distinct new issue to repost. No existing P0/P1 inline finding remains to carry.

Gabriel39
Gabriel39 previously approved these changes Oct 6, 2026

@Gabriel39 Gabriel39 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The updated managed-branch guard addresses the Unicode compatibility issue from my earlier review: it checks URI/path safety without relying on the JDK to recognize every character accepted by Lance. The added coverage includes the existing managed branch scan and search paths. Approved based on static review; tests were not run locally.

zy-kkk added 4 commits October 8, 2026 12:11
…ector_search and full_text_search

vector_search() and full_text_search() take optional version, timestamp,
tag and branch properties with the semantics of FOR VERSION AS OF, FOR
TIME AS OF, @tag and @Branch on a Lance table. A tag stands alone,
version and timestamp exclude each other, branch may be combined with
either, and branch=main is the main chain. version takes only a number,
since tag names a tag without ambiguity, and an empty value is an error
rather than an absent selector.

The selector goes through the same catalog snapshot resolution as table
scans, including REST Namespace managed tables, and the schema, search
column, fragments and index metadata come from the selected snapshot.
Every execution resolves the selector again; nothing falls back to the
latest version. EXPLAIN of a search node shows the catalog type, the
managed flag and the branch, and planner errors name the branch.

full_text_search rejects prepared-statement placeholders, which used to
reach it as the literal text "?". A managed table's branch is opened by
a URI Doris joins, so Doris now checks the branch name against Lance's
rules before joining it. The time-resolution error no longer names FOR
TIME AS OF, since the timestamp property reaches it too.

The BE rejects search splits and row-id fetches without a positive
version, rejects FE fragment ids the opened snapshot lacks (lance-c
would skip them silently), names the dataset version and URI in
execution errors, and records the snapshot it read in the profile.

New regression fixtures search_snapshot.lance,
search_snapshot_pruned.lance and search_snapshot_evolved.lance cover
index history, deletes, an index rebuild, tags, a branch overlapping
main's versions, cleanup, and a column added and renamed after the
index was built.
The FE plans a vector-search split without index segments as a flat
search, and EXPLAIN and the profile count it as one. The BE forwarded
use_index only when the user set it, so Lance searched such a split with
the first index on the column in manifest order, which need not be the
index the FE selected, for example while an index is rebuilt under a new
name, or an index the FE could not plan with at all. The BE now sets
use_index to whether the split carries index segments.

A split that carries index segments while the request says
use_index=false is rejected. The FE plans segments only when use_index
allows them, so such a split contradicts itself, and the BE would
otherwise have to override either the plan or the request.
…h_snapshot

branch=dev with version 4 selects the fork point, whose rows and indexes
match main version 4, and the branch timestamp case asked for a future
time, which selects the branch head even if the time were ignored. Add
assertions that tell the branch apart from main and the time from the
head: dev version 5 returns the branch's rows, STRICT full-text search
succeeds on dev version 6 (its index was rebuilt over fragment 2) and
fails on main version 6, and a time between dev versions 5 and 6 selects
dev version 5.
…fetch

A vector index written before Lance 0.20 carries no index details. Lance
infers them from the index files when it loads the index, so the FE
plans it and the BE searches it through the planned segment rather than
flat. legacy_vector_index.lance, built by pylance 0.18.2 and carried over
as-is by the catalog build, and test_lance_vector_search_legacy_index
check that it is planned with one segment and returns the flat search's
rows.

The tag two-phase case now fetches row 3, which version 5 still has and
version 6 deleted, so a fetch from a later version would return fewer
rows, and it asserts the materialization node.
@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 84.54% (82/97) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.92% (31936/42628)
Line Coverage 59.27% (357695/603550)
Region Coverage 55.95% (298191/532927)
Branch Coverage 56.83% (135219/237916)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 55.07% (76/138) 🎉
Increment coverage report
Complete coverage report

@Gabriel39
Gabriel39 merged commit da03ee7 into apache:branch-4.1 Oct 9, 2026
31 checks passed
Gabriel39 pushed a commit to apache/doris-website that referenced this pull request Oct 9, 2026
…search functions (#4185)

Documents the `version`, `timestamp`, `tag` and `branch` parameters of
`vector_search()` and `full_text_search()`, added by apache/doris#68707
(issue apache/doris#68625).

Changes to `lakehouse/catalogs/lance-catalog.mdx` (4.x, English and
Chinese):

- Time Travel limitations: the search functions now select a snapshot
with their own parameters. Index inspection (`SHOW INDEX`,
`lance_index_entries()`) still uses the latest version of `main`.
- `vector_search()` and `full_text_search()` parameter tables: the four
selector parameters.
- Vector index selection: when several compatible indexes exist, Doris
uses one per query, the first by index name, and fragments it does not
cover use Flat Search. Indexes Doris cannot plan with, including legacy
indexes without index details, are not used.
- Catalog refresh: a refresh with cache invalidation clears the FE's
Lance Session only, not the BEs' caches.
- New section "Search a Historical Version or Branch": combination
rules, what is bound to the selected snapshot, no fallback to the latest
version, selection per execution, per relation and in `CREATE TABLE AS
SELECT`, the `EXPLAIN` and Profile fields, and known limitations
(branches of branches, a branch or Dataset recreated at the same
location, FTS indexes without token positions).
- Execution flow: the snapshot step names the selectors.

## Versions 

- [ ] dev
- [x] 4.x
- [ ] 3.x
- [ ] 2.1 or older (not covered by version/language sync gate)

## Languages

- [x] Chinese
- [x] English

## Docs Checklist

- [x] Checked by AI
- [ ] Test Cases Built
- [x] Updated required version and language counterparts, or explained
why not: the dev docs have no Lance catalog page yet, so only 4.x
changes.
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync
yiguolei pushed a commit that referenced this pull request Oct 10, 2026
…ector_search and full_text_search (#68707)

### What problem does this PR solve?

Issue Number: close #68625

Related PR: #68453 (time travel, tags, branches and managed versioning
for Lance table scans)

Problem Summary:

`vector_search()` and `full_text_search()` always searched the latest
version of `main`, and their `table` argument rejected `FOR VERSION AS
OF`, `@tag` and `@branch`. A search could not be reproduced against a
released tag, compared across versions, or run on an experimental
branch.

Both TVFs now take four optional properties, with the semantics of the
table syntax from #68453:

| Property | Value | Table syntax | Combines with |
|-|-|-|-|
| `version` | positive integer | `FOR VERSION AS OF N` | `branch` |
| `timestamp` | `yyyy-MM-dd HH:mm:ss[.SSS]`, session time zone | `FOR
TIME AS OF` | `branch` |
| `tag` | tag name | `@tag(name)` | none |
| `branch` | branch name, `main` is the main chain | `@branch(name)` |
`version` or `timestamp` |

```sql
SELECT id, _distance FROM vector_search(
    "table" = "lance_catalog.db.documents", "column" = "embedding",
    "query_vector" = "[0.1, 0.2, 0.3]", "version" = "3")
ORDER BY _distance;

SELECT id, _score FROM full_text_search(
    "table" = "lance_catalog.db.documents", "column" = "content", "query" = "database",
    "branch" = "dev", "timestamp" = "2026-09-20 12:00:00")
ORDER BY _score DESC;
```

How it works:

- **FE.** The TVF parses the properties into a `LanceRefSelector` and
loads its metadata through the same
`LanceCatalogClient.readTableSnapshot` as table scans, so
storage-versioned and namespace-managed tables follow the #68453 rules
unchanged. Schema, search column and field id, vector type and
dimension, fragments and index segments all come from the selected
snapshot, and planning never reopens the dataset. The splits carry the
branch directory URI and the version, so no thrift change is needed.
- **Each execution resolves again.** A bound TVF is never cached,
prepared statements re-analyze on every `EXECUTE`, and TVF statements
skip the SQL cache, so the latest version, a future time, a branch head
and a moved tag are resolved per execution and stay fixed within its
plan. `CREATE TABLE AS SELECT` plans its query twice (schema, then
rows), so there a moving selector can resolve twice, as it already can
for a table scan with an explicit selector; pinning `version` gives an
exact copy. This is documented.
- **No fallback.** A snapshot that cannot be selected, or whose index
files are missing, is an error; nothing falls back to the latest
version.
- **EXPLAIN** of a search node now shows `lanceCatalogType`,
`lanceManagedVersioning` and `lanceBranch` next to `lanceVersion`, as
table scans do. Planner errors that name "dataset version N" also name
the branch, since branch versions overlap main's.

Decisions worth calling out:

- `version` takes only a number. `FOR VERSION AS OF 'x'` names a tag
because SQL has one clause for both; the TVF has a `tag` property, and
Lance allows tag names made of digits, which `FOR VERSION AS OF` cannot
reach.
- An empty selector value is an error rather than "not set", so an empty
client-side variable cannot silently search the latest version of
`main`.
- The selector properties must be constants. Accepting `?` would leave
PREPARE without a snapshot to take the result schema from.
`full_text_search` now rejects `?` in any property: until now a
placeholder reached it as the literal text "?", so `'query'=?` searched
for "?".
- Each search relation resolves its own snapshot, as each explicit
selector already does for table scans. `vector_search(t) JOIN t` can
read two versions if a commit lands in between; pinning both with
`version` / `FOR VERSION AS OF` avoids that. This is documented.
- A managed table's branch is opened by a URI Doris joins
(`<table>/tree/<branch>`), which Lance does not validate on that path,
so Doris now rejects names that could leave the table's `tree/`
directory or change that URI: empty segments, `..`, `\`, and ASCII
characters other than letters, digits, `.`, `-`, `_` and `/`, so
`../other` and a percent-encoded dot segment are rejected before the
branch is opened. Outside ASCII only whitespace and control characters
are rejected, so no name Lance accepts is rejected, whichever Unicode
version the JDK and Lance's Rust use. Other tables check a branch out
through Lance, which validates the name itself.

BE changes:

- A search split or a two-phase row fetch without a positive version is
rejected instead of opening the latest version.
- Errors from opening the dataset, reading batches, preparing the FTS
query context and the two-phase row fetch name the snapshot: "... at
Lance dataset version N at <uri>", with the query string and any
password removed from the URI. Missing index files surface at execution
because Lance loads indexes lazily.
- Fragment ids from the FE must exist in the opened snapshot. lance-c
silently skips unknown fragment ids, which would drop rows without an
error.
- The profile records `LanceDatasetVersion` and `LanceDatasetUri`, the
snapshot an execution actually read. EXPLAIN cannot show this for moving
selectors because it resolves again.
- **Behavior change for every vector search:** a split without index
segments now always runs flat (`use_index=false`). The FE plans such
splits as flat search, and EXPLAIN and the profile count them as flat,
but Lance used to search them with the first index on the column in
manifest order, which need not be the index the FE selected (for example
while an index is rebuilt under a new name). Indexes the FE cannot plan
with (no fragment coverage metadata, no recorded metric, or a legacy
index whose details Lance cannot infer from its files, which the FE
skips; `lanceVectorIndexStatus=UNKNOWN_COVERAGE` / `METRIC_MISMATCH` /
`NO_MATCH`) are therefore no longer used implicitly, and such searches
become exact flat searches. Results can change only where the unplanned
index was approximate; latency can grow on such datasets. Legacy indexes
that lack details but whose details Lance infers, such as those written
by Lance 0.10 to 0.24, are planned and searched as before
(`test_lance_vector_search_legacy_index`).
- A vector split that carries index segments while the request says
`use_index=false` is rejected. The FE plans segments only when
`use_index` allows them, so it never sends such a split; the BE now
takes `use_index` from the segments and would otherwise have to override
either the plan or the request.

Known limitations (documented):

- A branch of a branch, or a branch of a shallow clone, can fail to find
the index files it inherited, because Lance writers before v13 record
them under the intermediate branch (lance-format/lance#9176). The search
fails with an error.
- After a branch is deleted and recreated under the same name, or a
dataset is replaced at the same URI, version numbers repeat. Since
#68615 the BE's Lance keys the index list and the row-id index by the
manifest e_tag (lance-format/lance#8904, #8930), but the BE's
process-wide Lance session still keys two entries by version alone,
until eviction or restart. On datasets with stable row ids, the live-row
mask that vector and full-text search prefilter with can come from the
old data, so rows deleted in the new data can reappear or live rows can
be missing (lance-format/lance#9079, open). For a manifest without an
e_tag (V1 naming, or a store that returns no ETag), the index list can
come from the old data, on the BE and in the FE's Lance session until
the catalog is refreshed. Neither is introduced by this PR.
- `SHOW INDEX` and `lance_index_entries()` keep showing the latest
version of `main`.

### Release note

`vector_search()` and `full_text_search()` accept `version`,
`timestamp`, `tag` and `branch` to search a historical version, the
version at a time, a tag or a branch of a Lance table, including REST
Namespace managed tables. Vector search no longer uses an index the
planner did not select, and `full_text_search()` rejects
prepared-statement placeholders.

### Check List (For Author)

- Test: Regression test / Unit Test
- Regression: `external_table_p0/lance/test_lance_search_snapshot`
(filesystem catalog) and `test_lance_rest_search_snapshot` (REST,
storage-versioned and managed) on a new fixture `search_snapshot.lance`.
It has seven versions (appends, a vector index, an FTS index, an append
no index covers, a delete, a vector index rebuild), a tag, and a branch
`dev` whose versions and fragment ids overlap main's with different
rows, plus a tag at a non-latest branch version.
`search_snapshot_pruned.lance` lost a version to cleanup and the index
files of a tagged version. `search_snapshot_evolved.lance` adds a column
and renames the vector column after its index is built, then appends
data the index does not cover. The suites check rows, distances,
two-phase reads (on a branch, a tag, the columns of a version before and
after a rename, and a managed table through REST), EXPLAIN, full-text
search across a delete, `use_index=false` with a selector, every
selector error, and a managed branch named with a Unicode 16 letter
(`unicode_branch.lance`, built by pylance 12.0.0) through a table scan,
a search and a two-phase read. `test_lance_rest_catalog.out` and
`test_lance_rest_time_travel.out` list the new REST tables.
`test_lance_vector_search_legacy_index` searches a vector index written
by pylance 0.18.2 without index details and checks that it is planned
(`USED`, one segment) and returns the flat search's rows. All 30 suites
in `external_table_p0/lance` pass locally.
- FE unit tests: `LanceSearchSelectorTest` (property rules, time zones,
selector forwarding), `LancePreparedSearchTest` (both TVFs pass their
selector to the table; each analysis resolves it exactly once, and a
moved tag is resolved again on the next `EXECUTE`; search statements
bypass the SQL cache; placeholders rejected), `LanceScanNodeTest`
(EXPLAIN, errors naming the branch), `LanceCatalogClientTest` (branch
names, including letters newer than JDK 17's Unicode tables), and, with
real datasets and locally only like the other JNI tests,
`LanceSearchSnapshotTest` (historical index coverage, branches, tags,
schema evolution, cleanup, a snapshot planned before later commits) and
`LanceManagedVersioningTest` (managed versions, tags, branches,
unfinalized heads through the search entry points).
- BE unit tests in `lance_reader_test.cpp`: missing fragment ids on
normal, vector and FTS splits, unpinned versions, error and profile
contents, a split without index segments searching flat although an
index covers it, and a split with index segments under `use_index=false`
being rejected. The indexed variants of the existing multi-vector tests
now pass the index segments the FE would plan.
- Behavior changed: Yes. New TVF properties; vector splits without index
segments always search flat; `full_text_search` rejects `?`; managed
branch names are validated; the error for a time in removed history now
reads "Lance cannot select the version at or before '...'" instead of
"Lance cannot resolve FOR TIME AS OF '...'", because the TVF reaches it
too; BE errors name the dataset version and URI.
- Does this need documentation: Yes (apache/doris-website#4185)
hello-stephen pushed a commit that referenced this pull request Oct 14, 2026
… at graceful exit (#68942)

Problem Summary:

**In short.** The NonConcurrent regression pipeline fails a few percent
of its runs with `Lsan fail` while every case passes. LeakSanitizer
finds, at the BE's graceful exit, one stream load's `ResourceContext`
that was never released, but its report only shows allocation stacks,
which are the same for every stream load, so nobody can tell which load
it was. This PR makes the BE print, at graceful exit in memleak-check
mode, every `ResourceContext` still alive with its task id, the
references still held, its memory tracker and workload group, and
whether its `QueryContext` is already gone. Nothing changes unless
`enable_graceful_exit_check=true`.

**Background.**
- With `enable_graceful_exit_check=true` (memleak-check mode, used by
the regression pipelines on the ASAN build), `doris_main` stops every
service and runs `ExecEnv::destroy()` instead of `_exit()`, so that
LeakSanitizer checks the heap when the process exits. The NonConcurrent
pipeline greps `be.out` for `detected memory leaks` after `stop_be.sh
--grace` and reports `Lsan fail`.
- Every query or load has a `ResourceContext`: CPU/memory/IO contexts, a
task controller, the query memory tracker and the workload group. The
`QueryContext` owns it, and the components that work for the task (load
channel, memtables, scanner context, `RuntimeQueryStatisticsMgr`,
threads attached to the task, ...) hold references of their own while
they run.

**The problem, and what it cost.**
- `Lsan fail` hits 1–5% of NonConcurrent runs depending on the month
(448 of the last 14000, back to March 2026), on PRs that have nothing to
do with it; authors rerun or skip. Every report TeamCity still keeps (10
runs from 2026-10-05 to 2026-10-12, on #68707, #68281, #68791, #68803,
#68798, #68543, #68901, #68919, #68879) has the same shape, down to the
byte counts, and all of its leaks are indirect:

  | Leaked | Size | Origin |
  |---|---|---|
| one `ResourceContext` with its CPU/memory/IO contexts,
`QueryTaskController` and both trackers (`Load#Id=…`,
`Memtable#Load#Id=…`) | 6.5 KB, 94 objects | an HTTP stream load:
`StreamLoadAction::_process_put` → `StreamLoadExecutor` →
`FragmentMgr::_get_or_create_query_ctx` |
| the `normal` workload group created by
`WorkloadGroupMgr::create_internal_wg()` | 6.1 KB + 4.3 KB | kept alive
by `ResourceContext::_workload_group` |
| the `std::make_shared` block of the load's `QueryContext` | 2.5 KB |
kept allocated by the `weak_ptr` in `QueryTaskController` |

- The `QueryContext` itself was destroyed: in `be.INFO`, each of the
21894 `query_id: … use wg:` lines of the run has its `Query …
deconstructed`. So one reference to the load's `ResourceContext` is
never released, from a place LeakSanitizer does not scan (a bthread
stack, whose memory brpc maps itself) or a count that is never
decremented. LeakSanitizer cannot tell which of these two it is.
- A NonConcurrent run issues about 430 HTTP stream loads (about 380
through the load channel, 150 of them audit-log loads), and the report
cannot say which one leaked: the stacks are identical, and the one thing
that would identify the load, the tracker label with its id, is inside
the leaked memory, which LeakSanitizer does not print. Without the load
there is nothing to follow, so the flake stays unexplained.

**How this PR fixes it.**
- In memleak-check mode each `ResourceContext` registers itself in a
process-wide set when it is constructed and removes itself when it is
destroyed. The set stores the addresses bitwise inverted, so
LeakSanitizer does not take it for a reference and still reports a
leaked context.
- `ResourceContext::describe_live_contexts()` returns one line per
context still alive: address, strong references still held, task id,
finished flag, running time, memory tracker (label, consumption, peak),
workload group and, for a query, whether its `QueryContext` is still
alive.
- `doris_main` calls `ResourceContext::log_live_contexts()` right after
`ExecEnv::destroy()` on the graceful exit path. It writes the
description to `be.WARNING` and to `be.out`, where the LeakSanitizer
report follows once `main()` returns, or one INFO line when no context
is alive. The printed address falls inside one of the `Objects leaked
above` ranges of that report.
- The description is built on a short-lived thread, and the main thread
only handles the resulting text. Describing a context touches it; done
on the main thread, it could leave a pointer to a leaked context in a
dead slot of the main thread's stack, which LeakSanitizer scans at exit
— it would then take the context for referenced and not report it, so
the run that reproduces the leak could pass. The stack of a finished
thread is not scanned.

**Results.** What the next `Lsan fail` will tell, compared with today:

| | Before | After |
|---|---|---|
| Which query or load leaked | unknown | `task_id=…`; its label, table
and test are then in `be.INFO` |
| How many references are left | unknown | `references=N` |
| Whether the `QueryContext` is gone | only by pairing ~20k log lines |
`query_context=destroyed` |
| Where to look | the LeakSanitizer report in `be.out` | the same
`be.out`, just before that report (the build log's `head -n 100 be.out`
shows it) |
| Cost with `enable_graceful_exit_check=false` (default) | — | one
boolean check per `ResourceContext` construction and destruction |

A line looks like this (values made up):

```
1 resource contexts are still alive. LeakSanitizer reports a leaked context as the memory block that holds it, so match the addresses below against the ranges it prints under 'Objects leaked above'.
  0x7c0000001010: references=1, task_id=1eac0c4ec4ec0000-0000000000000003, finished=true, running_ms=7569, mem_tracker=Load#Id=1eac0c4ec4ec0000-0000000000000003 (consumption=0, peak=17.18 MB), workload_group=normal(1), query_context=destroyed
```

**Classes, and how they call each other.**
- `ResourceContext` (`resource_context.{h,cpp}`): registers in the
constructor and unregisters in the destructor when memleak-check mode is
on; `describe_live_contexts()` formats the live ones;
`log_live_contexts()` runs that on its own thread and logs the result.
- `QueryTaskController`: new `is_query_context_alive()`.
- `doris_main`: one call to `log_live_contexts()` after
`ExecEnv::destroy()`.

```
doris_main (graceful exit, enable_graceful_exit_check=true)
  ├─ ExecEnv::destroy()
  └─ ResourceContext::log_live_contexts() ──► be.WARNING + be.out (or one INFO line)
     └─ [short-lived thread] describe_live_contexts(100)
        ├─ registry of disguised addresses ◄── ResourceContext() / ~ResourceContext()
        └─ per live context:
              ├─ TaskController: task id, finished, running time
              ├─ MemoryContext → MemTrackerLimiter: label, consumption, peak
              ├─ WorkloadGroup: name, id
              └─ QueryTaskController::is_query_context_alive()
```

No holder of `ResourceContext` changes, and neither do the LeakSanitizer
options or suppressions. Once the leaked load is known, the fix for the
leak itself will be a separate PR.


---------

Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
yiguolei pushed a commit that referenced this pull request Oct 15, 2026
…69126)

### What problem does this PR solve?

Related PR: #69124, #68707

Apply the fixture portability fix from #69124 to `branch-4.2`.

The Unicode Lance branch fixture tracks five files below `tree/devᲉ/`,
causing checkout failures on filesystems that cannot represent that
directory name. Store these files under `tree/dev_u1c89/` in Git and
restore the original Unicode object prefix when uploading to MinIO.
Fixture payloads and runtime Unicode branch coverage remain unchanged.
Update the fixture checker and regeneration instructions to use the
portable layout, and fail initialization if upload or relocation fails.

### Release note

None

### Check List (For Author)

- Test
    - [x] Manual test (details below)
- Behavior changed:
    - [x] No. Runtime fixture paths and contents are unchanged.
- Does this need documentation?
    - [x] No. Fixture regeneration instructions are updated inline.

Validation on this branch:

- All tracked paths are ASCII and contain no literal backslashes.
- Fresh checkout of all 10 Unicode fixture files with
`core.protectNTFS=true` and `core.protectHFS=true` on Linux.
- `lance_build_unicode_branch.py --check` passes with pylance 12.0.0.
- Python, YAML and shell syntax checks and `git diff --check` pass.
- Verified the complete Iceberg fixture/setup subtree is identical to
#69124. That change was validated against temporary MinIO with two
initialization runs, matching all 515 original object keys and blob
hashes, and successful reads of both Unicode branch versions and vector
search. The MinIO integration test was not repeated for this identical
port.

Full Doris regression tests and a native Windows checkout were not run
locally.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
yiguolei pushed a commit that referenced this pull request Oct 16, 2026
### What problem does this PR solve?

Related PR: #68707

The Unicode Lance branch fixture introduced by #68707 tracks five files
below `tree/devᲉ/`. This causes checkout failures on filesystems that
cannot represent that directory name.

Store these files under the ASCII directory `tree/dev_u1c89/` and
restore the original Unicode object prefix when uploading fixtures to
MinIO. The fixture payloads remain byte-for-byte identical, and the
existing Unicode branch regression coverage is preserved. Update the
fixture checker and regeneration instructions to use the portable Git
layout. Initialization exits on upload or relocation failure instead of
continuing with an incomplete fixture.

### Release note

None

### Check List (For Author)

- Test
    - [x] Manual test (details below)
- Behavior changed:
    - [x] No. Runtime fixture paths and contents are unchanged.
- Does this need documentation?
    - [x] No. Fixture regeneration instructions are updated inline.

Validation:

1. Scanned all tracked paths: no non-ASCII names or literal backslashes
remain. Checked out all 10 files in the Unicode fixture using a fresh
Git index with `core.protectNTFS=true` and `core.protectHFS=true` on
Linux.
2. Ran `lance_build_unicode_branch.py --check` on the renamed fixture
with pylance 12.0.0.
3. Executed the upload commands extracted from `iceberg.yaml.tpl`
against a temporary MinIO instance with the pinned mc image. Ran
initialization twice, injecting a stale object before the second run.
After each run, all 515 object keys and blob hashes matched the original
fixtures exactly; neither the ASCII alias nor the stale object remained.
4. Read the main snapshot and both Unicode branch versions with pylance
12.0.0. Confirmed 8 rows in branch version 1, 12 rows in version 2, and
vector search returning the expected branch-only rows.
5. Passed YAML parsing, shell syntax validation, and `git diff --check`.

Full Doris regression tests and a native Windows checkout were not run
locally.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
morningman added a commit to morningman/doris that referenced this pull request Oct 25, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 26, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 26, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 27, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 27, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 28, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 28, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 29, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 29, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 31, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Oct 31, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Nov 2, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Nov 2, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Nov 5, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690

Problem Summary:

Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2
cabcf45) to master. Master links liblance_c but has no Lance reader:
file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no
Lance path, so every FORMAT_LANCE scan fails today. This change is BE
only. The FE connector plugin and the regression suites follow
separately, so nothing on master plans a Lance scan yet and the BE unit
tests carry the proof.

- Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.*
  (plain scans, vector and full-text search, Substrait and runtime
  filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT,
  profiles), copied from branch-4.2 except one local renamed for
  -Wshadow.
- Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2
  dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use
  FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss
  do: there is no V1 Lance reader. The second phase of TopN lazy
  materialization reads Lance rows through dataset take_rows.
- Row locations: GlobalRowLoacationV2 gets a version 1 layout with a
  64-bit Lance row id (the version 0 layout is unchanged),
  PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and
  the materialization operator and RowIdStorageReader decode and
  validate by version.
- Thrift: the external search request body, TLanceFileDesc 4-6 and
  TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids, in place of the
  comments that reserve them on master (apache#70075); field 6 stays master's
  paimon_options.
  TVectorSearchOptions 6 is branch-4.2's approx_mode (apache#69852), reserved
  in a comment.
- Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the
  nullable serde reads an Arrow Null array as NULLs.
- Nine lance_* BE configs, a BE-wide Lance session with an optional
  Foyer data cache and 10 lance_session_* metrics; macOS links IOKit.
- Tests: the Lance BE unit tests and their fixtures, including the
  docker fixture all_types.lance that one of them reads. Under master's
  -ffile-prefix-map the session test locates its fixture from the
  repository root, and the reader tests shrink the data cache to 64 MiB
  under the UT temp dir: the default 100 GiB disk tier is 51200 block
  files that stay open for the rest of the test process.

Not included: the MaterializationOperator::push timer fix (apache#69579), the
Lance index metadata scanner, the FE side, regression suites, and the
4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815).

### Release note

Add nine lance_* BE configs and lance_session_* BE metrics. They take
effect once the FE can plan Lance scans.

### Check List (For Author)

- Test: Unit Test
    - BE UT: the Lance reader, nested Null, runtime filter and session
      suites, plus the row-location protocol and FileScannerV2 cases
- Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the
  second phase of TopN lazy materialization rejects row locations of an
  unknown version or size.
- Does this need documentation: No (documented with the FE connector)

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
morningman added a commit to morningman/doris that referenced this pull request Nov 5, 2026
… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
yiguolei pushed a commit that referenced this pull request Nov 6, 2026
### What problem does this PR solve?

Issue Number: None

Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110
#67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690
#68813 #68815 #68707 #68649

Problem Summary:

**In short.** This ports the Lance BE read path of branch-4.1 /
branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus
the BE side of three 4.x changes merged after that snapshot: the search
filter `prefilter` flag and profile layout (#68813), distance bounds for
`vector_search` (#68815), and the split-level index rule of #68707 that
#68815 relies on. It gathers the BE parts of the PRs listed above.
Master links `liblance_c` but has no Lance reader, so every
`FORMAT_LANCE` scan fails today. With this PR the BE reads Lance
datasets through FileScannerV2: plain scans, vector search and full-text
search, predicate and runtime-filter pushdown, the schema fetch of a
local Lance TVF, and the second phase of TopN lazy materialization. The
PR is BE only. The FE connector plugin and the regression suites come in
separate PRs, so until then nothing on master plans a Lance scan, and
the BE unit tests carry the proof.

Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju
@FANNG1

**Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` /
`s3()` with `format=lance`, and the `vector_search` / `full_text_search`
TVFs. On master the FE side is being rebuilt as a connector plugin
(`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift
ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5`
(`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added
after the snapshot, so a BE built from master also reads the Lance scan
ranges and search requests that a 4.x FE sends during a rolling upgrade.
The one exception is `TVectorSearchOptions` 6, branch-4.2's
`approx_mode` (#69852, merged into branch-4.2 after the snapshot and not
ported): it is reserved in a comment, and a master BE ignores it, so
during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate`
gets Lance's default (`normal`).

**The problem, and what it cost.**

| | master today | with this PR |
|---|---|---|
| A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan
fails with "FileScannerV2 does not support file format" | read by
`LanceTableReader` in FileScannerV2 |
| `fetch_table_schema` for a local Lance TVF | not handled | returns the
Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` |
| The row location of a Lance row in TopN lazy materialization |
`GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a
64-bit id within the snapshot | a version 1 layout carries the 64-bit
row id; phase two reads the rows with the dataset's `take_rows` |

**How this PR fixes it.**

- Reader: `be/src/format_v2/lance/*` and
`be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one
local renamed for master's `-Wshadow`. The unit tests construct
`RuntimeState(TQueryOptions(), globals)`, because master removed the
globals-only constructor.
- FileScannerV2: the reader dispatch and `_to_file_format` handle
`FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two
exhaustive switches. Since #69723 every file scan uses FileScannerV2, so
nothing else routes Lance scans (on branch-4.2 a Lance query fails with
NotSupported when `enable_file_scanner_v2` is off).
- Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0,
LANCE_DATASET_ROW_ID = 1}` and a union layout for
`GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static
asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2
varint, so the wire format is compatible). The materialization operator
decodes row locations by version, rejects an unknown version or a wrong
size, and stamps `row_location_version`. `RowIdStorageReader` validates
the counts and the version, range-checks internal row ids against
`uint32`, and groups Lance rows by plan node, snapshot version and
dataset URI, so each dataset is read with one `take_rows`. Master's
segment batching, scan-slot dedup and fetch concurrency stay as they
are.
- Thrift (`PlanNodes.thrift`): the external search request body
(`TVectorElementType`, `TVectorMetric`, `TSearchVector`,
`TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`,
`TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`,
`TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`),
`TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the
branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is
reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in
comments and added 6 (`paimon_options`); this PR replaces those comments
with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is
reserved in a comment for branch-4.2's `approx_mode` (#69852).
- After the snapshot (one commit each):
- #68813: `prefilter` decides whether the TVF search filter runs before
(true) or after (unset / false) candidate selection, for vector and
full-text search. The profile groups the Lance counters under
`LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`,
`LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan`
and `LanceDataCache`, uses the 4.x counter names, and registers every
scan metric before the statistics callback is installed, so the
callback, which Lance invokes from its execution streams, never adds
counters. #68813's FE tweak that hides `TUnit.NONE` labels in
`AggCounter` belongs to the FE port.
- #68707 (part): a vector split searches exactly the index segments FE
planned for it, and a split without segments is searched flat whatever
`use_index` says; a split with segments under `use_index = false` is
rejected. 4.x FEs plan segments only when `use_index` allows them.
- #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound`
(exclusive) are validated (finite FLOAT values, lower < upper at FLOAT
precision, single-vector only) and installed with
`lance_scanner_set_distance_range()`, which master's pinned lance-c
provides.
- Review fixes:
- A nested array goes to the Doris SerDes without a copy only when it
and every descendant start at offset 0 and reference exactly their child
ranges. Lance's legacy (v1) decoder can return a struct or list that is
tight at its own level over a Boolean child with a bit offset (a batch
that starts mid-byte in the bitmap), and the SerDes reject that under
Arrow input validation.
- A zero-field `STRUCT<>`, which Lance decodes, is reported as an
unsupported column at any nesting level, like the other unsupported
Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot
hold (its row count comes from its first child).
- A Lance Decimal256 field with precision <= 38 is narrowed to
Decimal128 by a safe Arrow cast before decoding. Doris types it
DECIMAL32/64/128I by precision, and those SerDes read only Decimal128
arrays; the BE used to crash on the first value, also for nested leaves.
Only values a reader can see are cast: before a nested leaf is narrowed,
every child slot that no visible row references (under a NULL list,
fixed-size-list, map or struct row, or outside a sliced parent) is
marked NULL, so a hidden value does not fail the cast and a visible
value outside the declared precision still does; map keys, which cannot
be NULL, stay checked.
- FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares
floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows
that Doris keeps.
- Runtime filters that mention INT64_MIN stay residual-only. The pinned
Lance SQL parser reads `-9223372036854775808` as the negation of a
magnitude that it parses as i64 first; the overflow turns the literal
into a Float64, which the planner cannot coerce to Int64 or Decimal, so
a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal
column failed planning and the query failed. Every other i64 is still
pushed.
- The shared Arrow DATE32 decoding
(`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside
0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`,
skips null slots and adds the epoch in 64 bits: the DATE32 range part of
branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and
paimon-rust reads.
- Child values hidden by a NULL list, fixed-size-list, map or struct row
are not decoded: the array, map and struct SerDes give the element SerDe
a validity bitmap that also clears those slots
(`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them
unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden
10000-01-01 under a NULL fixed-size list used to fail the DATE32 range
check above. Visible values are unchanged; this applies to ADBC, Arrow
stream load and paimon-rust as well. The masks come from ExecEnv's
tracked Arrow pool. A nested Lance array that is compacted before
decoding is masked the same way, at every level, before its
`ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision
decimal, Date64 or time no longer fails the batch while a visible one
still does. Since #69697 the same masks keep hidden DATETIMEV2 /
TIMESTAMPTZ values out of its range checks. A mask costs at most one
pass per batch: a NULL row that owns no child slots, or only NULL ones,
allocates nothing, and Arrow stream load masks each retained IPC batch
once (`mask_arrow_descendants_not_visible()`) before it hands row ranges
to the SerDes, so its per-block reads allocate nothing. The one
per-block mask left is for map keys under a NULL map row that still owns
entries, since a MapArray cannot hold NULL keys; Arrow's map builder
writes NULL map rows without entries.
- Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a
Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The
nullable serde reads an Arrow Null array as NULLs without touching
buffers that the array does not have.
- Configs, caches and metrics, with the 4.x names and defaults:

  | Config | Default |
  |---|---|
  | `lance_index_cache_size_bytes` | 10 GiB |
  | `lance_metadata_cache_size_bytes` | 1 GiB |
  | `enable_lance_data_cache` | true |
  | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` |
  | `lance_data_cache_disk_capacity_bytes` | 100 GiB |
  | `lance_data_cache_read_block_size_bytes` | 1 MiB |
  | `lance_io_buffer_size_bytes` (mutable) | 2 GiB |
  | `lance_batch_readahead` (mutable) | 5 |
  | `lance_fragment_readahead` (mutable) | 5 |

The BE-wide Lance session (index cache, metadata cache and the optional
Foyer disk cache) is created by the first Lance scan, so a BE that never
reads Lance creates no cache directory. The first six configs take
effect after a restart. A negative cache size (and, while the data cache
is enabled, a negative disk capacity or read block size) fails the first
Lance scan with a message that names the setting, as does any data-cache
value lance-c rejects; the rest of the BE keeps serving. Ten
`lance_session_*` metrics report the two in-memory caches. With the
defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from
the first Lance scan on (100 GiB in blocks of twice the read block
size), the same as on branch-4.1 / 4.2. Size
`lance_data_cache_disk_capacity_bytes` and
`lance_data_cache_read_block_size_bytes` against the BE's open-file
limit, or set `enable_lance_data_cache = false`.

The index and metadata caches live in lance-c memory: they are not
charged to a MemTracker or registered with CacheManager, so memory GC,
the cache capacity adjustment under memory pressure and
`/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache
= false` does not turn them off. lance-c has no call to clear, evict or
resize them, so size them with the two configs. The disk tier keys
blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS,
without the endpoint) and the object path
(`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve
the same bucket name and data-file path with different bytes (data files
registered under chosen names or replaced in place; Lance's writers use
random file names) share entries, also across a BE restart; such a
deployment should set `enable_lance_data_cache = false`. Both are as on
branch-4.1 / 4.2 and tracked as follow-ups.

A Lance scan's native working set, that is the I/O buffered up to
`lance_io_buffer_size_bytes` per scanner, the decode readahead and the
operators of a search plan such as the top-k of a flat vector search, is
allocated inside lance-c and is charged to neither the query nor its
workload group. It counts toward the BE process memory only, so memory
GC, which ranks queries by tracked memory, does not attribute it to the
Lance query. lance-c accepts no memory budget, so on BEs with little
memory headroom size `lance_io_buffer_size_bytes`,
`lance_batch_readahead` and `lance_fragment_readahead` for the number of
concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a
follow-up.
- Build: macOS links `-framework IOKit`. This is the first time objects
of `liblance_c.a` are pulled into `doris_be`, so the binary and the link
time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and
`util/threadpool.h` 589, the same as master `315c2e00331` alone (limits
275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562
(master alone 559, limit 660); the reach budgets are unchanged.

Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`);
the FE side, including the rolling-upgrade guard that keeps Lance lazy
materialization off while an old BE could decode a version 1 row
location as version 0; the regression suites; the rest of #68707
(searching historical versions, tags and branches, rejecting fragment
ids missing from the snapshot, naming the snapshot in errors and the
profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that
#67005 carried goes in a separate PR.

**Results.**

| | master | with this PR |
|---|---|---|
| A `FORMAT_LANCE` query | fails before reading anything | read by
`LanceTableReader` in FileScannerV2 |
| A Lance row in the second phase of TopN lazy materialization | cannot
be represented | a version 1 row location, read back with `take_rows` |
| A row location of an unknown version or size in phase two | decoded as
version 0 | rejected with an error |
| An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream
load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read
fails with `InvalidArgument` |

BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on
the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run,
1027 pass and 2 Parquet microbenchmarks skip by design. They cover the
Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10,
`LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the
row-location protocol (`MaterializationSharedStateTest`,
`RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`:
count mismatch, unknown version, a Lance version on an internal mapping,
an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and
Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`,
`ParquetScanTest` and the other `Parquet*` suites), whose row ids now go
through `file_local`; and the Arrow / protobuf / DATE serde suites with
the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader
suites that decode Arrow dates. Reverting any of the review fixes, the
#68813 flag, the #68707 split rule or the #68815 bounds makes its new
tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest`
10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow
SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310
pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The
ninth review round adds
`LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and
`LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`:
both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto
`339e087b11f`, and restoring `parse_int64()` in either check makes
exactly these two fail. On the head rebased onto master `315c2e00331`,
with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites
pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest`
(15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe
suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`,
`AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest`
and the Lance suites. Allocating at the first NULL row of each call
again fails the new `ArrowParentNullsTest` case and the four new
`LoadArrowPipeTest` block cases, dropping the per-batch mask fails the
LIST and STRUCT block cases, and returning the child unmasked fails the
new timestamp case, four `ArrowParentNullsTest` cases and
`LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`.

### Release note

Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They
take effect once the FE can plan Lance scans. Reading an Arrow DATE32
value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or
paimon-rust now fails instead of returning 0000-00-00.

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
- `run-be-ut.sh --run --filter=...` with the suites listed under
Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the
rebases, and the Lance and Arrow SerDe suites pass again with the
nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and
compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite
driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the
scanner selection and in `fetch_table_schema`, which now hands every
file format to `format::fetch_file_schema()`, so the Lance branch runs
ahead of that call; #69646 added a TopN row-group test that reads the
version 0 row location fields, which now go through `file_local` (the
earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the
scanner selection, the `FileFormat` enum, its test and the reach budget
table). The 95 TUs that this PR changes or that master changed since
`5caaeb3e3d4` were syntax-checked against the rebased tree with the UT
build flags, and CI runs the suites again. Rebased again onto master
`8d88f1cc6f7`: the only conflict was the budget table, where #65858
moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that
this PR changes or that master changed since `43f9ceb48d7` pass the same
syntax check. Rebased again onto master `339e087b11f`: the only conflict
was the budget table again, where #67292 moved the `util/rle_encoding.h`
baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent
masking in the map and struct readers and merged cleanly. Of the 114
sources that this PR changes or that master changed since `8d88f1cc6f7`,
111 pass the same syntax check. The other three are master's: two new
#67292 files under `storage/segment` fail the same way on master alone,
since they compile only inside their unity batch, and this PR adds no
source to that target; the #69818 benchmark is not in the UT build and
includes only `util/simd` headers that this PR does not touch. Reverting
any of the review fixes, the #68813 flag, the #68707 split rule or the
#68815 bounds makes its new tests fail. Rebased again onto master
`378404eeb12`, which removed the legacy file scanner (#69723): the
conflicts were in the scanner-selection code and tests that #69723
deleted, so this PR's Lance lines there went with them (including its
`LanceQueriesAlwaysUseScannerV2` test), in the profile keys of
`materialization_opertor.cpp`, and in the row fetch of
`rowid_fetcher.cpp`, whose non-Lance path is now only
`FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests
took this PR's `file_local` rename with them, and the
`util/rle_encoding.h` budget commit is dropped because master is no
longer near that limit. Of the 100 sources that this PR changes or that
master changed since `339e087b11f`, 99 pass the same syntax check; the
other, `format_v2/orc/orc_reader.cpp`, fails the same way on master
alone. On the rebased head the suites listed under Results run 1029
cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that
`DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master.
Rebased again onto master `315c2e00331`: the only conflict was
`TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had
reserved fields 2-5 in comments; the two commits that add the fields
replace those comments, and field 6 stays #70075's `paimon_options`.
#69717 cut transitive includes from shared headers: of the 160 sources
that this PR changes or that master changed since `378404eeb12`, 156
pass the same syntax check; the two `jemalloc_hook` tests have no
compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc
aliases that macOS cannot build (#69819), and
`format_v2/table_reader.cpp` alone fails the same way on master alone
while its unity batch compiles. The full `doris_be_test` builds and
links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases)
pass before the mask fix, and the 328 cases above with it.
- `build-support/check-build-hygiene.sh` (on this branch rebased onto
master `315c2e00331`), clang-format 16 and
`build-support/run-clang-tidy.sh` on the changed files: all pass; on the
changed lines clang-tidy (llvm 22) reports only
`readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of
the masking functions, the macro's expansion that master's existing
`DORIS_CHECK`s report too.
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- Behavior changed:
    - [ ] No.
- [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN
lazy materialization rejects row locations of an unknown version or
size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day
outside the Doris DATE range fails the read instead of becoming
`0000-00-00`.
- Does this need documentation?
    - [x] No. Lance is documented with the FE connector.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
Co-authored-by: Gabriel <liwenqiang@selectdb.com>
Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com>
Co-authored-by: jay <qingfengyouchen@163.com>
Co-authored-by: FANNG <fanng@apache.org>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants