Repository navigation
[docs](lance) Document time travel, tags, branches and managed versioning - #4172
Merged
Merged
Conversation
…ribe Describe how FOR TIME AS OF compares commit times without assuming they grow with version numbers and ignores history before a version removed by cleanup, and that for a managed table the BE reads with the location and storage configuration of the describe the Lance SDK opened the table with.
… versions and gaps Describe the redesigned managed-versioning read path: the FE and the BE read the version the Namespace selects by URI and version number, a managed table's access is not cached, a version recorded only at a staged manifest without its canonical manifest cannot be read, a version the Namespace no longer records between recorded ones cuts FOR TIME AS OF history, and the Namespace decides whether a branch of a managed table exists.
…h checks A managed table's branch is read from its directory without reading main, a table_uri may add a query to its location and an s3+ddb URI is rejected, and FOR TIME AS OF requires every version the Namespace records on the chain to be at a manifest Doris reads.
Gabriel39
pushed a commit
to apache/doris
that referenced
this pull request
Sep 29, 2026
…ce managed versioning (#68453) ### What problem does this PR solve? Issue Number: close #66491 Related PR: #65730 (baseline Lance read integration, which already carried `FOR VERSION AS OF` / `FOR TIME AS OF` on the FE without tests or documentation) Problem Summary: Lance catalogs could select a dataset version on the FE, but nothing proved it end to end, the error messages leaked SDK internals (`NumberFormatException: For input string`, storage paths and Rust source locations), and tables whose versions are managed by a REST Namespace (`DescribeTable` returns `managed_versioning = true`) were rejected outright with "not supported by the current BE reader". This PR completes the four scope items of the issue: 1. **Time travel** – `FOR VERSION AS OF <n>` and `FOR TIME AS OF '<timestamp>'` are covered by regression suites on a filesystem catalog and a REST catalog, with a committed three-version fixture (`time_travel.lance`) and the existing `multi_frag.lance` (three appends, then a delete) so that reading an older version across a deletion file is exercised on the BE. The semantics follow the Iceberg and Paimon integrations: the selected version is the one committed last at or before the timestamp (of versions committed at the same instant, the newer one), with the timestamp parsed in the session time zone with second or millisecond precision and compared with the full commit time, which Lance records below a millisecond, so a commit later within the requested millisecond is not selected; commit times are compared as recorded, without assuming they grow with version numbers. As Iceberg drops its snapshot log before a removed snapshot, only the versions newer than the most recent one removed by cleanup are candidates, so a timestamp earlier than all of them (before the first version, or in removed history) is an error. 2. **REST Namespace managed versioning** – the namespace decides which versions of such a table exist and which is the newest: the latest version comes from `ListTableVersions`, an explicit version is checked with `DescribeTableVersion`, and `FOR TIME AS OF` selects among the versions the namespace lists. The FE then reads the chosen version the way it reads any Lance dataset, and the way the BE does through lance-c: by the location and storage options of the table's `DescribeTable` and the version number. No BE or thrift change is needed. `EXPLAIN` shows `lanceManagedVersioning=true` for these tables. 3. **Selected version fixed for the whole statement** – schema binding, fragment planning, predicate pushdown, metadata COUNT and the BE scan all use the selected version; two references to the same table in one statement can select different versions (self join / union tests). 4. **Tags and branches** – `tbl@tag(name)` resolves the tag to the branch and version it points at, read from the dataset's `_refs/tags/` for both storage-versioned and managed tables, and only that tag's file is read (Lance's namespace tag APIs expose the same refs; the selected version of a managed table is then resolved through the namespace); a tag created on a branch selects that branch's version, not the same number on `main`. As for Paimon, and for tag names in Iceberg, a non-numeric `FOR VERSION AS OF 'v2'` names a tag; a tag name is never combined with `@branch`, `@branch(main)` included, because the tag already determines its branch. `@branch(main)` is `main`; `@tag` / `@branch` take exactly one name and reject extra parameters. Inside a branch, versions and times select among the branch's own versions, which start at the version it was created from. `tbl@branch(name)` opens the branch under `<table>/tree/<name>/`, optionally with `FOR VERSION AS OF` / `FOR TIME AS OF` inside the branch; the BE receives the branch directory as the dataset URI and needs no change (a branch is a shallow clone with its own `_versions/`). For managed tables Doris sends the branch in the `branch` field of the version APIs. Previously `@tag` / `@branch` were silently ignored and the query read the latest version of `main`. 5. **Errors** – a missing version (never committed, or removed by `cleanup_old_versions`) is reported as `Lance version N of db.tbl[ (tag 'x')] was not found[ in the namespace]` without the storage path; a missing tag or branch, an out-of-range or non-positive version (a signed number is a version, not a tag name), a timestamp before the first version and an unparsable timestamp have their own messages; a namespace that lists no versions is reported as such; a table that is only declared in the namespace fails with "declared in the namespace but has no data yet"; for a managed table, a namespace `location` that is empty or names another place than `table_uri` (a query string aside), or an `s3+ddb` URI, whose DynamoDB commit handler would record and finalize versions on read, fails the query before anything is read; every other failure keeps the sanitized provider message. Design decisions worth calling out: - For a managed table the namespace is the source of truth. A version whose manifest is still in storage but which the namespace does not record is reported as not found, and the table's latest version is the namespace's latest even when storage already holds a newer manifest. Reading unrecorded manifests would expose staged or rolled-back commits. - `FOR TIME AS OF` uses the commit times the manifests record, for storage-versioned and managed tables alike, as Lance's `asof` does; a managed table only selects among the versions its namespace lists. The namespace's `timestamp_millis` is not used: it is optional in the spec and its meaning varies (Lance's Directory namespace reports the manifest object's modification time, second-granular on S3 and reset when a dataset is copied), so the same dataset would otherwise resolve differently depending on how it is catalogued. A removed version takes its commit time with it, which is why history before it is not used. On a managed chain, a version number the namespace no longer records between versions it records is a removed version too (Lance numbers a chain's commits consecutively, and Lance's Directory namespace only accepts the next version), and so is a recorded version whose manifest is not in storage. - For a managed table every read of the latest version (a plain query, the search functions, a branch head) is pinned to the newest version the namespace records, asked for the way Lance's own reader asks (`ListTableVersions`, newest first, one entry); an empty history is an error. Opening the storage latest would expose a version the namespace never published. - The FE and the BE read a managed table the same way, by URI and version number, rather than the FE opening it through the SDK's namespace client (`Dataset.open().namespaceClient(..)`). The BE's lance-c can only open by URI and version, and with the FE in namespace mode the two readers resolved the store, the credentials and the manifest differently; earlier rounds of this PR kept adding code to align them (a wrapper around the namespace client, store identities, reconciliation of a second describe). Reading both sides by URI removes that class of mismatch. The namespace still decides the versions: Doris asks it directly for every managed read, and a managed table's access is never cached, so each read uses the location and options of its own `DescribeTable`, which the BE receives unchanged. The table describe and the version requests run outside the catalog's namespace lock, which keeps only the namespace and table listings: the REST and Directory namespaces are safe to call concurrently, a managed table makes these requests on every read, and Lance's REST client sets no request timeout, so a stalled request must not hold up every other table. - The BE reads a version by number at its canonical manifest, `<table root>[/tree/<branch>]/_versions/<u64::MAX - v>.manifest` (or the V1 `<v>.manifest`). Doris checks that the manifest the namespace records for the selected version is that path, or a staged manifest beside it (`<canonical>-<id>`, which a namespace that records the path Lance's client sends in `CreateTableVersion` keeps recording after the commit is finalized, since Lance's namespace store cannot update a record). Any other path means the namespace names a manifest neither reader would open (a manifest recorded elsewhere, or a table moved between the describe and the version lookup), and the read fails with both paths in the message and a hint to retry. `FOR TIME AS OF` checks every version the namespace records on the chain, since the commit times of those manifests decide the selection. The table root is derived from the location as lance-io derives it. - A version the namespace records only at a staged manifest whose canonical manifest does not exist (a commit that was never finalized, or whose manifest cleanup removed) cannot be read: the query fails and says so, instead of falling back to an older version, and Doris writes nothing on read. This is how Doris's Iceberg and Paimon connectors treat a snapshot their metadata names but they cannot load. Lance's own namespaces finalize a commit inside `CreateTableVersion`, before they list it, so they do not produce this state for committed versions. - `FOR VERSION AS OF` on a dataset whose directory is missing altogether reports "version N was not found", because the SDK probes the version's manifest path first; the provider's own message is kept in the exception cause, handled the same way as every other Lance error. - `org.lance:lance-core` is bumped from 11.0.0 to 12.0.0 (its `lance-namespace` client from 0.7.7 to 0.11.1). The bump is for the namespace client, whose `branch` field on the version requests is what managed-table branches need; `Dataset.checkout(Ref)` already existed. The FE code compiled unchanged and every Lance unit test class and regression suite passes on it. FE packaging replaces the Linux x86_64 JNI library in that JAR with a glibc 2.17 rebuild, so the pins in `docker/thirdparties/lance-jni-helpers.sh` move to the 12.0.0 rebuild published as [lance-jni-12.0.0-glibc2.17-r1](https://github.com/apache/doris-thirdparty/releases/tag/lance-jni-12.0.0-glibc2.17-r1) (built from unmodified Lance v12.0.0 with the procedure of the 11.0.0 asset; its 288 JNI exports match the upstream library, and the Lance v12 Java test suite passes with it on CentOS 7). The BE stays on lance-c 0.1.9 (Lance 11.0.0): the FE only reads metadata and never writes a dataset, so the newer FE release does not change what the BE can read. - On a storage-versioned table, an expired version (removed by cleanup) is indistinguishable from one that never existed, so both get the same message. On a managed table the namespace's record tells them apart: a version it does not record is "not found in the namespace", one it records at its canonical path but storage lacks is "not found", and one recorded only at a staged manifest "cannot be read". - A managed table's branch is read from its own directory (`<table>/tree/<branch>`, joined before any query string as Lance joins it) by URI and version, as the BE reads it, so it needs neither a recorded `main` version nor a readable `main` manifest. Every other selector except an explicit version on `main` and the latest `main` is a checkout from one open of the latest `main` in storage; for a managed table that open only supplies the tag files and the manifest listing, and the version finally read is always one the namespace records, checked as above. The namespace version list is fetched completely, once per read, without a page size: Lance's Directory namespace applies a limit without returning a page token, and neither the order nor monotonic commit times can be relied on to stop early. - Known limitations: a version recorded at a staged manifest is read from its canonical manifest when one exists; a namespace that lets one version number be committed twice (deleting records while the earlier commit's canonical manifest stays in storage) is not detected. While the newest version of a managed table is not finalized, reading the table without `FOR VERSION AS OF` fails, and so does loading its schema. Two references to the same branch in one statement resolve the branch head independently, and the optimizer's row count is that of the latest `main` (both shared with the Iceberg integration). `vector_search()`, `full_text_search()` and index inspection use the latest `main` and reject a version, tag or branch in their `table` argument. Index paths open by URI and are only reachable for filesystem catalogs, whose Directory namespace never manages versions; an assertion guards that. On a storage-versioned table, a branch whose metadata was deleted, or whose creation stopped partway, stays readable through `@branch` until its directory is cleaned up, because the Lance SDK checks a branch out by its directory; on a managed table the namespace decides whether the branch exists (a missing table reported by Lance's Directory namespace for a branch request counts as the missing branch). - A Lance branch is a shallow clone whose manifests record the parent's location as an absolute URI (`Manifest.base_paths`), so a branch is readable only where it was created. The committed `time_travel.lance` branch was therefore created against the fixture's final location `s3://warehouse/lance/time_travel.lance` (`lance_build_time_travel.py --create-branch`) and synced back; the fixture's self-check verifies that reference. Examples (mysql client against a 4.1 FE; `time_travel.lance` has versions 1..3 committed at 13:06:07.597 / 13:06:09.113 / 13:06:10.621 UTC with rows 1..3 / 4..6 / 7..9): ```sql mysql> SET enable_file_scanner_v2 = true; mysql> SET time_zone = 'UTC'; mysql> CREATE CATALOG lance_demo_fs PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "filesystem", -> "warehouse" = "s3://warehouse/lance", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.access_key" = "admin", -> "s3.secret_key" = "password", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT * FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:10'; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT a.row_id, a.tag AS tag_v1, b.tag AS tag_v3 -> FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 a -> JOIN lance_demo_fs.`default`.time_travel FOR VERSION AS OF 3 b ON a.row_id = b.row_id -> ORDER BY a.row_id; +--------+--------+--------+ | row_id | tag_v1 | tag_v3 | +--------+--------+--------+ | 1 | v1 | v1 | | 2 | v1 | v1 | | 3 | v1 | v1 | +--------+--------+--------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 99; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 99 of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:07'; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance table default.time_travel has no version at or before '2026-09-19 13:06:07' ``` A REST Namespace with managed versioning. `time_travel_managed` records versions 1..3, `time_travel_managed_lagging` records only 1..2 while storage already holds version 3, `time_travel_managed_partial` records 1 and 3, and `time_travel_managed_pending` records a fourth version only at a staged manifest that was never finalized: ```sql mysql> CREATE CATALOG lance_demo_rest PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "rest", -> "lance.rest.uri" = "http://127.0.0.1:19102", -> "lance.rest.security.type" = "bearer", -> "lance.rest.bearer-token" = "doris-lance-rest-test-token", -> "lance.namespace.root_database" = "default", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_lagging; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_lagging FOR VERSION AS OF 3; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 3 of default.time_travel_managed_lagging was not found in the namespace mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_partial FOR TIME AS OF '2026-09-19 13:06:10'; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance cannot resolve FOR TIME AS OF '2026-09-19 13:06:10' on default.time_travel_managed_partial: version 2, which may hold the state at that time, no longer exists mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_pending; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 4 of default.time_travel_managed_pending cannot be read: the namespace records it at a staged manifest, and its canonical manifest, which Doris reads, does not exist (its commit was not finalized, or cleanup removed it) mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_pending FOR VERSION AS OF 3; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 9 | 9 | +----------+-------------+ mysql> EXPLAIN SELECT row_id FROM lance_demo_rest.`default`.time_travel_managed_lagging; ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_rest.default.time_travel_managed_lagging lanceCatalogType=rest lanceVersion=2 lanceManagedVersioning=true lanceFragments=2 lanceFragmentGrouping=FRAGMENT ``` Tags and branches (`time_travel.lance` carries tags `v1`..`v3` and a branch `dev` forked from version 2 with one extra row; on `time_travel_managed_partial` the tag `v2` points at the version the namespace no longer records): ```sql mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@tag(v2) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | +--------+------+ mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@branch(dev) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | | 100 | dev | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel@branch(dev) FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@tag(no_such_tag); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance tag 'no_such_tag' of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@branch(nope); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance branch 'nope' of default.time_travel was not found mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@branch(dev); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 7 | 100 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@tag(v1); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 3 | 3 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_partial@tag(v2); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 2 of default.time_travel_managed_partial (tag 'v2') was not found in the namespace mysql> EXPLAIN SELECT row_id FROM lance_demo_fs.`default`.time_travel@branch(dev); ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_fs.default.time_travel lanceCatalogType=filesystem lanceVersion=3 lanceManagedVersioning=false lanceBranch=dev lanceFragments=3 lanceFragmentGrouping=FRAGMENT ``` Test environment changes: the Lance REST fixture (`lance_rest_server.py`) gains `ListTableVersions` / `DescribeTableVersion` for tables listed in `LANCE_REST_MANAGED_TABLES_JSON`; `time_travel.lance` is a committed three-version dataset with tags and a branch, carried over as-is by `lance_build_preinstalled_catalog.py` because the suites hard-code its commit times (`lance_build_time_travel.py` regenerates it and prints the times to update; `--create-branch` forks the branch at the uploaded location). The stub additionally serves the `branch` parameter of the version APIs, and can record a version at a staged manifest (`"staged": true`) or at any path (`"manifest_path"`), which the `_pending`, `_staged` and `_misrecorded` fixture tables use. ### Release note Lance catalogs support `FOR VERSION AS OF`, `FOR TIME AS OF`, `@tag(name)` and `@branch(name)`, including tables whose versions are managed by a REST Namespace (`managed_versioning = true`). Missing versions, tags and branches and invalid selectors now report clear errors. ### Check List (For Author) - Test: Regression test / Unit Test - `external_table_p0/lance/test_lance_time_travel`, `test_lance_rest_time_travel`, `test_lance_rest_catalog` (Lance REST fixture + MinIO) - `LanceManagedVersioningTest` (real SDK dataset behind a REST stub: managed / partial / lagging / untimed / growing / server error / location checks / declared-only / cleaned-up version / tags / branches / paging / empty version list / relocated table / table moved after it was described / version recorded at a staged manifest, finalized or not, on `main` and on a branch, newest or older / finalized manifest outside its canonical path / the BE receiving the options of the read's describe / history gaps / namespace times that differ from manifest times), plus `test_lance_vector_search` and `test_lance_full_text_search` re-run on lance-core 12.0.0, `LanceSnapshotTest` (including sub-millisecond commit times, and a check of time selection against its rule on random histories with microsecond commit times and versions a namespace records but storage lacks), `LanceManifestPathsTest` (canonical and staged paths in both naming schemes, branches, rejected paths, a non-ASCII-digit locale, location paths as lance-io derives them), `LanceTableAccessCacheTest` (including a managed table described on every read), `LanceRestCatalogTest`, `LanceScanNodeTest`, `LanceMetadataLoaderTest`, `LanceCatalogLifecycleTest` - Behavior changed: Yes. Tables with `managed_versioning = true` were rejected and are now readable; error messages for invalid or missing versions changed to the user-facing form above; `@tag` / `@branch` on a Lance table now select the tag or branch (including a tag that points into a branch) instead of silently reading the latest version of `main`; FE dependency `lance-core` 11.0.0 → 12.0.0. - Does this need documentation: Yes (apache/doris-website#4172: feature table, REST catalog caution, new "Time Travel" section with the managed-versioning rules and limitations)
This was referenced Sep 29, 2026
yiguolei
pushed a commit
to apache/doris
that referenced
this pull request
Oct 8, 2026
…ce managed versioning (#68453) ### What problem does this PR solve? Issue Number: close #66491 Related PR: #65730 (baseline Lance read integration, which already carried `FOR VERSION AS OF` / `FOR TIME AS OF` on the FE without tests or documentation) Problem Summary: Lance catalogs could select a dataset version on the FE, but nothing proved it end to end, the error messages leaked SDK internals (`NumberFormatException: For input string`, storage paths and Rust source locations), and tables whose versions are managed by a REST Namespace (`DescribeTable` returns `managed_versioning = true`) were rejected outright with "not supported by the current BE reader". This PR completes the four scope items of the issue: 1. **Time travel** – `FOR VERSION AS OF <n>` and `FOR TIME AS OF '<timestamp>'` are covered by regression suites on a filesystem catalog and a REST catalog, with a committed three-version fixture (`time_travel.lance`) and the existing `multi_frag.lance` (three appends, then a delete) so that reading an older version across a deletion file is exercised on the BE. The semantics follow the Iceberg and Paimon integrations: the selected version is the one committed last at or before the timestamp (of versions committed at the same instant, the newer one), with the timestamp parsed in the session time zone with second or millisecond precision and compared with the full commit time, which Lance records below a millisecond, so a commit later within the requested millisecond is not selected; commit times are compared as recorded, without assuming they grow with version numbers. As Iceberg drops its snapshot log before a removed snapshot, only the versions newer than the most recent one removed by cleanup are candidates, so a timestamp earlier than all of them (before the first version, or in removed history) is an error. 2. **REST Namespace managed versioning** – the namespace decides which versions of such a table exist and which is the newest: the latest version comes from `ListTableVersions`, an explicit version is checked with `DescribeTableVersion`, and `FOR TIME AS OF` selects among the versions the namespace lists. The FE then reads the chosen version the way it reads any Lance dataset, and the way the BE does through lance-c: by the location and storage options of the table's `DescribeTable` and the version number. No BE or thrift change is needed. `EXPLAIN` shows `lanceManagedVersioning=true` for these tables. 3. **Selected version fixed for the whole statement** – schema binding, fragment planning, predicate pushdown, metadata COUNT and the BE scan all use the selected version; two references to the same table in one statement can select different versions (self join / union tests). 4. **Tags and branches** – `tbl@tag(name)` resolves the tag to the branch and version it points at, read from the dataset's `_refs/tags/` for both storage-versioned and managed tables, and only that tag's file is read (Lance's namespace tag APIs expose the same refs; the selected version of a managed table is then resolved through the namespace); a tag created on a branch selects that branch's version, not the same number on `main`. As for Paimon, and for tag names in Iceberg, a non-numeric `FOR VERSION AS OF 'v2'` names a tag; a tag name is never combined with `@branch`, `@branch(main)` included, because the tag already determines its branch. `@branch(main)` is `main`; `@tag` / `@branch` take exactly one name and reject extra parameters. Inside a branch, versions and times select among the branch's own versions, which start at the version it was created from. `tbl@branch(name)` opens the branch under `<table>/tree/<name>/`, optionally with `FOR VERSION AS OF` / `FOR TIME AS OF` inside the branch; the BE receives the branch directory as the dataset URI and needs no change (a branch is a shallow clone with its own `_versions/`). For managed tables Doris sends the branch in the `branch` field of the version APIs. Previously `@tag` / `@branch` were silently ignored and the query read the latest version of `main`. 5. **Errors** – a missing version (never committed, or removed by `cleanup_old_versions`) is reported as `Lance version N of db.tbl[ (tag 'x')] was not found[ in the namespace]` without the storage path; a missing tag or branch, an out-of-range or non-positive version (a signed number is a version, not a tag name), a timestamp before the first version and an unparsable timestamp have their own messages; a namespace that lists no versions is reported as such; a table that is only declared in the namespace fails with "declared in the namespace but has no data yet"; for a managed table, a namespace `location` that is empty or names another place than `table_uri` (a query string aside), or an `s3+ddb` URI, whose DynamoDB commit handler would record and finalize versions on read, fails the query before anything is read; every other failure keeps the sanitized provider message. Design decisions worth calling out: - For a managed table the namespace is the source of truth. A version whose manifest is still in storage but which the namespace does not record is reported as not found, and the table's latest version is the namespace's latest even when storage already holds a newer manifest. Reading unrecorded manifests would expose staged or rolled-back commits. - `FOR TIME AS OF` uses the commit times the manifests record, for storage-versioned and managed tables alike, as Lance's `asof` does; a managed table only selects among the versions its namespace lists. The namespace's `timestamp_millis` is not used: it is optional in the spec and its meaning varies (Lance's Directory namespace reports the manifest object's modification time, second-granular on S3 and reset when a dataset is copied), so the same dataset would otherwise resolve differently depending on how it is catalogued. A removed version takes its commit time with it, which is why history before it is not used. On a managed chain, a version number the namespace no longer records between versions it records is a removed version too (Lance numbers a chain's commits consecutively, and Lance's Directory namespace only accepts the next version), and so is a recorded version whose manifest is not in storage. - For a managed table every read of the latest version (a plain query, the search functions, a branch head) is pinned to the newest version the namespace records, asked for the way Lance's own reader asks (`ListTableVersions`, newest first, one entry); an empty history is an error. Opening the storage latest would expose a version the namespace never published. - The FE and the BE read a managed table the same way, by URI and version number, rather than the FE opening it through the SDK's namespace client (`Dataset.open().namespaceClient(..)`). The BE's lance-c can only open by URI and version, and with the FE in namespace mode the two readers resolved the store, the credentials and the manifest differently; earlier rounds of this PR kept adding code to align them (a wrapper around the namespace client, store identities, reconciliation of a second describe). Reading both sides by URI removes that class of mismatch. The namespace still decides the versions: Doris asks it directly for every managed read, and a managed table's access is never cached, so each read uses the location and options of its own `DescribeTable`, which the BE receives unchanged. The table describe and the version requests run outside the catalog's namespace lock, which keeps only the namespace and table listings: the REST and Directory namespaces are safe to call concurrently, a managed table makes these requests on every read, and Lance's REST client sets no request timeout, so a stalled request must not hold up every other table. - The BE reads a version by number at its canonical manifest, `<table root>[/tree/<branch>]/_versions/<u64::MAX - v>.manifest` (or the V1 `<v>.manifest`). Doris checks that the manifest the namespace records for the selected version is that path, or a staged manifest beside it (`<canonical>-<id>`, which a namespace that records the path Lance's client sends in `CreateTableVersion` keeps recording after the commit is finalized, since Lance's namespace store cannot update a record). Any other path means the namespace names a manifest neither reader would open (a manifest recorded elsewhere, or a table moved between the describe and the version lookup), and the read fails with both paths in the message and a hint to retry. `FOR TIME AS OF` checks every version the namespace records on the chain, since the commit times of those manifests decide the selection. The table root is derived from the location as lance-io derives it. - A version the namespace records only at a staged manifest whose canonical manifest does not exist (a commit that was never finalized, or whose manifest cleanup removed) cannot be read: the query fails and says so, instead of falling back to an older version, and Doris writes nothing on read. This is how Doris's Iceberg and Paimon connectors treat a snapshot their metadata names but they cannot load. Lance's own namespaces finalize a commit inside `CreateTableVersion`, before they list it, so they do not produce this state for committed versions. - `FOR VERSION AS OF` on a dataset whose directory is missing altogether reports "version N was not found", because the SDK probes the version's manifest path first; the provider's own message is kept in the exception cause, handled the same way as every other Lance error. - `org.lance:lance-core` is bumped from 11.0.0 to 12.0.0 (its `lance-namespace` client from 0.7.7 to 0.11.1). The bump is for the namespace client, whose `branch` field on the version requests is what managed-table branches need; `Dataset.checkout(Ref)` already existed. The FE code compiled unchanged and every Lance unit test class and regression suite passes on it. FE packaging replaces the Linux x86_64 JNI library in that JAR with a glibc 2.17 rebuild, so the pins in `docker/thirdparties/lance-jni-helpers.sh` move to the 12.0.0 rebuild published as [lance-jni-12.0.0-glibc2.17-r1](https://github.com/apache/doris-thirdparty/releases/tag/lance-jni-12.0.0-glibc2.17-r1) (built from unmodified Lance v12.0.0 with the procedure of the 11.0.0 asset; its 288 JNI exports match the upstream library, and the Lance v12 Java test suite passes with it on CentOS 7). The BE stays on lance-c 0.1.9 (Lance 11.0.0): the FE only reads metadata and never writes a dataset, so the newer FE release does not change what the BE can read. - On a storage-versioned table, an expired version (removed by cleanup) is indistinguishable from one that never existed, so both get the same message. On a managed table the namespace's record tells them apart: a version it does not record is "not found in the namespace", one it records at its canonical path but storage lacks is "not found", and one recorded only at a staged manifest "cannot be read". - A managed table's branch is read from its own directory (`<table>/tree/<branch>`, joined before any query string as Lance joins it) by URI and version, as the BE reads it, so it needs neither a recorded `main` version nor a readable `main` manifest. Every other selector except an explicit version on `main` and the latest `main` is a checkout from one open of the latest `main` in storage; for a managed table that open only supplies the tag files and the manifest listing, and the version finally read is always one the namespace records, checked as above. The namespace version list is fetched completely, once per read, without a page size: Lance's Directory namespace applies a limit without returning a page token, and neither the order nor monotonic commit times can be relied on to stop early. - Known limitations: a version recorded at a staged manifest is read from its canonical manifest when one exists; a namespace that lets one version number be committed twice (deleting records while the earlier commit's canonical manifest stays in storage) is not detected. While the newest version of a managed table is not finalized, reading the table without `FOR VERSION AS OF` fails, and so does loading its schema. Two references to the same branch in one statement resolve the branch head independently, and the optimizer's row count is that of the latest `main` (both shared with the Iceberg integration). `vector_search()`, `full_text_search()` and index inspection use the latest `main` and reject a version, tag or branch in their `table` argument. Index paths open by URI and are only reachable for filesystem catalogs, whose Directory namespace never manages versions; an assertion guards that. On a storage-versioned table, a branch whose metadata was deleted, or whose creation stopped partway, stays readable through `@branch` until its directory is cleaned up, because the Lance SDK checks a branch out by its directory; on a managed table the namespace decides whether the branch exists (a missing table reported by Lance's Directory namespace for a branch request counts as the missing branch). - A Lance branch is a shallow clone whose manifests record the parent's location as an absolute URI (`Manifest.base_paths`), so a branch is readable only where it was created. The committed `time_travel.lance` branch was therefore created against the fixture's final location `s3://warehouse/lance/time_travel.lance` (`lance_build_time_travel.py --create-branch`) and synced back; the fixture's self-check verifies that reference. Examples (mysql client against a 4.1 FE; `time_travel.lance` has versions 1..3 committed at 13:06:07.597 / 13:06:09.113 / 13:06:10.621 UTC with rows 1..3 / 4..6 / 7..9): ```sql mysql> SET enable_file_scanner_v2 = true; mysql> SET time_zone = 'UTC'; mysql> CREATE CATALOG lance_demo_fs PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "filesystem", -> "warehouse" = "s3://warehouse/lance", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.access_key" = "admin", -> "s3.secret_key" = "password", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT * FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:10'; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT a.row_id, a.tag AS tag_v1, b.tag AS tag_v3 -> FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 a -> JOIN lance_demo_fs.`default`.time_travel FOR VERSION AS OF 3 b ON a.row_id = b.row_id -> ORDER BY a.row_id; +--------+--------+--------+ | row_id | tag_v1 | tag_v3 | +--------+--------+--------+ | 1 | v1 | v1 | | 2 | v1 | v1 | | 3 | v1 | v1 | +--------+--------+--------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 99; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 99 of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:07'; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance table default.time_travel has no version at or before '2026-09-19 13:06:07' ``` A REST Namespace with managed versioning. `time_travel_managed` records versions 1..3, `time_travel_managed_lagging` records only 1..2 while storage already holds version 3, `time_travel_managed_partial` records 1 and 3, and `time_travel_managed_pending` records a fourth version only at a staged manifest that was never finalized: ```sql mysql> CREATE CATALOG lance_demo_rest PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "rest", -> "lance.rest.uri" = "http://127.0.0.1:19102", -> "lance.rest.security.type" = "bearer", -> "lance.rest.bearer-token" = "doris-lance-rest-test-token", -> "lance.namespace.root_database" = "default", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_lagging; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_lagging FOR VERSION AS OF 3; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 3 of default.time_travel_managed_lagging was not found in the namespace mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_partial FOR TIME AS OF '2026-09-19 13:06:10'; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance cannot resolve FOR TIME AS OF '2026-09-19 13:06:10' on default.time_travel_managed_partial: version 2, which may hold the state at that time, no longer exists mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_pending; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 4 of default.time_travel_managed_pending cannot be read: the namespace records it at a staged manifest, and its canonical manifest, which Doris reads, does not exist (its commit was not finalized, or cleanup removed it) mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_pending FOR VERSION AS OF 3; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 9 | 9 | +----------+-------------+ mysql> EXPLAIN SELECT row_id FROM lance_demo_rest.`default`.time_travel_managed_lagging; ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_rest.default.time_travel_managed_lagging lanceCatalogType=rest lanceVersion=2 lanceManagedVersioning=true lanceFragments=2 lanceFragmentGrouping=FRAGMENT ``` Tags and branches (`time_travel.lance` carries tags `v1`..`v3` and a branch `dev` forked from version 2 with one extra row; on `time_travel_managed_partial` the tag `v2` points at the version the namespace no longer records): ```sql mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@tag(v2) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | +--------+------+ mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@branch(dev) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | | 100 | dev | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel@branch(dev) FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@tag(no_such_tag); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance tag 'no_such_tag' of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@branch(nope); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance branch 'nope' of default.time_travel was not found mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@branch(dev); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 7 | 100 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@tag(v1); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 3 | 3 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_partial@tag(v2); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 2 of default.time_travel_managed_partial (tag 'v2') was not found in the namespace mysql> EXPLAIN SELECT row_id FROM lance_demo_fs.`default`.time_travel@branch(dev); ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_fs.default.time_travel lanceCatalogType=filesystem lanceVersion=3 lanceManagedVersioning=false lanceBranch=dev lanceFragments=3 lanceFragmentGrouping=FRAGMENT ``` Test environment changes: the Lance REST fixture (`lance_rest_server.py`) gains `ListTableVersions` / `DescribeTableVersion` for tables listed in `LANCE_REST_MANAGED_TABLES_JSON`; `time_travel.lance` is a committed three-version dataset with tags and a branch, carried over as-is by `lance_build_preinstalled_catalog.py` because the suites hard-code its commit times (`lance_build_time_travel.py` regenerates it and prints the times to update; `--create-branch` forks the branch at the uploaded location). The stub additionally serves the `branch` parameter of the version APIs, and can record a version at a staged manifest (`"staged": true`) or at any path (`"manifest_path"`), which the `_pending`, `_staged` and `_misrecorded` fixture tables use. ### Release note Lance catalogs support `FOR VERSION AS OF`, `FOR TIME AS OF`, `@tag(name)` and `@branch(name)`, including tables whose versions are managed by a REST Namespace (`managed_versioning = true`). Missing versions, tags and branches and invalid selectors now report clear errors. ### Check List (For Author) - Test: Regression test / Unit Test - `external_table_p0/lance/test_lance_time_travel`, `test_lance_rest_time_travel`, `test_lance_rest_catalog` (Lance REST fixture + MinIO) - `LanceManagedVersioningTest` (real SDK dataset behind a REST stub: managed / partial / lagging / untimed / growing / server error / location checks / declared-only / cleaned-up version / tags / branches / paging / empty version list / relocated table / table moved after it was described / version recorded at a staged manifest, finalized or not, on `main` and on a branch, newest or older / finalized manifest outside its canonical path / the BE receiving the options of the read's describe / history gaps / namespace times that differ from manifest times), plus `test_lance_vector_search` and `test_lance_full_text_search` re-run on lance-core 12.0.0, `LanceSnapshotTest` (including sub-millisecond commit times, and a check of time selection against its rule on random histories with microsecond commit times and versions a namespace records but storage lacks), `LanceManifestPathsTest` (canonical and staged paths in both naming schemes, branches, rejected paths, a non-ASCII-digit locale, location paths as lance-io derives them), `LanceTableAccessCacheTest` (including a managed table described on every read), `LanceRestCatalogTest`, `LanceScanNodeTest`, `LanceMetadataLoaderTest`, `LanceCatalogLifecycleTest` - Behavior changed: Yes. Tables with `managed_versioning = true` were rejected and are now readable; error messages for invalid or missing versions changed to the user-facing form above; `@tag` / `@branch` on a Lance table now select the tag or branch (including a tag that points into a branch) instead of silently reading the latest version of `main`; FE dependency `lance-core` 11.0.0 → 12.0.0. - Does this need documentation: Yes (apache/doris-website#4172: feature table, REST catalog caution, new "Time Travel" section with the managed-versioning rules and limitations)
3 of 4 tasks
morningman
pushed a commit
that referenced
this pull request
Oct 16, 2026
Document the physical `lance_index_entries` table-valued function for Lance Filesystem Catalog tables added in apache/doris#66671. This closes the documentation item deferred in that PR and tracked in apache/doris#66497 ("Show index names, UUIDs, columns, types, and dataset versions ... user documentation for the physical `lance_index_entries` TVF remains outstanding"). It also resolves the forward reference to `lance_index_entries()` already present in the Lance time-travel limitations (#4172). ## What Changed - `lakehouse/catalogs/lance-catalog.mdx` (EN + zh-CN) - Feature Overview: metadata access now lists the `lance_index_entries` TVF alongside `SHOW INDEX` (both Filesystem Catalogs only); the system-tables row points users to both surfaces for index metadata. - New "List Physical Index Entries with lance_index_entries" subsection under "Inspect Lance Indexes": why the physical view exists (UUID and dataset version do not fit the fixed MySQL-style 13-column `SHOW INDEX` schema), the six fixed columns, the dataset-version semantics recorded on each entry, read/validation semantics (all-or-error, 16,384 raw-entry cap, reserved system indexes validated but not displayed, ordered by `(IndexName, IndexUuid)`), ordinary-predicate filtering, the `SHOW` privilege requirement, and the fixed `lance_index_entries is not supported for Lance REST catalogs` error. - `sql-manual/sql-functions/table-valued-functions/lance-index-entries.md` (EN + zh-CN, new) - Standalone TVF reference: description, syntax, required `table` parameter (fully qualified three-part name, backtick quoting), the fixed return columns, and examples including a predicate filter. Linked back to the Lance Catalog page. - `versioned_sidebars/version-4.x-sidebars.json` - Registered the new page in the table-valued-functions list (alphabetical, between `jobs` and `local`). `docs/` (dev) is intentionally untouched: the Lance catalog documentation is not present in dev because apache/doris master has no Lance code (the feature lives on branch-4.1), matching #4090. ## Verified versions and validation Lance Catalog is available from 4.1.4; `lance_index_entries` is available from 4.1.5. Verified against `branch-4.1`, the 4.1.4 and 4.1.5-release source tags, the TVF argument/authorization implementation, physical-entry validation and the existing Lance regression suite. Master has no Lance catalog implementation, so current docs are intentionally not added. Rebased onto current website master. Both languages include version markers, prerequisites, parameter/empty-result behavior and expected example results. `git diff --check` and sidebar JSON validation pass. No source build or live cluster execution was performed in this audit. ## Versions - [ ] dev - [x] 4.x ## Languages - [x] Chinese - [x] English --------- Co-authored-by: u70b3 <u70b3@users.noreply.github.com> Co-authored-by: morningman <moringman@apache.org>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents Lance time travel, tags, branches and REST Namespace managed versioning, added by apache/doris#68453 (issue apache/doris#66491).
Changes to
lakehouse/catalogs/lance-catalog.mdx(4.x, English and Chinese):FOR VERSION AS OF,FOR TIME AS OF,@tag(...),@branch(...)), including tables whose versions are managed by a REST Namespace.org.lance:lance-core:12.0.0; the BE reader is unchanged.Versions
Languages
Docs Checklist