[Feature](lance) Support additional Arrow and Lance data types by wangzhaobo957-cloud · Pull Request #67325 · apache/doris

wangzhaobo957-cloud · 2026-08-31T03:14:44Z

What problem does this PR solve?

Issue Number: close #66496

Problem Summary:

The Lance reader previously reported several Arrow and Lance-specific types as UNSUPPORTED, preventing Doris from correctly discovering schemas or reading these columns.

This PR adds Doris-side support for:

Arrow Null as Doris NULL
Arrow Duration as Doris BIGINT
Arrow and Lance JSON extensions as Doris JSON
Lance BFloat16 as Doris FLOAT

Implementation details:

FE recognizes the supported Arrow and Lance extension metadata and validates each extension's physical storage type.
BE maps the same logical types during schema discovery and scan execution.
BFloat16 values are widened to Float32 without precision loss, including values nested in arrays and other complex types.
Ordinary nested Arrow columns bypass reconstruction when no BFloat16 or registered extension array requires normalization.
Only top-level Arrow Null fields are supported. Nested Null fields remain unsupported because the current complex-type deserialization path cannot handle them safely.

Release note

Add Doris support for Arrow Null and Duration types, and Lance JSON and BFloat16 extensions.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test

Coverage includes:

FE type-conversion tests for Null, Duration, JSON, and BFloat16.
BE schema and value tests, including nested BFloat16 and the nested-column no-op normalization path.
test_lance_catalog_all_types
test_lance_s3_tvf

Validation:

Production-source diff checks passed.
A complete local FE test run is blocked because the standalone fe-core build cannot resolve Doris internal SNAPSHOT dependencies.
A complete local BE test run is blocked by the stale macOS build directory and third-party headers; CI will run the full suite.
Behavior changed:
- No.
- Yes. Supported Lance schemas are mapped to Doris types instead of UNSUPPORTED.
Does this need documentation?
- No.
- Yes. A follow-up documentation PR will be submitted separately.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-08-31T03:14:49Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zhangstar333 · 2026-09-01T06:33:52Z

/review

github-actions

Requesting changes for two issues:

Correctness / external compatibility: local local(..., format="lance") schema discovery imports the logical Blob v2 dataset schema, but the new BE validator accepts only the five-field scan descriptor, so a valid Blob dataset is rejected before planning.
Performance: BFloat normalization recursively traverses and allocates Arrow wrapper metadata for every ordinary nested column on every scanner and take_rows batch, even when the subtree contains no BFloat16 extension.

Critical checkpoints:

Type and reader correctness: reviewed top-level Null, raw-count Duration, Arrow/Lance JSON, nested BFloat16, Blob descriptor materialization, catalog/S3 scans, and vector take_rows; no other distinct correctness defect remains. The local-TVF Blob boundary above is blocking.
Lifecycle and malformed input: reviewed schema projection, nulls/slices, whole-STRUCT versus subcolumn lazy fetch, row-ID ordering/count, EOF/error cleanup, and invalid extension layouts; no separate lifecycle or malformed-input issue remains.
Performance and observability: the per-batch no-op normalization is blocking; no separate I/O amplification or profile-counter regression was found.
Tests: the current BE/unit and catalog/S3 coverage does not exercise local BE schema discovery with Blob v2 or keep the ordinary nested no-extension batch path allocation-free. The inline comments request focused coverage. Tests were not run in this review-only environment.
Documentation / release note: the live PR description says Blob v2 maps to VARBINARY and reads a materialized binary payload, while the patch and expected outputs expose STRUCT<kind,position,size,blob_id,blob_uri> and explicitly never materialize the payload. Please align the description/release note before merge.
User focus: no additional focus point was supplied.
Completion: complete and converged after three review rounds on head 5807f583490e9a2fc28f3a5b2e7b11497bfda3a9; all candidates were accepted, deduplicated, or dismissed with evidence.

github-actions · 2026-09-01T07:10:27Z

+        };
+        // lance-c returns the Blob v2 descriptor as this fixed 5-field struct (Lance
+        // BLOB_V2_DESC_FIELDS): where a Blob lives and how large it is, never its bytes.
+        const bool descriptor_layout = fields.size() == 5 &&


[P1] Keep local TVF schema discovery compatible with logical Blob v2 schemas. fetch_schema() imports lance_dataset_schema() directly, and the pinned lance-c v0.1.7 implementation returns the dataset's logical Blob schema there (struct<data,uri> or the ranged four-field form), not the five-field scan descriptor. Both valid layouts therefore reach this branch and fail fields.size() == 5, so local(..., format="lance") reports the Blob column as unsupported before a query can run; the new catalog/S3 tests use FE schema discovery and do not cover this path. Please map the logical layouts to the advertised descriptor during dataset-schema conversion (while keeping the strict check for scan output), and add a local Lance TVF schema/query regression with Blob v2.

github-actions · 2026-09-01T07:10:27Z

+                field->name(), child_fields.size(), child_data.size());
+    }
+
+    auto normalized_data = storage_array->data()->Copy();


[P2] Keep no-op nested normalization out of the batch hot path. _fill_block_from_record_batch() calls this routine for every projected column, and for each ordinary LIST/MAP/STRUCT node with no BFloat16 descendant we still allocate an ArrayData copy and field vector, wrap every child with MakeArray(), and build replacement Fields before changed remains false and all of that work is discarded. This regresses every existing nested Lance scan once per record batch, proportional to schema width/depth, even though the new conversion is needed only for extension-bearing subtrees. Please precompute/gate which fields need normalization, or defer ancestor allocations until a child actually changes, and cover the no-extension path with a wide/deep nested benchmark or allocation-sensitive test.

wangzhaobo957-cloud · 2026-09-01T08:23:00Z

/review

zhangstar333 · 2026-09-04T09:23:00Z

/review

github-actions

Requesting changes for one performance issue: nested BFloat16 normalization can widen logically unused values retained by a sliced offset container on stable-row-ID Lance scans with a pushed LIMIT.

Critical checkpoints:

Type and reader correctness: reviewed top-level Null, all Duration units, Arrow/Lance JSON, registered extensions, exact BFloat16 bits, nested container semantics, sequential scans, and row-ID take; no other distinct correctness issue remains.
Lifecycle and malformed input: reviewed schema RPC serialization, nullable defaults, unsupported and malformed extensions, projection/missing-column checks, EOF/error cleanup, and C Data ownership; no separate lifecycle issue remains.
Performance and observability: the inline P2 is blocking. The older ordinary nested no-op allocation issue is fixed on this head and was not duplicated. No separate I/O or profile-counter regression was found.
Tests: current BE, FE, and catalog/S3 coverage exercises the main conversions, but it lacks the stable-row-ID limited slice that exposes this amplification. Tests were not run in this review-only environment.
Documentation / release note: the current description matches the supported Null, Duration, JSON, and BFloat16 scope; no separate documentation blocker was found.
User focus: no additional focus point was supplied.
Completion: complete and converged after two review rounds on head d17b403; all candidates were accepted, deduplicated, or dismissed with evidence.

github-actions · 2026-09-04T10:05:41Z

+        if (!child_requires_normalization) {
+            continue;
+        }
+        auto child_array = arrow::MakeArray(child_data[child_idx]);


[P2] Normalize only the visible child range of sliced offset containers. For a stable-row-ID Lance dataset, an unfiltered pushed LIMIT cannot become a pre-scan range, so pinned Lance keeps GlobalLimitExec over the full projection and DataFusion returns a RecordBatch slice. Arrow List/LargeList/Map slices retain the complete values/entries child, and this line recurses into that unsliced child; convert_bfloat16_array then widens every backing value. LIMIT 1 can therefore allocate Float32 storage for a whole batch's nested BFloat16 values per fragment (about 267 MiB for an 8160 x 8192 child), even though only one row is visible. Please normalize only the child interval covered by the visible parent offsets and rebase the parent, with a stable-row-ID LIMIT or explicitly sliced offset-container regression.

wangzhaobo957-cloud · 2026-09-06T08:35:14Z

/review

zhangstar333 · 2026-09-07T02:39:05Z

run buildall

zhangstar333 · 2026-09-07T02:40:31Z

hi @wangzhaobo957-cloud
need another PR in doris-website, update the lance catalog type support in. Thanks
https://github.com/apache/doris-website/blob/master/versioned_docs/version-4.x/lakehouse/catalogs/lance-catalog.mdx

hello-stephen · 2026-09-07T04:31:24Z

FE UT Coverage Report

Increment line coverage 66.67% (14/21) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-07T06:54:51Z

BE UT Coverage Report

Increment line coverage 77.22% (305/395) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.61% (25831/43333)
Line Coverage	44.22% (265457/600307)
Region Coverage	40.17% (210918/525000)
Branch Coverage	41.62% (97293/233761)

Gabriel39 · 2026-09-10T05:30:58Z

run buildall

Gabriel39 · 2026-09-10T05:31:08Z

/review

github-actions

Requesting changes for three distinct P2 correctness and rolling-upgrade issues on head e0bfe8668a4d8c29cb7859d03064cc75fb84171b:

Sliced scalar Duration/arrow.json arrays can reach Doris with a nonzero Arrow offset and then fail validation or materialize preceding rows.
Rebased sliced Map/Struct parents retain nonzero offsets on ordinary siblings next to BFloat16 children, with the same reject-or-wrong-row outcome.
Shared-storage local Lance TVF schema discovery is not tied to execution capability, so a supported mixed-BE rolling upgrade can discover the new types on a new BE and execute them on an old BE.

Critical checkpoints: same-version FE/BE mappings and pinned BFloat16, Duration, and logical JSON semantics are otherwise consistent; the raw lance.json scan path is converted to logical arrow.json by pinned Lance; residual schema walking and hidden-child widening are already covered by existing threads and were duplicate-fenced. The added tests are homogeneous and offset-zero, or use Arrow's offset-aware accessors, so they do not cover these failures. No additional user focus was supplied. Per the review prompt, I did not run builds or tests.

github-actions · 2026-09-10T06:33:55Z

                DORIS_CHECK(col_type != nullptr);
                PTypeDesc* type_desc = result->add_column_types();
-                if (col_type->get_primitive_type() == INVALID_TYPE) {
+                if (col_type->is_null_literal()) {


[P2] Keep schema discovery and execution on the same BE capability set. During Doris's documented BE-first rolling upgrade, local(..., "shared_storage"="true", "format"="lance") picks a random backendIdForRequest for this RPC, but TVFScanNode leaves execution on every BE when the public backendId is -1. A new BE can therefore expose these ordinary NULL_TYPE/FLOAT/JSONB/BIGINT nodes to an old FE, which generically accepts them, and the one Lance split can then run on an old BE. BFloat16 is a concrete failure: the old reader sends FixedSizeBinary(2) directly to the FLOAT SerDe, which rejects the two-byte buffer under default Arrow validation. Please pin execution to the schema BE or carry/filter an explicit BE capability until all eligible BEs support the new materialization, and add a mixed-BE route test.

github-actions · 2026-09-10T06:33:55Z

+        RETURN_IF_ERROR(rebase_lance_parent_data(array, &rebased_data));
+        for (int child_idx = 0; child_idx < static_cast<int>(struct_array->fields().size());
+             ++child_idx) {
+            rebased_data->child_data[child_idx] = struct_array->field(child_idx)->data();


[P2] Rebase every child when resetting a sliced Struct parent to offset zero. Arrow 24 StructArray::field() applies the parent's slice, so in the changed sliced-Map case the ordinary String key installed here still has offset 2; only the BFloat16 item is rebuilt at offset zero. DataTypeMapSerDe then asks the key SerDe for [0,3), which fails the default zero-offset validation (or reads k0/k1/k2 rather than k2/k3/k4 with validation disabled). The test's direct GetString() assertions are offset-aware and therefore miss the Doris failure. Please compact/rebase unchanged siblings too, and materialize the mixed Map/Struct through _fill_block_from_record_batch.

github-actions · 2026-09-10T06:33:56Z

+    const auto& child_fields = storage_type->fields();
+    const auto& child_data = storage_array->data()->child_data;
+    if (child_fields.empty()) {
+        *normalized = std::move(storage_array);


[P2] Rebase sliced scalar storage before handing it to Doris. Vector pagination forwards a nonzero offset to Lance/DataFusion, whose limit stream can return a sliced RecordBatch; for an already-present projected/refine Duration or arrow.json column, this leaf branch returns the storage unchanged while _fill_block_from_record_batch() always reads [0,row_count). With default Arrow validation, the Nullable/Number/JSON SerDes reject the nonzero offset; without it, Duration and 32-bit JSON storage index buffers from row zero and return preceding values. This is distinct from the existing hidden-child BFloat16 allocation thread. Please add a sliced Duration/JSON materialization test with leading sentinels and either compact here or make the SerDes offset-aware.

hello-stephen · 2026-09-10T10:24:37Z

BE Regression && UT Coverage Report

Increment line coverage 76.68% (296/386) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.40% (31417/42226)
Line Coverage	58.63% (349940/596872)
Region Coverage	55.26% (291541/527579)
Branch Coverage	56.15% (131631/234410)

Gabriel39 · 2026-09-11T02:36:13Z

run buildall

Gabriel39 · 2026-09-11T02:36:19Z

/review

github-actions

Requesting changes for three additional P2 correctness, compatibility, and availability issues on exact head 90d2b22b022b14067315c5eb425ad2bf23f2ceee:

A sliced nested container containing registered arrow.json fails in Arrow builder construction before the child extension can be unwrapped.
Catalog/S3 Lance can schedule the newly advertised JSON/BFloat16 types on a smooth-upgrade source BE whose reader predates their materialization support.
Shared-storage local Lance can successfully discover schema on a backend that the new exact execution policy subsequently rejects.

Critical checkpoints:

Type and reader correctness: reviewed top-level and nested Null, all Duration units, Arrow/Lance JSON, registered extensions, exact BFloat16 conversion, scalar and nested slices, malformed storage, and Doris SerDe contracts. The first inline P2 is the remaining distinct reader failure; other slice defects are already covered by existing threads.
Lifecycle, configuration, and routing: reviewed schema RPC ownership, scanner/batch cleanup, local versus catalog discovery, exact-ID filtering, compute/mix roles, query/load/decommission/blacklist state, compute groups, and scan-range assignment. The third inline P2 is blocking; the repeated exact-policy initialization only overcounts an internal node field on the current non-batch path and was dismissed with evidence.
Compatibility: current-version FE/BE type contracts agree, but the catalog/S3 smooth-upgrade-source path remains ungated as described by the second inline P2. This is distinct from the existing local-TVF mixed-BE thread.
Performance and observability: hidden-child and no-op normalization costs are already fenced by existing inline threads; no additional distinct I/O, allocation, or profile-counter issue survived the final sweep.
Tests: current tests use metadata-only or homogeneous current-version shapes and do not cover production C Data nested-extension rehydration, a smooth-source BE, or composed schema/execution eligibility. Per the review contract, no builds or tests were run.
Documentation / release note: no separate documentation blocker was substantiated in this pass.
User focus: no additional review focus was supplied.
Completion: complete and converged after two review rounds; every candidate was accepted, duplicate-fenced, or dismissed with concrete evidence.

github-actions · 2026-09-11T03:14:32Z

+        return Status::InvalidArgument("validate sliced Lance array failed: {}",
+                                       validation.message());
+    }
+    auto builder_result = arrow::MakeBuilder(array->type(), arrow::default_memory_pool());


[P2] Normalize registered extension children before compacting this parent. Arrow 24's C Data importer rehydrates the built-in arrow.json type recursively, so a sliced List/Struct/Map can still have an ExtensionType child here. MakeBuilder(array->type()) recursively asks for child builders, and Arrow returns NotImplemented for extension types; the scan therefore fails before the later child-unwrapping loop runs. Please recurse/unwrap before parent compaction (or build from storage-typed children) and add a sliced nested arrow.json import/materialization regression.

github-actions · 2026-09-11T03:14:32Z

+        if (field.getDictionary() != null) {
            return Type.UNSUPPORTED;
        }
+        if (extensionName != null && !extensionName.isEmpty()) {


[P2] Gate catalog/S3 execution on BE support for these new encodings. This converter can now advertise JSONB/FLOAT from FE metadata, but LanceScanNode still schedules through the ordinary backend policy, including a smooth-upgrade source BE running the previous reader. BFloat16 then reaches that reader as FixedSizeBinary(2) and is rejected by the FLOAT SerDe (JSON similarly lacks the new normalization). The local-TVF pin does not cover this FE-discovery path; please filter/reject old source BEs when these types are projected and add a mixed-version catalog regression.

github-actions · 2026-09-11T03:14:32Z

                .assignExpectBeNum(Config.min_backend_num_for_external_table)
-                .addPreLocations(preLocations);
+                .addPreLocations(preLocations)
+                .addRequiredBackendIds(requiredBackendIds);


[P2] Select the schema BE from the same eligibility set used here. Shared-storage local Lance currently picks backendIdForRequest from all alive nodes, but this standard policy additionally rejects compute-only nodes when prefer_compute_node_for_external_table=false (the default), plus query/load-disabled, decommissioning, blacklisted, or out-of-group nodes. If one of those served schema discovery, the required-ID filter leaves no candidates and planning fails with No available backends. Please align schema selection with execution eligibility (or make the exact-ID role semantics explicit while retaining real availability checks) and cover a compute-only/ineligible selected BE.

wangzhaobo957-cloud · 2026-09-11T03:41:04Z

/review

Gabriel39 · 2026-09-13T04:01:27Z

run buildall

Gabriel39 · 2026-09-13T04:01:32Z

/review

github-actions

Static review at exact head b5f9e1f2ec3430bde7849fff53ce0ff29eca6925. Per the review contract, I did not run builds or tests.

Requesting changes for one P2 resource-accounting defect: the new Lance normalization copies allocate full Arrow buffers outside Doris's query MemTracker, so sufficiently wide BFloat16 or sliced nested batches can exceed the query memory limit without failing or cancelling at the allocation. The inline comment identifies both builder paths.

Critical checkpoint conclusions:

Architecture/layering: FE schema discovery and BE normalization responsibilities are consistent; local Lance pins schema and execution to the same eligible BE, and catalog/S3 compatibility checks cover the intended routes.
Reader lifecycle and Arrow layout: registered extensions, visible sliced ranges, nested physical-type reconstruction, and Null/Duration/JSON/BFloat16 materialization were traced through the Doris SerDes. No unique issue remains beyond existing threads and the memory-accounting finding below.
Schema/type contract: FE and BE agree on top-level Null, signed Duration counts in each declared unit, JSON storage validation, recursive BFloat16 mapping, and rejection of nested Null or malformed/unknown extensions.
Compatibility: remaining mixed-version and schema-BE concerns are already covered by existing inline threads, so they are not duplicated here.
Performance/resource accounting: existing no-op and hidden-backing-copy concerns are already threaded; the distinct new blocker is that even necessary visible-only normalization buffers bypass query tracking.
Tests: the added BE, FE, and regression coverage exercises values, slices, routing, and session cleanup, but it does not verify normalization under a low query memory limit. No test result was independently produced in this review.
User focus: no additional review focus was provided.

github-actions · 2026-09-13T04:39:26Z

+        check_arrow_fixed_width_buffer(*fixed_binary, sizeof(uint16_t));
+    }
+
+    arrow::FloatBuilder builder;


[P2] Charge these normalization buffers to the query memory tracker. This parameter-free FloatBuilder and compact_lance_array()'s MakeBuilder(..., arrow::default_memory_pool()) both materialize a complete Arrow array through Arrow's system pool rather than Doris's ArrowMemoryPool/Allocator path. A wide BFloat16 vector or compacted nested slice can therefore hold an extra batch-sized buffer outside the query's MemTracker while the destination Doris column is allocated, allowing the query to exceed its memory limit instead of failing or cancelling at this allocation. Please thread a Doris-tracked Arrow pool through both builder paths (with a test-safe fallback or injection) and cover the behavior under a low query memory limit.

hello-stephen · 2026-09-13T05:44:16Z

FE UT Coverage Report

Increment line coverage 58.51% (55/94) 🎉
Increment coverage report
Complete coverage report

…66496

Gabriel39 · 2026-09-13T10:57:30Z

run buildall

Gabriel39 · 2026-09-13T10:57:35Z

/review

github-actions

Static review completed on exact head 598b0818f6b58b6fb2a5096e06aca1cec9aab27d against base d6fdff126d0edbee79ae765de0b788cd58328aa1. No new actionable finding remains, so this is a summary-only comment review with no new inline comments. Existing inline threads were treated as hard duplicate fences.

Critical checkpoints:

Type and reader correctness: traced top-level Arrow Null, every Duration unit, Arrow/Lance JSON, registered extensions, recursive BFloat16 conversion, scalar and nested slices, malformed storage, and Doris SerDe materialization. The current patch unwraps registered extensions before compaction, copies visible sliced ranges to offset zero, preserves BFloat16 bits/nulls, rebuilds parent types consistently, and uses the Doris-tracked Arrow pool for materializing normalization allocations. No distinct reader defect survived the final sweep.
Lifecycle, architecture, and performance: reviewed both Lance RecordBatch producers, schema import, projection, multi-batch append/lifetime behavior, error propagation, and the no-op versus materializing normalization paths. No new lifecycle, ownership, I/O-amplification, or resource-accounting issue remains beyond the existing inline threads.
FE/BE contract and compatibility: FE and BE agree on the new logical mappings and storage validation. Catalog/S3 checks cover materialized tuple slots and selected smooth-upgrade source BEs; local Lance pins schema discovery and execution to the same eligible backend. The exact backend initializer is called twice, but the base TVF path already had the same unguarded double-call topology, and no new observable planning or execution failure was substantiated.
Tests and CI: reviewed the changed BE/FE unit tests and both catalog/S3 regression suites statically. Per the review contract, I did not run builds or tests. The author reports unit/regression coverage with full local runs blocked; at submission freeze, GitHub reports both coverage contexts failing, while BE UT, FE UT, compile, and code-review are still pending. Passing style/license and Cloud/performance contexts are external CI evidence only.
Documentation and release note: the PR description and release note match the implemented Null, Duration, JSON, and BFloat16 scope and identify follow-up documentation.
User focus: no additional review focus was supplied.
Completion: complete and converged after one round consisting of independent BE, FE/integration, and cross-layer risk passes. Each returned NO_NEW_VALUABLE_FINDINGS; every main candidate was dismissed with evidence or duplicate-fenced.

hello-stephen · 2026-09-13T14:43:45Z

BE UT Coverage Report

Increment line coverage 80.13% (371/463) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	59.79% (25953/43409)
Line Coverage	44.51% (267859/601837)
Region Coverage	40.37% (212562/526511)
Branch Coverage	41.83% (98192/234751)

hello-stephen · 2026-09-13T17:35:49Z

BE Regression && UT Coverage Report

Increment line coverage 79.91% (366/458) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.46% (31446/42234)
Line Coverage	58.68% (350372/597047)
Region Coverage	55.44% (292540/527716)
Branch Coverage	56.29% (132005/234495)

np

### What problem does this PR solve? Follow-up to #67325; partially addresses #66496. Lance columns containing Arrow Null inside lists, structs, or map values are currently reported as unsupported. This change maps nullable Null leaves recursively, including deeper list/map/struct combinations, materializes them without accessing nonexistent Arrow validity/value buffers, and preserves their logical `NULL_TYPE` through nested schema RPC serialization. Array offsets, parent nullability, empty collections, and sliced inputs remain intact. Unknown extensions and non-nullable Null leaves are rejected before materialization. Generic `file()` scans preserve the delegate’s reader compatibility requirements and Local schema-backend affinity during rolling upgrades. Search TVFs retain reader compatibility requirements for deferred columns across TopN lazy materialization. Both read phases are checked without adding deferred fields back to the first-phase tuple. ### Tests - Lazy search mixed-version translation and Lance scan tests: 19 passed in an isolated runner using cached dependencies, including both search TVFs, aliased deferred fields, and ordinary projections. Both new rejection tests failed before the fix. - Generic-file upgrade scheduling and external TVF tests: 13 passed in an isolated runner using cached dependencies. Both new safety tests failed before the fix. - FE type-conversion tests: 12 passed in an isolated runner using existing dependencies; the new nested mapping test failed before the change. - BE nested Null tests: 10 passed in an isolated harness using the changed routines and compatible cached dependencies. The original schema conversion, RPC type preservation, and nested read failures were reproduced. - All affected C++ translation units passed syntax checks against the current source headers and generated schemas. - FE `mvn checkstyle:check`, clang-format 16 checks, and `git diff --check` passed. - The canonical fixture rebuild generates and validates the nested-null dataset. Its full check passes against the committed fixture; a temporary rebuild passes with `--repin` (one existing HNSW discriminator changes on retraining), and the check rejects a missing nested-null dataset. - Added S3 TVF regression coverage and a reproducible seven-column Lance fixture, verified by PyLance round-trip reading. Full FE Maven tests are blocked locally by a Thrift compiler/runtime version mismatch (installed compiler 0.16 versus branch dependency 0.24). The complete BE suite and S3 regression have not been run locally and require CI. ### Release note Support Arrow Null elements and fields nested in Lance arrays, structs, and map values.

…gs (#4182) Filtered Lance retrieval can repeat scalar-index work when scalar segments cross search-split boundaries. Add an English/Chinese guide explaining filter placement, physical-segment split planning, fragment-coverage alignment, construction and maintenance workflows, shared-cache pressure, and profile diagnosis. The guide distinguishes fragment-scoped V2 scalar pruning from legacy reader paths, treats segment sizing as a workload experiment, and explains why logical indexes do not automatically align different columns. It includes filtered vector/FTS SQL and a clearly labeled SDK orchestration pattern with separate per-index commits. Also correct the Catalog type mapping to document Arrow/Lance JSON as Doris JSON, together with the already-supported Duration, nullable Null, and BFloat16 mappings. Preserve storage-type checks, unsupported-extension restrictions, and the distinction between JSON extensions and ordinary strings containing JSON text. Remove JSON from the unsupported-column exclusion example. Type behavior was checked against the current branch-4.1 converter and reader, including apache/doris#67325. Validation: - Docusaurus 3.6.3 MDX compilation passed for all six touched pages. - Front matter, Markdown structure, new-page SEO, targeted links/anchors, and diff checks passed. - Checked bilingual SQL/Python examples, Python syntax, metric names, heading anchors, sidebar JSON/reference, and sensitive-data absence in additions. - The i18n checker reports only missing `current` counterparts. These pages intentionally follow the existing 4.x-only Lance Catalog placement and retain its documented experimental/4.2 availability boundary. - No full-site build, dataset execution, or cluster benchmark was performed. No performance improvement is claimed as measured. Self-review: - Goal: covers filtered retrieval tuning and aligned construction; corrects the requested JSON support statement. - Scope: two new localized pages, focused Catalog corrections, entry links, and one sidebar item. - Information architecture: follows the existing 4.x Lakehouse Best Practices layout; locales are synchronized. - Links/navigation: existing URLs and headings preserved; new page linked from the sidebar, Catalog, and cache guide. - UI/configuration: no component, styling, routing configuration, or build-script changes; sidebar JSON remains valid. - Verification: targeted checks above cover document parsing and navigation; runtime validation limits are explicit. - Other issues: no additional correctness, accessibility, SEO, or maintainability issues found in self-review.

### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

### What problem does this PR solve? Issue Number: None Related PR: apache#65730 apache#66608 apache#66805 apache#66841 apache#66999 apache#67005 apache#67011 apache#67110 apache#67268 apache#67289 apache#67325 apache#67650 apache#67921 apache#68028 apache#68335 apache#68375 apache#68615 apache#68690 Problem Summary: Port the Lance BE read path of branch-4.1/4.2 (as of branch-4.2 cabcf45) to master. Master links liblance_c but has no Lance reader: file_scan_operator keeps FORMAT_LANCE off FileScannerV2 and V1 has no Lance path, so every FORMAT_LANCE scan fails today. This change is BE only. The FE connector plugin and the regression suites follow separately, so nothing on master plans a Lance scan yet and the BE unit tests carry the proof. - Reader: be/src/format_v2/lance/* and format_v2/table/lance_reader.* (plain scans, vector and full-text search, Substrait and runtime filter pushdown, scalar index segments, COUNT(*) from metadata, LIMIT, profiles), copied from branch-4.2 except one local renamed for -Wshadow. - Hooks: FileFormat::LANCE and its exhaustive switches, FileScannerV2 dispatch, and fetch_table_schema for FORMAT_LANCE. Lance queries use FileScannerV2 whatever enable_file_scanner_v2 says, as ADBC and fluss do: there is no V1 Lance reader. The second phase of TopN lazy materialization reads Lance rows through dataset take_rows. - Row locations: GlobalRowLoacationV2 gets a version 1 layout with a 64-bit Lance row id (the version 0 layout is unchanged), PRequestBlockDesc.row_id becomes uint64 (varint, wire compatible), and the materialization operator and RowIdStorageReader decode and validate by version. - Thrift: the external search request body, TLanceFileDesc 4-6 and TExternalSearchRequest 2-4 with the branch-4.1/4.2 ids, in place of the comments that reserve them on master (apache#70075); field 6 stays master's paimon_options. TVectorSearchOptions 6 is branch-4.2's approx_mode (apache#69852), reserved in a comment. - Core: PTypeDesc keeps NULL_TYPE for a NULL literal type, and the nullable serde reads an Arrow Null array as NULLs. - Nine lance_* BE configs, a BE-wide Lance session with an optional Foyer data cache and 10 lance_session_* metrics; macOS links IOKit. - Tests: the Lance BE unit tests and their fixtures, including the docker fixture all_types.lance that one of them reads. Under master's -ffile-prefix-map the session test locates its fixture from the repository root, and the reader tests shrink the data cache to 64 MiB under the UT temp dir: the default 100 GiB disk tier is 51200 block files that stay open for the rest of the test process. Not included: the MaterializationOperator::push timer fix (apache#69579), the Lance index metadata scanner, the FE side, regression suites, and the 4.x changes after cabcf45 (apache#68707, apache#68813, apache#68815). ### Release note Add nine lance_* BE configs and lance_session_* BE metrics. They take effect once the FE can plan Lance scans. ### Check List (For Author) - Test: Unit Test - BE UT: the Lance reader, nested Null, runtime filter and session suites, plus the row-location protocol and FileScannerV2 cases - Behavior changed: Yes. PRequestBlockDesc.row_id is uint64, and the second phase of TopN lazy materialization rejects row locations of an unknown version or size. - Does this need documentation: No (documented with the FE connector) Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@zhangstar333

### What problem does this PR solve? Issue Number: None Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110 #67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690 #68813 #68815 #68707 #68649 Problem Summary: **In short.** This ports the Lance BE read path of branch-4.1 / branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus the BE side of three 4.x changes merged after that snapshot: the search filter `prefilter` flag and profile layout (#68813), distance bounds for `vector_search` (#68815), and the split-level index rule of #68707 that #68815 relies on. It gathers the BE parts of the PRs listed above. Master links `liblance_c` but has no Lance reader, so every `FORMAT_LANCE` scan fails today. With this PR the BE reads Lance datasets through FileScannerV2: plain scans, vector search and full-text search, predicate and runtime-filter pushdown, the schema fetch of a local Lance TVF, and the second phase of TopN lazy materialization. The PR is BE only. The FE connector plugin and the regression suites come in separate PRs, so until then nothing on master plans a Lance scan, and the BE unit tests carry the proof. Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju @FANNG1 **Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` / `s3()` with `format=lance`, and the `vector_search` / `full_text_search` TVFs. On master the FE side is being rebuilt as a connector plugin (`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5` (`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added after the snapshot, so a BE built from master also reads the Lance scan ranges and search requests that a 4.x FE sends during a rolling upgrade. The one exception is `TVectorSearchOptions` 6, branch-4.2's `approx_mode` (#69852, merged into branch-4.2 after the snapshot and not ported): it is reserved in a comment, and a master BE ignores it, so during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate` gets Lance's default (`normal`). **The problem, and what it cost.** | | master today | with this PR | |---|---|---| | A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan fails with "FileScannerV2 does not support file format" | read by `LanceTableReader` in FileScannerV2 | | `fetch_table_schema` for a local Lance TVF | not handled | returns the Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` | | The row location of a Lance row in TopN lazy materialization | `GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a 64-bit id within the snapshot | a version 1 layout carries the 64-bit row id; phase two reads the rows with the dataset's `take_rows` | **How this PR fixes it.** - Reader: `be/src/format_v2/lance/*` and `be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one local renamed for master's `-Wshadow`. The unit tests construct `RuntimeState(TQueryOptions(), globals)`, because master removed the globals-only constructor. - FileScannerV2: the reader dispatch and `_to_file_format` handle `FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two exhaustive switches. Since #69723 every file scan uses FileScannerV2, so nothing else routes Lance scans (on branch-4.2 a Lance query fails with NotSupported when `enable_file_scanner_v2` is off). - Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0, LANCE_DATASET_ROW_ID = 1}` and a union layout for `GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2 varint, so the wire format is compatible). The materialization operator decodes row locations by version, rejects an unknown version or a wrong size, and stamps `row_location_version`. `RowIdStorageReader` validates the counts and the version, range-checks internal row ids against `uint32`, and groups Lance rows by plan node, snapshot version and dataset URI, so each dataset is read with one `take_rows`. Master's segment batching, scan-slot dedup and fetch concurrency stay as they are. - Thrift (`PlanNodes.thrift`): the external search request body (`TVectorElementType`, `TVectorMetric`, `TSearchVector`, `TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`, `TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`, `TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`), `TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in comments and added 6 (`paimon_options`); this PR replaces those comments with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is reserved in a comment for branch-4.2's `approx_mode` (#69852). - After the snapshot (one commit each): - #68813: `prefilter` decides whether the TVF search filter runs before (true) or after (unset / false) candidate selection, for vector and full-text search. The profile groups the Lance counters under `LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan` and `LanceDataCache`, uses the 4.x counter names, and registers every scan metric before the statistics callback is installed, so the callback, which Lance invokes from its execution streams, never adds counters. #68813's FE tweak that hides `TUnit.NONE` labels in `AggCounter` belongs to the FE port. - #68707 (part): a vector split searches exactly the index segments FE planned for it, and a split without segments is searched flat whatever `use_index` says; a split with segments under `use_index = false` is rejected. 4.x FEs plan segments only when `use_index` allows them. - #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound` (exclusive) are validated (finite FLOAT values, lower < upper at FLOAT precision, single-vector only) and installed with `lance_scanner_set_distance_range()`, which master's pinned lance-c provides. - Review fixes: - A nested array goes to the Doris SerDes without a copy only when it and every descendant start at offset 0 and reference exactly their child ranges. Lance's legacy (v1) decoder can return a struct or list that is tight at its own level over a Boolean child with a bit offset (a batch that starts mid-byte in the bitmap), and the SerDes reject that under Arrow input validation. - A zero-field `STRUCT<>`, which Lance decodes, is reported as an unsupported column at any nesting level, like the other unsupported Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot hold (its row count comes from its first child). - A Lance Decimal256 field with precision <= 38 is narrowed to Decimal128 by a safe Arrow cast before decoding. Doris types it DECIMAL32/64/128I by precision, and those SerDes read only Decimal128 arrays; the BE used to crash on the first value, also for nested leaves. Only values a reader can see are cast: before a nested leaf is narrowed, every child slot that no visible row references (under a NULL list, fixed-size-list, map or struct row, or outside a sliced parent) is marked NULL, so a hidden value does not fail the cast and a visible value outside the declared precision still does; map keys, which cannot be NULL, stay checked. - FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows that Doris keeps. - Runtime filters that mention INT64_MIN stay residual-only. The pinned Lance SQL parser reads `-9223372036854775808` as the negation of a magnitude that it parses as i64 first; the overflow turns the literal into a Float64, which the planner cannot coerce to Int64 or Decimal, so a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal column failed planning and the query failed. Every other i64 is still pushed. - The shared Arrow DATE32 decoding (`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside 0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`, skips null slots and adds the epoch in 64 bits: the DATE32 range part of branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and paimon-rust reads. - Child values hidden by a NULL list, fixed-size-list, map or struct row are not decoded: the array, map and struct SerDes give the element SerDe a validity bitmap that also clears those slots (`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden 10000-01-01 under a NULL fixed-size list used to fail the DATE32 range check above. Visible values are unchanged; this applies to ADBC, Arrow stream load and paimon-rust as well. The masks come from ExecEnv's tracked Arrow pool. A nested Lance array that is compacted before decoding is masked the same way, at every level, before its `ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision decimal, Date64 or time no longer fails the batch while a visible one still does. Since #69697 the same masks keep hidden DATETIMEV2 / TIMESTAMPTZ values out of its range checks. A mask costs at most one pass per batch: a NULL row that owns no child slots, or only NULL ones, allocates nothing, and Arrow stream load masks each retained IPC batch once (`mask_arrow_descendants_not_visible()`) before it hands row ranges to the SerDes, so its per-block reads allocate nothing. The one per-block mask left is for map keys under a NULL map row that still owns entries, since a MapArray cannot hold NULL keys; Arrow's map builder writes NULL map rows without entries. - Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The nullable serde reads an Arrow Null array as NULLs without touching buffers that the array does not have. - Configs, caches and metrics, with the 4.x names and defaults: | Config | Default | |---|---| | `lance_index_cache_size_bytes` | 10 GiB | | `lance_metadata_cache_size_bytes` | 1 GiB | | `enable_lance_data_cache` | true | | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` | | `lance_data_cache_disk_capacity_bytes` | 100 GiB | | `lance_data_cache_read_block_size_bytes` | 1 MiB | | `lance_io_buffer_size_bytes` (mutable) | 2 GiB | | `lance_batch_readahead` (mutable) | 5 | | `lance_fragment_readahead` (mutable) | 5 | The BE-wide Lance session (index cache, metadata cache and the optional Foyer disk cache) is created by the first Lance scan, so a BE that never reads Lance creates no cache directory. The first six configs take effect after a restart. A negative cache size (and, while the data cache is enabled, a negative disk capacity or read block size) fails the first Lance scan with a message that names the setting, as does any data-cache value lance-c rejects; the rest of the BE keeps serving. Ten `lance_session_*` metrics report the two in-memory caches. With the defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from the first Lance scan on (100 GiB in blocks of twice the read block size), the same as on branch-4.1 / 4.2. Size `lance_data_cache_disk_capacity_bytes` and `lance_data_cache_read_block_size_bytes` against the BE's open-file limit, or set `enable_lance_data_cache = false`. The index and metadata caches live in lance-c memory: they are not charged to a MemTracker or registered with CacheManager, so memory GC, the cache capacity adjustment under memory pressure and `/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache = false` does not turn them off. lance-c has no call to clear, evict or resize them, so size them with the two configs. The disk tier keys blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS, without the endpoint) and the object path (`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve the same bucket name and data-file path with different bytes (data files registered under chosen names or replaced in place; Lance's writers use random file names) share entries, also across a BE restart; such a deployment should set `enable_lance_data_cache = false`. Both are as on branch-4.1 / 4.2 and tracked as follow-ups. A Lance scan's native working set, that is the I/O buffered up to `lance_io_buffer_size_bytes` per scanner, the decode readahead and the operators of a search plan such as the top-k of a flat vector search, is allocated inside lance-c and is charged to neither the query nor its workload group. It counts toward the BE process memory only, so memory GC, which ranks queries by tracked memory, does not attribute it to the Lance query. lance-c accepts no memory budget, so on BEs with little memory headroom size `lance_io_buffer_size_bytes`, `lance_batch_readahead` and `lance_fragment_readahead` for the number of concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a follow-up. - Build: macOS links `-framework IOKit`. This is the first time objects of `liblance_c.a` are pulled into `doris_be`, so the binary and the link time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and `util/threadpool.h` 589, the same as master `315c2e00331` alone (limits 275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562 (master alone 559, limit 660); the reach budgets are unchanged. Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`); the FE side, including the rolling-upgrade guard that keeps Lance lazy materialization off while an old BE could decode a version 1 row location as version 0; the regression suites; the rest of #68707 (searching historical versions, tags and branches, rejecting fragment ids missing from the snapshot, naming the snapshot in errors and the profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that #67005 carried goes in a separate PR. **Results.** | | master | with this PR | |---|---|---| | A `FORMAT_LANCE` query | fails before reading anything | read by `LanceTableReader` in FileScannerV2 | | A Lance row in the second phase of TopN lazy materialization | cannot be represented | a version 1 row location, read back with `take_rows` | | A row location of an unknown version or size in phase two | decoded as version 0 | rejected with an error | | An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read fails with `InvalidArgument` | BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run, 1027 pass and 2 Parquet microbenchmarks skip by design. They cover the Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10, `LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the row-location protocol (`MaterializationSharedStateTest`, `RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`: count mismatch, unknown version, a Lance version on an internal mapping, an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`, `ParquetScanTest` and the other `Parquet*` suites), whose row ids now go through `file_local`; and the Arrow / protobuf / DATE serde suites with the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader suites that decode Arrow dates. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest` 10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310 pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The ninth review round adds `LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and `LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`: both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto `339e087b11f`, and restoring `parse_int64()` in either check makes exactly these two fail. On the head rebased onto master `315c2e00331`, with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest` (15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`, `AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest` and the Lance suites. Allocating at the first NULL row of each call again fails the new `ArrowParentNullsTest` case and the four new `LoadArrowPipeTest` block cases, dropping the per-batch mask fails the LIST and STRUCT block cases, and returning the child unmasked fails the new timestamp case, four `ArrowParentNullsTest` cases and `LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`. ### Release note Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They take effect once the FE can plan Lance scans. Reading an Arrow DATE32 value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or paimon-rust now fails instead of returning 0000-00-00. ### Check List (For Author) - Test  - [ ] Regression test - [x] Unit Test - `run-be-ut.sh --run --filter=...` with the suites listed under Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the rebases, and the Lance and Arrow SerDe suites pass again with the nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the scanner selection and in `fetch_table_schema`, which now hands every file format to `format::fetch_file_schema()`, so the Lance branch runs ahead of that call; #69646 added a TopN row-group test that reads the version 0 row location fields, which now go through `file_local` (the earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the scanner selection, the `FileFormat` enum, its test and the reach budget table). The 95 TUs that this PR changes or that master changed since `5caaeb3e3d4` were syntax-checked against the rebased tree with the UT build flags, and CI runs the suites again. Rebased again onto master `8d88f1cc6f7`: the only conflict was the budget table, where #65858 moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that this PR changes or that master changed since `43f9ceb48d7` pass the same syntax check. Rebased again onto master `339e087b11f`: the only conflict was the budget table again, where #67292 moved the `util/rle_encoding.h` baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent masking in the map and struct readers and merged cleanly. Of the 114 sources that this PR changes or that master changed since `8d88f1cc6f7`, 111 pass the same syntax check. The other three are master's: two new #67292 files under `storage/segment` fail the same way on master alone, since they compile only inside their unity batch, and this PR adds no source to that target; the #69818 benchmark is not in the UT build and includes only `util/simd` headers that this PR does not touch. Reverting any of the review fixes, the #68813 flag, the #68707 split rule or the #68815 bounds makes its new tests fail. Rebased again onto master `378404eeb12`, which removed the legacy file scanner (#69723): the conflicts were in the scanner-selection code and tests that #69723 deleted, so this PR's Lance lines there went with them (including its `LanceQueriesAlwaysUseScannerV2` test), in the profile keys of `materialization_opertor.cpp`, and in the row fetch of `rowid_fetcher.cpp`, whose non-Lance path is now only `FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests took this PR's `file_local` rename with them, and the `util/rle_encoding.h` budget commit is dropped because master is no longer near that limit. Of the 100 sources that this PR changes or that master changed since `339e087b11f`, 99 pass the same syntax check; the other, `format_v2/orc/orc_reader.cpp`, fails the same way on master alone. On the rebased head the suites listed under Results run 1029 cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that `DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master. Rebased again onto master `315c2e00331`: the only conflict was `TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had reserved fields 2-5 in comments; the two commits that add the fields replace those comments, and field 6 stays #70075's `paimon_options`. #69717 cut transitive includes from shared headers: of the 160 sources that this PR changes or that master changed since `378404eeb12`, 156 pass the same syntax check; the two `jemalloc_hook` tests have no compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc aliases that macOS cannot build (#69819), and `format_v2/table_reader.cpp` alone fails the same way on master alone while its unity batch compiles. The full `doris_be_test` builds and links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases) pass before the mask fix, and the 328 cases above with it. - `build-support/check-build-hygiene.sh` (on this branch rebased onto master `315c2e00331`), clang-format 16 and `build-support/run-clang-tidy.sh` on the changed files: all pass; on the changed lines clang-tidy (llvm 22) reports only `readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of the masking functions, the macro's expansion that master's existing `DORIS_CHECK`s report too. - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - Behavior changed: - [ ] No. - [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN lazy materialization rejects row locations of an unknown version or size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day outside the Doris DATE range fails the read instead of becoming `0000-00-00`. - Does this need documentation? - [x] No. Lance is documented with the FE connector. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label  --------- Co-authored-by: zhangstar333 <zhangsida@selectdb.com> Co-authored-by: Gabriel <liwenqiang@selectdb.com> Co-authored-by: zhaobo wang <wangzhaobo957@gmail.com> Co-authored-by: jay <qingfengyouchen@163.com> Co-authored-by: FANNG <fanng@apache.org> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: zy-kkk <zhongyk10@gmail.com>

wangzhaobo957-cloud force-pushed the feature/lance-additional-data-types branch from eecec32 to 57a5835 Compare August 31, 2026 03:32

github-actions Bot requested changes Sep 1, 2026

View reviewed changes

github-actions Bot requested changes Sep 4, 2026

View reviewed changes

wangzhaobo957-cloud force-pushed the feature/lance-additional-data-types branch 2 times, most recently from 4abb063 to e0bfe86 Compare September 9, 2026 12:21

wangzhaobo957-cloud marked this pull request as ready for review September 10, 2026 03:40

wangzhaobo957-cloud requested a review from yiguolei as a code owner September 10, 2026 03:40

github-actions Bot requested changes Sep 10, 2026

View reviewed changes

wangzhaobo957-cloud force-pushed the feature/lance-additional-data-types branch from e0bfe86 to 90d2b22 Compare September 10, 2026 16:53

github-actions Bot requested changes Sep 11, 2026

View reviewed changes

wangzhaobo957-cloud force-pushed the feature/lance-additional-data-types branch from 90d2b22 to 7ea58fd Compare September 11, 2026 03:36

github-actions Bot previously requested changes Sep 13, 2026

View reviewed changes

wangzhaobo957-cloud added 5 commits September 13, 2026 18:17

[fix](lance) Handle sliced values and mixed-BE execution apache#66496

1ce1312

[fix](lance) Handle registered extensions and upgrade routing apache#…

cfd7e5d

…66496

[fix](lance) Complete additional type upgrade compatibility apache#66496

025fb7e

[test](lance) Fix exact backend selection test apache#66496

7889ba5

[fix](lance) Track Arrow normalization memory apache#66496

598b081

wangzhaobo957-cloud force-pushed the feature/lance-additional-data-types branch from c51cf70 to 598b081 Compare September 13, 2026 10:19

github-actions Bot reviewed Sep 13, 2026

View reviewed changes

yiguolei approved these changes Sep 14, 2026

View reviewed changes

yiguolei merged commit 5445bdf into apache:branch-4.1 Sep 14, 2026
33 of 36 checks passed

Gabriel39 mentioned this pull request Sep 14, 2026

[Feature](lance) Support nested Arrow Null types #67921

Merged

Gabriel39 mentioned this pull request Sep 30, 2026

[docs] Add Lance hybrid search best practices and correct type mappings apache/doris-website#4182

Merged

This was referenced Sep 30, 2026

[Feature](lance) Support additional Lance and Arrow data types #66496

Open

[Tracking] Lance feature matrix for Doris 4.2 #66340

Open

morningman mentioned this pull request Oct 25, 2026

[feature](be) Read Lance datasets with FileScannerV2 #69581

Merged

3 of 12 tasks

Conversation

wangzhaobo957-cloud commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 31, 2026

Uh oh!

zhangstar333 commented Sep 1, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

wangzhaobo957-cloud commented Sep 1, 2026

Uh oh!

zhangstar333 commented Sep 4, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

wangzhaobo957-cloud commented Sep 6, 2026

Uh oh!

zhangstar333 commented Sep 7, 2026

Uh oh!

zhangstar333 commented Sep 7, 2026

Uh oh!

hello-stephen commented Sep 7, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 7, 2026

BE UT Coverage Report

Uh oh!

Gabriel39 commented Sep 10, 2026

Uh oh!

Gabriel39 commented Sep 10, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Sep 10, 2026

BE Regression && UT Coverage Report

Uh oh!

Gabriel39 commented Sep 11, 2026

Uh oh!

Gabriel39 commented Sep 11, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

wangzhaobo957-cloud commented Sep 11, 2026

Uh oh!

wangzhaobo957-cloud commented Aug 31, 2026 •

edited

Loading