[Bug](lance) test_lance_vector_search boundary_single_probe returns a one-sided candidate set on branch-4.1 #67026

Activity

1. Doris-Breakwater commented on Aug 21, 2026

   

   Doris-Breakwater

   on Aug 21, 2026

   Collaborator

   More actions

   Initial assessment

   This is most likely a fixture/object-store mismatch, not a confirmed Doris vector-search bug. The reported candidate set exactly identifies the newer IVF_PQ fixture that was merged after 5dcf1066af4. A clean Doris build does not make the persistent Iceberg/MinIO data clean.

   Verified facts

   • I ran the fixture self-check from 5dcf1066af4 with its pinned Python dependencies. Its vs_ivf_pq_f32 index UUID is 4231299b-169a-4694-b440-7fb7396b1188; the selected IVF partition contains rows 1-259, and the nprobes=1 boundary query returns the old golden sequence (256,255,257,254,258,253,259,252,251,...).
   • The later branch-4.1 commit f44b3fa49e2 ([test](lance) Add query coverage for the remaining ANN index types and vector element types #66512) regenerates this non-reproducible IVF index. The table path changes from c325fd55_doris$vs_ivf_pq_f32 to 3adf6edd_doris$vs_ivf_pq_f32, and the index UUID changes to f7df7f0a-77f9-428a-abe2-fe510132cfa0.
   • Running that newer fixture's self-check selects the partition containing rows 255-510 and returns exactly the issue's actual rows: 256,255,257,258,259,260,261,262,263.
   • The same later commit changes test_lance_vector_search.out to exactly those rows. It is a descendant of 5dcf1066af4, is already on current branch-4.1, and does not change the relevant FE/BE production path.
   • The Iceberg compose file bind-mounts MinIO storage at docker/thirdparties/docker-compose/iceberg/data/input/minio_data. run-thirdparties-docker.sh -c iceberg brings the stack down but does not clear that directory; startup recursively copies the checked-out fixture into the existing warehouse/lance prefix without first removing stale objects. Therefore switching commits or rebuilding Doris can leave code, golden, manifest, and index objects from different revisions.
   • Independently, nprobes=1 on a four-partition IVF_PQ index is approximate. refine_factor=10 recomputes exact L2 distances only for candidates obtained from the probed partition; it cannot recover neighbours assigned to another partition. A one-sided result at a partition boundary is therefore not, by itself, a correctness violation. The test's functional discriminator still succeeds because this distance sequence differs from flat search.

   At 5dcf1066af4, distributed execution does change the physical path: FE creates a split for the physical index segment and its two fragments, and BE enables prefilter before nearest() while passing the explicit fragment IDs and index-segment UUID. That path deserves isolation only if the old fixture is confirmed. The current evidence does not establish that it changed the old fixture's candidate set.

   Missing information

   Please provide/confirm:

   1. The physical vs_ivf_pq_f32 table directory and index UUID actually present in MinIO during the failing run. For an exact 5dcf1066af4 fixture these should be c325fd55_doris$vs_ivf_pq_f32 and 4231299b-169a-4694-b440-7fb7396b1188, not the newer values above.
   2. Whether docker/thirdparties/docker-compose/iceberg/data/input/minio_data (or an equivalent isolated MinIO data root) was recreated after checking out 5dcf1066af4. “Clean build” alone is insufficient.
   3. A rerun on current branch-4.1 with a fresh, matching fixture. Current branch already contains the newer fixture and matching golden.
   4. Only if the old UUID is confirmed and still fails: the EXPLAIN output and BE profile for this query, showing two searched fragments, one index segment, LanceUseIndex=true, and LanceFragmentCount=2; plus results for the same fixture with nprobes=4 and use_index=false.

   Recommended next steps

   1. Recreate the Iceberg/MinIO environment from an empty, isolated data root at the same Git revision as the test and rerun the suite.
   2. If the old fixture then returns the old golden, close this as environment/fixture skew. On current branch-4.1, f44b3fa49e2 already couples the regenerated fixture to the reported candidate set.
   3. If Doris still returns the reported rows against the confirmed old UUID, run 5dcf1066af4^ against the same unchanged object-store contents. If only 5dcf1066af4 differs, isolate the two new Lance inputs with a focused lance-c test: unrestricted whole-dataset search versus with_fragments(all) plus the same single with_index_segments(...) selection. That will distinguish a prefilter/segment-planning change from IVF approximation before considering production code changes.
   4. Longer term, add a fixture identity/fingerprint assertion near this boundary test. The durable contract should be result count, exact distances after refinement, and a distance sequence different from flat search; exact row IDs for nprobes=1 are valid only when explicitly tied to the frozen index bytes.

   No production code change is recommended from the current evidence.

   Breakwater-GitHub-Analysis-Slot: slot_a27bbd1ec9da
2. FANNG1 commented on Aug 21, 2026

   

   FANNG1

   on Aug 21, 2026

   ContributorAuthor

   More actions

   Correction: I can no longer reproduce this, so the "deterministic" characterisation above is wrong and I want to flag that before anyone spends time on it.

   What I have now, on the same machine and the same docker environment:

   • test_lance_vector_search passes, three times in a row — once as part of -d external_table_p0/lance (all 7 suites green, including test_lance_vector_search_two_phase) and twice run on its own.
   • The boundary_single_probe result now matches the golden, i.e. neighbours on both sides of the query point.

   What changed in between was FE/BE rebuilds and process restarts, plus an unrelated FE-side refactor of mine that cannot affect the search path. Nothing touched the fixture datasets or the index.

   So the accurate statement is: the failure was real and reproduced four times, including on a clean build of 5dcf1066af4 with none of my changes present, and then stopped reproducing after rebuilding and restarting. That makes it state-dependent rather than deterministic, and I do not know what the trigger is — the earlier runs were stable enough (byte-identical output across three runs) that it did not look like ordinary flakiness at the time.

   Two things still stand and are why I am leaving this open rather than closing it:

   • When it did fail, the returned candidate set was one-sided (row_id >= 256 only), which is a search-result difference and not a golden-ordering nit.
   • It reproduced without any of my changes, so whatever the trigger is, it is not specific to [fix](lance) Pass namespace-vended storage options through to the BE #66805.

   @zhangstar333 — sorry for the noise. If this does not ring a bell as a known state-dependent behaviour in the distributed search path, it is probably not worth chasing on my report alone, and I am happy for this to be closed. I will reopen with a proper reproduction if I hit it again.
3. FANNG1

   closed this as completedon Aug 22, 2026