Skip to content

Find no row, not an error, for a missing rowid on rescore and DiskANN columns - #342

Open
MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:point-missing-rowid
Open

MayCXC wants to merge 1 commit into
asg017:mainfrom
MayCXC:point-missing-rowid

Conversation

@MayCXC

@MayCXC MayCXC commented Oct 23, 2026 •

Copy link
Copy Markdown

A point query on a rowid the table does not hold returns no rows on a flat column, and fails on a rescore or DiskANN column:

create virtual table t using vec0(embedding float[8] indexed by rescore(quantizer=bit));
insert into t(rowid, embedding) values (1, ?);   -- any float[8]
select rowid from t where rowid = 999;
-- Error: Could not fetch vector data for 999 from rescore vectors

The same happens with quantizer=int8, for a rowid that was deleted, and on a DiskANN column (Could not fetch vector data for 999 from DiskANN vectors table). An IVF column already returns no rows.

vec0_get_vector_data() looks the rowid up in _rowids with vec0_get_chunk_position(), whose SQLITE_EMPTY vec0Filter_point() reads as the end of the results, but only on the flat path; the DiskANN and rescore branches come first and read their own tables by rowid, where a missing row is an error. Every index type registers its rows in _rowids (the insert's first step, an all-DiskANN table included, which skips only the chunk placement), so the lookup now comes before every index's read. Its other callers, vec0Column_fullscan() and vec0Column_knn(), read rowids taken from _rowids or from a KNN result.

Test. tests/test-rescore.py (both quantizers), tests/test-diskann.py and tests/test-ivf.py each check a never-inserted and a deleted rowid. On main the rescore and DiskANN tests fail with the errors above; on this branch they pass, and the IVF one passes on both. The Python tests pass: 242 passed and 67 skipped as make loadable builds, 305 passed and 4 skipped with -DSQLITE_VEC_EXPERIMENTAL_IVF_ENABLE=1. It merges cleanly with #327 to #331 and #336 to #341.

Origin. The rescore port to photostructure/sqlite-vec, photostructure#8 (cd4bb09), lists this among its differences from upstream, with the lookup first in vec0_get_vector_data() as here, and the DiskANN port, photostructure#11 (cd53413), tests it for DiskANN columns; the change and its tests are written here for asg017's main. No open or closed PR or issue here makes the same change.

… columns

A point query on a rowid the table does not hold returns no rows on a flat
column: vec0_get_vector_data() looks the rowid up in _rowids first, and
vec0Filter_point() reads its SQLITE_EMPTY as the end of the results. A
rescore column read the rowid's row of _rescore_vectors, and a DiskANN
column its row of _vectorsNN, before that lookup, so the same query failed:

    create virtual table t using vec0(
      embedding float[8] indexed by rescore(quantizer=bit));
    insert into t(rowid, embedding) values (1, ?);
    select rowid from t where rowid = 999;
    -- Could not fetch vector data for 999 from rescore vectors

A rowid deleted from the table fails the same way, and a DiskANN column
reports "Could not fetch vector data for 999 from DiskANN vectors table".
An IVF column already returns no rows.

Every index type registers its rows in _rowids, an all-DiskANN table
included, so vec0_get_vector_data() now looks the rowid up there before
reading any index's vectors. tests/test-rescore.py, tests/test-diskann.py
and tests/test-ivf.py check a never-inserted and a deleted rowid on each;
the rescore and DiskANN tests fail on main.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant