Repository navigation
Add a table manifest to the experimental vortex-table - #11559
joseph-isaacs wants to merge 8 commits into
Conversation
An empty, unpublished crate that will hold a proof of concept of a minimal multi-file table, built to demonstrate Vortex's scanning API: a manifest of data files and their footers, stats pruning across files, and a planner that scans the kept files under one plan driver. It is experimental and not a table format; nothing in the workspace depends on it. The manifest, pruning, planner and an example follow in separate changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The manifest is a Vortex file with one row per data file: its path, size, row count, footer region offset and length, an XXH3 hash of that region, and optionally the region itself. With the footer inlined, a later scan can open every file and prune on its statistics without reading it; the hash fixes the exact bytes an out-of-file index can bind to. ManifestBuilder reads each file's exact footer region, and the manifest is read back through a v2 scan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
b5b65bc to
4c2cb00
Compare
|
Generated by Claude Code |
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> # Conflicts: # Cargo.lock # vortex-table/Cargo.toml # vortex-table/src/lib.rs
Drop the hand-rolled footer reader, the storage trait and the separate writer module: the footer comes from open_options(), and its region is segments_end() through the end of the file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The footer offset, length and hash all follow from the inlined region and the file size, so the manifest stores only path, size, row count and the region. ManifestEntry::read replaces the builder. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Reading a file whose columns have the manifest's names but other types panicked in as_slice; check the dtype first. Drop footer_hash, which no planned part consumes, footer_offset and row_count, which only tests used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
|
The red Generated by Claude Code |
Merging this PR will not alter performance1 warning
Comparing Footnotes
|
This is the first piece of the experimental, proof-of-concept table that demonstrates the scanning API. A manifest is itself a Vortex file, with one row per data file holding its path, size, row count and footer region.
With the footers inlined, the table planner that follows can open every file and prune on its statistics without reading any data file. The footer is found through the normal open path, and the region is everything from
Footer::segments_end()to the end of the file, so its offset is derived rather than stored. The manifest is read back through a v2 scan with develop'sScanDriver, so nothing here depends on the new plan drivers.Follows #11537.
Generated by Claude Code