Skip to content

Add a table manifest to the experimental vortex-table - #11559

Open
joseph-isaacs wants to merge 8 commits into
developfrom
ji/tb-manifest
Open

joseph-isaacs wants to merge 8 commits into
developfrom
ji/tb-manifest

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

This is the first piece of the experimental, proof-of-concept table that demonstrates the scanning API. A manifest is itself a Vortex file, with one row per data file holding its path, size, row count and footer region.

With the footers inlined, the table planner that follows can open every file and prune on its statistics without reading any data file. The footer is found through the normal open path, and the region is everything from Footer::segments_end() to the end of the file, so its offset is derived rather than stored. The manifest is read back through a v2 scan with develop's ScanDriver, so nothing here depends on the new plan drivers.

Follows #11537.


Generated by Claude Code

joseph-isaacs and others added 3 commits November 17, 2026 13:10
An empty, unpublished crate that will hold a proof of concept of a
minimal multi-file table, built to demonstrate Vortex's scanning API: a
manifest of data files and their footers, stats pruning across files,
and a planner that scans the kept files under one plan driver. It is
experimental and not a table format; nothing in the workspace depends
on it. The manifest, pruning, planner and an example follow in separate
changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The manifest is a Vortex file with one row per data file: its path, size,
row count, footer region offset and length, an XXH3 hash of that region,
and optionally the region itself. With the footer inlined, a later scan
can open every file and prune on its statistics without reading it; the
hash fixes the exact bytes an out-of-file index can bind to.
ManifestBuilder reads each file's exact footer region, and the manifest
is read back through a v2 scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Nov 17, 2026 — with Claude
@joseph-isaacs joseph-isaacs self-assigned this Nov 17, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Copy link
Copy Markdown
Contributor Author

Rust tests (miri) failed on an AWS Spot-instance interruption, not on this change. I'll re-run it once the workflow run finishes. The earlier Rust (docs) failure (a redundant intra-doc link in manifest.rs) is fixed in 4c2cb00.


Generated by Claude Code

Base automatically changed from ji/tb-crate to develop November 17, 2026 18:02
joseph-isaacs and others added 4 commits November 17, 2026 18:02
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

# Conflicts:
#	Cargo.lock
#	vortex-table/Cargo.toml
#	vortex-table/src/lib.rs
Drop the hand-rolled footer reader, the storage trait and the separate
writer module: the footer comes from open_options(), and its region is
segments_end() through the end of the file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The footer offset, length and hash all follow from the inlined region and
the file size, so the manifest stores only path, size, row count and the
region. ManifestEntry::read replaces the builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Reading a file whose columns have the manifest's names but other types
panicked in as_slice; check the dtype first. Drop footer_hash, which no
planned part consumes, footer_offset and row_count, which only tests used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBB5Ee9dvNfjZPnyuUN
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Copy link
Copy Markdown
Contributor Author

The red Java check on 36f9a02 isn't a test failure. It's the aggregator job reporting that its sub-jobs were cancelled. About 25 jobs in run 44187447921 were cancelled within 3 seconds of being queued and never got a runner, so nothing ran. I'll re-run the failed jobs once the run finishes. The earlier Windows (x64) failure on 3e95c67 was another AWS Spot interruption.


Generated by Claude Code

@codspeed

codspeed Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 1975 untouched benchmarks
⏩ 53 skipped benchmarks1


Comparing ji/tb-manifest (3e95c67) with develop (3ebf68d)2

Open in CodSpeed

Footnotes

  1. 53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on develop (ef1dc20) during the generation of this report, so 3ebf68d was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant