Repository navigation
feat(compressor): compress a leaf's validity in the scheme that stores it - #11576
Open
joseph-isaacs wants to merge 1 commit into
Open
joseph-isaacs wants to merge 1 commit into
joseph-isaacs wants to merge 1 commit into
Conversation
…s it The compressor compressed every leaf's validity before choosing a scheme. That work was wasted for schemes that drop the validity (RunEnd, Sparse, Sequence, constants), and a scheme that passes it to a child had it decompressed and compressed again one level down. A scheme that stores its input's validity now compresses it itself: BitPacking, Delta, Pco, ALP-RD, Zstd, ZstdBuffers, FSST and OnPair call the new `CascadingCompressor::with_compressed_validity` on their input, and VarBin compresses the bitmap it rebuilds with `compress_validity`, which it used to write raw. A leaf no scheme compresses, a constant's mask, and a child left uncompressed at the cascade limit have their validity compressed by the compressor. The other schemes pass the validity to a child, which compresses it at the level it ends up on. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBC7U83BNkbAphq6H4L
robert3005
approved these changes
Nov 17, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The compressor compresses every leaf's validity before it chooses a scheme. That work is wasted when the winning scheme drops the validity: RunEnd keeps nulls in its values, Sparse in its patches, and Sequence and the constant shortcut have no bitmap. When a scheme passes the validity to a child, the compressed bitmap is decompressed when the child is canonicalized, then compressed again one level down.
With this change, a scheme that stores its input's validity compresses that validity itself. Schemes that drop it no longer pay for it. This is the first of two PRs. The second, #11494, adds
Scheme::encodeon top of it, which will let a scheme register its validity with the compressor directly.Changes
Compressor (
vortex-compressor/src/compressor)CascadingCompressor::with_compressed_validity(&ArrayRef)returns a leaf with its validity compressed;CascadingCompressor::compress_validity(Validity)compresses a validity the scheme built itself. It was already the function behind container validity, and is now public.Schemes (
vortex-btrblocks/src/schemes)compress_validity.Tests
test_bitpacking_compresses_its_validityfails if BitPacking stops compressing its input's validity.test_varbin_validity_is_compressedfails ondevelop, where VarBin's validity stays raw.Checks run
cargo test -p vortex-compressor -p vortex-btrblocks --all-features.cargo clippy -p vortex-compressor -p vortex-btrblocks --all-targets --all-features -- -D warnings.RUSTDOCFLAGS="-D warnings" cargo doc -p vortex-compressor -p vortex-btrblocks --no-deps.cargo +nightly-2026-11-13 fmton both crates.The CUDA and Python bindings were not run.
API Changes
Additions only:
CascadingCompressor::with_compressed_validity, which is new;CascadingCompressor::compress_validity, which already existed and is now public.A scheme outside this repository that stores its input's validity no longer gets that validity pre-compressed. Its output stays correct, but its validity is stored uncompressed until the scheme calls
with_compressed_validity.Generated by Claude Code