Skip to content

feat(compressor): compress a leaf's validity in the scheme that stores it - #11576

Open
joseph-isaacs wants to merge 1 commit into
developfrom
ji/scheme-internal-validity
Open

joseph-isaacs wants to merge 1 commit into
developfrom
ji/scheme-internal-validity

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

The compressor compresses every leaf's validity before it chooses a scheme. That work is wasted when the winning scheme drops the validity: RunEnd keeps nulls in its values, Sparse in its patches, and Sequence and the constant shortcut have no bitmap. When a scheme passes the validity to a child, the compressed bitmap is decompressed when the child is canonicalized, then compressed again one level down.

With this change, a scheme that stores its input's validity compresses that validity itself. Schemes that drop it no longer pay for it. This is the first of two PRs. The second, #11494, adds Scheme::encode on top of it, which will let a scheme register its validity with the compressor directly.

Changes

Compressor (vortex-compressor/src/compressor)

  • A leaf's validity is no longer compressed before choosing a scheme.
  • The compressor still compresses it in the places where no scheme does:
    • a leaf no scheme beats;
    • the validity mask of a constant array;
    • a canonical child returned as it is at the cascade limit.
  • A rejected constant encoding now reports the input as unchanged, not as compressed, so the leaf's validity still gets compressed.
  • Two public helpers for schemes:
    • CascadingCompressor::with_compressed_validity(&ArrayRef) returns a leaf with its validity compressed;
    • CascadingCompressor::compress_validity(Validity) compresses a validity the scheme built itself. It was already the function behind container validity, and is now public.

Schemes (vortex-btrblocks/src/schemes)

  • Store the input's validity: BitPacking, Delta, Pco, ALP-RD, Zstd, ZstdBuffers, FSST and OnPair now compress their input's validity first, so the encoding stores the compressed bitmap, as it did before.
  • VarBin: it rebuilds the bitmap through a builder and used to write it raw. It now compresses it with compress_validity.
  • Unchanged: the other schemes either drop the validity or pass it to a child, which compresses it at the level it ends up on.

Tests

  • test_bitpacking_compresses_its_validity fails if BitPacking stops compressing its input's validity.
  • test_varbin_validity_is_compressed fails on develop, where VarBin's validity stays raw.
  • All golden snapshots are unchanged.

Checks run

  • cargo test -p vortex-compressor -p vortex-btrblocks --all-features.
  • cargo clippy -p vortex-compressor -p vortex-btrblocks --all-targets --all-features -- -D warnings.
  • RUSTDOCFLAGS="-D warnings" cargo doc -p vortex-compressor -p vortex-btrblocks --no-deps.
  • cargo +nightly-2026-11-13 fmt on both crates.

The CUDA and Python bindings were not run.

API Changes

Additions only:

  • CascadingCompressor::with_compressed_validity, which is new;
  • CascadingCompressor::compress_validity, which already existed and is now public.

A scheme outside this repository that stores its input's validity no longer gets that validity pre-compressed. Its output stays correct, but its validity is stored uncompressed until the scheme calls with_compressed_validity.


Generated by Claude Code

…s it

The compressor compressed every leaf's validity before choosing a scheme.
That work was wasted for schemes that drop the validity (RunEnd, Sparse,
Sequence, constants), and a scheme that passes it to a child had it
decompressed and compressed again one level down.

A scheme that stores its input's validity now compresses it itself:
BitPacking, Delta, Pco, ALP-RD, Zstd, ZstdBuffers, FSST and OnPair call the
new `CascadingCompressor::with_compressed_validity` on their input, and VarBin
compresses the bitmap it rebuilds with `compress_validity`, which it used to
write raw. A leaf no scheme compresses, a constant's mask, and a child left
uncompressed at the cascade limit have their validity compressed by the
compressor. The other schemes pass the validity to a child, which compresses
it at the level it ends up on.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBC7U83BNkbAphq6H4L

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants