Repository navigation
Streaming chunked decompression via a composable vtable method - #9614
joseph-isaacs wants to merge 25 commits into
Conversation
Merging this PR will degrade performance by 2.75%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ❌ | Simulation | decompress[for_bp_u64] |
44.9 µs | 54.9 µs | -18.3% |
| ❌ | Simulation | bitpacked_decompress_u32 |
51.4 µs | 61.7 µs | -16.64% |
| ❌ | Simulation | baseline_lt[4, 1024] |
75.2 µs | 86.9 µs | -13.51% |
| ❌ | WallTime | dict_canonicalize_gt_u8_avx2[1000000] |
422.4 µs | 480.2 µs | -12.05% |
| ❌ | Simulation | decompress_rd[f32, (2000, 0.0)] |
87.5 µs | 98.8 µs | -11.4% |
| ❌ | Simulation | filter_list_primitive_short[AllSelected] |
110.3 µs | 123 µs | -10.3% |
| ⚡ | Simulation | density_sweep_single_slice[0.9] |
50.8 µs | 32.6 µs | +55.85% |
| ⚡ | Simulation | decompress[alp_for_bp_f64] |
77 µs | 61.7 µs | +24.69% |
| 🆕 | Simulation | materialize_fused |
N/A | 789.7 µs | N/A |
| 🆕 | Simulation | materialize_levelwise |
N/A | 900 µs | N/A |
| 🆕 | Simulation | materialize_streaming |
N/A | 799.3 µs | N/A |
| 🆕 | Simulation | sum_fused |
N/A | 455.9 µs | N/A |
| 🆕 | Simulation | sum_levelwise |
N/A | 1.1 ms | N/A |
| 🆕 | Simulation | sum_streaming |
N/A | 468.4 µs | N/A |
| 🆕 | Simulation | materialize_fused |
N/A | 405.4 µs | N/A |
| 🆕 | Simulation | materialize_levelwise |
N/A | 378.5 µs | N/A |
| 🆕 | Simulation | materialize_streaming |
N/A | 381.5 µs | N/A |
| 🆕 | Simulation | sum_fused |
N/A | 227 µs | N/A |
| 🆕 | Simulation | sum_levelwise |
N/A | 451 µs | N/A |
| 🆕 | Simulation | sum_streaming |
N/A | 192.1 µs | N/A |
| ... | ... | ... | ... | ... | ... |
ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/decompression-chunked-iteration-u3kcns (e7b7c8d) with develop (9a08b82)
Footnotes
-
503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Introduce VTable::decompress_chunks: a push-based streaming decompression API that walks an array's decompressed values in cache-resident ~1024 element chunks without materializing the full array. Chunks flow through a ChunkSink chain: leaf encodings stream their existing unpack scratch buffers, and wrapper encodings compose by interposing a stack-allocated sink adapter and recursing into their child through the erased ArrayRef entry point, so the mechanism composes across arbitrary encoding trees with no per-element dynamic dispatch and no heap state on the way down. The default implementation executes to canonical and streams the result (the two-pass baseline), so the method is always available. BitPacked overrides it to stream FastLanes blocks straight from its unpack scratch (applying patches per block via a cursor), and FoR overrides it by adding the reference value in place per chunk before forwarding. On 4Mi u32 FoR-over-BitPacked, a streaming sum through the vtable path runs ~1.7x faster than fused decompress-then-iterate, within ~22% of a fully monomorphized loop doing identical per-chunk work; the pure dynamic dispatch cost measures ~0.06ns per element. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
When a FoR array's child is BitPacked (and the reference is unsigned, mirroring the dispatch in decompress), FoR::decompress_chunks now streams each FastLanes block through the fused FoRStrategy unchecked_unfor_pack kernel instead of wrapping the child's chunk stream with an in-place add pass. Patch values get the reference applied when the patch cursor list is built. The generic sink-composition path remains for any other child. The block streaming and patch-cursor loop is shared between BitPacked's plain path and the fused FoR path via stream_unpacked_chunks, generic over the UnpackStrategy. On 4Mi u32 FoR-over-BitPacked, a streaming sum through the vtable path now matches the hand-written monomorphized fused loop within noise (1.35ms vs 1.36ms median), i.e. the same speed as the unpack_map-style kernels, while the generic composition (measured via a signed reference) stays available for arbitrary children. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Constant streams by filling a single stack-resident 1024-element scratch chunk with the constant and re-emitting it across the array (refilled per emission, since sinks may mutate chunks in place), so the array is never materialized. Patched streams its inner child through the erased decompress_chunks entry point and interposes a sink adapter that overwrites patched rows in each chunk before forwarding. The lane-transposed patch layout is flattened once up front into row-sorted (row, value) pairs so the per-chunk work is a single cursor advance. This composes with any inner encoding: Patched(Constant) patches each re-emitted constant chunk, Patched(BitPacked) patches each unpacked FastLanes block. On 4Mi u32 Patched-over-Constant with sparse patches, a streaming sum runs 3.6x faster than execute-then-iterate (0.59ms vs 2.12ms median) since the 16MB base buffer is never written. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Add benches comparing the chunked decompress_chunks path on Patched(Constant) against the sparse decompression baseline (execute: canonicalize the constant into a full buffer, then scatter patches), for both pure materialization and materialize-then-sum. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The default VTable::decompress_chunks previously fell back to executing the array to canonical and streaming the materialized result, which silently defeated the purpose of the API: decompressing and transforming blocks while they are L1-resident. Streaming is now advertised via VTable::supports_decompress_chunks (default false); wrapper encodings propagate the check through the children they stream from, so support of the whole tree is decided before any chunk is emitted. ArrayRef::decompress_chunks errors on unsupported trees without emitting anything, and the materializing two-pass fallback moves to the explicitly named ArrayRef::decompress_chunks_or_materialize. Primitive gains a real streaming implementation (it is already decompressed; chunks are copied through one reusable L1 scratch buffer), Constant and BitPacked advertise support directly, FoR advertises support when its fused BitPacked path applies or its child supports streaming, and Patched advertises support when its inner array does. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Add a stream-to-canonical shortcut to the iterative executor: when the whole encoding tree supports chunked decompression (the capability cascades — each encoding only advertises support when the children it streams from do) and the tree is at least two streaming levels deep, execute_until decompresses blocks straight into the canonical builder via execute_via_chunks while each block is L1-resident, instead of materializing a full intermediate buffer per encoding level. A lone streaming leaf keeps its normal execute path, which already decodes directly into the output. Benchmarked on 4Mi-element trees via execute::<PrimitiveArray> with the shortcut toggled on/off in one process (medians): - FoR(signed ref) over BitPacked: 3.87ms -> 3.05ms (-21%) - Patched(FoR(BitPacked)) with 0.1% exceptions: 4.72ms -> 3.30ms (-30%) - FoR(unsigned ref) over BitPacked (fused execute baseline): 4.41ms -> 3.01ms (-32%) A process-global toggle (set_chunked_execute_enabled, doc-hidden) exists so benchmarks and tests can compare both executor paths; an equivalence test checks the shortcut against level-wise execution on a nullable multi-level tree. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Honor VORTEX_CHUNKED_EXECUTE=0 at process start so benchmark binaries can compare the executor with and without the stream-to-canonical shortcut without recompiling; set_chunked_execute_enabled still overrides at runtime. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
VORTEX_CHUNKED_EXECUTE_TRACE=1 prints, for every non-canonical array reaching the executor's step 2c check, whether the stream-to-canonical shortcut fires along with the tree's length, root encoding, and child encodings. Used to establish which TPC-H scan trees are streaming-capable; to be removed (or demoted to tracing) once the investigation concludes. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The TPC-H trace established its conclusion: on Q1/Q6 the shortcut fires 288 times, almost entirely on 65536-row FoR(BitPacked) trees, but that is only ~6% of non-canonical executor work. The dominant trees are wrapped in vortex.filter (1656, including 736x Filter(BitPacked)) and vortex.decimal_byte_parts (828), neither of which implements decompress_chunks yet — which explains the SQL-neutral benchmark result as an encoding-coverage gap rather than a batch-size effect. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
2d9d407 to
9e5874f
Compare
Filter now streams: the child streams its blocks, each block is compacted in place against the mask slice covering it (output index is always <= input index within a block), and the surviving prefix is forwarded. The mask's selection is resolved once up front and walked with a cursor, with a mean-run-length heuristic choosing between run-copying and per-index gathering, so no per-chunk mask slicing or allocation happens. Benchmarking this answered the question it was meant to answer: streaming a filter is NOT faster than the existing path. On Filter(BitPacked) at 65,536 rows (the dominant TPC-H scan tree), streaming matches level-wise for materialization and is 10-35% slower for consumption except at very high selectivity. The reason is structural and now understood: the level-wise path decodes the child directly into the output buffer via decode_into and compacts in place within it, so there is no intermediate for streaming to eliminate -- streaming only adds a scratch-to-output copy. Two consequences: - should_execute_via_chunks now requires a cardinality-preserving child, which keeps the executor from picking the streaming path for Filter trees (it would have regressed them by up to 2.6x). - The executor shortcut is disabled by default and enabled with VORTEX_CHUNKED_EXECUTE=1. Re-measuring on a quiet machine (three rounds) shows it is only a reliable win where the level-wise path does an extra full-buffer pass: signed FoR(BitPacked), which runs a separate wrapping-add pass, is 10-25% faster streaming, while fused FoR(BitPacked) and Patched(FoR(BitPacked)) are ~5-10% slower. The earlier 21-32% figures were measured on a loaded machine and did not reproduce. The streaming consumption win (sum without materializing: 1.29ms vs 3.36ms two-pass on 4Mi rows) is unaffected and remains the API's value, along with Patched(Constant) where the base buffer is never written. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Deeper stacks are where streaming pays: each additional encoding level costs level-wise execution a full-length intermediate buffer, but costs streaming only one more pass over an L1-resident block. Measured on 4Mi-row nested FoR-over-BitPacked stacks (medians, repeated rounds; marginal cost per added level 0.93ms level-wise vs 0.51ms streaming): chain len 2: 2.57ms vs 2.58ms parity chain len 3: 3.14ms vs 2.74ms 1.15x chain len 5: 5.26ms vs 4.00ms 1.32x chain len 9: 9.07ms vs 6.16ms 1.47x Streaming is also far more predictable: p100/median is ~1.2x versus 2.5-5x for level-wise, which repeatedly allocates full-length buffers. So instead of a blanket on/off default, the executor now measures the tree: streaming_chain_len counts consecutive streaming-capable nodes from the root, following only children that stream, are not already canonical, and preserve row count, and the shortcut fires at MIN_STREAMING_CHAIN nodes. The cardinality rule keeps selection encodings such as Filter out of the executor path without a special case, since their level-wise kernels already decode into the output and compact in place. VORTEX_CHUNKED_EXECUTE=0 remains as a kill switch. Equivalence tests now drive execute_via_chunks directly so they exercise streaming at every depth rather than depending on the heuristic, and a new test pins the depth rule (including that filter push-down beneath a deep stack still streams the levels above it). Also rewrites the chunk_iter module docs to state the vtable API contract and the cost model in one place. Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk> Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
9e5874f to
377d029
Compare
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
Port chunked decompression onto the blocked FoR encoding, which now has one reference per 1024-element chunk and an offset into the first chunk: - Fused FoR+BitPacked streaming now unpacks with `unchecked_unfor_pack` per chunk, for signed and unsigned types alike, whenever the references are constant or the BitPacked chunks line up with the FoR chunks, mirroring `decompress`. - Generic composition adds each FoR chunk's reference in place, splitting a streamed child chunk that spans two FoR chunks. - BitPacked streaming uses the caller-owned unpack scratch API. - Outputs are allocated through the execution context's allocator. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
…ed and Slice Extend `decompress_chunks` beyond FoR, BitPacked, Patched, Filter, Constant and Primitive: - Leaves that generate their values stream from one stack scratch chunk (`stream_from_fn`): Sparse (fill plus patches), RunEnd (runs expanded per chunk) and Sequence. - Wrappers transform their child's stream in place: ALP decodes integers into floats and patches them, ZigZag decodes into signed integers. `ChunkMut::retype` re-tags a chunk as a type of the same width. - Dict gathers each chunk of streamed codes from its values, tolerating out-of-bounds codes only where the code is null. - RLE and Delta decode whole FastLanes blocks. `BlockDecodeSink` regroups the child's stream into aligned 1024-row blocks, decodes each into scratch and forwards the rows within the array's offset window. - Chunked streams each chunk in turn. Slice resolves itself through the child's slice kernel, as the executor would, or else forwards only the sliced rows of its child's stream. - `ChunkPatches` resolves patches once and applies them with a cursor; BitPacked, FoR, Sparse, ALP and Patched share it. - Chunks hold at most 1024 rows, checked in debug builds, so adapters that change the value type convert into a fixed stack scratch chunk. - The executor checks the same-length, non-canonical spine below an array reaches the streaming depth before asking any encoding whether it streams, so shallow trees skip the support checks. Every encoding is tested against level-wise execution, including nulls, slices and unaligned child chunks, through `test_harness::assert_streams_like_execute`. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
…w ns Re-measured on nested FoR over BitPacked with mimalloc, streaming runs 1.07x, 1.10x and 1.13x as fast as level-wise execution at chains of 3, 4 and 9 nodes for 64Ki rows, and 1.64x to 2.64x for 4Mi rows. Chains of 2 lose 11% at 64Ki rows, so `MIN_STREAMING_CHAIN` drops from 4 to 3. That streams ALP over FoR over BitPacked, the common float tree, which the 2048-row `decompress[alp_for_bp_f64]` benchmark measures 21% faster. The executor's check now costs a few nanoseconds on trees it does not stream, so the other encoding-tree benchmarks are within 1% of develop: - non-primitive arrays are turned away by their dtype first, - the structural walk tests `is_canonical`, which costs ~36ns, only for children deep enough to count. RLE's per-chunk bounds check reduced its indices after widening each to `usize`, which does not vectorize. Reducing in the index type makes RLE decoding 1.7x faster level-wise and streaming alike. RunEnd streaming resolves its run ends once and walks them like a slice. Tests compute their level-wise reference with `without_chunked_execute`, which disables the shortcut on the calling thread only, since the executor now streams some of the trees they check. Also fix two rustdoc links. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
`vortex/benches/streaming_decompress.rs` measures eight encoding trees
six ways, `{materialize, sum} x {streaming, levelwise, fused}`, where
`fused` is a kernel written by hand for the tree, monomorphized with no
dynamic dispatch. Delta over FoR over BitPacked compares against a
special kernel that unpacks, undeltas and untransposes each block in L1.
Every fused kernel is checked against execution before timing.
The FastLanes bench keeps only what is specific to the executor: the
FoR depth sweep, now at 64Ki and 4Mi rows with mimalloc, and Filter over
BitPacked.
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
`execute_until::<M>` lets callers stop at an encoding they can use as is: the Arrow exporter executes until a run-end or dictionary array to export it zero-copy. With chains of three streaming, a run-end array behind two lazy slices streamed straight to canonical, skipping the run-end array the exporter was executing toward and failing `vortex-arrow`'s `nested_wrappers_reveal_run_end_fast_path`. The streaming chain now takes the activation's done predicate and the executor does not stream when a node the chain would decode through satisfies it, so that node is reached level-wise. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
|
CodSpeed Performance Analysis is red on 09fc129. The four Simulation regressions it flags come from comparing across different runtime environments (CodSpeed shows that warning on this run), not from this PR's code. Instruction counts measured locally with callgrind (per iteration, the difference between 201-iteration and 1-iteration runs of the same bench binary; develop vs this branch):
CodSpeed reports −16% and −19% for those two.
There is no code fix to make here, and a re-run won't change the environment mismatch, so these need acknowledging in CodSpeed. Generated by Claude Code |
When streaming materializes an array, producers wrote each chunk into a stack scratch chunk and the builder sink then copied it into the output buffer. That copy left streaming slower than level-wise decoding for single-pass producers (sparse fills, run-end runs, dictionary gathers). `ChunkSink` gains `destination`, which offers the memory the sink would copy rows into, and `accept_written`, which accepts rows the producer wrote there. `emit_with` decodes into the destination when one is offered and into scratch otherwise, so leaf producers (`stream_from_fn`, bit-packed and FoR unpacking, whole-block decoders, dictionary gathers) write the output buffer directly. In-place adapters (patches, FoR references, ALP, ZigZag, chunked offsets and slices) forward their inner sink's destination and transform the written values when they accept them. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
Scans cast columns to their table schema's nullability, so real TPC-H stacks put a lazy cast between FoR and its bit-packed values, e.g. FoR(Cast(Slice(BitPacked))). Casts did not stream, which cut these chains short of the streaming threshold. A cast now streams when it cannot fail: every value converts losslessly (the widenings the primitive cast kernel already treats as infallible) and the nullability only widens. Each chunk converts while L1-resident, straight into the output when materializing; a same-ptype cast forwards its child's chunks. Casts that check values against the target range keep the level-wise kernel. On TPC-H SF1 this streams the FoR(Cast(Slice|Filter(BitPacked))) stacks, about 40M of the rows the queries decode. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
A census of the arrays the TPC-H SF1 queries execute found the trees that stream: FoR(Cast(Filter(BitPacked))) and FoR(Cast(Slice(BitPacked))), where a scan's filter or slice and its nullability cast are pushed into FoR's values. Measure them, at three filter selectivities, against level-wise execution, and check at startup that they stream. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
Streaming a filter built the mask's runs and then its indices up front, a pass over the whole mask each, and compacted every chunk with scalar run copies or index gathers. On TPC-H that cost more than streaming saved: callgrind of SF1 Q21 counted 9.5% more instructions with streaming on, nearly all of it in the filter, while the level-wise filter compacts straight from the bitmap with the SIMD compress kernels. Each chunk now compacts in place against the mask bits covering its rows with the kernel the level-wise filter would pick for the mask, chosen once from its density, and falls back to the scalar bitmap walk where no kernel applies. The kernels' mask walk is generic over `MaskWords`, so the same code serves a whole mask and the bits of one chunk. Q21 now executes 0.1% more instructions with streaming on than off. Streaming Filter(BitPacked) at 64Ki rows runs 0.84x, 0.85x, 0.86x and 0.97x as long as level-wise execution at selectivities 1/16, 4/16, 8/16 and 12/16 (before: 0.83x, 1.16x, 1.30x and 1.34x). Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
Producers decoded into stack scratch chunks that were zeroed up front, 8 KiB each per stream. Since chunks decode straight into a materializing sink's destination, the scratch is often never touched, yet the zeroing remained: callgrind of the CodSpeed `decompress[alp_for_bp_f64]` benchmark (2048 rows) counted 32.0K instructions per iteration, 8.2K of them in memset, against 26.7K on develop. `ScratchChunk` zeroes itself on first use, and the bit-packed, leaf, block, dictionary and cast producers use it for their fallback scratch. The benchmark now counts 23.8K instructions per iteration. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
|
Following up on CodSpeed Performance Analysis for 01e5296. Everything else on this head is green.
The earlier "Rust build (with tokio dispatcher)" failure was the runner killing the build (exit 137, no compiler error); its retry passed. Generated by Claude Code |
Streaming a run-end array collected its ends into a Vec<usize> before expanding the runs, a separate pass and a full-length allocation that level-wise decoding avoids by reading the ends lazily. At 64Ki rows in runs of 16 that pass cost 49K of the 374K instructions per decode, which left streaming slower than level-wise decoding. Each run's end is now read from the stored ends as the stream reaches it. Materializing streams in 23.5us against 27.0us level-wise and 23.7us for a hand-fused kernel (before: 29.8us), and folding in 31us against 35us level-wise (before: 40us). Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
Hand-written kernels fused per encoding tree ran up to 1.86x faster than
streaming the same tree through `ChunkSink`s, mostly in consumers that fold
values rather than materialize them. Profiles put the cost in code that saw
chunks as slices of unknown length and in per-chunk validation, not in the
dynamic dispatch:
- `ChunkMut::as_block` views a full chunk as a fixed-size block, and
`emit_with` hands full chunks to producers' decode closures as blocks, so
both sides of the sink compile the common full chunk with a constant
length, as a kernel written for whole blocks does. Dictionary gathers
reslice their codes to the output's length for the same reason.
- RLE decodes bit-packed indices block by block itself rather than
streaming them up through BitPacked and regrouping them. `emit_block`,
which `BlockDecodeSink` already used, is public for decoders like it.
- RLE skips the per-chunk index bounds check when the indices' encoding
bounds them (bit-packed indices without patches are below
`1 << bit_width`), and otherwise reduces with AVX2 where available, which
has an unsigned 16-bit max the baseline target lacks. Level-wise RLE
decoding takes the bound too.
- `execute_via_chunks` treats a non-nullable tree as all valid without
asking its encodings, and does not re-check support its caller checked.
Wall time at 64Ki rows (medians of five runs), streaming time relative to
the hand-fused kernel, materialize / sum:
FoR(BitPacked) 1.05 / 1.23 -> 1.06 / 1.03
BitPacked + patches 0.96 / 1.26 -> 0.98 / 1.04
ALP(FoR(BitPacked)) 1.02 / 1.04 -> 0.99 / 1.04
Delta(FoR(BitPacked)) 1.05 / 1.05 -> 1.03 / 1.04
Dict(BitPacked codes) 0.91 / 1.15 -> 0.91 / 0.91
RLE(BitPacked indices) 1.25 / 1.31 -> 1.05 / 1.03
RunEnd 0.98 / 1.11 -> 0.98 / 0.79
Sparse 0.91 / 1.85 -> 0.94 / 0.93
Streaming now beats level-wise execution on every tree, Sparse sums by
1.9x.
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
- Dict: take the unchecked gather only when a chunk's smallest code is non-negative as well as its largest in bounds, so a negative signed code can no longer wrap to an out-of-bounds index. For unsigned codes the extra check folds away. - execute_via_chunks: the builder sink now requires chunks in order and checks the stream covered every row before finishing the builder, in release builds too, so a stream that skips rows is an error rather than uninitialized output. - BlockDecodeSink::finish rejects a child stream that ends part way through a block, which RLE and Delta call instead of silently dropping the trailing rows. - Wrappers stream their children through decompress_child_chunks, which skips the up-front support walk their own support check already did. - FoR's support check no longer special-cases BitPacked, which always streams. Benches compare against level-wise execution with the scoped without_chunked_execute; RLE tests use `?` instead of unwrap. Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
Streaming carried primitive values only, so decimal trees always executed level by level. Chunks now carry a fixed-width ValueType: a primitive, or the i128/i256 integers wide decimals store their values in, and any ChunkValue (every native primitive, i128, i256) can be streamed, gathered, filtered or written into a destination. The vtable's supports_decompress_chunks -> bool becomes decompress_chunks_type -> Option<ValueType>: the physical type an array would stream. A primitive array streams its dtype's ptype; a decimal array streams the integer type executing it stores its values in, which the dtype alone does not determine, so materializing a streamed decimal keeps the storage type level-wise execution would. - Decimal: the canonical leaf streams its values. - DecimalByteParts: without lower parts, forwards the most significant part's stream; with them, streams the most significant part and assembles each chunk with its lower words, straight into the output. - Constant, Dict, Chunked, Slice and Filter stream decimals. Chunked converts chunks stored in another type to the smallest type for the precision, as its level-wise builder does; Slice only takes a resolved slice that streams the same type. - execute_via_chunks builds a PrimitiveArray or DecimalArray, and the executor streams decimal trees whose streaming chain is deep enough. Materializing a decimal(38, 2) DecimalByteParts over FoR(BitPacked) runs at 0.75x of level-wise execution, summing it at 0.68x; the primitive trees are unchanged (geometric mean 0.996x, identical instruction counts). Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
|
This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
Summary
Adds
decompress_chunks, a vtable method that streams an encoding tree's decompressed fixed-width values (primitives, and the integers decimals are stored in) through aChunkSinkin L1-resident chunks of at most 1024 rows. Leaf encodings stream blocks straight out of their decode kernel; wrapper encodings interpose a stack-allocated sink adapter and recurse into their child, so a whole tree decodes and transforms one block at a time instead of materializing a full-length buffer per level. When the consumer materializes, sinks offer their output memory, so producers decode straight into the final buffer and the only full-length write is the output itself.This buys two things:
decompress_chunksis 1.1–1.9x faster than executing it and reading the result back on every tree benchmarked, and on par with a kernel written by hand for each tree (geometric mean 0.995x of the hand-fused time across all 16 materialize and sum cases, range 0.84–1.11x).The vtable API
ArrayRef::decompress_chunksrejects an unsupported tree without emitting a partial stream. Wrappers stream their children throughdecompress_child_chunks, which skips the walk their own support check already did. Callers who want a materializing fallback ask by name:decompress_chunks_or_materialize.i32), so streaming one to canonical keeps the storage level-wise execution would produce. AnyChunkValue(every native primitive,i128,i256) can be streamed, and the structural encodings (Slice, Filter, Chunked, Dict, Constant) are generic over it.ChunkMutis a type-erased&mut [T]tagged with itsValueType; sinks recover the typed slice once per chunk, or a&[T; 1024]for a full chunk (as_block), so per-chunk loops run a constant number of times.emit_withdecodes a chunk into the sink's destination when it offers one and into scratch otherwise. In-place wrappers forward their inner sink's destination and transform the written values inaccept_written, so a materialized tree is written once, by its leaf, and then transformed while L1-resident.0..lenexactly, hold at most 1024 rows and carry the array'sdecompress_chunks_type(all debug-checked; materializing checks order and coverage in release builds too, before exposing the builder's memory); validity is not streamed; primitive and decimal dtypes only.Changes
Encodings that stream
unchecked_unfor_pack, signed and unsigned alike. Otherwise adds each chunk's reference in place, splitting child chunks that span two FoR chunks.i128/i256, straight into the destination; the lower parts, full words that rarely compress, are read from their executed buffers.Shared machinery in
vortex-array/src/chunk_iter.rs:ValueType,ChunkValueandmatch_each_value_type!,emit_with(decode into the destination or scratch),ScratchChunk(stack scratch zeroed on first use),ChunkPatches(patches resolved once, applied with a cursor),stream_from_fn(leaf producers),BlockDecodeSink(decoders of whole aligned FastLanes blocks over an arbitrary child stream;finishrejects a stream that ends part way through a block) andemit_block(one such block, for decoders that read their input blocks themselves),ChunkMut::{retype, retype_to, narrow, as_block}. The filter's SIMD compress kernels now walk mask words through a smallMaskWordstrait, so the same kernels serve a whole mask and one chunk's bits (ChunkCompactor).Executor:
execute_untilcanonicalizes by streaming when the tree's streaming chain reachesMIN_STREAMING_CHAIN = 3nodes. The chain only follows children that stream, are not canonical and preserve row count. It never streams past a node the currentexecute_untiltarget matches, so callers that stop at an encoding to use it as is, such as the Arrow exporter's run-end and dictionary fast paths, still reach it. The check is a dtype test plus a structural walk that defers the ~36nsis_canonicalcall.VORTEX_CHUNKED_EXECUTE=0disables the shortcut.Also: RLE's per-chunk bounds check widened every index to
usizebefore reducing, which does not vectorize. Reducing in the index type makes RLE decoding 1.7x faster for level-wise execution as well (45 → 26µs per 64Ki rows). The reduction uses AVX2 where the CPU has it (runtime-detected), and is skipped where the indices' bit width already bounds them.Tests: every encoding is checked against level-wise execution through
test_harness::assert_streams_like_execute, covering nulls, slices with offsets, non-chunk-aligned lengths, patches, per-chunk FoR references (aligned and misaligned with the BitPacked chunks), child chunks that straddle RLE/Delta blocks, Slice with and without a slice kernel, casts that may and may not stream, filters of every density and run shape at four element widths (including masks sliced mid-byte), negative signed Dict codes, decimals of every storage type (i8toi256) through the Decimal leaf, Slice, Filter, Chunked, Dict and Constant, DecimalByteParts with and without lower parts (nullable, sliced, and chunked with mixed storage types), and the error-without-emitting contract. The harness also checks that a decimal streams the storage type executing it produces. Unit tests cover the builder sink rejecting gaps and the block decoder rejecting a partial trailing block. Executor tests cover the depth rule, that the shortcut matches level-wise execution, and that it stops at anexecute_untiltarget behind lazy slices.What real queries stream
A census of every array the 22 TPC-H SF1 queries execute (instrumentation not included in this PR) found the common stacks one or two levels deep:
FoR(BitPacked), bareBitPacked,DecimalByteParts(BitPacked), and comparisons over them. The stacks that do stream areFoR(Cast(Slice(BitPacked)))andFoR(Cast(Filter(BitPacked))): a scan's slice or filter and its nullability cast pushed into FoR's values. That needed two of the changes above: casts had to stream, and streaming a filter had to stop building the mask's indices and runs. Before the filter change, Q21 executed 9.5% more instructions with streaming on (callgrind); after, 0.1%. About 40M of the 1.2B rows the queries decode stream.TPC-H SF1 end to end (DataFusion, Vortex files, 4-core VM, medians of 32 runs per query, streaming on vs
VORTEX_CHUNKED_EXECUTE=0in ABBA order): total 1655 → 1662 ms (+0.4%). The 11 queries that stream anything run at a 0.987 geometric mean ratio, the 11 that stream nothing at 1.009: neutral within the ±5% per-query noise, as expected when 3% of decoded rows stream.The census's
DecimalByteParts(BitPacked)columns are narrow (no lower parts), whose level-wise execution already reuses the most significant part's buffer as the decimal values, so streaming them to canonical runs at parity; folding them without materializing is 1.6x faster (below).Benchmarks
Streaming vs level-wise vs hand-fused (
vortex/benches/streaming_decompress.rs, 64Ki rows, µs).fusedis a kernel written for exactly that tree, monomorphized end to end with no dynamic dispatch, decoding each block through every level in L1. For Delta(FoR(BitPacked)) it is the special kernel:unchecked_unfor_pack→undelta→untransposeper block.Wall time: 4-core cloud VM, median of the divan medians of five runs; bold where streaming beats level-wise. On this VM the same code's wall time moves up to ±40% between builds with code and data layout (callgrind instruction counts identical), so instruction counts follow.
Across the 16 cases with a hand-fused kernel, streaming runs at a 0.995x geometric mean of its time (0.84–1.11x); across all 18, at 0.81x of level-wise execution.
Callgrind instructions per iteration (streaming vs level-wise; the fused kernels are omitted):
Streaming executes about as many instructions as level-wise execution; where it is faster, it is faster because it removes the intermediate buffers' memory traffic. Dict executes more instructions streaming, a scalar gather from L1 in place of the level-wise SIMD gather, and still runs faster.
Decimals:
DecimalBytePartsover a FoR(BitPacked) most significant part, 64Ki rows, µs, five runs (no hand-fused kernel):i128)Streaming the wide one assembles each chunk of most significant parts with its lower words while L1-resident, instead of materializing the parts and then assembling. Generalizing from primitives to fixed-width values leaves the primitive trees unchanged: 0.996x geometric mean wall time over the 18 cases above, identical instruction counts.
Real scan stacks from the TPC-H census (64Ki rows, µs, streaming vs level-wise):
The 4/16 level-wise range spans three builds with identical instruction counts.
Filter(BitPacked) at 64Ki rows, streaming time relative to level-wise, before → after compacting with the SIMD kernels: 1/16 selected 0.83 → 0.84, 4/16 1.16 → 0.85, 8/16 1.30 → 0.86, 12/16 1.34 → 0.97.
Executor depth sweep: nested FoR over BitPacked, streaming speedup over level-wise (
encodings/fastlanes/benches/chunked_decompress.rs, mimalloc):Chain 2 is FoR over BitPacked, where both paths run the same fused kernel, so the executor keeps streaming to chains of three.
Against develop (2048-row CodSpeed benches, callgrind instructions per iteration):
decompress[alp_for_bp_f64]26,740 → 23,772 (−11%);decompress[for_bp_u64]9,412 → 9,526 andbitpacked_decompress_u329,894 → 9,897, the executor's structural check. CodSpeed's Simulation runs flagfor_bp_u64,bitpacked_decompress_u32,baseline_ltanddecompress_rdby 11–19% on every revision, against a base measured in a different runtime environment, which the instruction counts do not reproduce. It also flaggedtake[all_null_len65536_run4_take2048](−11%), which runs no streaming code: 15,014 instructions per iteration on develop, 15,082 here (+0.45%), with equal cache misses.Where streaming does not win. Materializing FoR(BitPacked), BitPacked and other chain-2 trees is at parity with level-wise execution, so the executor does not stream them. Against the hand-fused kernels, materializing RLE, Delta and FoR(BitPacked) is 5–11% slower across runs: the dynamic boundary between encodings costs an indirect call and a destination handshake per 1024-row block, which a fused kernel inlines away.
Checks
cargo nextest run: vortex-array, vortex-fastlanes, vortex-alp, vortex-zigzag, vortex-sparse, vortex-runend, vortex-sequence, vortex-decimal-byte-parts (--features vortex-array/_test-harness): 5197 passed, 1 skipped;cargo test --docon vortex-array and vortex-decimal-byte-parts.cargo clippy --all-targets --all-features -- -D warningson all eight crates above,cargo clippy -p vortex --bench streaming_decompress -- -D warnings.cargo check -p vortex-array --target aarch64-unknown-linux-gnufor the NEON compress kernels, now also instantiated for 16- and 32-byte values.cargo +nightly-2026-09-10 fmt,typoson changed files.API Changes
New public API in
vortex-array: thechunk_itermodule (ValueType,ChunkValue,match_each_value_type!,ChunkSinkwithaccept,destinationandaccept_written,ChunkMut,ScratchChunk,emit_with,emit_block,ChunkPatches,BlockDecodeSink,stream_from_fn,stream_slice_chunks,execute_via_chunks,decompress_chunks_via_canonical,DECOMPRESS_CHUNK_LEN),ArrayRef::{decompress_chunks_type, supports_decompress_chunks, decompress_chunks, decompress_child_chunks, decompress_chunks_or_materialize}, two defaultedVTablemethods (decompress_chunks_type,decompress_chunks), andtest_harness::assert_streams_like_execute. Behavior change:execute_untilcanonicalizes primitive and decimal trees whose streaming chain reaches three nodes by streaming (same logical results; disable withVORTEX_CHUNKED_EXECUTE=0).🤖 Generated with Claude Code
https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7