Skip to content

Streaming chunked decompression via a composable vtable method - #9614

Closed
joseph-isaacs wants to merge 25 commits into
developfrom
claude/decompression-chunked-iteration-u3kcns
Closed

joseph-isaacs wants to merge 25 commits into
developfrom
claude/decompression-chunked-iteration-u3kcns

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds decompress_chunks, a vtable method that streams an encoding tree's decompressed fixed-width values (primitives, and the integers decimals are stored in) through a ChunkSink in L1-resident chunks of at most 1024 rows. Leaf encodings stream blocks straight out of their decode kernel; wrapper encodings interpose a stack-allocated sink adapter and recurse into their child, so a whole tree decodes and transforms one block at a time instead of materializing a full-length buffer per level. When the consumer materializes, sinks offer their output memory, so producers decode straight into the final buffer and the only full-length write is the output itself.

This buys two things:

  1. Consumers that never materialize fold values straight out of L1. Summing a 64Ki-row tree through decompress_chunks is 1.1–1.9x faster than executing it and reading the result back on every tree benchmarked, and on par with a kernel written by hand for each tree (geometric mean 0.995x of the hand-fused time across all 16 materialize and sum cases, range 0.84–1.11x).
  2. Deep encoding trees canonicalize faster, because each level costs level-wise execution a whole intermediate buffer but costs streaming one more pass over an L1 block. The executor streams fixed-width trees whose streaming chain reaches three nodes: ALP over FoR over BitPacked, the common float tree, the FoR(Cast(Filter|Slice(BitPacked))) stacks TPC-H scans produce, and wide decimals as DecimalByteParts over FoR(BitPacked), which materialize 1.34x faster.

The vtable API

pub trait VTable {
    /// The type of values this encoding (recursively) streams without materializing,
    /// or None if it cannot. Default: None.
    fn decompress_chunks_type(array: ArrayView<'_, Self>) -> Option<ValueType> { None }

    /// Stream decompressed values through `sink` in cache-resident chunks.
    /// Default: reports unsupported, never a silent materializing fallback.
    fn decompress_chunks(
        array: ArrayView<'_, Self>,
        ctx: &mut ExecutionCtx,
        sink: &mut dyn ChunkSink,
    ) -> VortexResult<()>;
}

/// A primitive, or a 128- or 256-bit integer, as wide decimals are stored.
pub enum ValueType { Primitive(PType), I128, I256 }

pub trait ChunkSink {
    fn accept(&mut self, chunk: ChunkMut<'_>, row_range: Range<usize>) -> VortexResult<()>;

    /// Offer the memory the sink would copy these rows into. Default: none.
    fn destination(&mut self, row_range: Range<usize>) -> Option<ChunkMut<'_>> { None }
    /// Accept rows the producer wrote into the destination.
    fn accept_written(&mut self, row_range: Range<usize>) -> VortexResult<()>;
}
  • Support and the value type are answerable up front and cascade: a wrapper only advertises support when the children it streams from do, so ArrayRef::decompress_chunks rejects an unsupported tree without emitting a partial stream. Wrappers stream their children through decompress_child_chunks, which skips the walk their own support check already did. Callers who want a materializing fallback ask by name: decompress_chunks_or_materialize.
  • Fixed-width, not just primitive. A primitive array streams its dtype's ptype. A decimal array streams the integer type executing it would store its values in, which its dtype does not determine (the compressor stores a decimal(38, 2) column of small values as i32), so streaming one to canonical keeps the storage level-wise execution would produce. Any ChunkValue (every native primitive, i128, i256) can be streamed, and the structural encodings (Slice, Filter, Chunked, Dict, Constant) are generic over it.
  • Dispatch is per chunk, never per value. ChunkMut is a type-erased &mut [T] tagged with its ValueType; sinks recover the typed slice once per chunk, or a &[T; 1024] for a full chunk (as_block), so per-chunk loops run a constant number of times.
  • Chunks are mutable producer scratch, so wrappers transform in place (FoR adds references, ALP decodes ints to floats, Patched overwrites rows) with no extra buffer.
  • Destinations remove the final copy. emit_with decodes a chunk into the sink's destination when it offers one and into scratch otherwise. In-place wrappers forward their inner sink's destination and transform the written values in accept_written, so a materialized tree is written once, by its leaf, and then transformed while L1-resident.
  • Contract: chunks are in order, contiguous, cover 0..len exactly, hold at most 1024 rows and carry the array's decompress_chunks_type (all debug-checked; materializing checks order and coverage in release builds too, before exposing the builder's memory); validity is not streamed; primitive and decimal dtypes only.

Changes

Encodings that stream

Encoding How
BitPacked Unpacks each FastLanes block into the destination (or scratch); patches applied per block with a cursor.
FoR Over BitPacked (constant references, or per-chunk references whose chunks line up), unpacks with unchecked_unfor_pack, signed and unsigned alike. Otherwise adds each chunk's reference in place, splitting child chunks that span two FoR chunks.
ALP Decodes each chunk of integers into floats in place, then patches it.
ZigZag Decodes into signed integers in place.
Delta Regroups the deltas' stream into whole blocks, undeltas each against its lane bases and untransposes it in L1.
RLE Over bit-packed indices, unpacks each block of indices itself and gathers its run values, skipping the bounds check when the bit width already bounds the indices. Over other indices, regroups their stream into whole blocks.
Dict Gathers each chunk of streamed codes from the primitive or decimal values; out-of-bounds (or negative) codes are allowed only where null.
Cast Lossless primitive widenings with widening nullability only (casts that cannot fail); converts each chunk, or forwards a same-ptype cast's chunks. Other casts keep the level-wise kernel's range checks.
RunEnd, Sparse, Sequence, Constant, Primitive, Decimal Leaves: fill each chunk in the destination (or scratch). RunEnd reads its ends in their stored type. A decimal Constant streams in the type canonicalizing it stores.
DecimalByteParts Without lower parts, forwards the most significant part's stream: its integers are the values. With lower parts, streams the most significant part and assembles each chunk with its rows' lower words into i128/i256, straight into the destination; the lower parts, full words that rarely compress, are read from their executed buffers.
Patched Overwrites patched rows of each streamed chunk.
Filter Compacts each block in place with the SIMD compress kernel the level-wise filter would pick for the mask, straight from the mask's bits.
Chunked Streams each chunk with its rows shifted. Decimal chunks stored in another type than the array's (the smallest for its precision when there are several chunks, as the level-wise builder uses) are converted per chunk.
Slice Resolves itself through the child's slice kernel, as the executor would, when the resolved slice streams the same type, or forwards only the sliced rows of the child's stream.

Shared machinery in vortex-array/src/chunk_iter.rs: ValueType, ChunkValue and match_each_value_type!, emit_with (decode into the destination or scratch), ScratchChunk (stack scratch zeroed on first use), ChunkPatches (patches resolved once, applied with a cursor), stream_from_fn (leaf producers), BlockDecodeSink (decoders of whole aligned FastLanes blocks over an arbitrary child stream; finish rejects a stream that ends part way through a block) and emit_block (one such block, for decoders that read their input blocks themselves), ChunkMut::{retype, retype_to, narrow, as_block}. The filter's SIMD compress kernels now walk mask words through a small MaskWords trait, so the same kernels serve a whole mask and one chunk's bits (ChunkCompactor).

Executor: execute_until canonicalizes by streaming when the tree's streaming chain reaches MIN_STREAMING_CHAIN = 3 nodes. The chain only follows children that stream, are not canonical and preserve row count. It never streams past a node the current execute_until target matches, so callers that stop at an encoding to use it as is, such as the Arrow exporter's run-end and dictionary fast paths, still reach it. The check is a dtype test plus a structural walk that defers the ~36ns is_canonical call. VORTEX_CHUNKED_EXECUTE=0 disables the shortcut.

Also: RLE's per-chunk bounds check widened every index to usize before reducing, which does not vectorize. Reducing in the index type makes RLE decoding 1.7x faster for level-wise execution as well (45 → 26µs per 64Ki rows). The reduction uses AVX2 where the CPU has it (runtime-detected), and is skipped where the indices' bit width already bounds them.

Tests: every encoding is checked against level-wise execution through test_harness::assert_streams_like_execute, covering nulls, slices with offsets, non-chunk-aligned lengths, patches, per-chunk FoR references (aligned and misaligned with the BitPacked chunks), child chunks that straddle RLE/Delta blocks, Slice with and without a slice kernel, casts that may and may not stream, filters of every density and run shape at four element widths (including masks sliced mid-byte), negative signed Dict codes, decimals of every storage type (i8 to i256) through the Decimal leaf, Slice, Filter, Chunked, Dict and Constant, DecimalByteParts with and without lower parts (nullable, sliced, and chunked with mixed storage types), and the error-without-emitting contract. The harness also checks that a decimal streams the storage type executing it produces. Unit tests cover the builder sink rejecting gaps and the block decoder rejecting a partial trailing block. Executor tests cover the depth rule, that the shortcut matches level-wise execution, and that it stops at an execute_until target behind lazy slices.

What real queries stream

A census of every array the 22 TPC-H SF1 queries execute (instrumentation not included in this PR) found the common stacks one or two levels deep: FoR(BitPacked), bare BitPacked, DecimalByteParts(BitPacked), and comparisons over them. The stacks that do stream are FoR(Cast(Slice(BitPacked))) and FoR(Cast(Filter(BitPacked))): a scan's slice or filter and its nullability cast pushed into FoR's values. That needed two of the changes above: casts had to stream, and streaming a filter had to stop building the mask's indices and runs. Before the filter change, Q21 executed 9.5% more instructions with streaming on (callgrind); after, 0.1%. About 40M of the 1.2B rows the queries decode stream.

TPC-H SF1 end to end (DataFusion, Vortex files, 4-core VM, medians of 32 runs per query, streaming on vs VORTEX_CHUNKED_EXECUTE=0 in ABBA order): total 1655 → 1662 ms (+0.4%). The 11 queries that stream anything run at a 0.987 geometric mean ratio, the 11 that stream nothing at 1.009: neutral within the ±5% per-query noise, as expected when 3% of decoded rows stream.

The census's DecimalByteParts(BitPacked) columns are narrow (no lower parts), whose level-wise execution already reuses the most significant part's buffer as the decimal values, so streaming them to canonical runs at parity; folding them without materializing is 1.6x faster (below).

Benchmarks

Streaming vs level-wise vs hand-fused (vortex/benches/streaming_decompress.rs, 64Ki rows, µs). fused is a kernel written for exactly that tree, monomorphized end to end with no dynamic dispatch, decoding each block through every level in L1. For Delta(FoR(BitPacked)) it is the special kernel: unchecked_unfor_pack → undelta → untranspose per block.

Wall time: 4-core cloud VM, median of the divan medians of five runs; bold where streaming beats level-wise. On this VM the same code's wall time moves up to ±40% between builds with code and data layout (callgrind instruction counts identical), so instruction counts follow.

tree materialize: fused level-wise streaming sum: fused level-wise streaming
FoR(BitPacked) u32 11.4 12.1 12.2 15.2 23.6 15.5
BitPacked + patches u32 13.3 13.2 13.0 16.3 24.5 17.0
ALP(FoR(BitPacked)) f64 45.1 49.0 46.8 80.4 97.5 83.6
Delta(FoR(BitPacked)) u32 79.4 87.9 84.0 87.4 100.3 90.2
Dict(BitPacked codes) i64 25.9 26.6 23.6 29.9 34.5 26.7
RLE(BitPacked indices) u32 20.3 25.8 22.5 28.3 37.3 30.1
RunEnd i64 23.7 27.1 23.8 27.4 35.5 23.2
Sparse i64 15.8 15.6 14.9 13.4 23.7 12.3
FoR(Cast(Slice(BitPacked + patches))) i64 – 53.1 48.3 – 61.6 36.6

Across the 16 cases with a hand-fused kernel, streaming runs at a 0.995x geometric mean of its time (0.84–1.11x); across all 18, at 0.81x of level-wise execution.

Callgrind instructions per iteration (streaming vs level-wise; the fused kernels are omitted):

tree materialize: level-wise streaming sum: level-wise streaming
FoR(BitPacked) u32 107K 109K (+2%) 254K 234K (-8%)
BitPacked + patches u32 98K 108K (+10%) 246K 233K (-5%)
ALP(FoR(BitPacked)) f64 521K 516K (-1%) 611K 607K (-1%)
Delta(FoR(BitPacked)) u32 1333K 1352K (+1%) 1481K 1478K (-0%)
Dict(BitPacked codes) i64 205K 325K (+58%) 320K 406K (+27%)
RLE(BitPacked indices) u32 304K 307K (+1%) 452K 433K (-4%)
RunEnd i64 336K 346K (+3%) 450K 427K (-5%)
Sparse i64 109K 81K (-25%) 223K 163K (-27%)
FoR(Cast(Slice(BitPacked + patches))) i64 348K 366K (+5%) 461K 433K (-6%)

Streaming executes about as many instructions as level-wise execution; where it is faster, it is faster because it removes the intermediate buffers' memory traffic. Dict executes more instructions streaming, a scalar gather from L1 in place of the level-wise SIMD gather, and still runs faster.

Decimals: DecimalByteParts over a FoR(BitPacked) most significant part, 64Ki rows, µs, five runs (no hand-fused kernel):

tree materialize: level-wise streaming sum: level-wise streaming
decimal(18, 2), no lower parts 24.1 23.5 32.0 20.1
decimal(38, 2), one lower part (i128) 83.8 62.4 124.0 83.9

Streaming the wide one assembles each chunk of most significant parts with its lower words while L1-resident, instead of materializing the parts and then assembling. Generalizing from primitives to fixed-width values leaves the primitive trees unchanged: 0.996x geometric mean wall time over the 18 cases above, identical instruction counts.

Real scan stacks from the TPC-H census (64Ki rows, µs, streaming vs level-wise):

real stack materialize: level-wise streaming sum: level-wise streaming
FoR(Cast(Filter(BitPacked))), 1/16 selected 23.8 23.4 24.4 22.1
FoR(Cast(Filter(BitPacked))), 4/16 selected 50.0–73.0 36.8 51.7–73.3 35.3
FoR(Cast(Filter(BitPacked))), 12/16 selected 55.1 44.5 63.9 42.3
FoR(Cast(Slice(BitPacked + patches))) 53.1 48.3 61.6 36.6

The 4/16 level-wise range spans three builds with identical instruction counts.

Filter(BitPacked) at 64Ki rows, streaming time relative to level-wise, before → after compacting with the SIMD kernels: 1/16 selected 0.83 → 0.84, 4/16 1.16 → 0.85, 8/16 1.30 → 0.86, 12/16 1.34 → 0.97.

Executor depth sweep: nested FoR over BitPacked, streaming speedup over level-wise (encodings/fastlanes/benches/chunked_decompress.rs, mimalloc):

rows chain 2 chain 3 chain 4 chain 5 chain 9
64Ki 1.00x 1.15x 1.18x 1.19x 1.15x
4Mi 0.94x 1.46x 1.85x 2.10x 2.45x

Chain 2 is FoR over BitPacked, where both paths run the same fused kernel, so the executor keeps streaming to chains of three.

Against develop (2048-row CodSpeed benches, callgrind instructions per iteration): decompress[alp_for_bp_f64] 26,740 → 23,772 (−11%); decompress[for_bp_u64] 9,412 → 9,526 and bitpacked_decompress_u32 9,894 → 9,897, the executor's structural check. CodSpeed's Simulation runs flag for_bp_u64, bitpacked_decompress_u32, baseline_lt and decompress_rd by 11–19% on every revision, against a base measured in a different runtime environment, which the instruction counts do not reproduce. It also flagged take[all_null_len65536_run4_take2048] (−11%), which runs no streaming code: 15,014 instructions per iteration on develop, 15,082 here (+0.45%), with equal cache misses.

Where streaming does not win. Materializing FoR(BitPacked), BitPacked and other chain-2 trees is at parity with level-wise execution, so the executor does not stream them. Against the hand-fused kernels, materializing RLE, Delta and FoR(BitPacked) is 5–11% slower across runs: the dynamic boundary between encodings costs an indirect call and a destination handshake per 1024-row block, which a fused kernel inlines away.

Checks

  • cargo nextest run: vortex-array, vortex-fastlanes, vortex-alp, vortex-zigzag, vortex-sparse, vortex-runend, vortex-sequence, vortex-decimal-byte-parts (--features vortex-array/_test-harness): 5197 passed, 1 skipped; cargo test --doc on vortex-array and vortex-decimal-byte-parts.
  • cargo clippy --all-targets --all-features -- -D warnings on all eight crates above, cargo clippy -p vortex --bench streaming_decompress -- -D warnings.
  • cargo check -p vortex-array --target aarch64-unknown-linux-gnu for the NEON compress kernels, now also instantiated for 16- and 32-byte values.
  • cargo +nightly-2026-09-10 fmt, typos on changed files.
  • Not run: workspace-wide clippy/doc, other SQL benchmark suites.

API Changes

New public API in vortex-array: the chunk_iter module (ValueType, ChunkValue, match_each_value_type!, ChunkSink with accept, destination and accept_written, ChunkMut, ScratchChunk, emit_with, emit_block, ChunkPatches, BlockDecodeSink, stream_from_fn, stream_slice_chunks, execute_via_chunks, decompress_chunks_via_canonical, DECOMPRESS_CHUNK_LEN), ArrayRef::{decompress_chunks_type, supports_decompress_chunks, decompress_chunks, decompress_child_chunks, decompress_chunks_or_materialize}, two defaulted VTable methods (decompress_chunks_type, decompress_chunks), and test_harness::assert_streams_like_execute. Behavior change: execute_until canonicalizes primitive and decimal trees whose streaming chain reaches three nodes by streaming (same logical results; disable with VORTEX_CHUNKED_EXECUTE=0).

🤖 Generated with Claude Code

https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7

@codspeed

codspeed Bot commented Aug 25, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 2.75%

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 2 improved benchmarks
❌ 6 regressed benchmarks
✅ 2095 untouched benchmarks
🆕 108 new benchmarks
⏩ 503 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ Simulation decompress[for_bp_u64] 44.9 µs 54.9 µs -18.3%
❌ Simulation bitpacked_decompress_u32 51.4 µs 61.7 µs -16.64%
❌ Simulation baseline_lt[4, 1024] 75.2 µs 86.9 µs -13.51%
❌ WallTime dict_canonicalize_gt_u8_avx2[1000000] 422.4 µs 480.2 µs -12.05%
❌ Simulation decompress_rd[f32, (2000, 0.0)] 87.5 µs 98.8 µs -11.4%
❌ Simulation filter_list_primitive_short[AllSelected] 110.3 µs 123 µs -10.3%
⚡ Simulation density_sweep_single_slice[0.9] 50.8 µs 32.6 µs +55.85%
⚡ Simulation decompress[alp_for_bp_f64] 77 µs 61.7 µs +24.69%
🆕 Simulation materialize_fused N/A 789.7 µs N/A
🆕 Simulation materialize_levelwise N/A 900 µs N/A
🆕 Simulation materialize_streaming N/A 799.3 µs N/A
🆕 Simulation sum_fused N/A 455.9 µs N/A
🆕 Simulation sum_levelwise N/A 1.1 ms N/A
🆕 Simulation sum_streaming N/A 468.4 µs N/A
🆕 Simulation materialize_fused N/A 405.4 µs N/A
🆕 Simulation materialize_levelwise N/A 378.5 µs N/A
🆕 Simulation materialize_streaming N/A 381.5 µs N/A
🆕 Simulation sum_fused N/A 227 µs N/A
🆕 Simulation sum_levelwise N/A 451 µs N/A
🆕 Simulation sum_streaming N/A 192.1 µs N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing claude/decompression-chunked-iteration-u3kcns (e7b7c8d) with develop (9a08b82)

Open in CodSpeed

Footnotes

  1. 503 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@joseph-isaacs joseph-isaacs changed the title Add composable chunked decompression via a vtable method Streaming chunked decompression via a composable vtable method Aug 27, 2026
Introduce VTable::decompress_chunks: a push-based streaming decompression
API that walks an array's decompressed values in cache-resident ~1024
element chunks without materializing the full array. Chunks flow through a
ChunkSink chain: leaf encodings stream their existing unpack scratch
buffers, and wrapper encodings compose by interposing a stack-allocated
sink adapter and recursing into their child through the erased ArrayRef
entry point, so the mechanism composes across arbitrary encoding trees
with no per-element dynamic dispatch and no heap state on the way down.

The default implementation executes to canonical and streams the result
(the two-pass baseline), so the method is always available. BitPacked
overrides it to stream FastLanes blocks straight from its unpack scratch
(applying patches per block via a cursor), and FoR overrides it by adding
the reference value in place per chunk before forwarding.

On 4Mi u32 FoR-over-BitPacked, a streaming sum through the vtable path
runs ~1.7x faster than fused decompress-then-iterate, within ~22% of a
fully monomorphized loop doing identical per-chunk work; the pure dynamic
dispatch cost measures ~0.06ns per element.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
When a FoR array's child is BitPacked (and the reference is unsigned,
mirroring the dispatch in decompress), FoR::decompress_chunks now streams
each FastLanes block through the fused FoRStrategy unchecked_unfor_pack
kernel instead of wrapping the child's chunk stream with an in-place add
pass. Patch values get the reference applied when the patch cursor list is
built. The generic sink-composition path remains for any other child.

The block streaming and patch-cursor loop is shared between BitPacked's
plain path and the fused FoR path via stream_unpacked_chunks, generic over
the UnpackStrategy.

On 4Mi u32 FoR-over-BitPacked, a streaming sum through the vtable path now
matches the hand-written monomorphized fused loop within noise (1.35ms vs
1.36ms median), i.e. the same speed as the unpack_map-style kernels, while
the generic composition (measured via a signed reference) stays available
for arbitrary children.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Constant streams by filling a single stack-resident 1024-element scratch
chunk with the constant and re-emitting it across the array (refilled per
emission, since sinks may mutate chunks in place), so the array is never
materialized.

Patched streams its inner child through the erased decompress_chunks
entry point and interposes a sink adapter that overwrites patched rows in
each chunk before forwarding. The lane-transposed patch layout is
flattened once up front into row-sorted (row, value) pairs so the
per-chunk work is a single cursor advance. This composes with any inner
encoding: Patched(Constant) patches each re-emitted constant chunk,
Patched(BitPacked) patches each unpacked FastLanes block.

On 4Mi u32 Patched-over-Constant with sparse patches, a streaming sum
runs 3.6x faster than execute-then-iterate (0.59ms vs 2.12ms median)
since the 16MB base buffer is never written.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Add benches comparing the chunked decompress_chunks path on
Patched(Constant) against the sparse decompression baseline (execute:
canonicalize the constant into a full buffer, then scatter patches), for
both pure materialization and materialize-then-sum.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The default VTable::decompress_chunks previously fell back to executing
the array to canonical and streaming the materialized result, which
silently defeated the purpose of the API: decompressing and transforming
blocks while they are L1-resident.

Streaming is now advertised via VTable::supports_decompress_chunks
(default false); wrapper encodings propagate the check through the
children they stream from, so support of the whole tree is decided before
any chunk is emitted. ArrayRef::decompress_chunks errors on unsupported
trees without emitting anything, and the materializing two-pass fallback
moves to the explicitly named ArrayRef::decompress_chunks_or_materialize.

Primitive gains a real streaming implementation (it is already
decompressed; chunks are copied through one reusable L1 scratch buffer),
Constant and BitPacked advertise support directly, FoR advertises support
when its fused BitPacked path applies or its child supports streaming,
and Patched advertises support when its inner array does.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Add a stream-to-canonical shortcut to the iterative executor: when the
whole encoding tree supports chunked decompression (the capability
cascades — each encoding only advertises support when the children it
streams from do) and the tree is at least two streaming levels deep,
execute_until decompresses blocks straight into the canonical builder
via execute_via_chunks while each block is L1-resident, instead of
materializing a full intermediate buffer per encoding level. A lone
streaming leaf keeps its normal execute path, which already decodes
directly into the output.

Benchmarked on 4Mi-element trees via execute::<PrimitiveArray> with the
shortcut toggled on/off in one process (medians):

- FoR(signed ref) over BitPacked:      3.87ms -> 3.05ms  (-21%)
- Patched(FoR(BitPacked)) with 0.1%
  exceptions:                          4.72ms -> 3.30ms  (-30%)
- FoR(unsigned ref) over BitPacked
  (fused execute baseline):            4.41ms -> 3.01ms  (-32%)

A process-global toggle (set_chunked_execute_enabled, doc-hidden) exists
so benchmarks and tests can compare both executor paths; an equivalence
test checks the shortcut against level-wise execution on a nullable
multi-level tree.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Honor VORTEX_CHUNKED_EXECUTE=0 at process start so benchmark binaries can
compare the executor with and without the stream-to-canonical shortcut
without recompiling; set_chunked_execute_enabled still overrides at
runtime.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
VORTEX_CHUNKED_EXECUTE_TRACE=1 prints, for every non-canonical array
reaching the executor's step 2c check, whether the stream-to-canonical
shortcut fires along with the tree's length, root encoding, and child
encodings. Used to establish which TPC-H scan trees are streaming-capable;
to be removed (or demoted to tracing) once the investigation concludes.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The TPC-H trace established its conclusion: on Q1/Q6 the shortcut fires
288 times, almost entirely on 65536-row FoR(BitPacked) trees, but that is
only ~6% of non-canonical executor work. The dominant trees are wrapped
in vortex.filter (1656, including 736x Filter(BitPacked)) and
vortex.decimal_byte_parts (828), neither of which implements
decompress_chunks yet — which explains the SQL-neutral benchmark result
as an encoding-coverage gap rather than a batch-size effect.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs force-pushed the claude/decompression-chunked-iteration-u3kcns branch from 2d9d407 to 9e5874f Compare August 28, 2026 11:29
Filter now streams: the child streams its blocks, each block is compacted
in place against the mask slice covering it (output index is always <=
input index within a block), and the surviving prefix is forwarded. The
mask's selection is resolved once up front and walked with a cursor, with
a mean-run-length heuristic choosing between run-copying and per-index
gathering, so no per-chunk mask slicing or allocation happens.

Benchmarking this answered the question it was meant to answer: streaming
a filter is NOT faster than the existing path. On Filter(BitPacked) at
65,536 rows (the dominant TPC-H scan tree), streaming matches level-wise
for materialization and is 10-35% slower for consumption except at very
high selectivity. The reason is structural and now understood: the
level-wise path decodes the child directly into the output buffer via
decode_into and compacts in place within it, so there is no intermediate
for streaming to eliminate -- streaming only adds a scratch-to-output
copy.

Two consequences:

- should_execute_via_chunks now requires a cardinality-preserving child,
  which keeps the executor from picking the streaming path for Filter
  trees (it would have regressed them by up to 2.6x).
- The executor shortcut is disabled by default and enabled with
  VORTEX_CHUNKED_EXECUTE=1. Re-measuring on a quiet machine (three
  rounds) shows it is only a reliable win where the level-wise path does
  an extra full-buffer pass: signed FoR(BitPacked), which runs a separate
  wrapping-add pass, is 10-25% faster streaming, while fused
  FoR(BitPacked) and Patched(FoR(BitPacked)) are ~5-10% slower. The
  earlier 21-32% figures were measured on a loaded machine and did not
  reproduce.

The streaming consumption win (sum without materializing: 1.29ms vs
3.36ms two-pass on 4Mi rows) is unaffected and remains the API's value,
along with Patched(Constant) where the base buffer is never written.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Deeper stacks are where streaming pays: each additional encoding level
costs level-wise execution a full-length intermediate buffer, but costs
streaming only one more pass over an L1-resident block.

Measured on 4Mi-row nested FoR-over-BitPacked stacks (medians, repeated
rounds; marginal cost per added level 0.93ms level-wise vs 0.51ms
streaming):

  chain len 2:  2.57ms vs 2.58ms   parity
  chain len 3:  3.14ms vs 2.74ms   1.15x
  chain len 5:  5.26ms vs 4.00ms   1.32x
  chain len 9:  9.07ms vs 6.16ms   1.47x

Streaming is also far more predictable: p100/median is ~1.2x versus
2.5-5x for level-wise, which repeatedly allocates full-length buffers.

So instead of a blanket on/off default, the executor now measures the
tree: streaming_chain_len counts consecutive streaming-capable nodes from
the root, following only children that stream, are not already canonical,
and preserve row count, and the shortcut fires at MIN_STREAMING_CHAIN
nodes. The cardinality rule keeps selection encodings such as Filter out
of the executor path without a special case, since their level-wise
kernels already decode into the output and compact in place.

VORTEX_CHUNKED_EXECUTE=0 remains as a kill switch. Equivalence tests now
drive execute_via_chunks directly so they exercise streaming at every
depth rather than depending on the heuristic, and a new test pins the
depth rule (including that filter push-down beneath a deep stack still
streams the levels above it).

Also rewrites the chunk_iter module docs to state the vtable API contract
and the cost model in one place.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs force-pushed the claude/decompression-chunked-iteration-u3kcns branch from 9e5874f to 377d029 Compare August 28, 2026 11:33
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Sep 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR was closed because it has been inactive for 7 days since being marked as stale.

@github-actions github-actions Bot closed this Sep 27, 2026
Port chunked decompression onto the blocked FoR encoding, which now has
one reference per 1024-element chunk and an offset into the first chunk:

- Fused FoR+BitPacked streaming now unpacks with `unchecked_unfor_pack`
  per chunk, for signed and unsigned types alike, whenever the
  references are constant or the BitPacked chunks line up with the FoR
  chunks, mirroring `decompress`.
- Generic composition adds each FoR chunk's reference in place, splitting
  a streamed child chunk that spans two FoR chunks.
- BitPacked streaming uses the caller-owned unpack scratch API.
- Outputs are allocated through the execution context's allocator.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
@joseph-isaacs joseph-isaacs reopened this Oct 2, 2026
@joseph-isaacs joseph-isaacs removed the stale This PR is stale and will be auto-closed soon label Oct 2, 2026 — with Claude
joseph-isaacs and others added 4 commits October 2, 2026 21:34
…ed and Slice

Extend `decompress_chunks` beyond FoR, BitPacked, Patched, Filter,
Constant and Primitive:

- Leaves that generate their values stream from one stack scratch chunk
  (`stream_from_fn`): Sparse (fill plus patches), RunEnd (runs expanded
  per chunk) and Sequence.
- Wrappers transform their child's stream in place: ALP decodes integers
  into floats and patches them, ZigZag decodes into signed integers.
  `ChunkMut::retype` re-tags a chunk as a type of the same width.
- Dict gathers each chunk of streamed codes from its values, tolerating
  out-of-bounds codes only where the code is null.
- RLE and Delta decode whole FastLanes blocks. `BlockDecodeSink` regroups
  the child's stream into aligned 1024-row blocks, decodes each into
  scratch and forwards the rows within the array's offset window.
- Chunked streams each chunk in turn. Slice resolves itself through the
  child's slice kernel, as the executor would, or else forwards only the
  sliced rows of its child's stream.
- `ChunkPatches` resolves patches once and applies them with a cursor;
  BitPacked, FoR, Sparse, ALP and Patched share it.
- Chunks hold at most 1024 rows, checked in debug builds, so adapters
  that change the value type convert into a fixed stack scratch chunk.
- The executor checks the same-length, non-canonical spine below an array
  reaches the streaming depth before asking any encoding whether it
  streams, so shallow trees skip the support checks.

Every encoding is tested against level-wise execution, including nulls,
slices and unaligned child chunks, through
`test_harness::assert_streams_like_execute`.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
…w ns

Re-measured on nested FoR over BitPacked with mimalloc, streaming runs
1.07x, 1.10x and 1.13x as fast as level-wise execution at chains of 3, 4
and 9 nodes for 64Ki rows, and 1.64x to 2.64x for 4Mi rows. Chains of 2
lose 11% at 64Ki rows, so `MIN_STREAMING_CHAIN` drops from 4 to 3. That
streams ALP over FoR over BitPacked, the common float tree, which the
2048-row `decompress[alp_for_bp_f64]` benchmark measures 21% faster.

The executor's check now costs a few nanoseconds on trees it does not
stream, so the other encoding-tree benchmarks are within 1% of develop:

- non-primitive arrays are turned away by their dtype first,
- the structural walk tests `is_canonical`, which costs ~36ns, only for
  children deep enough to count.

RLE's per-chunk bounds check reduced its indices after widening each to
`usize`, which does not vectorize. Reducing in the index type makes RLE
decoding 1.7x faster level-wise and streaming alike. RunEnd streaming
resolves its run ends once and walks them like a slice.

Tests compute their level-wise reference with `without_chunked_execute`,
which disables the shortcut on the calling thread only, since the
executor now streams some of the trees they check. Also fix two rustdoc
links.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
`vortex/benches/streaming_decompress.rs` measures eight encoding trees
six ways, `{materialize, sum} x {streaming, levelwise, fused}`, where
`fused` is a kernel written by hand for the tree, monomorphized with no
dynamic dispatch. Delta over FoR over BitPacked compares against a
special kernel that unpacks, undeltas and untransposes each block in L1.
Every fused kernel is checked against execution before timing.

The FastLanes bench keeps only what is specific to the executor: the
FoR depth sweep, now at 64Ki and 4Mi rows with mimalloc, and Filter over
BitPacked.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
`execute_until::<M>` lets callers stop at an encoding they can use as
is: the Arrow exporter executes until a run-end or dictionary array to
export it zero-copy. With chains of three streaming, a run-end array
behind two lazy slices streamed straight to canonical, skipping the
run-end array the exporter was executing toward and failing
`vortex-arrow`'s `nested_wrappers_reveal_run_end_fast_path`.

The streaming chain now takes the activation's done predicate and the
executor does not stream when a node the chain would decode through
satisfies it, so that node is reached level-wise.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
@joseph-isaacs joseph-isaacs added the changelog/feature A new feature label Oct 2, 2026 — with Claude

Copy link
Copy Markdown
Contributor Author

CodSpeed Performance Analysis is red on 09fc129. The four Simulation regressions it flags come from comparing across different runtime environments (CodSpeed shows that warning on this run), not from this PR's code.

Instruction counts measured locally with callgrind (per iteration, the difference between 201-iteration and 1-iteration runs of the same bench binary; develop vs this branch):

Benchmark develop branch Δ
bitpacked_decompress_u32 9,894 9,897 +0.04%
decompress[for_bp_u64] 9,910 9,994 +0.85%

CodSpeed reports −16% and −19% for those two.

  • baseline_lt[4, 1024] (one BitPacked → Primitive execute, then a compare) and decompress_rd (ALP-RD, which doesn't stream) go through the same non-streaming path. That path adds one cheap structural check per executor step and nothing else.
  • On the same run, decompress[alp_for_bp_f64] (+13%) is the streaming path itself; locally it measures 21% faster.

There is no code fix to make here, and a re-run won't change the environment mismatch, so these need acknowledging in CodSpeed.


Generated by Claude Code

When streaming materializes an array, producers wrote each chunk into a stack
scratch chunk and the builder sink then copied it into the output buffer. That
copy left streaming slower than level-wise decoding for single-pass producers
(sparse fills, run-end runs, dictionary gathers).

`ChunkSink` gains `destination`, which offers the memory the sink would copy
rows into, and `accept_written`, which accepts rows the producer wrote there.
`emit_with` decodes into the destination when one is offered and into scratch
otherwise, so leaf producers (`stream_from_fn`, bit-packed and FoR unpacking,
whole-block decoders, dictionary gathers) write the output buffer directly.
In-place adapters (patches, FoR references, ALP, ZigZag, chunked offsets and
slices) forward their inner sink's destination and transform the written
values when they accept them.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
joseph-isaacs and others added 4 commits October 3, 2026 00:23
Scans cast columns to their table schema's nullability, so real TPC-H stacks
put a lazy cast between FoR and its bit-packed values, e.g.
FoR(Cast(Slice(BitPacked))). Casts did not stream, which cut these chains
short of the streaming threshold.

A cast now streams when it cannot fail: every value converts losslessly (the
widenings the primitive cast kernel already treats as infallible) and the
nullability only widens. Each chunk converts while L1-resident, straight into
the output when materializing; a same-ptype cast forwards its child's chunks.
Casts that check values against the target range keep the level-wise kernel.

On TPC-H SF1 this streams the FoR(Cast(Slice|Filter(BitPacked))) stacks,
about 40M of the rows the queries decode.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
A census of the arrays the TPC-H SF1 queries execute found the trees that
stream: FoR(Cast(Filter(BitPacked))) and FoR(Cast(Slice(BitPacked))), where
a scan's filter or slice and its nullability cast are pushed into FoR's
values. Measure them, at three filter selectivities, against level-wise
execution, and check at startup that they stream.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
Streaming a filter built the mask's runs and then its indices up front, a pass
over the whole mask each, and compacted every chunk with scalar run copies or
index gathers. On TPC-H that cost more than streaming saved: callgrind of
SF1 Q21 counted 9.5% more instructions with streaming on, nearly all of it in
the filter, while the level-wise filter compacts straight from the bitmap with
the SIMD compress kernels.

Each chunk now compacts in place against the mask bits covering its rows
with the kernel the level-wise filter would pick for the mask, chosen once
from its density, and falls back to the scalar bitmap walk where no kernel
applies. The kernels' mask walk is generic over `MaskWords`, so the same code
serves a whole mask and the bits of one chunk.

Q21 now executes 0.1% more instructions with streaming on than off. Streaming
Filter(BitPacked) at 64Ki rows runs 0.84x, 0.85x, 0.86x and 0.97x as long as
level-wise execution at selectivities 1/16, 4/16, 8/16 and 12/16 (before:
0.83x, 1.16x, 1.30x and 1.34x).

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
Producers decoded into stack scratch chunks that were zeroed up front, 8 KiB
each per stream. Since chunks decode straight into a materializing sink's
destination, the scratch is often never touched, yet the zeroing remained:
callgrind of the CodSpeed `decompress[alp_for_bp_f64]` benchmark (2048 rows)
counted 32.0K instructions per iteration, 8.2K of them in memset, against
26.7K on develop.

`ScratchChunk` zeroes itself on first use, and the bit-packed, leaf, block,
dictionary and cast producers use it for their fallback scratch. The
benchmark now counts 23.8K instructions per iteration.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7

Copy link
Copy Markdown
Contributor Author

Following up on CodSpeed Performance Analysis for 01e5296. Everything else on this head is green.

  • decompress[alp_for_bp_f64] is now +20.5%. On 78adcb6 it regressed to −11.7%. Callgrind traced that to an 8 KiB scratch chunk each stream zeroed up front: 8.2K of 32.0K instructions per iteration, against 26.7K on develop. 01e5296 zeroes scratch only on first use, and it now runs 23.8K per iteration.
  • filter_list_primitive_short[AllSelected] (−10%, Simulation). List arrays never take the streaming path (the executor's check stops at the dtype). Callgrind across all six masks of this benchmark, 09fc129 vs 01e5296: 317,873 vs 322,771 instructions per iteration (+1.5%). The extra instructions sit in unchanged code (list-to-listview conversion, row-wise numeric kernels), with symbols renamed between builds, i.e. inlining and identical-code folding in the recompiled crate.
  • mul_u64_nonnull_neon, mul_i64_nonnull_neon, multiply_shapes_neon (WallTime, arm64). This PR does not touch the binary-op kernels, and CodSpeed warns its WallTime instrument is unreliable on hosted runners.
  • for_bp_u64, bitpacked_decompress_u32 and baseline_lt are the cross-environment flags explained above. Callgrind still counts +1.2% and +0.04% for the first two.

The earlier "Rust build (with tokio dispatcher)" failure was the runner killing the build (exit 137, no compiler error); its retry passed.


Generated by Claude Code

@joseph-isaacs
joseph-isaacs marked this pull request as ready for review October 3, 2026 01:50
joseph-isaacs and others added 4 commits October 3, 2026 08:38
Streaming a run-end array collected its ends into a Vec<usize> before
expanding the runs, a separate pass and a full-length allocation that
level-wise decoding avoids by reading the ends lazily. At 64Ki rows in runs
of 16 that pass cost 49K of the 374K instructions per decode, which left
streaming slower than level-wise decoding.

Each run's end is now read from the stored ends as the stream reaches it.
Materializing streams in 23.5us against 27.0us level-wise and 23.7us for a
hand-fused kernel (before: 29.8us), and folding in 31us against 35us
level-wise (before: 40us).

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
Hand-written kernels fused per encoding tree ran up to 1.86x faster than
streaming the same tree through `ChunkSink`s, mostly in consumers that fold
values rather than materialize them. Profiles put the cost in code that saw
chunks as slices of unknown length and in per-chunk validation, not in the
dynamic dispatch:

- `ChunkMut::as_block` views a full chunk as a fixed-size block, and
  `emit_with` hands full chunks to producers' decode closures as blocks, so
  both sides of the sink compile the common full chunk with a constant
  length, as a kernel written for whole blocks does. Dictionary gathers
  reslice their codes to the output's length for the same reason.
- RLE decodes bit-packed indices block by block itself rather than
  streaming them up through BitPacked and regrouping them. `emit_block`,
  which `BlockDecodeSink` already used, is public for decoders like it.
- RLE skips the per-chunk index bounds check when the indices' encoding
  bounds them (bit-packed indices without patches are below
  `1 << bit_width`), and otherwise reduces with AVX2 where available, which
  has an unsigned 16-bit max the baseline target lacks. Level-wise RLE
  decoding takes the bound too.
- `execute_via_chunks` treats a non-nullable tree as all valid without
  asking its encodings, and does not re-check support its caller checked.

Wall time at 64Ki rows (medians of five runs), streaming time relative to
the hand-fused kernel, materialize / sum:

    FoR(BitPacked)          1.05 / 1.23  ->  1.06 / 1.03
    BitPacked + patches     0.96 / 1.26  ->  0.98 / 1.04
    ALP(FoR(BitPacked))     1.02 / 1.04  ->  0.99 / 1.04
    Delta(FoR(BitPacked))   1.05 / 1.05  ->  1.03 / 1.04
    Dict(BitPacked codes)   0.91 / 1.15  ->  0.91 / 0.91
    RLE(BitPacked indices)  1.25 / 1.31  ->  1.05 / 1.03
    RunEnd                  0.98 / 1.11  ->  0.98 / 0.79
    Sparse                  0.91 / 1.85  ->  0.94 / 0.93

Streaming now beats level-wise execution on every tree, Sparse sums by
1.9x.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
- Dict: take the unchecked gather only when a chunk's smallest code is
  non-negative as well as its largest in bounds, so a negative signed
  code can no longer wrap to an out-of-bounds index. For unsigned codes
  the extra check folds away.
- execute_via_chunks: the builder sink now requires chunks in order and
  checks the stream covered every row before finishing the builder, in
  release builds too, so a stream that skips rows is an error rather
  than uninitialized output.
- BlockDecodeSink::finish rejects a child stream that ends part way
  through a block, which RLE and Delta call instead of silently
  dropping the trailing rows.
- Wrappers stream their children through decompress_child_chunks, which
  skips the up-front support walk their own support check already did.
- FoR's support check no longer special-cases BitPacked, which always
  streams. Benches compare against level-wise execution with the scoped
  without_chunked_execute; RLE tests use `?` instead of unwrap.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
Streaming carried primitive values only, so decimal trees always
executed level by level. Chunks now carry a fixed-width ValueType: a
primitive, or the i128/i256 integers wide decimals store their values
in, and any ChunkValue (every native primitive, i128, i256) can be
streamed, gathered, filtered or written into a destination.

The vtable's supports_decompress_chunks -> bool becomes
decompress_chunks_type -> Option<ValueType>: the physical type an
array would stream. A primitive array streams its dtype's ptype; a
decimal array streams the integer type executing it stores its values
in, which the dtype alone does not determine, so materializing a
streamed decimal keeps the storage type level-wise execution would.

- Decimal: the canonical leaf streams its values.
- DecimalByteParts: without lower parts, forwards the most significant
  part's stream; with them, streams the most significant part and
  assembles each chunk with its lower words, straight into the output.
- Constant, Dict, Chunked, Slice and Filter stream decimals. Chunked
  converts chunks stored in another type to the smallest type for the
  precision, as its level-wise builder does; Slice only takes a
  resolved slice that streams the same type.
- execute_via_chunks builds a PrimitiveArray or DecimalArray, and the
  executor streams decimal trees whose streaming chain is deep enough.

Materializing a decimal(38, 2) DecimalByteParts over FoR(BitPacked)
runs at 0.75x of level-wise execution, summing it at 0.68x; the
primitive trees are unchanged (geometric mean 0.996x, identical
instruction counts).

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FzEAvoztg2xuTKsoG4ZVX7
@joseph-isaacs
joseph-isaacs marked this pull request as draft October 5, 2026 10:49
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Oct 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR was closed because it has been inactive for 7 days since being marked as stale.

@github-actions github-actions Bot closed this Oct 30, 2026
@robert3005
robert3005 deleted the claude/decompression-chunked-iteration-u3kcns branch November 10, 2026 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature stale This PR is stale and will be auto-closed soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant