Repository navigation
feat: add fused block executor vtable method - #11541
joseph-isaacs wants to merge 10 commits into
Conversation
The iterative executor descends an encoding tree with ExecuteSlot frames, so when it reaches a leaf, the frames on its stack are the parents waiting for that leaf to be canonical. Level-wise execution writes a full-length buffer at every level of that chain. The executor now fuses it instead: the leaf emits blocks of at most 1024 rows (BlockSource), each waiting parent transforms a block (BlockOp) in L1 scratch, and only the outermost parent writes a full-length buffer. An encoding opts in explicitly. It asks for the last child its execute waits for with the new ExecutionStep::ExecuteSpine (execute_spine, require_spine!), promising that every other child it needs is executed and that once this child is canonical it only transforms the child's values row by row, as its VTable::block_op does. Leaves implement VTable::block_source. The executor only fuses through spine frames. execute_until tries fusion before executing an encoded leaf, after its execute_parent probes so no parent kernel is skipped, and when a canonical child has two or more spine parents waiting above it. A leaf with no block source but an already canonical child uses that child as the source with its own op on top. Anything else executes level-wise as before. Validity is not streamed; it is taken from the outermost parent. Implemented for Primitive and BitPacked (sources) and FoR: a source over BitPacked when their chunks line up, running the fused unfor_pack per chunk, and a reference-adding op over any other child. FoR requests its encoded child as its spine. Ops map blocks with the same lane kernels level-wise execution uses. Disable with ExecutionCtx::with_block_fusion(false) or VORTEX_BLOCK_FUSION=0. Each fused chain logs a vortex_array::block debug event. Adds encodings/fastlanes/benches/block_fusion.rs, comparing fused and level-wise execution on the same arrays, and documents block fusion in the execution internals guide. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Drop FoR's own block source, which duplicated the BitPacked unpack with FoR references threaded through it. A leaf without a block source now fuses its own op over any child that has one, not only a canonical child, so FoR over BitPacked fuses as BitPacked's chunk-by-chunk source with FoR's add-reference op on top, and BitPacked's source no longer knows about FoR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
FoR over BitPacked with one global reference is a block source again, unpacking each chunk with FastLanes' fused unpack-and-add, as FoR's own execute does, instead of unpacking and then adding the reference in a second pass over each block. The BitPacked block source is now generic over the existing UnpackStrategy the level-wise unpack uses: BitPacked uses BitPackingStrategy and FoR a new FoRStrategy, so the chunk loop is not duplicated. FoR with per-chunk references still fuses as BitPacked's source with FoR's op. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will regress 3 benchmarks1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing Footnotes
|
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.980x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.009x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.995x ➖, 0↑ 0↓)
No file size changes detected. |
Its fused and level-wise pairs were for measuring the change during development; fusion can be toggled with VORTEX_BLOCK_FUSION=0 on any existing benchmark. The existing decompress[alp_for_bp_f64] encoding-tree benchmark covers the tree that fuses in practice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (0.994x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.005x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.996x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 1↑ 0↓)
duckdb / parquet / ns (1.012x ➖, 0↑ 3↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.987x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.994x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.928x ➖, 4↑ 3↓)
datafusion / parquet / ns (0.963x ➖, 2↑ 1↓)
duckdb / vortex-file-compressed / ns (1.128x ➖, 1↑ 3↓)
duckdb / parquet / ns (0.916x ➖, 2↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 1↓)
duckdb / parquet / ns (0.992x ➖, 1↑ 0↓)
No file size changes detected. |
BlockOp::process now takes a Block that is either Copy (an input and separate output memory) or InPlace (one buffer the op overwrites), and returns a Written proof. Block::map writes every value with the lane kernels for either shape, so ops that only map, like FoR's, need no change for in-place blocks, and an op can only skip values with unsafe code, closing the hole where a safe op left output uninitialized. The executor overwrites scratch in place wherever an op keeps its width, so a chain needs one scratch block, and only the last op copies into the output. A canonical leaf it holds the only reference to is decoded in its own buffer when every op keeps the width, as level-wise execution does, instead of into a second full-length buffer. execute_until releases its input once optimized so that leaf can be uniquely owned. Planning finds the chain's input before building any op, so a leaf that cannot fuse costs no op construction. Primitive no longer needs a block source: the executor reads canonical arrays itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The trace harness records ExecuteSpine as its own step, each fused chain (the outermost array, whether its input is a source or a canonical leaf, and how many frames and ops it consumed), and, for the first block, how each op ran: in place or copying, and which memory it read and wrote (the source's own memory, a scratch block, the output, or the leaf's buffer). Trace events now render the arrays they describe when recorded instead of holding clones of them, so tracing does not keep arrays alive and change what it observes: a leaf the executor owns uniquely is still decoded in place while traced. New tests assert the schedules of FoR stacks over BitPacked, FoR over a shared and an owned canonical leaf, and FoR with misaligned per-chunk references. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Replace planning from the leaf with fusion that follows the executor's trampoline. The loop descends with ExecuteSpine frames as before. On the way back up, a child that can emit blocks (a block source, or a canonical primitive array) starts a block program when the spine parent waiting for it has a block op, and the parent becomes a Fused array carrying the program. Each spine parent above extends it with its op. When the next parent cannot extend it, or at the root, executing the Fused array drives the whole program in one pass, and execution carries on. Fused is an internal encoding with no slots that never leaves execute_until. Each fused parent takes its validity and statistics from its own vtables with the child back in its slot, so validity is derived per level, as level-wise execution does. The leaf-time planner, its canonical-leaf special case and its minimum-length rules go away. BlockSource::fill now takes the rows to emit, and block_offset reports the source's grid, so the executor chooses the blocks: on the source's grid, a BitPacked range lies in one chunk and is lent from it; a range across chunks is assembled. Block is #[non_exhaustive]. Traces record fuse start, fuse extend and drive. New executor tests with test encodings assert the build and drive order, copying and in-place schedules, width changes, ops without in-place support, a non-spine parent driving the chain below it, the block grid, owned and shared canonical leaves, disabled fusion and empty arrays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- In-place ops wrote through pointers derived from a shared borrow, which is undefined behaviour. BlockSource::fill now returns a SourceBlock: values written into the executor's scratch are addressed through the executor's own mutable borrow and may be overwritten in place, while values lent from the source's memory are only read. The read-only BlockBuf::view is gone. - A fused array is never done for the target or a frame's predicate, so it is always driven before execute_until returns, whatever the target matches. - A parent with execute-parent kernels of its own, or one the next parent up has a kernel for, is not fused, so those kernels still probe it with its child executed. One try_fuse helper serves both call sites. - Fused results materialize validity at drive, as level-wise execution does. - fuse_into_parent checks the child is primitive and has its parent's length before asking for the op, and finds the program's input before building the op. - The second scratch block is allocated only when an op copies between scratch blocks. - BlockPatches keeps indices in their stored type, checks indices and values match in length and values are all valid. - FoR with a zero global reference does not fuse, keeping its zero-copy level-wise path. - check_written reports lengths and types with vortex_ensure_eq. - Shared-leaf tests check the leaf's values are untouched; new tests cover a target matching a fused array and a parent with kernels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
execute_encoding_unchecked called the encoding through a shared borrow of the array's data while also moving in the ArrayRef that owns it. When that ArrayRef was the last reference, the step dropped the array while the borrow was still a live function argument, which Miri reports as undefined behaviour. It was latent while execute_until held its input for the whole loop, and reachable now that it releases it. DynArrayData::execute_fn returns the step as a function taking the array by value, so the borrow ends before the step runs. The checked execute_encoding applies its postconditions around it, and DynArrayData::execute and execute_unchecked go away. The block tests pass under Miri. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
A block source claims its scratch holds initialized values when it returns SourceBlock::Scratch, which safe code could do without writing them. The variant is now non-exhaustive and built with the unsafe SourceBlock::scratch, as op outputs already need Written::assume_init. BlockPatches::try_new returns None for null patch values instead of failing, so such an array executes level-wise, which patches validity too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
Part 1 of 2. This PR adds the executor machinery and the BitPacked and FoR implementations. The follow-up PR, #11542, adds ALP, lossless Cast and ZigZag ops on top. It revisits the 1024-row streaming idea from #9614, built into the iterative executor instead of a separate recursive vtable path.
The iterative executor descends an encoding tree with
ExecuteSlotframes. When it reaches a leaf, the frames on its stack are the parents waiting for that leaf to be canonical. Level-wise execution writes a full-length buffer at every level of that chain. This PR fuses the chain instead:BlockSource);BlockOp), with intermediates in two L1 scratch buffers used in turn;Changes
ExecutionStep::ExecuteSpine(ExecutionResult::execute_spine,require_spine!). An encoding uses it for the last child itsexecutewaits for. It promises that every other child it needs is already executed, and that once this child is canonical it only transforms the child's values row by row. Stack frames record the flag, and the executor only fuses through spine frames. The single-step executor treats it likeExecuteSlot.VTable::block_sourceandVTable::block_op(slot, child_dtype), both defaulted.block_opmust produce exactly whatexecuteproduces once the slot is canonical. A leaf'sblock_sourceis also where an encoding offers a fused kernel over its child.execute_untiltries fusion in two places:execute_parentprobes, so no parent kernel is skipped;A leaf with no block source uses the source of a child it has a
block_opfor, with its own op on top. Anything else falls back to level-wise execution unchanged. Validity is not streamed; it is taken from the outermost parent.Encodings:
UnpackStrategythat level-wise unpacking uses.encodedas its spine.FoRStrategyruns FastLanes' fusedunchecked_unfor_packper chunk, the same kernel FoR's ownexecuteuses.map_intolane kernels as level-wise execution.Disable with
ExecutionCtx::with_block_fusion(false)orVORTEX_BLOCK_FUSION=0. Each fused chain logs avortex_array::blockdebug event.Docs:
docs/developer-guide/internals/execution.mdgains a Block Fusion section.Tests
The tests in
encodings/fastlanes/src/block.rsexecute each tree with fusion on and off and compare the results. They cover:They also count the chains actually fused, so a test can't pass just because fusion never ran.
Benchmark
Measured locally with a fused/level-wise benchmark that is not committed, 40 rounds × 100 samples on the current head. Speedup is level-wise ÷ fused, paired within each round, on a noisy 4-core VM. FoR(BP) alone fuses nothing and serves as the control. No new benchmark is added; set
VORTEX_BLOCK_FUSION=0to A/B any existing one.Checks
cargo test -p vortex-array -p vortex-fastlanes -p vortex-alp -p vortex-zigzagcargo clippy -p vortex-array -p vortex-fastlanes --all-targets --all-features -- -D warningscargo +nightly-2026-11-13 fmttypos, docs build, workspace-wide checks.API Changes
vortex_array::blockmodule.ExecutionStep::ExecuteSpine. This breaks exhaustive matches onExecutionStep.ExecutionResult::execute_spineandrequire_spine!.VTable::block_source/VTable::block_opwith defaults.ExecutionCtx::with_block_fusion/block_fusion.https://claude.ai/code/session_01UeniB9bmdcvP43mdgntV8U
Generated by Claude Code