Skip to content

Move the shared scan engine into vortex-scan - #11557

Draft
gatesn wants to merge 13 commits into
developfrom
ngates/consolidate-scan-engine
Draft

gatesn wants to merge 13 commits into
developfrom
ngates/consolidate-scan-engine

Conversation

@gatesn

@gatesn gatesn commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Keep native layout lowering, segment decoding, and legacy statistics conversion in vortex-layout while consolidating plans, preparation, execution, caches, and the default driver in vortex-scan. Preserve existing paths through compatibility exports and update engine consumers to the new ownership boundaries.

Keep native layout lowering, segment decoding, and legacy statistics conversion in vortex-layout while consolidating plans, preparation, execution, caches, and the default driver in vortex-scan. Preserve existing paths through compatibility exports and update engine consumers to the new ownership boundaries.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Construct source-independent plans for node, optimizer, pipeline, and scan execution tests in vortex-scan. Keep native lowering and file-backed comparisons in vortex-layout, and remove the cross-crate scan test harness and its public inspection APIs.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
@joseph-isaacs

Copy link
Copy Markdown
Contributor

This is quite a lot more than a refactor

Move record-batch scan output to an ArrowScanExt trait in vortex-arrow so the scan engine has no direct Arrow dependencies. Expose generic worker-side batch mapping to preserve parallel conversion and decompression.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
The scan engine does not use the sequence encoding, paste, or termtree. Remove the stale dependencies left by the extraction so the engine only declares the lower-level primitives and utilities it actually uses.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Validate native legacy rewrites through rule sets, decode delivered bytes in scan fixtures, and restore checked raw statistics fixtures and driver concurrency. Keep internal construction and source inspection APIs crate-private without changing crate dependencies.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Restore the original legacy-schema constructor and tests beside the native schema helper. Retain generic evaluator tests in vortex-scan and remove the duplicated native schema validation from its fixtures.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Restore the native ZoneMap representation and schema helper implementations without wrappers or test-only fields. Keep generic scan evaluation independent and replace qualified body paths introduced by the reorganization with module imports.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Restore the native builder to scan::v2 and the existing public scan definitions to the crate root. Keep legacy field-name lookup in the native zone map, where it is still needed for older files.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Restore limits, metrics, and byte-range partitioning to their original modules. Share the existing row block size across the crate boundary and keep internal scan helpers private.

Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>
Signed-off-by: Nicholas Gates <nick@nickgates.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants