Repository navigation
test(sqllogictest): check the encodings of the TPC-H lineitem and orders tables - #11564
Draft
joseph-isaacs wants to merge 1 commit into
Draft
joseph-isaacs wants to merge 1 commit into
joseph-isaacs wants to merge 1 commit into
Conversation
…ers tables Add encoding checks for the TPC-H data `data-gen` writes: - `tables.check`: every column of every table has a zone map with a constant null count. - `lineitem.check` and `orders.check`: dictionary layouts for the low-cardinality string columns, no uncompressed chunk roots, string compression for free text, bit-packed dictionary codes, FoR + bit-packed dates, byte-part decimals (dictionary-encoded for `l_quantity`), FoR + bit-packed `o_orderkey`, constant `o_shippriority`. Units are selected with `data*`, which matches chunked and unchunked columns, so the checks hold whatever chunking a scale factor produces. Free text accepts FSST or OnPair rather than pinning one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBDxK1z3cP4gAzkh8c8 Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
This was referenced Nov 17, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The third PR in the stack, on top of #11544, which adds
vortex-filecheck(itself on top of #11563). This PR adds encoding checks for the TPC-H tables thatdata-genwrites for the SLT suite.Changes
tpch/filecheck/tables.check(files: tpch/data/.*\.vortex, so it runs against all 8 tables): every column has a zone map whose null count is constant.tpch/filecheck/lineitem.check:l_quantity, whose most significant part is dictionary-encoded.tpch/filecheck/orders.check:o_orderstatus,o_orderpriorityando_clerk;o_orderkey;o_shippriority;o_totalprice;Design choices:
data*, which matches every chunk of a chunked column and the single unit of an unchunked one. The checks therefore don't depend on how many chunks a scale factor produces.regex:variable. The compressor already picks differently per column (l_commentis FSST,c_commentis OnPair), and pinning one would make these checks churn on harmless scoring changes.l_orderkeymixes run-end, frame-of-reference and plain bit-packing across chunks. It is only covered by the "no uncompressed chunk" rule, and is worth a separate look.Testing: each check passes with
vx filecheckagainst freshdata-genoutput at SF 0.1:tables.checkon all 8 tables;lineitem.check: 9 sections, 66 chunked units;orders.check: 8 sections.The SLT runner itself could not be run here because the DuckDB download is blocked.
Generated by Claude Code