Skip to content

test(sqllogictest): check the encodings of the TPC-H lineitem and orders tables - #11564

Draft
joseph-isaacs wants to merge 1 commit into
ji/sleepy-albattani-o2d43nfrom
ji/filecheck-tpch-checks
Draft

joseph-isaacs wants to merge 1 commit into
ji/sleepy-albattani-o2d43nfrom
ji/filecheck-tpch-checks

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

The third PR in the stack, on top of #11544, which adds vortex-filecheck (itself on top of #11563). This PR adds encoding checks for the TPC-H tables that data-gen writes for the SLT suite.

Changes

  • tpch/filecheck/tables.check (files: tpch/data/.*\.vortex, so it runs against all 8 tables): every column has a zone map whose null count is constant.
  • tpch/filecheck/lineitem.check:
    • dictionary layouts for the four low-cardinality string columns;
    • no chunk stores an uncompressed root;
    • free text is FSST or OnPair;
    • dictionary codes are bit-packed;
    • dates are frame-of-reference then bit-packed;
    • decimals use byte parts, with a bit-packed most significant part, except l_quantity, whose most significant part is dictionary-encoded.
  • tpch/filecheck/orders.check:
    • dictionary layouts for o_orderstatus, o_orderpriority and o_clerk;
    • no uncompressed chunk roots;
    • frame-of-reference + bit-packed o_orderkey;
    • constant o_shippriority;
    • frame-of-reference + bit-packed dates;
    • byte-part o_totalprice;
    • string-compressed free text;
    • bit-packed dictionary codes.

Design choices:

  • Units are selected with data*, which matches every chunk of a chunked column and the single unit of an unchunked one. The checks therefore don't depend on how many chunks a scale factor produces.
  • Free text accepts FSST or OnPair, through a regex: variable. The compressor already picks differently per column (l_comment is FSST, c_comment is OnPair), and pinning one would make these checks churn on harmless scoring changes.
  • The checks pin encodings that matter for performance: sequences, frame-of-reference, dictionary layouts and constants.
  • l_orderkey mixes run-end, frame-of-reference and plain bit-packing across chunks. It is only covered by the "no uncompressed chunk" rule, and is worth a separate look.

Testing: each check passes with vx filecheck against fresh data-gen output at SF 0.1:

  • tables.check on all 8 tables;
  • lineitem.check: 9 sections, 66 chunked units;
  • orders.check: 8 sections.

The SLT runner itself could not be run here because the DuckDB download is blocked.


Generated by Claude Code

…ers tables

Add encoding checks for the TPC-H data `data-gen` writes:

- `tables.check`: every column of every table has a zone map with a
  constant null count.
- `lineitem.check` and `orders.check`: dictionary layouts for the
  low-cardinality string columns, no uncompressed chunk roots, string
  compression for free text, bit-packed dictionary codes, FoR +
  bit-packed dates, byte-part decimals (dictionary-encoded for
  `l_quantity`), FoR + bit-packed `o_orderkey`, constant
  `o_shippriority`.

Units are selected with `data*`, which matches chunked and unchunked
columns, so the checks hold whatever chunking a scale factor produces.
Free text accepts FSST or OnPair rather than pinning one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBDxK1z3cP4gAzkh8c8
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant