Repository navigation
benchmarks: BENCH_INSTRUMENT=2026-11 turns on the default-off switches and the agreed headline changes - #413
Merged
Merged
Conversation
…s and the agreed headline changes (#412) Sparse warm-up, idle-state hold, dense id index, official arcadedb-driver as the served headline with a lean twin cell, Arrow reads for l4 q_high and q_groupby with the plain read timed beside it, and no -Xms pin on embedded ArcadeDB arms. Unset BENCH_INSTRUMENT is unchanged; explicit variables override in both instruments. The runner now forwards BENCH_INSTRUMENT to the cell. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…int from the timed set (#412) Extra dataset queries when present, else a fixed-seed sample of documents used as queries; the row stamps the count and the source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ckDB's Arrow rows are read by position (#412) The 2026-11 gate run found DuckDB's q_groupby typed digest (7ab90d6f43a25460) differing from fetchall() (18c9985de67433e6): the unaliased avg(uu) column is not named v, so dict rows digested <missing>. ArrowResult reads DuckDB's table positionally; the row now carries <query>_plain_digest and _match, and the cell ends if they differ. Test: test_l4_arrow_digest_gate.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #412. Under BENCH_INSTRUMENT=2026-11 the default-off switches and the agreed headline changes become the behaviour; unset BENCH_INSTRUMENT is exactly 2026-10, and an explicit variable overrides the default in both instruments. Rows stamp what they used.
What changes under 2026-11: BENCH_SPARSE_WARMUP defaults to 1,000 untimed searches for every sparse engine, on queries disjoint from the timed set (the dataset's extra queries, else a fixed-seed sample of documents used as queries; rows stamp sparse_warmup_queries and sparse_warmup_source) and BENCH_CPU_DMA_LATENCY_US to a hold at 0 (the values are bench_common constants;
offturns either off); BENCH_DENSE_ID_INDEX defaults on for the seven engines that lack an index; served ArcadeDB arms use the official arcadedb-driver 0.2.0 (sync) for /query and /command, with the lean HTTP client run as a twin cell (run_id ends_clientlean, rowarcadedb_http_client_role=sensitivity; lean, not lean2, because lean is what the October rows used); l4 q_high and q_groupby are read with ArcadeDB to_arrow and DuckDB to_arrow_table (SQLite stays on fetchall), with to_list / fetchall timed beside them as<query>_plain_ms; embedded ArcadeDB arms leave -Xms unset. BENCH_ARCADEDB_TS_NDJSON stays off.JVM initial heap per engine: ArcadeDB embedded unset (changed); ArcadeDB served -Xms = -Xmx via ARCADEDB_OPTS_MEMORY (unchanged); Elasticsearch -Xms = -Xmx, as its docs say; Neo4j initial = max heap, as its docs say; QuestDB gets no setting from the harness. Links are in PROTOCOL.md. Note that the ArcadeDB server script itself leaves the heap to the JVM unless set, so embedded and served ArcadeDB now differ in -Xms; that is a one-line change in runner.py if wanted.
Also fixed: the runner did not forward BENCH_INSTRUMENT to the cell, so a container would have stamped and followed 2026-10 whatever the launcher set. It is forwarded now (unset stays unforwarded).
Precondition: the l4 to_arrow digest gate (the native time-series datetime column) must pass in a smoke before any 2026-11 stage runs.
Not built: the PostgreSQL wire sensitivity row (the harness has no adapter) and the official driver for bulk loads, which stay on HTTP batch. The typed-read digest check on the native time-series datetime column still has to pass on the first 2026-11 stage. The client image needs a rebuild to get arcadedb-driver. No engine was run; the 27 new tests in test_next_instrument_switches.py cover each switch under both instruments, and the full benchmarks/experiments suite passes (797 passed, 15 skipped).
CAMPAIGN rows 81, 82, 84 and the note below the table, PROTOCOL, FAIRNESS, PAGE-SPEC and COMPARATORS are updated.
🤖 Generated with Claude Code