Skip to content

[ruby] Optimizations for massive improvement in SQL statements and CPU use - #1566

Open
brandur wants to merge 6 commits into
masterfrom
brandur-ruby-drain-scaling
Open

[ruby] Optimizations for massive improvement in SQL statements and CPU use#1566
brandur wants to merge 6 commits into
masterfrom
brandur-ruby-drain-scaling

Conversation

@brandur

@brandur brandur commented Oct 30, 2026

Copy link
Copy Markdown
Contributor

This one comes by way of @jpcamara -- it turns out that we were doing some
unreasonably inefficient stuff in Ruby that was significantly impacting
possible through.

After fixes, we observe these improvements in a benchmark:

  • SQL statements/job: 15.677 → 1.292
  • Total CPU/job: 4.007 → 0.947 ms, about 76% lower
  • Connection resets/run: 19,906 → 0
  • Follow-up job reads/run: 40,000 → 0

Most of the slowdown was the result of these problems:

  • We were using .connection to lease a connection a job's thread. Each job
    thread was short-lived, and when ActiveRecord eventually reclaimed it, it had
    to reset by issueing a whole bunch of queries:

    • DISCARD ALL
    • Five session setting SET statements
    • Four queries against pg_type for type information

    The fix is to switch to with_transaction to return the borrowed
    connection to the pool without needing a reset.

  • We weren't doing RETURNING river_job.*, instead returning job information
    via ID and doing a fetch. Fix saves two reads per job.

  • Single statement operations were using explicit transaction commands.
    Previously, claim and completion were wrapping themselves with BEGIN and
    COMMIT. With the reads above removed, that can go away, saving many
    operations issued.

jpcamara and others added 6 commits October 30, 2026 17:01
@jpcamara @claude @brandur
[ruby] don't lease Active Record connections to job threads
05360b4 
The Active Record driver read the class's connection, which leases it to
the calling thread for the thread's lifetime. Jobs complete on their own
short-lived thread, so every job left a leased connection behind. When the
pool ran dry, Active Record reaped each one from its dead thread and reset
it: DISCARD ALL, the session SETs, and a reload of the type map, then
fresh prepared statements. Borrow the connection for each call instead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Px5N6n2zvu9WqL1Nk8hruU
@jpcamara @claude @brandur
[ruby] Keep maintenance error backoff separate from producers
8aca9b9 
Current master already waits on a dedicated condition for normal maintenance,
with generation tracking for leadership notifications. Preserve that loop
and also use the dedicated condition during error backoff, so completions
and insertions cannot trigger an immediate retry. Stop still wakes both
conditions.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Px5N6n2zvu9WqL1Nk8hruU
@jpcamara @claude @brandur
[ruby] return job rows from claim and finish updates
233a231 
Claims, completions, and state changes returned ids and then read each
job again by id. They now return the rows from the UPDATE itself, as Go's
JobGetAvailable and JobSetStateIfRunningMany do. Each driver decodes the
returned rows with its existing decoders, and SQLite returns
SQLITE_JOB_COLUMNS. The claim is a single statement, so it needs no
transaction and no reads after it commits.

Finishing a running job locks it by primary key in a CTE and tests the
locked row's state there, like Go's JobRetry and JobUpdate. With the state
test in the statement's own WHERE, Postgres planned it on the fetching
index, and the scan read every running entry, live and dead.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Px5N6n2zvu9WqL1Nk8hruU
@brandur
[ruby] Adapt connection borrowing to the current runtime
3f02b5a 
Borrow connections during driver initialization and transaction handling,
including the transaction-joining path added since the original drain
optimization. Keep an application's existing lease and rollback behavior.

Exercise initialization, insertion, claiming, completion, reads, successful
transactions, rollback, and exceptions from a short-lived thread on both
Postgres and file-backed SQLite. Verify that nested driver calls preserve
the caller's lease, and that shared claim/completion paths return rows
without rereading jobs by ID.

Move the changelog entries into Unreleased and retain the notification-aware
maintenance loop. The rebased completion SQL also retains current-attempt
checks used by transactional completion.
@brandur
[ruby] Add a reproducible Active Job drain benchmark
f47cb5e 
Add a multi-process drain harness using riverqueue-rails' Active Job adapter
and worker. Verify every serialized job is delivered exactly once and every
persisted attempt counter is one. Capture pg_stat_statements and process CPU
before stopping the workers, and emit the measurements as JSON.

Boot workers in fresh Ruby interpreters to avoid a native libpq fork crash
on macOS. Worker boot and job insertion are outside the measured drain.
Use a dedicated local Postgres server because the harness resets its SQL
statistics server-wide. Each run uses and removes its own migrated schema.

Compared upstream e898054 with the adapted changes at 3f02b5a on an Apple
M4 (10 CPU cores, 24 GiB), Ruby 4.0.2, Active Record 8.1.4, and Postgres 18.0.
The dedicated server used pg_stat_statements, 128 MB shared_buffers, 500
max_connections, and fsync/synchronous_commit enabled.

Each run drains 20,000 otherwise no-op Active Job deliveries with 20 worker
threads per process. fetch_poll_interval stays at its 1.0 second default.
Run three paired rounds for each configuration, alternating revision order:
baseline/candidate, candidate/baseline, baseline/candidate. All 36 runs
passed delivery and attempt checks (720,000 jobs total).

Throughput in jobs/s, median [min-max]:

Processes x threads | Cooldown | Upstream              | This branch           | Ratio
8 x 20              | 0.1      |   733 [731-752]        |   740 [739-755]        | 1.01x
8 x 20              | 0.01     | 2,229 [2,185-2,275]    | 5,424 [5,420-5,865]    | 2.43x
8 x 20              | 0.001    | 2,125 [2,067-2,126]    | 8,525 [8,469-8,659]    | 4.01x
16 x 20             | 0.1      | 1,426 [1,423-1,459]    | 1,466 [1,461-1,506]    | 1.03x
16 x 20             | 0.01     | 2,060 [2,041-2,108]    | 7,222 [7,162-7,401]    | 3.51x
16 x 20             | 0.001    | 2,006 [1,931-2,074]    | 7,271 [7,062-7,310]    | 3.62x

Median costs at 8 x 20 and fetch_cooldown=0.001:

Metric                         | Upstream | This branch
SQL statements/job             |   15.677 |       1.292
DISCARD ALL calls/run          |   19,906 |           0
Follow-up job reads/run         |   40,000 |           0
Completion buffers/call         |   83.123 |      33.031
Completion mean execution (ms)  |    0.553 |       0.169
Ruby CPU/job (ms)               |    2.381 |       0.564
Postgres CPU/job (ms)           |    1.627 |       0.383
Total CPU/job (ms)              |    4.007 |       0.947

CPU figures are OS process-time estimates; Postgres includes the dedicated
server's background and monitoring work. Each cell is an independent median.
The per-delivery audit adds a small, identical in-memory cost to both versions.
The default cooldown remains the throughput limit. At shorter cooldowns the
branch uses about 76% less total CPU/job; on this machine, 8 processes reach
higher peak throughput than 16.

To reproduce one run, select the checkout with BUNDLE_GEMFILE while using
the same harness for both revisions:

  BUNDLE_GEMFILE=<checkout>/ruby/rails/riverqueue-rails/Gemfile \
    ruby ruby/bench/drain.rb \
    --database-url postgres://localhost:55432/river_bench \
    --jobs 20000 --processes 8 --threads 20 --cooldown 0.001

Validation: Ruby make test (3,055 examples, zero failures), make lint, and
make type-check pass. Core and both driver suites retain 100% line and
branch coverage. The benchmark matrix exercises the final harness against
both revisions.
@brandur
[ruby] Consolidate drain optimization changelog entry
66e5c0a
@brandur
brandur force-pushed the brandur-ruby-drain-scaling branch from a64b3b9 to 66e5c0a Compare October 31, 2026 00:05
bgentry
bgentry approved these changes Oct 31, 2026
View reviewed changes

@bgentry bgentry left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice fixes.

I was gonna ask if you’d had an agent check other ports to see if they’re doing this correctly, but then it made me think we should probably do some kind of regular scans against recent changes to do parity checks. I don’t think it would be deterministic enough to be used in a CI check, gonna think about the best mechanism.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bgentry bgentry bgentry approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@brandur @bgentry @jpcamara