Skip to content

feat(linux): in-process JPEG for screenshot, with region and the Swift arguments (#32 item 5) - #82

Merged
chaodu-agent merged 4 commits into
mainfrom
feat/linux-screenshot-jpeg
Nov 7, 2026
Merged

chaodu-agent merged 4 commits into
mainfrom
feat/linux-screenshot-jpeg

Conversation

@chaodu-agent

@chaodu-agent chaodu-agent commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Refs #32. This PR implements item 5 only and does not close the issue.

  • 5. In-process JPEG: Connect-shaped call returns image/jpeg ≤ 300 KB at 1080p q0.6 (measured 50,546 B); structuredContent.points {width,height} present.

What changes

Debian's grim is built without libjpeg, so every Linux frame was PNG (~2 MB at 1080p). For JPEG the daemon now asks grim for raw PPM (-t ppm) and encodes it in-process with jpeg-encoder 0.7. That crate is pure Rust and has no dependencies. grim still does the crop (-g) and the scaling (-s), so there is no in-process resampling. I picked jpeg-encoder over image because we only need an encoder: there is nothing to decode (PPM is parsed in ~40 lines) and nothing to resize.

screenshot now follows the Swift ScreenshotTool contract:

Argument Linux before Linux now macOS
format png default, jpeg silently returned PNG jpeg default, png jpeg default, png
quality integer 1–100, default 80; 0.6 was ignored number 0–1, default 0.7, floored at 0.1 (1 = 100%); above 1 up to 100 is read as a percentage 0–1, default 0.7
scale 0.05–2.0, default 0.5 unchanged 0.1–2.0, default 1.0
region — {x,y,width,height} in display pixels, grown to whole pixels and clipped to the output layout; fully outside → error same (points)
display ignored 0 only; other values error index

structuredContent adds display, points (layout size), region (the effective crop, or the whole layout), and image {width,height,bytes}. The old top-level bytes and format stay. The result also carries a text caption after the image, as on macOS. points and the clip come from seat::layout(): a live virtual pointer's layout when one exists (so screenshot and mouse use one coordinate space), otherwise a short Wayland connection that binds only wl_output. A screenshot never creates an input device.

A malformed argument (non-number scale, unknown format, quality outside 0–100, bad region) now gets -32602. Before, it fell back to the default without saying so.

The Linux default scale stays 0.5. Connect always sends scale 1, and the brief asked only for format and quality to match macOS.

Not touched: src/attach/client.rs, src/http.rs. Cargo.toml gains one line, placed right under [dependencies] to keep rebases with #66 easy. Cargo.lock gains one package.

Measurements

a Raspberry Pi labwc seat, 1920×1080, branch binary run as a scratch instance on loopback:

Call Result Bytes Time
Connect-shaped display 0, scale 1, quality 0.6, format jpeg image/jpeg 1920×1080 50,546 0.07 s
scale 1, format png image/png 1920×1080 2,065,950 1.03 s
defaults (jpeg q0.7, scale 0.5) image/jpeg 960×540 19,612 0.40 s
scale 0.5, format png (old default) image/png 960×540 484,433 0.70 s
region 0,0 800×600, scale 1 image/jpeg 800×600 17,394 0.03 s
region 1800,1000 400×400, scale 2 (clipped to 120×80) image/jpeg 240×160 1,560 0.02 s

The desktop in this test is mostly wallpaper, so a text-heavy page will encode larger. The ≤ 300 KB target still has about 6× headroom.

Release binary (aarch64, cargo build --release): 2,607,408 → 2,824,552 B unstripped (+217 KB), 1,840,088 → 2,036,696 B stripped (+196 KB). The release tarball is a plain glibc cargo build --release, and its build is unchanged.

Tests

New unit tests:

  • frame.rs: PPM parse with comments, malformed PPM rejected, encoding a known buffer (SOI/EOI and SOF0 size checked), quality monotonic, >65535 px refused, PNG IHDR read.
  • screen.rs: Swift defaults, Connect-shaped args, quality fraction and percent mapping, invalid args → -32602 / display 1 → -32000, region clip (inside, fractional, partial, huge, outside, negative layout origin).
  • wlroots.rs: grim argv.

Ran on a Raspberry Pi (aarch64): cargo fmt --check, cargo clippy --release --all-targets -- -D warnings, cargo test --release (50 passed), and markdownlint-cli2@0.17.2 "docs/**/*.md" (0 errors).

Docs: poc/reverse-attach-linux/README.md (drops the "PNG only" limitation), docs/linux-setup.md, the top-level README platform table, and the docs/parity-checklist.md screenshot row.

Release note: quality: 1 means 100% (fraction form, as on macOS); integers 2–100 are percentages.

chaodu-agent and others added 2 commits November 6, 2026 19:21
…t arguments (#32 item 5)

Debian's grim has no libjpeg, so every Linux frame was PNG (~2 MB at 1080p).
For JPEG the daemon now asks grim for raw PPM and encodes it with
jpeg-encoder 0.7 (pure Rust, zero deps); grim still crops (-g) and scales (-s).

screenshot now takes the macOS contract: format jpeg (default) | png,
quality 0-1 (default 0.7; 1-100 accepted), scale (default 0.5), region
{x,y,width,height} clipped to the output layout, display 0. Invalid
arguments are -32602 instead of silently defaulting. structuredContent adds
points / region / image {width,height,bytes} / display (bytes and format kept),
plus a text caption like Swift. points and the clip come from the virtual
pointer's layout, so they share mouse's coordinate space.

Verified on rpi1 (aarch64): cargo fmt --check, cargo clippy --release
--all-targets -D warnings, cargo test --release (50 passed),
OAB_RELEASE_VERSION=0.8.2 cargo build --release; markdownlint-cli2 0.17.2
docs/**/*.md 0 errors. Real seat on rpi2 (labwc, 1920x1080): Connect-shaped
call q0.6 scale 1 -> 50,546 B JPEG in 0.07 s vs 2,065,950 B PNG in 1.03 s.
Release binary +217 KB unstripped (+196 KB stripped).

Refs #32
…ointer (#32 item 5)

Review round 1 (opus5):
- MEDIUM: seat::layout() went through with_pointer, so a screenshot could
  connect a zwlr_virtual_pointer_v1 (input device, 400 ms settle, pointer
  mutex, region failing without virtual-pointer support). layout() now
  returns a live pointer's layout when the slot is free (so screenshot and
  mouse agree) and otherwise runs query_layout(): a short Wayland connection
  that binds only wl_output. The wl_output event parsing is shared with
  Pointer::connect via apply_output_event.
- LOW: quality 0-1 is a fraction floored at 0.1 (Swift's minimum), 10-100 a
  percentage, and 1 < q < 10 is rejected as ambiguous. Tests, schema and
  README updated.
- LOW (hotplug): screenshot now follows the outputs unless a pointer is live;
  docs say so.

Verified: rpi1 (aarch64) cargo fmt --check, cargo clippy --release
--all-targets -D warnings, cargo test --release (50 passed, 1 ignored),
OAB_RELEASE_VERSION=0.8.2 cargo build --release (stripped 2,036,696 B,
unchanged); markdownlint-cli2 0.17.2 docs/**/*.md 0 errors. rpi2 labwc seat:
ignored test query_layout_reads_a_live_seat passes; scratch instance
screenshot: 1080p q0.6 JPEG 50,518 B vs PNG 2,065,818 B, region clip and
error cases as before.
@chaodu-agent

Copy link
Copy Markdown
Contributor Author

Review round 1 → c04d512

Reviewers on b0bdebd: sol LGTM; opus5 CHANGES_REQUESTED (1 MEDIUM, 2 LOW).

Fixed

  • MEDIUM: screenshot could create the virtual pointer. seat::layout() went through with_pointer, so a screenshot with no live pointer ran Pointer::connect() (input device, 400 ms settle, waiting on the pointer mutex, and region failing on compositors without virtual-pointer support). layout() now returns a live pointer's layout when the slot is free, so mouse and screenshot still agree, and otherwise calls a new query_layout(): a short Wayland connection that binds only wl_output and closes. The wl_output event parsing is shared with Pointer::connect (apply_output_event). Note: the daemon's startup warm_up already creates the pointer on main, unchanged by this PR; this fix removes screenshot's own dependency on it. Gating warm_up by profile is out of scope here.
  • LOW: quality just above 1. 0–1 is a fraction floored at 0.1 (Swift's minimum), 10–100 is a percentage, and 1 < q < 10 is rejected with -32602 as ambiguous. Tests, schema and README updated.
  • LOW: stale layout after hotplug. Screenshot now follows the outputs unless a pointer is live. Docs updated to say so.

Verified: rpi1 aarch64 fmt, clippy -D warnings, cargo test --release (50 passed, 1 ignored), release build (stripped size unchanged at 2,036,696 B), markdownlint 0 errors. On the rpi2 labwc seat, the new ignored test query_layout_reads_a_live_seat passes, and a scratch instance gave the same frames as before (1080p q0.6 JPEG 50,518 B vs PNG 2,065,818 B).

HEAD: c04d51234e6b1b3566ffada496166357ed3f8096

…ge (#32 item 5)

Round-2 review: quality values strictly between 1 and 10 returned -32602.
0..=1 stays a fraction floored at 0.1 (1 = 100%, as on macOS); values above
1 through 100 are now a rounded percentage (at least 1). Schema text, crate
README and tests updated. The crate README's old rpi1 verification line no
longer claims the JPEG call returns PNG.

Checked on rpi1 (aarch64): cargo fmt --check OK; cargo clippy --release
--all-targets -D warnings OK; cargo test --release 50 passed, 1 ignored;
cargo build --release OK. markdownlint-cli2 0.17.2 docs/**/*.md: 0 errors.
@chaodu-agent

Copy link
Copy Markdown
Contributor Author

Review round 2 (opus5 LGTM, sol CHANGES_REQUESTED on c04d512). New HEAD: 71bfc7f81de98422ed8ffb9da797707d3a63242f.

Fixed

  • MEDIUM (sol; opus5 LOW 2): quality strictly between 1 and 10 returned -32602. Now 0–1 is a fraction floored at 0.1 (1 = 100%, as on macOS) and values above 1 through 100 are a rounded percentage, so quality: 5 is 5%. Tests cover 1.0001, 1.5, 5, 9.99; schema text, crate README and PR body updated; release note added to the description.
  • LOW (sol): the crate README's old rpi1 verification line said the format jpeg call returned PNG. It now says that was PNG at the time and is JPEG since this change.

Declined (follow-up)

  • LOW (opus5): after an output hotplug, points and the region clip use the virtual pointer's connect-time layout until it reconnects. This is pre-existing (startup warm-up), documented in the crate README and linux-setup, and keeps screenshot and mouse in one coordinate space. Resyncing on layout mismatch is a separate change.

Checked on rpi1: cargo fmt --check, cargo clippy --release --all-targets -D warnings, cargo test --release (50 passed, 1 ignored live-seat test), cargo build --release; markdownlint-cli2 0.17.2 on docs/**/*.md: 0 errors.

@chaodu-agent

Copy link
Copy Markdown
Contributor Author

Final review round → 71bfc7f

Both reviewers approve 71bfc7f81de98422ed8ffb9da797707d3a63242f (the current head): opus5 LGTM, sol LGTM. No HIGH or MEDIUM findings remain. This took three rounds (b0bdebd → c04d512 → 71bfc7f).

Last round

  • Confirmed fixed from earlier rounds: screenshot reads the outputs with a short connection that binds only wl_output, so it never creates a virtual pointer. quality 0–1 is a fraction floored at 0.1 (1 = 100%), and values above 1 up to 100 are a percentage.
  • Re-checked: jpeg-encoder 0.7.1 is the only new dependency (pure Rust, no transitive deps). PNG bypasses the encoder. src/attach/client.rs and src/http.rs are untouched. The release build is still glibc.
  • Remaining LOW notes, no code change:
    • quality mapping differs from Swift above 1 (Swift clamps to 1.0, Linux reads a percentage). Documented, and Connect sends 0.6. Covered by the release note in the description.
    • After an output hotplug, points and the region clip use the warm-up pointer's connect-time layout until it reconnects. This is pre-existing and documented. A possible follow-up is to compare against query_layout() and drop the pointer on mismatch.

Verification at this head: cargo fmt --check, cargo clippy --release --all-targets -- -D warnings, cargo test --release (50 passed, 1 ignored live-seat test, which also passed when run against a live labwc seat), release build on aarch64, markdownlint 0 errors. PR CI is green and the PR is MERGEABLE / CLEAN.

Measured (1080p labwc seat): Connect-shaped call 50,518 B JPEG vs 2,065,818 B PNG (~41× smaller, ~15× faster). Stripped aarch64 binary +196,608 B (+10.7%).

Ready to merge. Refs #32 (item 5 only).

Conflicts:
- poc/reverse-attach-linux/README.md: kept this branch's mouse/screenshot
  layout bullet (replacing the PNG-only bullet) and main's updated wss://
  and live-pod bullets.
- poc/reverse-attach-linux/Cargo.lock: took main's and let cargo add
  jpeg-encoder 0.7.1.
@chaodu-agent
chaodu-agent merged commit 9c926fb into main Nov 7, 2026
7 checks passed
@chaodu-agent
chaodu-agent deleted the feat/linux-screenshot-jpeg branch November 7, 2026 01:29
@chaodu-agent chaodu-agent mentioned this pull request Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant