Skip to content

Linux hands node: ui_snapshot and ui_action via AT-SPI (deferred) #102

Open

Description

@chaodu-agent

Split out of #67, which delivered both tools on macOS: ui_snapshot (0.9.0, #93) and ui_action (0.10.0, #96, with the sheet fix #98/#99 in 0.10.1).

Status: deferred. We'll pick this up when a Linux hands node actually needs it. Until then, Linux callers use screenshot + mouse/key, and browser_snapshot for web pages.

Scope when picked up

  • ui_snapshot on the Rust Linux port (poc/reverse-attach-linux) over AT-SPI (D-Bus), with the same output contract as macOS: role, name, value, states, frame in screenshot coordinates, a per-snapshot ref and snapshot_id, node/depth limits, a deadline, truncated/timed_out, and secure (password) fields masked.
  • ui_action(snapshot_id, ref, action) via AT-SPI Action/EditableText/Component: press, set_value, focus. Stale refs are refused, the cursor never moves, it is owner-only, and secure values are never logged. A switchboard secure set_value refusal applies, as on macOS.
  • Evaluate AT-SPI coverage on our wlroots seats (labwc/sway): which toolkits expose a usable tree (GTK, Qt, Electron/Chromium with accessibility enabled), and whether coordinates match the grim screenshot space.
  • Shared conformance: extend conformance/profile_vectors.json; add parity-checklist rows.

Open questions

  • Is the AT-SPI bus available to the daemon's systemd --user session on a headless or --headless-seat node?
  • Do Wayland compositors report usable global coordinates through AT-SPI?

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestp3Priority 3: RFC, roadmap, nice-to-have or deferred

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions