fix(approval): a desktop approval rereads the app's window, not the whole screen

.github/workflows/desktop-replay.yml

@@ -23,6 +23,7 @@ on:
       - packages/flow/tests/e2e/test_windows_hosted_replay.py
       - packages/flow/tests/test_xfreerdp.py
       - packages/flow/tests/test_windows_attempt_ledger.py
+      - packages/flow/tests/test_approval_form_window.py
       - packages/flow/benchmark/rdp_pool/**
       - packages/flow/pyproject.toml
       - packages/flow/uv.lock
@@ -69,13 +70,13 @@ jobs:
           python ../../scripts/ci/locked_install.py --lock .github/ci-tools:win-agent
           --lock 'packages/flow[dev]'
           --editable packages/types --editable packages/flow
-      - name: Windows file-system guards (the attempt ledger)
+      - name: Windows file-system guards (the attempt ledger, the approval window)
         # The default temp folder is in the user's profile, where the ledger
         # must live; the runner's temp drive is not.
         run: |
           mkdir -p runs
           pytest -v -rs -p no:cacheprovider tests/test_windows_attempt_ledger.py \
-            --junitxml=runs/windows-guards-junit.xml
+            tests/test_approval_form_window.py --junitxml=runs/windows-guards-junit.xml
       - name: Record, compile and replay, judged from the record
         env:
           OAFLOW_WINDOWS_HOSTED_E2E: "1"

llms-full.txt

@@ -2340,6 +2340,7 @@ ledger settlement atomic and extends zero-config verifier generation.
 
 - Programs certified before the `Anchor.form_identity` and `TokenTemplate.bearing` fields were added load again. When the current workflow-contract digest does not match, the loader tries the older form, leaving out only fields on the reviewed sealed-omission list for the bundle's format version that are unset and `None`, and accepts it only if it reproduces the bound digest exactly. A program whose content changed is still refused.
 - Ledger settlements record the journal revision they observed, so a stale or concurrent observation cannot release a hold, grant a retry, complete a recheck or hand back an older result. Rechecks are bound to the original receipt and the exact set of validated later receipts.
+- A person's approval of a desktop save is no longer voided by text that changes by itself outside the app, such as the taskbar clock ticking over a minute while the run waits. Off the browser, the approval's "form unchanged" check read every word on the screen. On Windows it now reads the app window the item's verified steps were in, in reading order, with the window's frame in the digest, so any change inside that window, or a moved window, still stops the save. When a window of the app reaches outside that frame, the whole screen is read. A resumed run names the window under the earlier steps' recorded points, so an approval given during a hold still holds when the form is unchanged. Elsewhere the whole screen is read as before ([#1697](https://github.com/OpenAdaptAI/OpenAdapt/issues/1697)).
 
 ### Features
 

packages/flow/CHANGELOG.md

@@ -32,6 +32,7 @@ ledger settlement atomic and extends zero-config verifier generation.
 
 - Programs certified before the `Anchor.form_identity` and `TokenTemplate.bearing` fields were added load again. When the current workflow-contract digest does not match, the loader tries the older form, leaving out only fields on the reviewed sealed-omission list for the bundle's format version that are unset and `None`, and accepts it only if it reproduces the bound digest exactly. A program whose content changed is still refused.
 - Ledger settlements record the journal revision they observed, so a stale or concurrent observation cannot release a hold, grant a retry, complete a recheck or hand back an older result. Rechecks are bound to the original receipt and the exact set of validated later receipts.
+- A person's approval of a desktop save is no longer voided by text that changes by itself outside the app, such as the taskbar clock ticking over a minute while the run waits. Off the browser, the approval's "form unchanged" check read every word on the screen. On Windows it now reads the app window the item's verified steps were in, in reading order, with the window's frame in the digest, so any change inside that window, or a moved window, still stops the save. When a window of the app reaches outside that frame, the whole screen is read. A resumed run names the window under the earlier steps' recorded points, so an approval given during a hold still holds when the form is unchanged. Elsewhere the whole screen is read as before ([#1697](https://github.com/OpenAdaptAI/OpenAdapt/issues/1697)).
 
 ### Features
 

packages/flow/docs/design/DECOMPOSITION.md

@@ -599,7 +599,7 @@ that:
 <!-- size-ratchet:begin -->
 | File | Line budget |
 |---|---:|
-| `packages/flow/openadapt_flow/runtime/replayer.py` | 20175 |
+| `packages/flow/openadapt_flow/runtime/replayer.py` | 20138 |
 | `packages/flow/openadapt_flow/__main__.py` | 14501 |
 | `packages/flow/openadapt_flow/qualification.py` | 5963 |
 | `packages/flow/openadapt_flow/ir.py` | 6711 |

packages/flow/openadapt_flow/approval_mode/form_state.py

@@ -11,7 +11,11 @@
   Password fields contribute only whether they're empty.
 - Elsewhere it reads the text on the screen with the engine's local OCR,
   as a sorted list of words, so a cursor blink or hover doesn't count as a
-  change but a changed value does.
+  change but a changed value does. When the backend names the app window the
+  item's verified steps were in (``approval_form_region``), only that window
+  is read, in reading order and with its frame in the digest, so text
+  elsewhere on the screen that changes by itself (a clock in a taskbar, a
+  notification) is not mistaken for an edited form.
 
 Only a digest leaves this module. The digest stays in the run directory; it
 never enters the report or anything sent to Cloud.
@@ -20,12 +24,15 @@
 from __future__ import annotations
 
 import hashlib
+import io
 import json
 import re
-from typing import Any, Optional
+from typing import Any, Optional, Sequence
 
 #: (kind, "sha256:<hex>")
 FormState = tuple[str, str]
+#: x, y, width, height in frame pixels.
+Region = tuple[int, int, int, int]
 
 _DOM_FORM_STATE_JS = """() => {
   const rows = [];
@@ -99,30 +106,65 @@ def dom_form_state(backend: Any) -> Optional[FormState]:
     return _digest("dom", payload)
 
 
-def screen_text_state(vision: Any, frame_png: bytes) -> Optional[FormState]:
-    """Sorted words on the screen, or ``None`` when OCR found nothing."""
-
-    try:
-        lines = vision.ocr(frame_png)
-    except Exception:  # noqa: BLE001 - unreadable is not "unchanged"
-        return None
+def _words(vision: Any, png: bytes) -> list[str]:
+    lines = vision.ocr(png)
     words: list[str] = []
     for line in lines or []:
         text = str(getattr(line, "text", "") or "")
         words.extend(word.casefold() for word in _WORD.findall(text))
-    if not words:
+    return words
+
+
+def _crop(png: bytes, region: Region) -> bytes:
+    import io
+
+    from PIL import Image
+
+    x, y, w, h = region
+    with Image.open(io.BytesIO(png)) as image:
+        out = io.BytesIO()
+        image.crop((x, y, x + w, y + h)).save(out, format="PNG")
+    return out.getvalue()
+
+
+def screen_text_state(
+    vision: Any, frame_png: bytes, regions: Optional[Sequence[Region]] = None
+) -> Optional[FormState]:
+    """Sorted words on the screen, or ``None`` when OCR found nothing.
+
+    With ``regions`` (the app's window), only those parts of the frame are
+    read, and the words keep their reading order, so swapped values change
+    the digest. The regions are part of the digest.
+    """
+
+    try:
+        if not regions:
+            words = _words(vision, frame_png)
+            if not words:
+                return None
+            return _digest("screen_text", sorted(words))
+        rows = [
+            [list(region), _words(vision, _crop(frame_png, region))]
+            for region in regions
+        ]
+    except Exception:  # noqa: BLE001 - unreadable is not "unchanged"
         return None
-    return _digest("screen_text", sorted(words))
+    if not any(words for _region, words in rows):
+        return None
+    return _digest("screen_text", {"rows": rows})
 
 
 def capture_form_state(
-    backend: Any, vision: Any, frame_png: Optional[bytes] = None
+    backend: Any,
+    vision: Any,
+    frame_png: Optional[bytes] = None,
+    regions: Optional[Sequence[Region]] = None,
 ) -> Optional[FormState]:
     """The best available field state for this backend, or ``None``.
 
     A backend may provide ``form_state()`` returning JSON-serializable data;
     a browser backend is read through its page; any other backend falls back
-    to the words on the screen.
+    to the words on the screen, in ``regions`` when they are given.
     """
 
     custom = getattr(backend, "form_state", None)
@@ -138,11 +180,58 @@ def capture_form_state(
             frame_png = backend.screenshot()
         except Exception:  # noqa: BLE001
             return None
-    return screen_text_state(vision, frame_png)
+    return screen_text_state(vision, frame_png, regions)
+
+
+def approval_window_regions(
+    backend: Any,
+    frame_png: Optional[bytes],
+    save_step: Optional[str],
+    results: Sequence[Any] = (),
+) -> Optional[list[Region]]:
+    """The app window's part of the frame, for a screen-text form state.
+
+    A backend that knows the window this item's verified steps were in
+    (``approval_form_region``) bounds the read to that window, so the whole
+    form the person saw is read, and the clock or a notification outside it
+    is not. The save step is left out, so the approval request (taken before
+    the save resolves) and the recheck after the answer ask about the same
+    window; its frame is part of the digest, so a moved or resized window
+    halts the save. The earlier steps' points (``results``, the run's step
+    results so far) let a resumed run, which verified nothing in this process
+    yet, name the same window. None, and the whole screen is read, when the
+    backend cannot say. Never raises.
+    """
+
+    region_of = getattr(backend, "approval_form_region", None)
+    if not callable(region_of) or frame_png is None:
+        return None
+    try:
+        points = [
+            (int(prior.resolution.point[0]), int(prior.resolution.point[1]))
+            for prior in results
+            if prior.step_id != save_step and prior.resolution is not None
+        ]
+        region = region_of(save_step, points)
+        if region is None:
+            return None
+        from PIL import Image
+
+        with Image.open(io.BytesIO(frame_png)) as image:
+            width, height = image.size
+        x, y, w, h = (int(v) for v in region)
+    except Exception:  # noqa: BLE001 - no window means the whole screen
+        return None
+    left, top = max(0, x), max(0, y)
+    right, bottom = min(width, x + w), min(height, y + h)
+    if right <= left or bottom <= top:
+        return None
+    return [(left, top, right - left, bottom - top)]
 
 
 __all__ = [
     "FormState",
+    "approval_window_regions",
     "capture_form_state",
     "dom_form_state",
     "screen_text_state",

packages/flow/openadapt_flow/backends/win_agent/server.py

@@ -1765,6 +1765,7 @@ def __init__(self) -> None:
                 [wintypes.HWND, wintypes.LPWSTR, ctypes.c_int],
                 ctypes.c_int,
             ),
+            ("GetSystemMetrics", [ctypes.c_int], ctypes.c_int),
         ):
             fn = getattr(dll, name)
             fn.argtypes = args
@@ -1876,6 +1877,15 @@ def title(self, hwnd: int) -> str:
         self._dll.GetWindowTextW(hwnd, buffer, 512)
         return buffer.value
 
+    def virtual_origin(self) -> tuple[int, int]:
+        """Top-left of the virtual desktop (SM_XVIRTUALSCREEN and
+        SM_YVIRTUALSCREEN): negative when a monitor sits left of or above
+        the primary. The screenshot (mss ``monitors[0]``) starts there."""
+        return (
+            int(self._dll.GetSystemMetrics(76)),
+            int(self._dll.GetSystemMetrics(77)),
+        )
+
     def is_cloaked(self, hwnd: int) -> Optional[bool]:
         """A cloaked window (another virtual desktop, a suspended UWP app) is
         'visible' to IsWindowVisible but not on screen. None: unknown."""
@@ -2035,7 +2045,7 @@ def work() -> None:
 
 
 _MAX_OWNER_CHAIN = 32
-_WINDOW_OPERATIONS = frozenset({"window-root-at", "owned-window-over"})
+_WINDOW_OPERATIONS = frozenset({"window-root-at", "window-bounds", "owned-window-over"})
 
 
 def _intersects(
@@ -2091,6 +2101,70 @@ def _window_operation(
             "status": "ok",
             "window": {"hwnd": root, "pid": user32.pid_of(root)},
         }
+    if operation == "window-bounds":
+        data = _exact_object(
+            payload, required=frozenset({"hwnd", "pid"}), label=operation
+        )
+        hwnd, pid = data["hwnd"], data["pid"]
+        if any(
+            not isinstance(v, int) or isinstance(v, bool) or v <= 0 for v in (hwnd, pid)
+        ):
+            raise AgentRequestError(400, "invalid_schema", "hwnd and pid required")
+        if (
+            not user32.is_window(hwnd)
+            or user32.pid_of(hwnd) != pid
+            or not user32.is_visible(hwnd)
+        ):
+            return {"status": "ok", "window": None}
+        rect = user32.rect(hwnd)
+        if rect is None or rect[2] <= rect[0] or rect[3] <= rect[1]:
+            return {"status": "ok", "window": None}
+        # The caller crops the screenshot at these bounds as pixels. That
+        # holds only when the virtual desktop starts at (0, 0); with a monitor
+        # left of or above the primary the crop would be offset, so answer
+        # None (and the caller reads the whole screen). Unknown is None too.
+        origin = getattr(user32, "virtual_origin", None)
+        try:
+            at_zero = callable(origin) and tuple(origin()) == (0, 0)
+        except Exception:  # noqa: BLE001 - unknown origin is no frame
+            at_zero = False
+        if not at_zero:
+            return {"status": "ok", "window": None}
+        # The frame bounds the whole form only if every window the app shows
+        # sits inside it: a dialog or second window of the app that reaches
+        # outside the frame (or is on another monitor) answers None, and the
+        # caller reads the whole screen.
+        for candidate in user32.top_level_windows():
+            if candidate == hwnd or user32.pid_of(candidate) != pid:
+                continue
+            if not user32.is_visible(candidate):
+                continue
+            cloaked = user32.is_cloaked(candidate)
+            if cloaked is None:
+                return {"status": "ok", "window": None}
+            if cloaked:
+                continue
+            other = user32.rect(candidate)  # its visible frame, as for hwnd
+            if other is None:
+                return {"status": "ok", "window": None}
+            if other[2] <= other[0] or other[3] <= other[1]:
+                continue
+            if (
+                other[0] < rect[0]
+                or other[1] < rect[1]
+                or other[2] > rect[2]
+                or other[3] > rect[3]
+            ):
+                return {"status": "ok", "window": None}
+        return {
+            "status": "ok",
+            "window": {
+                "hwnd": hwnd,
+                "pid": pid,
+                # The visible frame (DWM bounds), in screenshot pixels.
+                "bounds": [rect[0], rect[1], rect[2] - rect[0], rect[3] - rect[1]],
+            },
+        }
     data = _exact_object(
         payload, required=frozenset({"hwnd", "pid", "region"}), label=operation
     )
@@ -2791,6 +2865,7 @@ def do_GET(self) -> None:  # noqa: N802 - stdlib naming
                             "guarded_input_v1",
                             "uia_v1",
                             "uia_read_v1",
+                            "window_bounds_v1",
                             "uia_authoring_tree_v1",
                             *(
                                 ["legacy_waa_input_v1"]
@@ -2832,6 +2907,7 @@ def do_POST(self) -> None:  # noqa: N802 - stdlib naming
                 "/uia/authoring-tree": ("uia", "authoring-tree"),
                 "/window/root-at": ("uia", "window-root-at"),
                 "/window/owned-over": ("uia", "owned-window-over"),
+                "/window/bounds": ("uia", "window-bounds"),
                 "/uia/text-at-point": ("uia", "text-at-point"),
                 "/uia/focused-field-label": ("uia", "focused-field-label"),
                 "/uia/identity-candidates-at": ("uia", "identity-candidates-at"),