Repository navigation
Windows: agent CLI output in ANSI/GBK codepage crashes subprocess reader threads (fix: errors="replace") #64
Copy link
Copy link
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
### Environment
- lh-harness 0.1.6 (PyPI), Windows 10 (10.0.26200)
- Python 3.13.14
- Agent backend: `codex` (third-party build whose `--version` output contains GBK bytes on zh-CN Windows)
- Windows "Use Unicode UTF-8 for worldwide language support" **enabled** (system ACP = 65001), so Python's text-mode subprocess defaults to UTF-8
### What happened
`lh-harness run` starts, creates the run directory, enters Round 1, then both
subprocess reader threads die and the manager episode ends in
`UnicodeDecodeError`. The run is then misreported as `provider_network` /
`provider_model_unavailable`:
```
── Round 1/25 ──
[manager] running...
Exception in thread Thread-6 (_readerthread):
...
File "subprocess.py", line 1615, in _readerthread
buffer.append(fh.read())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte
Exception in thread Thread-7 (_readerthread):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb3 in position 0: invalid start byte
[manager] failed · 308.8s
Stopped: provider_network
```
### Root cause
`utils/agent_cli.py` validates the agent binary with:
```python
subprocess.run([path, "--version"], capture_output=True, text=True, ...)
```
On zh-CN Windows, agent CLIs (and Windows helpers such as `taskkill`) may emit
ANSI/GBK-encoded output while the harness decodes pipes as UTF-8 (system UTF-8
codepage or `PYTHONUTF8=1`). With two captured pipes this spawns two reader
threads; both crash, the version probe returns nothing, and the failure is
misattributed to the provider.
### Fix
Make every text-mode subprocess call tolerant of non-UTF-8 bytes by adding
`errors="replace"`. Four files (five call sites) on 0.1.6:
```diff
--- a/src/lh_harness/utils/agent_cli.py
+++ b/src/lh_harness/utils/agent_cli.py
@@ def _probe (subprocess.run([path, "--version"], ...)
capture_output=True,
text=True,
+ errors="replace",
timeout=timeout,
--- a/src/lh_harness/model_catalog.py
+++ b/src/lh_harness/model_catalog.py
@@ _codex_app_server_models
stderr=subprocess.DEVNULL,
text=True,
+ errors="replace",
bufsize=1,
--- a/src/lh_harness/utils/process_group.py
+++ b/src/lh_harness/utils/process_group.py
@@ _run_quiet
capture_output=True,
text=True,
+ errors="replace",
timeout=15,
--- a/src/lh_harness/plugins/npm.py
+++ b/src/lh_harness/plugins/npm.py
@@ two subprocess.run call sites
capture_output=True,
text=True,
+ errors="replace",
```
(`adapters/deepseek_runner.py` already passes `errors="replace"` — these five
sites just missed it.)
With this patch (plus PR #29 for the control-bus blocker), a full
Manage→Execute→Audit run completes on Windows — verified with `claude_code`
backend, 2 rounds, auditor confirmed against the filesystem independently.
### Two related observations
1. **`doctor` blind spot:** on stock 0.1.6, `lh-harness doctor` prints
`Doctor result: ready` even though `run` is hard-broken on Windows by the
POSIX-only control-bus hardening (`O_NOFOLLOW` / `O_DIRECTORY` / `dir_fd`,
see `supervisor/control_bus.py:127`). PR #29 fixes the run path. It would
help if `doctor` probed control-bus capability on Windows and warned
accordingly, since "ready" currently over-promises.
2. **Misattributed abort reason:** when an agent CLI probe dies in a reader
thread, the run is reported as `provider_network`. The actual cause (local
subprocess decode crash) only shows in the episode `agent.log`. A stderr
pattern check for `UnicodeDecodeError` could surface the real reason.
Happy to turn the `errors="replace"` changes into a PR if the maintainers
prefer.Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
tasks: PR numbers for ops (#64, mcp-cognizioware #65) and auth-broker (…
Upper9527 commentedon Oct 21, 2026
The decoding fix is merged in #71: the five missing text-mode subprocess sites now use
errors="replace", and the GBK banner regression successfully extracts the version instead of crashing. The focused regression fails on the old main and passes after the patch; the full combined suite passed (457 passed, 1 skipped).The native Windows bootstrap/doctor capability gap is still covered by #70; fixing output decoding does not make all Windows run paths supported. The README platform statement is corrected in #92. The patch is on main, not yet a new PyPI release.
Upper9527 commentedon Oct 21, 2026
The subprocess output-decoding fix from #71 is now published on PyPI in v0.1.8. The affected text-mode subprocess calls now use
errors="replace"so undecodable output does not crash reader threads.Native Windows bootstrap/support remains tracked separately in #70; WSL2 is still the recommended Windows environment.
Upgrade with pip:
For a uv tool installation:
Verify the installed version with
lh-harness --version.v0.1.8 release notes
fix: tolerate non-UTF-8 output from agent CLIs in text-mode subproces…