Repository navigation
[feat] Map proxy budget refusals to a clear starter-credits message - #6168
Conversation
…ssage A run funded by starter credits routes through a budgeted model proxy. When the organization's allowance is spent, the proxy refuses at admission with a 429 whose body names the key and its spend. The condenser had no branch for it, so the raw provider JSON reached the chat — the user saw a key fragment and a cost figure, and no way to read "you are out of credits, add your own key". Classify these refusals on the BODY, never on the status alone: 429 covers budget refusals, per-key throttling, and upstream provider quota alike, and telling a user their credits are gone when they were merely throttled is worse than saying nothing. `conciseError` now returns the message half of `classifyRunError`, which also yields a stable code the error event carries to the client. Copy is placeholder.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe runner now classifies errors with stable codes, handles Pi retry-only output as swallowed errors, and includes codes in error events. Vercel adapters validate and propagate these codes while preserving schema-conforming output. ChangesRunner error classification and propagation
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: 🔵 Low · up to The PR replaces raw proxy budget errors with safe user messaging and stable error codes. It is mergeable with owner awareness because the code validation currently permits a trailing newline, which could prevent clients from recognizing an otherwise valid failure state. Sequence Diagram(s)sequenceDiagram
participant Runner
participant classifyRunError
participant AgentEvent
participant VercelAdapter
participant VercelSDK
Runner->>classifyRunError: classify run error
classifyRunError-->>Runner: message and stable code
Runner->>AgentEvent: emit error event
AgentEvent->>VercelAdapter: forward message and code
VercelAdapter->>VercelAdapter: validate code slug
VercelAdapter->>VercelSDK: emit schema-conforming error frame
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1⚔️ Resolve merge conflicts 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
… hiding it
Live QA on the dev stack surfaced the exact frame a spent funded key produces, and
two things about it that the mapping alone did not cover.
The refusal arrives status-prefixed onto a bare body (not `{"error": {...}}`-wrapped),
and names the key twice: `Key=` is the organization UUID that aliases it, and the
parenthesized token is the key's suffix. Both are pinned as a fixture and asserted
absent from the mapped message, along with the spend figures.
The bigger one: pi-acp renders Pi's auto-retry events as assistant message chunks, so
a retried refusal leaves `run.output()` holding "Retrying (attempt 1/3, waiting 2s)..."
and nothing else. The swallowed-error recovery is guarded on an empty output, so it was
skipped and the turn shipped that chatter as its answer — the mapped message never had
a chance to fire. A turn holding only retry notices now counts as the empty turn it is.
The three starter-credits messages spoke about "this organization's starter credits" and "the free starter credits program", which names nothing the reader recognises. They now say Agenta credits, so the person reading the blank turn knows whose credits ran out and what to do about it.
These two fixtures quote the runner's credit-exhaustion messages verbatim, so the stale wording read as the canonical one. They pass either way: each test asserts the adapter echoes back the string it was handed.
dosubot
Bot
added
size:L
Railway Preview Environment
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (5)
services/runner/src/engines/sandbox_agent/errors.ts (2)
55-73: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueExport the message constants so the copy rewrite stays a one-file change.
The five strings are marked as placeholders that the product owner will rewrite.
services/runner/tests/unit/sandbox-agent-errors.test.tspins each string verbatim (for example lines 186-188 and 199-201). The rewrite will therefore touch both files and can fail the suite for a copy-only change.Export the constants and assert against them in the tests. The tests then keep verifying the code-to-message mapping without pinning the wording.
♻️ Proposed change
-const STARTER_CREDITS_EXHAUSTED_MESSAGE = +export const STARTER_CREDITS_EXHAUSTED_MESSAGE = "Your free Agenta credits are used up. Add your own provider key to keep going."; -const STARTER_CREDITS_PROGRAM_PAUSED_MESSAGE = +export const STARTER_CREDITS_PROGRAM_PAUSED_MESSAGE = "Free Agenta credits are paused right now. Add your own provider key to continue."; -const RATE_LIMITED_MESSAGE = +export const RATE_LIMITED_MESSAGE = "Too many requests right now. Try again in a moment."; -const PROVIDER_RATE_LIMITED_MESSAGE = +export const PROVIDER_RATE_LIMITED_MESSAGE = "Too many requests to the model provider right now. Try again in a moment."; -const STARTER_CREDITS_UNAVAILABLE_MESSAGE = +export const STARTER_CREDITS_UNAVAILABLE_MESSAGE = "Agenta credits are temporarily unavailable. Try again in a moment.";
99-109: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
budget_exceededinPROXY_MARKERis unreachable.
classifyRunErrorreturns at line 141 for any body that matchesBUDGET_REFUSAL, which already containsbudget_exceeded. Execution reachesPROXY_MARKERonly after that branch does not match, so thebudget_exceededalternative can never contribute a match here.Remove the alternative to keep the marker set honest about what actually attributes a connection failure to the credits proxy.
♻️ Proposed change
-const PROXY_MARKER = /litellm|budget_exceeded|no_db_connection/i; +const PROXY_MARKER = /litellm|no_db_connection/i;sdks/python/agenta/sdk/agents/adapters/vercel/stream.py (1)
944-945: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDerive the default from
AgentRunFailed.failure_codeto keep one source of truth.
_error_partsat line 962 already falls back toAgentRunFailed.failure_codefor the same generic code.DEFAULT_RUNNER_FAILURE_CODErestates the literal"runner_error", so the two can drift apart.♻️ Proposed change
-#: What a runner-reported error is coded as when the runner names no class of its own. -DEFAULT_RUNNER_FAILURE_CODE = "runner_error" +#: What a runner-reported error is coded as when the runner names no class of its own. +DEFAULT_RUNNER_FAILURE_CODE = AgentRunFailed.failure_codesdks/python/oss/tests/pytest/unit/agents/adapters/test_vercel_stream_conformance.py (1)
400-419: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse
@pytest.mark.parametrizefor the bad-code cases.The loop runs all five cases inside one test. The first failing case stops the test, and the report does not name the offending value.
parametrizereports each case separately and removes thebad_code=bad_codedefault-argument workaround for the closure.♻️ Proposed change
+@pytest.mark.parametrize( + "bad_code", + [ + None, + "", + 429, + "Budget has been exceeded! Key=sk-EXAMPLE-not-a-real-key", + "Starter_Credits_Exhausted", + ], +) `@pytest.mark.asyncio` -async def test_a_non_slug_runner_code_falls_back_to_the_generic_code() -> None: +async def test_a_non_slug_runner_code_falls_back_to_the_generic_code(bad_code) -> None: """The code field is a stable slug, never prose. An older runner sends none, and a confused one could send a raw provider string — neither may reach the client as a code, or a consumer switching on it would branch on attacker- or vendor-controlled text. """ message = "something went wrong" - for bad_code in [ - None, - "", - 429, - "Budget has been exceeded! Key=sk-EXAMPLE-not-a-real-key", - "Starter_Credits_Exhausted", - ]: - - async def _events(bad_code=bad_code): - yield {"type": "error", "data": {"message": message, "code": bad_code}} - - parts = [ - part - async for part in agent_stream_to_vercel_stream( - _events(), trace_id="t-bad-code" - ) - ] - for part in parts: - assert_conforms(part) - assert_error_pair(parts, code="runner_error", error_text=message) + async def _events(): + yield {"type": "error", "data": {"message": message, "code": bad_code}} + + parts = [ + part + async for part in agent_stream_to_vercel_stream(_events(), trace_id="t-bad-code") + ] + for part in parts: + assert_conforms(part) + assert_error_pair(parts, code="runner_error", error_text=message)services/runner/src/engines/sandbox_agent/pi-error.ts (1)
36-37: 🩺 Stability & Availability | 🔵 TrivialAdd observability for retry-only recovery
pi-acp@0.0.29emits the literals matched byHARNESS_RETRY_NOTICE. Log when retry-only output triggers swallowed-error recovery so provider failures remain observable.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Organization UI (inherited)
Review profile: CHILL
Plan: Pro Plus
Run ID: 5ef1bb2c-c5f7-453c-a57b-bf711b6eca4a
📒 Files selected for processing (8)
sdks/python/agenta/sdk/agents/adapters/vercel/stream.pysdks/python/oss/tests/pytest/unit/agents/adapters/test_vercel_stream_conformance.pyservices/runner/src/engines/sandbox_agent/errors.tsservices/runner/src/engines/sandbox_agent/pi-error.tsservices/runner/src/engines/sandbox_agent/run-turn.tsservices/runner/src/protocol.tsservices/runner/tests/unit/sandbox-agent-errors.test.tsservices/runner/tests/unit/sandbox-agent-pi-error.test.ts
Included review availability: Your plan provides up to 8 included reviews per hour; 5 remain after this review.
| _FAILURE_CODE = re.compile(r"^[a-z][a-z0-9_]{0,63}$") | ||
|
|
||
| #: What a runner-reported error is coded as when the runner names no class of its own. | ||
| DEFAULT_RUNNER_FAILURE_CODE = "runner_error" | ||
|
|
||
|
|
||
| def _runner_failure_code(code: Any) -> str: | ||
| if isinstance(code, str) and _FAILURE_CODE.match(code): | ||
| return code | ||
| return DEFAULT_RUNNER_FAILURE_CODE |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Anchor the slug pattern with fullmatch so a trailing newline cannot pass.
In Python, $ also matches immediately before a trailing newline. re.match therefore accepts "rate_limited\n" and _runner_failure_code forwards it unchanged. The comment above states that no unvetted text may reach the client as a code, and a client that switches on the exact slug fails to match a value with embedded whitespace.
Use fullmatch, or end the pattern with \Z. The parameterized fallback test in sdks/python/oss/tests/pytest/unit/agents/adapters/test_vercel_stream_conformance.py lines 400-406 does not cover this input, so add a case for it.
🐛 Proposed fix
def _runner_failure_code(code: Any) -> str:
- if isinstance(code, str) and _FAILURE_CODE.match(code):
+ if isinstance(code, str) and _FAILURE_CODE.fullmatch(code):
return code
return DEFAULT_RUNNER_FAILURE_CODE📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| _FAILURE_CODE = re.compile(r"^[a-z][a-z0-9_]{0,63}$") | |
| #: What a runner-reported error is coded as when the runner names no class of its own. | |
| DEFAULT_RUNNER_FAILURE_CODE = "runner_error" | |
| def _runner_failure_code(code: Any) -> str: | |
| if isinstance(code, str) and _FAILURE_CODE.match(code): | |
| return code | |
| return DEFAULT_RUNNER_FAILURE_CODE | |
| _FAILURE_CODE = re.compile(r"^[a-z][a-z0-9_]{0,63}$") | |
| #: What a runner-reported error is coded as when the runner names no class of its own. | |
| DEFAULT_RUNNER_FAILURE_CODE = "runner_error" | |
| def _runner_failure_code(code: Any) -> str: | |
| if isinstance(code, str) and _FAILURE_CODE.fullmatch(code): | |
| return code | |
| return DEFAULT_RUNNER_FAILURE_CODE |
| | { | ||
| type: "error"; | ||
| message: string; | ||
| /** | ||
| * Stable string code, never a display string: the class of failure, so a client can render a | ||
| * purposeful state (add-your-own-key, retry) instead of parsing `message`. Optional and | ||
| * defaulted by the consumer, so an older runner that omits it stays readable. | ||
| * Values: see `RunErrorCode` in engines/sandbox_agent/errors.ts. | ||
| */ | ||
| code?: string; | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Locate wire.py and inspect how an error event's data is built from the wire payload.
fd -t f 'wire.py' --exec ast-grep outline {} --items all
fd -t f 'wire.py' --exec rg -n -C 6 'from_wire|"error"|reasonCode|code' {}
# Check whether any golden fixture pins the error event shape.
rg -rn --iglob '*golden*' -l . 2>/dev/null | head
fd -t d -i golden
rg -n '"type"\s*:\s*"error"' -g '*.json' -g '*.jsonl' .Repository: Agenta-AI/agenta
Length of output: 3933
🏁 Script executed:
#!/bin/bash
set -eu
echo '--- Python Event model and wire parsing ---'
rg -n -C 12 'class Event|def from_wire|Event\.from_wire|type.*error|code' \
sdks/python/agenta/sdk sdks/python/oss/tests/pytest/unit/agents/adapters/test_vercel_stream_conformance.py
echo '--- Vercel adapter use ---'
rg -n -C 8 'data\.get\("code"\)|runner_error|RunErrorCode|error' \
sdks/python/agenta/sdk/agents/adapters/vercel sdks/python/oss/tests/pytest/unit/agents/adapters/test_vercel_stream_conformance.py
echo '--- Error-event fixtures and contract tests ---'
rg -n --glob '*.json' --glob '*.jsonl' --glob '*.ts' --glob '*.tsx' --glob '*.py' \
'"type"[[:space:]]*:[[:space:]]*"error"|type[[:space:]]*[:=][[:space:]]*["'\'']error["'\'']|code[[:space:]]*[:=]' \
services/runner sdks/python 2>/dev/null | head -n 300
echo '--- Repository change summary ---'
git diff --statRepository: Agenta-AI/agenta
Length of output: 50373
🏁 Script executed:
#!/bin/bash
set -eu
echo '--- Event definitions ---'
rg -n -l 'class Event' sdks/python/agenta/sdk
rg -n -l 'def from_wire' sdks/python/agenta/sdk/agents
echo '--- Exact parser implementations ---'
while IFS= read -r file; do
echo "### $file"
rg -n -A 45 -B 8 'class Event|def from_wire' "$file"
done < <(rg -l 'class Event|def from_wire' sdks/python/agenta/sdk/agents)
echo '--- Adapter references ---'
rg -n -C 5 'data\.get\("code"\)|runner_error|RunErrorCode' \
sdks/python/agenta/sdk/agents/adapters/vercel \
sdks/python/oss/tests/pytest/unit/agents/adapters/test_vercel_stream_conformance.py
echo '--- Candidate fixtures and contract tests ---'
fd -t f -i 'golden|fixture|contract|protocol|wire' services/runner sdks/python | head -n 300
echo '--- Error event literals ---'
rg -n -C 3 '"type"[[:space:]]*:[[:space:]]*"error"|type[[:space:]]*=[[:space:]]*["'\'']error["'\'']' \
services/runner sdks/python --glob '*.json' --glob '*.jsonl' --glob '*.ts' --glob '*.tsx' --glob '*.py' \
| head -n 300Repository: Agenta-AI/agenta
Length of output: 346
🏁 Script executed:
#!/bin/bash
set -eu
echo '--- Python Event implementation ---'
rg -n -C 35 'class Event|def from_wire' sdks/python/agenta/sdk/agents/dtos.py
echo '--- Python adapter and conformance test ---'
rg -n -C 8 'data\.get\("code"\)|runner_error|RunErrorCode|error' \
sdks/python/agenta/sdk/agents/adapters/vercel/stream.py \
sdks/python/oss/tests/pytest/unit/agents/adapters/test_vercel_stream_conformance.py
echo '--- Relevant tracked files ---'
git ls-files | rg -i 'golden|fixture|contract|protocol|wire' | head -n 300
echo '--- Error event literals in relevant files ---'
git ls-files services/runner sdks/python | rg '\.(json|jsonl|ts|tsx|py)$' |
xargs -r rg -n -C 3 '"type"[[:space:]]*:[[:space:]]*"error"|type[[:space:]]*=[[:space:]]*["'\'']error["'\'']' |
head -n 300Repository: Agenta-AI/agenta
Length of output: 50373
🏁 Script executed:
#!/bin/bash
set -eu
echo '--- Tracked runner and SDK contract files ---'
git ls-files services/runner sdks/python | rg -i \
'(^|/)(golden|fixtures?|contract|protocol|wire)(/|\.|$)|test.*(protocol|wire|contract)|(?:protocol|wire).*test' \
| head -n 300
echo '--- Runner error-event contract references ---'
rg -n -C 5 'AgentEvent|type: "error"|type:.*error|RunErrorCode|error event|golden|fixture' \
services/runner --glob '*.ts' --glob '*.tsx' --glob '*.json' --glob '*.jsonl' \
| head -n 500
echo '--- Python wire and DTO tests ---'
rg -n -C 5 'Event|from_wire|error|code' \
sdks/python/oss/tests/pytest/unit/agents --glob '*test*.py' \
| head -n 500
echo '--- Current file change summary ---'
git diff --stat
git status --shortRepository: Agenta-AI/agenta
Length of output: 50373
🏁 Script executed:
#!/bin/bash
set -eu
echo '--- Error golden fixture ---'
cat sdks/python/oss/tests/pytest/unit/agents/golden/run_result.error.json
echo '--- Python shared golden contract test ---'
rg -n -A 30 -B 15 'run_result\.error|events|golden' \
sdks/python/oss/tests/pytest/unit/agents/test_wire_contract.py
echo '--- TypeScript result contract assertions ---'
sed -n '319,390p' services/runner/tests/unit/wire-contract.test.ts
echo '--- Protocol error variants and error-code declarations ---'
rg -n -A 20 -B 10 'RunErrorCode|type: "error"|type: "error"|error.*message|AgentEvent' \
services/runner/src/protocol.ts services/runner/src/engines/sandbox_agent/errors.tsRepository: Agenta-AI/agenta
Length of output: 39884
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
import json
from pathlib import Path
fixture = Path("sdks/python/oss/tests/pytest/unit/agents/golden/run_result.error.json")
payload = json.loads(fixture.read_text())
events = payload.get("events", [])
print("error fixture events:", events)
print("contains error event:", any(e.get("type") == "error" for e in events if isinstance(e, dict)))
PY
echo '--- All runner error-event producers ---'
rg -n -C 8 'type: "error"|emitError|persistError|classified.*error|classify.*error|code:' \
services/runner/src --glob '*.ts' | head -n 500
echo '--- Changes touching the protocol and contract surfaces ---'
git diff -- services/runner/src/protocol.ts services/runner/tests/unit/wire-contract.test.ts \
sdks/python/oss/tests/pytest/unit/agents/golden/run_result.error.json \
sdks/python/oss/tests/pytest/unit/agents/test_wire_contract.pyRepository: Agenta-AI/agenta
Length of output: 37777
🏁 Script executed:
#!/bin/bash
set -eu
echo '--- Error classification and event emission ---'
rg -n -C 25 'classifyRunError|ClassifiedRunError|record\(\{[[:space:]]*type: "error"|record\(\{ type: "error"|emit\(.*error|type: "error"' \
services/runner/src/engines/sandbox_agent services/runner/src/server.ts --glob '*.ts' \
| head -n 600
echo '--- Error fixture and result tests, exact current contents ---'
cat sdks/python/oss/tests/pytest/unit/agents/golden/run_result.ok.json
sed -n '1100,1140p' sdks/python/oss/tests/pytest/unit/agents/test_wire_contract.py
sed -n '319,375p' services/runner/tests/unit/wire-contract.test.ts
echo '--- Source-level parser invariant ---'
python3 - <<'PY'
from pathlib import Path
source = Path("sdks/python/agenta/sdk/agents/dtos.py").read_text()
needle = 'return cls(type=str(raw["type"]), data=raw)'
assert needle in source
print("Event.from_wire returns the raw mapping as data:", needle)
PYRepository: Agenta-AI/agenta
Length of output: 40658
Add an error event to the shared /run golden contract.
Event.from_wire preserves raw, so code reaches data.get("code"); no wire.py change is required. Existing fixtures and contract tests do not cover an error event with code. Add this case and assert it in the Python and TypeScript contract tests.
Source: Path instructions
What
A run funded by starter credits routes through a budgeted model proxy. When the organization's allowance is spent, the proxy refuses the request at admission with a 429 whose body names the key and its spend:
That is the verbatim frame from live QA on the dev stack (identifiers masked here, not in the fixture).
Key=is the organization UUID that aliases the key and the parenthesized token is the key's suffix; both identify the key and neither may reach a user.The runner's error condenser had no branch for it, so that raw provider JSON went straight to the chat. The user saw a key alias, a key suffix, and a cost figure, and no way to read the one thing that mattered: you are out of credits, add your own key to keep going.
This PR teaches the condenser to recognize those refusals and answer with a short plain line, plus a stable machine code the client can render a purposeful state from.
Before (what reaches the chat bubble today)
After
…carried alongside
code: "starter_credits_exhausted".Why match on the body, not the status
429 covers three unrelated situations: an admission-time budget refusal, per-key throttling, and the upstream provider's own quota. Telling someone their credits are gone when they were merely throttled is worse than saying nothing, so every branch here keys off the response body. A 429 that matches none of the known bodies is left exactly as it is today.
The mapping
budget_exceedednaming a keystarter_credits_exhaustedbudget_exceedednaming a team / program budgetstarter_credits_program_pausedrate_limit_error, max parallel request, RPM/TPM)rate_limitedRESOURCE_EXHAUSTED/ quota exceededrate_limitedno_db_connection, or a connection failure the body attributes to the proxystarter_credits_unavailablerunner_errorTwo orderings are deliberate and pinned by tests:
RateLimitErrorwhose body says "exceeded your current quota" — that is a billing stop, and "try again in a moment" would send the operator into a loop.An unattributed
ECONNREFUSEDis not read as a credits outage; it could be any host the run touched (an MCP server, a tool endpoint). The body has to name the proxy. There is a TODO to key this off the resolved model connection once a run knows it is on the funded connection.The code, and how it reaches the client
conciseErrorclassified failures internally and then threw the classification away, returning only prose. It is now the message half of a newclassifyRunError, which also returns aRunErrorCode. Existing call sites are untouched.The code rides the error event's new optional
codefield (protocol.ts), mirroringattachment_delivery.reasonCode's "stable string code, never a display string" contract. The SDK adapter, which hardcodedfailure_code="runner_error"on runner-reported errors, now reads that field — defaulting torunner_errorwhen it is absent, so an older runner is unchanged. The event shape is otherwise untouched, and the code lands on thedata-agent-errorpart that already carries one.The adapter only accepts a conservative slug (
^[a-z][a-z0-9_]{0,63}$). Anything else on the wire falls back torunner_error, so no vendor- or attacker-controlled text can reach a client that switches on the code.The harness retries this 429 first, and its chatter used to win
Two things had to be true for the mapped message to actually reach the user.
The user waits about six seconds. Pi treats a 429 as retryable and retries three times with exponential backoff (2s, then 4s) before the failure surfaces. The retryable set is a hardcoded regex in
@earendil-works/pi-ai'sretry.js:RETRYABLE_PROVIDER_ERROR_PATTERNmatches the literal"429", and the companionNON_RETRYABLE_PROVIDER_LIMIT_ERROR_PATTERN— which already excludesinsufficient_quota,quota exceeded,billing, andout of budget— has no entry for LiteLLM'sbudget_exceeded/ "Budget has been exceeded". So a refusal that can never succeed on retry is retried anyway.Not fixed here, and worth a follow-up: a budget refusal is deterministic, so short-circuiting it would return the message immediately instead of after ~6s of dead waiting. Two ways in, neither implemented in this PR — add the budget literal to that non-retryable pattern upstream, or have the runner write Pi's
retry.*settings (retry.enabled,retry.maxRetries,retry.baseDelayMs, all supported by Pi's settings schema). Note the runner has no such knob today: it authors onlymodels.jsonand copies the operator'ssettings.jsonverbatim, so there is nothing to configure yet.The retry chatter counted as an answer. pi-acp renders Pi's
auto_retry_start/auto_retry_endevents as ordinary assistant message chunks — the same channel as real model output. The swallowed-error recovery inrun-turn.ts, which reads Pi's transcript when a turn produces nothing, is guarded on!run.output().trim(). Retry notices made that output non-empty, so the recovery was skipped and the run returnedok: truewith"Retrying (attempt 1/3, waiting 2s)..."as its entire visible answer — exactly what QA saw. The mapping in this PR would never have fired on that path.So the guard now also treats a turn holding nothing but retry notices as empty (
isOnlyHarnessRetryNoticesinpi-error.ts, matched against pi-acp's literal shapes rather than a loose/retry/iso a genuine answer that mentions retrying is never discarded). This is the one behavior change here beyond the mapping itself: a Pi turn whose only output was retry chatter and whose transcript records a real error now fails loud instead of shipping the chatter. Classification also reads the whole error rather than its first line, so a leading retry notice cannot displace the mapped message.Copy is a placeholder
The five strings are marked
TODO(copy: owner)behind named constants in one block. They are the first product copy the runner puts in front of an end user — every other line in that file is an operator hint — so the final wording belongs to the product owner. They are deliberately not prefixed with the harness name the way the operator hints are; "claude:" is noise to the person chatting.No user-visible line ever contains the raw body, a key fragment, a spend figure, or a URL. All five are constants, and a test asserts it against every fixture.
Follow-up
The web app does not read
data-agent-error.data.codeyet — it renders only the plain error frame. Consuming these codes to render a purposeful state (an "add your own key" call to action onstarter_credits_exhausted, a retry affordance onrate_limited) is the next piece of work and is not in this PR.Tests
services/runner/tests/unit/sandbox-agent-errors.test.ts— one case per mapping with realistic LiteLLM bodies, the verbatim live refusal from dev-stack QA (asserted to map tostarter_credits_exhausted, with a leak test naming its organization UUID, key suffix, and both spend figures), the retry-notice-first variant, the two ordering cases above, an unrelated 429 that must not map to exhaustion, and an unattributed connection failure that must not map to an outage.services/runner/tests/unit/sandbox-agent-pi-error.test.ts—isOnlyHarnessRetryNoticesrecognizes pi-acp's notices alone and in sequence, and never discards a turn that produced real text alongside them.sdks/python/oss/tests/pytest/unit/agents/adapters/test_vercel_stream_conformance.py— the code reachesdata-agent-erroron both twins, and a non-slug code falls back torunner_error.Runner:
pnpm test2254 passed / 7 expected-fail across 131 files;pnpm run typecheckclean.SDK:
pytest oss/tests/pytest/unit/agents/adapters/130 passed.