Skip to content

validate-agent.sh: don't abort at the first warning (set -e + ((x++))) and stop false-flagging valid agents - #89404

Open
bcherny wants to merge 1 commit into
mainfrom
boris/triage-fix-83803
Open

bcherny wants to merge 1 commit into
mainfrom
boris/triage-fix-83803

Conversation

@bcherny

@bcherny bcherny commented Aug 25, 2026

Copy link
Copy Markdown
Collaborator

Fixes public issue #83803

The plugin-dev skill's validate-agent.sh failed on plugin-dev's own agent files. Three root causes, all set -euo pipefail interactions:

  1. Abort at the first warning. ((warning_count++)) / ((error_count++)) evaluate the arithmetic expression, and ((expr)) returns a nonzero exit status when the expression's value is 0 — so the first increment from 0 killed the script under set -e, mid-run, with exit 1. All increments now use count=$((count + 1)), which is an assignment and always returns 0.

  2. Abort on any absent frontmatter field. Extractions like TOOLS=$(echo "$FRONTMATTER" | grep '^tools:' | ...) propagate grep's exit 1 (no match) into the assignment, aborting the script instead of reporting the missing field. Each extraction now ends in || true.

  3. False "missing <example> blocks" warning. DESCRIPTION was extracted with grep '^description:', which only captures the first physical line — but plugin-dev's own agents use multi-line descriptions whose <example> blocks sit on later lines, so valid agents were flagged. The extraction now captures the full multi-line value (from description: up to the next top-level agent key).

Verification (all three plugin-dev agents previously died at the first warning with exit 1):

  • agents/agent-creator.md, agents/plugin-validator.md, agents/skill-reviewer.md → all checks pass, exit 0
  • A valid agent file that only triggers warnings → runs to the summary, exit 0
  • An intentionally invalid file (bad name, missing color) → all 3 errors reported, exit 1

New validate-agent.test.sh next to the script covers all three cases as a regression test.

Fixed validate-agent.sh aborting at the first warning and rejecting valid agent files

🤖 Generated with Claude Code

…) and stop false-flagging valid agents

Two defects made the validator fail on plugin-dev's own agent files
(#83803):

1. Under `set -e`, `((warning_count++))` / `((error_count++))` return a
   nonzero status when the counter was 0, so the script died at the first
   warning or error instead of finishing the run. Increments now use
   `count=$((count + 1))`, which always returns 0.

2. Field extractions like `TOOLS=$(... | grep '^tools:' ...)` aborted the
   script under `set -e` when the field was absent (grep exits 1 on no
   match), instead of reporting the missing field. They now end in
   `|| true`.

3. The description check only read the first physical line of the
   `description:` value, so multi-line descriptions with <example> blocks
   (as in plugin-dev's own agents) were false-flagged as missing examples.
   The extraction now captures the full multi-line value.

Adds validate-agent.test.sh: plugin-dev's own agents must exit 0, a
warning-only file must complete with exit 0, and an invalid file must
still exit 1 with all errors reported.

No-Verification-Needed: standalone shell script in the public repo; driven end-to-end directly plus new regression harness
@fsc-eriker

Copy link
Copy Markdown

Wouldn't a simpler fix be to switch to preincrement ((++warning_count))

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

description continuation only stops at name|model|color|tools, but valid agent frontmatter also includes keys such as permissionMode, disallowedTools, mcpServers, and maxTurns. Since YAML key order is free, a folded description followed by one of those keys absorbs it into DESCRIPTION and can false-flag later validation. Stop on any new top-level frontmatter key (or parse YAML) and add a regression with description: > followed by permissionMode:.

@sylvesterkaczmarek

Copy link
Copy Markdown

Thanks, that evidence changes the right fix. I agree that simply extending the stop-key list is not sufficient. The absorption bug I reported is real, but if the plugin’s own fixtures are not valid frontmatter to Claude Code, then a validator test that treats them as valid is also testing the wrong language. I would align this validator with the product’s actual frontmatter/YAML parsing rules, or at minimum use fixtures that the product itself accepts, and then validate the extracted description. That resolves the key-order ambiguity without inventing a second parser grammar in awk.

konsta95 added a commit to konsta95/claude-code that referenced this pull request Sep 9, 2026
…idate

validate-agent.sh no longer decides parse-ability with an awk classifier of
its own. The file is copied into a throwaway plugin and run through
`claude plugin validate --json` (plain-report fallback for Claude Code older
than 2.1.259; exit 2 "not verified" when no claude can run), so the verdict
is the loader's own parser, Bun YAML plus Claude Code's quoting/tab retry,
and the product's message is what the user reads. The script keeps what the
product does not check: an empty, null, numeric or boolean description
(dropped by the project loader; placeholder or digits for a plugin agent),
and the style checks, which now run on the text the plugin loader hands the
model: block scalars with their chomping, plain and quoted scalars folded,
double-quoted escapes decoded, then trimmed. CRLF and BOM files are read the
way the loader reads them instead of being rejected at the first line.
`validate-agent.sh --description <file>` prints that text.

validate-agent.test.sh asserts against the product: 44 description shapes go
through `claude plugin validate` once and the script's verdict must equal the
product's plus the stated policy (44/44); the extracted text must equal the
runtime's on 27 shapes observed in the agent listing of Claude Code 2.1.266
or replayed through Bun 1.4.1 (27/27). Pointed at PR anthropics#89404's script the
corpus check fails on 19 shapes, at the previous version of this script on
20. The suite passes under gawk, mawk and busybox awk.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@konsta95 konsta95 left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.

konsta95 added a commit to konsta95/claude-code that referenced this pull request Sep 20, 2026
…t 92-shape corpus

The own-grammar note still gave the control figures from a 44-shape corpus (19 and 20 of 44, Claude Code 2.1.266). Re-measured on Claude Code 2.1.278 with the corpus at 92 shapes: VALIDATOR pointed at PR anthropics#89404's script fails the corpus check on 35 of 92 (57/92 agree); this script's own-grammar version (e830ff6) on 37 of 92 (55/92 agree). The old figures stay as history. The suite itself is unchanged: 39 pass on 2.1.274 and 2.1.278, 92/92 shapes agree.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
hippoley added a commit to hippoley/CounterProof that referenced this pull request Oct 7, 2026
Publish a reviewer-supplied external-oracle contradiction artifact for anthropics/claude-code#89404, bind it into the claim matrix, Reality Lab, README flagship narrative, and regression tests without claiming an independent Claude runtime replay.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants