Repository navigation
Conversation
…) and stop false-flagging valid agents Two defects made the validator fail on plugin-dev's own agent files (#83803): 1. Under `set -e`, `((warning_count++))` / `((error_count++))` return a nonzero status when the counter was 0, so the script died at the first warning or error instead of finishing the run. Increments now use `count=$((count + 1))`, which always returns 0. 2. Field extractions like `TOOLS=$(... | grep '^tools:' ...)` aborted the script under `set -e` when the field was absent (grep exits 1 on no match), instead of reporting the missing field. They now end in `|| true`. 3. The description check only read the first physical line of the `description:` value, so multi-line descriptions with <example> blocks (as in plugin-dev's own agents) were false-flagged as missing examples. The extraction now captures the full multi-line value. Adds validate-agent.test.sh: plugin-dev's own agents must exit 0, a warning-only file must complete with exit 0, and an invalid file must still exit 1 with all errors reported. No-Verification-Needed: standalone shell script in the public repo; driven end-to-end directly plus new regression harness
|
Wouldn't a simpler fix be to switch to preincrement |
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
description continuation only stops at name|model|color|tools, but valid agent frontmatter also includes keys such as permissionMode, disallowedTools, mcpServers, and maxTurns. Since YAML key order is free, a folded description followed by one of those keys absorbs it into DESCRIPTION and can false-flag later validation. Stop on any new top-level frontmatter key (or parse YAML) and add a regression with description: > followed by permissionMode:.
|
Thanks, that evidence changes the right fix. I agree that simply extending the stop-key list is not sufficient. The absorption bug I reported is real, but if the plugin’s own fixtures are not valid frontmatter to Claude Code, then a validator test that treats them as valid is also testing the wrong language. I would align this validator with the product’s actual frontmatter/YAML parsing rules, or at minimum use fixtures that the product itself accepts, and then validate the extracted |
…idate validate-agent.sh no longer decides parse-ability with an awk classifier of its own. The file is copied into a throwaway plugin and run through `claude plugin validate --json` (plain-report fallback for Claude Code older than 2.1.259; exit 2 "not verified" when no claude can run), so the verdict is the loader's own parser, Bun YAML plus Claude Code's quoting/tab retry, and the product's message is what the user reads. The script keeps what the product does not check: an empty, null, numeric or boolean description (dropped by the project loader; placeholder or digits for a plugin agent), and the style checks, which now run on the text the plugin loader hands the model: block scalars with their chomping, plain and quoted scalars folded, double-quoted escapes decoded, then trimmed. CRLF and BOM files are read the way the loader reads them instead of being rejected at the first line. `validate-agent.sh --description <file>` prints that text. validate-agent.test.sh asserts against the product: 44 description shapes go through `claude plugin validate` once and the script's verdict must equal the product's plus the stated policy (44/44); the extracted text must equal the runtime's on 27 shapes observed in the agent listing of Claude Code 2.1.266 or replayed through Bun 1.4.1 (27/27). Pointed at PR anthropics#89404's script the corpus check fails on 19 shapes, at the previous version of this script on 20. The suite passes under gawk, mawk and busybox awk. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…t 92-shape corpus The own-grammar note still gave the control figures from a 44-shape corpus (19 and 20 of 44, Claude Code 2.1.266). Re-measured on Claude Code 2.1.278 with the corpus at 92 shapes: VALIDATOR pointed at PR anthropics#89404's script fails the corpus check on 35 of 92 (57/92 agree); this script's own-grammar version (e830ff6) on 37 of 92 (55/92 agree). The old figures stay as history. The suite itself is unchanged: 39 pass on 2.1.274 and 2.1.278, 92/92 shapes agree. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Publish a reviewer-supplied external-oracle contradiction artifact for anthropics/claude-code#89404, bind it into the claim matrix, Reality Lab, README flagship narrative, and regression tests without claiming an independent Claude runtime replay.
Fixes public issue #83803
The plugin-dev skill's
validate-agent.shfailed on plugin-dev's own agent files. Three root causes, allset -euo pipefailinteractions:Abort at the first warning.
((warning_count++))/((error_count++))evaluate the arithmetic expression, and((expr))returns a nonzero exit status when the expression's value is 0 — so the first increment from 0 killed the script underset -e, mid-run, with exit 1. All increments now usecount=$((count + 1)), which is an assignment and always returns 0.Abort on any absent frontmatter field. Extractions like
TOOLS=$(echo "$FRONTMATTER" | grep '^tools:' | ...)propagate grep's exit 1 (no match) into the assignment, aborting the script instead of reporting the missing field. Each extraction now ends in|| true.False "missing <example> blocks" warning.
DESCRIPTIONwas extracted withgrep '^description:', which only captures the first physical line — but plugin-dev's own agents use multi-line descriptions whose<example>blocks sit on later lines, so valid agents were flagged. The extraction now captures the full multi-line value (fromdescription:up to the next top-level agent key).Verification (all three plugin-dev agents previously died at the first warning with exit 1):
agents/agent-creator.md,agents/plugin-validator.md,agents/skill-reviewer.md→ all checks pass, exit 0New
Fixed validate-agent.sh aborting at the first warning and rejecting valid agent filesvalidate-agent.test.shnext to the script covers all three cases as a regression test.🤖 Generated with Claude Code