Implement broad skill catalog and verified small model training with governed evaluation #3603

Objective

Build a versioned catalog of discoverable Agent Skills and evaluate a compact Ruflo router, followed conditionally by a small execution model trained on verified trajectories.

Architecture: ADR-402 (proposed).

“All available” means discoverable sources in a declared snapshot, not unrestricted execution or universal training eligibility. Catalog coverage, permission to execute and eligibility to train must remain separate.

Deliverables and milestones

M0: Contracts and baseline

• Inspect current routing, hooks, AgentDB/RuVector and MetaHarness integration against a pinned commit.
• Verify active ADR-322A promotion enforcement and any sequential evidence implementation.
• Define versioned skill, task, trajectory, dataset, candidate and evaluation schemas.
• Reproduce current baseline and ADR-391 benchmark or document discrepancies.
• Choose initial domain, hardware, source adapters, retention policy and training license policy.
• Measure a small batch and record a bounded compute/data budget before paid work.

M1: Broad catalog

• Add bounded, resumable discovery with pinned source commits and CLI versions.
• Parse instructions and resources without running imported scripts or lifecycle hooks.
• Record failures and coverage instead of claiming a complete internet census.
• Preserve source provenance while grouping duplicate families.
• Build tenant filtered lexical and RuVector retrieval.
• Test archive traversal, symlink escapes, resource limits, revoked sources and tenant isolation.

M2: Learned routing

• Create independently labeled tasks including abstention and conflicting prerequisites.
• Split by repository and skill family before generating synthetic examples.
• Compare current routing, lexical retrieval, vector retrieval and a trained classifier/reranker.
• Calibrate confidence on development data.
• Preserve ADR-391 default routing gates; publish losing and inconclusive results.

M3: Verified trajectories

• Select 20 to 50 reviewed skills and attempt 500 pilot training trajectories.
• Use disposable environments, hidden final state verifiers and capped tool/network/spend budgets.
• Capture actions, tool results, policy decisions, failures, recovery and actual costs.
• Redact before export; do not collect hidden reasoning.
• Audit label quality and report verified trajectory yield.
• Keep evaluation data separate and quarantine unreliable verifier families.

M4: Optional execution adapter

• Select a licensed 1B to 3B base model only after pilot evidence.
• Train a reproducible adapter; preserve an untrained model control.
• Evaluate retrieved current instructions, unseen families, stale versions and escalation.
• Evaluate the actual quantized deployment artifact separately.
• Expand to 5,000 to 20,000 verified traces only when yield, learning curves and budget support it.

M5: Governed rollout

• Run shadow evaluation without side effects.
• Register source bound receipts through existing promotion mechanisms.
• Demonstrate complete bundle rollback before a separately authorized canary.
• Keep dependencies optional and the feature disabled by default.

Evaluation and completion criteria

Run identical fixtures, tools, permissions and budgets across current Ruflo, capable model plus retrieval, untrained small model, trained small model, trained model plus fallback, and learned router with existing executor.

Start with 500 evaluation tasks across at least 100 unseen families where available. Determine final sample size using development discordance and family clustering; 500 is not assumed sufficient.

For proposed opt in activation, require all:

1. One sided 95% lower bound on success difference exceeds minus 2 percentage points versus the strong retrieval baseline.
2. One sided 95% upper bound on fully loaded cost per verified success ratio is at most 0.70.
3. End to end p95 latency is at most 1.10 times baseline.
4. Zero observed executed permission violations in the frozen adversarial suite and all deterministic boundary checks pass.
5. Reproducible receipts and demonstrated rollback.
6. All applicable existing repository gates, including ADR-391 for default router changes.

A crossed confidence bound means inconclusive. No repeated peeking or gate relaxation. Report per family results, retries, failed runs, teacher fallback, hosting, verification and amortized training costs.

Non goals and scope

This issue tracks implementation, not merely ADR acceptance. Merging its documentation PR must not close this issue. No trained checkpoint, measured improvement, production activation or paid training is claimed or authorized by the documentation change.

Risks

Primary risk is instruction imitation without execution competence. Independent final state verification and unseen family tests are required. Additional risks: corpus license uncertainty, poisoning, tenant leakage, stale skills, correlated evaluation cases and fallback erasing savings.

Related architecture: ADR-353, ADR-391, ADR-322A and ADR-381. Preserve their scope and verify implementation rather than assuming a proposal is deployed.