Skip to content

fix(metaharness): use the @metaharness/* and ruflo CLI already installed instead of npm-installing at tool-call time (#3366) - #3367

Merged
ruvnet merged 1 commit into
ruvnet:mainfrom
MAMDV:fix/metaharness-plugin-pins-and-npx
Sep 26, 2026
Merged

ruvnet merged 1 commit into
ruvnet:mainfrom
MAMDV:fix/metaharness-plugin-pins-and-npx

Conversation

@HF-teamdev

Copy link
Copy Markdown
Contributor

Hi @ruvnet and @proffesor-for-testing,

This finishes the job _harness.mjs's own SECURITY + PERF block started: pin the version, resolve what is installed, and stop touching the registry per call. Three paths in the same plugin never got that treatment, and the two pins that decide what "installed" means had drifted two minors behind the CLI. @proffesor-for-testing — both regression tests are hermetic (stub packages, a node_modules layout built in a temp dir, an npm/npx PATH trap), and the Tests section lists what survives deleting each new guard, including the four assertions that also pass on main.

What this fixes

Fixes #3366. On a stock ruflo 3.42.4 install, 14 of the 15 metaharness_* MCP tools that run a subprocess reach the npm registry at tool-call time, and offline they degrade or return silently empty results — although ruflo ships metaharness 0.4.2, @metaharness/darwin 0.10.2 and @metaharness/redblue 0.1.4, and the @claude-flow/cli the memory tools need is the one that spawned them.

Why the two halves ship together. They are the same six files, and each alone leaves the bug half-fixed:

  • The pin bump alone makes metaharness resolve locally, but darwin still runs npx -y -p @metaharness/darwin@~0.10.2 on every call, redblue still npm-installs, and the memory tools still shell to npx @claude-flow/cli@latest.
  • The no-npx change alone teaches _darwin.mjs / _redblue.mjs to prefer an installed copy — but with the old ~0.8.0 pin the shipped darwin 0.10.2 does not satisfy it (satisfiesTildeRange('0.10.2', '~0.8.0') is false), so darwin's "installed copy" branch is dead on a stock install and every call attempts a darwin-cache-0.8.0 install instead. The pin fix is what makes the new resolution path resolvable.

Measured end to end through the real compiled MCP handlers, offline: 15 npm/npx spawns → 0, with all 15 tools returning real results.

The fix

1. Pins (_harness.mjs, _darwin.mjs, check-metaharness-pins.mjs)

  • METAHARNESS_PIN_VERSION '~0.3.0' → '~0.4.1'. For 0.x, ~0.4.1 and the CLI's ^0.4.1 are the same set (>=0.4.1 <0.5.0), so every copy a clean install can produce is accepted: 0.4.2 (the ruflo tree), 0.4.1 (v3/pnpm-lock.yaml), 0.4.16 (npm latest).
  • DARWIN_PIN_VERSION '~0.8.0' → '~0.10.2', identical to the CLI's optionalDependency. gepa.mjs derives its cache pin from this, so gepa's fallback moves with it; on a stock install gepa's bare import (0.10.2) and the bin resolution now agree.
  • REDBLUE_PIN_VERSION is unchanged: ~0.1.4 still admits npm latest (0.1.6), and the CLI does not declare redblue.
  • check-metaharness-pins.mjs now also reads each helper's *_PIN_VERSION. Three rules: the pin must be a ~X.Y.Z range — _invoke.satisfiesTildeRange(), the predicate the helpers actually apply, parses nothing else, so a ^/exact/latest pin would reject every installed copy while covers() called the row green; when the CLI declares the package the helper range must cover the declared range (outer.lo <= inner.lo && inner.hi <= outer.hi, no network, so it runs under --offline); otherwise it must admit npm latest, like the existing rows. An unreadable constant stays fatal. satisfies() now shares a bounds() helper with covers(), and the existing rows print byte-identically. metaharness-pin-drift.yml also triggers on the three helper files.

2. No npm/npx at tool-call time (_darwin.mjs, _redblue.mjs, the four memory scripts, _invoke.mjs)

  • _invoke.mjs gains three shared helpers: resolvePackageBin(pkgDir, binName) (the package's own bin map entry, realpath'd), packageBinRelPath(), and runRufloCli(args) = spawnSync(process.execPath, [<cli>/bin/cli.js, ...args], { shell: false }). <cli> is the @claude-flow/cli that owns the plugin: <scripts>/../../.. published, or <scripts>/../../../v3/@claude-flow/cli in a checkout, both derived from import.meta.url — never from cwd or PATH. A candidate counts only if dist/src/index.js exists next to bin/cli.js (mcp-launch.cjs L42-L47's unbuilt-checkout guard) and its package.json is named @claude-flow/cli — a check this PR adds, so an unrelated directory two levels up can never qualify. In the published layout that is the same CLI whose MCP server spawned the tool, because locatePluginScripts() L78-L95 finds the plugin by walking up from that server's own dist/.
  • _darwin.mjs resolves metaharness-darwin the way _harness.mjs resolves its bins: installed copy satisfying the pin → the one-time darwin-cache-<pin> that gepa already shares → process.execPath <abs bin> with shell:false. If neither resolves, the result is the same {degraded:true, reason:'metaharness-darwin-not-available'} the npx failure produced.
  • _redblue.mjs checks the installed copy first and realpaths the CLI path for both the local and the cache copy (the isMain guard above). Its spawn moves from 'node' + shell: win32 to process.execPath + shell:false: that argv carries MCP-supplied --config / --out / --in, and under shell:true Node hands cmd.exe one unquoted string, so x.yaml & whoami would run as a command (and an installed path containing a space would break).
  • audit-list / audit-trend / oia-audit / similarity call runRufloCli(['memory', …]). Argument lists and result handling are unchanged. drift-from-history drops its unused CLI_PKG. Kept: CLI_CORE=1 still means npx @claude-flow/cli-core@alpha (the ADR-100 opt-in), and a layout with no usable local CLI still falls back to npx @claude-flow/cli@latest.
  • Two SKILL.md lines that documented darwin's npx -y @metaharness/darwin@~0.8.0 shell-out now describe the resolution; five more skills/agents/commands docs get the new current-pin numbers.

3. Where the installed-copy lookup may search (please sanity-check this call)

findLocalPackageDir() at base walks up node_modules from the plugin's own directory and from process.cwd() (L106-L132), and its only caller is _harness.mjs. The MCP handler spawns the scripts with no cwd, so they inherit the server's cwd — the user's project. Wiring _darwin.mjs and _redblue.mjs into the same helper would have extended that cwd walk to two paths that execute a package bin with MCP-supplied argv. That is reachable whenever the plugin-directory walk finds nothing in range — which, with the old darwin pin, is every stock install. It is covered by the new test: revert the option and Phase 5 fails with darwin in the tool's cwd node_modules is NOT executed (got stub cwd-darwin) and redblue in the tool's cwd node_modules is NOT executed, i.e. the planted stub runs.

So findLocalPackageDir keeps its name, its signature and its default, and takes an option: the two new callers pass { fromCwd: false } and search only the ruflo install that owns the plugin, falling back to the pinned cache install. This costs nothing measurable — the 15 → 0 spawn result, the plugin suites and smoke are all measured with it in place — because on a stock install the shipped copies are ancestors of the plugin directory anyway.

_harness.mjs is deliberately not changed: metaharness_score / genome / mcp_scan / threat_model have resolved a project-local metaharness since the pinned-cache refactor, RUFLO_METAHARNESS_SKIP_LOCAL=1 still disables it, and narrowing it is a behaviour change beyond this fix. If you want that one narrowed too, say so and I will push it as a follow-up.

Alternatives considered

  • Read the declared range from @claude-flow/cli/package.json at runtime instead of pinning in the helpers. Single source of truth, but the helpers would need caret support (satisfiesTildeRange is tilde-only), a different relative path per layout, and a policy for peer vs optional ranges — and it drops the explicit "only bumped deliberately" posture _harness.mjs documents for supply-chain reasons. Explicit pins plus a guard that fails on divergence keep that posture.
  • Widen the helper pins (>=0.3.0 <0.5.0). Accepts releases CI never exercises, and the matcher only supports ~.
  • Have the MCP server pass its own bin in an env var (RUFLO_CLI_BIN). Exact for MCP calls, but it needs TypeScript changes in both spawners (metaharness-tools.ts, commands/metaharness.ts) and does nothing for direct node scripts/x.mjs runs. The location-derived CLI is that same package in the published layout and covers both entry points with a plugin-only change. An env override can still be added later.
  • Pin npx @claude-flow/cli@<version> or add --prefer-offline. Still npx: the first call per version downloads the whole CLI and it still depends on npx cache state ([verification] MEDIUM: doctor health check output not captured — concurrent npx runs corrupt shared npm cache #3190, ruflo@3.41.2 reports latest while executing cached @claude-flow/cli@3.33.0: wrapper dependency permits stale runtime #3306).
  • Import the memory API in-process. Pulls the CLI's heavy module graph (agentdb, ONNX) into every plugin script and breaks the subprocess isolation the plugin is built around.
  • Leave darwin on npx with --prefer-offline. Stays inconsistent with _harness.mjs/gepa and can still run a darwin other than the one gepa imports.

Tests

Two new hermetic tests, both wired into smoke.sh (so they run in metaharness-ci.yml → "Plugin structural smoke"). Neither needs the network or a real MetaHarness package; both skip on win32, because the npm/npx trap is a POSIX sh script. Every run below used a scratch HOME, a scratch RUFLO_METAHARNESS_CACHE_BASE and that trap — never ~/.ruflo.

test-pin-alignment.mjs (smoke step 17z74a) — each helper pin accepts the floor of the CLI-declared range and a later patch, using the runtime predicate satisfiesTildeRange; then, in a published layout (<p>/node_modules/@claude-flow/cli/plugins/…/scripts next to a stub <p>/node_modules/metaharness at the declared floor), metaharnessResolution() must report source: 'local', score.mjs must return the installed copy's output, and no npm/npx may be spawned.

On main (the test copied onto e558f0c0f, everything else base) — 3 passed, 9 failed:

Failures:
  - _harness.mjs pin ~0.3.0 accepts 0.4.1 (floor of declared ^0.4.1)
  - _harness.mjs pin ~0.3.0 accepts 0.4.101 (a later patch inside declared ^0.4.1)
  - _darwin.mjs pin ~0.8.0 accepts 0.10.2 (floor of declared ~0.10.2)
  - _darwin.mjs pin ~0.8.0 accepts 0.10.102 (a later patch inside declared ~0.10.2)
  - metaharnessResolution() → source 'local' (got {"ok":false,"reason":"metaharness-not-available"})
  - resolved metaharness bin is the installed copy next to @claude-flow/cli
  - score.mjs is not degraded (reason: metaharness-not-available)
  - score.mjs ran the installed metaharness 0.4.1
  - no npm / npx process spawned (got: npm install --no-audit --no-fund --no-package-lock --prefix …/empty-cache-base/metaharness-cache-0.3.0 metaharness@~0.3.0)

On this branch: 12 passed, 0 failed.

test-no-tool-time-npx.mjs (smoke step 17r2) — builds a published layout with a stub @claude-flow/cli (a file-backed memory store|list|retrieve that logs its calls), stub metaharness and @metaharness/darwin at the pin floors, and a stub @metaharness/redblue symlinked from a pnpm-like store that keeps upstream's isMain guard. Phase 1 darwin (both success branches: the installed copy, and the versioned cache with no installed copy); Phase 2 redblue through the symlink; Phase 3 the oia-audit → audit-list → audit-trend/similarity round trip through the shipping CLI; Phase 4 the kept contracts; Phase 5 the scoping and repair guards.

On main (same treatment) — 7 passed, 19 failed. The 19:

  - bench verify exits 0 with success (got 0, reason metaharness-darwin-not-available)
  - bench verify ran the installed metaharness-darwin (sync path)
  - bench verify: no npm / npx spawned (got: npx -y -p @metaharness/darwin@~0.8.0 metaharness-darwin bench verify …)
  - security bench is not degraded (reason metaharness-darwin-not-available)
  - security bench (async path): no npm / npx spawned (got: npx -y -p @metaharness/darwin@~0.8.0 metaharness-darwin security bench --population 1 --cycles 1)
  - no installed copy → the existing darwin-cache-0.8.0 runs (got stub none)
  - bench verify via darwin-cache: no npm / npx spawned (got: npx -y -p @metaharness/darwin@~0.8.0 …)
  - redblue attack exits 0 (got 0, reason metaharness-redblue-not-available)
  - redblue CLI actually dispatched (isMain held: resolved bin is realpath'd)
  - redblue attack: no npm / npx spawned (got: npm install … --prefix …/empty-cache-b…)
  - oia-audit persisted the record (got {"ok":false,…,"error":"npm ERR! npx blocked by test-no-tool-time-npx trap\n"})
  - audit-list returns the persisted record (got 0)
  - audit-trend reads both records back (exit 2)
  - similarity reads both records back (exit 2, reason a: key not found in namespace metaharness-audit: …)
  - the shipping CLI served store + list + retrieve (saw: nothing)
  - memory round trip: no npm / npx spawned (got: npx @claude-flow/cli@latest memory store --namespace metaharness-audit …)
  - a cache with package.json but no bin triggers one reinstall (trap: npx -y -p @metaharness/darwin@~0.8.0 …)
  - _darwin.mjs spawns the resolved bin with shell:false
  - _redblue.mjs spawns the resolved bin with shell:false

On this branch: 26 passed, 0 failed.

What survives deleting each guard (each mutation applied to a scratch copy of scripts/, then test-no-tool-time-npx.mjs re-run):

mutation result (out of 26)
findLocalPackageDir's fromCwd option ignored (cwd walk always on) 22/4 — the two cwd assertions and both half-cache assertions fail (the cwd stub wins before the cache is consulted)
darwin's half-written-cache reinstall removed 25/1 — "a cache with package.json but no bin triggers one reinstall"
_redblue.mjs spawn back to shell: process.platform === 'win32' 25/1 — "_redblue.mjs spawns the resolved bin with shell:false"
both realpathSync calls removed 25/1 — "redblue CLI actually dispatched (isMain held …)"
darwin's cache success branch deleted 25/1 — "no installed copy → the existing darwin-cache-0.10.2 runs"
pj.name === '@claude-flow/cli' check dropped 25/1 — "a bin/cli.js whose package.json is not @claude-flow/cli is rejected"

Honest gaps in that table. Four assertions pass on main as well as here, so they are guards against this PR's new surface rather than regression tests for the old bug: the two "cwd node_modules is NOT executed" assertions (base has no installed-copy lookup for darwin/redblue at all, so nothing local ever runs), "a bin/cli.js whose package.json is not @claude-flow/cli is rejected" (base always goes to npx), and "…still degrades with exit 0" (base degrades too). The mutation table above is what shows they are load-bearing.

Pin-drift guard, run four ways:

# branch guard against main's helper files, --offline → exit 1
| METAHARNESS_PIN_VERSION (plugins/ruflo-metaharness/scripts/_harness.mjs) | ~0.3.0 (vs ^0.4.1) | plugin | — | NOT-COVERED |
| DARWIN_PIN_VERSION (plugins/ruflo-metaharness/scripts/_darwin.mjs)       | ~0.8.0 (vs ~0.10.2) | plugin | — | NOT-COVERED |

# branch guard on this branch, online (`node scripts/check-metaharness-pins.mjs` from the repo root — what CI runs) → exit 0
| METAHARNESS_PIN_VERSION (plugins/ruflo-metaharness/scripts/_harness.mjs) | ~0.4.1 (vs ^0.4.1) | plugin | — | covers |
| DARWIN_PIN_VERSION (plugins/ruflo-metaharness/scripts/_darwin.mjs)       | ~0.10.2 (vs ~0.10.2) | plugin | — | covers |
| REDBLUE_PIN_VERSION (plugins/ruflo-metaharness/scripts/_redblue.mjs)     | ~0.1.4 | plugin | 0.1.6 | current |
✓ All metaharness pins are declared installably and admit the current npm `latest`; plugin helper pins cover them.

# main's guard on main → exit 0. That is the blind spot this PR closes.

# mutated pins on this branch (_harness ^0.4.1, _redblue 'latest'), --offline → exit 1
| METAHARNESS_PIN_VERSION (…/_harness.mjs) | ^0.4.1 | plugin | — | NOT-TILDE |
| REDBLUE_PIN_VERSION (…/_redblue.mjs)     | latest | plugin | — | NOT-TILDE |
(main's guard reports exit 0 for the same mutation.)

End to end through the real compiled MCP handlers. A copy of an installed ruflo 3.42.4 tree with only plugins/ruflo-metaharness swapped, scratch HOME/cwd/cache base, npm/npx trapped (offline). All 15 subprocess-backed tools, in one process:

MCP tool upstream e558f0c0f this PR
score / genome / mcp_scan / threat_model degraded, 1 npm install metaharness@~0.3.0 each (4) 0 — harnessFit 56; needs-work/0.395; worst info; worst info
learn degraded, 1 0 — checkout-required
oia_audit degraded, 1 0 — worst clean, persisted: true
audit_list / audit_trend / similarity 1 npx @claude-flow/cli@latest memory … each (3) 0 — records 1/1; near-identical; overall 0.15
gepa ok, 0 ok, 0 — valid
bench / security_bench / evolve degraded, 1 npx … @metaharness/darwin@~0.8.0 each (3) 0 — suite written; gates 8/12; evolve improved
redblue degraded, 1 npm install @metaharness/redblue@~0.1.4 0 — upstream ok, 906 B of stdout
drift_from_history degraded, 2 (npx … memory list + npm install metaharness@~0.3.0) 0 — similarity 1
total npm/npx spawns 15 0
End-to-end harness used for that table (mcp-e2e.mjs)

--ruflo is any directory whose node_modules/ holds an installed ruflo tree: $(npm root -g)/ruflo for a global install, or the prefix of npm i --prefix <dir> ruflo@3.42.4. The tree is only read. --plugin is a checkout's plugins/ruflo-metaharness (upstream or this branch).

node mcp-e2e.mjs --ruflo "$(npm root -g)/ruflo" --plugin <checkout>/plugins/ruflo-metaharness --work "$(mktemp -d)" --label <name>
#!/usr/bin/env node
// mcp-e2e.mjs — call ruflo's real metaharness_* MCP tool handlers against an
// installed ruflo tree, with ONLY plugins/ruflo-metaharness swapped for the
// variant under test, and record every npm/npx the tool calls try to spawn.
//
//   node mcp-e2e.mjs --ruflo <installed ruflo pkg dir> --plugin <plugin dir> \
//                    --work <empty scratch dir> [--label X] [--json out.json]
//
// Isolation: nothing outside --work is written. HOME, cwd and
// RUFLO_METAHARNESS_CACHE_BASE all point into --work, and PATH is prefixed
// with a trap dir whose `npm` / `npx` log their argv and exit 1 ("offline").
// The installed ruflo tree is only READ (symlinked), never modified.
import { cpSync, existsSync, mkdirSync, readdirSync, readFileSync, rmSync, symlinkSync, writeFileSync } from 'node:fs';
import { join, resolve } from 'node:path';
import { spawnSync } from 'node:child_process';
import { pathToFileURL } from 'node:url';

const A = {};
for (let i = 2; i < process.argv.length; i += 2) A[process.argv[i].replace(/^--/, '')] = process.argv[i + 1];
const RUFLO = resolve(A.ruflo);
const PLUGIN = resolve(A.plugin);
const WORK = resolve(A.work);
const LABEL = A.label || 'variant';
if (existsSync(WORK) && readdirSync(WORK).length) { console.error(`--work must be empty: ${WORK}`); process.exit(2); }

// 1. Fixture tree: <WORK>/node_modules mirrors <RUFLO>/node_modules via
//    symlinks, except @claude-flow/cli, which is a real directory so the MCP
//    tool module's own __dirname (and therefore locatePluginScripts()) lives
//    in the fixture and finds the swapped plugin — exactly the published
//    layout <cli>/plugins/ruflo-metaharness/scripts.
const SRC_NM = join(RUFLO, 'node_modules');
const NM = join(WORK, 'node_modules');
mkdirSync(join(NM, '@claude-flow'), { recursive: true });
for (const e of readdirSync(SRC_NM)) {
  if (e === '@claude-flow' || e.startsWith('.')) continue;
  symlinkSync(join(SRC_NM, e), join(NM, e));
}
for (const e of readdirSync(join(SRC_NM, '@claude-flow'))) {
  if (e !== 'cli') symlinkSync(join(SRC_NM, '@claude-flow', e), join(NM, '@claude-flow', e));
}
const CLI_SRC = join(SRC_NM, '@claude-flow', 'cli');
const CLI = join(NM, '@claude-flow', 'cli');
mkdirSync(CLI);
for (const e of ['package.json', 'bin', 'dist']) cpSync(join(CLI_SRC, e), join(CLI, e), { recursive: true });
if (existsSync(join(CLI_SRC, 'node_modules'))) symlinkSync(join(CLI_SRC, 'node_modules'), join(CLI, 'node_modules'));
mkdirSync(join(CLI, 'plugins'));
cpSync(PLUGIN, join(CLI, 'plugins', 'ruflo-metaharness'), { recursive: true });

// 2. Trap, HOME, cache base, throwaway project (cwd for every tool call).
const TRAP = join(WORK, 'trap-bin');
mkdirSync(TRAP);
for (const t of ['npm', 'npx']) {
  writeFileSync(join(TRAP, t), `#!/bin/sh\nprintf '%s\\t%s\\n' "${t}" "$*" >> "\${TRAP_LOG:-/dev/null}"\necho "npm ERR! ${t} blocked by test trap (offline)" >&2\nexit 1\n`, { mode: 0o755 });
}
const HOME = join(WORK, 'home');
const CACHE = join(WORK, 'empty-cache-base');
const PROJ = join(WORK, 'project');
for (const d of [HOME, CACHE, PROJ]) mkdirSync(d);
writeFileSync(join(PROJ, 'package.json'), JSON.stringify({ name: 'mh-e2e-project', version: '1.0.0', scripts: { test: 'node -e "process.exit(0)"' } }, null, 2));
writeFileSync(join(PROJ, 'index.js'), 'module.exports = () => 42;\n');
process.env.HOME = HOME;
process.env.PATH = `${TRAP}:${process.env.PATH}`;
process.env.RUFLO_METAHARNESS_CACHE_BASE = CACHE;
delete process.env.RUFLO_METAHARNESS_SKIP_LOCAL;
delete process.env.CLI_CORE;
for (const [cmd, args] of [['git', ['init', '-q']], ['git', ['add', '-A']], ['git', ['-c', 'user.name=e2e', '-c', 'user.email=e2e@example.invalid', 'commit', '-qm', 'init']],
  [process.execPath, [join(CLI, 'bin', 'cli.js'), 'memory', 'init']]]) {
  const r = spawnSync(cmd, args, { cwd: PROJ, encoding: 'utf-8', env: process.env });
  if (r.status !== 0) { console.error(`setup ${cmd} ${args.join(' ')} failed: ${r.stderr}`); process.exit(2); }
}
process.chdir(PROJ);

// 3. Import the REAL compiled MCP tool module from the fixture and call tools.
const { metaharnessTools } = await import(pathToFileURL(join(CLI, 'dist', 'src', 'mcp-tools', 'metaharness-tools.js')).href);
const tool = (n) => metaharnessTools.find((t) => t.name === n);
const rows = [];
let callNo = 0;
async function call(name, input, summarize) {
  const log = join(WORK, `trap-${String(++callNo).padStart(2, '0')}-${name}.log`);
  process.env.TRAP_LOG = log;
  const t0 = Date.now();
  let r;
  try { r = await tool(name).handler(input); } catch (e) { r = { threw: String(e?.message ?? e) }; }
  const ms = Date.now() - t0;
  const calls = existsSync(log) ? readFileSync(log, 'utf-8').trim().split('\n').filter(Boolean) : [];
  let summary;
  try { summary = summarize ? summarize(r) : ''; } catch (e) { summary = `summarize failed: ${e.message}`; }
  rows.push({ name, success: r?.success, degraded: r?.degraded, reason: r?.data?.reason ?? null, exitCode: r?.exitCode, ms, npmNpxCalls: calls.length, firstCall: calls[0] ?? null, summary });
  return r;
}

await call('metaharness_score', { path: '.' }, (r) => `harnessFit=${r.data?.harnessFit} grade=${r.data?.grade ?? '-'}`);
await call('metaharness_genome', { path: '.' }, (r) => `verdict=${r.data?.verdict} risk=${r.data?.risk_score}`);
await call('metaharness_mcp_scan', { path: '.' }, (r) => `findings=${r.data?.findings?.length ?? '-'} worst=${r.data?.worst ?? r.data?.summary?.overallSeverity ?? '-'}`);
await call('metaharness_threat_model', { path: '.' }, (r) => `worst=${r.data?.worst ?? '-'}`);
const oia = await call('metaharness_oia_audit', { path: '.' }, (r) => `worst=${r.data?.composite?.worst ?? '-'} persisted=${r.data?.persisted?.ok ?? '-'}`);
const list = await call('metaharness_audit_list', {}, (r) => `records=${r.data?.returned ?? '-'}/${r.data?.totalInNamespace ?? '-'}`);
const key = list?.data?.records?.[0]?.key ?? oia?.data?.persisted?.key ?? 'audit-missing';
await call('metaharness_audit_trend', { baselineKey: key, currentKey: key }, (r) => `verdict=${r.data?.verdict ?? r.data?.delta?.structuralDistance?.verdict ?? '-'}`);
await call('metaharness_similarity', { aKey: key, bKey: key }, (r) => `overall=${r.data?.overall ?? '-'}`);
await call('metaharness_gepa', { op: 'genome' }, (r) => `valid=${r.data?.valid ?? '-'}`);
await call('metaharness_learn', {}, (r) => `status=${r.data?.status ?? '-'}`);
await call('metaharness_bench', { op: 'create', repo: '.', out: 'bench-suite.json' }, (r) => `suite written=${existsSync('bench-suite.json')} op=${r.data?.data?.op ?? '-'}`);
await call('metaharness_security_bench', { population: 1, cycles: 1 }, (r) => `gates passed=${r.data?.data?.gates?.passed ?? '-'}/${r.data?.data?.gates?.total ?? '-'}`);
await call('metaharness_evolve', { repo: '.', confirm: true, sandbox: 'mock', mutator: 'deterministic', generations: 1, children: 1, concurrency: 1 }, (r) => `success=${r.data?.success} improved=${r.data?.data?.improved ?? '-'}`);
await call('metaharness_redblue', { subcommand: 'attack', family: 'prompt', count: 1 }, (r) => `upstream ok=${r.data?.ok} stdout=${(r.data?.stdout ?? '').length}B`);
await call('metaharness_drift_from_history', { dryRun: true, threshold: 0.5 }, (r) => `similarity=${r.data?.drift?.structuralDistance?.overall ?? '-'}`);

const total = rows.reduce((n, r) => n + r.npmNpxCalls, 0);
console.log(`\n## ${LABEL}  (ruflo tree: ${RUFLO})\n`);
console.log('| MCP tool | success | degraded | reason | npm/npx spawned | result | ms |');
console.log('|---|---|---|---|---:|---|---:|');
for (const r of rows) console.log(`| ${r.name} | ${r.success} | ${r.degraded} | ${r.reason ?? ''} | ${r.npmNpxCalls} | ${r.summary} | ${r.ms} |`);
console.log(`\nTotal npm/npx spawns: ${total}`);
const allCalls = readdirSync(WORK).filter((f) => f.startsWith('trap-') && f.endsWith('.log')).sort()
  .flatMap((f) => readFileSync(join(WORK, f), 'utf-8').trim().split('\n').filter(Boolean).map((l) => `${f.replace(/^trap-\d+-|\.log$/g, '')}: ${l}`));
if (allCalls.length) { console.log('\nTrapped calls:'); for (const c of [...new Set(allCalls)]) console.log(`  ${c}`); }
if (A.json) writeFileSync(A.json, JSON.stringify({ label: LABEL, rows, total, allCalls }, null, 2));
process.exit(0);

Neighbouring suites, upstream vs this branch, in the same environment: a plain git archive checkout of each side (no build), scratch HOME, an empty scratch cache base and the npm/npx trap. Nothing was seeded, so these numbers reproduce on a cold machine:

  • smoke.sh: upstream 107 passed / 16 failed, this branch 109 / 16. The failing-step sets are identical (diff of the two FAIL lists is empty); the +2 are the new 17r2 and 17z74a. Trapped npm/npx calls: 37 on both — in a bare checkout nothing is installed on the walk-up path, so both sides cache-install (the only difference in the log is metaharness@~0.3.0 → metaharness@~0.4.1), and the memory tools fall back to npx because the checkout has no built CLI.
  • test-graceful-degradation 16/16, test-similarity 102/0, _spike-similarity ✓, test-parallel-pipeline 25/25 — identical on both. test-pipeline-roundtrip needs a real metaharness and exits 2 offline on both.
  • test-mcp-tools (the same checkouts with v3/@claude-flow/cli/dist built, so the memory path can resolve a local CLI): 168 passed / 0 failed on both, and its npm/npx spawns drop 21 → 15. Upstream: 12 × metaharness@~0.3.0, 6 × npx @claude-flow/cli@latest memory … (3 list, 3 retrieve), 1 × npx -y -p @metaharness/darwin@~0.8.0, 1 × darwin cache install, 1 × redblue cache install. This branch: 12 × metaharness@~0.4.1, 2 × darwin cache install, 1 × redblue cache install. The six memory calls are gone; darwin's per-call npx becomes a cache install. The pinned cache installs remain because a bare checkout has nothing installed on the walk-up path.
  • ADR-150 drill from no-metaharness-smoke.yml (7 skills, SKIP_LOCAL=1, empty cache, unreachable registry): identical on both — every skill emits its degraded payload and exits 0, except drift-from-history, which exits 3 by design.
  • tsc --noEmit -p v3/@claude-flow/cli: 4 errors, all pre-existing TS6305 from the unbuilt @claude-flow/swarm project references. Byte-identical to the same run with the two touched TypeScript files reverted to e558f0c0f.
  • CLI vitest (--no-file-parallelism) for distill-tuning, harness-verify, neural-router, guidance-brain, agenticow-tools: 59 passed / 8 skipped / 5 failed. All five failures are environmental or pre-existing and unrelated: four in distill-tuning are better-sqlite3 failing to load its native binding (no Node 26 prebuild on this machine), and the fifth is the known guidance-brain › classifies every live registered MCP tool without duplicates. This PR changes no TypeScript behaviour — only two comments.

Performance / compatibility

  • A tool call no longer pays a first-call npm install (up to 180 s) or a per-call npx resolution when ruflo's own copies are present. In the offline end-to-end run above, metaharness_audit_list takes 648 ms (local CLI start plus the list) against the 90 s CI timeout the cold-npx path is documented to hit. I did not measure the old online path, because that means running npx against the registry.
  • Existing ~/.ruflo/metaharness-cache-0.3.0 and darwin-cache-0.8.0 directories are left alone; they are simply never selected again. If no installed copy qualifies (a stripped install, or a CI checkout), the one-time cache install now fetches ~0.4.1 / ~0.10.2.
  • A cache directory that holds package.json but not the bin — npm killed by the 180 s install timeout, or two first calls racing into the same prefix — used to count as installed forever, because ensureCachedInstall's default probe is package.json. npx repaired that by re-resolving; _darwin.mjs now re-runs the install once, probing the bin file itself, and then degrades as before.
  • Windows: the new spawns use process.execPath with shell:false, so oia-audit's JSON --value and redblue's --config/--out/--in no longer go through cmd.exe quoting. The npx fallbacks keep shell: win32, as before. This is not tested on Windows — the new tests skip on win32 because the trap is POSIX sh.
  • ADR-150 is unchanged: no static @metaharness/* import is added, no new dependency, and every unresolvable path returns the same degraded payload with exit 0 (drill above).

How it was found

A cross-CLI check of the MCP tool surface on a real three-client setup (Claude Code, Codex CLI and Grok Build CLI sharing one global ruflo 3.42.4). The metaharness_* tools downloaded packages on first use while ruflo doctor reported metaharness 0.4.2 installed. Pointing the versioned cache directories at an installed metaharness made the pin-backed tools work with nothing downloaded — and an offline end-to-end run of all 15 tools through the real MCP handlers still logged spawns, which is how the second half surfaced: darwin's per-call npx, redblue's cache install, and npx @claude-flow/cli@latest memory.

Notes for the maintainer

  • CI changes metaharness version. metaharness-ci.yml's score and mcp-scan jobs check out the repo with no install step, so the plugin has no local copy to resolve and always takes the cache-install path — today that is 0.3.2, the same release users get; after this PR it is the newest 0.4.x (0.4.16). The same applies to the drift round-trip's oia-audit.mjs, and to oia-audit-weekly.yml, which also runs on the post-merge push because its push paths include _harness.mjs. I ran those gates on both trees, with the pinned caches really installed from npm (~0.3.0 → 0.3.2, ~0.4.1 → 0.4.16) and SKIP_LOCAL=1 to mimic a runner:

    gate main (0.3.2) this PR (0.4.16)
    empty-fixture genome (ruflo metaharness genome exits 2 on any non-ready repo — wrapper treats verdict exit code as fatal error #2626): blocked / exit 2 / risk ≥ 0.7 blocked / 2 / 0.72 blocked / 2 / 0.72
    score --alert-on-fit-below 70 harnessFit 82, exit 0 harnessFit 82, exit 0
    genome --alert-on-risk-above 0.5 risk 0.18, exit 0 risk 0.23, exit 0
    mcp-scan --fail-on high worst low, 1 finding, exit 0 worst medium, 3 findings, exit 0

    The mcp-scan delta is two new risky-bash-allow medium findings for Bash(node .claude/*) and Bash(node:*) in .claude/settings.json, a rule that landed after 0.4.2. --fail-on high still exits 0, so nothing goes red, but the uploaded artifact's worst moves low → medium, and so does the oia-audit composite. The first post-merge weekly drift step compares against a 0.3.2-era artifact. The cross-check that CI really runs 0.3.2 today is that my 0.3.2 column above reproduces the artifacts of main's last green metaharness-ci run at this exact base (run 35277579682, head_sha e558f0c0f, conclusion success): harnessFit 82, and one unpinned-deps low from mcp-scan.

  • The cwd question in "The fix" §3 is the one judgement call I would like confirmed: I scoped the two new lookups to ruflo's own tree and left _harness.mjs's existing cwd walk alone. The opposite choice (let all three search cwd, and document it) is one line.

  • Two comments in TypeScript are updated because this PR is what makes them wrong: distill-oracle.ts's MH_DARWIN_PIN block and metaharness-tools.ts's ADR-153 header both named the plugin's ~0.8.0 pin. Comment-only; tsc output is byte-identical.

  • Still stale, deliberately left: the cache-dir examples in _invoke.mjs now name the new pins, but historical mentions ("metaharness@0.3.0 / darwin@0.8.0" in CLAUDE.md, MCP tool descriptions, gepa/learn headers) record when a feature landed and are left as history. skills/harness-oia-audit/SKILL.md still tells people to run npx @claude-flow/cli@latest memory list by hand — that is a user-facing example, not a code path.

  • Found while testing, not changed here: similarity.mjs --a-key/--b-key hands the whole oia-audit record to similarity(), which reads score / genome at the top level, so identical records score 0.15 while their .fingerprint scores 1.0. It was invisible while the memory path failed. audit-trend and drift-from-history use the fingerprint correctly.

  • Separate follow-up, not changed here: runScript() in metaharness-tools.ts maps a subprocess killed by its 120 s timer (code === null) to exitCode: 0, success: true (L144-L164). After this PR a stock install no longer hits it on first call, but any slow tool still would.

  • metaharness pin drift — a pinned dependency is behind npm latest #3219 (the guard's tracking issue for the ~0.9.0 darwin row) was fixed by fix(metaharness): bump the darwin pin to 0.10.2 (unblocks pin-drift + clean-install on main) #3262 and is still open.

  • Overlap with open PRs: only fix: validate skill frontmatter as YAML #3075 touches a file here. It edits the frontmatter description of skills/harness-threat-model/SKILL.md (line 3, +1/-1) and scripts/audit-skill-frontmatter.mjs, which this PR does not touch; this PR edits line 16 of that SKILL.md, so there is no conflict. fix(metaharness): honor genome verdict exit codes instead of treating them as failure #2644 and fix(metaharness): oia-audit wrappers preserve findings on non-zero exit (#2750) #2751 edit the genome/mcp-scan/threat-model scripts only.

  • ci.yml is flaky on main at this exact base: of the three ci.yml runs at e558f0c0f, the push run and the 2026-09-18 scheduled run passed and the 2026-09-19 scheduled run (35415064637) failed — its ratchet reported reasoningbank-adapter.test.ts and memory-search-scores-3327.test.ts. This PR adds no vitest-collected file, so a red ci.yml here is not from this change.

Checklist

  • Regression tests fail before / pass after (test-pin-alignment.mjs 3/9 → 12/0; test-no-tool-time-npx.mjs 7/19 → 26/0), with a mutation table for every new guard
  • Neighbouring tests pass (smoke, plugin suites, test-mcp-tools, ADR-150 drill, tsc: same results as upstream)
  • No new dependencies
  • Style matches the surrounding code (explanatory header comments, #3366 markers for the issue number, resolution order mirrored from _harness.mjs)

🤖 Generated with Claude Code

https://claude.ai/code/session_01KTedGmPKpEdArVBMMn3JaP

@HF-teamdev

Copy link
Copy Markdown
Contributor Author

Note on the red Test Suite (ubuntu-latest) check — I don't believe it comes from this PR, and here is the evidence rather than an assertion.

The ratchet flags two files:

  + v3/@claude-flow/cli/__tests__/memory-search-scores-3327.test.ts
  + v3/@claude-flow/cli/__tests__/router-bandit.test.ts

Neither appears in this PR's diff (25 files, all under plugins/ruflo-metaharness/, plus scripts/check-metaharness-pins.mjs, metaharness-tools.ts, distill-oracle.ts, and the pin-drift workflow). Run directly on this branch they both pass:

Test Files  2 passed (2)
     Tests  21 passed (21)

memory-search-scores-3327.test.ts also fails on main itself — run 35415064637, at e558f0c0f, with no PR involved — alongside reasoningbank-adapter.test.ts. And the sibling PRs opened in this same batch each trip a different file in that job while three of them pass it outright, which is the signature of flake rather than a regression.

Happy to rebase or re-run if you'd rather see it green before reviewing.

…led instead of npm-installing at tool-call time (ruvnet#3366)

On a stock ruflo 3.42.4 install every metaharness_* MCP tool reached for the
npm registry at tool-call time, although the packages it needs are already on
disk. Two causes, in the same six helper files, which is why they ship
together: the helper PINS rejected what ruflo installs, and three code paths
never looked for an installed copy at all. Fixing only the pins leaves darwin
and the memory tools on npx; fixing only the lookups leaves them resolving a
pin that nothing on disk satisfies.

1. Pins (metaharness, @metaharness/darwin)

The metaharness_* tools resolve MetaHarness through the plugin helpers' own
tilde pins, not through package.json. Those pins were never bumped:
_harness.mjs stayed at ~0.3.0 after the CLI declared metaharness ^0.4.1
(b8ec03f), and _darwin.mjs stayed at ~0.8.0 while the CLI moved
@metaharness/darwin to ~0.9.0 (ruvnet#2958) and ~0.10.2 (ruvnet#3262). On a stock install
(metaharness 0.4.2, darwin 0.10.2) findLocalPackageDir() therefore rejected
the bundled metaharness, and every score / genome / mcp_scan / threat_model /
oia_audit / learn call ran
`npm install --prefix ~/.ruflo/metaharness-cache-0.3.0 metaharness@~0.3.0`
at tool-call time — a download of the older 0.3.2 (180 s install budget inside
a 120 s MCP subprocess budget), degraded offline. The darwin tools fetched
darwin 0.8.x through npx while gepa imported the bundled 0.10.2.

- _harness.mjs: METAHARNESS_PIN_VERSION ~0.3.0 -> ~0.4.1 (= CLI ^0.4.1)
- _darwin.mjs:  DARWIN_PIN_VERSION      ~0.8.0 -> ~0.10.2 (= CLI ~0.10.2)
- scripts/check-metaharness-pins.mjs: the watcher compared only the CLI's
  declared ranges (and MH_DARWIN_PIN), so it reported "all current" while the
  pins that decide what runs were two minors behind. It now reads each
  helper's *_PIN_VERSION: the pin must be a `~X.Y.Z` range (the only form
  _invoke.satisfiesTildeRange accepts), and must cover the declared range, or
  admit npm latest when the CLI declares nothing (redblue). An unreadable
  constant is fatal. `satisfies` now shares a `bounds()` helper with the new
  `covers()`; the results for the existing rows are unchanged.
- metaharness-pin-drift.yml also triggers on the three helper files.
- The two current-pin comments in TypeScript (distill-oracle.ts MH_DARWIN_PIN,
  metaharness-tools.ts ADR-153 header) named the old ~0.8.0 plugin pin; they
  now name ~0.10.2. Comments only, no behaviour.

2. No npm/npx at tool-call time (darwin, redblue, memory)

- _darwin.mjs ran `npx -y -p @metaharness/darwin@<pin> metaharness-darwin` on
  every evolve / bench / security_bench call: npm resolved the range each
  time, it ignored the optionalDependency the CLI ships, and it could run a
  different darwin than the gepa import in the same plugin. It now resolves
  like _harness.mjs — installed copy satisfying the pin, then the one-time
  versioned darwin-cache-<pin> that gepa already shares, then
  `process.execPath <abs bin>` with shell:false.
- _redblue.mjs always npm-installed into redblue-cache-<pin> on first use,
  never looking at the installed copy (ruflo 3.42.4 carries 0.1.4). It now
  checks the installed copy first. The chosen CLI path is realpath'd, because
  redblue only dispatches when import.meta.url === file://argv[1]; a
  pnpm-symlinked package dir (or a cache base under /tmp on macOS) otherwise
  makes it exit 0 having done nothing. Its spawn moves to process.execPath
  with shell:false: the argv carries MCP-supplied --config/--out/--in, which
  under the previous `shell: win32` went through cmd.exe unquoted.
- audit-list / audit-trend / oia-audit / similarity ran
  `npx @claude-flow/cli@latest memory ...` — an npm-registry resolution per
  call (the check _harness.mjs already removed for its own npx path), and
  possibly a different CLI version than the server that spawned the tool
  (ruvnet#3306; cold-cache npx cost ruvnet#3145/ruvnet#3154). Without registry access oia_audit
  reported persisted:false and audit_list silently showed 0 records. They now
  call _invoke.runRufloCli(): process.execPath + bin/cli.js of the
  @claude-flow/cli that ships the plugin (<cli>/plugins/ruflo-metaharness, or
  v3/@claude-flow/cli in a checkout). A candidate needs dist/src/index.js next
  to bin/cli.js (mcp-launch.cjs's unbuilt-checkout guard) plus a package.json
  named @claude-flow/cli (a new check, so an unrelated parent directory can
  never qualify). CLI_CORE=1 still opts into `npx @claude-flow/cli-core@alpha`,
  and a layout with no usable local CLI still falls back to
  `npx @claude-flow/cli@latest`.
- drift-from-history.mjs: drop its unused CLI_PKG constant.

The two new lookups end in a spawn of a package bin with MCP-supplied argv, so
they pass findLocalPackageDir(..., { fromCwd: false }) and search only the
ruflo install that owns the plugin — never the tool's cwd or its ancestors.
The pre-existing metaharness lookup keeps its cwd walk unchanged.

Tests (both hermetic, published layout, stub packages, npm/npx PATH trap):

- test-pin-alignment.mjs (+ smoke 17z74a): 3 passed / 9 failed on main,
  12 / 0 here.
- test-no-tool-time-npx.mjs (+ smoke 17r2): 7 passed / 19 failed on main,
  26 / 0 here. It covers both darwin success branches (installed copy and the
  versioned cache), the memory round trip through the shipping CLI, the kept
  contracts (CLI_CORE, npx fallback, a bin/cli.js that is not
  @claude-flow/cli, darwin absent -> degraded exit 0), and the hardening
  above: a darwin/redblue in the tool's own cwd node_modules is never
  executed, a cache dir with package.json but no bin gets one reinstall
  instead of degrading forever, and neither helper spawns the resolved bin
  through a shell.

ADR-150 is unchanged: no static @metaharness import is added, no new
dependency, and every unresolvable path still returns the same
{degraded:true, reason} with exit 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KTedGmPKpEdArVBMMn3JaP
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants