Skip to content

dream(intelligence): #3301 gate findSimilar()'s confidence on reliability, not query-similarity (evaluated, ACCEPT) - #3302

Merged
ruvnet merged 3 commits into
mainfrom
dream/2026-09-12-intelligence
Sep 15, 2026
Merged

ruvnet merged 3 commits into
mainfrom
dream/2026-09-12-intelligence

Conversation

@ruvnet

@ruvnet ruvnet commented Sep 12, 2026

Copy link
Copy Markdown
Owner

1. Hypothesis

Given LocalReasoningBank.findSimilar() (v3/@claude-flow/cli/src/memory/intelligence.ts:604-638) returning pattern objects whose confidence field is overwritten with the per-query cosine similarity score, when the candidate change stops overwriting confidence and instead returns the cosine score as a distinct similarity field (satisfying the pre-existing, already-exported PatternMatch extends Pattern { similarity: number } contract at line 1195 that findSimilarPatterns() was already trying, and failing, to satisfy via an unsafe type-cast fallback), then (a) distillLearning()'s "only distill from high-confidence matches" gate (match.confidence < 0.5, line 357) should correctly test the pattern's learned reliability rather than its similarity to the current trajectory step, and (b) findSimilarPatterns()'s public API should return genuinely distinct confidence/similarity values instead of two copies of the same number — relative to baseline where both are conflated, subject to:

  1. memory-bridge.ts's search-ranking consumer (~line 2189), which reads the match score for relevance ranking, is migrated to prefer the new similarity field (falling back to confidence defensively) so its ranking behavior isn't silently switched from relevance to reliability;
  2. endTrajectory()'s RL-update consumer (line 269-289), which already re-fetches pattern.confidence via bank.get() rather than reading the match object's field, is unaffected;
  3. all existing tests remain green;
  4. fully deterministic, $0 evaluation cost, no LLM calls.

Frozen before evaluation; not modified after seeing results.

2. Candidate

v3/@claude-flow/cli/src/memory/intelligence.ts (+18/-5): findSimilar()'s return type changed from StoredPattern[] to (StoredPattern & { similarity: number })[]; the map body changed return { ...s.pattern, confidence: s.score } → return { ...s.pattern, similarity: s.score } (confidence now flows through untouched via the spread). findSimilarPatterns()'s mapping now reads the real r.similarity field instead of the unsafe (r as unknown as {similarity?:number}).similarity ?? r.confidence ?? 0.5 cast.

v3/@claude-flow/cli/src/memory/memory-bridge.ts (+5/-1): the one downstream consumer that actually wanted the old query-similarity semantics (bridgeSearchPatterns's ranking score, ~line 2189) is migrated to score: r.similarity ?? r.confidence ?? r.score ?? 0 — ?? (not ||) so a legitimate 0 similarity isn't incorrectly overridden.

3. Evaluation Receipt

evaluated: accepted. Real evaluator: Vitest 4.1.8, deterministic (hand-crafted embeddings, no randomness), zero LLM calls, $0 cost. New file: v3/@claude-flow/cli/__tests__/intelligence-confidence-similarity-conflation.test.ts (4 tests).

Baseline vs. candidate, isolated via git stash of just the 2 source files (test file kept): all 4 tests fail against baseline for exactly the predicted mechanism — expected 1 to be 0.42 (identical-embedding confidence overwritten by cosine 1.0), expected 0.44998... to be 0.9 (partial-match confidence overwritten by cosine ~0.45), expected 0.30100000000000005 to be 0.30000000000000004 (the DISTILL gate's inverted-inclusion bug — a low-reliability pattern wrongly included receives an extra LoRA bump it shouldn't get), expected +0 to be 0.85 (public API's conflated field). All pass on candidate.

Full @claude-flow/cli package suite, run both ways: baseline 91 failed/149 passed/1 skipped (241 files), candidate 90 failed/150 passed/1 skipped — the only difference in the failed-file sets (comm diff) is the new test file itself. The other 90 failing files are identical, pre-existing, environmental failures (unbuilt monorepo sibling packages — e.g. @claude-flow/neural/@claude-flow/memory Vite module-resolution failures — unrelated to this diff, same class documented in #3221/#3243). tsc --noEmit: 463 pre-existing errors, byte-identical count and lines both ways.

4. Baseline Comparison

Baseline Candidate
findSimilar() result .confidence Overwritten with query-cosine score Real stored reliability, untouched
findSimilar() result .similarity Does not exist Real per-query cosine score
distillLearning() gate tests Query-similarity (inverted intent) Learned reliability (correct)
findSimilarPatterns() confidence vs similarity Always numerically identical Genuinely distinct
Full package suite 149 passed / 91 failed (241 files) 150 passed / 90 failed (241 files)
tsc --noEmit 463 errors 463 errors (same lines)

5. Darwin Lineage

Skipped — binary correctness/API-contract fix (confidence either reflects reliability or it doesn't), not a continuous/categorical parameter with a fitness gradient for Darwin's real interface (npx ruvector harness darwin --help, confirmed available tonight, @metaharness/darwin@0.9.2) to search over. Same class as several recent nights (#3110, #3160, #3184, #3221, #3243).

6. Flywheel Evidence

No .claude-flow/flywheel/ state or signed @metaharness/flywheel bundle exists in this repo (confirmed via npx ruvector harness flywheel --help). Evidence retained as: the 4 new tests, the gist, issue #3301, the stash-isolated baseline/candidate comparison, and the independent adversarial critique — consistent with every accepted night since 2026-08-18.

7. Reward Hack Check

No standalone reward-hack CLI reachable this session (@metaharness/weight-eft is a LoRA-distillation tool, not a diff/benchmark scanner). Manual checklist, independently re-verified by the adversarial critic: no existing test weakened (new file only), no gold/expected data touched (none exists on this path), no cherry-picking (full suite reported both ways, all 90 pre-existing failures disclosed not hidden), no seed manipulation (deterministic hand-crafted embeddings throughout), zero cost, discriminating power confirmed real (fails on baseline for the stated reason, not a flake).

8. Security Review

Not security-sensitive: in-process ranking/confidence bookkeeping inside a local pattern store, no new I/O, network, credential, or filesystem surface. Independently confirmed by the adversarial critic (no fs/http/exec/env changes in the diff). The memory-bridge.ts migration's ?? fallback chain was specifically checked against a || footgun on a legitimate 0 similarity value — ?? is correct, || would have silently regressed ranking.

Independent adversarial critique (separate subagent, no authoring context): verdict CONFIRMED. Re-derived the bug from the raw diff independently, re-ran the discriminating test both ways itself (reproducing 4/4 pass on the fix, 4/4 fail on baseline for the claimed reasons), grepped every .findSimilar( call site in v3/ and confirmed no consumer was missed, verified endTrajectory()'s RL-update loop is unaffected, and verified the ?? vs || correctness claim itself. One cosmetic-only nit (a doc line-number off by 10, corrected). No blocking caveats.

9. Regression Analysis

endTrajectory()'s RL-update loop (lines 269-289) reads pattern.confidence via a separate bank.get() call, never the match object's field — confirmed unaffected by both this session and the independent critic. Non-findSimilar() code paths are untouched. The one behavioral migration (memory-bridge.ts) preserves its consumer's existing ranking semantics rather than silently changing them.

10. ADR

None — this is a scoped RETRIEVE-stage API-contract/correctness fix (an already-declared interface PatternMatch.similarity the implementation never honored), matching the explicit exclusion criteria used by prior nights for this class of finding (e.g. 2026-09-03 MMR fix/#3169, 2026-09-09 AgentPool fix/#3243).

11. Research Gist

docs/dream-cycle/dream-gist-2026-09-12.md (this branch) — full SOTA report (5 parallel research roles: 2026 self-learning-agent literature, capabilities scan, memory scan, competitor analysis, independent architecture review), competitor comparison table (LangGraph, Microsoft Agent Framework, CrewAI, OpenAI Agents SDK, Google ADK, plus Letta/GEPA/ACE as more specific comparisons), and 5 additional scored candidates not selected tonight. No gist-creation tool is reachable in this execution environment (GitHub MCP tools cover issues/PRs/repos, not gists) — committed to docs/dream-cycle/ instead, matching established repo convention.

12. Issue

Closes evidence trail documented in #3301 (full 15-section report: ledger check with GitHub-verified fates for the last 7 nights, research, evaluation receipt, adversarial critique, reward-hack check, security review, capabilities/memory scan findings, competitor comparison, witness).

13. Witness

Field Value
Session commit 39e0b0540c9b018174955fc8a21f355bbac26c6a
Gist SHA-256 (pre-witness content) cb5ee8eff4a9ef7d0123ea74e3d6c30dba3bf0c2922dd1db0235a55ffc633a09
Witness stamp cb60b43b6c2130b41626657fc19395214d2ee929e0aff4fd4ee05468cd4bca57

Verifier procedure: fetch docs/dream-cycle/dream-gist-2026-09-12.md from this branch, strip the witness table's filled values back to PENDING, SHA-256 it, concatenate with the session commit above, SHA-256 again — result must equal the witness stamp.

14. Merge Policy

Human review required. Do not self-merge. Do not autonomously promote Flywheel state.


🤖 Generated with RuFlo

https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z


Generated by Claude Code

ruvnet added a commit that referenced this pull request Sep 12, 2026
Records tonight's finding, verified GitHub fates for the trailing 7
nights (including a newly-discovered merge conflict on #3232), and the
evaluation verdict.

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z
claude and others added 3 commits September 14, 2026 20:02
…tion

LocalReasoningBank.findSimilar() overwrote every returned pattern's
confidence (learned reliability) with the per-query cosine similarity
score, instead of returning the score under the already-declared
PatternMatch.similarity field. This silently broke distillLearning()'s
"only distill from high-confidence matches" gate (it tested
query-similarity, not reliability) and forced findSimilarPatterns()'s
public API into an unsafe type-cast fallback that made confidence and
similarity numerically identical for every caller.

Fix: findSimilar() now preserves the real stored confidence and returns
the cosine score as a distinct similarity field; findSimilarPatterns()
reads the real field instead of casting; memory-bridge.ts's search-
ranking consumer (which wants query-relevance, not reliability) is
migrated to prefer similarity, falling back to confidence.

Evaluated: baseline-fails/candidate-passes via git-stash isolation (4
new discriminating tests), full @claude-flow/cli suite identical
90-vs-91-failed-file sets (only delta is the new test file; all other
failures are pre-existing unbuilt-sibling-package issues), tsc
--noEmit byte-identical. Independent adversarial critique: CONFIRMED,
no blocking caveats.

Darwin: skipped (binary correctness fix, no fitness gradient).
Security: not sensitive, in-process bookkeeping only.

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z
Records tonight's finding, verified GitHub fates for the trailing 7
nights (including a newly-discovered merge conflict on #3232), and the
evaluation verdict.

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z
…nistic

CI's "Test Suite (ubuntu-latest)" job (root-level ci-test-ratchet.mjs)
flagged this new test file as an unexpected failure: the "confidence
and similarity differ for a partial match" case called the real
findSimilarPatterns(), which tries a live ONNX embedding (falling back
to a hash) before it can compare fields. In the CI runner that path
silently returned [] (no cached model / restricted network),
so `hit` was undefined.

Mock both embedding sources (memory-bridge.js's bridgeGenerateEmbedding,
memory-initializer.js's generateEmbedding) to a fixed deterministic
vector. This matches the evaluation's own stated claim ("fully
deterministic, $0 evaluation cost") that this one test wasn't actually
honoring. Reproduced the exact CI invocation locally (root-level
vitest, not the cli package's own scoped config) — all 4 tests now
pass; local package-scoped suite unaffected (7/7 across this file and
the adjacent hooks-intelligence-learning.test.ts).

Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z
@ruvnet
ruvnet force-pushed the dream/2026-09-12-intelligence branch from 92d0308 to 8fa7419 Compare September 15, 2026 00:04

ruvnet commented Sep 15, 2026

Copy link
Copy Markdown
Owner Author

CI status: red, but not this PR's — standing down per the drive-to-green policy after one re-run.

The Test Suite (ubuntu-latest) job has now failed twice on this exact head (8fa7419), each time on a different file, neither of which this PR touches:

  1. First failure (run 34911604119, attempt 1): v3/@claude-flow/plugins/__tests__/ruvector-quantization.test.ts — AssertionError: expected 0 to be greater than or equal to 0.1 in the binary-quantization search-recall test. This exact commit passed CI cleanly on 2026-09-12 (all checks green, "Test Suite" included), and main's own most recent push-triggered run passed too — ruled out as a base-branch issue. Re-ran once (per policy) to confirm flake vs. regression.
  2. Second failure (attempt 2, after the automatic re-run): v3/@claude-flow/cli/__tests__/doctor-native-binding-2968.test.ts — AssertionError: expected 'fail' to be 'warn'. This asserts on whether the doctor command's structural check reports warn vs fail for a missing better-sqlite3 native binding — directly dependent on whether that native binding happens to be built/available in the runner at test time, not on anything this diff changes.

Neither failing file is imported, touched, or logically connected to intelligence.ts/memory-bridge.ts/the new intelligence-confidence-similarity-conflation.test.ts. Two different, unrelated, environment-state-dependent tests failing on two consecutive runs of the same commit is consistent with resource contention / native-binding availability flakiness in this CI runner's ~90-job matrix (the same class of issue already disclosed as pre-existing/environmental in #3221, #3243, and elsewhere in this repo's history), not a regression introduced here.

No fix exists to port for either (native-binding availability isn't a code bug in this PR's scope, and the quantization recall test's exact-zero result looks like a timing/build-order race rather than a deterministic logic error). Per policy, having used the one re-run, I'm standing down here rather than re-running again or widening this PR's scope to touch unrelated packages. This PR's own diff, and its own new test file, remain green in every run so far — 4/4 pass locally and 4/4 passed in the one CI run so far that got far enough to report the ratchet result. Will keep this PR watched; a future genuinely-red result on intelligence.ts/memory-bridge.ts/this PR's own test file is a different situation and I'll act on it directly.


Generated by Claude Code

@ruvnet
ruvnet marked this pull request as ready for review September 15, 2026 00:34
@ruvnet
ruvnet merged commit 2abc929 into main Sep 15, 2026
120 of 122 checks passed
This was referenced Sep 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants