Repository navigation
dream(intelligence): #3301 gate findSimilar()'s confidence on reliability, not query-similarity (evaluated, ACCEPT) - #3302
Conversation
Records tonight's finding, verified GitHub fates for the trailing 7 nights (including a newly-discovered merge conflict on #3232), and the evaluation verdict. Co-Authored-By: RuFlo <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z
…tion LocalReasoningBank.findSimilar() overwrote every returned pattern's confidence (learned reliability) with the per-query cosine similarity score, instead of returning the score under the already-declared PatternMatch.similarity field. This silently broke distillLearning()'s "only distill from high-confidence matches" gate (it tested query-similarity, not reliability) and forced findSimilarPatterns()'s public API into an unsafe type-cast fallback that made confidence and similarity numerically identical for every caller. Fix: findSimilar() now preserves the real stored confidence and returns the cosine score as a distinct similarity field; findSimilarPatterns() reads the real field instead of casting; memory-bridge.ts's search- ranking consumer (which wants query-relevance, not reliability) is migrated to prefer similarity, falling back to confidence. Evaluated: baseline-fails/candidate-passes via git-stash isolation (4 new discriminating tests), full @claude-flow/cli suite identical 90-vs-91-failed-file sets (only delta is the new test file; all other failures are pre-existing unbuilt-sibling-package issues), tsc --noEmit byte-identical. Independent adversarial critique: CONFIRMED, no blocking caveats. Darwin: skipped (binary correctness fix, no fitness gradient). Security: not sensitive, in-process bookkeeping only. Co-Authored-By: RuFlo <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z
Records tonight's finding, verified GitHub fates for the trailing 7 nights (including a newly-discovered merge conflict on #3232), and the evaluation verdict. Co-Authored-By: RuFlo <ruv@ruv.net> Claude-Session: https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z
…nistic
CI's "Test Suite (ubuntu-latest)" job (root-level ci-test-ratchet.mjs)
flagged this new test file as an unexpected failure: the "confidence
and similarity differ for a partial match" case called the real
findSimilarPatterns(), which tries a live ONNX embedding (falling back
to a hash) before it can compare fields. In the CI runner that path
silently returned [] (no cached model / restricted network),
so `hit` was undefined.
Mock both embedding sources (memory-bridge.js's bridgeGenerateEmbedding,
memory-initializer.js's generateEmbedding) to a fixed deterministic
vector. This matches the evaluation's own stated claim ("fully
deterministic, $0 evaluation cost") that this one test wasn't actually
honoring. Reproduced the exact CI invocation locally (root-level
vitest, not the cli package's own scoped config) — all 4 tests now
pass; local package-scoped suite unaffected (7/7 across this file and
the adjacent hooks-intelligence-learning.test.ts).
Co-Authored-By: RuFlo <ruv@ruv.net>
Claude-Session: https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z
92d0308 to
8fa7419
Compare
|
CI status: red, but not this PR's — standing down per the drive-to-green policy after one re-run. The
Neither failing file is imported, touched, or logically connected to No fix exists to port for either (native-binding availability isn't a code bug in this PR's scope, and the quantization recall test's exact-zero result looks like a timing/build-order race rather than a deterministic logic error). Per policy, having used the one re-run, I'm standing down here rather than re-running again or widening this PR's scope to touch unrelated packages. This PR's own diff, and its own new test file, remain green in every run so far — 4/4 pass locally and 4/4 passed in the one CI run so far that got far enough to report the ratchet result. Will keep this PR watched; a future genuinely-red result on Generated by Claude Code |
1. Hypothesis
Frozen before evaluation; not modified after seeing results.
2. Candidate
v3/@claude-flow/cli/src/memory/intelligence.ts(+18/-5):findSimilar()'s return type changed fromStoredPattern[]to(StoredPattern & { similarity: number })[]; the map body changedreturn { ...s.pattern, confidence: s.score }→return { ...s.pattern, similarity: s.score }(confidence now flows through untouched via the spread).findSimilarPatterns()'s mapping now reads the realr.similarityfield instead of the unsafe(r as unknown as {similarity?:number}).similarity ?? r.confidence ?? 0.5cast.v3/@claude-flow/cli/src/memory/memory-bridge.ts(+5/-1): the one downstream consumer that actually wanted the old query-similarity semantics (bridgeSearchPatterns's ranking score, ~line 2189) is migrated toscore: r.similarity ?? r.confidence ?? r.score ?? 0—??(not||) so a legitimate0similarity isn't incorrectly overridden.3. Evaluation Receipt
evaluated: accepted. Real evaluator: Vitest 4.1.8, deterministic (hand-crafted embeddings, no randomness), zero LLM calls, $0 cost. New file:
v3/@claude-flow/cli/__tests__/intelligence-confidence-similarity-conflation.test.ts(4 tests).Baseline vs. candidate, isolated via
git stashof just the 2 source files (test file kept): all 4 tests fail against baseline for exactly the predicted mechanism —expected 1 to be 0.42(identical-embedding confidence overwritten by cosine 1.0),expected 0.44998... to be 0.9(partial-match confidence overwritten by cosine ~0.45),expected 0.30100000000000005 to be 0.30000000000000004(the DISTILL gate's inverted-inclusion bug — a low-reliability pattern wrongly included receives an extra LoRA bump it shouldn't get),expected +0 to be 0.85(public API's conflated field). All pass on candidate.Full
@claude-flow/clipackage suite, run both ways: baseline 91 failed/149 passed/1 skipped (241 files), candidate 90 failed/150 passed/1 skipped — the only difference in the failed-file sets (commdiff) is the new test file itself. The other 90 failing files are identical, pre-existing, environmental failures (unbuilt monorepo sibling packages — e.g.@claude-flow/neural/@claude-flow/memoryVite module-resolution failures — unrelated to this diff, same class documented in #3221/#3243).tsc --noEmit: 463 pre-existing errors, byte-identical count and lines both ways.4. Baseline Comparison
findSimilar()result.confidencefindSimilar()result.similaritydistillLearning()gate testsfindSimilarPatterns()confidencevssimilaritytsc --noEmit5. Darwin Lineage
Skipped — binary correctness/API-contract fix (confidence either reflects reliability or it doesn't), not a continuous/categorical parameter with a fitness gradient for Darwin's real interface (
npx ruvector harness darwin --help, confirmed available tonight,@metaharness/darwin@0.9.2) to search over. Same class as several recent nights (#3110, #3160, #3184, #3221, #3243).6. Flywheel Evidence
No
.claude-flow/flywheel/state or signed@metaharness/flywheelbundle exists in this repo (confirmed vianpx ruvector harness flywheel --help). Evidence retained as: the 4 new tests, the gist, issue #3301, the stash-isolated baseline/candidate comparison, and the independent adversarial critique — consistent with every accepted night since 2026-08-18.7. Reward Hack Check
No standalone reward-hack CLI reachable this session (
@metaharness/weight-eftis a LoRA-distillation tool, not a diff/benchmark scanner). Manual checklist, independently re-verified by the adversarial critic: no existing test weakened (new file only), no gold/expected data touched (none exists on this path), no cherry-picking (full suite reported both ways, all 90 pre-existing failures disclosed not hidden), no seed manipulation (deterministic hand-crafted embeddings throughout), zero cost, discriminating power confirmed real (fails on baseline for the stated reason, not a flake).8. Security Review
Not security-sensitive: in-process ranking/confidence bookkeeping inside a local pattern store, no new I/O, network, credential, or filesystem surface. Independently confirmed by the adversarial critic (no
fs/http/exec/env changes in the diff). Thememory-bridge.tsmigration's??fallback chain was specifically checked against a||footgun on a legitimate0similarity value —??is correct,||would have silently regressed ranking.Independent adversarial critique (separate subagent, no authoring context): verdict CONFIRMED. Re-derived the bug from the raw diff independently, re-ran the discriminating test both ways itself (reproducing 4/4 pass on the fix, 4/4 fail on baseline for the claimed reasons), grepped every
.findSimilar(call site inv3/and confirmed no consumer was missed, verifiedendTrajectory()'s RL-update loop is unaffected, and verified the??vs||correctness claim itself. One cosmetic-only nit (a doc line-number off by 10, corrected). No blocking caveats.9. Regression Analysis
endTrajectory()'s RL-update loop (lines 269-289) readspattern.confidencevia a separatebank.get()call, never the match object's field — confirmed unaffected by both this session and the independent critic. Non-findSimilar()code paths are untouched. The one behavioral migration (memory-bridge.ts) preserves its consumer's existing ranking semantics rather than silently changing them.10. ADR
None — this is a scoped RETRIEVE-stage API-contract/correctness fix (an already-declared interface
PatternMatch.similaritythe implementation never honored), matching the explicit exclusion criteria used by prior nights for this class of finding (e.g. 2026-09-03 MMR fix/#3169, 2026-09-09 AgentPool fix/#3243).11. Research Gist
docs/dream-cycle/dream-gist-2026-09-12.md(this branch) — full SOTA report (5 parallel research roles: 2026 self-learning-agent literature, capabilities scan, memory scan, competitor analysis, independent architecture review), competitor comparison table (LangGraph, Microsoft Agent Framework, CrewAI, OpenAI Agents SDK, Google ADK, plus Letta/GEPA/ACE as more specific comparisons), and 5 additional scored candidates not selected tonight. No gist-creation tool is reachable in this execution environment (GitHub MCP tools cover issues/PRs/repos, not gists) — committed todocs/dream-cycle/instead, matching established repo convention.12. Issue
Closes evidence trail documented in #3301 (full 15-section report: ledger check with GitHub-verified fates for the last 7 nights, research, evaluation receipt, adversarial critique, reward-hack check, security review, capabilities/memory scan findings, competitor comparison, witness).
13. Witness
39e0b0540c9b018174955fc8a21f355bbac26c6acb5ee8eff4a9ef7d0123ea74e3d6c30dba3bf0c2922dd1db0235a55ffc633a09cb60b43b6c2130b41626657fc19395214d2ee929e0aff4fd4ee05468cd4bca57Verifier procedure: fetch
docs/dream-cycle/dream-gist-2026-09-12.mdfrom this branch, strip the witness table's filled values back toPENDING, SHA-256 it, concatenate with the session commit above, SHA-256 again — result must equal the witness stamp.14. Merge Policy
Human review required. Do not self-merge. Do not autonomously promote Flywheel state.
🤖 Generated with RuFlo
https://claude.ai/code/session_013LQSLa9h1uLdYsQBP5r36z
Generated by Claude Code