[Dream Cycle 2026-09-03] memory: SmartRetrieval's MMR used token-Jaccard instead of embedding-cosine (evaluated, ACCEPT) + plugins,automation scan #3168

1. Tonight's Rotation

2. Ledger Check

Inspected main's ledger (ends 2026-08-19) plus the last several nights' actual state via direct GitHub MCP queries and git ls-remote (never inferred from the sparse table alone, per STEP 1's anti-inference rule):

• 2026-08-14..19: dream(swarm): power-of-two-choices mesh load-balancing (evaluated, REJECTED) #3027/dream(performance): HNSWIndex efSearch query-time default (evaluated, REJECTED) #3034/dream(security): #3043 advisory scanner for untrusted settings.json hooks/allow-rules (evaluated, ACCEPT) #3044/dream(intelligence): #3048 discounted Thompson sampling for model-router bandit (evaluated, ACCEPT-scoped) #3049/dream(memory): #3056 hybridSearch reachable via explicit opt-in (evaluated, ACCEPT-scoped) #3057/dream(swarm): #3061 bound MessageBus retry attempts, fix unreachable message.failed (evaluated, ACCEPT) #3062 — 08-16..19 batch-merged 2026-08-21; 08-14/15 (REJECT-evidence) correctly still unmerged.
• 2026-08-20..23: confirmed (again) genuine no-run gap — no branch/PR/issue for any of these 4 dates.
• 2026-08-24..09-02: 9 nights, all evaluated ACCEPT, all still open/draft (dream(swarm): #3085 weightedConsensus() trust weights now reach the vote tally (evaluated, ACCEPT) #3086, dream(performance): #3093 wire productQuantizeDistance() into HNSW search path (evaluated, ACCEPT) #3094, dream(security): #3102 bind ADR-377 caller-identity verification into live authorizeMcpTool chokepoint (evaluated, ACCEPT-with-caveats) #3103, dream(intelligence): #3109 wire computeConfidencePenalty/updateFisherFromConfidences into distillLearning's EWC gate (evaluated, ACCEPT-scoped, corrected post-review) #3110, dream(memory): #3118 wire HybridBackend's dead weights field into weighted-RRF fusion (evaluated, ACCEPT) #3119, dream(swarm): #3122 fix TopologyManager.rebalanceHybrid() one-directional adjacency (evaluated, ACCEPT) #3123, dream(performance): #3129 remove unwired diskann-backend.ts, fix stale HNSW claims in generated CLAUDE.md (evaluated, ACCEPT) #3130, dream(security): #3138 opt-in MCP governance policy enforcement (evaluated, ACCEPT) #3139, dream(security): #3151 sliding-window reset for maxToolCallsPerTurn (evaluated, ACCEPT) #3152, dream(intelligence): #3159 LearningBridge.consolidate() reward-blind fix (evaluated, ACCEPT) #3160) — no batch-merge since 08-21. Each PR's diff already contains its own LEDGER.md row (STEP 25 ran correctly every night); the rows just haven't reached main.

This exact gap has now been flagged as a candidate automation fix in 3 consecutive gists (08-19, 08-28, 09-02) without ever being acted on. Per STEP 1.1 ("if the same finding appears 3+ times, reject duplicate research direction"), tonight does not write a 4th recovery note — it ships the fix (§11).

3. Deep Dive Findings

Five parallel research roles (Deep Researcher, Competitor Analyst, 2 Scan Researchers for plugins/automation, Ruflo Architecture Reviewer) fanned out. Full findings, grading, and citations in the committed gist (docs/dream-cycle/dream-gist-2026-09-03.md).

Ground-truthed by direct code read: smart-retrieval.ts's mmrRerank() (MMR diversity, phase 4 of SmartRetrieval's 5-phase pipeline) computed similarity via token-Jaccard text overlap — the file's own header names it "MMR Diversity (token-Jaccard proxy)." memory-initializer.ts's searchEntries() computes real embedding-cosine similarity at 2 live sites for the primary relevance score, then discards the parsed embedding (return type never included one). Deep Researcher confirmed this is not a live 2025-2026 research question — LangChain's reference MMR, Qdrant's native Mmr query (shipped Sept 2025), and Weaviate's MMR reranker (v1.37, Apr 2026) all use embedding-cosine; no source found treats lexical-only diversity as a serious production option once embeddings exist. Architecture Reviewer independently verified the exact bug mechanism, corrected scope to include a second instance (controller-registry.ts's toCands(), different root cause — embedding available but never copied), estimated patch size (~40-60 LOC, additive/optional), and flagged a sibling gap (deferred, not fixed tonight): searchEntries() also never returns metadata/createdAt/updatedAt, silently no-opping 2 of SmartRetrieval's other 3 phases (recency boost, session round-robin) for every CLI/MCP smart:true search.

4. Hypothesis

Given SmartRetrieval's MMR diversity re-ranking step (mmrRerank) operating on SearchCandidate objects sourced from searchEntries(), when the per-pair similarity term is changed from token-Jaccard text overlap to embedding-cosine similarity (reusing the ONNX embedding searchEntries() already computes but previously discarded), falling back to token-Jaccard when either candidate lacks an embedding or dimensions mismatch, then MMR-selected diversity should correctly suppress low-token-overlap semantic near-duplicates that a lexical-only proxy misses, subject to: (1) non-MMR pipeline phases unchanged; (2) all existing tests remain green; (3) $0 evaluation cost; (4) no leak into CLI/MCP JSON responses.

Frozen before evaluation; not modified after seeing results.

5. Evaluation Receipt

evaluated: accepted. Real evaluator: Vitest, deterministic, zero LLM calls, $0 cost. Discriminating test: seed A; B (paraphrase, 1/11 Jaccard overlap, cosine≈1.0 to A); C (different topic, 0 Jaccard overlap, cosine≈0, lower relevance score than B). Baseline (git stash-isolated, test file kept): picks B second (0.2154 > 0.21 under Jaccard) — reproduced live, confirmed failing. Candidate: picks C second. A second, non-discriminating fallback-regression test passes both ways. Full @claude-flow/memory suite: 460/461 passing identically with/without the candidate (controller-registry.test.ts+graceful-retrieval.test.ts: 75/75); the 1 failure is the pre-existing chmod/root-sandbox environmental failure documented since 08-15. tsc --noEmit on @claude-flow/memory: zero errors. @claude-flow/cli's tsc --noEmit: 455 errors both with and without this candidate (verified via git stash, identical count — pre-existing, unbuilt workspace packages, unrelated to this diff).

6. Darwin Results

Skipped — scope mismatch, same class as every recent night. Correctness/similarity-metric fix with a discriminating unit test, not a continuous parameter with a gold-labeled corpus to search a fitness gradient over.

7. Flywheel Evidence

No signed @metaharness/flywheel bundle — deterministic Vitest evidence, same class as every accepted night since 08-18. Classified: OBSERVATION (token-Jaccard proxy confirmed by direct code read + header comment) / MEASUREMENT (baseline-fails/candidate-passes via stash isolation, 460/461 full-suite green) / INFERENCE (embedding-cosine correctly distinguishes paraphrase-near-duplicates from genuinely-different topics; does not touch non-MMR phases) / DECISION (ship as a small, additive, fallback-preserving fix) / REJECTION (DPP-based diversification — thin single-paper evidence, not adopted by any production vector DB, too high-risk for tonight's small-patch bias).

8. Reward Hack Check

No reward-hack CLI reachable this session. Manual checklist clear: no test weakened (additive only); no benchmark/gold-answer path (no LLM); no cherry-picking (full 461-test suite run both ways); no cost/latency increase (pure in-memory float math, embedding was already parsed in scope — no new I/O); no seed manipulation; no undocumented caching; no threshold changes; embedding does not leak into CLI/MCP JSON responses (checked directly — both mappers whitelist output fields).

9. Security Review

In-process ranking/fusion logic only. No new network/filesystem-boundary/credential/MCP-authority surface. No concerns.

10. Scan Findings: plugins

Confirmed and sharpened the 2026-08-28 finding (still open, not yet fixed): PluginManager.installFromNpm() (v3/@claude-flow/cli/src/plugins/manager.ts) shells straight to npm install --prefix <dir> <name>[@version], never touching IPFS or comparing the download against the plugin registry's checksum/cid fields. discovery.ts does real, fail-closed Ed25519 verification on the registry listing — but that integrity story stops at the metadata layer; trustLevel/minTrustLevel/blockedPlugins/compatibilityMatrix/minClaudeFlowVersion are all defined and displayed (plugins list/info) but never enforced as an install gate. Several registry checksum demo values are literal placeholder strings (e.g. "sha256:abc123neural"). Shell-injection on package names is already well-mitigated (VALID_PACKAGE_RE + execFile array-form); no-sandbox execution is already self-disclosed via console.warn. External: npm's Sigstore-backed provenance/Trusted Publishing has been GA since mid-2025 (Grade A/B) — the closest real-world analog to what this registry's Ed25519 signing already does one layer up, but isn't cross-checked at install time. Nothing in the field has fundamentally changed since the May-July ledger's plugin-supply-chain findings; this is a specific, still-open, cheap-to-fix code-path gap, not a new standard.

11. Scan Findings: automation

Root cause of the recurring "ledger gap" (§2), confirmed for the third time and finally acted on: not a STEP-25 execution failure — every unmerged PR's diff already contains its LEDGER.md row. The mechanism: each dream-cycle branch becomes a draft PR requiring human review; rows only reach main when that PR merges. No batch-merge since 2026-08-21 → 9-night backlog. Shipped tonight: .github/workflows/dream-cycle-backlog-guard.yml — scheduled (daily, 07:11 UTC) + workflow_dispatch, lists open dream/* PRs via gh pr list, flags any older than a threshold (default 3 days) in the job summary with a ::warning:: annotation. Modeled on the existing oia-audit-weekly.yml pattern. This does not fix the append mechanism itself (that requires editing the Dream Cycle prompt, which lives outside this repo) — it makes the backlog visible on a schedule instead of requiring the next Dream Cycle session to re-derive the same diagnosis. Validated: YAML parses; the age/breach logic was run locally against a fixture matching real current PR data (10-day and 20-day-old PRs correctly flagged; a 1-day-old PR correctly not flagged).

12. Competitors Reviewed

Full "why the gap exists" analysis in the gist.

13. Gist

docs/dream-cycle/dream-gist-2026-09-03.md (committed on this branch; no gh gist create capability available in this session — consistent with every dream-cycle night since 2026-08-14).

14. Witness

Verifier procedure: fetch docs/dream-cycle/dream-gist-2026-09-03.md from this branch as it existed before the witness table was filled in, SHA-256 it, concatenate with the session commit above, SHA-256 again — result must equal the witness stamp.

15. Recommendation

1. Merge the linked draft PR — small (~110 net lines), additive, fully reversible fix matching Qdrant/Weaviate's converged production pattern, with an honest baseline-fails/candidate-passes discriminating test.
2. Follow-up (deferred, larger scope): extend searchEntries() to also return metadata/createdAt/updatedAt (needs 2 SQL SELECTs extended, not just plumbing an already-computed value) so recency-boost and session-round-robin stop silently no-opping on CLI/MCP-driven smart searches.
3. Follow-up (plugins, 08-28 finding still open): wire registry.checksum verification into installFromNpm(), or drop the decorative checksum/trust fields from the registry schema.
4. Human review/merge of the 08-24..09-02 ACCEPT-evidence backlog — the new backlog-guard workflow will surface this automatically going forward; it does not replace a human batch-merge.