Reality probe: is red→green evidence misleading when the test oracle disagrees with the product?

@konsta95

This is a CounterProof Reality Probe about evidence semantics, not a request to change anthropics/claude-code#89404.

@konsta95 @sylvesterkaczmarek — your review thread exposed a case I think is more important than ordinary “tests pass” verification.

The PR adds validate-agent.test.sh and reports 5/5 passing tests. But the review established two stronger facts:

the test suite asserts exit 0 for agent files that the actual product's claude plugin validate rejects as invalid YAML/frontmatter;
reverting the multi-line description extraction can still leave the 5/5 suite green while the false <example> warning returns.

So even a perfect before/after replay of the submitted test could produce a technically real red→green result while still being weak evidence for the actual product behavior.

Why I am asking

CounterProof currently has three distinct evidence outcomes from real external PRs:

Regression Witness — same changed test is PASS on HEAD / FAIL on BASE.
Proof Integrity — surface when the PR changes the CI/test-support machinery producing its own evidence.
INCONCLUSIVE / NOT WITNESSED — refuse to turn compile/setup/judge changes into a success claim.

#89404 suggests a fourth problem:

the submitted test can be internally consistent but use the wrong oracle.

For this PR, claude plugin validate --json appears to be closer to the authoritative oracle than validate-agent.sh's own notion of valid frontmatter.

Question

If you were reviewing an AI-assisted PR like #89404, would the useful evidence shape be something like:

Submitted regression:
  BASE -> FAIL
  HEAD -> PASS

Authoritative oracle probe:
  product parser on claimed-valid fixture -> REJECT

Conclusion:
  regression is real relative to the submitted judge,
  but the judge is not aligned with product behavior

Or is there a different distinction that would be more useful in practice?

I am specifically trying not to build an “oracle alignment” feature until a real reviewer says what information would actually reduce review uncertainty. A short answer is plenty.

Yes. I would treat this as a separate evidence class from an ordinary regression witness.

A BASE-fail / HEAD-pass replay only proves that the submitted judge distinguishes the two revisions. Before using that as evidence for product correctness, the judge itself has to be checked against the authoritative product behavior.

The useful output would be:

submitted regression: BASE fail / HEAD pass;
authoritative oracle probe: expected fixture accepted or rejected by the product;
oracle alignment: aligned / contradicted / unverified;
conclusion: witnessed relative to the submitted test, but product claim proven / contradicted / still unproven.

For #89404, if the product validator rejects a fixture that the regression labels valid, I would want the result surfaced as “regression witnessed, oracle contradicted”, not as a successful proof. That distinction would materially reduce review uncertainty.

@sylvesterkaczmarek

@sylvesterkaczmarek — I took your “separate evidence class” suggestion literally and made a manual oracle-alignment artifact before building anything:

https://github.com/hippoley/CounterProof/blob/reality/reviewer-requested-matrices/docs/reality/claude-code-89404-oracle-matrix.md

The shape is:

submitted regression
authoritative oracle probe
oracle alignment = aligned / contradicted / unverified
claim status

For #89404 the important row is deliberately CONTRADICTED: the submitted suite treats the plugin-dev fixtures as valid, while the product validator rejects them. The multi-line extraction protection stays UNPROVEN because the review showed the suite remains green when that fix is reverted.

If this is close to what you meant, I’ll use this artifact rather than a single PASS badge as the design target. If not, I’d rather change the manual format now than automate the wrong abstraction.

This is close to the design target.

The one change I would make before automating it is to avoid SUPPORTED, oracle not separately challenged as a status. It is softer than the rest of the evidence vocabulary and can be read as a product-level endorsement.

I would keep the axes mechanical:

submitted-test evidence: WITNESSED / NOT WITNESSED / UNPROVEN;
oracle alignment: ALIGNED / CONTRADICTED / UNVERIFIED;
overall claim: derived from those two.

So for the first row, use UNVERIFIED for oracle alignment unless there is an actual product-oracle probe. For the fixture-validity row, CONTRADICTED is exactly right. For the multi-line extraction row, UNPROVEN is exactly right.

That keeps a green submitted suite from becoming stronger evidence merely because nobody has challenged its oracle yet.

@sylvesterkaczmarek

@sylvesterkaczmarek — the oracle axis you asked for is now encoded in the merged minimal claim-matrix artifact:

#38

The implementation deliberately keeps oracle authority external. CounterProof only validates:

submitted-test evidence = WITNESSED / NOT_WITNESSED / UNPROVEN;
oracle alignment = ALIGNED / CONTRADICTED / UNVERIFIED;
overall claim is mechanically derived.

For a contradiction, the oracle wins regardless of an internally green submitted judge.

The #89404 acceptance fixture is here:
https://github.com/hippoley/CounterProof/blob/main/examples/claim_matrix/claude-code-89404.yml

I also removed the softer SUPPORTED vocabulary exactly as you suggested.

If you would actually use this shape in review, great; if the machine-readable version still overstates anything, I would rather change it now than grow more features around the wrong contract.

The mechanical derivation now matches what I meant, including the removal of SUPPORTED and the rule that a contradicted product oracle wins.

One trust-boundary refinement before building more on it: for ALIGNED or CONTRADICTED, I would require auditable oracle provenance, not only free-text oracle_probe. At the moment a manifest can produce PROVEN with an arbitrary probe string and no oracle_source_url. Requiring an oracle source URL (or an equivalent explicit authority/provenance field) for non-UNVERIFIED oracle states would make the derived status inspectable rather than merely asserted.

With that provenance tightened, I would use this machine-readable shape directly in review.

This addresses the trust-boundary concern I raised. Requiring an explicit absolute HTTP(S) oracle_source_url whenever the oracle state is ALIGNED or CONTRADICTED makes those asserted states auditable, while keeping UNVERIFIED free of a false provenance requirement.

The additional WITNESSED consistency check is also useful: a manifest can no longer label arbitrary BASE/HEAD outcomes as a regression witness. I would keep the URL check framed as provenance validation rather than oracle verification, since syntactic URL validity does not establish that the source is authoritative or that its content supports the claim. With that distinction, the follow-up preserves the evidence semantics I intended.

@sylvesterkaczmarek

@sylvesterkaczmarek @konsta95 — closing the loop on the oracle-alignment thread you both shaped.

The #89404 case is now published as a provenance-bound negative flagship rather than a generic “green test” example:

https://github.com/hippoley/CounterProof/blob/main/reality/claude-code-89404/ORACLE_DISAGREEMENT.md

Machine receipt:
https://github.com/hippoley/CounterProof/blob/main/examples/claim_matrix/receipts/claude-code-89404-oracle.json

The current artifact keeps the exact boundary you asked for:

submitted judge: green;
authoritative product oracle: rejects the claimed-valid fixtures;
oracle alignment: CONTRADICTED;
overall claim: CONTRADICTED;
provenance is explicit and auditable;
it explicitly says CounterProof did not independently execute the Claude binary.

I’ve stopped adding new proof types for now. The only question I’m trying to answer next is whether this final artifact shape is actually useful in review.

If you were reviewing a PR like #89404 today, is there still one piece of information missing from this artifact that you would need before relying on it?

If the answer is “no”, I’ll treat this seam as complete and move effort to external delivery/adoption rather than adding more schema.

For this seam, no — I do not think anything essential is missing now. In review I would rely on this artifact as evidence that the submitted regression and the product oracle disagree, provided the receipt pins the exact BASE/HEAD revisions, preserves the authoritative oracle source/provenance, and keeps the explicit caveat that CounterProof did not itself execute the product binary. That is enough for me to treat the product claim as contradicted rather than merely unproven. I would stop adding schema here and move effort to adoption/delivery.

Navigation Menu

Repository navigation

Reality probe: is red→green evidence misleading when the test oracle disagrees with the product? #16

Why I am asking

Question

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Reality probe: is red→green evidence misleading when the test oracle disagrees with the product? #16

Description

Why I am asking

Question

Activity

sylvesterkaczmarek commented on Sep 24, 2026

hippoley commented on Sep 24, 2026

sylvesterkaczmarek commented on Sep 24, 2026

hippoley commented on Sep 24, 2026

sylvesterkaczmarek commented on Sep 24, 2026

sylvesterkaczmarek commented on Sep 24, 2026

hippoley commented on Oct 7, 2026

sylvesterkaczmarek commented on Oct 7, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions