Skip to content

feat(simulate): mark failed scenarios that were already failing before the run - #1039

Draft
u9g wants to merge 2 commits into
mainfrom
sim-historically-red
Draft

u9g wants to merge 2 commits into
mainfrom
sim-historically-red

Conversation

@u9g

@u9g u9g commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Problem

When lk agent simulate reports a failed scenario, the output looks the same whether the change under test broke it or the scenario was already failing in earlier runs of the same agent. Coding agents that read this output (CI logs, the TUI report file) then spend their time chasing failures that predate their change.

Change

The server now marks each job with its scenario's recent history (SimulationRun_Job.prior_passed, SimulationRun_Job.historically_red; see the protocol PR below). The CLI only reads the server's historically_red flag. It does not recompute the threshold. The marker is shown only for failed scenarios that the server flags as historically red:

  • CI / plain output and the TUI report file: the job's ::group:: header and its ::error:: line get a suffix:
    ::group::✗ Refund request (sample 1/2) [historically red: failed 8 of last 10 runs] (SRJ_a1)
    ::error::Job 1 failed: agent hung up [historically red: failed 8 of last 10 runs]
    
    N counts failures in prior_passed, and M is its length.
  • Run summary: one extra line that counts scenarios, not samples, using the same grouping as the pass/fail verdict:
    1 of 2 failed scenarios were already failing before this run (historically red): Refund request
    
  • TUI: a dim historically red tag on the failed row in the scenario list, and in the scenario and job detail headers.
  • --export JSON: protojson already includes prior_passed and historically_red, so no code change was needed.

The first commit is a pure refactor. It splits the per-scenario verdict out of scenarioFailureCounts so the report can check whether a single scenario failed.

Dependencies

@u9g

u9g commented Oct 15, 2026

Copy link
Copy Markdown
Contributor Author

Demo with a fixture run, because the server does not fill these fields yet: CI output marks the two failures that were already red (header, ::error:: line, and summary line) and leaves the new failure unmarked. The TUI scenario list then shows the dim historically red tag on the failed scenario.

historically-red.mp4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant