Repository navigation
fix(agent-evals): skip commits that landed on the base branch - #4471
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Agent evals❌ 6 failed · ✅ 13 passed ❌ Failed (6)
✅ Passed (13)
Audited 47a3511. This check only reports. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CodeQL flags exported functions that pass a caller-provided array to `git` (js/second-order-command-line-injection). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Description
For local sessions, the provider audits the changes since the commit the session started on. If the branch is rebased onto the base branch during the session, or recreated from it (for example, after an earlier pull request from the same session lands), every commit that landed on the base branch in between gets audited too. In the session behind #4470, the diff contained 42 files instead of 8. The
changesetsrule then reported missing change files for five unrelated packages.The diff now starts at the later of two commits: the session's start commit, or where the branch forks off the base branch. That excludes commits already on the base branch. The base branch is:
baseBranchoption, if set;refs/remotes/<remote>/HEAD;git ls-remote --symref). Clones often don't set<remote>/HEAD; this one doesn't.The start commit is still used when:
Commands, tool calls and the transcript still cover the whole session, so rules based on them are unchanged.
metadata.baseCommitnow reports the commit that the diff starts from.@rnx-kit/tools-gitThe base branch lookup uses
getDefaultBranch()andgetBaseCommit()from@rnx-kit/tools-git, instead of reimplementing them. They previously only worked in the current directory withoriginandHEAD. UnderCIthey returned a hard-coded fallback, and otherwise they threw iforigin/HEADdidn't exist. They now:cwdandremote(andheadforgetBaseCommit());<remote>/HEADbefore falling back on CI;Existing calls behave the same, except that a missing
origin/HEADno longer throws outside of CI. The existinggetDefaultBranch()test failed in clones withoutorigin/HEADfor that reason.agent-evalsstill has its own asyncgit()helper for everything else. Moving it totools-gitneeds an async, throwing, untrimmed variant, so it's left for a separate change.Types in
github.ts,grader.tsandreport.tswere moved above the functions, with no other changes.Test plan
yarn lintandyarn buildinincubator/agent-evalsandincubator/tools-git: passedyarn test --experimental-test-coverage 'test/**/*.test.ts':incubator/agent-evals: 145/145 passed (previously 140)incubator/tools-git: 10/10 passed (previously 4/5;getDefaultBranch()failed withoutorigin/HEAD)agent-evals/src/provider/provider.tsagent-evals, all filestools-git/src/branch.tstools-git, all filesFrom the repository root:
node scripts/lint-tsconfig.ts: passedyarn knip: passedyarn constraints: passednode scripts/lint-node-version.ts: passed (it also usestools-git)I also checked the #4470 session directly.
forkPoint()with its start commit (20c1480) and branchagent-evals/azure-devops-hostreturns f7e9528 (#4467, the tip oforigin/main) through thels-remotefallback, instead of 20c1480.