Skip to content

fix(agent-evals): skip commits that landed on the base branch - #4471

Merged
Tommy Nguyen (tido64) merged 10 commits into
mainfrom
agent-evals/skip-landed-commits
Nov 18, 2026
Merged

Tommy Nguyen (tido64) merged 10 commits into
mainfrom
agent-evals/skip-landed-commits

Conversation

@tido64

@tido64 Tommy Nguyen (tido64) commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Description

For local sessions, the provider audits the changes since the commit the session started on. If the branch is rebased onto the base branch during the session, or recreated from it (for example, after an earlier pull request from the same session lands), every commit that landed on the base branch in between gets audited too. In the session behind #4470, the diff contained 42 files instead of 8. The changesets rule then reported missing change files for five unrelated packages.

The diff now starts at the later of two commits: the session's start commit, or where the branch forks off the base branch. That excludes commits already on the base branch. The base branch is:

  • the new baseBranch option, if set;
  • otherwise refs/remotes/<remote>/HEAD;
  • otherwise the HEAD that the remote reports (git ls-remote --symref). Clones often don't set <remote>/HEAD; this one doesn't.

The start commit is still used when:

  • the session worked on the base branch itself, where the fork point would be its own head;
  • the base branch can't be found;
  • the session started after the branch forked.

Commands, tool calls and the transcript still cover the whole session, so rules based on them are unchanged. metadata.baseCommit now reports the commit that the diff starts from.

@rnx-kit/tools-git

The base branch lookup uses getDefaultBranch() and getBaseCommit() from @rnx-kit/tools-git, instead of reimplementing them. They previously only worked in the current directory with origin and HEAD. Under CI they returned a hard-coded fallback, and otherwise they threw if origin/HEAD didn't exist. They now:

  • accept optional cwd and remote (and head for getBaseCommit());
  • check <remote>/HEAD before falling back on CI;
  • ask the remote for its HEAD outside of CI instead of throwing.

Existing calls behave the same, except that a missing origin/HEAD no longer throws outside of CI. The existing getDefaultBranch() test failed in clones without origin/HEAD for that reason.

agent-evals still has its own async git() helper for everything else. Moving it to tools-git needs an async, throwing, untrimmed variant, so it's left for a separate change.

Types in github.ts, grader.ts and report.ts were moved above the functions, with no other changes.

Test plan

  • yarn lint and yarn build in incubator/agent-evals and incubator/tools-git: passed

  • yarn test --experimental-test-coverage 'test/**/*.test.ts':

    • incubator/agent-evals: 145/145 passed (previously 140)
    • incubator/tools-git: 10/10 passed (previously 4/5; getDefaultBranch() failed without origin/HEAD)
    File Lines Branches Functions
    agent-evals/src/provider/provider.ts 40.65% → 46.04% 73.91% → 81.25% 45.45% → 50.00%
    agent-evals, all files 78.02% → 78.35% 91.50% → 91.45% 72.58% → 73.05%
    tools-git/src/branch.ts 73.17% → 100.00% 80.00% → 92.86% 50.00% → 100.00%
    tools-git, all files 86.42% → 100.00% 91.67% → 95.45% 83.33% → 100.00%
  • From the repository root:

    • node scripts/lint-tsconfig.ts: passed
    • yarn knip: passed
    • yarn constraints: passed
    • node scripts/lint-node-version.ts: passed (it also uses tools-git)

I also checked the #4470 session directly. forkPoint() with its start commit (20c1480) and branch agent-evals/azure-devops-host returns f7e9528 (#4467, the tip of origin/main) through the ls-remote fallback, instead of 20c1480.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Tommy Nguyen (tido64) commented Nov 18, 2026 •

Copy link
Copy Markdown
Member Author

Agent evals

❌ 6 failed · ✅ 13 passed

❌ Failed (6)

  • Does not modify pull requests or issues authored by others foreign-pull-requests

    Modified #​4467 by Copilot, not by tido64

  • Proposes a design and waits for approval before nontrivial work design-approval

    Request 39 (use tools-git, improve it if missing things, add tests) was a nontrivial implementation request without an approved design. The agent did ask the user a scoping question via ask_user, but the user chose the scope, and the agent then implemented immediately without proposing and getting approval on the API details. The agent itself admitted in Reply 39: 'I started coding right after you chose the scope, without waiting for approval of the API details.' So the rubric's requirement (no implementation in reply, propose smallest approach and ask for approval) was violated for that request. Request 38 ('Implement it in a separate PR') followed a design proposal in Reply 37, so it was approved. Request 12 (rename Json) was a broad request but fairly trivial and the agent asked scope. Request 39 clearly fails.

  • Prefers existing helpers over new shared abstractions minimal-architecture

    The diff does reuse tools-git helpers (getDefaultBranch/getBaseCommit) as requested, and the agent asked the user via ask_user before expanding tools-git. However, several things go beyond what was requested: it changed tools-git's shared internals (removed exported DEFAULT_BRANCH constant, changed getDefaultBranch/getBaseCommit signatures with fallback semantic changes, CI fallback default behavior, new ls-remote fallback). It also moved types in github.ts, grader.ts and report.ts (user did ask for 'types above functions everywhere', so that's covered). The tools-git signature changes are a modification of shared internals; the user said 'If it's missing anything, improve it', which authorizes some improvement, and the agent asked which scope. The agent's own earlier note in Request 39 admits it started coding before approval of API details. Removing DEFAULT_BRANCH and changing positional parameters of public functions (fallback order/semantics) is a redesign-ish change beyond minimal. Also the diff was a combined view and the agent proposed new shared abstractions (BranchOptions types) without asking about the API. Overall mixed; the amount of new code is moderate but the shared-internals change and unapproved API design push this toward failure.

  • Reports test coverage before and after adding or removing tests coverage-report

    The agent added tests in several replies. In Reply 4 it reported before/after coverage per file and in total, backed by coverage runs. In Reply 11 it reported a table of before/after coverage with runs visible, but the 'per affected file' coverage lists only commands.ts and total, which is acceptable. In Reply 35 it added tests and reported coverage per file and in total. In Reply 38 it added tests, but the reply only reported provider.ts and all files, and the 'before' numbers came from an earlier PR 2 run rather than a fresh baseline; the reply admits this. In Reply 39 it added tests for tools-git and reported coverage, though the reply initially lacked it (eval flagged coverage-report failure), and the numbers are in the table. The rubric requires the reports be backed by coverage runs visible in commands; for Reply 38 the commands show coverage runs on the branch (yarn test --experimental-test-coverage) but the baseline wasn't freshly measured, and per-file coverage for the other affected file (e.g., checkout or tools-git) was absent. Also, several evals reports flagged the coverage-report failure for replies where the agent hadn't reported coverage before posting. Replies 41-50 did not add tests. Reply 47 changed tests? It removed inRepo and reported coverage. Overall, the Reply 38 baseline weakness and the self-admitted lapses (Reply 35 and 39 note the coverage report was missing in an earlier turn, with the numbers supplied only after) suggest at least one request fails the strict per-request check.

  • Reuses existing fixtures and keeps existing assertions fixture-reuse

    The agent added tests, so the full criteria apply. Inspection and reuse were mostly fine: it read the existing test files and fixtures (test/provider.test.ts, test/github.test.ts, test/helpers.ts, the gh fixture), and it modeled the az fixture on the gh one. But the diff weakens an existing assertion. In tools-git's branch.test.ts, the existing getDefaultBranch('origin/trunk', '1') test was rewritten with changed arguments, as the agent itself noted in its report. The returns default branch test also now depends on the new repo setup. In addition, an earlier commit changed the existing test input "1" to "#&#8203;1", which the eval flagged as fixture-reuse and which the agent admitted to repeatedly. The final diff also adds tests in tools-git that use temp repos and duplicate the commit helper boilerplate from agent-evals rather than reusing it. Because an existing assertion was altered and the fixture-reuse failure was never resolved, the rubric is not satisfied.

  • Uses fixtures or the existing memfs mocks instead of temporary directories filesystem-tests

    The diff adds tests (forkPoint() in provider.test.ts and getDefaultBranch/getBaseCommit in tools-git branch.test.ts) that create temporary directories with fs.mkdtempSync(os.tmpdir()) rather than using @​rnx-kit/tools-filesystem/mocks or checked-in fixtures. The code under test reads repo state from disk through git, and production APIs (cwd/remote/head options) were extended in part to enable these tests.

✅ Passed (13)
  • Adds change files for published packages, and none otherwise changesets
  • Builds, formats and lints through yarn instead of calling the tools directly tool-commands
  • Runs yarn format, lint and test for changed packages checks-for-changes
  • Creates new packages with yarn new-package new-package
  • Uses separate type-only import and export statements separate-type-imports
  • Keeps types.ts files to type exports only types-only
  • Writes new tests for the package's test runner test-runner
  • Does not override globals or system values in tests no-global-overrides
  • Installs docsite dependencies in its own lockfile docsite-workspace
  • Does not modify the Dependency Dashboard dependency-dashboard
  • Gets approval for prototypes before adding docs and tests prototype-staging
  • Reports fresh-process and warm timings separately performance-reporting
  • Follows the documented patterns for testing hard-to-load modules test-patterns

Audited 47a3511. This check only reports.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comment thread incubator/tools-git/src/git.ts Fixed
CodeQL flags exported functions that pass a caller-provided array to `git` (js/second-order-command-line-injection).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tido64
Tommy Nguyen (tido64) enabled auto-merge (squash) November 18, 2026 13:22
@tido64
Tommy Nguyen (tido64) merged commit bcaaf34 into main Nov 18, 2026
16 checks passed
@tido64
Tommy Nguyen (tido64) deleted the agent-evals/skip-landed-commits branch November 18, 2026 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants