Rebasing a stack dismisses "stale review" of downstream PRs

[feosuna1]

If I have an approved stack in a repo that has "Dismiss stale pull request approvals when new commits are pushed" enabled. When I rebased the entire stack, the first PR in the stack retained it's approval state meanwhile the downstream PRs dismissed their approval states and I have to re-request approvals.

[liias]

Same issue after merging the first PR in the stack.

[imanmahjoubi]

Thank you so much for sharing this! forwarded this to the team so we can work on it 🙏🏼

[praseodym]

And worse, trying to re-review a rebased PR using "Changes since your last review" will show a "We went looking everywhere, but couldn’t find those commits. Sometimes commits can disappear after a force-push." error page.

[gugahoi]

This is still not working particularly well. When rebasing a stack that is fully approved by clicking the "rebase stack" in the UI, all approvals get dismissed and it mostly loses the entire benefit of having a stack.

[spencerbeggs-cf]

We narrowed down when this happens. Merging a lower PR dismisses an approval above it only when the upper PR changes lines within a few lines of the lower PR's change, even though the upper PR's diff is identical before and after the rebase. Approvals on upper PRs whose changes are farther away are kept, as the stacked PRs reference describes.

Setup. A ruleset on main with dismiss_stale_reviews_on_push: true, require_last_push_approval: true, require_code_owner_review: true, 1 required approval and squash only. We merge the bottom PR through the merge-async API (direct_merge, squash), and GitHub then rebases the rest of the stack (automatic_base_change_succeeded, then head_ref_force_pushed).

Minimal repro (two PRs, one file):

1. On main, add f.txt with lines line 1 to line 200.
2. PR A (base main): change line 70.
3. PR B (base A, in the same stack): change line 71.
4. Approve A and B, then merge A.
5. B's approval is dismissed with "The merge-base changed after approval." B's tree after GitHub's rebase is identical to its tree before, and so is its diff.

Change step 3 to edit line 170 instead, and B's approval is kept.

What we tested (each a stack of 2 or 3 PRs, with the bottom PR merged and the PRs above checked):

Case	Upper approvals
Changes in different files, or far apart in the same file (1 to 40 files, 1 or 2 commits per PR)	Kept
Approver is a bot, a person or a code owner	Kept (only the line overlap matters)
allow_update_branch or delete_branch_on_merge on	Kept
Upper PR edits a line next to a lower PR's change (one line, or across 30 files)	Dismissed

We saw the same dismissal on a real stack: a reviewer re-approved, the lower PR merged and the approval was dismissed, all with the upper PR's tree unchanged. It looks as though the stale check compares the diff hunks and their context against the old merge-base instead of comparing the diff against the new base. A stack where each PR builds on the one below edits nearby lines all the time, for example imports at the top of a file, so in practice most approvals above a merge are lost.

We'd expect an approval to be kept whenever the rebased PR's diff against its new base is identical to the diff that was approved.

[spencerbeggs-cf]

Follow-up: this reproduces under an organization-level ruleset as well, not only a repository ruleset. With only the org ruleset applying (dismiss_stale_reviews_on_push: true, require_last_push_approval: true, code-owner review, 1 approval, squash only), merging the bottom PR through merge-async:

• Upper PR changes the line right after the lower PR's change: approval dismissed ("The merge-base changed after approval."), with an identical tree and diff.
• Upper PR changes a line 80 lines away in the same file: approval kept.