[mbani01]

Summary

• The nightly mirror sweep now counts only mirrored repos against its limit. Skipped repos (renamed, not found, Nango or GitHub errors) no longer use up the batch: the sweep keeps taking the next candidates until the limit is mirrored or the list runs out. Each group is sized to what is still needed, so it never goes past the shadow cap.
• The schedule's default batchSize goes from 1000 to 2500. The last nights each tried 1000 and mirrored 858–926; the skips were what made up the difference.

Rollout

1. The schedule is create-only, so the live one keeps batchSize: 1000 until it is updated by hand with temporal schedule update ({ "batchSize": 2500, "shadowCap": 5000 }).
2. Deploy connectors-worker outside 03:15–03:30Z. Only the activity changed, but a deploy during the sweep cancels the rest of that night's batch.
3. Next night: @event:mirror_sweep_summary should show mirrored equal to the room, not attempted.

Notes

• shadowCap 5000 is what limits the batch now: 2613 repos were in shadow on 2026-10-30, so the room is about 2387 and shrinks as the shadow fills.
• When nearly every repo is being skipped (a GitHub rate limit or a Nango outage), the sweep now works through all ~12.8k candidates instead of stopping at the limit. At about 1000 repos per 3 minutes that is roughly 40 minutes, inside the 90-minute activity timeout.

Verification

• Vitest: the batch test now has to try 7 candidates to mirror 5 with 2 skips, and must not touch the 8th. Under the old code it would mirror only 4.
• tsc -b services/apps/connectors_worker and oxlint --deny-warnings are clean.

Refs linuxfoundation/lfx-cdp-insights-ops#934

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[cursor]

PR Summary

Medium Risk
Changes long-running nightly mirror batch semantics and may scan many more candidates during widespread skip storms, though still bounded by activity timeout; schedule default only affects new registrations.

Overview
The nightly mirror sweep now stops when mirrored hits the batch limit, not when it has attempted that many repos. Skipped outcomes (404, Nango/GitHub errors, etc.) no longer consume quota: mirrorCandidates walks the full activity-ordered list in concurrent groups of up to five, sizing each group to limit - result.mirrored so it never overshoots the cap.

mirrorBatch passes the entire ordered candidate list into that loop instead of pre-slicing to limit. The Vitest case is updated so a limit of 5 requires 7 attempts (5 mirrored, 2 skipped) and never touches the eighth repo—behavior the old slice-first logic would not achieve.

The Temporal schedule registration default batchSize changes from 1000 to 2500 (existing schedules keep their args until updated manually).

^{Reviewed by Cursor Bugbot for commit 725af57. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

🟢 Approval recommended

No active review issues were identified in the mirror-sweep changes to count successful mirrors toward the batch limit.

No open issues 🎉

What changed in this PR

Updates the connectors worker’s nightly mirror sweep to count successful mirrors toward its limit, so skipped repositories no longer reduce the batch’s yield.

Changes:

• Continues through ordered candidates until the mirror target is reached or candidates run out, sizing each group to the remaining target.
• Raises the default batch size from 1,000 to 2,500 for newly created schedules; existing schedules require a manual update.
• Adjusts the batch test to verify that seven attempts produce five mirrors despite two skips.

File	Description
services/​apps/​connectors_worker/​src/​schedules/​mirrorSweep.ts	Raises the schedule’s default batch size to 2,500.
services/​apps/​connectors_worker/​src/​activities/​mirrorSweepActivities.ts	Refills skipped candidates while respecting the successful-mirror limit.
services/​apps/​connectors_worker/​src/​activities/​mirrorSweepActivities.test.ts	Updates the test to verify skip refilling and stopping at the target.

_{Review effort: Balanced}

[Copilot]

🟡 Changes recommended

Persistent Nango 503 errors can keep the serial refill loop running until the 90-minute activity timeout because it lacks an elapsed-time budget or circuit breaker.

1 open issue

• Bound refill retries during persistent Nango outages · New

_{Review effort: Balanced}