Repository navigation
MAX_ENQUEUED_REMOVALS insufficient for gas limit errors. Datasets stuck #283
Description
We've observed an instance where the gas limit is being hit upon processing nextProvingPeriod. From @TippyFlitsUK:
Short version: while removing the duplicate pieces from Covalent's Storacha migration data set (1245), the scheduled removals made the next nextProvingPeriod tx too big to fit in a block. Gas estimation reverts, so nothing ever hits the chain... proving just silently stops. Every removal call was inside the allowed limits (500 per call, ~2k per proving period) - the caps happily let you build a state the chain can't actually execute.
This has always been my biggest outstanding worry about pdp. The guess at a safe value for the constant was wrong and so we are now observing a liveness issue. Moreover any guess for this constant is going to be wrong since PDPVerifier has no control over the listener contract which may happily eat up a large part of the message/block gas limit.
Here are some solution ideas
external processRemovals call processes the queue before nextProvingPeriod
One way to handle this is to allow partial queue draining via a separate call processRemovals which processes a user specified number of queue entries. When nextProvingPeriod fails with out of gas then callers like curio can move over to using this resumable queue draining call.
This maintains the invariant that all pieces added to the queue will eventually be processed while enusring livenss of nextProvingPeriod under the assumption that the listener call maintains liveness.
A listener call must be added for removalsProcessed so that it can handle tracking of removed pieces.
This new call would need to be handled with care to avoid grinding. It must be enforced that the proving set is not modified between challenge epoch sampling and proving. To accomplish this we will need a compliant listener that asserts that either proving has been missed or proving has completed this period before processing these calls. This is readily achievable with existing state in the filecoin-services contracts.
external cancelScheduledRemovals call empties queue before nextProvingPeriod
The other way to handle this is to simply cancel the queued removals to make space for nextProvingPeriod.
This removes the invariant that all pieces added to the queue are deleted before nextProvingPeriod which means grinding needs to be considered. Considering this further this operation is always safe. All manipulations on the proving set must occur between start of the current proving period and the call to nextProvingPeriod and operate only on the enqueued removals without impacting any proving state between challenge sampling and proving.
Note we'll still need a listener call scheduledRemovalsCancelled to help the listener revert any tracking that occured in the scheduling listener callback.
For this reason we could make cancleScheduledRemovals take in a count parameter or a list of piece ids and things would still be safe. But for simplicity I recommend just doing a single drop operation as an escape hatch guaranteeing liveness.
Recommendation
Cancellation is overall simpler for verifier and listener and reflects the fact that this is an escape hatch not a common case path. I also suspect it will be cleaner in curio.
We should also consider doing measurements on filecoin-services listener listener gas costs and PDPVerifier nextProvingPeriod gas costs and reduce the constant to something that better reflects a safe threshold in today's environment.
Activity
snadrus commentedon Jul 2, 2026
The always-safe option sounds best to me
So Curio could send 1 remove when it gets the notice that Bulk Removes are cancelled, to verify. Then apply some edge-finding algorithm to find the smaller bulk level that'll work.
rvagg commentedon Jul 3, 2026
Mostly this should be in the SP's hands to solve to keep things functioning. But, they do get paid for removals, so there should be ample remuneration for handling the hassle. One thing we should be aware of when making this flexible is that the removal fee is queued when
schedulePieceDeletionsis run, notnextProvingPeriod. So if we're going to allow the SP to de-schedule removals then we need to either move that fee or have a reverse rail or some other kind of debt .. we probably shouldn't move the fee intonextProvingPeriod, this issue shows why we try to keep things out of there.I like a separate
processRemovals(n)call prior tonextProvingPeriodif we can make it work with the proving system. That would solve the whole thing, if we shunt removals entirely to a separate call (makenextProvingPeriodblock if there are removals pending) then we could even properly gas-account for the cost of removals and build that into any pricing update in the future, that would be neat, but would mean mandatory extra calls; so .. tradeoffs.Another possibility might be a
nextProvingPeriod(n)variant where the SP could choose to process some number of removals less than the total amount queued. But we'd need to consider how to hold the SP accountable for actually making the removals happen over time and not just collecting the removal fee, making a vague promise to the user, but letting pieces sit for many days, or even the entire term of the data set. The differential between "I want to delete these pieces" and when/if they are deleted is already annoying with the 2000 limitation; at least it's clear though, "no more than 2000 per period" is at least a coherent rule, if confusing for users.Kubuxu commentedon Jul 13, 2026
I was going to suggest something similar, but with different semantics. As in the SP has to drain the removal queue to get the proving period assigned. The
nextProvingPeriodwould have to return a flag meaning "call me again".ZenGround0 commentedon Jul 13, 2026
This does seem pretty clean. I'm leaning in this direction.
TippyFlitsUK commentedon Jul 22, 2026
Second stuck data set: 1187 (mainnet, 976 pieces, ~35 GiB) with 386 queued removals. Small set, so not gas.
eth_callonnextProvingPeriod:The
PiecesRemovedevent blows the FVM event cap, so the real threshold is ~250 removals per flush regardless of gas or set size. Same silent presentation as 1245; curio walked it to 4 of 5 failures before we caught it.Happy to test against 1187 when there's something deployable.
rvagg commentedon Jul 23, 2026
filecoin-project/curio#1372; I'm not sure whether the original curio design intended to cap at 200 total or 200 on-chain plus 200 incoming, maybe it should have been 200 total in which case the errant 386 shouldn't have got through in the first place.
Kubuxu commentedon Jul 23, 2026
How do we end up with 12000 events for 400 removals? That is a 30x multiplier.
rvagg commentedon Jul 24, 2026
12480 > 8192is bytes, not events.386 uint256's = 12,352 bytes plus overhead. So we're hitting a single event ceiling here that's just above 250 pieces.
On top of this there's also some fixed size events that can be hit in the same tx:
DataSetEmpty,NextProvingPeriod, then in FWSSFaultRecordif there were faults, then there's even a bunch of FP events on top of that:AccountLockupSettled/RailRateModified/RailLockupModified/RailOneTimePaymentProcessed.~easy initial fix is to batch
PiecesRemovedto something like 100. That lifts that particular limit.rvagg commentedon Jul 24, 2026
argh, it's actually even worse. Current foc-observer numbers show a 132M gas per removal; that makes for a hard cap of ~70 removals per
nextProvingPeriod, not 250, not even the 200 Curio is encoding. Unfortunately our state growth is hurting us by growing this gas cost of ~5% per month so we'll be down to 50 removals per nextPP in a year if we don't deal with gas.We'd better fix this fairly urgently. Even if you did 70 removals for a nextPP then you're spending over 9B gas to do it.
TippyFlitsUK commentedon Aug 7, 2026
Quick update from the operator side after upgrading our mainnet SP to Curio v1.28.3.
Data set 1187 is still wedged as described above: 386 scheduled removals,
nextProvingPeriodreverting withtotal event value lengths exceeded the max size: 12480 > 8192, last successful proof 2 July. Five-plus weeks unproven now.One thing has changed that raises the urgency for other SPs. Curio v1.28.3 removed the per-dataset proving backoff, so this failure is no longer swallowed silently - the task now fails and logs the full revert every couple of minutes. Any SP carrying a wedged set will start seeing it loudly the moment they upgrade to v1.28.3. The invisibility that let this sit unnoticed for weeks on our node is gone, which is good, but it also means more operators are about to find sets in this state.
nalacateria commentedon Aug 13, 2026
@ZenGround0 @LexLuthr - can you give a status update on this? This is affecting production SPs with significant error messages.
ZenGround0 commentedon Aug 17, 2026
Status update:
The bright side is that we are handling these datasets more correctly now -- they are not getting silently deleted but are instead loudly complaining to the SP that they are in a bad state and need remediation.
ZenGround0 commentedon Aug 17, 2026
@snadrus and I did a search through better stack and I'm actually only seeing this issue present itself on Tippy's dataset 1187. For this reason we are not planning to do any emergency curio releases with #1439 and will focus our efforts on getting the pdp protocol update fixing the root cause issue all the way through the stack
TippyFlitsUK commentedon Aug 18, 2026
Hey, Zen 👋
Careful with the Better Stack read - it only covers SPs shipping logs there, and pre-v1.28.3 nodes swallow this revert silently, so a quiet search doesn't mean quiet chain.
I swept
getScheduledRemovalsacross all 796 live mainnet data sets this morning: 48 carry a queue, 24 are above the ~70 per flush gas ceiling, 15 above the ~250 event cap. Set 245 is sitting on a full 2,000. Those sets can never resume proving as things stand.Also worth knowing 1187 disappears as a sample soon - I terminated it on 10 Aug and it deletes after epoch 6,355,075 (~9 Sep).
Many thanks, buddy 🙏
ZenGround0 commentedon Aug 18, 2026
In that case can you please link me to reports of SPs dealing with this problem? We are already working to fix this at the root. If this is causing SPs enough trouble to report an issue then we can consider shipping an emergency release.
TippyFlitsUK commentedon Aug 18, 2026
I'm fully aligned with your root-fix-focus suggestion above. I don't think we need to worry about an emergency release here.
The datasets I turned up are on nodes that went dark back in March and April, and 1187 is my own and already on its way out. 🚀