Repository navigation
feat: add metrics skeleton with lean_head_slot gauge - #10
Merged
Merged
Conversation
MegaRedHand
marked this pull request as ready for review
January 8, 2026 18:56
tcoratger
pushed a commit
to tcoratger/ethlambda
that referenced
this pull request
Oct 30, 2026
… per seat (lambdaclass#10) * perf(beacon): stop rescanning the validator registry per attester and per seat A mainnet follower on this branch could not import a single block. It sat at 100% CPU for over ten minutes per block and never finished one, which also froze the actor's ticks, so `store.time()` stopped advancing and head recomputation never ran. Four places asked an O(registry size) question inside a loop whose iteration count also grows with the registry or the committee: - `get_flag_index_deltas` called `get_base_reward` per eligible validator, and that helper's second factor is `get_base_reward_per_increment`, an unconditional registry scan. At 2.36M validators that is O(n squared), and it runs three times per epoch boundary. Measured at 2^15 validators: ~11.9s unhoisted against ~384us hoisted. - `process_attestation` did the same per attester. - `process_attestation` also called `get_committee_count_per_slot` and `get_beacon_committee` per named committee, two more scans each, for up to `MAX_COMMITTEES_PER_SLOT` committees per attestation. One `EpochCommittees` now shares the single scan their common `(state, epoch)` pair actually costs. - `process_attestation` recomputed `get_attesting_indices` for an attestation `get_indexed_attestation` had just resolved against the same unmutated state. - `process_sync_aggregate` resolved each of `SYNC_COMMITTEE_SIZE` seats with its own `state.validators().iter().position(...)`, roughly 1.2 billion 48-byte pubkey comparisons per block at mainnet scale. It now builds the seat map first and makes one pass over the registry, keyed pubkey to a list of seats because the committee is drawn with replacement. Every substitution is an equivalence, not a behaviour change, and both hoists keep the helper's own order of operations so the arithmetic stays bit-identical: these feed consensus rewards. The minimal-preset beacon spec suite (40009 cases) passes unchanged, which is what proves it. Measured on the mainnet follower, anchored around slot 15171200 at 2,361,891 validators: | build | per block | |------------------------|------------------| | before | no import in 600s| | the four hoists | ~12.4s | | plus the seat inversion | 8.11s median | 8.11s against a 12s slot is the difference between falling behind and catching up. `EpochCommittees` is ported from `feat/mainnet-network` without that branch's shuffle-permutation cache, which lives in a `shuffling.rs` change not taken here, so it shares the registry scan but still pays one shuffle per committee. Adding the cache is a drop-in change to its internals. * feat(beacon): measure what one fork-choice head computation costs Nothing timed `get_head`, so the only evidence that it was the follower's bottleneck came from attaching `perf` to a live node: 62% of the process's cycles sat in that call, over half of them inside `SipHash` on the block index's own keys, while imports ran at 14s against 12s slots. A histogram makes that visible from the dashboard instead. Buckets run to a whole mainnet slot because that is the question being asked: a head computation costing seconds is one the chain actor cannot afford between blocks. The guard is taken in `recompute_beacon_head`, ahead of the early return on failure, since a computation that fails still spent the time and the failing one was the expensive one. * perf(beacon): weigh the whole tree in one pass, not once per candidate `get_weight` is the specification's per-root definition: for one root it scans the whole validator registry and walks each voter's latest message back to that root's slot. `get_head` called it once per candidate child at every level of its descent, so a node with the justified checkpoint tens of slots behind head re-walked every vote tens of times to pick one head. At mainnet's 2.36M validators that dominated the follower outright. `perf` on the live node put 62% of all cycles in that descent, 54% of the total inside `SipHash` alone, hashing the block index's 32-byte keys once per voter per hop. Merkleizing the state twice per import, the previously suspected cost, was 17%. Imports took ~14s against 12s slots, so the follower lost ground every slot and could never converge. `compute_weights` produces every root's weight in one bottom-up fold, since a vote counts for a root exactly when the voted block descends from it: sum each vote at its own block, then fold each block's total into its parent, highest slot first so a child is complete before its parent reads it. Parent links always point at a strictly earlier slot, which makes that a valid topological order. One index lookup per voter over a map small enough to stay in cache, and no registry scan at all. Two behaviours worth naming. Membership is `is_active_validator` against the justified checkpoint's state rather than `get_active_validator_indices`, which allocates a two-million-entry vector to answer a membership question. And a vote for a block no longer indexed weighs nothing instead of raising: `promote_beacon_anchor` prunes below the oldest kept anchor, and a validator whose freshest vote is down there keeps it until it attests again. Such a vote cannot separate candidates above the justified checkpoint anyway, and the specification's version aborting the whole descent on it is what once pinned a live follower's head for an hour. `get_weight` stays exactly as the specification writes it, and is what the new pass is tested against, root by root, across a fork with the boost applied. The 151 beacon `fork_choice` fixture cases still pass. Ported from `feat/mainnet-network` (c09ad18c), where the same descent was found by the same method. * perf(beacon): hoist the base reward out of altair's and deneb's attestation loops `38e2945d` took `feat/mainnet-network`'s `2f3ef3e1` for electra only. Altair and deneb carry the same loop: `process_attestation` calls `get_base_reward` per attester per flag, and that helper's second factor is `get_base_reward_per_increment`, which reaches `get_total_active_balance`, scans the whole validator registry and allocates the active set on every call, with no cache anywhere beneath it. The read phase does not mutate `state`, so that factor is constant across the loop. It is hoisted once and `get_base_reward` inlined against it in the helper's own order of operations, so the arithmetic stays bit-identical: these feed consensus rewards. Nothing changes on the live mainnet follower, which checkpoint-syncs and so only ever transitions fulu blocks. This is for a node that replays altair or deneb history, and for keeping the three copies of this loop saying the same thing rather than leaving two of them as the version that could not finish a block at mainnet scale. Both presets pass unchanged: 5705 mainnet and 40009 minimal fixture cases. * perf(beacon): record a block's post-state root in the state, don't recompute it An import merkleizes a full state twice. `state_transition` hashes the post-state to check the block's committed `state_root`, and one slot later `process_slot` hashes that same state again to record it as the pre-state root. At mainnet's registry size each is over a second, so a third of an import goes on hashing bytes this node has already hashed and already checked. `process_slot` fills `latest_block_header.state_root` with exactly the value it just computed. So `on_block` fills it a slot early instead, from the block field `state_transition` accepted the block for matching, and the new `BeaconState::compute_state_root` takes it back out rather than merkleizing. The cached root is used only while the state is still inside that header's own slot, which is the shape a stored post-state has and the shape nothing the specification produces ever has, since the field is written on the way out of that slot. Every other state, fixture ones included, merkleizes as before. This is what lean has always done: `store::on_block` writes the root into the same field in the same position, and lean's `process_slots` reads it back the same way, minus the slot equality, which it does not need because it keeps no `state_roots` vector. Doing it in the state rather than beside it means the value survives a restart, needs no cache and no second lookup, and reaches `checkpoint_state` and `should_override_forkchoice_update` without either of them asking for it. `get_forkchoice_store` establishes the same field for the anchor, the one block that never goes through `state_transition`, by clearing it, merkleizing, and writing the result back rather than trusting whatever a checkpoint provider sent. `Store::init_store` normalizes a lean anchor the same way. The state stored under a block root now differs from the state that block commits to, by this field. Nothing reads it back expecting the committed shape; serving states over an API would be the change that makes it matter. Rather than assert the equivalence in a test of its own, the `sanity`, `finality`, `random` and `transition` runners now cache each block's state root for the next one, the way `on_block` does. Every multi-block case in both suites then drives the cached arm, and each one's `post` comparison is the assertion that the state comes out the same.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dashboard image: