Repository navigation
fix: store justifications data in state transition - #22
Merged
Merged
Conversation
MegaRedHand
added a commit
that referenced
this pull request
Jan 13, 2026
This reverts commit 1ca257e.
tcoratger
pushed a commit
to tcoratger/ethlambda
that referenced
this pull request
Oct 30, 2026
…aclass#22) A mainnet follower spent 23.5% of its CPU deriving the same committees over and over. An Electra attestation names up to MAX_COMMITTEES_PER_SLOT committees, and every one of them was derived from scratch: a scan of the 2.4M-entry validator registry, then a SHUFFLE_ROUND_COUNT-round shuffle *per member* of the committee. A block carries up to MAX_ATTESTATIONS_ELECTRA of those, `process_attestation` walks the committees a second time to check the aggregation-bit lengths, and fork choice walks them a third time when it replays the block's attestations into the latest-message store. Nothing was shared between any of it. Two changes make it one derivation per epoch: `compute_shuffled_indices` computes a whole epoch's permutation in one pass, sharing each round's pivot and window hashes across every position instead of rehashing them per position. `EpochCommittees` stores the active set through that permutation, so a committee becomes a slice of it: no hashing and no allocation per lookup. Ported from feat/mainnet-network, which wrote the same function against a per-(seed, index_count) memo. `CommitteeCache` then keeps those shufflings across calls, keyed by epoch and the block root that last could have changed them: the last slot of epoch E - 2, which is lighthouse's AttestationShufflingId. An epoch's committees depend only on its active set and its seed, the seed is fixed once E - 2 ends, and every write to activation_epoch or exit_epoch lands at least MAX_SEED_LOOKAHEAD epochs out, so two states agreeing on that root agree on the committees whatever else they disagree on. That is what makes sharing sound where a key of epoch or seed alone would not be, since sibling branches can share both while disagreeing on who is active. The cache is owned by the chain actor and threaded down through fork choice and the state transition, rather than kept in a global: how many shufflings are worth keeping resident, at ~19 MB each on mainnet, is the owner's decision and not something a leaf helper can answer. A caller with no cache of its own passes a fresh one and gets exactly the old behaviour, which is what every fixture runner does. Per attestation at mainnet's ~2.4M validators, naming a full slot's 64 committees: 6.75M hashes and 64 registry scans before, 844k hashes and one scan on a cold epoch, none of either on a warm one. `get_beacon_committee` keeps its signature and still builds a fresh `EpochCommittees` per call, so a single-committee caller is unaffected.
tcoratger
pushed a commit
to tcoratger/ethlambda
that referenced
this pull request
Oct 30, 2026
…ambdaclass#25) * feat(types): write beacon scalars the way the Beacon API reads them * refactor(types): serialize beacon scalars without allocating * docs(types): correct the hex_array allocation rationale The paragraph was adapted from quoted_or_bare's and kept its example: no Validator field routes through hex_array, so the per-validator cost it claimed does not exist. The fixed-width byte types it serves appear a handful of times per state. * feat(types): add the collection and bitfield serde adapters * refactor(types): make a mis-annotated ssz_hex field a compile error The adapter took any SszEncode, so annotating a container field with it would compile and emit an opaque hex blob where a JSON object belongs. The guard test planned for the container rollout walks for bare numbers and cannot see that, so nothing else would catch it across 84 hand annotations. Bound it to the bitfield and byte-list families instead, and pin the bitfield encodings the tests only sampled by prefix. * feat(types): serialize the beacon byte newtypes and uint256 Gives the five byte newtypes and U256 their own Serialize, so container fields holding them need no attribute at all in the annotation pass. U256 writes decimal, not hex, which the Beacon API requires and which no primitive integer can hold: it goes through long division over the big-endian bytes. * feat(types): serialize the shared and phase0 containers as JSON Annotates the first 28 of the 84 containers and adds the guard test the rest of the pass is verified by: it walks a serialized block and fails on any bare JSON number, naming the field path, since a forgotten attribute is valid JSON and silently wrong. Drops the SszHexEncodable marker trait added alongside ssz_hex: the adapter takes any SszEncode again, so an ssz_hex annotation on a non-bitfield field is a hand-checked invariant rather than a compile error. * test(types): make the interior-zero-byte case actually interior The test set only byte 17, so every nonzero byte sat at one end and the docstring's claim of nonzero bytes above and below the zero run was not true of the value being asserted. Set the lowest byte too, so sixteen zero bytes sit between two nonzero ones and the carry genuinely has to cross them. * feat(types): serialize the altair containers as JSON Altair's additions need the three adapters told apart carefully: the sync committee bits are a bitvector and go out as hex, the participation lists are ParticipationFlags, a u8 alias, and go out as quoted integers, and the committee pubkeys are a vector of a type that serializes itself. * test(types): make the JSON guard reach what it claimed to The guard built its block from BeaconBlockBody::default(), so every list serialized as [] and the walker never descended into an element type, and no test constructed a BeaconState at all. It covered 4 of 28 structs. Populates every list with real elements, one level into nested lists, and adds a state test per fork, taking the reach to 20/28 for phase0+shared and 6/11 for altair; the rest are gossip-only types no block or state can contain. Also adds a test that the walker fails on an unannotated integer, so a green suite cannot be confused with a walker that matches nothing. * feat(types): serialize the bellatrix containers as JSON Bellatrix brings the execution payload, and with it the first fields the adapter table does not answer on its own. extra_data and logs_bloom are byte strings that must go out as one hex string each, though both would also compile as arrays of numbers. transactions is a sequence of byte lists, which no single adapter produces: it composes ssz_hex per element through seq, locally, until a second fork needs the same shape. * feat(types): serialize capella, and share the byte-string sequence adapter Capella imports bellatrix's own Transactions type, so the composition bellatrix kept private became a second call site. Promotes it to serde_helpers as ssz_hex_seq and repoints bellatrix at it, which is a net 44 lines out of that file. The bound is Deref<Target: SszEncode> rather than SszEncode: serde hands serialize_with a reference, and unlike Display and Serialize, SszEncode has no blanket impl for &T. * feat(types): serialize the deneb containers as JSON The blob commitments are the field worth naming: a sequence of KzgCommitment, which is a byte newtype that already writes itself as hex, so it takes seq rather than ssz_hex_seq. Both spellings emit an array of hex strings, so the guard test cannot tell them apart and the choice has to come from the element type. * feat(types): serialize the electra and fulu containers as JSON Completes the annotation pass: all 84 containers now derive Serialize. Fulu is where both sequence-of-hex spellings appear side by side. The data column is a list of Cell, a foreign byte vector with no Serialize of its own, so it takes ssz_hex_seq; the commitments and proofs beside it are byte newtypes that write themselves, so they take seq. A single Cell, as in MatrixEntry, is neither and takes ssz_hex. Electra turns out to import its execution payload from deneb, and fulu has no block types at all, so their fixtures reuse rather than duplicate. * feat(types): serialize lean signed blocks as JSON Lets /lean/v0/blocks/finalized answer JSON when a caller asks for it. Integers stay bare: that is lean's own encoding and the rpc crate's tests assert it. The proof blob takes the 0x-prefixed convention the roots and bitlists use, rather than the prefix-less one the fixed-width XMSS pubkeys use, since it is opaque variable-length data and not an identity. * feat(types): serialize the container enums Neither enum adds a variant tag: the Beacon API carries the fork in the response envelope's version field and its Eth-Consensus-Version header, never inside the data object. The two get there differently, and the asymmetry is the point. The block enum derives untagged, because its Lean variant really is served as JSON on /lean/v0/blocks/finalized. The state enum cannot: a derive would demand a JSON encoding for lean's State, which is deliberately SSZ-only, so it is hand-written and the lean variant returns a serde error instead. * feat(rpc): negotiate response encoding on Accept The Beacon API serves JSON unless the caller asks for application/octet-stream, which is the order lighthouse serves too. The lean surface defaults the other way on its two SSZ endpoints, so the default stays the caller's to pass rather than a constant here. Ranks the header's entries by their q weight: a client that accepts both states its preference that way, and ignoring it would serve SSZ to a caller who merely tolerates it. base.rs's private ssz_response moves here rather than being duplicated, since both surfaces now need it. * feat(types): report a block's body root on either chain /eth/v1/beacon/headers/{id} answers with a SignedBeaconBlockHeader, whose body_root is the one field the existing accessors cannot produce: signed_beacon_block_accessors! hands back a field verbatim, and every fork stores a different body container, so what the caller wants is the merkle root of whichever one this is. Dispatched including lean, since lean's Block has a body too and /lean/v0 wants the same number. The phase0 test computes the expected root through the raw libssz_merkle trait rather than this crate's convenience wrapper, so a body_root that only worked on the lean arm would fail rather than pass by construction. * feat(rpc): parse and resolve beacon block ids Parsing is chain-agnostic, resolution is not. resolve_beacon reads only accessors that answer on either chain: Store::head_state and head_slot panic on a beacon store, so the lean slot lookup stays where it is rather than being repointed at BlockRoots, which covers the branch ending at the head rather than everything a lean store was bootstrapped with. genesis resolves to the store's anchor rather than to slot 0: a checkpoint-synced directory has no genesis block, and its anchor is the earliest block it can answer for at all. * feat(types): serialize the beacon chain config as JSON /eth/v1/config/spec echoes the chain configuration in the Beacon API's encoding: SCREAMING_SNAKE_CASE keys, every integer a quoted decimal, byte strings 0x-prefixed hex. Config could only be deserialized, which is how --network <dir> loads a network. 74 fields move from deserialize_with = "m::deserialize" to with = "m", which resolves to the same function on read, so the existing YAML parsing tests staying green is the evidence that nothing about loading a network moved. Six fields keep their own arrangement: genesis_time stays skipped since it is not a config.yaml key and /eth/v1/beacon/genesis reports it; terminal_total_difficulty and terminal_block_hash already have Serialize on their own types; max_blobs_per_block_deneb keeps its rename; and blob_schedule pairs its hand-written deserializer with seq::serialize, since its two halves do not live in one module. * feat(rpc): add the beacon API envelope and error shape Every fork-versioned Beacon API payload travels in {version, execution_optimistic, finalized, data}. The fork sits in the envelope rather than in the data because the containers serialize untagged, so version is how a caller knows which shape it just parsed. The error body is {code, message}, deliberately not the lean surface's {error}. They are two different APIs and each consumer parses the shape its own specification names, so unifying them would break one of the two. The submodule declarations and routes() land with the files they name, rather than being written now and commented out. * feat(rpc): serve beacon blocks by id in JSON and SSZ /eth/v2/beacon/blocks/{block_id} and its /root sibling, JSON by default and SSZ on Accept: application/octet-stream, both tagged with Eth-Consensus-Version so an SSZ caller can tell which fork's container it received. genesis is now refused rather than resolved to the anchor slot. Table::BlockRoots is written only by update_checkpoints, which computes its delta by walking from the old head to the new one and returns early when they are the same root. At bootstrap they always are, so the anchor's own slot is never indexed and the previous spelling could only ever 404. Saying so is better than a lookup that silently never works. The fixture moves its head through update_checkpoints rather than poking the backend, so BlockRoots is populated by its real writer and the slot-resolution tests exercise the path a running node takes. get_block_root loads the block rather than only resolving the id: it needs the slot to answer finalized honestly instead of hardcoding it. * feat(rpc): serve beacon block headers /eth/v1/beacon/headers/{block_id}. Every field of the message comes from SignedBeaconBlock's accessors, which dispatch including lean, plus body_root(); only the envelope and the signature are beacon-specific, which is what makes this the one endpoint genuinely shared with the lean surface. canonical is asked of BlockRoots rather than assumed, even for a block reached by its own root, so a sibling or a block at a slot the index does not cover answers false instead of being taken on trust. The test pins both directions: the head is canonical, the anchor is not, since its slot is never indexed. The quoting comes from ethlambda_types::beacon::serde_helpers rather than a to_string() per field, so the rule lives in one place. * feat(rpc): serve beacon states and finality checkpoints /eth/v2/debug/beacon/states/{state_id} is the path checkpoint_sync.rs already fetches from other clients, so serving it makes this client checkpoint-syncable from itself. The SSZ test decodes the body back through slot_from_ssz, since round-tripping is the whole job. A 0x... state_id is refused with a message naming the ids that do work. States are keyed by block root here with no reverse index, so the alternative to refusing is answering with the wrong state, or being right only when the two roots happen to coincide. finality_checkpoints reports the state's own three checkpoints rather than the store's fork-choice view: the endpoint is defined as a read of the state the id names, and the state is the only place a previous justified checkpoint is kept at all. JSON serializes through the Arc rather than cloning, since a mainnet state runs to hundreds of megabytes. * feat(rpc): serve beacon genesis and the spec config /eth/v1/beacon/genesis reads genesis_validators_root off the finalized anchor's state rather than looking for slot 0: it is a property of the chain that every state carries, and a checkpoint-synced directory has no genesis block to read it from. /eth/v1/config/spec echoes the Config the store was bootstrapped with, so a node started with --network <dir> reports that network's constants rather than mainnet's. GENESIS_TIME is absent from it by design, since it is not a config.yaml key and the genesis endpoint reports it. * feat(rpc): serve the beacon node endpoints syncing reads beacon_head rather than head_slot, which is lean-only and panics on a beacon store, and computes the wall slot in milliseconds the way /lean/v0/node/syncing does: slot_duration_ms is the value a loaded network can actually change, and seconds_per_slot would truncate a sub-second cadence to zero. health is 206 while syncing and 200 once caught up. 503 would mean uninitialized, and a store that answers at all is initialized. identity reports peer_id and metadata only. enr, p2p_addresses and discovery_addresses are empty, which is not spec-valid: the ENR is built for discv5 and owned by the P2P actor, and BuiltSwarm hands run_node only a local_peer_id, so serving the record means widening the ethlambda-p2p surface. Deliberate and recorded as a deviation. * feat(rpc): serve the beacon API from a beacon node Which HTTP surface a node serves now follows from the store's own chain tag rather than from the sub-command, so the two cannot disagree. Until now a beacon node bound --api-port to the lean router off a beacon store. Those handlers reach for lean state variants and metadata keys a beacon directory never carries, so calling one panicked that request. The beacon router replaces it rather than extending it, and the integration test pins both halves: /eth/v1/node/version answers, /lean/v0/node/syncing is a 404. build_beacon_api_router does not merge the metrics router, for the same reason build_api_router does not: start_http_servers serves it, and merging a path twice makes axum panic at startup. start_beacon_rpc_server takes no AggregatorController and no EventBus. A follower has no aggregator duty to toggle, and the chain-events stream is part of the lean surface. * feat(rpc): serve the lean finalized block as JSON on request SSZ stays the default here, which is the opposite of the beacon surface and deliberately so: checkpoint_sync.rs reads these bytes, and other clients' lean checkpoint sync may send no Accept header at all. Moving the default would break every one of them, so JSON is opt-in. Not routed through Encoding::from_accept, which defaults to JSON. The asymmetry is the point, so the branch is spelled out here rather than hidden behind a shared name that means the other thing. The existing SSZ test already pinned the default; the new test pins JSON on request, and that lean integers stay bare rather than picking up the beacon surface's quoting. * docs: describe the beacon API surface and its deviations docs/rpc.md gains the endpoint table, the encoding rules (JSON by default, SSZ on Accept, quoted integers, Eth-Consensus-Version) and the three ids that are refused. docs/spec_deviations.md records two: the partial node identity, and the ids that cannot be named. The second is a property of Table::BlockRoots rather than a choice, so it is written down where the reason lives. CLAUDE.md said a /lean/v0 call on a beacon node panics the request and that giving the follower its own surface was still a change of its own. Both stopped being true with this branch. * fix(rpc): resolve the anchor's own slot, which checkpoint sync asks for Found by running a mainnet follower and pointing a second one at its API: the second died with "peer served no block at the anchor slot 15265888" and never started. checkpoint_sync.rs reads a peer's finalized state, takes the anchor slot from it, and asks that peer for /eth/v2/beacon/blocks/{slot}. That is exactly the one slot Table::BlockRoots never holds, because update_checkpoints is its only writer and writes nothing when the head does not move, which is the situation at bootstrap. So the endpoint 404'd and this node was unusable as a checkpoint-sync source for any client, not only for itself. On an index miss the lookup now tries the roots the store can name and accepts one only when block_entry confirms its block really sits at the slot asked for, so a slot the store holds nothing at still 404s and no request can be answered with the nearest checkpoint instead. The test that pinned the old 404 asserted a bug, so it now asserts the anchor resolves, with a sibling holding the line on slots below it. The docs claimed a slot at or below the anchor was refused; only genesis and a state-root id are. * refactor(blockchain): move the empty-custody check to the node that configures one The per-block fence in data_availability_for cannot distinguish a misconfigured node from the replay harness, which legitimately supplies every block itself and has no columns to wait on. A node is checked once, at startup, where the custody set is computed. * feat(blockchain): expose an unspawned import entry for offline replay on_block already takes &mut self and no Context, and returns the import verdict, so a caller can drive the real import path without a mailbox. Two public items rather than a benchmark module inside this crate. * feat(cli): add the import benchmark's corpus manifest An SSZ corpus rather than a prepared store directory, so it stays readable and portable across revisions of the store format. Empty slots are recorded as gaps in the manifest rather than treated as a truncated range. * feat(cli): resolve a corpus anchor across an empty slot The state at from-1 is slot-advanced and keyed to no block when that slot is empty, so walk the first block's parent link instead. Written against a trait so the rule is tested without a live server. * feat(cli): stream a beacon block range into an SSZ corpus One state is held at a time and blocks are written as each response lands, so the footprint does not scale with the range and the range needs no cap. * refactor(cli): split the benchmark report by workload A pure move: the shared statistics go to report::common, today's report to report::synthetic, schema_version unchanged. * feat(cli): add the import workload's report and an at-most-once phase timer An import runs a subset of BLOCK_IMPORT_PHASES, so an unobserved phase is omitted rather than recorded as a zero that reads as instant work. * feat(cli): replay a corpus through the real import path Drives BlockChainServer::import_block directly: no mailbox, no tick loop, no p2p, and the clock placed per block from the store's own config so a run is reproducible. A block that does not import aborts the run, since every later block descends from it. * feat(cli): add benchmark import fetch and replay Wires the two phases to the command line and drops the dead_code allows the earlier slices needed while nothing reached this code. * docs: describe the import benchmark workload Records why the corpus exists: a live-follower A/B costs a restart and a moving denominator per leg, and the work being measured is deterministic. * fix(blockchain): publish a replayed import's sections under their own source import_block left ImportTimings.source unset to keep replayed blocks out of the gossip and sync percentiles, but BlockImportReport::observe publishes nothing for a sourceless import. The replay harness reads its per-phase numbers back from exactly those observations, so every phase came back absent and a mainnet replay reported wall time alone. BlockSource::Replay gives them a label no node ever writes: they stay out of the wire sources' percentiles and are still published. * fix(cli): make the import benchmark work against a live mainnet node Run against ethlambda-5's mainnet follower, a corpus whose first block's parent sat mid-epoch failed on that first block with `root in store.blocks`. get_forkchoice_store finalizes (epoch(anchor), anchor_root), and every import walks back to that epoch's first slot, which lies below a mid-epoch anchor. The spec fixtures anchor at genesis, so the tests never reached it. - fetch anchors on the block at the first slot of the epoch holding from - 1, stepping back past empty first slots, and records the blocks up to --from as warm-up that replay imports without sampling. replay refuses a corpus anchored anywhere else and says to re-fetch it. - A 404 means an empty slot, a slot past the head, or one before the source's history. fetch now refuses a --to past the head and checks every block's parent link, so a hole or a reorg mid-fetch stops the fetch instead of failing a replay minutes later. - A failed fetch removes the corpus directory it created. - Progress goes to stderr. fetch's tracing lines sat below the benchmark's WARN filter, and replay printed nothing for a mainnet block's seconds. - The report also samples prune, get_head and fcu, prints its inclusive range as [a, b], and drops the coefficient-of-variation warning: it named a flag replay does not have, and across a corpus it measures how blocks differ, not noise. * fix(blockchain): initialize the committee cache in for_replay Merging beacon-chain-integration brought in lambdaclass#22, which added a `committees` field to `BlockChainServer`. The text merge was clean, but `for_replay` builds the struct by hand and never set the new field, so the branch stopped compiling (E0063). Replay starts from an empty cache, the same as a freshly spawned node. * chore(docs): drop the beacon API spec and plans from the branch They are working notes from the design sessions, not documentation, and were swept into bfdbdc8c alongside an unrelated rpc fix. Nothing links to them. They stay recoverable from that commit. * refactor(cli): stop reading the state cache's occupancy from the benchmark The replay read the store's LRU length to report peak_states_cached and to assert that a long range stays within STATE_CACHE_CAPACITY. An LRU cannot exceed its capacity by construction, and the storage crate's own the_state_cache_is_bounded already pins the cap, so the number could never show anything that test does not. It also cannot see what the replay module doc warns about: an Arc<BeaconState> pinned outside the cache leaves len() unchanged. That is not worth a benchmark-only accessor on Store. * test(blockchain): delete the beacon_replay integration test The benchmark's replay tests in bin/ethlambda already drive BlockChainServer::for_replay and import_block end to end, and this one read consensus-spec-tests fixtures the Test (consensus) job never downloads, so it failed on every CI run. Its snap and serde_yaml_ng dev-dependencies go with it; nothing else in the crate uses them. * test(cli): delete the import benchmark's fixture-backed replay tests They read a consensus-spec-tests case the Test (node) job never downloads, so they failed on every CI run. Two of the five also asserted less than their names claim: the block roots they compared are hashed from the corpus files, so "the head advances" and "two runs agree" held whenever both replays finished. The spec-case corpus builder, ReplayOptions::for_tests and the snap dev-dependency only served these tests and go with them.
tcoratger
pushed a commit
to tcoratger/ethlambda
that referenced
this pull request
Oct 30, 2026
…nned eviction (lambdaclass#33) * fix(beacon): harden the committee cache and share it where it was missing Review of lambdaclass#22 found gaps between what the cache's documentation promised and what the code did. - Committee numbers use checked arithmetic. Fork choice's gossip path never bounds `data.index`, so a release build wrapped a hostile index into a real committee where the specification raises. - `EpochCommittees::committee` rejects a slot outside the epoch it was built for. Callers now name that epoch separately and share the result through the cache, so a mismatch would have sliced the wrong shuffle. - Epochs 0 and 1 are keyed on the genesis block, as lighthouse does, instead of rebuilding a whole-epoch shuffle on every lookup. - `get_beacon_committee` is back to the per-member derivation, so a single-committee caller pays what it did before this PR, and the tests holding the sliced form to it compare two independent derivations. - Phase0 epoch processing shares one shuffling per call rather than one whole-epoch shuffle per attestation, and the inclusion-delay deltas derive each attestation's attesters once rather than once per attester. - Pre-Electra `process_attestation` reads the committee count off the cached shuffling instead of scanning the registry per attestation, and reuses the indexed attestation's indices as Electra already did. - The fork-choice and sync fixture runners hold one cache per case, as the chain actor does across imports, so the fixtures' sibling branches test the cache key instead of passing whatever it does. * perf(beacon): shuffle in place and pin the head's shufflings in the committee cache The whole-list shuffle visited every position in every round, each visit a random lookup into that round's window hashes plus a 64-bit `% n`. Lighthouse's `shuffle_list` (protolambda's algorithm, which Prysm and Teku also use) walks each round's swap pairs instead: one decision per pair, in order, rehashing only on crossing into a new window. It shuffles the active set in place, so a build holds one validator-sized list instead of three. Benchmarked standalone on an M3 Pro at 2.4M validators: 1387 ms to 355 ms, peak heap 57.6 MB to 19.2 MB. The cache held two shufflings, exactly one block's worth, so two branches imported in turn evicted each other's entry on every import and rebuilt it on the next. It now holds `COMMITTEE_CACHE_CAPACITY` entries and evicts the way lighthouse does: oldest epoch first, never the canonical head's previous, current, or next shuffling. The chain actor pins those whenever fork choice moves the head, reading the head's post-state from the store's state cache only, so pinning never reconstructs a state on the import thread. `lean_beacon_committee_cache_lookups_total{result}` counts hits, misses, and unkeyable lookups, so the capacity can be checked against live forks.
tcoratger
pushed a commit
to tcoratger/ethlambda
that referenced
this pull request
Oct 30, 2026
* feat(beacon): judge a block by the rules that need no post-state Beacon gossip validation needs the `beacon_block` rules that can be applied before a block's own state transition runs, and the chain actor will want the same rules before it parks a block. They live in one place, `precheck::precheck_block`: a slot after its parent's, the proposer fulu's lookahead already fixed for that slot, and a valid proposer signature. A block whose parent has no post-state yet is judged against a recent state, for the signature alone. `bls::DST` is public and `test_state` gains `secret_key_for`, so a test can sign a block these rules accept. * feat(types): expose a beacon block's blob count and payload timestamp The beacon_block gossip rules bound the commitment count and check the payload timestamp before any state is read; both fields sit behind the fork enum. * feat(beacon): gossip verdict types and seen caches Gossip validation needs a verdict that says both what gossipsub does and whether the chain still wants the object (the spec's MAY-queue cases); fixed reason enums keep metric labels bounded. * build: fetch the spec's gossip validation vectors They first ship in v1.7.0-beta.1, so they come from a tree of their own instead of moving the main fixture pin. Only the fulu block and column handlers are extracted, since those are the topics this node validates. * feat(beacon): beacon_block gossip rules Split by cost so the p2p actor can run the state-free half inline and the rest off its loop; reuses precheck for the proposer and signature rules and reads states only from the cache. * test(beacon): run the spec's beacon_block gossip vectors One vector is skipped by design (a parent without a post-state is queued until a bad-block cache exists); everything else must match. * feat(beacon): data_column_sidecar gossip rules The proposer check reads the parent's lookahead instead of advancing a cloned state, so it is cheap enough for the gossip path; slots outside the window are queued as the spec allows. * test(beacon): run the spec's data_column_sidecar gossip vectors reject_parent_failed_validation is skipped for the same reason as the block runner's parent-not-verified vector: without a bad-block cache we cannot tell a failed parent from one still importing, so it is queued rather than rejected. * fix(beacon): judge the lookahead last and never reject on a failed ancestor walk Code-review fixes to the beacon gossip validation rules: - Block `stateful_checks` ran `precheck::fixed_proposer(..).is_none()` before `precheck_block`, the finalized-ancestry check and the payload-timestamp check. A block whose slot the lookahead cannot place at all (before the parent's own epoch, or a pre-fulu parent) short-circuited into `Queue(ShufflingUnavailable)` after only a head-state signature check, turning spec REJECTs (slot not after parent, finalized not an ancestor, bad timestamp) into IGNOREs. The lookahead gate now runs last, after `precheck_block` already checked slot order, known proposer and signature against the parent's own state. - `descends_from_finalized` collapsed a failed ancestor walk into "not an ancestor", but `fork_choice::get_checkpoint_block` erroring means a row is missing from `LiveChain` (for example after `invalidate_subtree` deletes a late-invalidated parent's rows while its cached state stays), not that the chain has forked away from the finalized checkpoint. Rejecting such a block penalizes peers who haven't yet learned of the invalidation. Replaced the bool with a three-way `FinalizedAncestry` (`Descends` / `Conflicts` / `Unknown`), used by both `gossip::block` and `gossip::column`; `Unknown` queues (`ParentNotReady`) instead of rejecting. - `column::verify_header_proposer` checked the lookahead before the signature, so a slot the lookahead cannot place returned `Queue(ShufflingUnavailable)` before ever checking the signature, even though the signature does not depend on the shuffling. The signature (and the known-proposer check) now run unconditionally; only the proposer-identity question is deferred to a `Queue` when the lookahead has no answer. Tests now cover the head-state signature path for an unknown parent, the lookahead queue path once the parent's own checks have run, and the regression case (a slot before the parent's own lookahead window used to mask a not-after-parent violation as a queue). The spec runner's config fallback now covers every fork up to the case's own rather than hardcoding fulu, and its comment cites the vectors' own README convention instead of a claim the design spec doesn't make. Clock-disparity magic numbers in the gossip unit tests are now derived from `MAXIMUM_GOSSIP_CLOCK_DISPARITY`. * feat(metrics): count beacon gossip verdicts and late verdicts Validation now gates propagation, so how often each rule fires, how long a verdict takes, and whether it beat gossipsub's cache eviction are the three things to watch. * feat(p2p): hold beacon gossip until it has a verdict Gossipsub forwarded every beacon message before anything checked it; lean keeps auto-forwarding because its handlers produce no verdicts. Beacon gossip stops propagating until the verdict plumbing lands; do not deploy this commit alone. * fix(beacon): check a queued column's header signature, and test every rule A queued column is written to the chain actor's PendingDataColumns the same way a queued block is, but only blocks got a signature check before being parked: a forged block is rejected up front, a forged column just sat there. Give columns the same treatment on both queue paths that precede the parent's own state (parent unknown, parent known but uncached), reusing the head-state check block.rs already has. Where the parent's state *is* already in hand but the finalized-ancestry walk itself fails, check the signature against that state instead of the head's, since it is the authoritative one. Also add the tests this rule and the surrounding ones were missing (one per outcome in the design's rule tables for both beacon_block and data_column_sidecar), fold the duplicated `store`/`seen`/`fulu_parent`/clock test helpers shared by block.rs and column.rs into a `test_support` module, and reword two doc comments whose sources had drifted from what they now claim. * feat(p2p): run stateful gossip checks off the actor loop The chain actor's mailbox waits reach seconds at p90 on the followers, past gossipsub's message-cache window, so verdicts come from bounded one-off blocking tasks that report back to the p2p actor instead. * feat(p2p): validate beacon gossip before propagating it Blocks and data column sidecars are judged by the fulu gossip rules and forwarded only on Accept; the other subscribed topics are ignored until they get validators. Undecodable payloads are now rejected instead of relayed. * refactor(p2p): decide a beacon gossip verdict before acting on it Each beacon gossip handler both decided a verdict and acted on it (calling verdict::report or verdict::spawn_stateful_checks inline), which meant the decision could only be exercised through a running actor's Context and never in a unit test. Split the two: the handlers now return a Dispatch describing what to do, and handle_beacon_gossip is the single place that reports or spawns. That single call site is also where a future cheap check that answered Queue would need to forward its object; today no cheap check does, so Report carries no object, and a debug_assert! catches the invariant instead of widening the type for a case that cannot happen yet. The verdict-time seen-cache re-check (first Accept wins) moves out of the actor's Handler impl into its own settle function for the same reason: it needs a &mut P2PServer, not a Context. Both the triage decisions and the settle re-check now have unit tests. A caught validation panic is also logged, since Ignore(Internal) in the metrics previously left no trace of what panicked. * fix(beacon): keep overloaded gossip, and check a column's signature first With every validation permit taken, a message was ignored and dropped. Gossipsub's duplicate cache then hides every other peer's copy, so the chain actor learned of the block only through a child's by-root fetch or range sync. It is still ignored, but now also handed to the chain actor, which judges it itself as it did before gossip validation existed. A column's header signature is now checked right after its parent's state is found, before the finalized-ancestry walk, the inclusion proof and KZG, so a forged header costs one BLS check rather than all three while holding a permit. The lookahead is judged last, as for blocks. The already-stored check read every stored sibling sidecar inline in the p2p actor; it is now a point lookup (`Store::has_data_column`). The p2p test helper ignored its config, so a triage test passed for a different reason than its comment gave; it now honours it. * docs: describe beacon gossip validation Every beacon message now waits for a verdict, so the wire guide, the metrics reference and the chain actor's notes on parked sidecars said things that are no longer true: that columns are deduplicated by a finality-pruned set, that fabricated headers park freely from gossip, and that the import's decode phase ends at the decode. * test(beacon): pass the gossip runner's on_block a committee cache lambdaclass#22 added the parameter, and the merge left the gossip conformance runner calling the old signature, so the spec-test binary no longer compiled. A fresh cache per call, as the fork-choice runner does. * build: derive the gossip fixture configs from the main tree's Each run reads its own preset from both fixture trees, so the two config lists always named the same presets and CI had to set both to agree. The gossip list now follows CONSENSUS_SPEC_TESTS_CONFIGS minus `general`, which has no `networking` runner to extract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.