Repository navigation
fix: add missing imports, references and clones - #25
Merged
Merged
Conversation
tcoratger
pushed a commit
to tcoratger/ethlambda
that referenced
this pull request
Oct 30, 2026
…ambdaclass#25) * feat(types): write beacon scalars the way the Beacon API reads them * refactor(types): serialize beacon scalars without allocating * docs(types): correct the hex_array allocation rationale The paragraph was adapted from quoted_or_bare's and kept its example: no Validator field routes through hex_array, so the per-validator cost it claimed does not exist. The fixed-width byte types it serves appear a handful of times per state. * feat(types): add the collection and bitfield serde adapters * refactor(types): make a mis-annotated ssz_hex field a compile error The adapter took any SszEncode, so annotating a container field with it would compile and emit an opaque hex blob where a JSON object belongs. The guard test planned for the container rollout walks for bare numbers and cannot see that, so nothing else would catch it across 84 hand annotations. Bound it to the bitfield and byte-list families instead, and pin the bitfield encodings the tests only sampled by prefix. * feat(types): serialize the beacon byte newtypes and uint256 Gives the five byte newtypes and U256 their own Serialize, so container fields holding them need no attribute at all in the annotation pass. U256 writes decimal, not hex, which the Beacon API requires and which no primitive integer can hold: it goes through long division over the big-endian bytes. * feat(types): serialize the shared and phase0 containers as JSON Annotates the first 28 of the 84 containers and adds the guard test the rest of the pass is verified by: it walks a serialized block and fails on any bare JSON number, naming the field path, since a forgotten attribute is valid JSON and silently wrong. Drops the SszHexEncodable marker trait added alongside ssz_hex: the adapter takes any SszEncode again, so an ssz_hex annotation on a non-bitfield field is a hand-checked invariant rather than a compile error. * test(types): make the interior-zero-byte case actually interior The test set only byte 17, so every nonzero byte sat at one end and the docstring's claim of nonzero bytes above and below the zero run was not true of the value being asserted. Set the lowest byte too, so sixteen zero bytes sit between two nonzero ones and the carry genuinely has to cross them. * feat(types): serialize the altair containers as JSON Altair's additions need the three adapters told apart carefully: the sync committee bits are a bitvector and go out as hex, the participation lists are ParticipationFlags, a u8 alias, and go out as quoted integers, and the committee pubkeys are a vector of a type that serializes itself. * test(types): make the JSON guard reach what it claimed to The guard built its block from BeaconBlockBody::default(), so every list serialized as [] and the walker never descended into an element type, and no test constructed a BeaconState at all. It covered 4 of 28 structs. Populates every list with real elements, one level into nested lists, and adds a state test per fork, taking the reach to 20/28 for phase0+shared and 6/11 for altair; the rest are gossip-only types no block or state can contain. Also adds a test that the walker fails on an unannotated integer, so a green suite cannot be confused with a walker that matches nothing. * feat(types): serialize the bellatrix containers as JSON Bellatrix brings the execution payload, and with it the first fields the adapter table does not answer on its own. extra_data and logs_bloom are byte strings that must go out as one hex string each, though both would also compile as arrays of numbers. transactions is a sequence of byte lists, which no single adapter produces: it composes ssz_hex per element through seq, locally, until a second fork needs the same shape. * feat(types): serialize capella, and share the byte-string sequence adapter Capella imports bellatrix's own Transactions type, so the composition bellatrix kept private became a second call site. Promotes it to serde_helpers as ssz_hex_seq and repoints bellatrix at it, which is a net 44 lines out of that file. The bound is Deref<Target: SszEncode> rather than SszEncode: serde hands serialize_with a reference, and unlike Display and Serialize, SszEncode has no blanket impl for &T. * feat(types): serialize the deneb containers as JSON The blob commitments are the field worth naming: a sequence of KzgCommitment, which is a byte newtype that already writes itself as hex, so it takes seq rather than ssz_hex_seq. Both spellings emit an array of hex strings, so the guard test cannot tell them apart and the choice has to come from the element type. * feat(types): serialize the electra and fulu containers as JSON Completes the annotation pass: all 84 containers now derive Serialize. Fulu is where both sequence-of-hex spellings appear side by side. The data column is a list of Cell, a foreign byte vector with no Serialize of its own, so it takes ssz_hex_seq; the commitments and proofs beside it are byte newtypes that write themselves, so they take seq. A single Cell, as in MatrixEntry, is neither and takes ssz_hex. Electra turns out to import its execution payload from deneb, and fulu has no block types at all, so their fixtures reuse rather than duplicate. * feat(types): serialize lean signed blocks as JSON Lets /lean/v0/blocks/finalized answer JSON when a caller asks for it. Integers stay bare: that is lean's own encoding and the rpc crate's tests assert it. The proof blob takes the 0x-prefixed convention the roots and bitlists use, rather than the prefix-less one the fixed-width XMSS pubkeys use, since it is opaque variable-length data and not an identity. * feat(types): serialize the container enums Neither enum adds a variant tag: the Beacon API carries the fork in the response envelope's version field and its Eth-Consensus-Version header, never inside the data object. The two get there differently, and the asymmetry is the point. The block enum derives untagged, because its Lean variant really is served as JSON on /lean/v0/blocks/finalized. The state enum cannot: a derive would demand a JSON encoding for lean's State, which is deliberately SSZ-only, so it is hand-written and the lean variant returns a serde error instead. * feat(rpc): negotiate response encoding on Accept The Beacon API serves JSON unless the caller asks for application/octet-stream, which is the order lighthouse serves too. The lean surface defaults the other way on its two SSZ endpoints, so the default stays the caller's to pass rather than a constant here. Ranks the header's entries by their q weight: a client that accepts both states its preference that way, and ignoring it would serve SSZ to a caller who merely tolerates it. base.rs's private ssz_response moves here rather than being duplicated, since both surfaces now need it. * feat(types): report a block's body root on either chain /eth/v1/beacon/headers/{id} answers with a SignedBeaconBlockHeader, whose body_root is the one field the existing accessors cannot produce: signed_beacon_block_accessors! hands back a field verbatim, and every fork stores a different body container, so what the caller wants is the merkle root of whichever one this is. Dispatched including lean, since lean's Block has a body too and /lean/v0 wants the same number. The phase0 test computes the expected root through the raw libssz_merkle trait rather than this crate's convenience wrapper, so a body_root that only worked on the lean arm would fail rather than pass by construction. * feat(rpc): parse and resolve beacon block ids Parsing is chain-agnostic, resolution is not. resolve_beacon reads only accessors that answer on either chain: Store::head_state and head_slot panic on a beacon store, so the lean slot lookup stays where it is rather than being repointed at BlockRoots, which covers the branch ending at the head rather than everything a lean store was bootstrapped with. genesis resolves to the store's anchor rather than to slot 0: a checkpoint-synced directory has no genesis block, and its anchor is the earliest block it can answer for at all. * feat(types): serialize the beacon chain config as JSON /eth/v1/config/spec echoes the chain configuration in the Beacon API's encoding: SCREAMING_SNAKE_CASE keys, every integer a quoted decimal, byte strings 0x-prefixed hex. Config could only be deserialized, which is how --network <dir> loads a network. 74 fields move from deserialize_with = "m::deserialize" to with = "m", which resolves to the same function on read, so the existing YAML parsing tests staying green is the evidence that nothing about loading a network moved. Six fields keep their own arrangement: genesis_time stays skipped since it is not a config.yaml key and /eth/v1/beacon/genesis reports it; terminal_total_difficulty and terminal_block_hash already have Serialize on their own types; max_blobs_per_block_deneb keeps its rename; and blob_schedule pairs its hand-written deserializer with seq::serialize, since its two halves do not live in one module. * feat(rpc): add the beacon API envelope and error shape Every fork-versioned Beacon API payload travels in {version, execution_optimistic, finalized, data}. The fork sits in the envelope rather than in the data because the containers serialize untagged, so version is how a caller knows which shape it just parsed. The error body is {code, message}, deliberately not the lean surface's {error}. They are two different APIs and each consumer parses the shape its own specification names, so unifying them would break one of the two. The submodule declarations and routes() land with the files they name, rather than being written now and commented out. * feat(rpc): serve beacon blocks by id in JSON and SSZ /eth/v2/beacon/blocks/{block_id} and its /root sibling, JSON by default and SSZ on Accept: application/octet-stream, both tagged with Eth-Consensus-Version so an SSZ caller can tell which fork's container it received. genesis is now refused rather than resolved to the anchor slot. Table::BlockRoots is written only by update_checkpoints, which computes its delta by walking from the old head to the new one and returns early when they are the same root. At bootstrap they always are, so the anchor's own slot is never indexed and the previous spelling could only ever 404. Saying so is better than a lookup that silently never works. The fixture moves its head through update_checkpoints rather than poking the backend, so BlockRoots is populated by its real writer and the slot-resolution tests exercise the path a running node takes. get_block_root loads the block rather than only resolving the id: it needs the slot to answer finalized honestly instead of hardcoding it. * feat(rpc): serve beacon block headers /eth/v1/beacon/headers/{block_id}. Every field of the message comes from SignedBeaconBlock's accessors, which dispatch including lean, plus body_root(); only the envelope and the signature are beacon-specific, which is what makes this the one endpoint genuinely shared with the lean surface. canonical is asked of BlockRoots rather than assumed, even for a block reached by its own root, so a sibling or a block at a slot the index does not cover answers false instead of being taken on trust. The test pins both directions: the head is canonical, the anchor is not, since its slot is never indexed. The quoting comes from ethlambda_types::beacon::serde_helpers rather than a to_string() per field, so the rule lives in one place. * feat(rpc): serve beacon states and finality checkpoints /eth/v2/debug/beacon/states/{state_id} is the path checkpoint_sync.rs already fetches from other clients, so serving it makes this client checkpoint-syncable from itself. The SSZ test decodes the body back through slot_from_ssz, since round-tripping is the whole job. A 0x... state_id is refused with a message naming the ids that do work. States are keyed by block root here with no reverse index, so the alternative to refusing is answering with the wrong state, or being right only when the two roots happen to coincide. finality_checkpoints reports the state's own three checkpoints rather than the store's fork-choice view: the endpoint is defined as a read of the state the id names, and the state is the only place a previous justified checkpoint is kept at all. JSON serializes through the Arc rather than cloning, since a mainnet state runs to hundreds of megabytes. * feat(rpc): serve beacon genesis and the spec config /eth/v1/beacon/genesis reads genesis_validators_root off the finalized anchor's state rather than looking for slot 0: it is a property of the chain that every state carries, and a checkpoint-synced directory has no genesis block to read it from. /eth/v1/config/spec echoes the Config the store was bootstrapped with, so a node started with --network <dir> reports that network's constants rather than mainnet's. GENESIS_TIME is absent from it by design, since it is not a config.yaml key and the genesis endpoint reports it. * feat(rpc): serve the beacon node endpoints syncing reads beacon_head rather than head_slot, which is lean-only and panics on a beacon store, and computes the wall slot in milliseconds the way /lean/v0/node/syncing does: slot_duration_ms is the value a loaded network can actually change, and seconds_per_slot would truncate a sub-second cadence to zero. health is 206 while syncing and 200 once caught up. 503 would mean uninitialized, and a store that answers at all is initialized. identity reports peer_id and metadata only. enr, p2p_addresses and discovery_addresses are empty, which is not spec-valid: the ENR is built for discv5 and owned by the P2P actor, and BuiltSwarm hands run_node only a local_peer_id, so serving the record means widening the ethlambda-p2p surface. Deliberate and recorded as a deviation. * feat(rpc): serve the beacon API from a beacon node Which HTTP surface a node serves now follows from the store's own chain tag rather than from the sub-command, so the two cannot disagree. Until now a beacon node bound --api-port to the lean router off a beacon store. Those handlers reach for lean state variants and metadata keys a beacon directory never carries, so calling one panicked that request. The beacon router replaces it rather than extending it, and the integration test pins both halves: /eth/v1/node/version answers, /lean/v0/node/syncing is a 404. build_beacon_api_router does not merge the metrics router, for the same reason build_api_router does not: start_http_servers serves it, and merging a path twice makes axum panic at startup. start_beacon_rpc_server takes no AggregatorController and no EventBus. A follower has no aggregator duty to toggle, and the chain-events stream is part of the lean surface. * feat(rpc): serve the lean finalized block as JSON on request SSZ stays the default here, which is the opposite of the beacon surface and deliberately so: checkpoint_sync.rs reads these bytes, and other clients' lean checkpoint sync may send no Accept header at all. Moving the default would break every one of them, so JSON is opt-in. Not routed through Encoding::from_accept, which defaults to JSON. The asymmetry is the point, so the branch is spelled out here rather than hidden behind a shared name that means the other thing. The existing SSZ test already pinned the default; the new test pins JSON on request, and that lean integers stay bare rather than picking up the beacon surface's quoting. * docs: describe the beacon API surface and its deviations docs/rpc.md gains the endpoint table, the encoding rules (JSON by default, SSZ on Accept, quoted integers, Eth-Consensus-Version) and the three ids that are refused. docs/spec_deviations.md records two: the partial node identity, and the ids that cannot be named. The second is a property of Table::BlockRoots rather than a choice, so it is written down where the reason lives. CLAUDE.md said a /lean/v0 call on a beacon node panics the request and that giving the follower its own surface was still a change of its own. Both stopped being true with this branch. * fix(rpc): resolve the anchor's own slot, which checkpoint sync asks for Found by running a mainnet follower and pointing a second one at its API: the second died with "peer served no block at the anchor slot 15265888" and never started. checkpoint_sync.rs reads a peer's finalized state, takes the anchor slot from it, and asks that peer for /eth/v2/beacon/blocks/{slot}. That is exactly the one slot Table::BlockRoots never holds, because update_checkpoints is its only writer and writes nothing when the head does not move, which is the situation at bootstrap. So the endpoint 404'd and this node was unusable as a checkpoint-sync source for any client, not only for itself. On an index miss the lookup now tries the roots the store can name and accepts one only when block_entry confirms its block really sits at the slot asked for, so a slot the store holds nothing at still 404s and no request can be answered with the nearest checkpoint instead. The test that pinned the old 404 asserted a bug, so it now asserts the anchor resolves, with a sibling holding the line on slots below it. The docs claimed a slot at or below the anchor was refused; only genesis and a state-root id are. * refactor(blockchain): move the empty-custody check to the node that configures one The per-block fence in data_availability_for cannot distinguish a misconfigured node from the replay harness, which legitimately supplies every block itself and has no columns to wait on. A node is checked once, at startup, where the custody set is computed. * feat(blockchain): expose an unspawned import entry for offline replay on_block already takes &mut self and no Context, and returns the import verdict, so a caller can drive the real import path without a mailbox. Two public items rather than a benchmark module inside this crate. * feat(cli): add the import benchmark's corpus manifest An SSZ corpus rather than a prepared store directory, so it stays readable and portable across revisions of the store format. Empty slots are recorded as gaps in the manifest rather than treated as a truncated range. * feat(cli): resolve a corpus anchor across an empty slot The state at from-1 is slot-advanced and keyed to no block when that slot is empty, so walk the first block's parent link instead. Written against a trait so the rule is tested without a live server. * feat(cli): stream a beacon block range into an SSZ corpus One state is held at a time and blocks are written as each response lands, so the footprint does not scale with the range and the range needs no cap. * refactor(cli): split the benchmark report by workload A pure move: the shared statistics go to report::common, today's report to report::synthetic, schema_version unchanged. * feat(cli): add the import workload's report and an at-most-once phase timer An import runs a subset of BLOCK_IMPORT_PHASES, so an unobserved phase is omitted rather than recorded as a zero that reads as instant work. * feat(cli): replay a corpus through the real import path Drives BlockChainServer::import_block directly: no mailbox, no tick loop, no p2p, and the clock placed per block from the store's own config so a run is reproducible. A block that does not import aborts the run, since every later block descends from it. * feat(cli): add benchmark import fetch and replay Wires the two phases to the command line and drops the dead_code allows the earlier slices needed while nothing reached this code. * docs: describe the import benchmark workload Records why the corpus exists: a live-follower A/B costs a restart and a moving denominator per leg, and the work being measured is deterministic. * fix(blockchain): publish a replayed import's sections under their own source import_block left ImportTimings.source unset to keep replayed blocks out of the gossip and sync percentiles, but BlockImportReport::observe publishes nothing for a sourceless import. The replay harness reads its per-phase numbers back from exactly those observations, so every phase came back absent and a mainnet replay reported wall time alone. BlockSource::Replay gives them a label no node ever writes: they stay out of the wire sources' percentiles and are still published. * fix(cli): make the import benchmark work against a live mainnet node Run against ethlambda-5's mainnet follower, a corpus whose first block's parent sat mid-epoch failed on that first block with `root in store.blocks`. get_forkchoice_store finalizes (epoch(anchor), anchor_root), and every import walks back to that epoch's first slot, which lies below a mid-epoch anchor. The spec fixtures anchor at genesis, so the tests never reached it. - fetch anchors on the block at the first slot of the epoch holding from - 1, stepping back past empty first slots, and records the blocks up to --from as warm-up that replay imports without sampling. replay refuses a corpus anchored anywhere else and says to re-fetch it. - A 404 means an empty slot, a slot past the head, or one before the source's history. fetch now refuses a --to past the head and checks every block's parent link, so a hole or a reorg mid-fetch stops the fetch instead of failing a replay minutes later. - A failed fetch removes the corpus directory it created. - Progress goes to stderr. fetch's tracing lines sat below the benchmark's WARN filter, and replay printed nothing for a mainnet block's seconds. - The report also samples prune, get_head and fcu, prints its inclusive range as [a, b], and drops the coefficient-of-variation warning: it named a flag replay does not have, and across a corpus it measures how blocks differ, not noise. * fix(blockchain): initialize the committee cache in for_replay Merging beacon-chain-integration brought in lambdaclass#22, which added a `committees` field to `BlockChainServer`. The text merge was clean, but `for_replay` builds the struct by hand and never set the new field, so the branch stopped compiling (E0063). Replay starts from an empty cache, the same as a freshly spawned node. * chore(docs): drop the beacon API spec and plans from the branch They are working notes from the design sessions, not documentation, and were swept into bfdbdc8c alongside an unrelated rpc fix. Nothing links to them. They stay recoverable from that commit. * refactor(cli): stop reading the state cache's occupancy from the benchmark The replay read the store's LRU length to report peak_states_cached and to assert that a long range stays within STATE_CACHE_CAPACITY. An LRU cannot exceed its capacity by construction, and the storage crate's own the_state_cache_is_bounded already pins the cap, so the number could never show anything that test does not. It also cannot see what the replay module doc warns about: an Arc<BeaconState> pinned outside the cache leaves len() unchanged. That is not worth a benchmark-only accessor on Store. * test(blockchain): delete the beacon_replay integration test The benchmark's replay tests in bin/ethlambda already drive BlockChainServer::for_replay and import_block end to end, and this one read consensus-spec-tests fixtures the Test (consensus) job never downloads, so it failed on every CI run. Its snap and serde_yaml_ng dev-dependencies go with it; nothing else in the crate uses them. * test(cli): delete the import benchmark's fixture-backed replay tests They read a consensus-spec-tests case the Test (node) job never downloads, so they failed on every CI run. Two of the five also asserted less than their names claim: the block roots they compared are hashed from the corpus files, so "the head advances" and "two runs agree" held whenever both replays finished. The spec-case corpus builder, ReplayOptions::for_tests and the snap dev-dependency only served these tests and go with them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.