feat: add multi-fork architecture with ForkProtocol and SpecRunner by tcoratger · Pull Request #638 · leanEthereum/leanSpec

tcoratger · 2026-04-18T21:34:56Z

We just start from devnet4 here to begin right now and not go back in time, which would be a mess.

Proposal from @leolara.

Summary

Introduce ForkProtocol(ABC) + SpecRunner for multi-fork dispatch
Move State and Store from subspecs/ into forks/devnet4/ (fork-specific code)
subspecs/ now contains only fork-agnostic shared libraries
Consumer modules import from lean_spec.forks (re-exports, fork-generic)
Construction goes through ForkProtocol (fork.generate_genesis(), fork.create_store())
Rename testing fork Devnet → Devnet4, add Devnet5 skeleton
Adding a new fork: create Devnet5State(State) + Devnet5Spec(Devnet4Spec), done

Based on Multi-Fork Architecture proposal.

Test plan

uvx tox -e all-checks passes (ruff, format, ty, codespell, mdformat)
uv run pytest --no-cov passes (all 3267 tests)
uv run fill --fork=devnet4 --clean -n auto generates fixtures

🤖 Generated with Claude Code

Introduce a protocol-and-fork-class architecture that enables multiple devnet forks to coexist in the codebase using Python class inheritance. Key changes: - Add ForkProtocol ABC defining the consensus interface each fork implements - Add SpecRunner for era-aware fork dispatch - Create Devnet4Spec wrapping current State/Store as the base fork - Create Devnet5Spec skeleton inheriting from Devnet4Spec - Move State and Store from subspecs/ into forks/devnet4/ (fork-specific) - Wire ForkProtocol into Node, __main__, genesis, and storage - Rename testing fork from Devnet to Devnet4, add Devnet5 - Update all consumer imports to use lean_spec.forks re-exports - Update all test markers from valid_until("Devnet") to valid_until("Devnet4") The architecture follows the coworker's proposal: subspecs/ contains only fork-agnostic shared libraries (ssz, xmss, networking, chain). Fork-specific processing logic (State, Store) lives in forks/devnetN/. Consumer modules import State/Store from lean_spec.forks (the package re-export). Construction goes through ForkProtocol (fork.generate_genesis(), fork.create_store()). Adding a new fork requires: 1. forks/devnet5/state.py — Devnet5State(State) overriding changed methods 2. forks/devnet5/spec.py — Devnet5Spec(Devnet4Spec) with generate_genesis/upgrade_state 3. Python virtual dispatch handles the rest — no consumer code changes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

tcoratger · 2026-04-18T21:38:41Z

 from pydantic import Field, field_validator, model_validator

-from lean_spec.subspecs.containers import State, Validator
+from lean_spec.forks import ForkProtocol, State


I'm not happy with the imports of State and Store like this one (couple of them inside the subspecs/ folder) because it's the defaults that are imported here (and therefore for now devnet4). Which obviously not fully generic since we ideally want to be devnet agnostic here!

But first of all this is the easiest solution and we could find a much cleaner solution in some followup PRs (not sure about the best design to adopt here to make a compromise between the generic aspect and strong typing).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…nericize ForkProtocol Relocate every consensus-dependent container (Attestation, Block, Checkpoint, Config, Slot, State, Validator) from subspecs/containers/ into forks/devnet4/containers/ so per-fork divergence of container shapes (e.g. a future Devnet5 Attestation) becomes physically possible without mutating devnet4 symbols. Pydantic inheritance across forks is documented as unsafe for SSZ hash_tree_root stability — the supported pattern is copy-then-diverge inside the new fork's package. Strip ForkProtocol to its irreducible surface: five ClassVars (NAME, VERSION, state_class, block_class, store_class) and three methods (generate_genesis, create_store, upgrade_state). The protocol module imports nothing from any devnet package and exposes SpecStateType / SpecStoreType structural Protocols for type-hinting the classmethod contract. Runner rejects duplicate NAMEs and non-strictly-increasing VERSIONs; adds DEFAULT_RUNNER convenience. Reinstate Devnet5Spec as a genuine registered placeholder: own NAME, own VERSION, inherits method logic from Devnet4Spec, binds container types via explicit aliases that are one-line swappable when divergence lands. The multi-fork pipeline (runner, fixture filler, test framework) now exercises two forks end to end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The previous test asserted the literal string 'devnet4' did not appear in forks/protocol.py, which tripped on the docstring example for the NAME ClassVar. Replace with an AST-based check that inspects Import and ImportFrom nodes only — docstrings and comments are free to reference fork names as examples. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Integrates the metrics→observability split (leanEthereum#667) with the multi-fork layout: - fork code (forks/devnet4/store.py, forks/devnet4/containers/state/state.py) imports only the vendor-neutral observe_on_attestation / observe_on_block / observe_state_transition hooks; no Prometheus-specific metrics land inside forks/, keeping the fork folder consensus-critical by construction - BlockLookup is now the plain dict[Bytes32, Block] alias from leanEthereum#667, so the former dict-subclass imports (Iterator, GetCoreSchemaHandler, ZERO_HASH) and the node's BlockLookup({...}) wrap are dropped - reorg-depth telemetry moves to sync/service.py's default_block_processor and stays off the spec side - consensus tests and fixture builders added in leanEthereum#663, leanEthereum#664, leanEthereum#665, leanEthereum#666 retarget lean_spec.subspecs.containers.* → lean_spec.forks.devnet4.containers.*; the verify_signatures tamper hook's in-function imports are pulled up to the top - drops tests/lean_spec/subspecs/containers/block/test_block_lookup.py (BlockLookup no longer has ancestors / reorg_depth methods to test) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Bundles four small, behavior-preserving improvements to the multi-fork skeleton from PR leanEthereum#638. Tests + all-checks green. * Expose `GOSSIP_DIGEST: ClassVar[str]` on `ForkProtocol`. The gossipsub fork digest is fork metadata, not a top-level constant. Devnet4Spec binds it to "devnet0" (preserving the existing network agreement); Devnet5Spec inherits. Removes the hardcoded `GOSSIP_FORK_DIGEST` constant and the now-unused `Final` import from `__main__.py`, and routes every consumer through `fork.GOSSIP_DIGEST`. * Use `DEFAULT_RUNNER` directly in `__main__.py` instead of building a second `SpecRunner(FORK_SEQUENCE)`. The forks package already exports the singleton; the redundant construction is dropped along with the unused `FORK_SEQUENCE` and `SpecRunner` imports. * Add `previous: ClassVar[type[ForkProtocol] | None]` linking each fork to its predecessor. `Devnet4Spec.previous = None`; `Devnet5Spec.previous = Devnet4Spec`. Sets up the foundation for a future registry that derives ordering from topology and for chained state migrations via `upgrade_state`. * Tidy `devnet5/spec.py`: drop the module-level `_Devnet5State` and `_Devnet5Store` identity aliases; bind the container classes directly on `Devnet5Spec`. When devnet5 grows its own `forks/devnet5/containers/` package, the swap is a file-level diff in the imports rather than an alias rebinding. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The class is a registry — it stores a sequence of forks and looks them up by name — not a dispatcher routing spec calls per fork. The old name implied the latter and set wrong expectations for follow-ups. * Renames the file `runner.py` -> `registry.py`. * Renames the class `SpecRunner` -> `ForkRegistry`. * Renames the singleton `DEFAULT_RUNNER` -> `DEFAULT_REGISTRY`. * Updates the docstring on `FORK_SEQUENCE` and the error message in the empty-list guard. * Updates the test class name and assertions. No behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Replaces the silent identity default with an abstract method, forcing every concrete fork to declare its migration explicitly. This prevents the silent-no-op footgun: a future fork that adds a State field but forgets to override upgrade_state would otherwise migrate by simply not migrating. * `ForkProtocol.upgrade_state` becomes `@abstractmethod` with a docstring-only body. * `Devnet4Spec.upgrade_state` is identity, documented as "root fork, no predecessor, no migration". * `Devnet5Spec.upgrade_state` is identity, documented as "currently mirrors devnet4; replace when devnet5's State diverges". * Adds a test that asserts `ForkProtocol()` raises TypeError, locking in the abstract-class contract. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Stage 1: tighten ForkProtocol surface (trivial items)

Absorbs 14 commits from main and resolves two textual conflicts in genesis/config.py and the matching test (PR leanEthereum#683 deleted the dead GenesisConfig.create_state helper that this branch had refactored to take a fork argument; production code goes through fork.generate_genesis directly, so the wrapper is genuinely dead). Reroutes two stale subspecs.containers imports left behind by the auto-merge: a new test file from PR leanEthereum#684 (Checkpoint __lt__) and a sync test fixture from PR leanEthereum#672. Both now resolve through forks.devnet4.containers. Tightens the docstring and inline comment introduced for update_safe_target by PR leanEthereum#680 to follow the project's documentation rules (short sentences, bullets over paragraphs). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…r/leanSpec into multi-fork-architecture

Absorbs PR leanEthereum#687 which split event_source.py into a package. Resolves one content conflict in event_source/live.py: - Kept the forks.devnet4.containers import paths from this branch. - Accepted upstream's removal of the snappy import (decompress usage moved to event_source/gossip.py during the split, so live.py no longer needs it). Reroutes the new event_source/gossip.py file's stale subspecs.containers imports to forks.devnet4.containers (rename-vs-add auto-merge took the file as-is from upstream). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Delete forks/devnet5/ entirely. We are running one devnet roughly per month, so per-devnet placeholders rot fast and lock the spec into a sequential upgrade story we are not committing to. Rename forks/devnet4/ to forks/lstar/ (via git mv, history preserved): - Devnet4Spec -> LstarSpec - Devnet4 (test BaseFork class) -> Lstar - "devnet4" / "Devnet4" string identifiers -> "lstar" / "Lstar" - lean_spec.forks.devnet4.* import paths -> lean_spec.forks.lstar.* - --fork=Devnet4 / --fork=devnet4 in CI workflows and docs -> Lstar / lstar Test surface follows: TestDevnet4Spec -> TestLstarSpec, TestDevnet5Spec removed entirely. Multi-fork ForkRegistry tests now use a synthetic in-test successor class instead of a real second fork. Untouched on purpose: - pyproject.toml lean-multisig-py branch="devnet4" (external repo, not ours). - tests/consensus/devnet/ folder name (generic test path, no version number). - GOSSIP_DIGEST="devnet0" — the cross-client gossip network name set by PR leanEthereum#622. Renaming it touches the whole networking layer and is a separate cleanup. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The value sitting on the gossipsub topic layer is the cross-client network name (currently "devnet0", set by PR leanEthereum#622), not a 4-byte fork digest hash. The "fork_digest" naming was a holdover from Ethereum mainline where it really is a digest; here it just confuses readers into expecting a hash where there is a string identifier. Renamed in the network-name layer: - ForkProtocol.GOSSIP_DIGEST -> ForkProtocol.NETWORK_NAME - GossipTopic.fork_digest -> GossipTopic.network_name - GossipTopic.{block,committee_aggregation,attestation_subnet} fork_digest parameter -> network_name - GossipTopic.validate_fork(expected_fork_digest=...) -> validate_fork(expected_network_name=...) - GossipTopic.from_string_validated(..., expected_fork_digest=...) -> expected_network_name - GossipHandler.fork_digest -> GossipHandler.network_name - LiveNetworkEventSource._fork_digest -> _network_name - LiveNetworkEventSource.set_fork_digest() -> set_network_name() - NetworkService.fork_digest -> NetworkService.network_name - NodeConfig.fork_digest -> NodeConfig.network_name Plus matching test renames: - test_gossip_digest -> test_network_name - test_gossip_topic_fork_digest_{matches,mismatch,...} -> test_gossip_topic_network_name_* - All test variables, parameters, and prose updated. Untouched on purpose: - Eth2Data.fork_digest: ForkDigest in subspecs/networking/enr/eth2.py is a real 4-byte ENR hash per the Ethereum p2p spec. The ForkDigest type stays. enr/enr.py, peer.py, discovery/routing.py all interact with this real digest and are unchanged. - JSON fixture keys "forkDigest" / "expectedForkDigest" in cross-client test vectors. Other clients consume those keys; renaming would break the wire format. Internal Python uses network_name; the JSON keys are read into network_name on the way in. - ForkMismatchError class name. A fork mismatch is still a fork mismatch semantically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

tcoratger · 2026-04-27T11:59:26Z

+
+    NAME: ClassVar[str] = "lstar"
+    VERSION: ClassVar[int] = 4
+    NETWORK_NAME: ClassVar[str] = "devnet0"


We should rename this to "lstar" probably. I haven't touch this since this is not the purpose of the PR but this is something we should do in a followup PR

tcoratger commented Apr 18, 2026

View reviewed changes

tcoratger and others added 3 commits April 18, 2026 23:39

ci: update --fork=Devnet to --fork=Devnet4 in workflows

2381a9c

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

tcoratger force-pushed the multi-fork-architecture branch from 66a3333 to 1ce888e Compare April 20, 2026 11:25

unnawut mentioned this pull request Apr 22, 2026

metrics: simplify and separate metrics logic from specs #667

Merged

3 tasks

tcoratger and others added 2 commits April 22, 2026 11:04

docs(forks): trim consensus-testing fork module docstring

cd7750a

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

shariqnaiyer mentioned this pull request Apr 22, 2026

Evaluate multi fork architecture proposed vs current feature flags implementation ReamLabs/ream#1350

Open

leolara and others added 3 commits April 27, 2026 13:43

This was referenced Apr 27, 2026

Multi-fork architecture: roadmap and tracking issue #686

Open

Stage 1: tighten ForkProtocol surface (trivial items) tcoratger/leanSpec#3

Merged

tcoratger and others added 6 commits April 27, 2026 11:59

Merge pull request #3 from leolara/multi-fork-stage1-tidy

6d23c9b

Stage 1: tighten ForkProtocol surface (trivial items)

Merge branch 'multi-fork-architecture' of https://github.com/tcoratge…

0a25f45

…r/leanSpec into multi-fork-architecture

tcoratger commented Apr 27, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add multi-fork architecture with ForkProtocol and SpecRunner#638

feat: add multi-fork architecture with ForkProtocol and SpecRunner#638
tcoratger wants to merge 15 commits intoleanEthereum:mainfrom
tcoratger:multi-fork-architecture

tcoratger commented Apr 18, 2026 •

edited

Loading

Uh oh!

tcoratger Apr 18, 2026

Uh oh!

tcoratger Apr 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tcoratger commented Apr 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

tcoratger Apr 18, 2026

Choose a reason for hiding this comment

Uh oh!

tcoratger Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tcoratger commented Apr 18, 2026 •

edited

Loading