Blocktest builder #2190

bshastry · 2025-09-23T10:09:38Z

🗒️ Description

🔗 Related Issues or PRs

N/A.

✅ Checklist

All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
```
uvx --with=tox-uv tox -e lint,typecheck,spellcheck,markdownlint
```
All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
All: Considered adding an entry to CHANGELOG.md.
All: Considered updating the online docs in the ./docs/ directory.
All: Set appropriate labels for the changes (only maintainers can apply labels).
Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.
Ported Tests: All converted JSON/YML tests from ethereum/tests or tests/static have been assigned @ported_from marker.

danceratopz · 2025-09-23T13:40:51Z

CLAUDE.md

+**IMPORTANT**: Always activate the virtual environment first before running any commands:
+```bash
+source .venv/bin/activate
+```
+


This is incorrect, you should run uv sync --all-extras, but this should be covered.

jtraglia · 2025-09-23T14:39:37Z

src/cli/fuzzer_bridge/README.md

+```
+┌──────────────┐     JSON      ┌──────────────┐      Fixtures     ┌──────────────┐
+│  Blocktest   │ ──────────▶   │   Fuzzer     │ ──────────────▶   │   Ethereum   │
+│   Fuzzer     │   (v2 format)  │   Bridge     │   (Blockchain     │   Clients    │
+└──────────────┘                └──────────────┘     Tests)        └──────────────┘
+```


The boxes for "fuzzer bridge" and "ethereum clients" are messed up.

danceratopz

Hey @bshastry, really nice idea to "dynamically" generate blockchain tests from fuzzer output using EEST!

I have a few suggestions below, very happy to help you get this in a form that is more compatible with EEST's libraries and pydantic models. I'll message you on Signal to figure out how we'd like to proceed!

danceratopz · 2025-09-23T13:42:39Z

CLAUDE.md

+**IMPORTANT**: Always activate the virtual environment first before running any commands:
+```bash
+source .venv/bin/activate
+```
+


We should remove this, the user should never manage .venv (uv manages it). All commands should be prefixed with uv run (as noted just above), then they'll use the .venv created by uv sync --all-extras (also above in this file in Setup; we could mention at that point that this command creates a venv there to help avoid Claude getting confused).

danceratopz · 2025-09-23T13:55:36Z

src/cli/fuzzer_bridge/README.md

+```
+┌──────────────┐     JSON      ┌──────────────┐      Fixtures     ┌──────────────┐
+│  Blocktest   │ ──────────▶   │   Fuzzer     │ ──────────────▶   │   Ethereum   │
+│   Fuzzer     │   (v2 format)  │   Bridge     │   (Blockchain     │   Clients    │
+└──────────────┘                └──────────────┘     Tests)        └──────────────┘
+```


Helpful! But could you ask Claude to add a mermaid diagram instead of ascii art? :)

danceratopz · 2025-09-29T08:13:32Z

src/cli/fuzzer_bridge/README.md

+1. **Private Keys**: Any account that sends transactions MUST include a `privateKey` field
+2. **Address-Key Match**: Private keys must generate the corresponding addresses
+3. **Environment**: Describes the environment for block 1 (genesis is automatically derived)
+4. **Version**: Use "2.0" for this format


Suggested change

1. **Private Keys**: Any account that sends transactions MUST include a `privateKey` field

2. **Address-Key Match**: Private keys must generate the corresponding addresses

3. **Environment**: Describes the environment for block 1 (genesis is automatically derived)

4. **Version**: Use "2.0" for this format

1. **Private Keys**: Any account that sends transactions MUST include a `privateKey` field.

2. **Address-Key Match**: Private keys must generate the corresponding addresses.

3. **Environment**: Describes the environment for block 1 (genesis is automatically derived).

4. **Version**: Use "2.0" for this format.

danceratopz · 2025-09-29T08:16:57Z

src/cli/fuzzer_bridge/README.md

@@ -0,0 +1,205 @@
+# Fuzzer Bridge for Execution Spec Tests


A minimal readme is ok, but I think this should rather point to a page in the online documentation for easier sharing.

Unfortunately, we don't have a great top-level section for this, but perhaps we can put it in https://eest.ethereum.org/main/writing_tests/

For this, you'll need to:

Create an .md in ./docs/writing_tests/.

Add an entry to the table of contents in docs/navigation.md linking to it.

But maybe, it could be a new top-level section in "Running Tests"? Wdyt?
https://eest.ethereum.org/main/running_tests/

danceratopz · 2025-09-29T08:26:29Z

src/cli/fuzzer_bridge/README.md

+```json
+{
+  "version": "2.0",
+  "fork": "Prague",


Unfortunately, this is called network in blockchain fixtures. I guess we can live with the inconsistency :)

danceratopz · 2025-09-29T08:40:29Z

src/cli/fuzzer_bridge/README.md

+
+## Production Test Suite
+
+See `production_test.py` for a complete example that:


production_test.py in missing from this PR.

danceratopz · 2025-09-29T08:41:28Z

src/cli/fuzzer_bridge/README.md

+## Contributing
+
+When modifying the fuzzer bridge:
+1. Update the version number in `__init__.py`


Please tell Claude to always punctuate his bullet points.

danceratopz · 2025-09-29T08:41:42Z

src/cli/fuzzer_bridge/README.md

+## License
+
+See the main execution-spec-tests repository for license information.


Unnecessary. Especially in the docs.

danceratopz · 2025-09-29T08:42:29Z

src/cli/fuzzer_bridge/README.md

+## Contributing
+
+When modifying the fuzzer bridge:
+1. Update the version number in `__init__.py`


There is no version number in init.py.

danceratopz · 2025-09-29T09:02:45Z

src/cli/fuzzer_bridge/blocktest_builder.py

@@ -0,0 +1,240 @@
+"""Build valid blocktests from fuzzer-generated transactions and pre-state."""


I mentioned pydantic models above, there's a lot of opportunity here to align this code with EEST code standards to streamline it and make it more maintainable and future proof. This is what I'd suggest (also with some ai-help):

Create Pydantic Models for Fuzzer Output:

Define FuzzerOutput Pydantic model to replace dict parsing.

Add nested models: FuzzerAccount, FuzzerTransaction, FuzzerEnvironment.

Implement validation and type coercion at the model level.

Add from_fuzzer Method to BlockchainTest in ./src/ethereum_test_specs/blockchain.py.

Create class method on BlockchainTest to directly construct from FuzzerOutput.

Move all conversion logic into this method for better encapsulation.

Eliminate the need for separate BlocktestBuilder class.

- Remove incorrect venv activation instructions from CLAUDE.md - Replace ASCII art with Mermaid diagram in README - Add fuzzer_bridge as CLI entry point in pyproject.toml - Create comprehensive documentation in docs/writing_tests/ - Add Pydantic models for type-safe fuzzer output parsing - Add BlockchainTest.from_fuzzer() classmethod for better integration - Simplify BlocktestBuilder to use new architecture - Fix README inconsistencies (punctuation, references, sections) This refactoring aligns the fuzzer bridge with EEST code standards and makes it more maintainable and future-proof. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Add ruff noqa comment for mixedCase variables in Pydantic models - Fix line length issues in performance_utils.py - Replace bare except with specific Exception handling - Apply automated formatting from ruff 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

- Fix markdown linting errors in fuzzer_bridge.md documentation - Apply ruff formatting to blockchain.py - Fix mypy type errors in fuzzer bridge modules: - Make orjson import optional with proper error handling - Fix TransitionTool instantiation to use GethTransitionTool - Correct EOA usage and add missing class attributes - Fix json.dump parameter usage to avoid kwargs issues - Fix docstring line lengths to comply with 79-char limit - Fix line length issues in function signatures All linting (ruff) and type checking (mypy) now pass successfully. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

Add support for two critical Ethereum features in the fuzzer bridge: 1. EIP-7702 Authorization Lists (Prague+) - Add FuzzerAuthorization model for authorization tuples - Parse authorizationList from transactions - Translate to AuthorizationTuple in from_fuzzer() - Enables testing of SetCode transactions (tx type 0x04) 2. EIP-4788 Parent Beacon Block Root (Cancun+) - Add parentBeaconBlockRoot as top-level field in FuzzerOutput - Pass to Block during creation in from_fuzzer() - Must be 32-byte hash from consensus layer - Enables testing of beacon root contract (EIP-4788) These changes close significant coverage gaps in geth core validation: - tx_setcode.go: Authorization validation paths - block_validator.go: Beacon root handling - state_transition.go: Beacon root system contract calls The implementation follows existing patterns (similar to blob sidecar handling) and maintains backwards compatibility through optional fields. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Add support for splitting fuzzer transactions across multiple blocks to enable testing of state transitions and block boundaries. New CLI Options: - --num-blocks: Number of blocks to generate (default: 1) - --block-strategy: Transaction distribution strategy - "distribute": Sequential chunks preserving nonce order - "first-block": All txs in first block, rest empty - --block-time: Seconds between blocks (default: 12) Implementation: - Sequential distribution maintains nonce ordering per account - Timestamps increment by block_time for each block - Only first block receives parent_beacon_block_root - Fully backward compatible (single-block default) Example Usage: fuzzer_bridge --num-blocks 3 --block-strategy distribute \ --fork Osaka input.json output/ This enables testing multi-block scenarios like: - State evolution across blocks - Transaction dependencies spanning blocks - Block time-sensitive operations - Cross-block state transitions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Add --random-blocks flag to enable intelligent automatic selection of block counts for comprehensive testing coverage. New Feature: - --random-blocks: Randomly choose number of blocks (1 to min(num_txs, 10)) Implementation: - choose_random_num_blocks() helper with uniform distribution - Each file gets independent random selection in parallel mode - Edge case handling: empty blocks (0 txs → 1 block) - Cap at 10 blocks to prevent fixture bloat with large tx counts Algorithm Rationale: - Uniform distribution provides equal coverage for testing - Max of 10 blocks balances thoroughness with practicality - Independent per-file randomization maximizes corpus diversity Example Usage: # Random mode - each file gets random block count fuzzer_bridge --random-blocks --fork Osaka input/ output/ # Still works with fixed mode fuzzer_bridge --num-blocks 3 --fork Osaka input/ output/ Benefits: - Automated testing of various block configurations - Discovers edge cases in block boundary handling - Comprehensive multi-block scenario coverage - Zero configuration needed for random testing 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

danceratopz reviewed Sep 23, 2025

View reviewed changes

jtraglia reviewed Sep 23, 2025

View reviewed changes

danceratopz reviewed Sep 29, 2025

View reviewed changes

danceratopz self-assigned this Sep 29, 2025

bshastry and others added 11 commits September 29, 2025 13:37

Initial commit

6bce89f

Some refactoring

390578a

Add batch mode

188500b

Support blob gas fields in genesis

dd13b3a

Don't default to gasPrice=1 to not error on blob tx

6ee2fd3

Fix setcodetx translation

14e5b25

Parallelize

5e18969

Per file progress meter

fe18f89

Fix auth translation

f7fb175

bshastry force-pushed the blocktest-builder branch from 862d70e to 5472e9b Compare September 29, 2025 11:48

bshastry and others added 5 commits September 29, 2025 14:35

Fix docs link

9876593

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Blocktest builder #2190

Blocktest builder #2190

Uh oh!

bshastry commented Sep 23, 2025

Uh oh!

danceratopz Sep 23, 2025

Uh oh!

jtraglia Sep 23, 2025

Uh oh!

danceratopz left a comment

Uh oh!

danceratopz Sep 23, 2025

Uh oh!

danceratopz Sep 23, 2025

Uh oh!

danceratopz Sep 29, 2025

Uh oh!

danceratopz Sep 29, 2025

Uh oh!

danceratopz Sep 29, 2025

Uh oh!

danceratopz Sep 29, 2025

Uh oh!

danceratopz Sep 29, 2025

Uh oh!

danceratopz Sep 29, 2025

Uh oh!

danceratopz Sep 29, 2025

Uh oh!

danceratopz Sep 29, 2025

Uh oh!

danceratopz Sep 29, 2025

Uh oh!

Uh oh!


		## Production Test Suite

		See `production_test.py` for a complete example that:

		## License

		See the main execution-spec-tests repository for license information. No newline at end of file

		@@ -0,0 +1,240 @@
		"""Build valid blocktests from fuzzer-generated transactions and pre-state."""

Blocktest builder #2190

Are you sure you want to change the base?

Blocktest builder #2190

Uh oh!

Conversation

bshastry commented Sep 23, 2025

🗒️ Description

🔗 Related Issues or PRs

✅ Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danceratopz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!