Skip to content

Commit c3f787e

Browse files
mempiratemempirate
authored
docs: update README and AGENTS (#116)
1 parent bd8a3c9 commit c3f787e

File tree

2 files changed

+253
-29
lines changed

2 files changed

+253
-29
lines changed

‎AGENTS.md‎

Lines changed: 235 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,257 @@
11
# Repository Guidelines
22

3-
This guide helps contributors deliver changes to `buildernet-orderflow-proxy` safely and predictably.
3+
This guide helps contributors deliver changes to `buildernet-orderflow-proxy-v2` safely and predictably.
44

55
## Project Structure & Module Organization
6+
7+
### Entry Points & Core Structure
68
- The entrypoint binary lives in `src/main.rs` and delegates to the library crate in `src/lib.rs`.
7-
- Domain modules are located under `src/`:
8-
- `ingress/`: Handles HTTP+JSON-RPC requests.
9-
- `priority/`: Scores queues.
10-
- `forwarder.rs`: Relays bundles.
11-
- `rate_limit.rs`: Implements throttling logic.
12-
- Shared helpers are in `types.rs` and `utils.rs`.
13-
- Integration tests are under `tests/`, with reusable fixtures in `tests/common/`.
9+
- CLI arguments are defined in `src/cli.rs` using the `clap` crate.
10+
- The `src/runner/` module handles application lifecycle and HTTP server setup.
11+
12+
### Domain Modules
13+
Core domain logic is organized under `src/`:
14+
- **`ingress/`**: HTTP handlers for three endpoints:
15+
- `user.rs`: Public API for bundle/transaction submission
16+
- `system.rs`: Internal API for system operations
17+
- `builder.rs`: Builder stats endpoint
18+
- Handles validation, entity scoring, rate limiting, and queuing
19+
- **`forwarder.rs`**: Relays bundles to local builder via HTTP and peer proxies via BuilderHub
20+
- **`priority/`**: Multi-level priority queue system (High/Medium/Low)
21+
- `mod.rs`: PriorityQueues implementation
22+
- `pchannel.rs`: Custom priority channels with backoff strategies
23+
- **`entity.rs`**: Entity tracking, scoring, and spam detection logic
24+
- **`rate_limit.rs`**: Per-entity rate limiting with sliding windows
25+
- **`validation.rs`**: EVM transaction validation (intrinsic gas, initcode, chain ID, EIP-4844/7702)
26+
- **`builderhub.rs`**: Peer discovery and management via BuilderHub integration
27+
- **`indexer/`**: Data persistence layer
28+
- `click/`: ClickHouse indexer with async batching
29+
- `parq.rs`: Parquet file-based storage
30+
- **`cache.rs`**: Order and signer caching with TTL-based LRU eviction
31+
- **`metrics.rs`**: Prometheus metrics for observability
32+
- **`primitives/`**: Bundle types and encoding utilities
33+
- **`jsonrpc.rs`**: JSON-RPC protocol handling
34+
- **`tasks/`**: Task execution and graceful shutdown coordination
35+
36+
### Supporting Code
37+
- **`src/utils.rs`**: Shared helper functions
38+
- **`benches/`**: Criterion performance benchmarks
39+
- **`fixtures/`**: SQL schema files for ClickHouse tables
40+
- **`simulation/`**: Load testing harness with Docker Compose
41+
- **`tests/`**: Integration tests with reusable fixtures in `tests/common/`
42+
43+
### Guidelines for New Code
1444
- Place new code in the closest domain module and expose it through `lib.rs`.
45+
- For cross-cutting concerns, consider adding to `utils.rs` or creating a new focused module.
46+
- Keep modules focused and cohesive; split large modules into submodules when they exceed ~500 lines.
1547
## Build, Test, and Development Commands
48+
49+
### Building
1650
- `cargo build` compiles the binary with the default dev profile.
51+
- `cargo build --release` produces an optimized build with LTO enabled.
52+
- `just build-reproducible` creates a verifiable production build (uses the `reproducible` profile).
1753
- `cargo run -- --help` prints the CLI flags defined in `src/cli.rs` and is the fastest smoke-test.
18-
- `cargo fmt --all` applies the formatting rules in `rustfmt.toml`.
19-
- `cargo clippy --all-targets --all-features -D warnings` enforces lints configured in `Cargo.toml`.
54+
55+
### Testing
56+
- `just test` runs the test suite using nextest (faster parallel test runner).
2057
- `cargo test` executes unit and integration tests; add `-- --nocapture` when debugging async failures.
58+
- `cargo bench` runs Criterion benchmarks (validation, signature verification, etc.).
59+
- Integration tests use `testcontainers` for ClickHouse and require Docker to be running.
60+
61+
### Code Quality
62+
- `just fmt` or `cargo +nightly fmt --all` applies formatting rules from `rustfmt.toml`.
63+
- `just clippy` or `cargo clippy --all-targets --all-features -- -D warnings` enforces lints.
64+
- All linting configuration lives in `Cargo.toml` under `[lints.clippy]`.
65+
66+
### Database Operations
67+
- `just provision-db` creates the ClickHouse database and tables from `fixtures/`.
68+
- `just reset-db` drops and recreates tables (destructive operation).
69+
- `just extract-data <FILE>` exports data to Parquet format.
70+
71+
### Justfile Commands
72+
Run `just --list` to see all available commands. The justfile automates common workflows and should be the preferred method for development tasks.
2173

2274
## Coding Style & Naming Conventions
23-
- Keep Rust code within 100 columns.
75+
- Keep Rust code within 100 columns (enforced by rustfmt).
2476
- Use `rustfmt` for formatting; it is authoritative and will reorder imports at crate granularity.
2577
- Use `snake_case` for modules and functions, `UpperCamelCase` for types, and reserve `SCREAMING_SNAKE_CASE` for constants.
2678
- Prefer structured errors (`thiserror`, `eyre`) over panics, and bubble fallible calls with `?`.
2779
- When adding public APIs, gate re-exports in `lib.rs`.
28-
- Document public APIs with concise rustdoc comments.
80+
- Document public APIs with concise rustdoc comments (`///` for public items).
81+
- Use `tracing::instrument` for function-level observability on hot paths and error scenarios.
82+
- Prefer immutable data structures; use `Arc` for shared ownership across async tasks.
83+
- Keep async functions small and focused; extract complex logic into sync helper functions when possible.
2984

3085
## Testing Guidelines
31-
Async flows rely on `tokio::test`, so keep helpers in `tests/common` to avoid duplication. Write integration tests alongside existing `ingress.rs` and `network.rs` files, naming functions with the behavior under test (e.g. `ingress_rejects_invalid_signature`). Run `cargo test --features ...` if you introduce optional features. New behavior must include either a targeted unit test in the owning module or an integration scenario covering success and error paths.
86+
87+
### Test Organization
88+
- Unit tests live in the same file as the code they test, in a `#[cfg(test)]` module.
89+
- Integration tests are in the `tests/` directory, organized by feature area.
90+
- Shared test utilities and fixtures belong in `tests/common/`.
91+
- Async flows rely on `#[tokio::test]`, so ensure all async test helpers are properly propagated.
92+
93+
### Naming Conventions
94+
Name test functions with the behavior under test, following the pattern:
95+
- `<module>_<scenario>_<expected_outcome>`
96+
- Examples: `ingress_rejects_invalid_signature`, `forwarder_retries_on_timeout`, `entity_scores_spam_bundle_low`
97+
98+
### Test Coverage Requirements
99+
New behavior must include either:
100+
1. A targeted unit test in the owning module covering the happy path and key edge cases, OR
101+
2. An integration test scenario covering success and error paths
102+
103+
### Running Tests
104+
- `just test` uses nextest for parallel execution (recommended).
105+
- `cargo test` for standard test runner.
106+
- `cargo test -- --nocapture` when debugging async failures or inspecting tracing output.
107+
- Run `cargo test --features ...` if you introduce optional features.
108+
109+
### Integration Tests
110+
- Integration tests may use `testcontainers` to spin up ClickHouse for indexer tests.
111+
- Ensure Docker is running before executing integration tests.
112+
- Clean up resources in test teardown to avoid port conflicts and leaked containers.
113+
114+
### Performance Tests
115+
- Benchmark critical paths using Criterion in `benches/`.
116+
- Focus on hot paths: validation, signature verification, entity scoring, serialization.
117+
- Run `cargo bench` to execute all benchmarks and generate performance reports.
32118

33119
## Commit & Pull Request Guidelines
34-
Follow the short, imperative style seen in history (`fix package name`); prefix with scope if it aids triage (e.g. `ingress:`). Each PR should explain motivation, outline the approach, and call out risky areas. Link to BuilderNet issues when available and attach logs or screenshots for user-facing changes. Confirm CI-ready state by running build, fmt, clippy, and tests before requesting review.
120+
121+
### Commit Message Style
122+
- Follow the short, imperative style seen in history (e.g., `fix package name`, `add builder metrics`).
123+
- Prefix with scope if it aids triage: `ingress:`, `forwarder:`, `entity:`, `validation:`, `chore:`, `fix:`, `feat:`.
124+
- Examples:
125+
- `feat(forwarder): add retry logic for peer connections`
126+
- `fix(ingress): validate max txs per bundle`
127+
- `chore: bump version to 1.1.3`
128+
129+
### Pull Request Requirements
130+
Each PR should include:
131+
1. **Motivation**: Why is this change needed? What problem does it solve?
132+
2. **Approach**: High-level overview of the implementation strategy.
133+
3. **Risky Areas**: Call out potential edge cases, performance impacts, or breaking changes.
134+
4. **Testing**: Describe how the change was tested (unit tests, integration tests, manual testing).
135+
5. **Issue Links**: Reference BuilderNet issues when available (e.g., `Fixes #123`).
136+
6. **Evidence**: Attach logs, metrics screenshots, or before/after comparisons for user-facing changes.
137+
138+
### CI Readiness Checklist
139+
Before requesting review, ensure:
140+
- [ ] `cargo build` succeeds
141+
- [ ] `just fmt` produces no changes
142+
- [ ] `just clippy` passes with no warnings
143+
- [ ] `just test` passes all tests
144+
- [ ] Integration tests pass if touching ingress, indexer, or forwarder
145+
146+
### Review Process
147+
- Address reviewer feedback promptly.
148+
- Keep commits focused; avoid mixing refactoring with feature work.
149+
- Rebase on `main` before merging to maintain a clean history.
35150

36151
## Configuration & Runtime Notes
37-
The proxy is configured through CLI flags defined in `OrderflowIngressArgs`; document any new flag, default, or environment interaction. Keep defaults safe for local testing (loopback listeners, disabled BuilderHub) and update `--help` text whenever behavior shifts.
152+
153+
### CLI Configuration
154+
The proxy is configured through CLI flags defined in `src/cli.rs` (`OrderflowIngressArgs` struct):
155+
- **Required**: `--user-listen-url`, `--system-listen-url`, `--builder-name`
156+
- **Optional**: `--builder-url`, `--builder-hub-url`, `--orderflow-signer`, `--metrics`, etc.
157+
- All flags support environment variable overrides (e.g., `USER_LISTEN_ADDR`, `SYSTEM_LISTEN_ADDR`).
158+
159+
### Configuration Guidelines
160+
When adding new configuration:
161+
1. Document the flag in the struct's `#[arg(...)]` annotation with clear help text.
162+
2. Choose safe defaults for local testing (loopback listeners, disabled external services).
163+
3. Update `--help` text to reflect behavior changes.
164+
4. Add validation logic in `src/runner/` if the flag affects startup behavior.
165+
5. Update this document's Configuration section if the change is significant.
166+
167+
### Runtime Behavior
168+
- The proxy uses Tokio's multi-threaded runtime for async operations.
169+
- Jemalloc is the default allocator for better memory performance.
170+
- Graceful shutdown is coordinated through `src/tasks/` using `TaskExecutor` and `TaskManager`.
171+
- HTTP servers bind to configured addresses and serve JSON-RPC requests via Axum.
172+
173+
### Observability
174+
- **Logs**: Structured logging via `tracing` (text or JSON format with `--log.json`).
175+
- **Metrics**: Prometheus metrics exposed at `--metrics` endpoint (default: disabled).
176+
- **Tracing**: Use `RUST_LOG` environment variable for log level control (e.g., `RUST_LOG=info,buildernet_orderflow_proxy=debug`).
177+
178+
### Performance Considerations
179+
- Rate limiting is disabled by default; enable with `--enable-rate-limiting`.
180+
- Gzip compression is disabled by default; enable with `--http.enable-gzip`.
181+
- Indexing is optional; ClickHouse/Parquet indexers only run if configured.
182+
- Entity caches (order, signer) use LRU eviction with configurable TTLs.
183+
- Priority queues have configurable capacities and backoff strategies.
184+
185+
## Architecture & Design Patterns
186+
187+
### Data Flow Overview
188+
1. **Ingress** receives bundles/transactions via JSON-RPC over HTTP
189+
2. **Validation** checks transaction intrinsics, signatures, and bundle structure
190+
3. **Entity Scoring** computes priority based on sender reputation and bundle characteristics
191+
4. **Priority Queues** buffer bundles by priority level (High/Medium/Low)
192+
5. **Forwarder** distributes bundles to local builder and peer proxies
193+
6. **Indexer** persists bundles and receipts for analytics (optional)
194+
195+
### Common Patterns
196+
197+
#### Error Handling
198+
- Use `eyre::Result` for most fallible operations.
199+
- Use `thiserror` for domain-specific error types with context.
200+
- Log errors at appropriate levels: `error!` for unexpected failures, `warn!` for recoverable issues.
201+
- Return structured errors from public APIs; log internal errors with context.
202+
203+
#### Async Patterns
204+
- Spawn long-running tasks using `TaskExecutor` and `TaskManager` for graceful shutdown.
205+
- Use `tokio::select!` for concurrent operations with cancellation support.
206+
- Prefer bounded channels (`tokio::sync::mpsc`) over unbounded to prevent memory growth.
207+
- Use `Arc<RwLock>` or `Arc<Mutex>` for shared mutable state; prefer immutable sharing when possible.
208+
209+
#### Observability Patterns
210+
- Instrument public functions with `#[tracing::instrument]` for automatic span creation.
211+
- Use structured fields in log messages: `tracing::info!(entity = %addr, "processing bundle")`.
212+
- Emit metrics for critical operations: request counts, latencies, queue depths, error rates.
213+
- Add custom metrics in `src/metrics.rs` using the `metrics` crate.
214+
215+
#### Testing Patterns
216+
- Mock external dependencies using trait abstractions (see `PeerStore` trait in `builderhub.rs`).
217+
- Use builder patterns for complex test fixtures.
218+
- Leverage `proptest` for property-based testing of validation logic.
219+
- Use `testcontainers` for integration tests requiring external services.
220+
221+
### Key Abstractions
222+
223+
- **`Entity`**: Represents a sender identified by signing address; tracks reputation and spam detection.
224+
- **`Priority`**: Enum for bundle priority levels (High, Medium, Low).
225+
- **`PriorityQueues`**: Multi-level queue system with per-level capacities and backoff.
226+
- **`Indexer`**: Trait for persisting bundles/receipts (implementations: ClickHouse, Parquet).
227+
- **`PeerStore`**: Trait for peer discovery (implementations: BuilderHub HTTP, local fallback).
228+
229+
### Dependencies & Ecosystem
230+
231+
Key dependencies and their roles:
232+
- **`alloy`**: Ethereum primitives (transactions, signatures, addresses)
233+
- **`rbuilder-primitives`**: BuilderNet-specific bundle types
234+
- **`axum`**: HTTP framework for JSON-RPC APIs
235+
- **`tokio`**: Async runtime
236+
- **`tracing`**: Structured logging and observability
237+
- **`metrics`**: Prometheus-compatible metrics collection
238+
- **`clickhouse`**: ClickHouse client for indexing
239+
- **`parquet`**: Parquet file format for indexing
240+
- **`moka`**: Async-aware LRU caching
241+
- **`criterion`**: Performance benchmarking
242+
243+
### Performance Hotspots
244+
Monitor and optimize these areas:
245+
1. **Signature Verification**: Uses `alloy` for ECDSA recovery; consider batching.
246+
2. **Entity Scoring**: Computed per bundle; cache aggressively.
247+
3. **Validation**: Runs on every transaction; keep logic fast and allocate minimally.
248+
4. **Serialization**: JSON-RPC parsing and encoding; consider message size limits.
249+
5. **Queue Operations**: High contention on shared queues; use per-level locks.
250+
251+
### Security Considerations
252+
- Validate all user inputs before processing (see `validation.rs`).
253+
- Rate limit by entity to prevent spam and DoS attacks.
254+
- Reject bundles from entities with low scores (spam detection).
255+
- Use TLS for peer communication (BuilderHub integration).
256+
- Sign bundles when forwarding to peers for authentication.
257+
- Never log private keys or sensitive configuration values.

‎README.md‎

Lines changed: 18 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,19 @@
11
# buildernet-orderflow-proxy-v2
2+
This is the second version of the [BuilderNet](https://buildernet.org) orderflow proxy, designed to be highly performant, reliable and efficient. The proxy is responsible for receiving, validating, prioritizing, and forwarding orderflow to all BuilderNet builders. It is built in Rust and uses the [Tokio](https://tokio.rs) asynchronous runtime.
23

3-
## Clickhouse
4+
## CLI
5+
```
6+
cargo run -- --help
7+
```
8+
9+
## Repository Structure
10+
- [`src/`](src/): Source code.
11+
- [`tests/`](tests/): Integration & e2e tests.
12+
- [`fixtures/`](fixtures/): Clickhouse database fixtures for indexing orders.
13+
- [`benches/`](benches/): Criterion benchmarks for performance testing.
14+
- [`simulation/`](simulation/): Simulation harness with [Shadow](https://shadow.github.io/) for testing the proxy at scale.
415

16+
## Provisioning Clickhouse
517
Install the Clickhouse client:
618
```bash
719
curl https://clickhouse.com/ | sh
@@ -12,23 +24,15 @@ Copy the example environment variables and fill in the values:
1224
cp .env.example .env
1325
```
1426

15-
Connect to Clickhouse:
27+
Then you can use the `just` commands to provision the database and create the tables:
1628
```bash
17-
source .env
18-
./clickhouse client --host $CLICKHOUSE_HOST --user $CLICKHOUSE_USER --secure --password $CLICKHOUSE_PASSWORD
29+
just provision-db
1930
```
2031

21-
Create the database:
22-
```sql
23-
CREATE DATABASE IF NOT EXISTS buildernet_orderflow_proxy COMMENT 'Buildernet orderflow proxy database';
24-
```
25-
Or
2632
```bash
27-
source .env
28-
./clickhouse client --host $CLICKHOUSE_HOST --user $CLICKHOUSE_USER --secure --password $CLICKHOUSE_PASSWORD --query "CREATE DATABASE IF NOT EXISTS buildernet_orderflow_proxy COMMENT 'Buildernet orderflow proxy database';"
33+
just reset-db
2934
```
3035

31-
Create the bundles table:
3236
```bash
33-
./clickhouse client --host $CLICKHOUSE_HOST --user $CLICKHOUSE_USER --secure --password $CLICKHOUSE_PASSWORD -d buildernet_orderflow_proxy --queries-file ./fixtures/create_bundles_table.sql
34-
```
37+
just extract-data "filename.parquet"
38+
```

0 commit comments

Comments
 (0)