Skip to content
This repository was archived by the owner on Jun 16, 2026. It is now read-only.

full-node.md

Latest commit

 

History

History
100 lines (77 loc) · 3.27 KB

full-node.md

File metadata and controls

100 lines (77 loc) · 3.27 KB
Raw

Full Node Architecture

Purpose: Full node, distributed workers, aggregation.
Source: snapshotter-core-edge.

Overview

The full node (snapshotter-core-edge) processes all pools per epoch (no slot selection). It uses a distributed architecture with multiple services:

  • Block Fetcher: Fetches and caches blocks.
  • TX Processor: Processes transactions, extracts events.
  • Epoch Syncer: Monitors ProtocolState for EpochReleased, triggers processing.
  • Processor Distributor: Coordinates work distribution.
  • Snapshot Workers: Process base snapshots (one per project type).
  • Aggregation Workers: Build aggregates from base snapshots.
  • Cacher Worker: Manages snapshot caching, Redis state, IPFS unpinning.

Data Pipeline

1. Epoch Syncer detects EpochReleased
    ↓
2. Processor Distributor receives event
    ↓
3. Preloaders execute (shared data preparation)
    ↓
4. Snapshot Workers process each project type
   - Load compute module from /computes/
   - Call compute() for ALL pools (no slot filter)
   - Store to IPFS, submit via Local Collector
    ↓
5. Aggregation Workers (after base snapshots finalized)
   - Fetch base snapshots from IPFS
   - Aggregate into higher-order snapshots (e.g., 24h stats, 7d top pairs)
   - Submit via Local Collector
    ↓
6. Cacher Worker
   - Updates Redis with snapshot CIDs
   - Manages aggregated data structures
   - IPFS unpinning schedules

Compute Module Interface (Full Node)

Full node uses GenericProcessorSnapshot from snapshotter.utils.callback_helpers:

async def compute(
    self,
    epoch: SnapshotProcessMessage,
    redis_conn: aioredis.Redis,
    rpc_helper: RpcHelper,
    anchor_rpc_helper: RpcHelper,
    ipfs_reader: AsyncIPFSClient,
    protocol_state_contract,
    task_type: str = None,
) -> List[Tuple[str, BaseSnapshot]]:
  • redis_conn: Redis for caching, state.
  • task_type: Distinguishes base vs aggregation tasks.
  • Returns: List of (project_data_source, snapshot) tuples.

Code: snapshotter-core-edge/snapshotter/utils/callback_helpers.py — GenericProcessorSnapshot.

Project Configuration

  • projects.json: Base snapshot tasks. Each project has processor.module, processor.class_name.
  • aggregator.json: Aggregation tasks. References base projects, defines aggregation logic.
  • preloader.json: Preloaders run before project processors; results passed via preloader_results.

Full Node vs Lite Node

Aspect Full Node Lite Node
Pools per epoch All One (slot-selected)
Architecture Distributed (Redis, Dramatiq) Single process
Compute load High Low (~12% of epochs)
Use case Complete data, aggregation Decentralized participation
Submission Same (gRPC → Collector → Gossipsub) Same

Key Paths

  • Processor Distributor: snapshotter-core-edge/snapshotter/processor_distributor.py
  • Snapshot Worker: snapshotter-core-edge/snapshotter/utils/snapshot_worker.py
  • Aggregation Worker: snapshotter-core-edge/snapshotter/utils/aggregation_worker.py
  • Generic Worker: snapshotter-core-edge/snapshotter/utils/generic_worker.py
  • Config: snapshotter-core-edge/config/projects.json, aggregator.json, preloader.json