IOTA

Incentivized Orchestrated Training Architecture (IOTA) is a framework for pretraining large language models across a network of heterogeneous, unreliable, permissionless and token incentivized machines. IOTA employs a data- and pipeline-parallel architecture to accelerate training and reduce hardware requirements for participants.

Overview

• The orchestrator distributes model layers across heterogeneous miners and streams activations between them.
• All network communication is mediated via the orchestrator, and a shared S3 bucket is used to store activations and layer weights.
• Miners compete to process as many activations as possible in the training stage.
• Miners periodically upload their local weights and merge their activations using a variant of Butterfly All-Reduce.
• Validators spot-check miners to ensure that work was performed as required.

For a more comprehensive overview, please refer to our technical paper here. You can also find the report on ArXiv

Current Run Information

• 1.5B parameter Llama-inspired architecture with uninterrupted residual flow (see paper for details)
• 3 layers, breaking the model into 3 distinct training sections (1 head, 1 tail, 1 body)

Future Run Information

1. Scaling the system to 15B, 50B, and 100B models
2. More advanced compression techniques to speed up training

Comprehensive Dashboard

Visualizing the state of the network, the number of miners, layers, and general metrics is paramount to understanding the training process. We provide a comprehensive dashboard here.

Installation

1. First install uv (https://docs.astral.sh/uv/)
2. Run bash setup.sh and choose Miner or Validator
3. Configure your .env file

Additional Miner Documentation

Running the miner is as easy as bash ./start_miner.sh. For more information, reference the official miner docs.

Use PM2 to run the miner in the background: pm2 start pm2/miner.config.js

Additional Validation Documentation

Running the validator ./start_validator.sh. For more information, reference the official validator docs

Use PM2 to run the validator in the background: pm2 start pm2/validator.config.js

Compute Requirements

The runs are currently in bfloat16, resulting in a total footprint of ~2GB for a 1B parameter model. As such, we recommend:

1. Cuda GPU with >= 16GB VRAM (RTX 4090, for example).
2. Ubuntu 22.04 (Jammy)