As Builderby, our mission is to construct a comprehensive, real-time OHLCV (Open, High, Low, Close, Volume) data platform for potentially all SPL tokens on Solana. This document outlines the core concepts we'll be working with and how we plan to tackle the inherent challenges.
To achieve our goal, Builderby will master and integrate the following foundational elements of the Solana ecosystem:
- SPL Tokens: These are the lifeblood of Solana's DeFi. Builderby's system will identify, track, and process data for this diverse range of fungible tokens, all adhering to the Solana Program Library (SPL) Token standard.
- Decentralized Exchanges (DEXs): These on-chain protocols (like Raydium, Orca, OpenBook, etc.) are where SPL token trading occurs. Builderby will interface directly with these DEXs to capture raw trade data.
- Liquidity Pools (LPs) / Automated Market Makers (AMMs): The dominant model on Solana DEXs involves AMMs, where liquidity is provided in pools. Builderby's data ingestors will monitor activity within these pools to understand token swaps.
- Price Discovery: We recognize that token prices are not globally fixed but are determined by trade activity within specific DEX markets. Builderby will accurately capture this market-specific price discovery.
- Transaction Logs & Events: Every trade on a Solana DEX is a transaction that emits vital information through logs or Anchor events. Builderby's ingestion layer will be engineered to meticulously parse these to extract details like tokens swapped, amounts, and involved accounts.
- Solana RPC Nodes: These are Builderby's indispensable gateways to the Solana blockchain. We will deploy and manage a robust, likely self-hosted, RPC node infrastructure. This will allow us to:
- Subscribe to real-time logs from target DEX programs using WebSockets.
- Efficiently fetch detailed transaction data.
- Accurately retrieve block times for timestamping.
- Query account information, such as pool states and token mint details.
- OHLCV Data Generation: The core output of Builderby's platform. We will calculate:
- Open: Price of the first trade for a pair in a defined time interval.
- High: Highest trade price observed within that interval.
- Low: Lowest trade price observed within that interval.
- Close: Price of the last trade within that interval.
- Volume: Total quantity of the base asset traded during the interval. Builderby will generate this data for various token pairs (e.g., BONK/SOL) across multiple configurable time intervals (e.g., 1m, 5m, 1h).
- Data Ingestion Pipeline: Builderby will design and implement a resilient data ingestion pipeline. This system will connect to our Solana RPC nodes, listen to targeted DEX program activity, parse the resulting logs/events, and extract raw, validated trade data.
- Data Processing & Aggregation: Raw trades will flow into Builderby's processing layer, where they will be efficiently aggregated into the structured OHLCV candles.
- Scalable Data Storage: Builderby will utilize a two-pronged database strategy: a time-series database (like TimescaleDB) for high-volume OHLCV data and a relational database (like PostgreSQL) for token and market metadata, ensuring both query performance and data integrity.
- API Provisioning: Builderby will develop a clean, well-documented API (likely using Express.js/Fastify/FastAPI) to provide users with access to the aggregated OHLCV data and relevant metadata of tokens, including top trending tokens all with a affordable plan pricing.
- Dynamic Token & Market Discovery: A critical component for achieving broad token coverage. Builderby will develop sophisticated mechanisms to continuously discover new SPL tokens and the DEX markets where they gain liquidity.
- **We will develop a system to automatically detect harmful/rug/scammy tokens and will label them accurately to ensure we keep the Solana and Crypto community safe.
We are fully aware of the complexities involved. Here's how Builderby plans to address the key challenges:
- DEX Integration Complexity:
- The Solana ecosystem hosts a variety of DEXs, each with unique on-chain programs and log structures.
- Builderby's Approach: We will develop a modular and extensible parsing framework. This will involve creating dedicated parsers for major DEXs (e.g., Raydium, Meteora, Orca) and their different pool types (AMM, CLMM, DLMM, DAMMv2). Our design will prioritize adaptability, allowing us to integrate new DEXs or updated program versions with manageable effort. We will leverage Anchor IDLs where available and employ advanced log analysis techniques for others.
- Token and Market Discovery (The "Any Token" Ambition):
- There's no central on-chain registry for all tokens or DEX markets.
- Builderby's Approach: We will implement a multi-faceted discovery service. This service will actively monitor known DEX programs for new pool creation events. We'll also integrate reputable token lists and explore on-chain analytics to identify emerging tokens and trading venues. This continuous discovery process is key to expanding our coverage towards the "any token" goal.
- Scalability & Performance:
- The sheer volume of Solana transactions and the need for real-time processing demand a high-performance architecture.
- Builderby's Approach: Our system is being designed from the ground up for scalability. This includes horizontally scalable ingestor and processor services, optimized database schemas and queries for TimescaleDB, and efficient management of our RPC node cluster to handle high loads. We will use message queues (like Kafka or Redis Streams) to decouple services and manage backpressure.
- Data Reliability & Accuracy:
- Ensuring data completeness and correctness is paramount.
- Builderby's Approach: We will implement robust error handling, retry mechanisms, and data validation checks throughout our pipeline. This includes strategies to minimize missed trades/swaps due to RPC issues, rigorous testing of log parsers, and careful handling of token decimals for accurate price/volume calculations. We will also monitor for and handle potential (though rare) blockchain reorganizations.
- Infrastructure Costs & Management:
- Self-hosting RPC nodes and managing database servers incurs operational overhead.
- Builderby's Approach: While self-hosted RPC nodes are preferred for performance and reliability, we will continuously evaluate cost-effectiveness. We will leverage containerization (Docker) and orchestration (Kubernetes, if feasible) to streamline deployment, management, and scaling of our infrastructure components.
- Historical Data Backfilling:
- Populating historical OHLCV data is RPC-intensive and requires significant processing.
- Builderby's Approach: We will develop dedicated backfilling scripts that efficiently use Solana's
getSignaturesForAddressandgetParsedTransactionRPC methods. This process will be designed to be resumable and to run in parallel where possible, with careful consideration for RPC rate limits and data storage growth.
- Maintenance & Adaptability:
- The Solana ecosystem is dynamic; DEXs upgrade, and new ones emerge.
- Builderby's Approach: We will establish a proactive monitoring system for DEX program updates. Our modular architecture is designed to facilitate quicker updates to parsers and the integration of new data sources. Continuous improvement and adaptation will be core to Builderby's operational philosophy.
By systematically addressing these concepts and challenges with well-thought-out strategies, Builderby is confident in its ability to deliver a valuable and reliable OHLCV data platform for the Solana, Crypto community to use.