Data Sources

Supported data types, collection pipeline, and storage architecture

Overview

ClyptQ supports four categories of data that can be combined as inputs to any trading strategy graph. All data flows through the same pipeline: declare what you need in TradingDataSpec, let the DataProvider fetch and normalize it, and consume it as FIELD inputs in the operator graph.

Supported Data Types

Category	Source	Granularity	Key Fields	Status
OHLCV (Crypto)	Binance, Bybit, Coinbase, OKX, Gate.io, Kraken, Hyperliquid, Aster	Tick to 1d	close, open, high, low, volume	Production
Funding Rates	Binance, Bybit, Coinbase, Kraken, Hyperliquid, Aster	1h to 8h	funding_rate	Production
Futures Metrics	Binance, Bybit, Gate.io	5m to 1h	OI, L/S ratio, liquidations	Production
Onchain (DeFiLlama)	DeFiLlama API (27 chains)	1d	TVL, DEX volume, fees, stablecoin supply	Production
Onchain (RPC)	Alchemy archive, native RPCs (16 chains)	1h	Gas, DeFi rates, exchange flow, whale transfers	Production
Macro (FRED)	Federal Reserve FRED API (38 series)	Daily to quarterly	Interest rates, CPI, GDP, VIX, money supply	Production
Sentiment	Fear & Greed Index	1d	Market sentiment score	Production
US Stocks	Tiingo, Alpaca, IBKR, SEC EDGAR	1d	OHLCV, fundamentals (40+ fields)	Developed (not in production)
Order Book	Binance (historical ZIP)	Tick / 1s	Bid/ask depth, spread, imbalance	Developed (not in production)

Data Flow

TradingDataSpec
    |
    |-- symbol_source_map (which symbols, which exchange)
    |-- observations      (OHLCVSpec, OnChainSpec, MacroSpec, ...)
    |-- start / end       (date range)
    |
    v
DataProvider
    |-- Routes to correct Collector (Binance, FRED, DeFiLlama, ...)
    |-- Fetches historical or subscribes to live stream
    |-- Normalizes to unified DataFrame schema
    |
    v
StatefulGraph FIELD Inputs
    |-- FIELD:binance:futures:ohlcv:close    (T, N) RollingBuffer
    |-- FIELD:fred:macro:DFF                 (T, 1) forward-filled
    |-- FIELD:defillama:onchain:tvl          (T, N) daily
    |
    v
Operators consume FIELDs via Input(source="FIELD:...", lookback=N)

Declaring Data Sources

OHLCV (Crypto)

from clyptq.apps.trading.spec.observation.crypto import OHLCVSpec

ohlcv = OHLCVSpec(
    exchange="binance",
    market_type="futures",
    timeframe="1m",
)

Multiple Exchanges

observations = [
    OHLCVSpec(exchange="binance", market_type="futures", timeframe="1m"),
    OHLCVSpec(exchange="bybit", market_type="linear", timeframe="1m"),
]

Onchain + Macro

from clyptq.apps.trading.spec.observation.onchain_global import OnChainSpec
from clyptq.apps.trading.spec.observation.alternative import MacroIndicatorSpec as MacroSpec

onchain = OnChainSpec(
    data_type="tvl",
    chains=["ethereum", "solana", "arbitrum"],
)

macro = MacroSpec(
    indicators=["DFF", "T10Y2Y", "VIXCLS"],
)

Storage

ClyptQ uses a layered storage system. Collectors write to storage; the graph engine reads from it during warmup and backtest.

Layer	Format	Use Case
Local Parquet	.parquet files on disk	Default for development and backtesting
S3	Parquet files in S3 buckets	Production, shared team data
API / Live	WebSocket streams, REST polling	Live trading, no storage needed

Chunk-Based Loading

For large datasets (multi-year, many symbols), the data system uses chunk-based loading to avoid memory exhaustion:

• Historical data is partitioned by time chunks (e.g., monthly Parquet files)
• The backtest engine loads only the chunk needed for the current simulation window
• Warmup buffers pre-load enough history for the longest lookback in the graph
• Garbage collection runs periodically between chunks to free memory

This means you can backtest over years of 1-minute data across hundreds of symbols without loading everything into RAM at once.

Collector Architecture

Every data source implements the UnifiedCollector interface:

class UnifiedCollector:
    def collect_historical(symbols, start, end, timeframe) -> Dict[str, DataFrame]
    def get_latest(symbols, timeframe) -> Dict[str, Dict]
    def subscribe(symbols, on_data, on_error, timeframe) -> Subscription
    def collect_and_save(symbols, start, end, timeframe) -> Dict[str, int]

This unified interface means all data sources — whether exchange candles, onchain metrics, or macro indicators — are consumed the same way by the graph engine.

Related Pages

• Onchain Data — DeFiLlama and RPC collector details
• FRED Macro Data — Federal Reserve economic indicators
• Data System (Engine) — How the graph engine consumes data