> ## Documentation Index > Fetch the complete documentation index at: https://docs.clypt.ai/llms.txt > Use this file to discover all available pages before exploring further. # Rust Core Engine > Migrating the compute engine to Rust for PyTorch-level performance with Python research ergonomics ## Vision ClyptQ's engine will be rewritten in **Rust** with **Python bindings via PyO3** — following the same architecture as PyTorch (C++/CUDA core, Python research interface). Builders write strategies in Python. The engine executes them at native speed. ```mermaid theme={null} %%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#1e293b', 'primaryTextColor': '#e2e8f0', 'primaryBorderColor': '#94F1E8', 'lineColor': '#5bb8b0', 'clusterBkg': '#0f172a', 'clusterBorder': '#1e3a3a', 'titleColor': '#94F1E8', 'edgeLabelBackground': 'transparent' }}}%% flowchart TB classDef highlight fill:#134e4a,stroke:#94F1E8,stroke-width:2.5px,color:#94F1E8 classDef action fill:#172554,stroke:#60a5fa,stroke-width:1.5px,color:#bfdbfe A["Python Layer
Strategy research, Jupyter notebooks,
operator definitions, TradingSpec"] B["PyO3 Bindings
Zero-copy numpy ↔ Rust array bridge"] C["Rust Core Engine
Graph execution, RollingBuffer,
TaggedArray ops, order matching,
risk management, data I/O"] A --> B --> C class B highlight class C action ``` **Goal:** All PyTorch-equivalent functionality in Rust. Python does research. Rust does computation. ## Why Rust? ### Current Bottlenecks (Python) | Component | Current | Issue | | ------------------------------- | ----------------- | ------------------------------------------------- | | Graph tick loop | Python `for` loop | GIL-bound, single-threaded | | TaggedArray operations | NumPy | Good for vectorized, slow for per-tick scalar ops | | RollingBuffer | NumPy arrays | Memory copies on `to_array()` reordering | | Order matching | Python classes | Object allocation per order, per tick | | Backtest (1yr, 100 symbols, 1m) | \~30 minutes | Too slow for parameter sweeps | ### Target Architecture (Rust) | Component | Target | Benefit | | ---------------------- | ------------------------- | ---------------------------- | | Graph tick loop | Rust `tokio` async loop | No GIL. True parallelism | | TaggedArray operations | `ndarray` + SIMD | Faster element-wise ops | | RollingBuffer | Zero-copy circular buffer | No reallocation. No copies | | Order matching | Stack-allocated structs | No heap allocation per order | ### Rust vs Alternatives | Option | Pros | Cons | | --------------- | ---------------------------------------------------------------------------- | --------------------------------------------------- | | **Rust + PyO3** | Zero-cost abstractions, memory safety, no GC, proven (PyTorch, Polars, Ruff) | Learning curve for contributors | | C++ + pybind11 | Mature ecosystem, CUDA support | Memory bugs, undefined behavior, complex build | | Cython | Easy Python integration | Still limited by GIL, modest speedup | | Numba/JAX | JIT compilation | Can't optimize control flow, limited to numeric ops | Rust is the clear choice: **memory safety without GC**, **zero-cost abstractions**, and a proven track record in Python ecosystem tools (Polars, Ruff, Pydantic V2, cryptography). ## Migration Strategy ### Phase 1: Hot Path (TaggedArray + RollingBuffer) **Target: TBD** (deferred — no Rust code written yet) The innermost loop — TaggedArray operations and buffer management — moves to Rust first. ```rust theme={null} // Rust: TaggedArray with SIMD operations #[pyclass] pub struct TaggedArray { pub value: Array1, pub exists: Array1, pub valid: Array1, pub updated: Array1, } #[pymethods] impl TaggedArray { /// Element-wise multiply with mask propagation fn mul(&self, other: &TaggedArray) -> TaggedArray { TaggedArray { value: &self.value * &other.value, exists: &self.exists & &other.exists, valid: &self.valid & &other.valid, updated: &self.updated | &other.updated, } } /// Combined mask: exists & valid #[getter] fn is_valid(&self) -> Array1 { &self.exists & &self.valid } } ``` ```rust theme={null} // Rust: Zero-copy RollingBuffer #[pyclass] pub struct RollingBuffer { value: Array2, exists: Array2, valid: Array2, updated: Array2, write_idx: usize, lookback: usize, } #[pymethods] impl RollingBuffer { /// O(n_symbols) append — no copies fn append(&mut self, tick: &TaggedArray) { let idx = self.write_idx % self.lookback; self.value.row_mut(idx).assign(&tick.value); self.exists.row_mut(idx).assign(&tick.exists); self.valid.row_mut(idx).assign(&tick.valid); self.updated.row_mut(idx).assign(&tick.updated); self.write_idx += 1; } /// Zero-copy view when possible, single concat when wrapped fn to_array(&self) -> TaggedArray { // If buffer hasn't wrapped, return view directly // If wrapped, single contiguous reorder } } ``` **Python interface unchanged:** ```python theme={null} # Builder code stays exactly the same class MyAlpha(Operator): def compute(self, data, timestamp, context): close = data["close"] # Now a Rust TaggedArray returns = close.value[-1] / close.value[0] - 1.0 # NumPy interop return TaggedArray(value=returns, exists=close.exists[-1], ...) ``` ### Phase 2: Graph Execution Engine **Target: TBD** (depends on Phase 1 completion) The `StatefulGraph.on_tick()` loop moves to Rust. Operators are still defined in Python, but the graph orchestration (buffer management, topological execution, input distribution) runs in Rust. ```rust theme={null} // Rust: Graph execution loop pub struct RustGraph { nodes: Vec, topo_order: Vec, buffers: HashMap<(usize, String), RollingBuffer>, } impl RustGraph { /// Execute one tick across all nodes pub fn on_tick(&mut self, field_data: &FieldInputs, py: Python) -> PyResult<()> { // 1. Distribute FIELD inputs to consumer buffers (Rust) self.distribute_fields(field_data); // 2. Execute operators in topological order for &node_idx in &self.topo_order { let node = &self.nodes[node_idx]; // Prepare inputs from buffers (Rust) let inputs = self.prepare_inputs(node_idx); // Call Python operator.compute() via PyO3 let output = node.operator.call_method1(py, "compute", (inputs, timestamp, context))?; // Store output in downstream buffers (Rust) self.distribute_output(node_idx, output); } Ok(()) } } ``` ### Phase 3: Built-in Operators in Rust **Target: TBD** (depends on Phase 2 completion) Common operators (SMA, EMA, RSI, MACD, all 101 Alpha operators) get Rust implementations. Python definitions become thin wrappers. ```rust theme={null} // Rust: SMA operator pub struct SMA { span: usize, } impl Operator for SMA { fn compute(&self, data: &InputData) -> TaggedArray { let close = data.get("close"); let n = close.value.shape()[0]; let n_symbols = close.value.shape()[1]; let mut result = Array1::::zeros(n_symbols); let mut valid = Array1::::from_elem(n_symbols, false); if n >= self.span { for s in 0..n_symbols { if close.is_valid.slice(s![n-self.span..n, s]).iter().all(|&v| v) { result[s] = close.value.slice(s![n-self.span..n, s]).mean().unwrap(); valid[s] = true; } } } TaggedArray { value: result, exists: close.exists.row(n-1).to_owned(), valid, .. } } } ``` ```python theme={null} # Python: Thin wrapper auto-selects Rust implementation class SMA(Operator): def __init__(self, span: int): self.span = span self._rust_impl = _clyptq_rs.SMA(span) # Rust backend def compute(self, data, timestamp, context): return self._rust_impl.compute(data) # Delegated to Rust ``` ### Phase 4: Order Matching & Risk Engine **Target: 2026 H2** The execution pipeline (intention → delta → order → fill) moves to Rust with zero-allocation order processing. ```rust theme={null} // Rust: Stack-allocated order processing pub struct OrderEngine { positions: HashMap, cash: f64, pending_orders: Vec, } impl OrderEngine { /// Process intention → order with zero heap allocation pub fn process_intention(&mut self, intention: &TradingIntention) -> Option { let current_pos = self.positions.get(&intention.symbol) .map(|p| p.quantity).unwrap_or(0.0); let delta = intention.target_position - current_pos; if delta.abs() < MIN_ORDER_SIZE { return None; } Some(Order { symbol: intention.symbol.clone(), side: if delta > 0.0 { Side::Buy } else { Side::Sell }, quantity: delta.abs(), .. }) } } ``` ### Phase 5: Data I/O & Storage **Target: 2027** Parquet reading/writing and live data WebSocket handling in Rust. Uses `arrow-rs` for zero-copy Parquet I/O. ```rust theme={null} // Rust: Arrow-based Parquet reader pub fn load_ohlcv(path: &Path, start: DateTime, end: DateTime) -> OHLCVData { let file = File::open(path).unwrap(); let reader = ParquetRecordBatchReaderBuilder::try_new(file) .unwrap() .with_row_filter(/* timestamp range */) .build() .unwrap(); // Zero-copy Arrow → numpy via PyO3 } ``` ## PyTorch Analogy | PyTorch | ClyptQ (Planned) | | ----------------------------- | ------------------------------------------- | | `torch.Tensor` (C++/CUDA) | `TaggedArray` (Rust) | | `torch.nn.Module` (Python) | `Operator` (Python) | | `torch.autograd` engine (C++) | `StatefulGraph.on_tick()` (Rust) | | `torch.optim` (Python + C++) | `Optimizer` operators (Python + Rust) | | `DataLoader` (Python + C++) | `TradingDriver.load_data()` (Python + Rust) | | Python for research/training | Python for strategy research | | C++/CUDA for execution | Rust for backtesting + live execution | **Key principle:** Just as PyTorch lets ML researchers write Python while CUDA handles the heavy lifting, ClyptQ lets quant researchers write Python strategies while Rust handles tick-by-tick execution. ## Developer Experience ### For Strategy Builders (No Change) ```python theme={null} # This code works identically before and after Rust migration class MomentumAlpha(Operator): role = OperatorRole.ALPHA def compute(self, data, timestamp, context): close = data["close"] returns = (close.value[-1] / close.value[0]) - 1.0 return TaggedArray( value=returns, exists=close.exists[-1], valid=close.valid[-1], updated=True ) ``` ### For Performance-Critical Operators (Optional Rust) ```python theme={null} # Builders CAN write Rust operators for maximum performance from clyptq_rs import register_rust_operator @register_rust_operator("path/to/my_operator.rs") class MyFastOperator(Operator): pass # Implementation in Rust file ``` ### For Data Research (Full Python) ```python theme={null} # Jupyter notebook research stays pure Python import clyptq as cq import pandas as pd import matplotlib.pyplot as plt # Load data (Rust Parquet reader under the hood) data = cq.load_ohlcv("binance:futures", ["BTC/USDT"], "2024-01-01", "2024-12-31") # Research with pandas/numpy/scipy (pure Python) returns = data.close.pct_change() rolling_sharpe = returns.rolling(252).mean() / returns.rolling(252).std() # Backtest (Rust engine under the hood) results = cq.backtest(spec) # Analyze results (pure Python) results.plot_equity_curve() ``` ## Timeline | Phase | Component | Target | | ----------- | ----------------------------------------- | ------- | | **Phase 1** | TaggedArray + RollingBuffer | 2025 H2 | | **Phase 2** | Graph execution engine | 2026 H1 | | **Phase 3** | Built-in operators (SMA, RSI, Alpha\_101) | 2026 H1 | | **Phase 4** | Order matching + risk engine | 2026 H2 | | **Phase 5** | Data I/O + storage | 2027 | ## Technology Stack | Layer | Technology | Role | | ------------------- | ---------------------------------------- | ---------------------------------------------- | | **Core compute** | Rust + `ndarray` | Array operations, SIMD | | **Python bindings** | PyO3 + `numpy` crate | Zero-copy numpy ↔ Rust bridge | | **Async runtime** | `tokio` | Concurrent tick processing, WebSocket handling | | **Serialization** | `serde` + `arrow-rs` | Parquet I/O, message passing | | **Testing** | `criterion` (Rust benchmarks) + `pytest` | Performance regression + correctness | | **Build** | `maturin` | Rust → Python wheel packaging |