> ## Documentation Index
> Fetch the complete documentation index at: https://docs.clypt.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Lookback Buffers

> Pre-allocated circular buffers that manage historical data for operators

## Why Lookback Buffers?

Most technical indicators need historical data. An SMA(20) needs 20 ticks of history. An RSI(14) needs 15 ticks. In vectorized frameworks, this means slicing into a large dataframe — which introduces lookahead risk and wastes memory.

ClyptQ takes a different approach: each operator input gets a **pre-allocated circular buffer** of exactly the size it needs. No more, no less.

## RollingBuffer

`RollingBuffer` is a fixed-size circular buffer that stores [TaggedArray](/engine/tagged-array) history:

```python theme={null}
# Created internally by the graph for each Input
buffer = RollingBuffer(lookback=20, n_symbols=3)
```

### Pre-allocated Arrays

On creation, 4 numpy arrays are allocated:

| Array     | Dtype     | Shape                   | Initial Value |
| --------- | --------- | ----------------------- | ------------- |
| `value`   | `float64` | `(lookback, n_symbols)` | `0.0`         |
| `exists`  | `bool`    | `(lookback, n_symbols)` | `False`       |
| `valid`   | `bool`    | `(lookback, n_symbols)` | `False`       |
| `updated` | `bool`    | `(lookback, n_symbols)` | `False`       |

No dynamic allocation during execution. The memory footprint is fixed at `lookback × n_symbols × (8 + 3)` bytes (float64 + 3 bools).

## Circular Write

New data is written at `write_idx % lookback`, then `write_idx` increments:

```
Tick 1: write at index 0    [X _ _ _ _]  write_idx=1
Tick 2: write at index 1    [X X _ _ _]  write_idx=2
Tick 3: write at index 2    [X X X _ _]  write_idx=3
  ...
Tick 5: write at index 4    [X X X X X]  write_idx=5  (buffer full)
Tick 6: write at index 0    [Y X X X X]  write_idx=6  (overwrites oldest)
Tick 7: write at index 1    [Y Y X X X]  write_idx=7
```

The oldest data is automatically overwritten — no shifting, no copying, no reallocation.

### append()

Every tick, the graph calls `buffer.append(tick)`:

```python theme={null}
def append(self, tick: TaggedArray):
    idx = self.write_idx % self.lookback
    self.value[idx]   = tick.value
    self.exists[idx]  = tick.exists
    self.valid[idx]   = tick.valid
    self.updated[idx] = tick.updated
    self.write_idx += 1
```

This is O(n\_symbols) per append — just array assignment.

## Reading Data

### get\_last(): Most Recent Tick

Returns the single most recent tick as a 1D TaggedArray:

```python theme={null}
last = buffer.get_last()
# Returns TaggedArray at (write_idx - 1) % lookback
# Shape: (n_symbols,)
```

Used when `lookback=1` — the operator only needs the current value.

### to\_array(): Full Buffer as 2D

Returns the entire buffer as a 2D TaggedArray, ordered oldest-to-newest:

```python theme={null}
history = buffer.to_array()
# Shape: (lookback, n_symbols) if buffer is full
# Shape: (write_idx, n_symbols) if buffer is not yet full
```

Since the buffer is circular, the raw array may be out of order. `to_array()` reorders using `np.concatenate`:

```python theme={null}
# If buffer is full and start != 0:
start = self.write_idx % self.lookback
ordered = np.concatenate([
    self.value[start:],    # Older data
    self.value[:start]     # Newer data
])
# Result: chronological order, oldest first
```

## Per-Consumer Buffers

Every (consumer, source) pair gets its own RollingBuffer. This is crucial:

```python theme={null}
# sma_20 needs 20 ticks of close
graph.add_node("sma_20", SMA(input=close, period=20),
    inputs=[Input("FIELD:binance:futures:ohlcv:close", "1m", lookback=20)])

# rsi_14 needs 15 ticks of close
graph.add_node("rsi_14", RSI(period=14),
    inputs=[Input("FIELD:binance:futures:ohlcv:close", "1m", lookback=15)])
```

This creates **two separate buffers**:

```
Buffer ("sma_20", "FIELD:binance:futures:ohlcv:close"): lookback=20
Buffer ("rsi_14", "FIELD:binance:futures:ohlcv:close"): lookback=15
```

Both receive the same FIELD data (appended by `_distribute_source_inputs`), but each maintains its own circular buffer with its own lookback size. SMA sees 20 ticks, RSI sees 15.

## Warmup: When Buffers Are Ready

A buffer needs `lookback` ticks before it's full. The graph tracks this automatically:

```python theme={null}
warmup = graph.compute_warmup_per_field()
# {"FIELD:binance:futures:ohlcv:close": 22}
# 22 ticks needed before the deepest consumer is ready
```

### Warmup Calculation Algorithm

The graph traces backwards from every node to its FIELD sources, accumulating lookback requirements:

```
signal(lookback=2) → sma_20(lookback=20) → FIELD:close
signal(lookback=2) → rsi_14(lookback=15) → FIELD:close

Path 1: 2 + 20 = 22 ticks
Path 2: 2 + 15 = 17 ticks
Max = 22 ticks for FIELD:close
```

**Timeframe ratio**: If the system clock is `"1m"` but an operator uses `"1h"` data with `lookback=5`, that's `5 × 60 = 300` clock ticks.

During warmup, operators may return `valid=False` — not enough data yet. The graph continues ticking, filling buffers, until all paths have enough data.

## Dynamic Axis Expansion

When new symbols are added at runtime, buffers expand:

```python theme={null}
buffer.pad(pad_count=1)  # Add one new symbol column
```

This extends all 4 arrays by `pad_count` columns:

* `value`: padded with `NaN`
* `exists`, `valid`, `updated`: padded with `False`

The new symbol starts with no data. As ticks arrive, its data fills in naturally.

### Lazy Resize

On the first `append()` call, the buffer checks if `tick.value.shape` matches `n_symbols`. If not, it resizes:

```python theme={null}
def _resize(self, new_size):
    # Reallocate all arrays to (lookback, new_size)
    # Preserve existing data, pad new columns with NaN/False
```

This handles the case where the axis size wasn't known at buffer creation time.

## Memory Efficiency

RollingBuffer is designed for minimal memory use:

```
Buffer for SMA(20) with 100 symbols:
  value:   20 × 100 × 8 bytes = 16 KB
  exists:  20 × 100 × 1 byte  =  2 KB
  valid:   20 × 100 × 1 byte  =  2 KB
  updated: 20 × 100 × 1 byte  =  2 KB
  Total: 22 KB per buffer
```

Compare to storing the full history in a dataframe:

```
Full history (525,600 minutes/year) × 100 symbols × 8 bytes = 420 MB
```

RollingBuffer uses **0.005%** of the memory. It only stores what's needed.

## Buffer Lifecycle

```mermaid theme={null}
%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#1e293b', 'primaryTextColor': '#e2e8f0', 'primaryBorderColor': '#94F1E8', 'lineColor': '#5bb8b0', 'clusterBkg': '#0f172a', 'clusterBorder': '#1e3a3a', 'titleColor': '#94F1E8', 'edgeLabelBackground': 'transparent' }}}%%
flowchart TB
    classDef highlight fill:#134e4a,stroke:#94F1E8,stroke-width:2.5px,color:#94F1E8
    classDef source fill:#0d3b3b,stroke:#94F1E8,stroke-width:1.5px,color:#e0faf7
    classDef action fill:#172554,stroke:#60a5fa,stroke-width:1.5px,color:#bfdbfe

    A["add_node()<br>RollingBuffer allocated<br>(lookback × n_symbols)"]
    B["on_tick() — warmup<br>buffer.append(tick)<br>valid=False until full"]
    C["on_tick() — normal<br>overwrites oldest entry<br>operator receives full lookback"]
    D["expand_axis()<br>new symbols appear<br>buffer.pad(pad_count)"]

    A --> B --> C
    C -.-> D

    class A highlight
    class C action
    class D source
```

## Gap Handling: skip / ffill / nan

Different asset classes have different trading schedules. Crypto trades 24/7, US stocks trade weekdays 9:30-16:00 ET, and macro data releases monthly. When these data sources are combined in a single graph, **gaps** appear — timestamps where some sources have data and others don't.

The `Input.gaps` parameter controls how each buffer handles missing rows:

```python theme={null}
from clyptq.system.graph import Input

# Crypto: 24/7, no gaps — default "nan" is optimal
close = Input("FIELD:binance:futures:ohlcv:close", timeframe="1h", lookback=20, gaps="nan")

# US Stocks: skip weekends/holidays — "skip" returns only valid rows
stock = Input("FIELD:alpaca:stock:ohlcv:close", timeframe="1d", lookback=20, gaps="skip")

# Macro data: monthly releases — "ffill" carries last value forward
macro = Input("FIELD:fred:macro:DFF", timeframe="1d", lookback=5, gaps="ffill")
```

| Mode              | Behavior                                                                                                                                           | Use Case                                                                                                                                                                                                              |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `"nan"` (default) | Keep all rows as-is. Non-existing rows have `exists=False`, `value=NaN`.                                                                           | 24/7 markets (crypto). Optimal when there are no calendar gaps.                                                                                                                                                       |
| `"skip"`          | Return only rows where `exists=True`. Buffer stores extra headroom (`lookback // 2`) to ensure `lookback` valid rows are returned after filtering. | Markets with regular gaps (stocks, forex). Operators see only trading-day data. Currently implemented and production-ready, but not yet actively exercised because US stock data collection is not yet in production. |
| `"ffill"`         | Forward-fill gap rows with the last valid value. `updated=False` for filled rows.                                                                  | Sparse data (macro indicators, monthly releases). Operators always see a value.                                                                                                                                       |

### Headroom for skip Mode

When `gaps="skip"`, the buffer allocates extra capacity so that after filtering out non-existing rows, at least `lookback` valid rows remain:

```
Input(lookback=20, gaps="skip")
  → capacity = lookback + headroom = 20 + 10 = 30
  → 30 rows stored, then filtered to 20 valid rows
```

This is transparent to operators — they always receive exactly `lookback` rows.

## Trading Calendars

The calendar system marks which timestamps are valid trading periods. It generates an `exists` mask that the data provider applies before the data enters the graph.

| Calendar           | Behavior                                                | Asset Class                |
| ------------------ | ------------------------------------------------------- | -------------------------- |
| `AlwaysOnCalendar` | All timestamps valid (24/7)                             | Crypto, perpetual futures  |
| `USEquityCalendar` | NYSE hours (9:30-16:00 ET), weekdays, holidays excluded | US stocks, ETFs            |
| `MonthlyCalendar`  | First timestamp of each month only                      | FRED macro data (GDP, CPI) |

Calendars are set at the observation spec level, not per operator. The graph combines multiple calendars when a strategy uses cross-asset data.

## Cross-Asset & Cross-Timezone Strategies

ClyptQ's tagged tensor architecture natively supports strategies that combine data from different asset classes, exchanges, and timezones:

```python theme={null}
# Cross-asset strategy: crypto + stocks + macro
observations = [
    OHLCVSpec(exchange="binance", market_type="futures", timeframe="1h"),  # 24/7 crypto
    StockOHLCVSpec(exchange="alpaca", timeframe="1d"),                     # US market hours
    MacroSpec(indicators=["DFF", "T10Y2Y", "VIXCLS"]),                   # Monthly/daily
]
```

The system handles timezone alignment automatically:

* **Crypto** (UTC, 24/7) → `AlwaysOnCalendar`, `gaps="nan"`
* **US Stocks** (ET, weekdays) → `USEquityCalendar`, `gaps="skip"`
* **Macro** (UTC, sparse) → `MonthlyCalendar`, `gaps="ffill"`

All data aligns to the system clock frequency. Lower-frequency sources (1d stock data on a 1h clock) are forward-filled or skip-filtered based on the `gaps` parameter. The `exists` mask ensures operators know which data points are real observations vs. filled values.

<Note>
  The calendar system and `gaps="skip"` mode are fully implemented and production-ready, but the cross-asset use case (crypto + stocks) is not yet exercised in production because US stock data collection has not been deployed. Once stock data is onboarded, cross-asset strategies will work without any engine changes — only the observation specs and `gaps` parameters need to be configured.
</Note>

## Relationship to Other Concepts

* **[TaggedArray](/engine/tagged-array)**: RollingBuffer stores and delivers TaggedArrays
* **[StatefulGraph](/engine/stateful-graph)**: The graph creates and manages all RollingBuffers
* **[FIELD & STATE](/engine/field-state)**: FIELD and STATE data are distributed to consumer buffers via the same mechanism
* **[Operator Protocol](/engine/operator-protocol)**: Operators receive buffered data from their Input specifications