diff --git a/rust_hft/docs/RESEARCH_DATA_SOURCES.md b/rust_hft/docs/RESEARCH_DATA_SOURCES.md new file mode 100644 index 000000000..017adeeb5 --- /dev/null +++ b/rust_hft/docs/RESEARCH_DATA_SOURCES.md @@ -0,0 +1,368 @@ +# Research Data Source Inventory + +_Last reviewed: 2026-08-24_ + +This document records external historical market-data resources that Monday can use to bootstrap research. It is a **research inventory**, not a declaration that any source is supported by a governed Data Mission, production connector, or live execution path. + +Monday should use third-party or venue-hosted archives to accelerate historical research, while retaining its own live collectors as production/evidence truth for exact receive timing, sequence integrity, order/fill telemetry, and venue-specific behavior. + +## Operating principle + +Use a three-layer model: + +1. **Research bootstrap** — public/free datasets and downloadable venue archives. +2. **Large-scale historical research** — paid or continuous historical providers where needed. +3. **Production truth** — Monday live collectors and private execution telemetry. + +Do not block research until Monday has accumulated years of self-recorded data. Do not treat historical snapshots from an external provider as proof of production latency, maker queue position, or fill probability. + +--- + +## Polymarket + +### 1. PolyOrderbooks Hugging Face: 1-second full L2 sample + +Source: +- https://huggingface.co/datasets/polyorderbooks/polymarket-crypto-5min-orderbooks + +Current published sample: +- 15 complete five-minute crypto markets +- BTC, ETH, SOL, XRP, BNB, DOGE, HYPE +- 1-second snapshots +- YES and NO books +- full resting bid/ask ladders +- 9,000 outcome snapshots +- winner attached +- CC BY 4.0 + +Best use: +- execution-aware replay +- spread/depth/OBI/microprice research +- liquidity withdrawal +- depth slope/convexity +- size-aware VWAP and slippage +- capacity estimation +- expiry-liquidity studies + +Limitation: +- a 1-second snapshot does not reconstruct every intra-second trade/cancel event; +- it is not enough for true maker queue modeling, sub-second latency research, or exact fill probability. + +### 2. PolyOrderbooks continuous historical API + +Sources: +- https://polyorderbooks.com/ +- https://polyorderbooks.com/features + +Advertised capability: +- historical Polymarket order books +- 1-second snapshots +- resolved markets remain queryable +- full bid/ask ladders +- REST API + +Use this as a candidate large-scale historical source. Monday should still validate sampled windows against its own collector before treating vendor data as canonical evidence. + +### 3. Large public Polymarket research corpus + +Source: +- https://huggingface.co/datasets/obadiaha/polymarket-crypto-5m-15m + +Published coverage includes: +- BTC, ETH, SOL, XRP +- 5-minute and 15-minute markets +- millions of order-book rows +- tens of millions of trades +- top-10 order-book snapshots at 10-second cadence +- market resolutions and underlying crypto prices + +Best use: +- large-sample alpha screening +- regime studies +- market-resolution joins +- coarse liquidity and lead/lag studies + +Do **not** use 10-second top-10 snapshots as an execution-truth substitute for 1-second/full-event L2. + +--- + +## Binance + +### 1. Binance Public Data — official downloadable archive + +Sources: +- https://data.binance.vision/ +- https://github.com/binance/binance-public-data + +Useful public datasets include daily/monthly: +- trades +- aggTrades +- klines +- Spot +- USD-M Futures +- COIN-M Futures + +Important parser note: +- Binance Spot archive timestamps changed to microseconds from 2025-01-01 onward; historical ingestion must normalize timestamp units explicitly. + +Best use: +- long-horizon trade-flow research +- taker buy/sell flow +- momentum / realized-volatility studies +- cross-sectional symbol selection +- long historical backfill + +Caution: +- Binance's easily downloadable public `bookDepth` artifacts should not be assumed to be execution-grade full incremental L2. For real order-book replay, prefer a validated incremental-L2 archive or Monday self-capture. +- verify the applicable Binance data-use terms before redistribution or commercial data resale. + +### 2. Tardis — Binance historical incremental L2 + +Source: +- https://docs.tardis.dev/historical-data-details/binance + +Available research types include: +- trades +- incremental_book_L2 +- book snapshots +- quotes +- raw historical WebSocket messages with local receive timestamps + +Tardis documents Binance depth capture at the fastest API update speed available at collection time and reconstructs initial snapshots using REST where required. + +Free bootstrap path: +- Tardis provides downloadable CSV samples for the first day of each month without an API key. + +Best use: +- order-book reconstruction +- cancellation/replenishment research +- microprice/OBI +- queue/flow proxies +- lead/lag against other exchanges +- realistic taker execution simulation + +--- + +## Bybit + +### 1. Bybit official historical public market data + +Sources: +- https://www.bybit.com/future-activity/developer +- https://www.bybit.com/en/learn/bybit-guide/how-to-create-a-bybit-api-key + +Bybit documents downloadable historical public market data including: +- order-book snapshots +- klines +- trades +- funding rates + +Use official downloads first for low-cost factor research where the required resolution is available. + +### 2. Tardis — Bybit historical incremental L2 + +Derivatives: +- https://docs.tardis.dev/historical-data-details/bybit + +Spot: +- https://docs.tardis.dev/historical-data-details/bybit-spot + +Published coverage includes: +- Bybit derivatives historical data for inverse contracts from 2019-11-07 +- linear contracts from 2020-05-28 +- Bybit Spot from 2021-12-04 +- trades +- incremental_book_L2 +- quotes +- derivative ticker +- liquidations for derivatives + +Tardis also exposes free first-day-of-month CSV samples. + +Best use: +- perp microstructure +- liquidation propagation +- funding/OI + L2 interaction +- Binance-vs-Bybit lead/lag +- high-volatility altcoin execution research + +--- + +## OKX + +### 1. OKX official historical data — unusually useful free source + +Source: +- https://www.okx.com/historical-data + +OKX currently documents downloadable: +- tick-level trade history from 2021-09 +- perpetual funding rates from 2022-03 +- high-resolution L2 order-book data from 2023-03 +- OHLC candles from 2023-07 +- borrowing rates from 2021-12 + +This should be one of Monday's highest-priority free sources because official historical L2 is directly available. + +Best use: +- multi-month L2 execution research without waiting for self-capture +- spread/depth/OBI/microprice +- funding + order-book regime studies +- cross-venue lead/lag with Binance/Bybit + +### 2. Tardis — OKX historical incremental L2 + +Spot: +- https://docs.tardis.dev/historical-data-details/okex + +Futures: +- https://docs.tardis.dev/historical-data-details/okex-futures + +Swap: +- https://docs.tardis.dev/historical-data-details/okex-swap + +Tardis provides historical incremental L2, trades, quotes, derivative ticker, liquidations, mark/index/open-interest channels depending on instrument and period. + +--- + +## Multi-venue historical provider: Tardis + +Overview: +- https://docs.tardis.dev/historical-data-details + +Useful supported venues include: +- Binance Spot / USD-M / COIN-M +- Bybit Spot / Derivatives / Options +- OKX Spot / Futures / Swap / Options +- Deribit +- Hyperliquid +- Bitget +- Gate.io +- Kraken +- Coinbase and others + +Why this matters for Monday: +- a normalized multi-venue adapter can enable the same feature and replay engine across CEXs; +- raw exchange-format replay remains available where venue semantics matter; +- local timestamps make receive-time/lead-lag research possible, subject to provider collection topology and clock assumptions. + +Do not blindly merge venues by event timestamp. Preserve at least: +- exchange event timestamp +- provider/local receive timestamp when available +- sequence/update IDs +- venue +- instrument +- source/provider +- schema version + +--- + +## Recommended Monday research tiers + +| Research question | First source | Upgrade source | Production truth | +|---|---|---|---| +| Polymarket L2 execution replay | PolyOrderbooks HF sample | PolyOrderbooks continuous history | Monday Polymarket WS + user/order telemetry | +| Polymarket large-sample alpha screening | public HF corpora | continuous historical API | Monday self-capture validation | +| Binance long-horizon trade/flow factors | Binance Public Data | Tardis if L2 required | Monday Binance collector | +| Binance true L2 microstructure | Tardis sample/full history | Tardis full history | Monday Binance LOB collector | +| Bybit perp microstructure | Bybit official download | Tardis incremental L2 | Monday Bybit collector | +| OKX L2 research | OKX official historical L2 | Tardis incremental L2 | Monday OKX collector | +| cross-venue lead/lag | official trades + aligned samples | Tardis normalized/raw multi-venue | synchronized Monday live collectors | +| maker fill / queue / latency | not reliable from coarse snapshots | event-level incremental L2 only | Monday private order/fill telemetry | + +--- + +## Suggested ingestion contract + +External historical sources should adapt into a provider-neutral research interface rather than leak provider schemas through the research stack. + +Conceptually: + +```rust +trait HistoricalMarketDataProvider { + fn instruments(&self) -> Result>; + fn trades(&self, request: TradeHistoryRequest) -> Result; + fn book_events(&self, request: BookHistoryRequest) -> Result; + fn funding(&self, request: FundingHistoryRequest) -> Result; +} +``` + +The canonical research event model should preserve enough provenance to distinguish snapshots from event-level deltas. + +Minimum provenance fields: +- `venue` +- `instrument` +- `event_time` +- `receive_time` when available +- `sequence_id` / update range when available +- `source_provider` +- `source_uri` +- `source_resolution` +- `source_schema_version` +- `ingested_at` +- raw artifact checksum + +Recommended book-event classes: +- `Snapshot` +- `Delta` +- `Trade` +- `Quote` +- `Funding` +- `OpenInterest` +- `Liquidation` +- `MarkPrice` + +Do not infer `Delta` from coarse snapshots and then present it as observed event truth. + +--- + +## Immediate research sequence + +1. **Polymarket execution unit tests** + - import the 1-second full-L2 HF sample; + - implement size-aware ladder walking; + - measure spread, depth, OBI, microprice, slippage, capacity, and time-to-expiry liquidity. + +2. **OKX free L2 pilot** + - download official L2 history; + - run the same canonical book/replay/features pipeline used for Polymarket; + - verify parser and sequence/completeness assumptions. + +3. **Binance / Bybit Tardis free-sample pilot** + - use first-day-of-month free incremental-L2 samples; + - prove one normalized replay path across Binance, Bybit, and OKX; + - compare OBI/microprice/depth-withdrawal features across venues. + +4. **Long-history alpha screening** + - use Binance Public Data trades/aggTrades/klines for broad symbol/time coverage; + - join funding/OI/liquidation sources where available; + - use event-level L2 only on shortlisted regimes/symbols to control data cost. + +5. **Shadow validation** + - compare historical-provider features with Monday's live self-capture on overlapping windows; + - quantify timestamp, depth, missing-event, and replay divergences before trusting a provider for execution research. + +--- + +## Data-quality gates + +Every imported archive should produce an evidence manifest with at least: +- source URL/provider +- requested and observed coverage window +- venue/instrument +- expected vs observed row/event counts where knowable +- checksum/hash +- timestamp unit and timezone +- duplicate count +- gap count +- crossed-book count +- sequence-gap count for event streams +- parser/schema version + +A dataset is research input, not evidence of production support. Missing events must remain missing; do not synthesize completeness for execution studies. + +## Strategic conclusion + +Monday no longer needs to wait for self-collected history before beginning market-microstructure and execution research. + +Use external historical data aggressively for **research iteration**. Keep Monday collectors for **production truth, exact event ordering, latency, fills, and validation**.