HF Market Data
26.5 billion rows of market data, no key required
A full-history downloader, a DuckDB-over-Parquet data lake and an open REST API serving intraday bars and complete options chains at www.hfmarketdata.io.
26.5B
Rows in the lake
558,814
Parquet files
350 GB
Lake size (zstd)
5,800+
Options underlyings
66/66
Options quarters since 2010
7
Asset classes
Overview
HF Market Data is an open high-frequency market data platform built end to end: ingestion, storage, API and documentation. A single-file downloader pulls everything FirstRate Data publishes — stocks, ETFs, futures, crypto, indices, FX and options — and converts it straight into a query-ready Parquet lake. The full ingestion completed as 745/745 jobs with zero data loss.
The data layer is deliberately radical: 26,464,775,098 rows across 558,814 zstd Parquet files, queried in place by DuckDB with predicate pushdown — no database server anywhere in the stack. Bars span five timeframes from 1-minute to daily, and options coverage runs 66/66 quarters since 2010 across 5,800+ US underlyings with quotes, implied volatilities and full Greeks.
On top sits an open, keyless REST API (FastAPI) serving JSON and CSV — up to 2 million rows per CSV request — with endpoints for bars, multi-ticker watchlists, point-in-time snapshots, options chains and contract histories. The same process serves a React 18 + Vite documentation platform with a full endpoint reference, request snippets in curl, Python and JavaScript, and live example responses.
Key Features
Full-history downloader
A single-file, fully resumable downloader ingests everything FirstRate Data publishes across a 745-job plan, with retries, a disk-space guard and a resume manifest.
DuckDB-over-Parquet lake
558,814 zstd Parquet files — one per instrument or segment — queried in place with predicate pushdown, no database server required.
Open, keyless REST API
Every endpoint is public and key-free, returning JSON or CSV with precise start/end windows and up to 2 million rows per CSV request.
Intraday to daily bars
OHLCV bars at 1min, 5min, 30min, 1hour and 1day across stocks, ETFs, futures, crypto, indices and FX, with split and dividend adjustments.
Complete options history
Every quarterly options archive since 2010 — quotes, bid/ask IV, open interest, volume and the full set of Greeks, 16 fields per row.
Point-in-time snapshots
Query a whole watchlist at one precise moment, or pull multi-ticker bars in a single call with per-ticker limits.
Zero-loss ingestion
CSV converts to Parquet via DuckDB with raw archives deleted on the fly — 745/745 jobs completed with zero data loss.
Live documentation platform
A React 18 + Vite docs site with parameter tables, curl/Python/JavaScript snippets, live responses and a real-time dataset status page.
How It Works
Ingest
frd_downloader.py works through a 745-job plan against the FirstRate Data API — options quarters since 2010, stocks and ETFs A–Z across all timeframes and adjustments, futures, crypto, indices and FX — fully resumable via a state manifest.
Convert
Each downloaded archive is converted CSV-to-Parquet through DuckDB with zstd compression, one file per instrument or segment, and the raw zips are deleted on the fly to keep the footprint at 350 GB.
Query in place
The API layer (FastAPI + DuckDB) queries the 26.5-billion-row lake directly with predicate pushdown into the Parquet row groups — no database server, no ETL into a warehouse.
Serve openly
Keyless endpoints expose the live inventory, ticker discovery, OHLCV bars, watchlist snapshots and full options chains with Greeks, in JSON or CSV up to 2 million rows.
Document and publish
The same process serves the React + Vite documentation platform, exposed publicly at www.hfmarketdata.io through ngrok.
Tech Stack
Data
API
Frontend
Ops
Highlights
- 26,464,775,098 rows queried in place — no database server anywhere in the stack
- 745/745 ingestion jobs completed with zero data loss
- Full US options history since 2010: 66/66 quarters, 5,800+ underlyings, quotes, IV and all Greeks
- Open and keyless — no signup, no API key, JSON and CSV out of the box
- Serves up to 2 million rows in a single CSV request
- The entire pipeline is two Python files: a 661-line downloader and a 480-line API
Explore HF Market Data
Open high-frequency market data platform — the full source is on GitHub.