LLM Index
The LLM leaderboard that can't be gamed
A discriminative, contamination-resistant, fully transparent LLM ranking built on IRT psychometrics and Bradley-Terry duels — updated live.
135
models ranked
12
evaluation domains
25
template item generators
3
cross-provider judges
95%
CI on every score
60+
published sources behind the design
Overview
Classic leaderboards fail twice: top models cluster above 95% on saturated benchmarks, giving zero discrimination, and fixed test sets leak into training data. LLM Index is engineered against both from the psychometrics up — every item carries a fitted difficulty and discrimination, ability is a 2PL IRT MAP estimate with Fisher-information standard errors, and raw accuracy is never the score.
There is no fixed test set to memorize: every scored batch is freshly generated from seeded template generators, and the fixed-versus-fresh gap is published per model as a contamination delta. Home-made agentic, terminal, and context-load benches are graded deterministically by simulators; writing, safety, and SVG logo duels use a position-swapped, never-self-judging 3-judge panel feeding a Bradley-Terry fit.
Everything is transparent: 135 models across 12 equally weighted domains, every score with a 95% confidence interval, and a page per model showing every answer on every test with judge verdicts, latency, and cost. Cost and latency live on a separate Pareto frontier — never blended into quality.
Key Features
IRT 2PL scoring
Each item has fitted difficulty and discrimination; ability theta is a MAP estimate with Fisher-information standard errors, and non-discriminating items are auto-retired.
Contamination resistance by design
Every scored batch is freshly generated from seeded template generators — values, paraphrases, structures — with the fixed-vs-fresh gap published per model.
Original agentic bench
Simulated tool-calling environments with distractor tools where a deterministic simulator computes the unique correct call sequence — graded by canonical-JSON equality, no judges.
Simulated terminal bench
A closed, unambiguous POSIX subset simulated in TypeScript: models predict exact pipeline stdout, file trees after mutations, and exit-code traces.
Bradley-Terry judged duels
SVG logo reproduction, writing, and safety duels scored by a position-swapped, cross-provider 3-judge panel that never self-judges, fitted with Bradley-Terry.
Vision OCR under clutter
Generated scenes with rotated codes, noise, and decoys rasterized to PNG; text-only models skip the domain and weights renormalize.
Live, one model at a time
Parallel evaluation lanes stream results; after every completed model, the IRT refit re-runs and the public leaderboard re-ranks in real time.
Total transparency
Every model page shows every answer on every test with judge verdicts, confidence, latency, and cost — each number traces to an immutable score run.
Public versioned API
Leaderboard, per-domain scores, model profiles, machine-readable methodology, and live run progress over a frozen v1 JSON API.
How It Works
apps/web
Next.js 14 serves the live leaderboard, per-model transparency pages, the public methodology, and the versioned /api/v1 endpoints.
apps/worker
The eval runner, duel runner, parallel benchmark orchestrator, and refit trigger — evaluation lanes run models in parallel with a full audit trail.
apps/psychometrics
Python fits the 2PL IRT model with MAP estimates and Fisher standard errors, the Bradley-Terry duel model, and calibration metrics.
packages/items
25 template generators plus the agentic, terminal, and ledger simulators and a lenient, robust answer-extraction cascade that never cheats models on formatting.
Scoring pipeline
OpenRouter catalog sync, seeded batch generation, temperature-0 scoring with consistency samples, per-domain 2PL fits, Bradley-Terry duels, and 95% CIs on every score.
Tech Stack
Frontend
Data
Psychometrics
Infrastructure
Highlights
- Domain weights are equal by design — the maximum-entropy prior — and per-domain scores are always published so anyone can re-weight.
- Cost and latency are never blended into quality: they live on a separate Pareto frontier.
- Answer keys and rubrics never ship to the client or public API; scored batches keep the frozen anchor subset at or below 20% of any run.
- Every displayed number traces to an immutable score-run row with item-set hash, model set, index version, and fit diagnostics.
- Methodology changes bump a semver INDEX_VERSION with a public changelog; weights and hyperparameters live in exactly one file, served machine-readable.
- Backed by a 19-page LaTeX white paper and 60+ published sources — yet every environment, item, simulator, and grader is original.
Explore LLM Index
Live, contamination-resistant LLM ranking — the full source is on GitHub.