Sources
Where the scores come from: the live sources of the automated catalogue, how each is pulled, what state it is in and under what licence. The thesis catalogue (v3) is a frozen snapshot of 51 other leaderboards; their details are in source_provenance.
Where the data comes from
live: last pull good stale or warning, old data kept error: pull or scraper failing, needs a person frozen (pulled once) or switched off
Arrow labels say how data arrives: file, API, or scraped (read out of a web page; breaks when the page changes). Colours come from the health checks after the last pull. "Live" means it worked the last time someone ran the pipeline; nothing is scheduled.
Source by source
ok stale or warning, old data kept broken, needs a person archived or disabled
| Source | How | Last pull | Data dated | Models | Scores | Licence |
|---|---|---|---|---|---|---|
|
upstream data dated 2025-10-03 (402 days old, limit 365)
|
file | 2026-11-08 unchanged |
2025-10-03 | 68 | 271 | Apache-2.0 (aider repository) |
|
upstream data dated 2025-01-10 (668 days old, limit 365)
|
file | 2026-11-08 unchanged |
2025-01-10 | 91 | 1,000 | Apache-2.0 (HELM project); the results bucket has no licence notice of its own. Credit: Stanford CRFM HELM. |
|
upstream data dated 2026-02-26 (256 days old, limit 120)
|
file | 2026-11-08 unchanged |
2026-02-26 | 253 | 313 | CC BY-NC 4.0 (swe-bench.github.io repository LICENSE): non-commercial only; credit SWE-bench. The SWE-bench code repo is MIT but is not the data we pull. |
| file | 2026-11-08 unchanged |
2025-12-16 | 82 | 2,458 | Apache-2.0 (BFCL-Result repository) | |
| file | 2026-11-08 unchanged |
2026-10-27 | 112 | 10,080 | MIT (Space metadata and BRIDGE code repo); no licence text for the scores themselves. Credit: Wu et al., BRIDGE, Nature Biomedical Engineering 2026, doi 10.1038/s41551-026-01719-2. We take scores only, never clinical text. | |
| file | 2026-11-08 good |
2026-11-06 | 1,238 | 4,712 | CC BY (epoch.ai/benchmarks/use-this-data): credit Epoch AI. Covers only benchmarks Epoch ran itself (named '(Epoch run)'); benchmarks named 'collected by Epoch' are external data that keeps its original licence. Public release: Epoch-run only. | |
| file | 2026-11-08 unchanged |
2025-11-24 | 68 | 408 | Apache-2.0 (HELM project); the results bucket has no licence notice of its own. Credit: Stanford CRFM HELM. | |
| file | 2026-11-08 unchanged |
2025-08-01 | 28 | 112 | Main repo MIT; Hugging Face Space MIT; website repo CC BY-SA 4.0 (README). Private use only: excluded from public releases (decision 8 Nov 2026). | |
| file | 2026-11-08 unchanged |
2026-11-06 | 456 | 26,732 | CC-BY-4.0 (dataset card) | |
| file | 2026-11-08 unchanged |
2026-01-19 | 13 | 494 | Apache-2.0 (HELM project); the results bucket has no licence notice of its own. Credit: Stanford CRFM HELM. | |
| scrape | 2026-11-08 good |
2026-11-06 | 109 | 182 | Terms for reuse not checked (UNKNOWN); private research use only | |
| scrape | 2026-11-08 unchanged |
2026-10-22 | 111 | 444 | Apache-2.0 (repository) | |
|
archived: pulled once, not re-checked
|
api | 2026-11-08 unchanged |
4,573 | 27,438 | Apache-2.0 (leaderboard Space); the contents and results datasets state no licence (checked 8 Nov 2026). Credit: Open LLM Leaderboard v2, Fourrier et al., Hugging Face 2024. |
Latest health checks per source
| Source | Check | Result | Message |
|---|---|---|---|
aider | unchanged | pass | sha256 equals pull 5; not parsed again |
aider | upstream_age | warn | upstream data dated 2025-10-03 (402 days old, limit 365) |
bfcl | unchanged | pass | sha256 equals pull 2; not parsed again |
bfcl | upstream_age | pass | upstream data dated 2025-12-16 (328 days old, limit 400) |
bridge_medical | unchanged | pass | sha256 equals pull 16; not parsed again |
bridge_medical | upstream_age | pass | upstream data dated 2026-10-27 (13 days old, limit 180) |
epoch | fetch | pass | 1 file(s), 2,283,861 bytes, HTTP 200 |
epoch | required_columns | pass | all 4 required columns/keys present |
epoch | columns_changed | pass | same columns as the last good pull |
epoch | row_count_range | pass | 5435 rows (plausible 1000-20000) |
epoch | row_count_change | pass | 5435 rows vs 5435 last good (0% change, limit 30%) |
epoch | score_range | pass | all scores within their declared range |
epoch | parse_notes | info | 3 note(s): fictionlivebench_external.csv: 1 rows name no model or agent; skipped; simplebench_external.csv: 2 rows name no model or agent; skipped; weirdml_external.csv: 1 rows name no model or agent; skipped |
epoch | parse_errors | pass | no parse errors |
epoch | upstream_age | pass | upstream data dated 2026-11-06 (3 days old, limit 30) |
helm_capabilities | unchanged | pass | sha256 equals pull 12; not parsed again |
helm_capabilities | upstream_age | pass | upstream data dated 2025-11-24 (350 days old, limit 365) |
helm_lite | unchanged | pass | sha256 equals pull 13; not parsed again |
helm_lite | upstream_age | warn | upstream data dated 2025-01-10 (668 days old, limit 365) |
livecodebench | unchanged | pass | sha256 equals pull 4; not parsed again |
livecodebench | upstream_age | pass | upstream data dated 2025-08-01 (464 days old, limit 540) |
lmarena_hf | unchanged | pass | sha256 equals pull 37; not parsed again |
lmarena_hf | upstream_age | pass | upstream data dated 2026-11-06 (3 days old, limit 60) |
medhelm | unchanged | pass | sha256 equals pull 14; not parsed again |
medhelm | upstream_age | pass | upstream data dated 2026-01-19 (294 days old, limit 365) |
open_llm_v2 | unchanged | pass | sha256 equals pull 9; not parsed again |
seal | fetch | pass | 6 file(s), 932,573 bytes, HTTP 200,200,200,200,200,200 |
seal | required_columns | pass | all 1 required columns/keys present |
seal | columns_changed | pass | same columns as the last good pull |
seal | row_count_range | pass | 182 rows (plausible 20-2000) |
seal | row_count_change | pass | 182 rows vs 182 last good (0% change, limit 30%) |
seal | score_range | pass | all scores within their declared range |
seal | parse_errors | pass | no parse errors |
seal | upstream_age | pass | upstream data dated 2026-11-06 (3 days old, limit 120) |
swebench | unchanged | pass | sha256 equals pull 6; not parsed again |
swebench | upstream_age | warn | upstream data dated 2026-02-26 (256 days old, limit 120) |
vectara | unchanged | pass | sha256 equals pull 15; not parsed again |
vectara | upstream_age | pass | upstream data dated 2026-10-22 (18 days old, limit 90) |
All pulls: pulls. All checks: source_checks.