MSbench
Thesis catalogue (v3)coming soon Automated catalogue Skills Sources Known issues Downloads & APIcoming soon About
DMSL @ UCY
Preview · the database itself is not public yet · data shown as built, issues documented on Known issues

Sources

Where the scores come from: the live sources of the automated catalogue, how each is pulled, what state it is in and under what licence. The thesis catalogue (v3) is a frozen snapshot of 51 other leaderboards; their details are in source_provenance.

Where the data comes from

file Berkeley Function-Calling Leaderboard (BFCL): Downloads the newest dated CSV from the GitHub repo HuanzhiMao/BFCL-Result. Berkeley Function-Calling Lea… live · data 2025-12-16 file BRIDGE medical leaderboard: Downloads the JSON files the Hugging Face Space reads to draw its table, pinned to the Space's commit. BRIDGE medical leaderboard live · data 2026-10-27 file Epoch AI Benchmarking Hub: Downloads one zip (benchmark_data.zip) with a CSV per benchmark and a metadata file naming each score column. Epoch AI Benchmarking Hub live · data 2026-11-06 file HELM Capabilities (Stanford CRFM): Finds the newest HELM Capabilities release in Stanford's public results bucket and downloads its accuracy table (JSON). HELM Capabilities (Stanford CRFM) live · data 2025-11-24 file LiveCodeBench (code generation): Downloads the JSON file behind the LiveCodeBench site from its GitHub repo. LiveCodeBench (code generation) live · data 2025-08-01 file LMArena leaderboards (Hugging Face dataset): Downloads Parquet files from the Hugging Face dataset lmarena-ai/leaderboard-dataset, pinned to its commit. LMArena leaderboards (Hugging… live · data 2026-11-06 file MedHELM (Stanford CRFM): Finds the newest MedHELM release in Stanford's public results bucket and downloads its accuracy table (JSON). MedHELM (Stanford CRFM) live · data 2026-01-19 scraped Scale AI SEAL leaderboards: Scraped: downloads each leaderboard's web page and reads the table data embedded in the page (no file or API is published). Scale AI SEAL leaderboards live · data 2026-11-06 scraped Vectara hallucination leaderboard (HHEM): Scraped: the leaderboard exists only as a Markdown table in the repo's README; the parser reads that table and its 'Last updated' line. Vectara hallucination leaderb… live · data 2026-10-22 file Aider polyglot leaderboard: Downloads the leaderboard YAML file from the aider GitHub repo. Aider polyglot leaderboard stale · data 2025-10-03 file HELM Lite (Stanford CRFM): Finds the newest HELM Lite release in Stanford's public results bucket and downloads its accuracy table (JSON). HELM Lite (Stanford CRFM) stale · data 2025-01-10 file SWE-bench leaderboards: Downloads the JSON file behind swebench.com from its GitHub repo. SWE-bench leaderboards stale · data 2026-02-26 API · pulled once, frozen Open LLM Leaderboard v2 (archived): Hugging Face datasets-server rows API, paged. Archived: pulled once. Open LLM Leaderboard v2 (archived) frozen Automated catalogue 2026-11-08_190934

live: last pull good   stale or warning, old data kept   error: pull or scraper failing, needs a person   frozen (pulled once) or switched off

Arrow labels say how data arrives: file, API, or scraped (read out of a web page; breaks when the page changes). Colours come from the health checks after the last pull. "Live" means it worked the last time someone ran the pipeline; nothing is scheduled.

Source by source

ok   stale or warning, old data kept   broken, needs a person   archived or disabled

SourceHowLast pullData datedModelsScoresLicence
Aider polyglot leaderboard
upstream data dated 2025-10-03 (402 days old, limit 365)
file 2026-11-08
unchanged
2025-10-03 68 271 Apache-2.0 (aider repository)
HELM Lite (Stanford CRFM)
upstream data dated 2025-01-10 (668 days old, limit 365)
file 2026-11-08
unchanged
2025-01-10 91 1,000 Apache-2.0 (HELM project); the results bucket has no licence notice of its own. Credit: Stanford CRFM HELM.
SWE-bench leaderboards
upstream data dated 2026-02-26 (256 days old, limit 120)
file 2026-11-08
unchanged
2026-02-26 253 313 CC BY-NC 4.0 (swe-bench.github.io repository LICENSE): non-commercial only; credit SWE-bench. The SWE-bench code repo is MIT but is not the data we pull.
Berkeley Function-Calling Leaderboard (BFCL)
file 2026-11-08
unchanged
2025-12-16 82 2,458 Apache-2.0 (BFCL-Result repository)
BRIDGE medical leaderboard
file 2026-11-08
unchanged
2026-10-27 112 10,080 MIT (Space metadata and BRIDGE code repo); no licence text for the scores themselves. Credit: Wu et al., BRIDGE, Nature Biomedical Engineering 2026, doi 10.1038/s41551-026-01719-2. We take scores only, never clinical text.
Epoch AI Benchmarking Hub
file 2026-11-08
good
2026-11-06 1,238 4,712 CC BY (epoch.ai/benchmarks/use-this-data): credit Epoch AI. Covers only benchmarks Epoch ran itself (named '(Epoch run)'); benchmarks named 'collected by Epoch' are external data that keeps its original licence. Public release: Epoch-run only.
HELM Capabilities (Stanford CRFM)
file 2026-11-08
unchanged
2025-11-24 68 408 Apache-2.0 (HELM project); the results bucket has no licence notice of its own. Credit: Stanford CRFM HELM.
LiveCodeBench (code generation)
file 2026-11-08
unchanged
2025-08-01 28 112 Main repo MIT; Hugging Face Space MIT; website repo CC BY-SA 4.0 (README). Private use only: excluded from public releases (decision 8 Nov 2026).
LMArena leaderboards (Hugging Face dataset)
file 2026-11-08
unchanged
2026-11-06 456 26,732 CC-BY-4.0 (dataset card)
MedHELM (Stanford CRFM)
file 2026-11-08
unchanged
2026-01-19 13 494 Apache-2.0 (HELM project); the results bucket has no licence notice of its own. Credit: Stanford CRFM HELM.
Scale AI SEAL leaderboards
scrape 2026-11-08
good
2026-11-06 109 182 Terms for reuse not checked (UNKNOWN); private research use only
Vectara hallucination leaderboard (HHEM)
scrape 2026-11-08
unchanged
2026-10-22 111 444 Apache-2.0 (repository)
Open LLM Leaderboard v2 (archived)
archived: pulled once, not re-checked
api 2026-11-08
unchanged
4,573 27,438 Apache-2.0 (leaderboard Space); the contents and results datasets state no licence (checked 8 Nov 2026). Credit: Open LLM Leaderboard v2, Fourrier et al., Hugging Face 2024.
Latest health checks per source
SourceCheckResultMessage
aiderunchangedpasssha256 equals pull 5; not parsed again
aiderupstream_agewarnupstream data dated 2025-10-03 (402 days old, limit 365)
bfclunchangedpasssha256 equals pull 2; not parsed again
bfclupstream_agepassupstream data dated 2025-12-16 (328 days old, limit 400)
bridge_medicalunchangedpasssha256 equals pull 16; not parsed again
bridge_medicalupstream_agepassupstream data dated 2026-10-27 (13 days old, limit 180)
epochfetchpass1 file(s), 2,283,861 bytes, HTTP 200
epochrequired_columnspassall 4 required columns/keys present
epochcolumns_changedpasssame columns as the last good pull
epochrow_count_rangepass5435 rows (plausible 1000-20000)
epochrow_count_changepass5435 rows vs 5435 last good (0% change, limit 30%)
epochscore_rangepassall scores within their declared range
epochparse_notesinfo3 note(s): fictionlivebench_external.csv: 1 rows name no model or agent; skipped; simplebench_external.csv: 2 rows name no model or agent; skipped; weirdml_external.csv: 1 rows name no model or agent; skipped
epochparse_errorspassno parse errors
epochupstream_agepassupstream data dated 2026-11-06 (3 days old, limit 30)
helm_capabilitiesunchangedpasssha256 equals pull 12; not parsed again
helm_capabilitiesupstream_agepassupstream data dated 2025-11-24 (350 days old, limit 365)
helm_liteunchangedpasssha256 equals pull 13; not parsed again
helm_liteupstream_agewarnupstream data dated 2025-01-10 (668 days old, limit 365)
livecodebenchunchangedpasssha256 equals pull 4; not parsed again
livecodebenchupstream_agepassupstream data dated 2025-08-01 (464 days old, limit 540)
lmarena_hfunchangedpasssha256 equals pull 37; not parsed again
lmarena_hfupstream_agepassupstream data dated 2026-11-06 (3 days old, limit 60)
medhelmunchangedpasssha256 equals pull 14; not parsed again
medhelmupstream_agepassupstream data dated 2026-01-19 (294 days old, limit 365)
open_llm_v2unchangedpasssha256 equals pull 9; not parsed again
sealfetchpass6 file(s), 932,573 bytes, HTTP 200,200,200,200,200,200
sealrequired_columnspassall 1 required columns/keys present
sealcolumns_changedpasssame columns as the last good pull
sealrow_count_rangepass182 rows (plausible 20-2000)
sealrow_count_changepass182 rows vs 182 last good (0% change, limit 30%)
sealscore_rangepassall scores within their declared range
sealparse_errorspassno parse errors
sealupstream_agepassupstream data dated 2026-11-06 (3 days old, limit 120)
swebenchunchangedpasssha256 equals pull 6; not parsed again
swebenchupstream_agewarnupstream data dated 2026-02-26 (256 days old, limit 120)
vectaraunchangedpasssha256 equals pull 15; not parsed again
vectaraupstream_agepassupstream data dated 2026-10-22 (18 days old, limit 90)

All pulls: pulls. All checks: source_checks.

DMSL @ UCY
MSbench, the LLM-MS capability catalogue · Data Management Systems Laboratory (DMSL), University of Cyprus · Scores belong to their leaderboards, see Sources
Powered by Datasette