MSbench
Thesis catalogue (v3)coming soon Automated catalogue Skills Sources Known issues Downloads & APIcoming soon About
DMSL @ UCY
Preview · the database itself is not public yet · data shown as built, issues documented on Known issues

Automated catalogue

A second catalogue, separate from v3, rebuilt from live leaderboards when the pipeline is run (started by hand, no schedule yet). The review's fixes are enforced as checks: a release goes out only when every blocking check passes, otherwise the previous one stays.

Current release automated_2026-11-08_190934

Built 2026-11-08 19:09 UTC from 13 sources with data · sources: 9 green 3 yellow 1 frozen · 2 more switched off

5,443models
529systems (not ranked)
74,644scores
355tests
149benchmarks
415open review items
Sources

Skills with evidence

SkillBenchmarks usedScoredRankedTop ranked
Open factual recallepoch_trivia_qa, helm_lite_naturalquestions_closed_book 11713ext/gpt-4-0613
Academic knowledge and reasoningepoch_gpqa_diamond, epoch_mmlu, epoch_open_book_qa, gpqa, helm_cap_gpqa, helm_cap_mmlu_pro, helm_lite_mmlu, helm_lite_openbookqa, mmlu_pro 509579ext/gemini-3-pro-preview
Mathematical problem solvingepoch_gsm8k, epoch_math_level_5, helm_cap_omni_math, helm_lite_gsm8k, helm_lite_math, math_level5 490266deepseek-ai/DeepSeek-V3
Passage-grounded question answeringhelm_lite_narrativeqa 910none yet
Compliance with explicit constraintshelm_cap_ifeval, ifeval 46240none yet
Valid structured formatshelm_cap_ifeval, ifeval 46240none yet
Multi-step logical reasoningbbh, epoch_bbh, musr 46090none yet
Medicinehelm_lite_medqa, helm_med_medqa 1000none yet
Lawhelm_lite_legalbench 910none yet
Tool and function callingbfcl 780none yet
Code editing and repository repairaider_polyglot, swe_bench_full 650none yet
Multiple-choice answerepoch_gpqa_diamond, epoch_mmlu, epoch_open_book_qa, gpqa, helm_cap_gpqa, helm_cap_mmlu_pro, helm_lite_medqa, helm_lite_mmlu, helm_lite_openbookqa, helm_med_medqa, mmlu_pro 509685ext/gpt-5-2025-08-07
Open short answerepoch_gsm8k, epoch_math_level_5, epoch_trivia_qa, helm_lite_gsm8k, helm_lite_naturalquestions_closed_book, math_level5 487942deepseek-ai/DeepSeek-V3
Context suppliedhelm_lite_narrativeqa 910none yet
Elementary difficultyepoch_gsm8k, epoch_open_book_qa, helm_lite_gsm8k, helm_lite_openbookqa 25417ext/gpt-4-0613
Advanced difficultyepoch_gpqa_diamond, epoch_math_level_5, gpqa, helm_cap_gpqa, helm_cap_omni_math, math_level5 493128ext/gemini-3-pro-preview

Rankings are provisional. A rank needs only 2 tests from 2 leaderboards, some sources are stale (see Sources), and the review queue is undecided. Entries below that bar are scored but not ranked. A new benchmark gets a skill only after a person approves it in the review queue.

What changed since the previous release

Most "scores changed" are small drifts in live scores such as LMArena Elo, not corrections.

models added0
models removed0
scores changed0
scores added0
scores removed0
review queue415
stale: aiderupstream data dated 2025-10-03 (402 days old, limit 365)
stale: helm_liteupstream data dated 2025-01-10 (668 days old, limit 365)
stale: swebenchupstream data dated 2026-02-26 (256 days old, limit 120)

Full changelog · all releases

Release checks
CheckResultDetail
at_least_one_live_sourcepass12 live sources with data
coverage_labels_agreepassone coverage label per benchmark per skill
duplicate_axes_droppedpass0 scores left on duplicate axes
every_score_traces_to_snapshotpass0 scores without a raw row; 0 snapshot problems
foreign_keyspass0 foreign-key violations
no_subset_next_to_wholepass0 used links of a subset next to its whole, or of a total
one_primary_axis_per_benchmarkpassone total weight per benchmark per source
one_run_per_variant_per_axispass0 variant/axis pairs with more than one score
rank_needs_two_axes_two_sourcespass0 ranked entries with fewer than 2 axes from 2 sources
systems_not_rankedpass0 skill scores belong to systems or unreviewed borderline entries
zero_never_next_to_realpass0 zeros kept next to a real score
review_queue (not blocking)pass415 open items
sources_healthy (not blocking)warningnot green: aider yellow, swebench yellow, helm_lite yellow
Rules built into every release
  • One run per model per test: a 0 next to a real score is treated as missing, precisions and quantisations stay separate variants, then the latest run is kept; runs more than 20% of the test's range apart are flagged.
  • A benchmark has one coverage label per skill, whichever leaderboard it comes from; a conflict stops the release.
  • A skill score needs at least 2 tests from 2 leaderboards to be ranked; an evidence-adjusted score is stored next to the mean.
  • One total weight per benchmark per leaderboard: no subset next to its whole, no total next to its parts, no identical duplicate tests.
  • Systems (agents, model pairs, best-of-N, test-time scaling) get their own ids and are never ranked or filed as the bare model.
  • Names are merged automatically only when they differ by capitals or match a Hugging Face id; the rest go to the review queue.
  • Missing is never zero: failed-run zeros and out-of-range values are recorded as missing.
  • Every score points to its raw row in a stored, hash-checked snapshot.
DMSL @ UCY
MSbench, the LLM-MS capability catalogue · Data Management Systems Laboratory (DMSL), University of Cyprus · Scores belong to their leaderboards, see Sources
Powered by Datasette