MSbench
Thesis catalogue (v3)coming soon Automated catalogue Skills Sources Known issues Downloads & APIcoming soon About
DMSL @ UCY
Preview · the database itself is not public yet · data shown as built, issues documented on Known issues

Known issues

Problems found in the November 2026 review of catalogue v3 that affect whether a ranking can be trusted, worst first. Each was checked with a query. Proposed fixes are in the full report (queries); the automated catalogue enforces most of them as build checks.

VERIFIED the number comes from a query    INFERRED a reasonable reading, not proven    UNKNOWN the catalogue cannot answer it

One archived leaderboard carries most of the evidence

10,744 of the 12,951 models the Index can rank have scores only from the archived Open LLM Leaderboard (v1 and v2, 2023-24) VERIFIED. Most rankings compare old open fine-tunes.

One test can win a skill

A skill score is the mean rank over whatever tests a model has, so one top score beats a broad record. #1 for truthfulness is Contamination/contaminated_proof_7b_v1.0; #1 for finance is 8xqmff94/slm with one accounting score VERIFIED. See any ranking on Skills.

Same benchmark, different labels

TruthfulQA, GSM8K and GPQA each have one test labelled "direct" while the same benchmark elsewhere, and at benchmark level, says "partial". 5,869 models' factual-recall score and 6,515 models' reasoning score rest on those single tests VERIFIED.

Repeat runs are averaged

34,481 model-test pairs average several runs, and 19,293 of those on Index tests disagree. Example: o3 on a BFCL tool-use test scores 0 and 86.5 and is shown as 43 VERIFIED. See v_axis_submissions.

Popularity votes counted as direct proof

22 LMArena preference ratings are linked "direct" to code, maths, law, medicine and finance, while their own notes call them "a rating, not a graded answer key" VERIFIED.

The same questions counted several times

Subsets next to full tests, totals next to their own parts (BFCL overall plus 24 sub-scores), and two LMArena columns that are exact copies, 140 of 140 scores equal VERIFIED.

Systems filed as plain models

The catalogue flags 18 non-single-model entries, but the Index does not exclude them, and more are unflagged: SWE-bench agents (CodeAct, swe-search) filed under Claude 3.5 Sonnet, an unflagged Grok multi-agent id, NexusSciAgent VERIFIED.

One model under many names

Claude Opus 4.7 appears under 5+ ids; 58 groups differ only by capital letters; 71 ext/ ids match a Hugging Face id VERIFIED. Evidence is split, so each id looks thinner.

Scores that look like missing data

A finance test where 18 of 21 models score 0; SWE-bench's metric column holds a submission name; 17 Index tests have fewer than 20 models VERIFIED.

Views that mislead

v_capability_coverage_summary counts at benchmark level and disagrees with the Index (factual recall 211 vs 6,080 models). v_unverified flags 7,243 of the ranked models, so it cannot be used as a filter as is VERIFIED.

Checked and fine

No broken links between tables (0 orphans across 56 foreign keys); every test the Index uses says higher is better, and none is an error rate or a cost; no mixed scales inside one test VERIFIED.

DMSL @ UCY
MSbench, the LLM-MS capability catalogue · Data Management Systems Laboratory (DMSL), University of Cyprus · Scores belong to their leaderboards, see Sources
Powered by Datasette