MSbench
Thesis catalogue (v3)coming soon Automated catalogue Skills Sources Known issues Downloads & APIcoming soon About
DMSL @ UCY
Preview · the database itself is not public yet · data shown as built, issues documented on Known issues

Who is best at what

For each skill, the Index ranks models within every test linked to it and averages those ranks. Click a skill to see its ranking. These are catalogue v3's rankings as built: many top spots rest on one test or one leaderboard, so compare the tests and leaderboards columns and read Known issues.

Skills

SkillTestsLeaderboardsModels with a score
Mathematical problem solving
Solving a stated mathematical problem, from grade-school word problems to competition and olympiad level.
33 9 12,019
Multi-step logical reasoning
Deduction, constraint puzzles and narrative-based inference.
6 4 11,169
Academic knowledge and reasoning
Examination questions in academic and professional subjects; often needs applying the knowledge.
56 7 10,599
Medicine
Clinical and biomedical questions, such as medical licensing examination problems.
19 5 6,631
Law
Legal reasoning over statutes, contracts, cases and legal procedure.
7 4 6,545
Finance
Reasoning about financial documents, quantities, accounting and markets.
5 3 6,420
Chemistry
Solving chemistry problems about substances, reactions and quantities.
6 4 6,300
Physics
Solving physics problems and explaining physical phenomena.
7 3 6,241
Open factual recall
Short factual answers to questions people ask, no options and no passage.
7 3 6,080
Truthfulness under misleading premises
Giving the true answer where a common misconception or a misleading cue invites a false one.
3 2 6,006
Compliance with explicit constraints
Meeting verifiable constraints on length, wording, layout or content.
6 5 4,980
Valid structured formats
Generating or converting JSON, YAML, CSV, HTML and similar so that it parses or renders.
12 5 4,668
Passage-grounded question answering
Answering a question from a supplied passage, including recognising that the passage has no answer.
7 4 1,870
Code generation and completion
Writing code that passes tests, from a docstring or a prose request.
16 3 703
Code editing and repository repair
Changing existing code to fix a reported issue or meet a new requirement, in a file or across a repository.
9 4 641
Reasoning about given code
Predicting what code does or locating code in a repository, without writing a solution.
19 4 533
Emotion understanding
Recognising and reasoning about emotions expressed in text or dialogue.
1 1 172
Faithfulness to a supplied source
Staying consistent with supplied text in summaries, dialogue or fact checking; detecting unsupported statements.
5 1 127
Tool and function calling
Choosing the function or tool a request needs and filling its arguments, or recognising that none of the offered tools fits.
25 1 81
Task characteristics (the form of a request, not a subject)

Examples are multiple choice, code stub and supplied context. The Index classifier uses 17 of the 19 skills above; tool_use and code_editing are left out, so their scores do not influence which model is picked VERIFIED.

CharacteristicTestsModels
Multiple-choice answer
The answer is one of listed options.
13 4,705
Open short answer
The answer is free text matched against references.
17 4,659
Context supplied
A passage, document or code is part of the request.
19 260
Elementary difficulty
School-level material.
5 124
Advanced difficulty
Graduate, professional or competition level.
5 4,560
Request written as prose
The task is described in natural language, not as a code stub or template.
2 171
Request given as a code stub
A function signature and docstring to complete.
2 198
Real user request
Taken from real conversations; often needs several abilities at once.
0 0
Chain-of-thought protocol
The evaluation asks the model to reason before answering; a property of the protocol, not of the question.
9 150

Ranks are recomputed here (each model's percentile within each test, then the mean per skill). The Index's exact tie-break and its 80% eligibility rule are not stored in the catalogue, so these rankings approximate the Index INFERRED. Each ranking also shows an evidence-adjusted score, 50 + (mean − 50) × n / (n + 2) with n the number of tests: our proposal for thin evidence, not what the Index does today.

DMSL @ UCY
MSbench, the LLM-MS capability catalogue · Data Management Systems Laboratory (DMSL), University of Cyprus · Scores belong to their leaderboards, see Sources
Powered by Datasette