Who is best at what
For each skill, the Index ranks models within every test linked to it and averages those ranks. Click a skill to see its ranking. These are catalogue v3's rankings as built: many top spots rest on one test or one leaderboard, so compare the tests and leaderboards columns and read Known issues.
Skills
| Skill | Tests | Leaderboards | Models with a score |
|---|---|---|---|
| Mathematical problem solving Solving a stated mathematical problem, from grade-school word problems to competition and olympiad level. |
33 | 9 | 12,019 |
| Multi-step logical reasoning Deduction, constraint puzzles and narrative-based inference. |
6 | 4 | 11,169 |
| Academic knowledge and reasoning Examination questions in academic and professional subjects; often needs applying the knowledge. |
56 | 7 | 10,599 |
| Medicine Clinical and biomedical questions, such as medical licensing examination problems. |
19 | 5 | 6,631 |
| Law Legal reasoning over statutes, contracts, cases and legal procedure. |
7 | 4 | 6,545 |
| Finance Reasoning about financial documents, quantities, accounting and markets. |
5 | 3 | 6,420 |
| Chemistry Solving chemistry problems about substances, reactions and quantities. |
6 | 4 | 6,300 |
| Physics Solving physics problems and explaining physical phenomena. |
7 | 3 | 6,241 |
| Open factual recall Short factual answers to questions people ask, no options and no passage. |
7 | 3 | 6,080 |
| Truthfulness under misleading premises Giving the true answer where a common misconception or a misleading cue invites a false one. |
3 | 2 | 6,006 |
| Compliance with explicit constraints Meeting verifiable constraints on length, wording, layout or content. |
6 | 5 | 4,980 |
| Valid structured formats Generating or converting JSON, YAML, CSV, HTML and similar so that it parses or renders. |
12 | 5 | 4,668 |
| Passage-grounded question answering Answering a question from a supplied passage, including recognising that the passage has no answer. |
7 | 4 | 1,870 |
| Code generation and completion Writing code that passes tests, from a docstring or a prose request. |
16 | 3 | 703 |
| Code editing and repository repair Changing existing code to fix a reported issue or meet a new requirement, in a file or across a repository. |
9 | 4 | 641 |
| Reasoning about given code Predicting what code does or locating code in a repository, without writing a solution. |
19 | 4 | 533 |
| Emotion understanding Recognising and reasoning about emotions expressed in text or dialogue. |
1 | 1 | 172 |
| Faithfulness to a supplied source Staying consistent with supplied text in summaries, dialogue or fact checking; detecting unsupported statements. |
5 | 1 | 127 |
| Tool and function calling Choosing the function or tool a request needs and filling its arguments, or recognising that none of the offered tools fits. |
25 | 1 | 81 |
Task characteristics (the form of a request, not a subject)
Examples are multiple choice, code stub and supplied context. The Index classifier uses 17 of the 19 skills above; tool_use and code_editing are left out, so their scores do not influence which model is picked VERIFIED.
| Characteristic | Tests | Models |
|---|---|---|
| Multiple-choice answer The answer is one of listed options. |
13 | 4,705 |
| Open short answer The answer is free text matched against references. |
17 | 4,659 |
| Context supplied A passage, document or code is part of the request. |
19 | 260 |
| Elementary difficulty School-level material. |
5 | 124 |
| Advanced difficulty Graduate, professional or competition level. |
5 | 4,560 |
| Request written as prose The task is described in natural language, not as a code stub or template. |
2 | 171 |
| Request given as a code stub A function signature and docstring to complete. |
2 | 198 |
| Real user request Taken from real conversations; often needs several abilities at once. |
0 | 0 |
| Chain-of-thought protocol The evaluation asks the model to reason before answering; a property of the protocol, not of the question. |
9 | 150 |
Ranks are recomputed here (each model's percentile within each test, then the mean per skill). The Index's exact tie-break and its 80% eligibility rule are not stored in the catalogue, so these rankings approximate the Index INFERRED. Each ranking also shows an evidence-adjusted score, 50 + (mean − 50) × n / (n + 2) with n the number of tests: our proposal for thin evidence, not what the Index does today.