Known issues
Problems found in the November 2026 review of catalogue v3 that affect whether a ranking can be trusted, worst first. Each was checked with a query. Proposed fixes are in the full report (queries); the automated catalogue enforces most of them as build checks.
VERIFIED the number comes from a query INFERRED a reasonable reading, not proven UNKNOWN the catalogue cannot answer it
One archived leaderboard carries most of the evidence
10,744 of the 12,951 models the Index can rank have scores only from the archived Open LLM Leaderboard (v1 and v2, 2023-24) VERIFIED. Most rankings compare old open fine-tunes.
One test can win a skill
A skill score is the mean rank over whatever tests a model has, so one top score beats a broad record. #1 for truthfulness is Contamination/contaminated_proof_7b_v1.0; #1 for finance is 8xqmff94/slm with one accounting score VERIFIED. See any ranking on Skills.
Same benchmark, different labels
TruthfulQA, GSM8K and GPQA each have one test labelled "direct" while the same benchmark elsewhere, and at benchmark level, says "partial". 5,869 models' factual-recall score and 6,515 models' reasoning score rest on those single tests VERIFIED.
Repeat runs are averaged
34,481 model-test pairs average several runs, and 19,293 of those on Index tests disagree. Example: o3 on a BFCL tool-use test scores 0 and 86.5 and is shown as 43 VERIFIED. See v_axis_submissions.
Popularity votes counted as direct proof
22 LMArena preference ratings are linked "direct" to code, maths, law, medicine and finance, while their own notes call them "a rating, not a graded answer key" VERIFIED.
The same questions counted several times
Subsets next to full tests, totals next to their own parts (BFCL overall plus 24 sub-scores), and two LMArena columns that are exact copies, 140 of 140 scores equal VERIFIED.
Systems filed as plain models
The catalogue flags 18 non-single-model entries, but the Index does not exclude them, and more are unflagged: SWE-bench agents (CodeAct, swe-search) filed under Claude 3.5 Sonnet, an unflagged Grok multi-agent id, NexusSciAgent VERIFIED.
One model under many names
Claude Opus 4.7 appears under 5+ ids; 58 groups differ only by capital letters; 71 ext/ ids match a Hugging Face id VERIFIED. Evidence is split, so each id looks thinner.
Scores that look like missing data
A finance test where 18 of 21 models score 0; SWE-bench's metric column holds a submission name; 17 Index tests have fewer than 20 models VERIFIED.
Views that mislead
v_capability_coverage_summary counts at benchmark level and disagrees with the Index (factual recall 211 vs 6,080 models). v_unverified flags 7,243 of the ranked models, so it cannot be used as a filter as is VERIFIED.
Checked and fine
No broken links between tables (0 orphans across 56 foreign keys); every test the Index uses says higher is better, and none is an error rate or a cost; no mixed scales inside one test VERIFIED.