Automated catalogue
A second catalogue, separate from v3, rebuilt from live leaderboards when the pipeline is run (started by hand, no schedule yet). The review's fixes are enforced as checks: a release goes out only when every blocking check passes, otherwise the previous one stays.
Current release automated_2026-11-08_190934
Built 2026-11-08 19:09 UTC from 13 sources with data · sources: 9 green 3 yellow 1 frozen · 2 more switched off
Skills with evidence
| Skill | Benchmarks used | Scored | Ranked | Top ranked |
|---|---|---|---|---|
| Open factual recall | epoch_trivia_qa, helm_lite_naturalquestions_closed_book | 117 | 13 | ext/gpt-4-0613 |
| Academic knowledge and reasoning | epoch_gpqa_diamond, epoch_mmlu, epoch_open_book_qa, gpqa, helm_cap_gpqa, helm_cap_mmlu_pro, helm_lite_mmlu, helm_lite_openbookqa, mmlu_pro | 5095 | 79 | ext/gemini-3-pro-preview |
| Mathematical problem solving | epoch_gsm8k, epoch_math_level_5, helm_cap_omni_math, helm_lite_gsm8k, helm_lite_math, math_level5 | 4902 | 66 | deepseek-ai/DeepSeek-V3 |
| Passage-grounded question answering | helm_lite_narrativeqa | 91 | 0 | none yet |
| Compliance with explicit constraints | helm_cap_ifeval, ifeval | 4624 | 0 | none yet |
| Valid structured formats | helm_cap_ifeval, ifeval | 4624 | 0 | none yet |
| Multi-step logical reasoning | bbh, epoch_bbh, musr | 4609 | 0 | none yet |
| Medicine | helm_lite_medqa, helm_med_medqa | 100 | 0 | none yet |
| Law | helm_lite_legalbench | 91 | 0 | none yet |
| Tool and function calling | bfcl | 78 | 0 | none yet |
| Code editing and repository repair | aider_polyglot, swe_bench_full | 65 | 0 | none yet |
| Multiple-choice answer | epoch_gpqa_diamond, epoch_mmlu, epoch_open_book_qa, gpqa, helm_cap_gpqa, helm_cap_mmlu_pro, helm_lite_medqa, helm_lite_mmlu, helm_lite_openbookqa, helm_med_medqa, mmlu_pro | 5096 | 85 | ext/gpt-5-2025-08-07 |
| Open short answer | epoch_gsm8k, epoch_math_level_5, epoch_trivia_qa, helm_lite_gsm8k, helm_lite_naturalquestions_closed_book, math_level5 | 4879 | 42 | deepseek-ai/DeepSeek-V3 |
| Context supplied | helm_lite_narrativeqa | 91 | 0 | none yet |
| Elementary difficulty | epoch_gsm8k, epoch_open_book_qa, helm_lite_gsm8k, helm_lite_openbookqa | 254 | 17 | ext/gpt-4-0613 |
| Advanced difficulty | epoch_gpqa_diamond, epoch_math_level_5, gpqa, helm_cap_gpqa, helm_cap_omni_math, math_level5 | 4931 | 28 | ext/gemini-3-pro-preview |
Rankings are provisional. A rank needs only 2 tests from 2 leaderboards, some sources are stale (see Sources), and the review queue is undecided. Entries below that bar are scored but not ranked. A new benchmark gets a skill only after a person approves it in the review queue.
What changed since the previous release
Most "scores changed" are small drifts in live scores such as LMArena Elo, not corrections.
| models added | 0 |
| models removed | 0 |
| scores changed | 0 |
| scores added | 0 |
| scores removed | 0 |
| review queue | 415 |
stale: aider | upstream data dated 2025-10-03 (402 days old, limit 365) |
stale: helm_lite | upstream data dated 2025-01-10 (668 days old, limit 365) |
stale: swebench | upstream data dated 2026-02-26 (256 days old, limit 120) |
Full changelog · all releases
Release checks
| Check | Result | Detail |
|---|---|---|
at_least_one_live_source | pass | 12 live sources with data |
coverage_labels_agree | pass | one coverage label per benchmark per skill |
duplicate_axes_dropped | pass | 0 scores left on duplicate axes |
every_score_traces_to_snapshot | pass | 0 scores without a raw row; 0 snapshot problems |
foreign_keys | pass | 0 foreign-key violations |
no_subset_next_to_whole | pass | 0 used links of a subset next to its whole, or of a total |
one_primary_axis_per_benchmark | pass | one total weight per benchmark per source |
one_run_per_variant_per_axis | pass | 0 variant/axis pairs with more than one score |
rank_needs_two_axes_two_sources | pass | 0 ranked entries with fewer than 2 axes from 2 sources |
systems_not_ranked | pass | 0 skill scores belong to systems or unreviewed borderline entries |
zero_never_next_to_real | pass | 0 zeros kept next to a real score |
review_queue (not blocking) | pass | 415 open items |
sources_healthy (not blocking) | warning | not green: aider yellow, swebench yellow, helm_lite yellow |
Rules built into every release
- One run per model per test: a 0 next to a real score is treated as missing, precisions and quantisations stay separate variants, then the latest run is kept; runs more than 20% of the test's range apart are flagged.
- A benchmark has one coverage label per skill, whichever leaderboard it comes from; a conflict stops the release.
- A skill score needs at least 2 tests from 2 leaderboards to be ranked; an evidence-adjusted score is stored next to the mean.
- One total weight per benchmark per leaderboard: no subset next to its whole, no total next to its parts, no identical duplicate tests.
- Systems (agents, model pairs, best-of-N, test-time scaling) get their own ids and are never ranked or filed as the bare model.
- Names are merged automatically only when they differ by capitals or match a Hugging Face id; the rest go to the review queue.
- Missing is never zero: failed-run zeros and out-of-range values are recorded as missing.
- Every score points to its raw row in a stored, hash-checked snapshot.