About
What the catalogue is for, the words it uses, and what state it is in.
What it is for
LLM-MS (LLM Meta-Search) is a decentralized edge architecture for discovering, selecting and running language models over a peer-to-peer network, developed at the Data Management Systems Laboratory (DMSL), University of Cyprus. It has three components: the Index ranks candidate models using their benchmark results and the hardware available, the Lookup finds the peers that host those models, and the Search combines the answers of several models under a token budget.
This catalogue is the Index's knowledge of the models: which public tests each model has taken, what those tests measure, and how it scored. For each skill, the Index takes the tests linked to it as direct evidence, ranks models within each test, and averages the ranks. Four tables decide this: axis_capability_coverage (which test counts for which skill), measurement_axes (the tests), score_observations (the scores) and capabilities (the skills). Every other table justifies a link, describes a model or documents provenance.
Catalogue v3 was built by Konstantin Krasovitskiy; the review and this site are by Ibrahim AlMafrachi, supervised by Demetrios Zeinalipour-Yazti.
Words used here
Benchmark = the exam (for example GSM8K). Leaderboard or source = a published table of results; one source can hold several benchmarks. Axis (also called a test) = one source + one benchmark + one metric; raw scores are comparable only within one axis. Skill = a capability such as maths or law; a task characteristic describes the form of a request instead. Index = the LLM-MS stage that uses these tests to pick models. Ranked = a model with at least 2 tests from 2 leaderboards for that skill.
Licences
The scores belong to the leaderboards they come from. Each source has its own licence, and some limit what can be republished. Before this goes public every one has to be settled; today the demo is private. Open points:
- Scale AI (SEAL): private research use only. Scale has been asked about reuse; no reply yet. This one blocks any public release.
- SWE-bench: CC BY-NC 4.0, non-commercial only.
- LiveCodeBench: kept for private use only and excluded from public releases (its website repository is CC BY-SA).
- Epoch AI: CC BY, but only for benchmarks Epoch ran itself. Results it merely collected from others follow the original owner's terms.
- Credit lines: BRIDGE, Open LLM Leaderboard v2 and HELM ask to be credited and are.
- Removed: Artificial Analysis (its terms forbid scraping). LiveBench is switched off until its terms are confirmed.
| Automated source | Licence |
|---|---|
| Aider polyglot leaderboard | Apache-2.0 (aider repository) |
| BRIDGE medical leaderboard | MIT (Space metadata and BRIDGE code repo); no licence text for the scores themselves. Credit: Wu et al., BRIDGE, Nature Biomedical Engineering 2026, doi 10.1038/s41551-026-01719-2. We take scores only, never clinical text. |
| Berkeley Function-Calling Leaderboard (BFCL) | Apache-2.0 (BFCL-Result repository) |
| Epoch AI Benchmarking Hub | CC BY (epoch.ai/benchmarks/use-this-data): credit Epoch AI. Covers only benchmarks Epoch ran itself (named '(Epoch run)'); benchmarks named 'collected by Epoch' are external data that keeps its original licence. Public release: Epoch-run only. |
| HELM Capabilities (Stanford CRFM) | Apache-2.0 (HELM project); the results bucket has no licence notice of its own. Credit: Stanford CRFM HELM. |
| HELM Lite (Stanford CRFM) | Apache-2.0 (HELM project); the results bucket has no licence notice of its own. Credit: Stanford CRFM HELM. |
| LMArena leaderboards (Hugging Face dataset) | CC-BY-4.0 (dataset card) |
| LiveCodeBench (code generation) | Main repo MIT; Hugging Face Space MIT; website repo CC BY-SA 4.0 (README). Private use only: excluded from public releases (decision 8 Nov 2026). |
| MedHELM (Stanford CRFM) | Apache-2.0 (HELM project); the results bucket has no licence notice of its own. Credit: Stanford CRFM HELM. |
| Open LLM Leaderboard v2 (archived) | Apache-2.0 (leaderboard Space); the contents and results datasets state no licence (checked 8 Nov 2026). Credit: Open LLM Leaderboard v2, Fourrier et al., Hugging Face 2024. |
| SWE-bench leaderboards | CC BY-NC 4.0 (swe-bench.github.io repository LICENSE): non-commercial only; credit SWE-bench. The SWE-bench code repo is MIT but is not the data we pull. |
| Scale AI SEAL leaderboards | Terms for reuse not checked (UNKNOWN); private research use only |
| Vectara hallucination leaderboard (HHEM) | Apache-2.0 (repository) |
The 51 leaderboards behind the thesis catalogue (v3) carry the licence recorded in its source_provenance table; 7 are not recorded.