summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2026-06-19 22:03:26 +0300
committerPaul Buetow <paul@buetow.org>2026-06-19 22:03:26 +0300
commit55a2b349727f527acc3e4294539e35a012c15784 (patch)
treef11554176b99ebd69616b6754ee4222cb24732de
parent9e9f16b4e7f7e125d891d6414b66d4d75ec940bc (diff)
llm-benchmark-comparison: split oversized SKILL.md into references/
Reorganize the skill to follow the f3s / blog-writing-style pattern: a slim SKILL.md (purpose, trigger, inputs, core workflow, reference index) plus three focused reference files. Pure reorganization — all content preserved verbatim. - references/search-and-sourcing.md: search endpoints, per-model queries, source trust order, source endpoint URLs (workflow steps 2-3) - references/benchmarks-catalog.md: per-model fields-to-capture table and the 2026 benchmark glossary - references/output-and-analysis.md: comparison table structure, analysis sections, anti-patterns, output format, output shape (workflow steps 4-7) SKILL.md drops from 225 to 86 lines. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
-rw-r--r--prompts/skills/llm-benchmark-comparison/SKILL.md211
-rw-r--r--prompts/skills/llm-benchmark-comparison/references/benchmarks-catalog.md52
-rw-r--r--prompts/skills/llm-benchmark-comparison/references/output-and-analysis.md103
-rw-r--r--prompts/skills/llm-benchmark-comparison/references/search-and-sourcing.md49
4 files changed, 240 insertions, 175 deletions
diff --git a/prompts/skills/llm-benchmark-comparison/SKILL.md b/prompts/skills/llm-benchmark-comparison/SKILL.md
index 7b55f6c..c9491df 100644
--- a/prompts/skills/llm-benchmark-comparison/SKILL.md
+++ b/prompts/skills/llm-benchmark-comparison/SKILL.md
@@ -19,6 +19,14 @@ into a blog post, RFC, or procurement doc.
- The user wants a model-selection report: benchmarks, pricing, context,
license, modalities — in one table.
+## Reference Files
+
+Detailed reference documentation is in the `references/` subfolder:
+
+- [Search & Sourcing](references/search-and-sourcing.md) — search endpoints (DuckDuckGo HTML, Brave), per-model queries, the source trust order (vendor blog → HF model card → Artificial Analysis → pricing pages → independent reviews), and the raw source endpoint URLs. Covers workflow steps 2–3.
+- [Benchmarks Catalog](references/benchmarks-catalog.md) — the per-model fields-to-capture table (where to find each one) and the 2026 benchmark glossary (SWE-Bench Pro/Verified, Terminal-Bench, BrowseComp, HLE, τ²-Bench, MCP-Atlas, GPQA-Diamond, AIME, AA Intelligence Index, AA-Omniscience, GDPval-AA Elo).
+- [Output & Analysis](references/output-and-analysis.md) — the copy-paste comparison table structure, the ordered analysis sections (quick reference, headline table, how-to-read, cost reality check, caveats, sources), the anti-patterns to avoid, the output-format rules, and the target output shape. Covers workflow steps 4–7.
+
## Inputs
Resolve these before searching; if the user gave you a list of models, use
@@ -39,187 +47,40 @@ point release every 1–3 months; `Kimi K2.6` is different from `Kimi K2.5` or
### 2. Search the web for each model
-Prefer the **HTML DuckDuckGo endpoint** (`https://html.duckduckgo.com/html/?q=…`)
-because the JSON API and the JS site return empty in this environment. Brave
-Search (`https://search.brave.com/search?q=…`) is a useful fallback. Bing
-and Google both gate on CAPTCHA.
-
-Useful per-model queries:
-
-- `"<model>" benchmark scores`
-- `"<model>" SWE-Bench Pro` (the de-facto coding test in 2026)
-- `"<model>" pricing context window`
-- `"<model>" site:huggingface.co` (model card)
-- `"<model>" site:artificialanalysis.ai` (independent evals)
-- `"<model>" review vs` (head-to-head pieces)
+Use the search endpoints and per-model queries in
+[references/search-and-sourcing.md](references/search-and-sourcing.md).
+Prefer the HTML DuckDuckGo endpoint; Brave is the fallback.
### 3. Pull authoritative sources
-In this order of trust:
-
-1. **Vendor blog / launch post** — gives headline numbers, but watch for
- cherry-picking and self-reported harnesses.
-2. **Hugging Face model card** — usually has the most complete benchmark
- table; check the license, params, context.
-3. **Artificial Analysis article** — independent evals (Intelligence
- Index, GDPval-AA Elo, AA-Omniscience hallucination rate).
-4. **OpenRouter / LLM-Stats pricing pages** — current $/M token rates.
-5. **Independent reviews** — Lushbinary, OfficeChai, Analytics India,
- Geeky Gadgets; useful for "what the numbers hide" and real-world
- anecdote.
-
-For each model, capture:
-
-| Field | Where to find it |
-|---|---|
-| Lab + release date | Vendor blog or HF model card |
-| Architecture (dense/MoE, total/active params, attention type) | HF model card or technical report |
-| Context window | Vendor page, often in the model card |
-| Modalities | HF model card, README |
-| License | HF model card (top of page) |
-| **SWE-Bench Pro** | Almost always in launch blog |
-| **SWE-Bench Verified** | Vendor blog, leaderboards |
-| **Terminal-Bench 2.0 / 2.1** | Vendor blog, "agentic" tables |
-| **BrowseComp** | Agentic browsing tests; check vendor and AA |
-| **HLE (Humanity's Last Exam)** | With and without tools, often reported both |
-| **GPQA-Diamond** | Reasoning benchmark; check leaderboards |
-| **AIME 2026** | Math reasoning; check leaderboards |
-| **τ²-Bench / τ²-Bench Telecom** | Tool-use / agentic |
-| **MCP-Atlas** | Tool-use over MCP |
-| **AA Intelligence Index** | artificialanalysis.ai |
-| **AA-Omniscience hallucination** | artificialanalysis.ai |
-| **Input $/M, Output $/M** | OpenRouter, vendor pricing page, LLM-Stats |
-
-If a number is missing after two passes, mark it `n/a` — do not invent.
+Follow the source trust order and capture the per-model fields. The trust
+order and source endpoints are in
+[references/search-and-sourcing.md](references/search-and-sourcing.md); the
+fields-to-capture table and benchmark glossary are in
+[references/benchmarks-catalog.md](references/benchmarks-catalog.md). If a
+number is missing after two passes, mark it `n/a` — do not invent.
### 4. Build the comparison table
-Use this structure (markdown, copy-paste-ready):
-
-```markdown
-| | **Model A** | **Model B** | **Model C** |
-|---|---|---|---|
-| Lab | ... | ... | ... |
-| Released | YYYY-MM-DD | ... | ... |
-| Architecture | ... | ... | ... |
-| Context | ... | ... | ... |
-| Modalities | ... | ... | ... |
-| License | ... | ... | ... |
-| Input $/M (promo/std) | ... | ... | ... |
-| Output $/M | ... | ... | ... |
-
-| Benchmark | Model A | Model B | Model C | Notes |
-|---|---:|---:|---:|---|
-| SWE-Bench Pro | ... | ... | ... | ... |
-| SWE-Bench Verified | ... | ... | ... | ... |
-| Terminal-Bench 2.x | ... | ... | ... | ... |
-| BrowseComp | ... | ... | ... | ... |
-| HLE (w/ tools) | ... | ... | ... | ... |
-| GPQA-Diamond | ... | ... | ... | ... |
-| AIME 2026 | ... | ... | ... | ... |
-| τ²-Bench Telecom | ... | ... | ... | ... |
-| MCP-Atlas | ... | ... | ... | ... |
-| AA Intelligence Index | ... | ... | ... | ... |
-```
-
-Right-align numeric columns (`---:`) so the digits line up. Round benchmark
-percentages to one decimal. Keep the Notes column short — one phrase, not
-a sentence.
+Use the copy-paste-ready table structure and formatting rules in
+[references/output-and-analysis.md](references/output-and-analysis.md).
### 5. Write the analysis
-After the table, include these sections in order:
-
-1. **Quick reference** — a one-row-per-model summary table covering the
- "what is it" fields (lab, release, arch, context, modalities, license,
- price). Optional but useful when the user has 3+ models.
-2. **Headline benchmark table** — the long table above.
-3. **How to read this** — 4–8 bullets, each one a claim grounded in the
- numbers:
- - "Coding is effectively a tie at 58–59% on SWE-Bench Pro"
- - "K2.6 is the most battle-tested for long-horizon work (13-hour
- exchange-core rewrite, 5-day autonomous ops agent)"
- - "M3's pitch is cheap 1M context, GLM 5.1's is breadth, K2.6's is
- tool-use + reasoning"
-4. **Cost reality check** — a per-task worked example. Standard
- workload: 500K input tokens + 100K output tokens. Compute
- `(0.5 × input_$/M) + (0.1 × output_$/M)` per model. Reference Claude
- Opus-class price as a baseline.
-5. **Caveats** — required. Cover at least:
- - Vendor-published numbers (most are)
- - Harness / scaffold differences (Claude Code, Terminus, OpenHands,
- Mini-SWE-Agent) — same model scores differently across harnesses
- - Token usage differences (K2.6 uses ~2× K2.5 tokens; M3's 1M context
- costs more tokens per turn)
- - Open-weights claim status (M3 weights + technical report promised
- ~10 days post-launch, not day-one in June 2026)
- - Independent verification status
-6. **Sources** — list the URLs you actually pulled from, not the
- search-results page.
-
-### 6. Anti-patterns to avoid
-
-- **Don't average across benchmarks** ("average score 73%") — hides the
- shape. SWE-Bench Pro 58% and AIME 95% are not commensurable.
-- **Don't quote vendor numbers without a harness caveat.** A 59% on
- SWE-Bench Pro run with Claude Code scaffolding is not the same as 59%
- with OpenHands.
-- **Don't pick a "winner" unless the user asks.** The whole point of the
- comparison is that the right model depends on the workload: long
- context vs. low cost vs. best tool use vs. strongest reasoning.
-- **Don't fabricate a score** to fill a cell. `n/a` is honest; "60%"
- invented is not.
-- **Don't bury the cost.** Pricing is often the deciding factor for
- agentic workloads and usually changes the ranking entirely.
-- **Don't ignore the "what the numbers hide" angle.** A Medium-style
- "I tested both on 15 real tasks" piece usually finds that the
- benchmark gap is much smaller than the practical gap (or vice versa).
- Worth citing at least one such piece per comparison.
-
-### 7. Output format
-
-- Plain markdown, ready to paste.
-- Tables first, prose after.
-- One section per analysis point, no walls of text.
-- Sources at the bottom as a bulleted URL list, not inline links (easier
- to copy and verify).
-
-## Quick benchmark glossary (2026)
-
-- **SWE-Bench Pro** — Real GitHub issues, harder subset of SWE-Bench.
- Industry-standard coding test in 2026. Currently ~58% is the open-weight
- SOTA bar.
-- **SWE-Bench Verified** — Human-verified easier subset. Top closed
- models hit 80%+.
-- **Terminal-Bench 2.0 / 2.1** — Real command-line agent tasks.
-- **BrowseComp** — Autonomous web browsing + information retrieval.
-- **HLE (Humanity's Last Exam)** — Hardest reasoning benchmark; usually
- reported with and without tools.
-- **τ²-Bench / τ²-Bench Telecom** — Tool-use in agentic loop, telecom
- domain.
-- **MCP-Atlas** — Tool use over Model Context Protocol.
-- **GPQA-Diamond** — Graduate-level science Q&A.
-- **AIME** — Math competition problems.
-- **AA Intelligence Index** — Composite index from Artificial Analysis
- combining many of the above; 54–57 is the current frontier band.
-- **AA-Omniscience** — Hallucination + abstention metric; lower
- hallucination rate is better.
-- **GDPval-AA Elo** — General agentic performance on knowledge-work
- tasks.
-
-## Example output shape
-
-A minimal good response is two tables + four short sections. A maximal
-good response is two tables + five sections + sources. Anything longer
-than that is padding.
-
-## References
-
-- `https://html.duckduckgo.com/html/?q=...` — primary search endpoint
-- `https://search.brave.com/search?q=...` — fallback search
-- `https://artificialanalysis.ai/articles/...` — independent evals
-- `https://huggingface.co/<org>/<model>` — model cards
-- `https://openrouter.ai/<provider>/<model>/benchmarks` — pricing +
- benchmarks in one place
-- `https://llm-stats.com/home/models/<model>` — pricing + benchmark
- snapshot
+Include the ordered analysis sections (quick reference, headline table,
+how-to-read, cost reality check, caveats, sources) from
+[references/output-and-analysis.md](references/output-and-analysis.md).
+
+### 6. Avoid the anti-patterns
+
+Don't average scores, don't quote vendor numbers without a harness caveat,
+don't pick a "winner" unless asked, don't fabricate scores, don't bury the
+cost, don't ignore the "what the numbers hide" angle. Full list with
+rationale in
+[references/output-and-analysis.md](references/output-and-analysis.md).
+
+### 7. Match the output format
+
+Plain markdown, tables first, sources at the bottom. The output-format rules
+and the target output shape (two tables + 4–5 short sections) are in
+[references/output-and-analysis.md](references/output-and-analysis.md).
diff --git a/prompts/skills/llm-benchmark-comparison/references/benchmarks-catalog.md b/prompts/skills/llm-benchmark-comparison/references/benchmarks-catalog.md
new file mode 100644
index 0000000..8bc29a5
--- /dev/null
+++ b/prompts/skills/llm-benchmark-comparison/references/benchmarks-catalog.md
@@ -0,0 +1,52 @@
+# Benchmark Catalog & Per-Model Fields
+
+What to capture for each model, and what each benchmark actually measures.
+
+## Per-model fields to capture
+
+For each model, capture:
+
+| Field | Where to find it |
+|---|---|
+| Lab + release date | Vendor blog or HF model card |
+| Architecture (dense/MoE, total/active params, attention type) | HF model card or technical report |
+| Context window | Vendor page, often in the model card |
+| Modalities | HF model card, README |
+| License | HF model card (top of page) |
+| **SWE-Bench Pro** | Almost always in launch blog |
+| **SWE-Bench Verified** | Vendor blog, leaderboards |
+| **Terminal-Bench 2.0 / 2.1** | Vendor blog, "agentic" tables |
+| **BrowseComp** | Agentic browsing tests; check vendor and AA |
+| **HLE (Humanity's Last Exam)** | With and without tools, often reported both |
+| **GPQA-Diamond** | Reasoning benchmark; check leaderboards |
+| **AIME 2026** | Math reasoning; check leaderboards |
+| **τ²-Bench / τ²-Bench Telecom** | Tool-use / agentic |
+| **MCP-Atlas** | Tool-use over MCP |
+| **AA Intelligence Index** | artificialanalysis.ai |
+| **AA-Omniscience hallucination** | artificialanalysis.ai |
+| **Input $/M, Output $/M** | OpenRouter, vendor pricing page, LLM-Stats |
+
+If a number is missing after two passes, mark it `n/a` — do not invent.
+
+## Quick benchmark glossary (2026)
+
+- **SWE-Bench Pro** — Real GitHub issues, harder subset of SWE-Bench.
+ Industry-standard coding test in 2026. Currently ~58% is the open-weight
+ SOTA bar.
+- **SWE-Bench Verified** — Human-verified easier subset. Top closed
+ models hit 80%+.
+- **Terminal-Bench 2.0 / 2.1** — Real command-line agent tasks.
+- **BrowseComp** — Autonomous web browsing + information retrieval.
+- **HLE (Humanity's Last Exam)** — Hardest reasoning benchmark; usually
+ reported with and without tools.
+- **τ²-Bench / τ²-Bench Telecom** — Tool-use in agentic loop, telecom
+ domain.
+- **MCP-Atlas** — Tool use over Model Context Protocol.
+- **GPQA-Diamond** — Graduate-level science Q&A.
+- **AIME** — Math competition problems.
+- **AA Intelligence Index** — Composite index from Artificial Analysis
+ combining many of the above; 54–57 is the current frontier band.
+- **AA-Omniscience** — Hallucination + abstention metric; lower
+ hallucination rate is better.
+- **GDPval-AA Elo** — General agentic performance on knowledge-work
+ tasks.
diff --git a/prompts/skills/llm-benchmark-comparison/references/output-and-analysis.md b/prompts/skills/llm-benchmark-comparison/references/output-and-analysis.md
new file mode 100644
index 0000000..ea76f5c
--- /dev/null
+++ b/prompts/skills/llm-benchmark-comparison/references/output-and-analysis.md
@@ -0,0 +1,103 @@
+# Output & Analysis Format
+
+How to build the comparison table, write the analysis, and avoid the common
+traps — steps 4–7 of the workflow plus anti-patterns and the target output
+shape.
+
+## Build the comparison table
+
+Use this structure (markdown, copy-paste-ready):
+
+```markdown
+| | **Model A** | **Model B** | **Model C** |
+|---|---|---|---|
+| Lab | ... | ... | ... |
+| Released | YYYY-MM-DD | ... | ... |
+| Architecture | ... | ... | ... |
+| Context | ... | ... | ... |
+| Modalities | ... | ... | ... |
+| License | ... | ... | ... |
+| Input $/M (promo/std) | ... | ... | ... |
+| Output $/M | ... | ... | ... |
+
+| Benchmark | Model A | Model B | Model C | Notes |
+|---|---:|---:|---:|---|
+| SWE-Bench Pro | ... | ... | ... | ... |
+| SWE-Bench Verified | ... | ... | ... | ... |
+| Terminal-Bench 2.x | ... | ... | ... | ... |
+| BrowseComp | ... | ... | ... | ... |
+| HLE (w/ tools) | ... | ... | ... | ... |
+| GPQA-Diamond | ... | ... | ... | ... |
+| AIME 2026 | ... | ... | ... | ... |
+| τ²-Bench Telecom | ... | ... | ... | ... |
+| MCP-Atlas | ... | ... | ... | ... |
+| AA Intelligence Index | ... | ... | ... | ... |
+```
+
+Right-align numeric columns (`---:`) so the digits line up. Round benchmark
+percentages to one decimal. Keep the Notes column short — one phrase, not
+a sentence.
+
+## Write the analysis
+
+After the table, include these sections in order:
+
+1. **Quick reference** — a one-row-per-model summary table covering the
+ "what is it" fields (lab, release, arch, context, modalities, license,
+ price). Optional but useful when the user has 3+ models.
+2. **Headline benchmark table** — the long table above.
+3. **How to read this** — 4–8 bullets, each one a claim grounded in the
+ numbers:
+ - "Coding is effectively a tie at 58–59% on SWE-Bench Pro"
+ - "K2.6 is the most battle-tested for long-horizon work (13-hour
+ exchange-core rewrite, 5-day autonomous ops agent)"
+ - "M3's pitch is cheap 1M context, GLM 5.1's is breadth, K2.6's is
+ tool-use + reasoning"
+4. **Cost reality check** — a per-task worked example. Standard
+ workload: 500K input tokens + 100K output tokens. Compute
+ `(0.5 × input_$/M) + (0.1 × output_$/M)` per model. Reference Claude
+ Opus-class price as a baseline.
+5. **Caveats** — required. Cover at least:
+ - Vendor-published numbers (most are)
+ - Harness / scaffold differences (Claude Code, Terminus, OpenHands,
+ Mini-SWE-Agent) — same model scores differently across harnesses
+ - Token usage differences (K2.6 uses ~2× K2.5 tokens; M3's 1M context
+ costs more tokens per turn)
+ - Open-weights claim status (M3 weights + technical report promised
+ ~10 days post-launch, not day-one in June 2026)
+ - Independent verification status
+6. **Sources** — list the URLs you actually pulled from, not the
+ search-results page.
+
+## Anti-patterns to avoid
+
+- **Don't average across benchmarks** ("average score 73%") — hides the
+ shape. SWE-Bench Pro 58% and AIME 95% are not commensurable.
+- **Don't quote vendor numbers without a harness caveat.** A 59% on
+ SWE-Bench Pro run with Claude Code scaffolding is not the same as 59%
+ with OpenHands.
+- **Don't pick a "winner" unless the user asks.** The whole point of the
+ comparison is that the right model depends on the workload: long
+ context vs. low cost vs. best tool use vs. strongest reasoning.
+- **Don't fabricate a score** to fill a cell. `n/a` is honest; "60%"
+ invented is not.
+- **Don't bury the cost.** Pricing is often the deciding factor for
+ agentic workloads and usually changes the ranking entirely.
+- **Don't ignore the "what the numbers hide" angle.** A Medium-style
+ "I tested both on 15 real tasks" piece usually finds that the
+ benchmark gap is much smaller than the practical gap (or vice versa).
+ Worth citing at least one such piece per comparison.
+
+## Output format
+
+- Plain markdown, ready to paste.
+- Tables first, prose after.
+- One section per analysis point, no walls of text.
+- Sources at the bottom as a bulleted URL list, not inline links (easier
+ to copy and verify).
+
+## Example output shape
+
+A minimal good response is two tables + four short sections. A maximal
+good response is two tables + five sections + sources. Anything longer
+than that is padding.
diff --git a/prompts/skills/llm-benchmark-comparison/references/search-and-sourcing.md b/prompts/skills/llm-benchmark-comparison/references/search-and-sourcing.md
new file mode 100644
index 0000000..8c05264
--- /dev/null
+++ b/prompts/skills/llm-benchmark-comparison/references/search-and-sourcing.md
@@ -0,0 +1,49 @@
+# Search & Sourcing
+
+How to find the numbers and which sources to trust, for steps 2–3 of the
+workflow.
+
+## Search the web for each model
+
+Prefer the **HTML DuckDuckGo endpoint** (`https://html.duckduckgo.com/html/?q=…`)
+because the JSON API and the JS site return empty in this environment. Brave
+Search (`https://search.brave.com/search?q=…`) is a useful fallback. Bing
+and Google both gate on CAPTCHA.
+
+Useful per-model queries:
+
+- `"<model>" benchmark scores`
+- `"<model>" SWE-Bench Pro` (the de-facto coding test in 2026)
+- `"<model>" pricing context window`
+- `"<model>" site:huggingface.co` (model card)
+- `"<model>" site:artificialanalysis.ai` (independent evals)
+- `"<model>" review vs` (head-to-head pieces)
+
+## Pull authoritative sources
+
+In this order of trust:
+
+1. **Vendor blog / launch post** — gives headline numbers, but watch for
+ cherry-picking and self-reported harnesses.
+2. **Hugging Face model card** — usually has the most complete benchmark
+ table; check the license, params, context.
+3. **Artificial Analysis article** — independent evals (Intelligence
+ Index, GDPval-AA Elo, AA-Omniscience hallucination rate).
+4. **OpenRouter / LLM-Stats pricing pages** — current $/M token rates.
+5. **Independent reviews** — Lushbinary, OfficeChai, Analytics India,
+ Geeky Gadgets; useful for "what the numbers hide" and real-world
+ anecdote.
+
+See [benchmarks-catalog.md](benchmarks-catalog.md) for the full list of
+fields to capture from these sources.
+
+## Source endpoints
+
+- `https://html.duckduckgo.com/html/?q=...` — primary search endpoint
+- `https://search.brave.com/search?q=...` — fallback search
+- `https://artificialanalysis.ai/articles/...` — independent evals
+- `https://huggingface.co/<org>/<model>` — model cards
+- `https://openrouter.ai/<provider>/<model>/benchmarks` — pricing +
+ benchmarks in one place
+- `https://llm-stats.com/home/models/<model>` — pricing + benchmark
+ snapshot