diff options
| author | Paul Buetow <paul@buetow.org> | 2026-06-19 22:03:26 +0300 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2026-06-19 22:03:26 +0300 |
| commit | 55a2b349727f527acc3e4294539e35a012c15784 (patch) | |
| tree | f11554176b99ebd69616b6754ee4222cb24732de /prompts | |
| parent | 9e9f16b4e7f7e125d891d6414b66d4d75ec940bc (diff) | |
llm-benchmark-comparison: split oversized SKILL.md into references/
Reorganize the skill to follow the f3s / blog-writing-style pattern: a slim
SKILL.md (purpose, trigger, inputs, core workflow, reference index) plus three
focused reference files. Pure reorganization — all content preserved verbatim.
- references/search-and-sourcing.md: search endpoints, per-model queries,
source trust order, source endpoint URLs (workflow steps 2-3)
- references/benchmarks-catalog.md: per-model fields-to-capture table and the
2026 benchmark glossary
- references/output-and-analysis.md: comparison table structure, analysis
sections, anti-patterns, output format, output shape (workflow steps 4-7)
SKILL.md drops from 225 to 86 lines.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diffstat (limited to 'prompts')
4 files changed, 240 insertions, 175 deletions
diff --git a/prompts/skills/llm-benchmark-comparison/SKILL.md b/prompts/skills/llm-benchmark-comparison/SKILL.md index 7b55f6c..c9491df 100644 --- a/prompts/skills/llm-benchmark-comparison/SKILL.md +++ b/prompts/skills/llm-benchmark-comparison/SKILL.md @@ -19,6 +19,14 @@ into a blog post, RFC, or procurement doc. - The user wants a model-selection report: benchmarks, pricing, context, license, modalities — in one table. +## Reference Files + +Detailed reference documentation is in the `references/` subfolder: + +- [Search & Sourcing](references/search-and-sourcing.md) — search endpoints (DuckDuckGo HTML, Brave), per-model queries, the source trust order (vendor blog → HF model card → Artificial Analysis → pricing pages → independent reviews), and the raw source endpoint URLs. Covers workflow steps 2–3. +- [Benchmarks Catalog](references/benchmarks-catalog.md) — the per-model fields-to-capture table (where to find each one) and the 2026 benchmark glossary (SWE-Bench Pro/Verified, Terminal-Bench, BrowseComp, HLE, τ²-Bench, MCP-Atlas, GPQA-Diamond, AIME, AA Intelligence Index, AA-Omniscience, GDPval-AA Elo). +- [Output & Analysis](references/output-and-analysis.md) — the copy-paste comparison table structure, the ordered analysis sections (quick reference, headline table, how-to-read, cost reality check, caveats, sources), the anti-patterns to avoid, the output-format rules, and the target output shape. Covers workflow steps 4–7. + ## Inputs Resolve these before searching; if the user gave you a list of models, use @@ -39,187 +47,40 @@ point release every 1–3 months; `Kimi K2.6` is different from `Kimi K2.5` or ### 2. Search the web for each model -Prefer the **HTML DuckDuckGo endpoint** (`https://html.duckduckgo.com/html/?q=…`) -because the JSON API and the JS site return empty in this environment. Brave -Search (`https://search.brave.com/search?q=…`) is a useful fallback. Bing -and Google both gate on CAPTCHA. - -Useful per-model queries: - -- `"<model>" benchmark scores` -- `"<model>" SWE-Bench Pro` (the de-facto coding test in 2026) -- `"<model>" pricing context window` -- `"<model>" site:huggingface.co` (model card) -- `"<model>" site:artificialanalysis.ai` (independent evals) -- `"<model>" review vs` (head-to-head pieces) +Use the search endpoints and per-model queries in +[references/search-and-sourcing.md](references/search-and-sourcing.md). +Prefer the HTML DuckDuckGo endpoint; Brave is the fallback. ### 3. Pull authoritative sources -In this order of trust: - -1. **Vendor blog / launch post** — gives headline numbers, but watch for - cherry-picking and self-reported harnesses. -2. **Hugging Face model card** — usually has the most complete benchmark - table; check the license, params, context. -3. **Artificial Analysis article** — independent evals (Intelligence - Index, GDPval-AA Elo, AA-Omniscience hallucination rate). -4. **OpenRouter / LLM-Stats pricing pages** — current $/M token rates. -5. **Independent reviews** — Lushbinary, OfficeChai, Analytics India, - Geeky Gadgets; useful for "what the numbers hide" and real-world - anecdote. - -For each model, capture: - -| Field | Where to find it | -|---|---| -| Lab + release date | Vendor blog or HF model card | -| Architecture (dense/MoE, total/active params, attention type) | HF model card or technical report | -| Context window | Vendor page, often in the model card | -| Modalities | HF model card, README | -| License | HF model card (top of page) | -| **SWE-Bench Pro** | Almost always in launch blog | -| **SWE-Bench Verified** | Vendor blog, leaderboards | -| **Terminal-Bench 2.0 / 2.1** | Vendor blog, "agentic" tables | -| **BrowseComp** | Agentic browsing tests; check vendor and AA | -| **HLE (Humanity's Last Exam)** | With and without tools, often reported both | -| **GPQA-Diamond** | Reasoning benchmark; check leaderboards | -| **AIME 2026** | Math reasoning; check leaderboards | -| **τ²-Bench / τ²-Bench Telecom** | Tool-use / agentic | -| **MCP-Atlas** | Tool-use over MCP | -| **AA Intelligence Index** | artificialanalysis.ai | -| **AA-Omniscience hallucination** | artificialanalysis.ai | -| **Input $/M, Output $/M** | OpenRouter, vendor pricing page, LLM-Stats | - -If a number is missing after two passes, mark it `n/a` — do not invent. +Follow the source trust order and capture the per-model fields. The trust +order and source endpoints are in +[references/search-and-sourcing.md](references/search-and-sourcing.md); the +fields-to-capture table and benchmark glossary are in +[references/benchmarks-catalog.md](references/benchmarks-catalog.md). If a +number is missing after two passes, mark it `n/a` — do not invent. ### 4. Build the comparison table -Use this structure (markdown, copy-paste-ready): - -```markdown -| | **Model A** | **Model B** | **Model C** | -|---|---|---|---| -| Lab | ... | ... | ... | -| Released | YYYY-MM-DD | ... | ... | -| Architecture | ... | ... | ... | -| Context | ... | ... | ... | -| Modalities | ... | ... | ... | -| License | ... | ... | ... | -| Input $/M (promo/std) | ... | ... | ... | -| Output $/M | ... | ... | ... | - -| Benchmark | Model A | Model B | Model C | Notes | -|---|---:|---:|---:|---| -| SWE-Bench Pro | ... | ... | ... | ... | -| SWE-Bench Verified | ... | ... | ... | ... | -| Terminal-Bench 2.x | ... | ... | ... | ... | -| BrowseComp | ... | ... | ... | ... | -| HLE (w/ tools) | ... | ... | ... | ... | -| GPQA-Diamond | ... | ... | ... | ... | -| AIME 2026 | ... | ... | ... | ... | -| τ²-Bench Telecom | ... | ... | ... | ... | -| MCP-Atlas | ... | ... | ... | ... | -| AA Intelligence Index | ... | ... | ... | ... | -``` - -Right-align numeric columns (`---:`) so the digits line up. Round benchmark -percentages to one decimal. Keep the Notes column short — one phrase, not -a sentence. +Use the copy-paste-ready table structure and formatting rules in +[references/output-and-analysis.md](references/output-and-analysis.md). ### 5. Write the analysis -After the table, include these sections in order: - -1. **Quick reference** — a one-row-per-model summary table covering the - "what is it" fields (lab, release, arch, context, modalities, license, - price). Optional but useful when the user has 3+ models. -2. **Headline benchmark table** — the long table above. -3. **How to read this** — 4–8 bullets, each one a claim grounded in the - numbers: - - "Coding is effectively a tie at 58–59% on SWE-Bench Pro" - - "K2.6 is the most battle-tested for long-horizon work (13-hour - exchange-core rewrite, 5-day autonomous ops agent)" - - "M3's pitch is cheap 1M context, GLM 5.1's is breadth, K2.6's is - tool-use + reasoning" -4. **Cost reality check** — a per-task worked example. Standard - workload: 500K input tokens + 100K output tokens. Compute - `(0.5 × input_$/M) + (0.1 × output_$/M)` per model. Reference Claude - Opus-class price as a baseline. -5. **Caveats** — required. Cover at least: - - Vendor-published numbers (most are) - - Harness / scaffold differences (Claude Code, Terminus, OpenHands, - Mini-SWE-Agent) — same model scores differently across harnesses - - Token usage differences (K2.6 uses ~2× K2.5 tokens; M3's 1M context - costs more tokens per turn) - - Open-weights claim status (M3 weights + technical report promised - ~10 days post-launch, not day-one in June 2026) - - Independent verification status -6. **Sources** — list the URLs you actually pulled from, not the - search-results page. - -### 6. Anti-patterns to avoid - -- **Don't average across benchmarks** ("average score 73%") — hides the - shape. SWE-Bench Pro 58% and AIME 95% are not commensurable. -- **Don't quote vendor numbers without a harness caveat.** A 59% on - SWE-Bench Pro run with Claude Code scaffolding is not the same as 59% - with OpenHands. -- **Don't pick a "winner" unless the user asks.** The whole point of the - comparison is that the right model depends on the workload: long - context vs. low cost vs. best tool use vs. strongest reasoning. -- **Don't fabricate a score** to fill a cell. `n/a` is honest; "60%" - invented is not. -- **Don't bury the cost.** Pricing is often the deciding factor for - agentic workloads and usually changes the ranking entirely. -- **Don't ignore the "what the numbers hide" angle.** A Medium-style - "I tested both on 15 real tasks" piece usually finds that the - benchmark gap is much smaller than the practical gap (or vice versa). - Worth citing at least one such piece per comparison. - -### 7. Output format - -- Plain markdown, ready to paste. -- Tables first, prose after. -- One section per analysis point, no walls of text. -- Sources at the bottom as a bulleted URL list, not inline links (easier - to copy and verify). - -## Quick benchmark glossary (2026) - -- **SWE-Bench Pro** — Real GitHub issues, harder subset of SWE-Bench. - Industry-standard coding test in 2026. Currently ~58% is the open-weight - SOTA bar. -- **SWE-Bench Verified** — Human-verified easier subset. Top closed - models hit 80%+. -- **Terminal-Bench 2.0 / 2.1** — Real command-line agent tasks. -- **BrowseComp** — Autonomous web browsing + information retrieval. -- **HLE (Humanity's Last Exam)** — Hardest reasoning benchmark; usually - reported with and without tools. -- **τ²-Bench / τ²-Bench Telecom** — Tool-use in agentic loop, telecom - domain. -- **MCP-Atlas** — Tool use over Model Context Protocol. -- **GPQA-Diamond** — Graduate-level science Q&A. -- **AIME** — Math competition problems. -- **AA Intelligence Index** — Composite index from Artificial Analysis - combining many of the above; 54–57 is the current frontier band. -- **AA-Omniscience** — Hallucination + abstention metric; lower - hallucination rate is better. -- **GDPval-AA Elo** — General agentic performance on knowledge-work - tasks. - -## Example output shape - -A minimal good response is two tables + four short sections. A maximal -good response is two tables + five sections + sources. Anything longer -than that is padding. - -## References - -- `https://html.duckduckgo.com/html/?q=...` — primary search endpoint -- `https://search.brave.com/search?q=...` — fallback search -- `https://artificialanalysis.ai/articles/...` — independent evals -- `https://huggingface.co/<org>/<model>` — model cards -- `https://openrouter.ai/<provider>/<model>/benchmarks` — pricing + - benchmarks in one place -- `https://llm-stats.com/home/models/<model>` — pricing + benchmark - snapshot +Include the ordered analysis sections (quick reference, headline table, +how-to-read, cost reality check, caveats, sources) from +[references/output-and-analysis.md](references/output-and-analysis.md). + +### 6. Avoid the anti-patterns + +Don't average scores, don't quote vendor numbers without a harness caveat, +don't pick a "winner" unless asked, don't fabricate scores, don't bury the +cost, don't ignore the "what the numbers hide" angle. Full list with +rationale in +[references/output-and-analysis.md](references/output-and-analysis.md). + +### 7. Match the output format + +Plain markdown, tables first, sources at the bottom. The output-format rules +and the target output shape (two tables + 4–5 short sections) are in +[references/output-and-analysis.md](references/output-and-analysis.md). diff --git a/prompts/skills/llm-benchmark-comparison/references/benchmarks-catalog.md b/prompts/skills/llm-benchmark-comparison/references/benchmarks-catalog.md new file mode 100644 index 0000000..8bc29a5 --- /dev/null +++ b/prompts/skills/llm-benchmark-comparison/references/benchmarks-catalog.md @@ -0,0 +1,52 @@ +# Benchmark Catalog & Per-Model Fields + +What to capture for each model, and what each benchmark actually measures. + +## Per-model fields to capture + +For each model, capture: + +| Field | Where to find it | +|---|---| +| Lab + release date | Vendor blog or HF model card | +| Architecture (dense/MoE, total/active params, attention type) | HF model card or technical report | +| Context window | Vendor page, often in the model card | +| Modalities | HF model card, README | +| License | HF model card (top of page) | +| **SWE-Bench Pro** | Almost always in launch blog | +| **SWE-Bench Verified** | Vendor blog, leaderboards | +| **Terminal-Bench 2.0 / 2.1** | Vendor blog, "agentic" tables | +| **BrowseComp** | Agentic browsing tests; check vendor and AA | +| **HLE (Humanity's Last Exam)** | With and without tools, often reported both | +| **GPQA-Diamond** | Reasoning benchmark; check leaderboards | +| **AIME 2026** | Math reasoning; check leaderboards | +| **τ²-Bench / τ²-Bench Telecom** | Tool-use / agentic | +| **MCP-Atlas** | Tool-use over MCP | +| **AA Intelligence Index** | artificialanalysis.ai | +| **AA-Omniscience hallucination** | artificialanalysis.ai | +| **Input $/M, Output $/M** | OpenRouter, vendor pricing page, LLM-Stats | + +If a number is missing after two passes, mark it `n/a` — do not invent. + +## Quick benchmark glossary (2026) + +- **SWE-Bench Pro** — Real GitHub issues, harder subset of SWE-Bench. + Industry-standard coding test in 2026. Currently ~58% is the open-weight + SOTA bar. +- **SWE-Bench Verified** — Human-verified easier subset. Top closed + models hit 80%+. +- **Terminal-Bench 2.0 / 2.1** — Real command-line agent tasks. +- **BrowseComp** — Autonomous web browsing + information retrieval. +- **HLE (Humanity's Last Exam)** — Hardest reasoning benchmark; usually + reported with and without tools. +- **τ²-Bench / τ²-Bench Telecom** — Tool-use in agentic loop, telecom + domain. +- **MCP-Atlas** — Tool use over Model Context Protocol. +- **GPQA-Diamond** — Graduate-level science Q&A. +- **AIME** — Math competition problems. +- **AA Intelligence Index** — Composite index from Artificial Analysis + combining many of the above; 54–57 is the current frontier band. +- **AA-Omniscience** — Hallucination + abstention metric; lower + hallucination rate is better. +- **GDPval-AA Elo** — General agentic performance on knowledge-work + tasks. diff --git a/prompts/skills/llm-benchmark-comparison/references/output-and-analysis.md b/prompts/skills/llm-benchmark-comparison/references/output-and-analysis.md new file mode 100644 index 0000000..ea76f5c --- /dev/null +++ b/prompts/skills/llm-benchmark-comparison/references/output-and-analysis.md @@ -0,0 +1,103 @@ +# Output & Analysis Format + +How to build the comparison table, write the analysis, and avoid the common +traps — steps 4–7 of the workflow plus anti-patterns and the target output +shape. + +## Build the comparison table + +Use this structure (markdown, copy-paste-ready): + +```markdown +| | **Model A** | **Model B** | **Model C** | +|---|---|---|---| +| Lab | ... | ... | ... | +| Released | YYYY-MM-DD | ... | ... | +| Architecture | ... | ... | ... | +| Context | ... | ... | ... | +| Modalities | ... | ... | ... | +| License | ... | ... | ... | +| Input $/M (promo/std) | ... | ... | ... | +| Output $/M | ... | ... | ... | + +| Benchmark | Model A | Model B | Model C | Notes | +|---|---:|---:|---:|---| +| SWE-Bench Pro | ... | ... | ... | ... | +| SWE-Bench Verified | ... | ... | ... | ... | +| Terminal-Bench 2.x | ... | ... | ... | ... | +| BrowseComp | ... | ... | ... | ... | +| HLE (w/ tools) | ... | ... | ... | ... | +| GPQA-Diamond | ... | ... | ... | ... | +| AIME 2026 | ... | ... | ... | ... | +| τ²-Bench Telecom | ... | ... | ... | ... | +| MCP-Atlas | ... | ... | ... | ... | +| AA Intelligence Index | ... | ... | ... | ... | +``` + +Right-align numeric columns (`---:`) so the digits line up. Round benchmark +percentages to one decimal. Keep the Notes column short — one phrase, not +a sentence. + +## Write the analysis + +After the table, include these sections in order: + +1. **Quick reference** — a one-row-per-model summary table covering the + "what is it" fields (lab, release, arch, context, modalities, license, + price). Optional but useful when the user has 3+ models. +2. **Headline benchmark table** — the long table above. +3. **How to read this** — 4–8 bullets, each one a claim grounded in the + numbers: + - "Coding is effectively a tie at 58–59% on SWE-Bench Pro" + - "K2.6 is the most battle-tested for long-horizon work (13-hour + exchange-core rewrite, 5-day autonomous ops agent)" + - "M3's pitch is cheap 1M context, GLM 5.1's is breadth, K2.6's is + tool-use + reasoning" +4. **Cost reality check** — a per-task worked example. Standard + workload: 500K input tokens + 100K output tokens. Compute + `(0.5 × input_$/M) + (0.1 × output_$/M)` per model. Reference Claude + Opus-class price as a baseline. +5. **Caveats** — required. Cover at least: + - Vendor-published numbers (most are) + - Harness / scaffold differences (Claude Code, Terminus, OpenHands, + Mini-SWE-Agent) — same model scores differently across harnesses + - Token usage differences (K2.6 uses ~2× K2.5 tokens; M3's 1M context + costs more tokens per turn) + - Open-weights claim status (M3 weights + technical report promised + ~10 days post-launch, not day-one in June 2026) + - Independent verification status +6. **Sources** — list the URLs you actually pulled from, not the + search-results page. + +## Anti-patterns to avoid + +- **Don't average across benchmarks** ("average score 73%") — hides the + shape. SWE-Bench Pro 58% and AIME 95% are not commensurable. +- **Don't quote vendor numbers without a harness caveat.** A 59% on + SWE-Bench Pro run with Claude Code scaffolding is not the same as 59% + with OpenHands. +- **Don't pick a "winner" unless the user asks.** The whole point of the + comparison is that the right model depends on the workload: long + context vs. low cost vs. best tool use vs. strongest reasoning. +- **Don't fabricate a score** to fill a cell. `n/a` is honest; "60%" + invented is not. +- **Don't bury the cost.** Pricing is often the deciding factor for + agentic workloads and usually changes the ranking entirely. +- **Don't ignore the "what the numbers hide" angle.** A Medium-style + "I tested both on 15 real tasks" piece usually finds that the + benchmark gap is much smaller than the practical gap (or vice versa). + Worth citing at least one such piece per comparison. + +## Output format + +- Plain markdown, ready to paste. +- Tables first, prose after. +- One section per analysis point, no walls of text. +- Sources at the bottom as a bulleted URL list, not inline links (easier + to copy and verify). + +## Example output shape + +A minimal good response is two tables + four short sections. A maximal +good response is two tables + five sections + sources. Anything longer +than that is padding. diff --git a/prompts/skills/llm-benchmark-comparison/references/search-and-sourcing.md b/prompts/skills/llm-benchmark-comparison/references/search-and-sourcing.md new file mode 100644 index 0000000..8c05264 --- /dev/null +++ b/prompts/skills/llm-benchmark-comparison/references/search-and-sourcing.md @@ -0,0 +1,49 @@ +# Search & Sourcing + +How to find the numbers and which sources to trust, for steps 2–3 of the +workflow. + +## Search the web for each model + +Prefer the **HTML DuckDuckGo endpoint** (`https://html.duckduckgo.com/html/?q=…`) +because the JSON API and the JS site return empty in this environment. Brave +Search (`https://search.brave.com/search?q=…`) is a useful fallback. Bing +and Google both gate on CAPTCHA. + +Useful per-model queries: + +- `"<model>" benchmark scores` +- `"<model>" SWE-Bench Pro` (the de-facto coding test in 2026) +- `"<model>" pricing context window` +- `"<model>" site:huggingface.co` (model card) +- `"<model>" site:artificialanalysis.ai` (independent evals) +- `"<model>" review vs` (head-to-head pieces) + +## Pull authoritative sources + +In this order of trust: + +1. **Vendor blog / launch post** — gives headline numbers, but watch for + cherry-picking and self-reported harnesses. +2. **Hugging Face model card** — usually has the most complete benchmark + table; check the license, params, context. +3. **Artificial Analysis article** — independent evals (Intelligence + Index, GDPval-AA Elo, AA-Omniscience hallucination rate). +4. **OpenRouter / LLM-Stats pricing pages** — current $/M token rates. +5. **Independent reviews** — Lushbinary, OfficeChai, Analytics India, + Geeky Gadgets; useful for "what the numbers hide" and real-world + anecdote. + +See [benchmarks-catalog.md](benchmarks-catalog.md) for the full list of +fields to capture from these sources. + +## Source endpoints + +- `https://html.duckduckgo.com/html/?q=...` — primary search endpoint +- `https://search.brave.com/search?q=...` — fallback search +- `https://artificialanalysis.ai/articles/...` — independent evals +- `https://huggingface.co/<org>/<model>` — model cards +- `https://openrouter.ai/<provider>/<model>/benchmarks` — pricing + + benchmarks in one place +- `https://llm-stats.com/home/models/<model>` — pricing + benchmark + snapshot |
