path: root/docs/llm-e2e-design.md

# LLM-Driven Cross-Stack E2E Test Design Proposal

**Status:** PROPOSAL — awaiting user approval before any implementation begins.
**Author:** Generated by Claude Code, task e6.
**Date:** 2026-05-18

---

## Overview

This document answers the seven design questions for the LLM-driven end-to-end
test layer and lists the follow-up implementation tasks to create after
approval. It covers the web stack only for now; the Android emulator path is
explicitly deferred (see Question 3).

The existing Playwright smoke suite at `player-server/test/e2e-web/` is a
deterministic, selector-driven harness. That suite is good at "did the page
render and did the API return 200?". The LLM-driven layer is for scenarios that
cross authentication boundaries, combine multiple API surfaces, or are too
cumbersome to express as a sequence of CSS selectors — the cross-stack flows
listed at the end of this document.

---

## 1. Scenario Format

**Recommendation: YAML front-matter + Markdown steps.**

Each scenario lives in a single `.md` file under `player-server/test/e2e-llm/scenarios/`.
The file begins with a YAML block that the harness agent parses to set up
pre-conditions and pick the right fixtures. The body is a numbered Markdown
list of steps written in plain English so a human can read, edit, and review
the scenario without touching agent code.

Example skeleton:

```yaml
---
id: S01
title: "Bootstrap → first-user"
tags: [auth, web]
preconditions:
  server_state: fresh          # server must have empty database
  fixtures: []
assertions:
  - db: "SELECT count(*) FROM users WHERE role='admin' = 1"
  - url_contains: /login.html
---
```

```markdown
1. Navigate to the root URL.
2. Confirm the page redirects to /bootstrap.html.
3. Fill in username "admin", password "hunter2", confirm "hunter2".
4. Submit the form.
5. Confirm the browser lands on /login.html.
6. Confirm the database contains exactly one user with role=admin.
```

**Trade-offs:**

| Option | Pro | Con |
|---|---|---|
| YAML-only | Machine-readable, diffable, strict schema | Hard to read; steps turn into nested maps |
| Prose-only | Easiest to write | Agent must interpret free text; brittle |
| **YAML front-matter + MD steps** | Humans read the steps; machine reads the pre/post conditions | Two syntaxes in one file; agent must parse both |

The mixed format wins because pre-conditions and post-conditions (what to
assert, which fixtures to load) need structure, while steps benefit from
natural-language flexibility. The agent receives the full file and interprets
both sections.

---

## 2. Browser Access

**Recommendation: Playwright CLI via Bash.**

Run `npx playwright test --config ...` (or a dedicated scenario runner script)
from inside a Claude Code Bash tool call. The agent drives scenarios by
calling the existing `player-server/test/e2e-web/` Playwright CLI for
structured assertions, and supplements with direct API calls (via `curl` or a
small helper script) where the UI is not involved.

**Why not Playwright MCP?**

The Anthropic Playwright MCP server provides rich, interactive browser access:
the agent can call `playwright_navigate`, `playwright_click`,
`playwright_screenshot`, and `playwright_get_visible_text` as structured tool
calls. That is excellent for exploratory or ambiguous scenarios. The trade-offs:

| Factor | Playwright MCP | Playwright CLI via Bash |
|---|---|---|
| Agent control | Fine-grained, action-by-action | Coarse — run a test file, read output |
| Step visibility | Full; agent sees each action result | Agent sees stdout/exit code only |
| Existing suite reuse | Cannot reuse .ts test files directly | Reuses the existing suite and helpers |
| Cost per scenario | Higher (many tool calls) | Lower (one Bash call per scenario file) |
| Failure detail | Agent sees DOM/screenshot inline | Agent reads HTML report or screenshot on failure |
| Setup complexity | Requires MCP server running alongside Claude | No additional server; Playwright already installed |

For LLM-driven e2e the recommended approach is a **hybrid**:

1. The harness agent invokes the Playwright CLI to run a scenario's structured
   assertions (fast, cheap, reuses existing infrastructure).
2. When a scenario step requires judgment ("does the media grid look right?"),
   the agent takes a screenshot via `playwright screenshot` and sends it to a
   secondary Claude call for visual inspection.
3. The Playwright MCP is kept as an **opt-in** for interactive debugging
   sessions (`/loop` invocations where a developer is watching), not for CI
   runs where cost per run matters.

**Concrete setup:** A thin Node/TypeScript runner at
`player-server/test/e2e-llm/runner/` will read scenario files, call Playwright
CLI with `--reporter=json`, parse results, and report pass/fail with the
per-step timing. The agent calls this runner via Bash; no additional server is
required in CI.

---

## 3. Android Emulator Access

**Decision: Deferred — unit tests only per task c6 (Android testing strategy).**

The `player-android/` scaffold contains no real screens yet. Running an Android
emulator on CI is expensive in both time (cold boot 3-5 min) and money, and
there is nothing useful to assert beyond "the scaffold compiles and the stub
home screen renders a text string".

Current boundary:

- `flutter test` (unit tests, no emulator) — done or in progress as part of
  task c6.
- Emulator-based integration tests — deferred until `player-android/lib/`
  contains real screens backed by actual API calls.

When emulator testing is unblocked, the design questions are:

- **Driver:** Flutter integration_test package + `flutter drive` (native,
  no extra toolchain). Playwright MCP cannot drive an Android emulator.
- **Harness access:** The agent calls `flutter drive --target=...` via Bash;
  same pattern as the Playwright CLI approach.
- **ADB fallback:** Direct ADB commands (`adb shell am instrument`) are
  available for lower-level checks (e.g., reading SharedPreferences after a
  sync), but should be last resort.

The optional Android skeleton scenario (S05) in this proposal is marked
`skip: until-integration-test-added` and will not run in CI until unblocked.

---

## 4. Oracle / Assertions

**Recommendation: Layered assertions — DB checks first, screenshot LLM last.**

Assertions are expensive in proportion to how much agent reasoning they
require. The cheapest oracle is a SQL query; the most expensive is an LLM
visual comparison. Use the cheapest oracle that can detect the failure:

```
Layer 1 — HTTP status codes         (zero cost, in every API call)
Layer 2 — JSON field assertions     (cheap; parse response body)
Layer 3 — DB state checks           (cheap; sqlite3 CLI in Bash)
Layer 4 — Playwright selector check (cheap; deterministic DOM query)
Layer 5 — Screenshot LLM comparison (expensive; use sparingly)
```

**Layer 5 details:** When visual validation is needed, the agent calls
`playwright screenshot` to capture a PNG, then submits it to a Claude
claude-haiku-4-x call (cheapest vision model) with a terse prompt: "Does this
screenshot show a media grid with at least one card? Answer yes/no and give a
one-sentence reason." Haiku is used (not Sonnet) specifically to keep per-run
cost manageable. Screenshot checks are applied only to the two scenarios where
visual confirmation is meaningful: `upload-verify-web` (S03) and
`share-link-round-trip` (S04).

**DB check implementation:** The player server uses SQLite at a configurable
path (`PLAYER_DB` env var). The harness agent runs `sqlite3 "$PLAYER_DB"
"<query>"` via Bash. The server must be stopped or the query must be
read-only (SQLite supports concurrent readers) before the check.

---

## 5. Trigger — When to Run

**Recommendation: Nightly via /schedule, not on every PR.**

Reasons not to run on every PR:

1. Cost — each run consumes Claude API tokens. At current pricing a full
   5-scenario run costs roughly $0.05-0.20 (see Question 6). Multiplied by
   dozens of PRs per week this adds up and creates friction against small
   commits.
2. Speed — LLM calls add latency. A PR gate should finish in under 5 minutes;
   an LLM e2e run will likely take 2-8 minutes depending on scenario count and
   whether visual assertions are needed.
3. Flakiness budget — LLM responses have inherent non-determinism. A PR gate
   that occasionally fails due to LLM variance is annoying.

**Recommended trigger matrix:**

| Trigger | How | Scenarios run |
|---|---|---|
| Nightly (automated) | `/schedule` cron | All 5 scenarios (S01-S04 + S05 when unblocked) |
| Manual (developer) | `/loop` invocation | One named scenario, developer watches |
| PR gate (optional, opt-in) | CI job, manual dispatch only | S01 bootstrap only (fastest, most critical) |

**Implementation:** The `/schedule` skill can create a routine that runs a
Claude Code agent with a fixed prompt: "Run the LLM e2e suite at
`player-server/test/e2e-llm/` against the production server and report results."
The agent reads scenario files, executes them, and annotates or opens a task on
failure (see Question 7).

---

## 6. Cost Envelope

**Estimates based on Claude Sonnet 4.x pricing ($3/M input, $15/M output)
and Claude Haiku 4.x pricing ($0.25/M input, $1.25/M output) as of 2026-05.**

Per-scenario token estimates:

| Scenario | Steps | Approx input tokens | Approx output tokens | Haiku screenshot? | Cost estimate |
|---|---|---|---|---|---|
| S01 bootstrap → first-user | 6 | 8 000 | 800 | No | ~$0.036 |
| S02 podcast subscribe → complete | 10 | 12 000 | 1 200 | No | ~$0.054 |
| S03 upload via API → verify web | 8 | 10 000 | 1 000 | Yes (+$0.003) | ~$0.048 |
| S04 share-link round-trip | 7 | 9 000 | 900 | Yes (+$0.003) | ~$0.042 |
| S05 Android skeleton (deferred) | 4 | 6 000 | 600 | No | ~$0.027 |

**Full nightly run (S01-S04, Sonnet for orchestration + Haiku for screenshots):**
~$0.18 per run. At one run per night that is ~$5.50/month.

**Sensitivity:**
- If the orchestrating agent is verbose (long context with full scenario file +
  Playwright output in context), input tokens could be 2-3x higher. Cap the
  Playwright CLI output forwarded to the agent at 4 000 characters to control
  this.
- Retries (see Question 7) add ~50% cost on a failing run.
- Visual screenshot checks (Layer 5) are routed to Haiku; cost is negligible
  (~$0.003 each).

**Budget recommendation:** Set a soft budget of $10/month for LLM e2e. If a
nightly run exceeds $0.50, flag it as anomalous (context window blowup or
excessive retries).

---

## 7. Failure Handling

**Recommendation: One retry, then open an `ask` task.**

```
Run scenario
├── Pass  → annotate task, continue
└── Fail
    ├── Retry once (same scenario, fresh browser context)
    │   ├── Pass  → annotate task with "flaky: passed on retry"
    │   └── Fail  → open ask task + include Playwright HTML report path
```

**Step-by-step:**

1. **Retry once** — most transient failures (timing, server not fully ready)
   resolve on a single retry with a 5-second wait. Do not retry more than once
   to avoid burning cost on a broken build.
2. **Open an `ask` task** — if the retry also fails, the agent runs:
   ```
   ask add "LLM e2e failure: <scenario title> — <one-line reason>"
   ```
   and annotates the task with the Playwright HTML report path and a snippet of
   the failure output. This creates a visible, trackable item without noisy
   notifications.
3. **File a Codeberg issue (opt-in)** — for nightly runs only, if the task has
   been open for more than 24 hours without human annotation, the agent can
   create a Codeberg issue via the `gh` CLI (configured to point at
   codeberg.org/snonux/player). This is opt-in and controlled by an environment
   variable `LLM_E2E_OPEN_ISSUE=true`.

**What the agent does NOT do:**

- Does not retry indefinitely — cost and noise.
- Does not send email or Slack notifications — out of scope for self-hosted setup.
- Does not auto-fix failing tests — the proposal layer only; implementation
  tasks are opened for humans to action.

---

## Initial Scenario Set

The five scenarios below are the minimum viable set. Each will be a separate
Markdown file under `player-server/test/e2e-llm/scenarios/`.

### S01 — Bootstrap → First User

**Goal:** Verify a brand-new server transitions correctly from empty database to
first admin account.

**Precondition:** Fresh SQLite database (temp file created by harness).

**Steps (abbreviated):**
1. Start server against temp DB.
2. Navigate to root → confirm redirect to `/bootstrap.html`.
3. Submit bootstrap form with valid credentials.
4. Confirm redirect to `/login.html`.
5. Log in.
6. Confirm home page loads and admin panel is accessible.

**Assertions:** DB has exactly one user; role is admin; session cookie is set.

---

### S02 — Podcast Subscribe → Download → Mark Complete

**Goal:** Verify the full podcast lifecycle: subscribe, episode appears, episode
is playable and progress is saved.

**Precondition:** Server running, admin account exists, a test podcast RSS URL
is configured (can use a local mock RSS served by the harness).

**Steps (abbreviated):**
1. POST to `/api/v1/podcast/feeds` with test RSS URL.
2. Trigger feed refresh.
3. Confirm at least one episode appears in `/api/v1/podcast/episodes`.
4. POST progress for the episode at 95% of its duration.
5. Confirm episode appears in the "in progress" list.

**Assertions:** Feed row in DB; episode rows in DB; progress row at >=60s
position.

---

### S03 — Upload via API → Verify on Web

**Goal:** Verify that a file uploaded via the admin API appears in the web UI
media grid.

**Precondition:** Server running, admin account exists, testdata/media fixture file
available.

**Steps (abbreviated):**
1. POST a small test audio file to the upload endpoint using the admin Bearer
   token.
2. Trigger a media rescan.
3. Open the web UI and navigate to the set containing the uploaded file.
4. Confirm the media card is visible in the grid.
5. Screenshot + Haiku visual check: "Is there a media card visible in the grid?"

**Assertions:** File exists in MEDIA_ROOT; DB has a media row; Playwright
selector finds the card; Haiku screenshot check passes.

---

### S04 — Share-Link Round-Trip

**Goal:** Verify that a share link grants unauthenticated access to exactly the
shared media.

**Precondition:** Server running, admin account exists, at least one media item
in DB.

**Steps (abbreviated):**
1. POST to `/api/v1/share` to generate a share token for a specific media item.
2. Open the share URL in a fresh, unauthenticated browser context.
3. Confirm the media player page loads without a login redirect.
4. Confirm the play button is present.
5. Screenshot + Haiku visual check: "Is there a media player visible?"
6. Confirm that navigating to the root URL from the same unauthenticated context
   redirects to `/login.html` (share does not grant full access).

**Assertions:** Share row in DB with correct media_id; unauthenticated context
can load share URL; unauthenticated context cannot access `/`.

---

### S05 — Android Skeleton Boots (Deferred)

**Goal:** Verify the Flutter skeleton builds and the stub home screen renders.

**Status:** `skip: until-integration-test-added` — will not run until
`player-android/integration_test/` is populated.

**Steps (abbreviated, for reference):**
1. `flutter build apk --debug`
2. `flutter drive --target=integration_test/app_test.dart` (emulator must be
   running).
3. Confirm the app displays "Player Android — connect to <baseUrl>".

**Assertions:** Build exits 0; integration test exits 0; Haiku screenshot check
on the rendered home screen.

---

## Follow-Up Implementation Tasks

After the user approves this proposal, the following tasks should be created via
`ask add`:

1. **e2e-llm: create harness runner** — implement
   `player-server/test/e2e-llm/runner/` (Node/TypeScript): scenario file parser,
   Playwright CLI invocation, JSON output parser, retry logic, `ask add` on
   failure. Target: ≤200 LOC, single file if possible.

2. **e2e-llm: write scenario files S01-S04** — create the four YAML+Markdown
   scenario files under `player-server/test/e2e-llm/scenarios/`. Include the
   mock RSS server fixture needed for S02.

3. **e2e-llm: integrate Haiku screenshot oracle** — add the optional Layer 5
   visual check. One small helper that takes a PNG path and a yes/no question,
   calls the Anthropic API with Haiku, returns bool. Gate it behind
   `LLM_E2E_SCREENSHOTS=true` env var so it is off by default in CI.

4. **e2e-llm: /schedule routine** — configure a nightly `/schedule` routine that
   runs the harness against the production server (or a staging server), captures
   exit code and token usage, and annotates an ongoing task with the result.

5. **e2e-llm: documentation** — add `player-server/test/e2e-llm/README.md`
   covering prerequisites, how to run a single scenario manually, how to add a
   new scenario, and the cost budget guidance from this proposal.

---

## Open Questions for User Decision

Before implementation begins, please confirm or override:

- [ ] **Format approved?** YAML front-matter + Markdown steps, or prefer pure YAML?
- [ ] **Browser access approved?** Playwright CLI (Bash) as primary, MCP as opt-in for interactive debugging?
- [ ] **Screenshot oracle approved?** Haiku for visual checks, gated behind env var?
- [ ] **Trigger approved?** Nightly via /schedule, optional PR gate for S01 only?
- [ ] **Budget approved?** ~$5.50/month for nightly runs; alert if >$0.50/run?
- [ ] **Failure handling approved?** One retry, then `ask add` task; Codeberg issue opt-in via env var?
- [ ] **Android deferred?** S05 marked skip until integration_test/ is populated?

---

*End of proposal. No code has been changed. This document is the sole deliverable of task e6.*