From 567cdef73eb4385d937ddd74a988e489890b3cc1 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Mon, 18 May 2026 18:57:39 +0300 Subject: Add e2e-llm harness runner and README Co-Authored-By: Claude Sonnet 4.6 --- player-server/test/e2e-llm/README.md | 313 +++++++++++++++++++++++++++++++++++ 1 file changed, 313 insertions(+) create mode 100644 player-server/test/e2e-llm/README.md (limited to 'player-server/test/e2e-llm/README.md') diff --git a/player-server/test/e2e-llm/README.md b/player-server/test/e2e-llm/README.md new file mode 100644 index 0000000..872efa9 --- /dev/null +++ b/player-server/test/e2e-llm/README.md @@ -0,0 +1,313 @@ +# Player — LLM-Driven End-to-End Tests + +This directory contains the LLM-driven end-to-end test harness for Player. +Unlike the deterministic Playwright suite at `../e2e-web/`, this layer uses a +Claude agent to interpret natural-language scenario steps, execute them against +a live server, and judge the results. It is designed for cross-stack flows that +are too complex to express as sequences of CSS selectors. + +For full background and design rationale, read +[`docs/llm-e2e-design.md`](../../../../docs/llm-e2e-design.md) at the project +root. + +--- + +## Directory layout + +``` +test/e2e-llm/ +├── README.md # this file +├── runner/ +│ ├── package.json # Node/TypeScript runner dependencies +│ ├── tsconfig.json # TypeScript configuration +│ └── index.ts # harness entry point +└── scenarios/ + ├── S01-bootstrap.md # Bootstrap → first user + ├── S02-podcast.md # Podcast subscribe → download → mark complete + ├── S03-upload.md # Upload via API → verify on web + └── S04-share-link.md # Share-link round-trip +``` + +--- + +## Prerequisites + +| Requirement | Version | Notes | +|---|---|---| +| Node.js | 18+ | Required for the runner | +| npm | 8+ | Required for the runner | +| Playwright (Chromium) | Latest via Playwright | Install via `../e2e-web/` setup | +| `ask` CLI | Any | Installed globally; used for failure tasks | +| `sqlite3` CLI | Any | For DB assertion checks | +| `ANTHROPIC_API_KEY` | Valid key | Required for screenshot oracle (see below) | + +The Player server must be running before any scenario is started. The harness +does not start or stop the server. + +### First-time runner setup + +```sh +# From this directory: +cd runner +npm install +npm run build +``` + +--- + +## Starting the server + +The harness defaults to `http://localhost:8080`. Override with `PLAYER_URL`. + +Start the server from `player-server/` so it can resolve embedded static assets: + +```sh +# From player-server/ — use testmedia/ as the media library: +MEDIA_ROOT=./testmedia \ +SECURE_COOKIES=false \ +DB_PATH=/tmp/player-e2e-llm.db \ +./player +``` + +Key server environment variables: + +| Variable | Value | Reason | +|---|---|---| +| `MEDIA_ROOT` | `./testmedia` | Provides pre-existing media for upload/verify scenarios | +| `SECURE_COOKIES` | `false` | Allows session cookie over plain HTTP | +| `DB_PATH` | `/tmp/player-e2e-llm.db` | Isolates the test database from production | + +--- + +## Environment variables (harness) + +| Variable | Required | Default | Description | +|---|---|---|---| +| `PLAYER_URL` | No | `http://localhost:8080` | Base URL of the Player server under test | +| `ANTHROPIC_API_KEY` | Only for screenshot oracle | — | API key for Claude Haiku visual checks | +| `LLM_E2E_SCREENSHOTS` | No | `false` | Set to `true` to enable Layer 5 screenshot + Haiku visual assertion | +| `LLM_E2E_OPEN_ISSUE` | No | `false` | Set to `true` to open a Codeberg issue if a failure task is older than 24 h | +| `PLAYER_DB` | No | Inferred from server config | Path to the SQLite database file; used for DB assertion checks | + +--- + +## Running all scenarios + +```sh +# From runner/ (after npm run build): +node dist/index.js + +# Or run directly during development: +npm run dev +``` + +This runs S01 through S04 in order. Each scenario is independent; a failure in +one scenario does not stop the remaining scenarios from running. + +To run against a non-default server: + +```sh +PLAYER_URL=http://myserver:9090 node dist/index.js +``` + +To enable screenshot assertions (Layer 5, costs extra API tokens): + +```sh +LLM_E2E_SCREENSHOTS=true ANTHROPIC_API_KEY=sk-ant-... node dist/index.js +``` + +--- + +## Running a single scenario manually + +Pass the scenario file path as the first argument: + +```sh +# From runner/: +node dist/index.js ../scenarios/S01-bootstrap.md +``` + +Or via `npm run dev` during development: + +```sh +npm run dev -- ../scenarios/S01-bootstrap.md +``` + +The runner prints a per-step summary to stdout and exits with code 0 on +success, 1 on failure. + +--- + +## Adding a new scenario + +Each scenario is a single Markdown file under `scenarios/`. The file has two +parts: a YAML front-matter block and a numbered Markdown step list. + +### File name convention + +``` +S-.md +``` + +Examples: `S05-android-skeleton.md`, `S06-bulk-sync.md`. + +### Skeleton + +```markdown +--- +id: S05 +title: "Short human-readable title" +tags: [auth, web] # free-form tags for filtering +preconditions: + server_state: running # "fresh" = empty DB, "running" = existing DB with data + fixtures: [] # list of fixture keys the harness should load +assertions: + - db: "SELECT count(*) FROM users WHERE role='admin'" + - url_contains: /login.html +skip: false # set to "until-" to exclude from CI runs +--- + +1. Navigate to the root URL. +2. Confirm the page redirects to /expected-page.html. +3. Do something meaningful with the UI or API. +4. Confirm the result. +``` + +### Front-matter fields + +| Field | Type | Description | +|---|---|---| +| `id` | string | Unique identifier, e.g. `S05` | +| `title` | string | Human-readable title used in logs and `ask` task names | +| `tags` | list | Free-form tags; not currently used for filtering but useful for documentation | +| `preconditions.server_state` | `fresh` or `running` | `fresh` means the harness creates a temp empty DB; `running` means it uses the existing one | +| `preconditions.fixtures` | list | Named fixture keys the harness loads before running steps | +| `assertions` | list | Post-condition checks the harness runs after all steps complete | +| `skip` | bool or string | `false` to run normally; a string like `"until-integration-test-added"` to skip in CI | + +### Assertion types + +| Key | Example | What it checks | +|---|---|---| +| `db` | `db: "SELECT id FROM users WHERE role='admin'"` | Runs a read-only SQLite query; passes if at least one row is returned | +| `url_contains` | `url_contains: /login.html` | Checks the browser's current URL | +| `selector_visible` | `selector_visible: ".media-card"` | Playwright DOM check | +| `status_code` | `status_code: "GET /api/v1/health 200"` | HTTP status assertion | + +### Writing good steps + +- Write steps as numbered plain-English sentences. The agent interprets them; no + special syntax is needed inside the Markdown body. +- Each step should do one thing: navigate, fill a form, call an API, or assert. +- Avoid writing assertions in the steps if you can express them in the YAML + `assertions` block — the YAML block is cheaper (no agent reasoning required). +- Visual checks ("does this look right?") belong in steps, not in the YAML block; + they trigger the Haiku screenshot oracle when `LLM_E2E_SCREENSHOTS=true`. + +--- + +## Assertion layers and cost + +The harness uses layered assertions from cheapest to most expensive. Use the +cheapest layer that can detect a failure: + +| Layer | Method | Cost | +|---|---|---| +| 1 | HTTP status codes | Zero — checked on every API call | +| 2 | JSON field assertions | Cheap — parse response body | +| 3 | DB state checks (`sqlite3` CLI) | Cheap — direct SQL query | +| 4 | Playwright selector checks | Cheap — deterministic DOM query | +| 5 | Screenshot + Haiku visual check | Expensive — only for S03 and S04; gated behind `LLM_E2E_SCREENSHOTS=true` | + +Layer 5 sends a PNG to Claude Haiku with a yes/no question such as "Is there a +media card visible in the grid?" Haiku is used (not Sonnet) to keep per-run +cost low. + +--- + +## Cost budget + +| Run type | Scenarios | Estimated cost | +|---|---|---| +| Full nightly (S01-S04) | 4 | ~$0.18 per run | +| Single scenario | 1 | ~$0.04-0.05 per run | +| Full nightly with retries | 4 + retries | ~$0.27 per run | + +At one nightly run the expected spend is **~$5.50/month** (Sonnet for +orchestration, Haiku for screenshots). + +**Alert threshold:** If a single run exceeds **$0.50**, treat it as anomalous. +Common causes are a context window blowup (Playwright output not capped) or +excessive retries. The runner caps Playwright CLI output forwarded to the agent +at 4 000 characters to help control this. + +**Monthly soft budget:** $10/month for the entire LLM e2e layer. If you +consistently see runs above $0.50, check for long scenario files, verbose +fixtures, or scenarios that retry repeatedly. + +--- + +## Failure handling + +The runner uses a one-retry policy: + +``` +Run scenario +├── Pass → log "PASS", continue to next scenario +└── Fail + ├── Wait 5 s, retry once (fresh browser context) + │ ├── Pass → log "FLAKY — passed on retry", continue + │ └── Fail → open ask task, log "FAIL", continue to next scenario +``` + +When a scenario fails after the retry, the runner opens a task: + +```sh +ask add "LLM e2e failure: — " +``` + +The task description includes the Playwright HTML report path and a snippet of +the failure output so a developer can reproduce the failure without re-reading +logs. + +The runner does **not**: +- retry more than once (cost control) +- stop the run on a single failure (other scenarios still run) +- send email or Slack notifications +- auto-fix failing scenarios + +To open a Codeberg issue automatically when a failure task is older than 24 h, +set `LLM_E2E_OPEN_ISSUE=true`. This is off by default. + +--- + +## Nightly schedule + +The nightly run is configured via the `/schedule` skill. It runs a Claude Code +agent with the prompt: + +> Run the LLM e2e suite at `player-server/test/e2e-llm/` against the +> production server and report results. + +The agent reads the scenario files, executes them via the runner, and annotates +an ongoing task with the result. To set up or modify the schedule, use: + +```sh +/schedule +``` + +--- + +## Relationship to the Playwright smoke suite + +| Dimension | Playwright smoke (`../e2e-web/`) | LLM e2e (this suite) | +|---|---|---| +| Assertion style | Deterministic CSS selectors | Natural-language steps + layered assertions | +| When to run | Every PR, fast | Nightly, slower | +| Best for | "Did the page render? Did the API return 200?" | Cross-stack flows, auth boundaries, multi-step sequences | +| Cost | Zero (no LLM calls) | ~$0.18/run | +| Visual checks | No | Yes, via Haiku (opt-in) | + +The two suites are complementary. The Playwright smoke suite is the first line +of defence for regressions. The LLM e2e suite catches higher-level integration +failures that selector-based tests cannot easily express. -- cgit v1.2.3