Add e2e-llm harness runner and README

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

7 files changed, 790 insertions, 0 deletions

diff --git a/player-server/test/e2e-llm/README.md b/player-server/test/e2e-llm/README.md
new file mode 100644
index 0000000..872efa9
--- /dev/null
+++ b/player-server/test/e2e-llm/README.md
@@ -0,0 +1,313 @@
+# Player — LLM-Driven End-to-End Tests
+
+This directory contains the LLM-driven end-to-end test harness for Player.
+Unlike the deterministic Playwright suite at `../e2e-web/`, this layer uses a
+Claude agent to interpret natural-language scenario steps, execute them against
+a live server, and judge the results. It is designed for cross-stack flows that
+are too complex to express as sequences of CSS selectors.
+
+For full background and design rationale, read
+[`docs/llm-e2e-design.md`](../../../../docs/llm-e2e-design.md) at the project
+root.
+
+---
+
+## Directory layout
+
+```
+test/e2e-llm/
+├── README.md              # this file
+├── runner/
+│   ├── package.json       # Node/TypeScript runner dependencies
+│   ├── tsconfig.json      # TypeScript configuration
+│   └── index.ts           # harness entry point
+└── scenarios/
+    ├── S01-bootstrap.md   # Bootstrap → first user
+    ├── S02-podcast.md     # Podcast subscribe → download → mark complete
+    ├── S03-upload.md      # Upload via API → verify on web
+    └── S04-share-link.md  # Share-link round-trip
+```
+
+---
+
+## Prerequisites
+
+| Requirement | Version | Notes |
+|---|---|---|
+| Node.js | 18+ | Required for the runner |
+| npm | 8+ | Required for the runner |
+| Playwright (Chromium) | Latest via Playwright | Install via `../e2e-web/` setup |
+| `ask` CLI | Any | Installed globally; used for failure tasks |
+| `sqlite3` CLI | Any | For DB assertion checks |
+| `ANTHROPIC_API_KEY` | Valid key | Required for screenshot oracle (see below) |
+
+The Player server must be running before any scenario is started. The harness
+does not start or stop the server.
+
+### First-time runner setup
+
+```sh
+# From this directory:
+cd runner
+npm install
+npm run build
+```
+
+---
+
+## Starting the server
+
+The harness defaults to `http://localhost:8080`. Override with `PLAYER_URL`.
+
+Start the server from `player-server/` so it can resolve embedded static assets:
+
+```sh
+# From player-server/ — use testmedia/ as the media library:
+MEDIA_ROOT=./testmedia \
+SECURE_COOKIES=false \
+DB_PATH=/tmp/player-e2e-llm.db \
+./player
+```
+
+Key server environment variables:
+
+| Variable | Value | Reason |
+|---|---|---|
+| `MEDIA_ROOT` | `./testmedia` | Provides pre-existing media for upload/verify scenarios |
+| `SECURE_COOKIES` | `false` | Allows session cookie over plain HTTP |
+| `DB_PATH` | `/tmp/player-e2e-llm.db` | Isolates the test database from production |
+
+---
+
+## Environment variables (harness)
+
+| Variable | Required | Default | Description |
+|---|---|---|---|
+| `PLAYER_URL` | No | `http://localhost:8080` | Base URL of the Player server under test |
+| `ANTHROPIC_API_KEY` | Only for screenshot oracle | — | API key for Claude Haiku visual checks |
+| `LLM_E2E_SCREENSHOTS` | No | `false` | Set to `true` to enable Layer 5 screenshot + Haiku visual assertion |
+| `LLM_E2E_OPEN_ISSUE` | No | `false` | Set to `true` to open a Codeberg issue if a failure task is older than 24 h |
+| `PLAYER_DB` | No | Inferred from server config | Path to the SQLite database file; used for DB assertion checks |
+
+---
+
+## Running all scenarios
+
+```sh
+# From runner/ (after npm run build):
+node dist/index.js
+
+# Or run directly during development:
+npm run dev
+```
+
+This runs S01 through S04 in order. Each scenario is independent; a failure in
+one scenario does not stop the remaining scenarios from running.
+
+To run against a non-default server:
+
+```sh
+PLAYER_URL=http://myserver:9090 node dist/index.js
+```
+
+To enable screenshot assertions (Layer 5, costs extra API tokens):
+
+```sh
+LLM_E2E_SCREENSHOTS=true ANTHROPIC_API_KEY=sk-ant-... node dist/index.js
+```
+
+---
+
+## Running a single scenario manually
+
+Pass the scenario file path as the first argument:
+
+```sh
+# From runner/:
+node dist/index.js ../scenarios/S01-bootstrap.md
+```
+
+Or via `npm run dev` during development:
+
+```sh
+npm run dev -- ../scenarios/S01-bootstrap.md
+```
+
+The runner prints a per-step summary to stdout and exits with code 0 on
+success, 1 on failure.
+
+---
+
+## Adding a new scenario
+
+Each scenario is a single Markdown file under `scenarios/`. The file has two
+parts: a YAML front-matter block and a numbered Markdown step list.
+
+### File name convention
+
+```
+S<nn>-<short-slug>.md
+```
+
+Examples: `S05-android-skeleton.md`, `S06-bulk-sync.md`.
+
+### Skeleton
+
+```markdown
+---
+id: S05
+title: "Short human-readable title"
+tags: [auth, web]          # free-form tags for filtering
+preconditions:
+  server_state: running     # "fresh" = empty DB, "running" = existing DB with data
+  fixtures: []              # list of fixture keys the harness should load
+assertions:
+  - db: "SELECT count(*) FROM users WHERE role='admin'"
+  - url_contains: /login.html
+skip: false                 # set to "until-<reason>" to exclude from CI runs
+---
+
+1. Navigate to the root URL.
+2. Confirm the page redirects to /expected-page.html.
+3. Do something meaningful with the UI or API.
+4. Confirm the result.
+```
+
+### Front-matter fields
+
+| Field | Type | Description |
+|---|---|---|
+| `id` | string | Unique identifier, e.g. `S05` |
+| `title` | string | Human-readable title used in logs and `ask` task names |
+| `tags` | list | Free-form tags; not currently used for filtering but useful for documentation |
+| `preconditions.server_state` | `fresh` or `running` | `fresh` means the harness creates a temp empty DB; `running` means it uses the existing one |
+| `preconditions.fixtures` | list | Named fixture keys the harness loads before running steps |
+| `assertions` | list | Post-condition checks the harness runs after all steps complete |
+| `skip` | bool or string | `false` to run normally; a string like `"until-integration-test-added"` to skip in CI |
+
+### Assertion types
+
+| Key | Example | What it checks |
+|---|---|---|
+| `db` | `db: "SELECT id FROM users WHERE role='admin'"` | Runs a read-only SQLite query; passes if at least one row is returned |
+| `url_contains` | `url_contains: /login.html` | Checks the browser's current URL |
+| `selector_visible` | `selector_visible: ".media-card"` | Playwright DOM check |
+| `status_code` | `status_code: "GET /api/v1/health 200"` | HTTP status assertion |
+
+### Writing good steps
+
+- Write steps as numbered plain-English sentences. The agent interprets them; no
+  special syntax is needed inside the Markdown body.
+- Each step should do one thing: navigate, fill a form, call an API, or assert.
+- Avoid writing assertions in the steps if you can express them in the YAML
+  `assertions` block — the YAML block is cheaper (no agent reasoning required).
+- Visual checks ("does this look right?") belong in steps, not in the YAML block;
+  they trigger the Haiku screenshot oracle when `LLM_E2E_SCREENSHOTS=true`.
+
+---
+
+## Assertion layers and cost
+
+The harness uses layered assertions from cheapest to most expensive. Use the
+cheapest layer that can detect a failure:
+
+| Layer | Method | Cost |
+|---|---|---|
+| 1 | HTTP status codes | Zero — checked on every API call |
+| 2 | JSON field assertions | Cheap — parse response body |
+| 3 | DB state checks (`sqlite3` CLI) | Cheap — direct SQL query |
+| 4 | Playwright selector checks | Cheap — deterministic DOM query |
+| 5 | Screenshot + Haiku visual check | Expensive — only for S03 and S04; gated behind `LLM_E2E_SCREENSHOTS=true` |
+
+Layer 5 sends a PNG to Claude Haiku with a yes/no question such as "Is there a
+media card visible in the grid?" Haiku is used (not Sonnet) to keep per-run
+cost low.
+
+---
+
+## Cost budget
+
+| Run type | Scenarios | Estimated cost |
+|---|---|---|
+| Full nightly (S01-S04) | 4 | ~$0.18 per run |
+| Single scenario | 1 | ~$0.04-0.05 per run |
+| Full nightly with retries | 4 + retries | ~$0.27 per run |
+
+At one nightly run the expected spend is **~$5.50/month** (Sonnet for
+orchestration, Haiku for screenshots).
+
+**Alert threshold:** If a single run exceeds **$0.50**, treat it as anomalous.
+Common causes are a context window blowup (Playwright output not capped) or
+excessive retries. The runner caps Playwright CLI output forwarded to the agent
+at 4 000 characters to help control this.
+
+**Monthly soft budget:** $10/month for the entire LLM e2e layer. If you
+consistently see runs above $0.50, check for long scenario files, verbose
+fixtures, or scenarios that retry repeatedly.
+
+---
+
+## Failure handling
+
+The runner uses a one-retry policy:
+
+```
+Run scenario
+├── Pass  → log "PASS", continue to next scenario
+└── Fail
+    ├── Wait 5 s, retry once (fresh browser context)
+    │   ├── Pass  → log "FLAKY — passed on retry", continue
+    │   └── Fail  → open ask task, log "FAIL", continue to next scenario
+```
+
+When a scenario fails after the retry, the runner opens a task:
+
+```sh
+ask add "LLM e2e failure: <scenario title> — <one-line reason>"
+```
+
+The task description includes the Playwright HTML report path and a snippet of
+the failure output so a developer can reproduce the failure without re-reading
+logs.
+
+The runner does **not**:
+- retry more than once (cost control)
+- stop the run on a single failure (other scenarios still run)
+- send email or Slack notifications
+- auto-fix failing scenarios
+
+To open a Codeberg issue automatically when a failure task is older than 24 h,
+set `LLM_E2E_OPEN_ISSUE=true`. This is off by default.
+
+---
+
+## Nightly schedule
+
+The nightly run is configured via the `/schedule` skill. It runs a Claude Code
+agent with the prompt:
+
+> Run the LLM e2e suite at `player-server/test/e2e-llm/` against the
+> production server and report results.
+
+The agent reads the scenario files, executes them via the runner, and annotates
+an ongoing task with the result. To set up or modify the schedule, use:
+
+```sh
+/schedule
+```
+
+---
+
+## Relationship to the Playwright smoke suite
+
+| Dimension | Playwright smoke (`../e2e-web/`) | LLM e2e (this suite) |
+|---|---|---|
+| Assertion style | Deterministic CSS selectors | Natural-language steps + layered assertions |
+| When to run | Every PR, fast | Nightly, slower |
+| Best for | "Did the page render? Did the API return 200?" | Cross-stack flows, auth boundaries, multi-step sequences |
+| Cost | Zero (no LLM calls) | ~$0.18/run |
+| Visual checks | No | Yes, via Haiku (opt-in) |
+
+The two suites are complementary. The Playwright smoke suite is the first line
+of defence for regressions. The LLM e2e suite catches higher-level integration
+failures that selector-based tests cannot easily express.
diff --git a/player-server/test/e2e-llm/runner/.gitignore b/player-server/test/e2e-llm/runner/.gitignore
new file mode 100644
index 0000000..b947077
--- /dev/null
+++ b/player-server/test/e2e-llm/runner/.gitignore
@@ -0,0 +1,2 @@
+node_modules/
+dist/
diff --git a/player-server/test/e2e-llm/runner/index.ts b/player-server/test/e2e-llm/runner/index.ts
new file mode 100644
index 0000000..fb973c9
--- /dev/null
+++ b/player-server/test/e2e-llm/runner/index.ts
@@ -0,0 +1,364 @@
+/**
+ * index.ts — LLM e2e harness runner.
+ *
+ * Reads YAML front-matter + Markdown scenario files from ../scenarios/,
+ * invokes the Playwright CLI (npx playwright test --reporter=json),
+ * parses JSON output to determine pass/fail, and retries once on failure.
+ * On double-failure it opens an `ask` task so the issue is tracked.
+ *
+ * Usage:
+ *   node dist/index.js                    # run all scenarios
+ *   node dist/index.js scenarios/S01.md  # run one scenario
+ *
+ * The runner drives tests by injecting the scenario id as SCENARIO_ID so
+ * the Playwright suite can filter on it when a scenario-specific test file
+ * exists in e2e-web/tests/.  If no matching test is found, Playwright exits 0
+ * with zero test results, which the runner treats as a skip.
+ */
+
+import * as fs from 'fs';
+import * as path from 'path';
+import * as yaml from 'js-yaml';
+import { spawnSync, SpawnSyncReturns } from 'child_process';
+
+// ---------------------------------------------------------------------------
+// Types
+// ---------------------------------------------------------------------------
+
+/** Parsed YAML front-matter from a scenario file. */
+interface ScenarioMeta {
+  id: string;
+  title: string;
+  tags?: string[];
+  skip?: string;         // non-empty string means skip; value is the reason
+  preconditions?: Record<string, unknown>;
+  assertions?: unknown[];
+}
+
+/** Holds the parsed content of a scenario file. */
+interface Scenario {
+  meta: ScenarioMeta;
+  steps: string;         // raw Markdown body (the numbered steps)
+  filePath: string;
+}
+
+/** Subset of the Playwright JSON reporter output that we care about. */
+interface PlaywrightReport {
+  stats: {
+    expected: number;
+    unexpected: number;
+    skipped: number;
+    flaky: number;
+  };
+  errors?: Array<{ message?: string }>;
+  suites?: PlaywrightSuite[];
+}
+
+interface PlaywrightSuite {
+  title: string;
+  specs?: PlaywrightSpec[];
+  suites?: PlaywrightSuite[];
+}
+
+interface PlaywrightSpec {
+  title: string;
+  ok: boolean;
+  tests?: Array<{ status: string; results?: Array<{ error?: { message?: string } }> }>;
+}
+
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+
+// Playwright config is at test/e2e-web/ — three levels up from runner/dist/.
+const PLAYWRIGHT_CONFIG = path.resolve(__dirname, '../../../e2e-web/playwright.config.ts');
+
+// Scenario files are at test/e2e-llm/scenarios/ — two levels up from runner/dist/.
+const SCENARIOS_DIR = path.resolve(__dirname, '../../scenarios');
+
+// How long to wait between a first failure and the retry, in ms.
+const RETRY_DELAY_MS = 5_000;
+
+// Max characters of Playwright output forwarded in the ask task description.
+// Keeps token costs manageable when the orchestrator reads the annotation.
+const MAX_OUTPUT_CHARS = 4_000;
+
+// ---------------------------------------------------------------------------
+// Scenario file parsing
+// ---------------------------------------------------------------------------
+
+/**
+ * parseFrontMatter splits a file into YAML front-matter and a Markdown body.
+ * Front-matter is delimited by leading and trailing `---` lines.
+ * Returns null if the file does not start with `---`.
+ */
+function parseFrontMatter(content: string): { meta: ScenarioMeta; steps: string } | null {
+  const lines = content.split('\n');
+  if (lines[0].trim() !== '---') return null;
+
+  const closeIdx = lines.findIndex((l, i) => i > 0 && l.trim() === '---');
+  if (closeIdx === -1) return null;
+
+  const yamlText = lines.slice(1, closeIdx).join('\n');
+  const body = lines.slice(closeIdx + 1).join('\n').trim();
+
+  const meta = yaml.load(yamlText) as ScenarioMeta;
+  if (!meta?.id || !meta?.title) {
+    throw new Error(`Scenario front-matter missing required fields 'id' and 'title'`);
+  }
+  return { meta, steps: body };
+}
+
+/**
+ * loadScenario reads and parses one scenario file.
+ */
+function loadScenario(filePath: string): Scenario {
+  const content = fs.readFileSync(filePath, 'utf8');
+  const parsed = parseFrontMatter(content);
+  if (!parsed) {
+    throw new Error(`${filePath}: does not start with YAML front-matter (--- delimiter)`);
+  }
+  return { ...parsed, filePath };
+}
+
+/**
+ * discoverScenarios returns all .md files in SCENARIOS_DIR sorted by filename.
+ * A specific file path can be passed to restrict the run to one scenario.
+ */
+function discoverScenarios(specificFile?: string): Scenario[] {
+  if (specificFile) {
+    // Resolve relative to cwd so callers can pass e.g. scenarios/S01.md
+    const abs = path.isAbsolute(specificFile)
+      ? specificFile
+      : path.resolve(process.cwd(), specificFile);
+    return [loadScenario(abs)];
+  }
+
+  if (!fs.existsSync(SCENARIOS_DIR)) {
+    console.warn(`[runner] Scenarios directory not found: ${SCENARIOS_DIR}`);
+    return [];
+  }
+
+  return fs
+    .readdirSync(SCENARIOS_DIR)
+    .filter(f => f.endsWith('.md'))
+    .sort()
+    .map(f => loadScenario(path.join(SCENARIOS_DIR, f)));
+}
+
+// ---------------------------------------------------------------------------
+// Playwright invocation
+// ---------------------------------------------------------------------------
+
+/**
+ * runPlaywright invokes `npx playwright test --reporter=json` with the
+ * scenario id injected as SCENARIO_ID so the test suite can filter.
+ * Returns the raw stdout string (JSON reporter output).
+ */
+function runPlaywright(scenario: Scenario): SpawnSyncReturns<string> {
+  const env: NodeJS.ProcessEnv = {
+    ...process.env,
+    SCENARIO_ID: scenario.meta.id,
+  };
+
+  // The Playwright config is in e2e-web/; we run npx from there so that
+  // node_modules/.bin/playwright is available without a separate install.
+  const cwd = path.dirname(PLAYWRIGHT_CONFIG);
+
+  return spawnSync(
+    'npx',
+    ['playwright', 'test', '--reporter=json', '--config', PLAYWRIGHT_CONFIG],
+    { cwd, env, encoding: 'utf8', maxBuffer: 10 * 1024 * 1024 },
+  );
+}
+
+// ---------------------------------------------------------------------------
+// Result parsing
+// ---------------------------------------------------------------------------
+
+/**
+ * parseReport extracts pass/fail information from Playwright JSON reporter
+ * output. Returns { passed, reason } where reason is populated on failure.
+ */
+function parseReport(stdout: string): { passed: boolean; reason: string } {
+  let report: PlaywrightReport;
+  try {
+    // Try the full output first (clean JSON). When Playwright emits debug lines
+    // before the JSON object, find the first newline-prefixed `{` instead —
+    // this avoids false matches on `{` inside warning text.
+    const trimmed = stdout.trim();
+    const jsonStr = trimmed.startsWith('{')
+      ? trimmed
+      : (() => {
+          const nlIdx = stdout.indexOf('\n{');
+          return nlIdx !== -1 ? stdout.slice(nlIdx + 1) : '';
+        })();
+    if (!jsonStr) return { passed: false, reason: 'No JSON object in Playwright output' };
+    report = JSON.parse(jsonStr) as PlaywrightReport;
+  } catch {
+    return { passed: false, reason: `Cannot parse Playwright JSON: ${stdout.slice(0, 200)}` };
+  }
+
+  const { unexpected, expected, skipped } = report.stats;
+
+  // Zero tests means the scenario filter matched nothing — treat as skip/pass
+  // so that adding a scenario file before its Playwright spec does not fail.
+  if (expected === 0 && unexpected === 0 && skipped === 0) {
+    return { passed: true, reason: 'no tests matched (scenario not yet implemented in e2e-web)' };
+  }
+
+  if (unexpected === 0) {
+    return { passed: true, reason: '' };
+  }
+
+  // Collect the first error message from the report hierarchy for the reason.
+  const reason = collectFirstError(report) ?? `${unexpected} test(s) failed`;
+  return { passed: false, reason };
+}
+
+/**
+ * collectFirstError walks the Playwright report tree to find the first
+ * failure error message. Returns undefined if none is found.
+ */
+function collectFirstError(report: PlaywrightReport): string | undefined {
+  // Top-level errors (e.g. global setup failures).
+  if (report.errors && report.errors.length > 0) {
+    return report.errors[0].message ?? undefined;
+  }
+  if (!report.suites) return undefined;
+
+  // Walk suites recursively.
+  const walkSuites = (suites: PlaywrightSuite[]): string | undefined => {
+    for (const suite of suites) {
+      if (suite.specs) {
+        for (const spec of suite.specs) {
+          if (!spec.ok && spec.tests) {
+            for (const t of spec.tests) {
+              if (t.results) {
+                for (const r of t.results) {
+                  if (r.error?.message) return r.error.message.slice(0, 300);
+                }
+              }
+            }
+          }
+        }
+      }
+      if (suite.suites) {
+        const found = walkSuites(suite.suites);
+        if (found) return found;
+      }
+    }
+    return undefined;
+  };
+
+  return walkSuites(report.suites);
+}
+
+// ---------------------------------------------------------------------------
+// Failure handling
+// ---------------------------------------------------------------------------
+
+/**
+ * openAskTask creates a tracked task via the `ask` CLI when a scenario fails
+ * on both the initial run and its retry. The task title includes the scenario
+ * title and the failure reason so it is actionable without further context.
+ */
+function openAskTask(scenario: Scenario, reason: string, playwrightOutput: string): void {
+  const truncated = playwrightOutput.slice(0, MAX_OUTPUT_CHARS);
+  const title = `+test-llm-e2e LLM e2e failure: ${scenario.meta.title} — ${reason}`;
+
+  // Use spawnSync to avoid shell injection from scenario titles that contain
+  // backticks, dollar signs, or quotes.
+  const result = spawnSync('ask', ['add', title], { stdio: 'inherit' });
+  if (result.status !== 0) {
+    console.error(`[runner] Failed to open ask task for scenario ${scenario.meta.id}`);
+  }
+
+  // Print the truncated output so CI logs capture it even if ask is unavailable.
+  console.error(`[runner] Playwright output (truncated to ${MAX_OUTPUT_CHARS} chars):`);
+  console.error(truncated);
+}
+
+// ---------------------------------------------------------------------------
+// Per-scenario run (with one retry)
+// ---------------------------------------------------------------------------
+
+/**
+ * runScenario executes a scenario once, retries on failure after a short
+ * delay, and calls openAskTask on double-failure.
+ * Returns true if the scenario ultimately passed (or was skipped).
+ */
+function runScenario(scenario: Scenario): boolean {
+  const { id, title, skip } = scenario.meta;
+
+  if (skip) {
+    console.log(`[runner] SKIP  ${id}: ${title} — ${skip}`);
+    return true;
+  }
+
+  console.log(`[runner] RUN   ${id}: ${title}`);
+
+  // First attempt.
+  const first = runPlaywright(scenario);
+  const firstResult = parseReport(first.stdout ?? '');
+
+  if (firstResult.passed) {
+    console.log(`[runner] PASS  ${id}: ${title}`);
+    return true;
+  }
+
+  console.warn(`[runner] FAIL  ${id}: ${title} — ${firstResult.reason}`);
+  console.warn(`[runner] Retrying in ${RETRY_DELAY_MS / 1000}s…`);
+
+  // Wait before retry to let transient server issues settle.
+  // spawnSync('sleep') is a reliable synchronous pause that works on any POSIX
+  // system (including the CI container) without relying on Atomics or timers.
+  spawnSync('sleep', [String(RETRY_DELAY_MS / 1000)]);
+
+  // Single retry.
+  const second = runPlaywright(scenario);
+  const secondResult = parseReport(second.stdout ?? '');
+
+  if (secondResult.passed) {
+    console.log(`[runner] PASS  ${id}: ${title} (passed on retry — flaky)`);
+    return true;
+  }
+
+  console.error(`[runner] FAIL  ${id}: ${title} — double-failure, opening ask task`);
+  openAskTask(scenario, secondResult.reason, second.stdout ?? '');
+  return false;
+}
+
+// ---------------------------------------------------------------------------
+// Entry point
+// ---------------------------------------------------------------------------
+
+/**
+ * main discovers scenarios (or uses the one passed as argv[2]), runs each in
+ * sequence, and exits non-zero if any scenario failed after its retry.
+ */
+function main(): void {
+  const specificFile = process.argv[2];
+  const scenarios = discoverScenarios(specificFile);
+
+  if (scenarios.length === 0) {
+    console.log('[runner] No scenario files found — nothing to run.');
+    process.exit(0);
+  }
+
+  console.log(`[runner] Running ${scenarios.length} scenario(s)…`);
+
+  let failures = 0;
+  for (const scenario of scenarios) {
+    const passed = runScenario(scenario);
+    if (!passed) failures++;
+  }
+
+  const total = scenarios.length;
+  const passed = total - failures;
+  console.log(`\n[runner] Results: ${passed}/${total} passed, ${failures} failed.`);
+
+  process.exit(failures > 0 ? 1 : 0);
+}
+
+main();
diff --git a/player-server/test/e2e-llm/runner/package-lock.json b/player-server/test/e2e-llm/runner/package-lock.json
new file mode 100644
index 0000000..df7de58
--- /dev/null
+++ b/player-server/test/e2e-llm/runner/package-lock.json
@@ -0,0 +1,76 @@
+{
+  "name": "player-e2e-llm-runner",
+  "version": "1.0.0",
+  "lockfileVersion": 3,
+  "requires": true,
+  "packages": {
+    "": {
+      "name": "player-e2e-llm-runner",
+      "version": "1.0.0",
+      "dependencies": {
+        "js-yaml": "^4.1.0"
+      },
+      "devDependencies": {
+        "@types/js-yaml": "^4.0.9",
+        "@types/node": "^22.0.0",
+        "typescript": "^5.8.0"
+      }
+    },
+    "node_modules/@types/js-yaml": {
+      "version": "4.0.9",
+      "resolved": "https://registry.npmjs.org/@types/js-yaml/-/js-yaml-4.0.9.tgz",
+      "integrity": "sha512-k4MGaQl5TGo/iipqb2UDG2UwjXziSWkh0uysQelTlJpX1qGlpUZYm8PnO4DxG1qBomtJUdYJ6qR6xdIah10JLg==",
+      "dev": true,
+      "license": "MIT"
+    },
+    "node_modules/@types/node": {
+      "version": "22.19.19",
+      "resolved": "https://registry.npmjs.org/@types/node/-/node-22.19.19.tgz",
+      "integrity": "sha512-dyh/xO2Fh5bYrfWaaqGrRQQGkNdmYw6AmaAUvYeUMNTWQtvb796ikLdmTchRmOlOiIJ1TDXfWgVx1QkUlQ6Hew==",
+      "dev": true,
+      "license": "MIT",
+      "dependencies": {
+        "undici-types": "~6.21.0"
+      }
+    },
+    "node_modules/argparse": {
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz",
+      "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==",
+      "license": "Python-2.0"
+    },
+    "node_modules/js-yaml": {
+      "version": "4.1.1",
+      "resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-4.1.1.tgz",
+      "integrity": "sha512-qQKT4zQxXl8lLwBtHMWwaTcGfFOZviOJet3Oy/xmGk2gZH677CJM9EvtfdSkgWcATZhj/55JZ0rmy3myCT5lsA==",
+      "license": "MIT",
+      "dependencies": {
+        "argparse": "^2.0.1"
+      },
+      "bin": {
+        "js-yaml": "bin/js-yaml.js"
+      }
+    },
+    "node_modules/typescript": {
+      "version": "5.9.3",
+      "resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
+      "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
+      "dev": true,
+      "license": "Apache-2.0",
+      "bin": {
+        "tsc": "bin/tsc",
+        "tsserver": "bin/tsserver"
+      },
+      "engines": {
+        "node": ">=14.17"
+      }
+    },
+    "node_modules/undici-types": {
+      "version": "6.21.0",
+      "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz",
+      "integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==",
+      "dev": true,
+      "license": "MIT"
+    }
+  }
+}
diff --git a/player-server/test/e2e-llm/runner/package.json b/player-server/test/e2e-llm/runner/package.json
new file mode 100644
index 0000000..de1ae9d
--- /dev/null
+++ b/player-server/test/e2e-llm/runner/package.json
@@ -0,0 +1,20 @@
+{
+  "name": "player-e2e-llm-runner",
+  "version": "1.0.0",
+  "description": "LLM-driven e2e harness runner: parses YAML+Markdown scenarios, invokes Playwright CLI, retries on failure, opens ask tasks on double-failure.",
+  "private": true,
+  "main": "dist/index.js",
+  "scripts": {
+    "build": "tsc",
+    "start": "node dist/index.js",
+    "dev": "tsc && node dist/index.js"
+  },
+  "dependencies": {
+    "js-yaml": "^4.1.0"
+  },
+  "devDependencies": {
+    "@types/js-yaml": "^4.0.9",
+    "@types/node": "^22.0.0",
+    "typescript": "^5.8.0"
+  }
+}
diff --git a/player-server/test/e2e-llm/runner/tsconfig.json b/player-server/test/e2e-llm/runner/tsconfig.json
new file mode 100644
index 0000000..98b51cd
--- /dev/null
+++ b/player-server/test/e2e-llm/runner/tsconfig.json
@@ -0,0 +1,15 @@
+{
+  "compilerOptions": {
+    "target": "ES2022",
+    "module": "CommonJS",
+    "lib": ["ES2022"],
+    "strict": true,
+    "esModuleInterop": true,
+    "skipLibCheck": true,
+    "outDir": "./dist",
+    "rootDir": ".",
+    "types": ["node"]
+  },
+  "include": ["**/*.ts"],
+  "exclude": ["node_modules", "dist"]
+}
diff --git a/player-server/test/e2e-llm/scenarios/.gitkeep b/player-server/test/e2e-llm/scenarios/.gitkeep
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/player-server/test/e2e-llm/scenarios/.gitkeep