summaryrefslogtreecommitdiff
path: root/player-server/test/e2e-llm/README.md
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2026-05-18 23:39:32 +0300
committerPaul Buetow <paul@buetow.org>2026-05-18 23:39:32 +0300
commit77e2670ac625c716526fdfdbbf0b4eb8e00e77b1 (patch)
tree28c8296ebc6c72a12879d59897c21c9d6cb428f0 /player-server/test/e2e-llm/README.md
parenta9b75513661cb9196cb6bd614f56c4bf09c970b5 (diff)
Switch oracle from Claude Haiku to Ollama cloud (qwen3-vl:235b-instruct)
Replaces the Anthropic SDK dependency in oracle.ts with direct fetch calls to the Ollama cloud API (https://ollama.com/v1/chat/completions). Uses qwen3-vl:235b-instruct — the strongest vision-language model available on the Ollama cloud — for screenshot analysis. The model is configurable via OLLAMA_MODEL and the endpoint via OLLAMA_BASE_URL so local Ollama instances can be used during development. OLLAMA_API_KEY carries the cloud bearer token. Smoke-tested: gate-off returns true without a network call; yes/no questions against a test PNG return correct answers via the cloud API. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Diffstat (limited to 'player-server/test/e2e-llm/README.md')
-rw-r--r--player-server/test/e2e-llm/README.md31
1 files changed, 23 insertions, 8 deletions
diff --git a/player-server/test/e2e-llm/README.md b/player-server/test/e2e-llm/README.md
index 872efa9..842db98 100644
--- a/player-server/test/e2e-llm/README.md
+++ b/player-server/test/e2e-llm/README.md
@@ -84,8 +84,10 @@ Key server environment variables:
| Variable | Required | Default | Description |
|---|---|---|---|
| `PLAYER_URL` | No | `http://localhost:8080` | Base URL of the Player server under test |
-| `ANTHROPIC_API_KEY` | Only for screenshot oracle | — | API key for Claude Haiku visual checks |
-| `LLM_E2E_SCREENSHOTS` | No | `false` | Set to `true` to enable Layer 5 screenshot + Haiku visual assertion |
+| `LLM_E2E_SCREENSHOTS` | No | `false` | Set to `true` to enable Layer 5 screenshot + vision model assertion |
+| `OLLAMA_BASE_URL` | No | `https://ollama.com` | Ollama API endpoint; override for a local instance |
+| `OLLAMA_API_KEY` | Yes (for cloud) | — | API key from ollama.com; omit only for local unauthenticated instances |
+| `OLLAMA_MODEL` | No | `qwen3-vl:235b-instruct` | Vision model on the cloud endpoint; override for a local model |
| `LLM_E2E_OPEN_ISSUE` | No | `false` | Set to `true` to open a Codeberg issue if a failure task is older than 24 h |
| `PLAYER_DB` | No | Inferred from server config | Path to the SQLite database file; used for DB assertion checks |
@@ -110,10 +112,23 @@ To run against a non-default server:
PLAYER_URL=http://myserver:9090 node dist/index.js
```
-To enable screenshot assertions (Layer 5, costs extra API tokens):
+To enable screenshot assertions (Layer 5, requires an ollama.com API key):
```sh
-LLM_E2E_SCREENSHOTS=true ANTHROPIC_API_KEY=sk-ant-... node dist/index.js
+LLM_E2E_SCREENSHOTS=true \
+ OLLAMA_API_KEY=<your-ollama.com-key> \
+ node dist/index.js
+```
+
+The oracle defaults to `llama3.2-vision` via `https://ollama.com`. To use a
+different model or a local Ollama instance:
+
+```sh
+# Different cloud model:
+LLM_E2E_SCREENSHOTS=true OLLAMA_API_KEY=<key> OLLAMA_MODEL=llava:13b node dist/index.js
+
+# Local Ollama (no API key needed):
+LLM_E2E_SCREENSHOTS=true OLLAMA_BASE_URL=http://localhost:11434 node dist/index.js
```
---
@@ -217,11 +232,11 @@ cheapest layer that can detect a failure:
| 2 | JSON field assertions | Cheap — parse response body |
| 3 | DB state checks (`sqlite3` CLI) | Cheap — direct SQL query |
| 4 | Playwright selector checks | Cheap — deterministic DOM query |
-| 5 | Screenshot + Haiku visual check | Expensive — only for S03 and S04; gated behind `LLM_E2E_SCREENSHOTS=true` |
+| 5 | Screenshot + Ollama vision check | Self-hosted cost only — gated behind `LLM_E2E_SCREENSHOTS=true` |
-Layer 5 sends a PNG to Claude Haiku with a yes/no question such as "Is there a
-media card visible in the grid?" Haiku is used (not Sonnet) to keep per-run
-cost low.
+Layer 5 sends a PNG to an Ollama vision model (`llama3.2-vision`) with a yes/no
+question such as "Is there a media card visible in the grid?" Using a self-hosted
+Ollama instance means no per-call API cost beyond compute.
---