# CI Pipeline Design Proposal **Status:** Proposal / Decision gate — requires user approval before implementation. **Date:** 2026-05-18 **Scope:** `codeberg.org/snonux/player` monorepo — `player-server/` (Go) and `player-android/` (Flutter) --- ## Summary of Recommendations | Question | Decision | |----------|----------| | Provider | Woodpecker CI (primary); GitHub Actions optional mirror | | Matrix | Go on Linux only; Flutter on Linux only | | Stages | Unit (fast gate) → Integration + Web smoke (parallel) → LLM e2e (nightly, guarded) | | Emulator | Deferred — `flutter analyze` + `flutter test` only (per c6 approval) | | Secrets | Woodpecker project secrets; LLM jobs on protected branch only | | Caching | Go module + build cache via Woodpecker volumes; Flutter pub cache via volume | --- ## 1. Provider ### The choice: Woodpecker CI on Codeberg The repository lives at `codeberg.org/snonux/player`. Codeberg's native CI system is Woodpecker CI, which reads pipeline definitions from `.woodpecker/` YAML files. Using Woodpecker as the primary CI system is the path of least friction: - No token-based mirroring required for basic push/PR triggers. - Codeberg's shared Woodpecker runners are available for free-tier public repos. - Configuration is near-identical to GitHub Actions conceptually (steps inside jobs inside pipelines) so migration later is low-cost. ### Trade-offs vs GitHub Actions | Factor | Woodpecker CI | GitHub Actions | |--------|---------------|----------------| | Native Codeberg integration | Yes (no mirror needed) | Requires mirror push | | Shared runner availability | Codeberg shared pool | Generous free tier | | KVM for Android emulator | Not guaranteed on shared runners | Available on `ubuntu-latest` | | Ecosystem / community docs | Smaller but sufficient | Very large | | Cost at current scale | Free (shared pool) | Free (within monthly quota) | | Self-hosted runner support | Yes (same agent) | Yes | ### Recommendation Use **Woodpecker CI as the sole provider** for now. The pipeline files live in `.woodpecker/`. Do not add a GitHub Actions mirror until there is a concrete reason (e.g., emulator tests that require GitHub's KVM-enabled runners, or GitHub Packages publishing). Adding both providers simultaneously doubles maintenance without adding value at this stage. If the emulator decision changes later and Codeberg shared runners cannot provide KVM, revisit a targeted GitHub Actions mirror for Android integration tests only — keeping the Go and Flutter unit pipelines on Woodpecker. --- ## 2. Matrix ### Go server: Linux only The Go server (`player-server/`) targets Linux (Docker image, k8s deployment). There is no Windows or macOS deployment path. Adding macOS/Windows to the test matrix would increase CI time and cost without testing anything relevant to production. The Go standard library and `modernc.org/sqlite` (pure-Go SQLite port used here) are cross-platform, but that portability is incidental. Tests should run on the same OS family as the deployment target. **Decision: Linux only.** Single-OS matrix for Go. ### Flutter Android: Linux only `flutter test` for unit tests runs on any OS with the Flutter SDK installed. The Android SDK and emulator are deferred (see Section 4). Linux suffices for `flutter analyze` and `flutter test`. **Decision: Linux only.** Single-OS matrix for Flutter. Revisit if macOS becomes necessary for real device testing (e.g., iOS), which is out of scope for this project. ### Go architecture: amd64 only The f3s homelab's Raspberry Pi 3 nodes are aarch64, but they are not suitable CI runner targets (arm64 cannot run an x86_64 Android emulator, and Go cross-compilation testing is a different concern from functional testing). The Beelink S12 Pro nodes (Intel N100, x86_64) are viable as self-hosted runners but are not needed at this stage — shared Codeberg runners handle the load. --- ## 3. Pipeline Stages The pipeline has four stages with clear entry conditions and failure semantics. ### Stage 1 — Unit (fast gate) **Trigger:** every push and pull request to any branch. **Jobs (run in parallel):** | Job | Command | Working dir | Estimated duration | |-----|---------|-------------|--------------------| | `server-unit` | `go test -race -count=1 ./...` | `player-server/` | ~10–15 s | | `android-unit` | `flutter analyze && flutter test` | `player-android/` | ~2–3 min | **Gate semantics:** Both jobs must pass before any downstream stage starts. A failure in either blocks integration and smoke stages. This keeps the feedback loop fast: the Go suite returns in under 15 seconds, so developers see a pass/fail signal on push within two minutes (dominated by Flutter SDK setup). **Why this order matters:** Unit tests have no external dependencies (no running server, no network, no database file). They can be parallelised freely and fail fast. Making them mandatory before anything else prevents wasted time running integration tests against broken code. ### Stage 2 — Integration + Web smoke (parallel, after Stage 1) **Trigger:** push to `main`; pull requests targeting `main`. Not every feature branch needs integration tests on every push. Scoping this to PRs targeting `main` (and pushes to `main`) strikes the right balance: fast feedback on unit tests during development, thorough integration checking at merge time. **Jobs (run in parallel after Stage 1 passes):** #### Job: `server-integration` Runs the Go integration tests against a real SQLite database. ``` go test -race -count=1 -tags=integration ./... ``` Working dir: `player-server/` Estimated duration: ~20–30 s (full API surface including auth methods and bulk sync). No running server process needed — integration tests start their own in-process server. #### Job: `web-smoke` Runs the Playwright smoke suite against a real running server with test media. ``` # Start server with test database and testdata/media MEDIA_ROOT=./testdata/media SECURE_COOKIES=false DB_PATH=/tmp/player-e2e-ci.db \ ./player & # Wait for server readiness (done inside the Playwright globalSetup hook) cd test/e2e-web npm ci npm run install-browsers CI=true npm test ``` Working dir: `player-server/` Estimated duration: ~3–5 min (Chromium install ~1 min, suite ~2–3 min). Requires building the server binary first (add a build step before the server is started, or use a compiled artifact from Stage 1's build output if the runner persists workspaces). **Trade-off for web-smoke:** The Playwright suite requires a running server, a real SQLite database, and Chromium. This makes it heavier than unit tests and unsuitable as a universal gate. Running it only on PRs to `main` and on `main` itself is the right scope. The suite is Chromium-only by design (see `playwright.config.ts` — adding Firefox/WebKit is deferred until cross-browser compatibility is a stated requirement). ### Stage 3 — LLM e2e (nightly, cost-guarded) **Trigger:** nightly cron on `main`; optionally on manual dispatch only. This stage covers any LLM-assisted end-to-end tests (currently a future concern, but included in the design to answer the secrets and cost questions explicitly). **Structure:** - Runs only on `main` branch (not on PRs). - Guarded by a branch protection check so secrets are not exposed to untrusted forks. - A per-run cost estimate should be logged to the job output so spend is visible. **Estimated cost:** Depends on the LLM API used and test volume. A simple benchmark: if the suite makes 50 API calls at GPT-4o pricing (~$0.005/call), one nightly run costs ~$0.25. This is low but should be logged and monitored as the suite grows. **Cost-control mechanism:** Set a hard token/call budget in the test harness. If the budget is exceeded, the test fails rather than silently consuming quota. The nightly job should emit a cost summary line that can be parsed by a monitoring step. **Current state:** No LLM e2e tests exist yet. This stage is a placeholder in the pipeline design so the implementation task (Section 6) wires it in correctly from the start. For now, the nightly job can be a stub that exits 0 until real tests are added. --- ## 4. Emulator Job Per the c6 decision (captured in `player-android/docs/testing-strategy.md`): - The Android CI job runs `flutter analyze` and `flutter test` only. - No emulator, no KVM, no `flutter drive`, no `integration_test` scaffolding. - This decision holds until real navigation (a second named route) lands in the app. The emulator question is therefore **deferred**. The pipeline design accommodates this by keeping Stage 1's `android-unit` job emulator-free and making no assumptions about KVM availability on the Woodpecker shared runners. When the emulator decision is revisited (concrete trigger: first real multi-screen navigation is implemented), the choices are: 1. **GitHub Actions KVM runners** — `ubuntu-latest` provides KVM; add a targeted GitHub Actions workflow for Android integration tests while keeping everything else on Woodpecker. 2. **f3s homelab Beelink x86 self-hosted runner** — Intel N100 supports VT-x; a self-hosted Woodpecker agent on one of the bhyve VMs could provide KVM. Setup cost is high (Android SDK, Flutter SDK, emulator AVD, runner agent maintenance). Viable long-term if cloud CI minutes become a concern. 3. **Raspberry Pi 3 nodes** — not viable for x86_64 Android emulator; ruled out. **For now:** no emulator job. The `android-unit` step in Stage 1 is the full extent of Android CI. --- ## 5. Secrets ### Where secrets live Woodpecker CI stores secrets at the project level (per-repository) via the Woodpecker UI or API. Secrets are injected as environment variables into pipeline steps that explicitly request them via the `secrets:` key in the step definition. For this project, two categories of secrets exist: #### Category A — Always available (build-time only) No secrets are needed for the Go or Flutter unit/integration tests. These jobs use only in-repo code, no network APIs, and no external services. Category A is empty at this stage. #### Category B — LLM API keys (nightly only, protected branch only) LLM API keys (e.g., `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`) are required only for the Stage 3 LLM e2e job. **Access control:** - Define the secret in Woodpecker with the "Protected" flag enabled. Woodpecker's protected secret mode restricts the secret to pipelines running on the repository's own pushes (not pull requests from forks). This prevents a malicious PR from exfiltrating the key by adding a print step. - The LLM e2e job should additionally check `CI_COMMIT_BRANCH == "main"` as a belt-and-suspenders guard; if not on `main`, the job skips rather than errors. - The nightly cron trigger is scoped to `main`, reinforcing the branch guard. **Why not GitHub Actions secrets:** If a GitHub Actions mirror is added later, GitHub's `secrets` context provides an equivalent mechanism with the same fork protection semantics. The design is portable. **No secrets in `.woodpecker/` YAML files.** All sensitive values come from the Woodpecker secret store. The YAML files are committed to the repository and must never contain raw keys. --- ## 6. Caching Caching reduces CI setup time significantly. The two dominant costs are Go module download and Flutter SDK + pub package download. ### Go module cache Woodpecker supports Docker volume mounts between pipeline steps. The Go module cache lives at `$GOPATH/pkg/mod` (typically `/root/go/pkg/mod` in the CI container). **Mechanism:** Use a named Woodpecker volume (e.g., `go-mod-cache`) mounted at `/root/go/pkg/mod`. Steps that run `go test` or `go build` populate the cache on first run; subsequent runs reuse it. The build cache (`$GOCACHE`, typically `/root/.cache/go-build`) should also be persisted via a named volume (`go-build-cache`). **Cache invalidation:** Go modules are content-addressed. No explicit invalidation is needed — `go.sum` ensures the right versions are fetched. The volume accumulates entries over time; a periodic volume purge (manual, or on a monthly timer) keeps disk usage bounded. **Estimated savings:** Go module download for this project (~15 deps) takes ~20–30 s on a cold cache. With a warm cache, `go test` starts in under 2 s. ### Flutter pub cache Flutter's package cache lives at `$PUB_CACHE` (typically `$HOME/.pub-cache`). A named Woodpecker volume mounted at `/root/.pub-cache` provides the same benefit as the Go module cache. The Flutter SDK itself is larger (~500 MB). If the CI runner image does not bundle Flutter, the SDK download dominates. Options: 1. **Pre-built Docker image with Flutter SDK baked in** — eliminates SDK download on every run. Maintain a `Dockerfile` in the repo (or use a community image like `cirrusci/flutter:stable`) as the CI container image. 2. **Volume cache for the SDK** — mount the SDK installation directory as a named volume. Works but is more brittle (SDK version pinning requires cache invalidation on version bump). **Recommendation:** Use a pre-built Flutter Docker image for the `android-unit` job. The `pubspec.lock` is committed, so `flutter pub get` with a warm pub cache runs in under 5 s. ### SQLite test fixtures The Go integration tests use `:memory:` SQLite databases opened fresh per test (`newTestStore` in `sqlite_test.go`). There are no persistent fixture files to cache — each test is hermetic by design. The Playwright smoke tests use a temporary file at `/tmp/player-e2e-ci.db`. This file is discarded at the end of the job. No caching applies. **No SQLite fixture caching is needed.** The hermetic design is correct and must be preserved. ### Playwright browser cache Chromium installation via `npm run install-browsers` downloads ~150 MB. This can be cached by persisting the Playwright browser directory (`/root/.cache/ms-playwright`). A named Woodpecker volume or a cache key based on the `@playwright/test` version in `package-lock.json` handles this. **Mechanism:** Named volume `playwright-chromium` mounted at `/root/.cache/ms-playwright`. The `install-browsers` step becomes a no-op when the binary is already present. ### Cache summary table | Cache | Volume name | Path | Key / Invalidation | |-------|-------------|------|--------------------| | Go modules | `go-mod-cache` | `/root/go/pkg/mod` | Content-addressed; no explicit invalidation | | Go build | `go-build-cache` | `/root/.cache/go-build` | Content-addressed; periodic purge | | Flutter pub | `flutter-pub-cache` | `/root/.pub-cache` | `pubspec.lock` change | | Flutter SDK | Baked into Docker image | — | Image rebuild on SDK version bump | | Playwright Chromium | `playwright-chromium` | `/root/.cache/ms-playwright` | `@playwright/test` version change | --- ## 7. Concrete Pipeline Shape The following is a prose description of the `.woodpecker/` YAML structure. This section describes *what to implement*, not the YAML itself (implementation is a follow-up task). ### Files ``` .woodpecker/ ├── server.yml # Go server pipeline: unit, integration, web-smoke ├── android.yml # Flutter Android pipeline: analyze + unit └── nightly.yml # LLM e2e pipeline: nightly cron on main ``` Three separate pipeline files avoid coupling the Go and Flutter pipelines. Each file declares its own trigger conditions and can be enabled/disabled independently. ### `server.yml` trigger conditions - Push to any branch: runs Stage 1 (unit) only. - Push to `main` or PR targeting `main`: runs Stage 1, then Stage 2 (integration + web-smoke) after Stage 1 passes. ### `android.yml` trigger conditions - Push to any branch touching `player-android/**`: runs `flutter analyze` and `flutter test`. - Path-scoped trigger so Go-only changes do not trigger the Flutter runner. ### `nightly.yml` trigger conditions - Woodpecker cron schedule: `0 2 * * *` (02:00 UTC daily, on `main`). - Manual dispatch (Woodpecker's `manual` trigger) for ad-hoc runs. - Protected secret access: LLM API keys only injected on `main`. ### Dependency graph ``` Push to any branch: [server-unit] ─┐ [android-unit] ─┤ (parallel, independent) └─ done Push to main / PR to main: [server-unit] ─────────────────────────────────────────────────┐ [android-unit] ─────────────────────────────────────────────────┤ │ (both pass) ┌─────────────────────┘ [server-integration] ──┐ (parallel) [web-smoke] ────────────┘ Nightly (main only): [server-unit] ──────────────────────────┐ │ (passes) [llm-e2e] (with LLM API keys) ``` --- ## 8. Honest Trade-offs and Risks ### Risk: Codeberg shared runner availability Codeberg's shared Woodpecker runners are a shared pool. Under load, jobs may queue. This is acceptable for a personal/hobby project but could become frustrating during active development sprints. **Mitigation:** The f3s homelab Beelink x86 nodes can run a self-hosted Woodpecker agent. If queue times become a problem, registering one of r0/r1/r2 as a self-hosted runner is a straightforward solution — the runner agent is a single Go binary. ### Risk: Flutter SDK version drift Pinning Flutter SDK via a Docker image means rebuilding the image when Flutter releases a new version. If the image is stale, CI uses an old SDK that may not match local development. **Mitigation:** Use a CI image tagged to a specific Flutter version (e.g., `ghcr.io/cirruslabs/flutter:3.x.y`) and update the tag in the pipeline YAML when bumping Flutter locally. This is a one-line change. ### Risk: Web smoke test flakiness The Playwright suite is configured with `retries: 1` on CI and generous timeouts (15 s navigation, 10 s action). Flakiness can still occur if the Go server takes longer than expected to start or if testdata/media scanning is slow. **Mitigation:** The Playwright `globalSetup` hook already waits for the server to respond before tests run (`waitForServer` helper in `tests/helpers/server.ts`). The server startup step in the pipeline YAML should also include a readiness probe (`curl --retry 10 --retry-delay 1 http://localhost:8080/`) before launching the test runner. ### Risk: LLM e2e cost creep If the nightly LLM e2e suite grows without a cost budget, API spend can accumulate unnoticed. **Mitigation:** The test harness must log total tokens used and estimated cost per run. Add a step that parses this and fails the job if cost exceeds a configured threshold (e.g., $1.00/run). The threshold is a commit-level configuration value, not a secret. --- ## 9. Follow-up Implementation Tasks The following tasks should be created via `ask add` after this proposal is approved. Each is scoped to produce one concrete artifact. 1. **Create `.woodpecker/server.yml`** — Go server pipeline: Stage 1 unit tests on every push; Stage 2 integration + web-smoke on `main` and PRs to `main`. Include Go module and build cache volumes. Include server readiness probe before Playwright. 2. **Create `.woodpecker/android.yml`** — Flutter Android pipeline: `flutter analyze` and `flutter test` on every push touching `player-android/**`. Use a pinned Flutter Docker image. Mount Flutter pub cache volume. 3. **Create `.woodpecker/nightly.yml`** — Nightly LLM e2e stub pipeline: runs `0 2 * * *` on `main`, injects LLM API key from protected Woodpecker secret, currently exits 0 as a placeholder. Add cost logging step. 4. **Build and publish a Flutter CI Docker image** — `Dockerfile.flutter-ci` in the repo root pinning Flutter SDK version; publish to Codeberg Container Registry or GitHub Container Registry. Update `android.yml` to use this image. 5. **Register a self-hosted Woodpecker runner on f3s (optional)** — If shared runner queue times become a problem, register one of the bhyve VMs (r0/r1/r2) as a self-hosted agent. Document the setup in `docs/ci-runner-setup.md`. 6. **Add Playwright Chromium cache volume to `server.yml`** — Named volume for `/root/.cache/ms-playwright`; invalidated by `@playwright/test` version. This is a separate task from task 1 because it requires testing cache hit behaviour.