skills: split f3s into six focused sibling skills (index pattern)

Execute plans/f3s-skill-split-plan.md. Carve the oversized f3s skill into six
f3s-prefixed siblings so each loads on its own triggers, and slim f3s to a hub
(38 reference files -> 10; 119 -> 73 lines):

- f3s-storage       ZFS/zrepl/CARP/NFS-stunnel, nfs-mount-monitor, backups
- f3s-k3s           cluster install, off-LAN access, ingress, etcd, r-node Rex
- f3s-observability Prometheus/Alloy/Loki/Tempo, FreeBSD node_exporter
- f3s-workloads     Immich, Garage, Player, yChat, goprecords/uptimed
- f3s-raspberry-pi  pi0/pi1 NetBSD static site, pi2/pi3 Pi-hole/LAN DNS
- f3s-dtail         dserver deployment/ops (SSH 2222)

f3s hub keeps the master Host-IP table, physical hosts, bhyve, WireGuard mesh,
and off-LAN access as the canonical inward-pointing context, plus a Related
skills block. Applies skill-maintenance best practices: fixes the three inlined
SKILL.md duplications (Pi/webserver, DTail) by moving prose to one canonical home
and keeping each new SKILL.md a slim index that points to its references rather
than re-inlining them. All cross-skill links rewritten to ../../<skill>/references/
form (incl. inbound links from pkgrepo and rocky-vm-setup, and two pre-existing
broken links); verified all 112 relative links resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

5 files changed, 636 insertions, 0 deletions

diff --git a/prompts/skills/f3s-workloads/references/garage.md b/prompts/skills/f3s-workloads/references/garage.md
new file mode 100644
index 0000000..52120df
--- /dev/null
+++ b/prompts/skills/f3s-workloads/references/garage.md
@@ -0,0 +1,158 @@
+# Garage
+
+Garage S3 runs as a 3-node cluster on FreeBSD hosts `f0`, `f1`, and `f2`.
+
+## Topology
+
+- Nodes: `f0.lan.buetow.org`, `f1.lan.buetow.org`, `f2.lan.buetow.org`
+- RPC: `:3901`
+- S3 API: `:3900`
+- Admin/metrics: `:3903`
+- Layout capacity target: `f0=8`, `f1=8`, `f2=4` (same ratio currently applied)
+- Zone: currently all in `dc1`
+- Garage version: `2.2.0` (cargo build)
+
+## Local Data and Service Setup
+
+- Encrypted ZFS datasets created per host:
+  - `zroot/garage/meta` mounted at `/var/db/garage/meta`
+  - `zroot/garage/data` mounted at `/var/db/garage/data`
+- Service enabled:
+  - `garage_enable=YES` in `/etc/rc.conf`
+- Config deployed by repo automation in `f3s/garage/`:
+  - `f3s/garage/Rexfile`
+  - `f3s/garage/Justfile`
+  - `f3s/garage/etc/garage.f0.toml`
+  - `f3s/garage/etc/garage.f1.toml`
+  - `f3s/garage/etc/garage.f2.toml`
+- Shared RPC secret is read from:
+  - `f3s/garage/secrets/rpc_secret` (intentionally gitignored)
+
+## Edge Domain and Frontend Routing
+
+- Public hostname: `garage.f3s.buetow.org`
+- Frontend wiring exists:
+  - Domain included in `frontends/Rexfile` f3s host list
+  - `relayd` backend table and host match added in `frontends/etc/relayd.conf.tpl`
+  - TLS certificate for `garage.f3s.buetow.org` is issued and served
+- Current routing health:
+  - DNS resolves on public edge hosts (`A` + `AAAA`)
+  - HTTPS returns expected S3 XML error for anonymous requests (`403 AccessDenied`)
+  - Authenticated S3 operations via external hostname are working
+
+## Critical Fix Applied
+
+The key issue was that Garage S3/admin listeners were IPv6-only in TOML:
+
+- old: `api_bind_addr = "[::]:3900"` and `"[::]:3903"`
+- fixed: `api_bind_addr = "0.0.0.0:3900"` and `"0.0.0.0:3903"` on `f0/f1/f2`
+
+After redeploy, `fishfinger` and `blowfish` can reach all Garage nodes on WireGuard IPv4:
+
+- `192.168.2.130:3900`
+- `192.168.2.131:3900`
+- `192.168.2.132:3900`
+
+This resolved the external edge path instability.
+
+## Existing Buckets / Keys
+
+- First bucket: `watchos-app`
+- First access key alias: `watchos-key`
+- Bucket permission: read/write granted to `watchos-key`
+- Authenticated external endpoint test is validated (PUT + LIST via `https://garage.f3s.buetow.org`)
+- Do not store key secrets in git; rotate if exposed in logs.
+
+## Prometheus
+
+- Scrape config for Garage admin endpoints was added under:
+  - `f3s/prometheus/additional-scrape-configs.yaml`
+  - `f3s/prometheus/manifests/additional-scrape-configs-secret.yaml`
+- Current state observed: targets present but `down` with `connection refused` on `:3903`.
+
+## Operational Notes (Important)
+
+- Use `ssh -p 22` for `f0/f1/f2` when running ad-hoc commands; default SSH config for `*.buetow.org` may route to a different port and fail for these hosts.
+- Garage 2.2 `node connect` expects `nodeid@host:port` format (not only `host:port`).
+- Ensure `/var/db/garage/meta` and `/var/db/garage/data` ownership allows Garage process access (`garage:garage`).
+- `garage.toml` is installed as `root:garage` mode `640` so service user can read it.
+- `f3s/garage/secrets/rpc_secret` must exist locally before deploy; keep it out of git.
+
+## Recovery Checklist (Public Endpoint Issues)
+
+When `https://garage.f3s.buetow.org` is broken, use this order:
+
+1. Confirm cluster health first (avoid debugging edge when backend is down):
+   - `ssh -p 22 paul@f0.lan.buetow.org 'doas garage status'`
+   - `ssh -p 22 paul@f0.lan.buetow.org 'doas garage stats -a'`
+2. Confirm Garage listeners are on IPv4 on all nodes:
+   - `ssh -p 22 paul@fN.lan.buetow.org 'sockstat -4 -l | grep 3900'`
+   - Expected: `*:3900` (and similarly `*:3903` for admin)
+3. Confirm edge hosts can reach WireGuard backends:
+   - from `fishfinger` and `blowfish`: `nc -zvw2 192.168.2.13{0,1,2} 3900`
+4. Confirm relayd syntax and reload state:
+   - `ssh rex@fishfinger.buetow.org 'doas relayd -n'`
+   - `ssh rex@blowfish.buetow.org 'doas relayd -n'`
+5. Confirm public DNS and TLS:
+   - `host garage.f3s.buetow.org`
+   - `curl -v https://garage.f3s.buetow.org/` (expect XML `403 AccessDenied` for anonymous)
+6. Run authenticated S3 external test:
+   - execute the authenticated PUT/LIST command from this document
+7. If listeners are wrong or config drifted:
+   - fix TOML in `f3s/garage/etc/garage.fN.toml`
+   - redeploy: `just -f f3s/garage/Justfile deploy`
+8. If relay changes were made:
+   - redeploy frontends from `frontends/` (`rex nsd httpd relayd`) and rerun ACME flow if keypair errors appear.
+
+## Useful Commands
+
+### Cluster health
+
+```sh
+ssh -p 22 paul@f0.lan.buetow.org 'doas garage status'
+ssh -p 22 paul@f0.lan.buetow.org 'doas garage stats -a'
+```
+
+### Local S3 check (no public endpoint)
+
+```sh
+ssh -p 22 paul@f0.lan.buetow.org 'curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:3900/'
+```
+
+### External endpoint checks
+
+```sh
+# Anonymous check should return 403 AccessDenied XML (expected)
+ssh rex@fishfinger.buetow.org 'curl -sS -D - https://garage.f3s.buetow.org/ | sed -n "1,20p"'
+
+# Reachability from edge hosts to Garage WG backends
+ssh rex@fishfinger.buetow.org 'for ip in 192.168.2.130 192.168.2.131 192.168.2.132; do nc -zvw2 $ip 3900; done'
+ssh rex@blowfish.buetow.org 'for ip in 192.168.2.130 192.168.2.131 192.168.2.132; do nc -zvw2 $ip 3900; done'
+```
+
+### Bucket and key workflow
+
+```sh
+ssh -p 22 paul@f0.lan.buetow.org 'doas garage bucket create <bucket>'
+ssh -p 22 paul@f0.lan.buetow.org 'doas garage key create <key-alias>'
+ssh -p 22 paul@f0.lan.buetow.org 'doas garage bucket allow <bucket> --read --write --key <key-alias>'
+ssh -p 22 paul@f0.lan.buetow.org 'doas garage key info <key-alias>'
+```
+
+### Authenticated S3 test via external hostname
+
+```sh
+ssh -p 22 paul@f0.lan.buetow.org '
+TMP=$(mktemp)
+doas garage key info watchos-key --show-secret > "$TMP"
+AK=$(awk -F": " "/Key ID:/ {print \$2}" "$TMP" | sed "s/^[[:space:]]*//;s/[[:space:]]*$//")
+SK=$(awk -F": " "/Secret key:/ {print \$2}" "$TMP" | sed "s/^[[:space:]]*//;s/[[:space:]]*$//")
+OBJ="e2e-external-$(date +%s).txt"
+echo "ok" >/tmp/$OBJ
+AWS_ACCESS_KEY_ID="$AK" AWS_SECRET_ACCESS_KEY="$SK" AWS_DEFAULT_REGION=garage \
+  aws --endpoint-url https://garage.f3s.buetow.org s3 cp /tmp/$OBJ s3://watchos-app/$OBJ
+AWS_ACCESS_KEY_ID="$AK" AWS_SECRET_ACCESS_KEY="$SK" AWS_DEFAULT_REGION=garage \
+  aws --endpoint-url https://garage.f3s.buetow.org s3 ls s3://watchos-app/$OBJ
+rm -f "$TMP" /tmp/$OBJ
+'
+```
diff --git a/prompts/skills/f3s-workloads/references/goprecords-uptimed.md b/prompts/skills/f3s-workloads/references/goprecords-uptimed.md
new file mode 100644
index 0000000..4ae4297
--- /dev/null
+++ b/prompts/skills/f3s-workloads/references/goprecords-uptimed.md
@@ -0,0 +1,130 @@
+# Uptimed / uprecords collection via goprecords
+
+Central uptime stats come from **`uptimed`** record files aggregated by **[goprecords](https://codeberg.org/snonux/goprecords)**. The live API is **`https://goprecords.f3s.buetow.org`** (k3s **services** namespace; stats PVC; auth DB **`goprecords-auth.db`**).
+
+## Daemon and keys
+
+- **Read API:** `GET /report` (Plaintext, Markdown, Gemtext, HTML).
+- **Upload API:** `PUT /upload/{HOSTNAME}/{kind}` with kinds `records`, `txt`, `cur.txt`, `os.txt`, `cpuinfo.txt`.
+- **Keys:** issued only on the server, e.g.  
+  `kubectl exec -n services deployment/goprecords -- goprecords --create-client-key HOST -stats-dir=/data/stats`  
+  Re-issuing replaces the previous token; update every client that uses that host name.
+
+The **`HOSTNAME`** in the URL must match the name passed to **`--create-client-key`**. Use stable short names (**`f0`**, **`pi2`**, **`fishfinger`**, …) consistent with stats file basenames. That short name is only for **`GOPRECORDS_HOST`** and upload URLs — not for SSH.
+
+## SSH / DNS (f3s)
+
+- **`f0.lan` is not a hostname** (it will not resolve). Use the **full** name **`f0.lan.buetow.org`**, or the **LAN IP** from the f3s table (**`192.168.1.130`** for **f0**, **`.131`–`.133`** for **f1**–**f3**).
+- **FreeBSD Beelinks and Pis:** **`ssh -p 22 paul@…`** (default SSH port). Your **`~/.ssh/config`** may use **port 2** for **OpenBSD frontends** only — that does **not** apply to **f0**–**f3** or **pi0**–**pi3**.
+- **Pis:** **`pi0.lan.buetow.org`** … **`pi3.lan.buetow.org`** (full FQDN), port **22**.
+- **Manual upload test over SSH:** Beelinks use **`doas env GOPRECORDS_HOST=fN /usr/local/bin/goprecords-upload-client.sh`** (not **`sudo`** — often absent). Rocky Pis use **`sudo env GOPRECORDS_HOST=piN …`**. NetBSD Pis use **`doas env PATH=/usr/pkg/bin:/usr/bin:/bin:/usr/sbin:/sbin GOPRECORDS_HOST=piN /usr/pkg/bin/goprecords-upload-client.sh`**; the explicit path is required so the script can find pkgsrc `curl` and `uprecords` under `doas`.
+
+## Where it is documented in-repo
+
+In **goprecords** **`README.md`**:
+
+- HTTP API and upload **`curl`** examples
+- **“Setting up a new upload client”** (generic)
+- **“Manual hourly upload (single host, not config-managed)”** — POSIX script **`contrib/goprecords-upload-client.sh`**, **FreeBSD** hourly **`cron`** (with **`PATH`**), **Linux** **`systemd`** **`oneshot` + `timer`**
+
+Install **`curl`** and **`uptimed`** on every client that uploads.
+
+## By host class (f3s)
+
+| Class | Hosts | Automation | Notes |
+|--------|--------|------------|--------|
+| OpenBSD frontends | **fishfinger**, **blowfish** | **Rex** **`goprecords_upload`** in **`~/git/conf/frontends`**; **`/etc/daily.local`** runs **`/usr/local/bin/goprecords-upload.sh`** once per **day** | Tokens in **geheim** **`secrets/etc/goprecords/<host>.token`**; template **`scripts/goprecords-upload.sh.tpl`** |
+| FreeBSD (Beelinks) | **f0**–**f3** (LAN **`192.168.1.130`–`133`**) | Manual **hourly** **root** **`cron`** calling **`goprecords-upload-client.sh`** with **`GOPRECORDS_HOST=f0`** … **`f3`** | **`/var/db/uptimed/records`**; SSH: **`fN.lan.buetow.org`** or **`192.168.1.(130+N)`** for **fN**, **`-p 22`** |
+| Raspberry Pi (Rocky) | **pi2**–**pi3** | Manual **hourly** **systemd** **timer** (see README) | **`/var/spool/uptimed/records`**; uptimed waits for chronyd via a systemd override (see below); SSH: **`piN.lan.buetow.org`**, **`-p 22`** |
+| Raspberry Pi (NetBSD) | **pi0**–**pi1** | Manual **hourly** **root** **`cron`** (no systemd) calling **`goprecords-upload-client.sh`** with **`GOPRECORDS_HOST=pi0`**/**`pi1`** | **`/var/spool/uptimed/records`**; `ntpdate=YES` and uptimed requires the `ntpdate` rc.d milestone; see [NetBSD Pi setup](../../f3s-raspberry-pi/references/bootstrap-netbsd-pi.md#uptimed-built-from-source--no-prebuilt-package); SSH: **`piN.lan.buetow.org`**, **`-p 22`** |
+| Fedora laptop | **earth** | **user** **systemd** **`oneshot` + hourly timer** `goprecords-upload-earth.{service,timer}` | Service sets **`Environment=GOPRECORDS_HOST=earth`** and runs **`~/.local/bin/goprecords-upload-earth.sh`**; token **`~/.config/goprecords-upload-earth/token`** |
+| Mac (uptimed) → published by earth | **mega-m3-pro** (raw host `MBDVXJ4XKH9C`) | Mac drops records into the **worktime** git repo; **earth** pushes them via a **second `ExecStart`** in `goprecords-upload-earth.service` | See [Mac / mega-m3-pro via earth](#mac--mega-m3-pro-via-earth) below |
+
+## OpenBSD frontends (Rex)
+
+From **`~/git/conf/frontends`**:
+
+```bash
+rex goprecords_upload
+# or full commons
+rex commons
+```
+
+See **`frontends/README.md`** (section **goprecords upload**).
+
+## Manual clients (FreeBSD + Pis + earth)
+
+The canonical unified script is **`scripts/goprecords-upload-client.sh`** (also mirrored in **`contrib/`**). It is POSIX sh and works on all host types:
+
+- **root** (FreeBSD/Linux): token at **`/etc/goprecords-upload.token`** (**`0600`**)
+- **non-root** (earth user session): token at **`$XDG_CONFIG_HOME/goprecords-upload-<HOST>/token`**
+
+Copy to **`/usr/local/bin/`** (system) or **`~/.local/bin/`** (user), set **`GOPRECORDS_HOST`** per machine (**cron** **`env`** or **`systemd`** **`Environment`**/**`EnvironmentFile`**). Full snippets: **goprecords** **`README.md`**.
+
+> **earth gotcha:** `goprecords-upload-earth.sh` is a *copy of the generic* `goprecords-upload-client.sh`, so it aborts with `set GOPRECORDS_HOST` unless the var is provided. The service therefore **must** carry `Environment=GOPRECORDS_HOST=earth`. (A missing env var silently broke earth uploads for ~a month — symptom: stale timestamp on the report, service `status=1/FAILURE` with `set GOPRECORDS_HOST` in `journalctl --user -u goprecords-upload-earth`.)
+
+## Rocky Pi uptimed clock synchronization
+
+**pi2** and **pi3** have no hardware RTC. At boot their clocks initially use a stale timestamp until chronyd synchronizes with NTP. The packaged `uptimed.service` only declares `After=time-sync.target`; that target does not guarantee chronyd has obtained valid time. If uptimed starts too early, it records the stale boot date and goprecords may omit the host's `*` active marker because activity is calculated from the newest record's boot time plus uptime, not from upload time.
+
+Both Pis have this override at **`/etc/systemd/system/uptimed.service.d/time-sync.conf`**:
+
+```ini
+[Unit]
+Wants=network-online.target chronyd.service
+After=network-online.target chronyd.service
+
+[Service]
+ExecStartPre=/usr/bin/chronyc waitsync 60 0.5
+```
+
+Apply or verify it with:
+
+```sh
+sudo systemctl daemon-reload
+sudo systemd-analyze verify uptimed.service
+sudo systemctl restart uptimed
+systemctl show uptimed -p ExecStartPre -p ExecMainStartTimestamp
+```
+
+`chronyc waitsync 60 0.5` allows up to 60 attempts and starts uptimed once the remaining clock correction is at most 0.5 seconds. It works as the service's unprivileged `daemon` user.
+
+If a Pi already has a stale current record, restart uptimed after NTP is synchronized, then run the normal upload immediately:
+
+```sh
+sudo systemctl restart uptimed
+sudo systemctl start goprecords-upload.service
+```
+
+Check the source and uploaded records with `uprecords -a`, `/var/spool/uptimed/records`, and the goprecords `LastUpdated` report. This repaired pi3 on **2026-07-16**: its uploader had been healthy, but its only stored boot timestamp was from **2025-12-02**; restarting uptimed added the correct **2026-06-25** boot timestamp and restored the `*` marker. A subsequent pi3 reboot verified the override: chronyd stepped the stale clock at **13:00:57 UTC**, `chronyc waitsync` succeeded, and uptimed started afterward at **13:01:03 UTC** with the correct current boot date.
+
+## Mac / mega-m3-pro via earth
+
+The Mac (Apple Silicon, **`Darwin`**) is **not** a direct upload client. Instead:
+
+The logic lives in a **fish helper kept in the (private) worktime repo**, **`~/git/worktime/scripts/uprecords-sync.fish`** (functions `worktime::uprecords::darwin::collect` / `…::import`), so host-specific details stay out of the public dotfiles repo. `dotfiles/fish/conf.d/worktime.fish` **`source`**s it, and **`worktime::supersync`** calls both functions.
+
+1. On the Mac, **`worktime::uprecords::darwin::collect`** copies the local **uptimed** records into the **worktime** git repo as
+   **`uprecords-MBDVXJ4XKH9C.records`** and **`uprecords-MBDVXJ4XKH9C.txt`**, and they get synced via `git` (part of **`worktime::supersync`**). Guards on `uname = Darwin`.
+2. On **earth** (the only host that publishes), **`worktime::uprecords::darwin::import`** reads those repo files and **`PUT`**s them to goprecords, **re-labelling** the raw host `MBDVXJ4XKH9C` → **`mega-m3-pro`**:
+   - `PUT /upload/mega-m3-pro/records` and `PUT /upload/mega-m3-pro/txt`
+   - token at **`~/.config/goprecords-upload-mega-m3-pro/token`** (`0600`).
+   - Guards on `hostname = earth`; exits cleanly (warning) if the token is missing.
+3. Automation: `goprecords-upload-earth.service` has a **second `ExecStart`** that runs the import hourly alongside earth's own upload:
+   ```ini
+   ExecStart=%h/.local/bin/goprecords-upload-earth.sh
+   ExecStart=/usr/local/sbin/fish -c worktime::uprecords::darwin::import
+   ```
+   The import is a **no-op off earth** (guards on `hostname = earth`) and exits cleanly with a warning if the `mega-m3-pro` token is absent, so it never fails the service.
+
+**Token note:** each token is bound to its host name server-side — the **earth** token returns **403** for `mega-m3-pro`. Issue a dedicated key (see *Daemon and keys* above):
+```sh
+kubectl exec -n services deployment/goprecords -- \
+  goprecords --create-client-key mega-m3-pro -stats-dir=/data/stats
+```
+Re-issuing **replaces** any previous `mega-m3-pro` token (the Mac used to upload directly until it switched to the repo route). When roaming, reach the cluster via the OpenBSD frontend jump (see [k3s remote access](../../f3s-k3s/references/remote-access.md)), then store the printed token in `~/.config/goprecords-upload-mega-m3-pro/token` on earth.
+
+## Related conf repo paths
+
+- Kubernetes Helm: **`conf/f3s/goprecords/`** (image, PVC, ingress **`goprecords.f3s.buetow.org`**)
+- OpenBSD Rex: **`conf/frontends/`** (**`Rexfile`**, **`scripts/goprecords-upload.sh.tpl`**)
diff --git a/prompts/skills/f3s-workloads/references/immich.md b/prompts/skills/f3s-workloads/references/immich.md
new file mode 100644
index 0000000..543fb8d
--- /dev/null
+++ b/prompts/skills/f3s-workloads/references/immich.md
@@ -0,0 +1,98 @@
+# Immich
+
+Immich runs in the `services` namespace. Config is in `f3s/immich/`.
+
+## Components
+
+- `immich-server` — main API and web UI (port 2283)
+- `immich-machine-learning` — ML inference for face detection, smart search, OCR (port 3003)
+- `immich-postgres` — PostgreSQL 16 with pgvecto-rs extension
+- `immich-valkey` — Redis-compatible queue backend (BullMQ)
+
+## Gathering Job Queue Stats
+
+Immich uses BullMQ via Valkey. To snapshot current queue counters:
+
+```sh
+kubectl exec -n services deploy/immich-valkey -- sh -c '
+for queue in thumbnailGeneration metadataExtraction videoConversion faceDetection smartSearch duplicateDetection backgroundTask storageTemplateMigration search sidecar library notification ocr migration; do
+  waiting=$(valkey-cli LLEN "immich_bull:${queue}:wait" 2>/dev/null)
+  active=$(valkey-cli LLEN "immich_bull:${queue}:active" 2>/dev/null)
+  delayed=$(valkey-cli ZCARD "immich_bull:${queue}:delayed" 2>/dev/null)
+  completed=$(valkey-cli ZCARD "immich_bull:${queue}:completed" 2>/dev/null)
+  failed=$(valkey-cli ZCARD "immich_bull:${queue}:failed" 2>/dev/null)
+  echo "${queue}: waiting=${waiting} active=${active} delayed=${delayed} completed=${completed} failed=${failed}"
+done
+'
+```
+
+## Saving and Comparing Snapshots
+
+Save a snapshot to the conf repo (persistent, not `/tmp`):
+
+```sh
+kubectl exec -n services deploy/immich-valkey -- sh -c '...' > ~/git/conf/f3s/immich/snapshots/immich-queues-$(date +%Y%m%d-%H%M%S).txt
+```
+
+To compare a previous snapshot with current state, re-run the command and diff:
+
+```sh
+diff ~/git/conf/f3s/immich/snapshots/immich-queues-<old>.txt ~/git/conf/f3s/immich/snapshots/immich-queues-<new>.txt
+```
+
+Decreasing `waiting` and stable/zero `failed` means healthy progress.
+
+### Always Check for Progress
+
+When gathering new stats, **always compare against the most recent saved snapshot** (check `~/git/conf/f3s/immich/snapshots/`). If a queue's `waiting` count has not decreased since the last snapshot, the queue is likely stuck — investigate immediately (see "Stuck job queue" in Troubleshooting below).
+
+## Job Control via API
+
+The API key is stored at `~/.immich_paul_key`. Use it to pause/resume jobs:
+
+```sh
+API_KEY=$(cat ~/.immich_paul_key)
+
+# Get all job statuses
+kubectl exec -n services deploy/immich-server -- curl -s \
+  -H "x-api-key: $API_KEY" http://localhost:2283/api/jobs
+
+# Pause a job (e.g. faceDetection)
+kubectl exec -n services deploy/immich-server -- curl -s -X PUT \
+  -H "x-api-key: $API_KEY" -H "Content-Type: application/json" \
+  -d '{"command":"pause","force":false}' \
+  http://localhost:2283/api/jobs/faceDetection
+
+# Resume a job
+kubectl exec -n services deploy/immich-server -- curl -s -X PUT \
+  -H "x-api-key: $API_KEY" -H "Content-Type: application/json" \
+  -d '{"command":"resume","force":false}' \
+  http://localhost:2283/api/jobs/faceDetection
+```
+
+### Throughput Optimization Strategy
+
+On the N100 (4-core) nodes, ML jobs compete for CPU. To speed up slow queues:
+
+1. **Pause faceDetection** (largest queue) to free CPU for OCR and smartSearch
+2. Resume faceDetection once OCR and smartSearch finish
+3. The anti-affinity in `values.yaml` prefers ML on a different node than both server and postgres
+
+## Troubleshooting
+
+- **Stuck job queue**: If a queue has `waiting` jobs but no progress since the last snapshot:
+  1. Check ML pod logs for activity: `kubectl logs -n services deploy/immich-machine-learning --tail=30`. Look for "Shutting down due to inactivity" — this means jobs are not being dispatched.
+  2. Check server/microservices logs: `kubectl logs -n services deploy/immich-server --tail=30`. If there's no job processing output (only version checks and websocket events), the worker is stuck.
+  3. A stale `active` job in Valkey can block the entire queue. Clear it:
+     ```sh
+     kubectl exec -n services deploy/immich-valkey -- valkey-cli DEL "immich_bull:<queue>:active"
+     ```
+  4. If clearing the stale job doesn't help, **restart the server deployment** — this is the most reliable fix:
+     ```sh
+     kubectl rollout restart deploy/immich-server -n services
+     kubectl rollout status deploy/immich-server -n services --timeout=120s
+     ```
+  5. After restart, wait ~20 seconds, then verify via the API that `isActive: true` and `waiting` is decreasing.
+- **Postgres crash loop**: Usually caused by liveness probe killing postgres during WAL recovery. Check `kubectl describe pod` for probe failures and postgres logs for "database system was interrupted while in recovery". Fix by relaxing probe timeouts/thresholds and adding resource limits.
+- **Server crash loop**: Often caused by postgres being unavailable. Fix postgres first.
+- **ML errors**: "Machine learning repository not been setup" is transient — resolves once the ML pod health check passes.
diff --git a/prompts/skills/f3s-workloads/references/player.md b/prompts/skills/f3s-workloads/references/player.md
new file mode 100644
index 0000000..2719792
--- /dev/null
+++ b/prompts/skills/f3s-workloads/references/player.md
@@ -0,0 +1,167 @@
+# Player Deployment
+
+Player is deployed on the f3s k3s cluster as a GitOps-managed service.
+
+## Repositories and paths
+
+- App source: `~/git/player`
+- f3s config source: `~/git/conf`
+- Helm chart: `~/git/conf/f3s/player/helm-chart`
+- ArgoCD app: `~/git/conf/f3s/argocd-apps/services/player.yaml`
+- External URL: `https://player.f3s.buetow.org`
+- Extra instance URL: `https://xplayer.f3s.buetow.org`
+- LAN URL: `https://player.f3s.lan.buetow.org`
+
+ArgoCD reads the chart from the in-cluster git-server repo:
+
+```sh
+http://git-server.cicd.svc.cluster.local/conf.git
+path: f3s/player/helm-chart
+```
+
+The secondary `xplayer` instance is managed by a separate ArgoCD app:
+
+```sh
+http://git-server.cicd.svc.cluster.local/conf.git
+path: f3s/xplayer/helm-chart
+```
+
+Keep `~/git/conf` pushed to both remotes after chart updates:
+
+```sh
+git push master master
+git push r0 master
+```
+
+## Build and push a new image
+
+Use the app git commit SHA as the immutable image tag.
+
+```sh
+cd ~/git/player
+go test ./...
+
+TAG=$(git rev-parse --short HEAD)
+podman build -t player:$TAG -t player:latest .
+podman tag player:$TAG r0.lan.buetow.org:30001/player:$TAG
+podman tag player:latest r0.lan.buetow.org:30001/player:latest
+podman push --tls-verify=false r0.lan.buetow.org:30001/player:$TAG
+podman push --tls-verify=false r0.lan.buetow.org:30001/player:latest
+```
+
+The registry is the f3s private registry on NodePort `30001` and is plain HTTP/insecure. In Kubernetes manifests, pods pull the image as:
+
+```text
+registry.lan.buetow.org:30001/player:<TAG>
+```
+
+The app must not run as root. The Dockerfile runtime stage uses `USER 65534:65534`, and the chart should keep:
+
+```yaml
+runAsNonRoot: true
+runAsUser: 65534
+runAsGroup: 65534
+fsGroup: 65534
+```
+
+## Update Helm and ArgoCD
+
+Update the same image tag in both Helm charts:
+
+`~/git/conf/f3s/player/helm-chart`:
+
+- `Chart.yaml`: `appVersion: "<TAG>"`
+- `templates/deployment.yaml`: `image: registry.lan.buetow.org:30001/player:<TAG>`
+
+`~/git/conf/f3s/xplayer/helm-chart`:
+
+- `Chart.yaml`: `appVersion: "<TAG>"`
+- `templates/deployment.yaml`: `image: registry.lan.buetow.org:30001/player:<TAG>`
+
+Validate locally:
+
+```sh
+cd ~/git/conf
+helm template player f3s/player/helm-chart >/tmp/player-helm-render.yaml
+helm template xplayer f3s/xplayer/helm-chart >/tmp/xplayer-helm-render.yaml
+kubectl apply --dry-run=client -f /tmp/player-helm-render.yaml
+kubectl apply --dry-run=client -f /tmp/xplayer-helm-render.yaml
+```
+
+Commit and push:
+
+```sh
+git add f3s/player/helm-chart f3s/xplayer/helm-chart
+git commit -m "Update player image tags"
+git push master master
+git push r0 master
+```
+
+Refresh ArgoCD and wait for rollout:
+
+```sh
+kubectl annotate application player -n cicd argocd.argoproj.io/refresh=normal --overwrite
+kubectl annotate application xplayer -n cicd argocd.argoproj.io/refresh=normal --overwrite
+kubectl rollout status deployment/player -n services --timeout=180s
+kubectl rollout status deployment/xplayer -n services --timeout=180s
+kubectl get application player -n cicd -o jsonpath='sync={.status.sync.status} health={.status.health.status} revision={.status.sync.revision}{"\n"}'
+kubectl get application xplayer -n cicd -o jsonpath='sync={.status.sync.status} health={.status.health.status} revision={.status.sync.revision}{"\n"}'
+```
+
+## Storage notes
+
+Player uses two static `hostPath` PVs that point at the NFS mount available on every k3s node:
+
+- `/data/nfs/k3svolumes/player/data` mounted at `/data`
+- `/data/nfs/k3svolumes/player/media` mounted at `/media`
+
+The `xplayer` instance uses separate static `hostPath` PVs under:
+
+- `/data/nfs/k3svolumes/xplayer/data` mounted at `/data`
+- `/data/nfs/k3svolumes/xplayer/media` mounted at `/media`
+
+The PVs must use:
+
+```yaml
+hostPath:
+  type: Directory
+```
+
+Do not change them to `DirectoryOrCreate`. `Directory` makes pod startup fail if the final path is missing, which helps avoid accidentally creating player data on a node when the intended NFS-backed path is unavailable.
+
+Create the paths before first deploy:
+
+```sh
+ssh -p 22 root@192.168.1.120 'mkdir -p /data/nfs/k3svolumes/player/{data,media}'
+ssh -p 22 root@192.168.1.120 'mkdir -p /data/nfs/k3svolumes/xplayer/{data,media}'
+```
+
+The NFS export may reject `chown` to UID 65534. Existing f3s writable service directories often use mode `777` when ownership cannot be changed:
+
+```sh
+ssh -p 22 root@192.168.1.120 'chmod 777 /data/nfs/k3svolumes/player /data/nfs/k3svolumes/player/data /data/nfs/k3svolumes/player/media'
+ssh -p 22 root@192.168.1.120 'chmod 777 /data/nfs/k3svolumes/xplayer /data/nfs/k3svolumes/xplayer/data /data/nfs/k3svolumes/xplayer/media'
+```
+
+## Verification
+
+```sh
+kubectl get pods,pvc,svc,ingress -n services | grep player
+kubectl logs -n services deploy/player --tail=100
+kubectl logs -n services deploy/xplayer --tail=100
+curl -fsS https://player.f3s.buetow.org/healthz
+curl -fsS https://xplayer.f3s.buetow.org/healthz
+curl -kfsS https://player.f3s.lan.buetow.org/healthz
+curl -kfsS https://player.f3s.lan.buetow.org/readyz
+```
+
+Verify the runtime UID and NFS write access:
+
+```sh
+POD=$(kubectl get pod -n services -l app=player -o jsonpath='{.items[0].metadata.name}')
+kubectl exec -n services "$POD" -- id
+kubectl exec -n services "$POD" -- sh -c 'touch /data/.write-test /media/.write-test && rm /data/.write-test /media/.write-test'
+XPOD=$(kubectl get pod -n services -l app=xplayer -o jsonpath='{.items[0].metadata.name}')
+kubectl exec -n services "$XPOD" -- id
+kubectl exec -n services "$XPOD" -- sh -c 'touch /data/.write-test /media/.write-test && rm /data/.write-test /media/.write-test'
+```
diff --git a/prompts/skills/f3s-workloads/references/ychat.md b/prompts/skills/f3s-workloads/references/ychat.md
new file mode 100644
index 0000000..98dc066
--- /dev/null
+++ b/prompts/skills/f3s-workloads/references/ychat.md
@@ -0,0 +1,83 @@
+# yChat Deployment
+
+yChat is a legacy (2007) C++ HTTP web chat server, revived to build in Docker
+with a mandatory embedded-SQLite backend. It is deployed on the f3s k3s
+cluster as a GitOps-managed service.
+
+> **Deployed.** The live LAN URL **https://ychat.f3s.lan.buetow.org/** serves
+> image tag `67babb2` (the DB-backed build), with a persistent volume
+> (`ychat-data-pvc`, hostPath-backed NFS share) mounted at `/app/data`, so
+> registered accounts survive pod restarts. The no-DB build that previously
+> ran live has been retired.
+
+This reference is the single home for f3s-specific deployment details. The
+public app repo (`ychat` on https://codeberg.org/snonux/ychat) deliberately
+keeps deployment/cluster specifics **out of scope** — everything below lives
+here instead.
+
+## Repositories and paths
+
+- App source: `~/git/ychat` (subproject `ychat/`; source on
+  https://codeberg.org/snonux/ychat)
+- f3s config source: `~/git/conf` (mirrored on the in-cluster git-server;
+  https://codeberg.org/snonux/conf)
+- Helm chart: `f3s/ychat/helm-chart`
+- ArgoCD app: `f3s/argocd-apps/services/ychat.yaml`
+- LAN URL: `https://ychat.f3s.lan.buetow.org/`
+
+## Build and push a new image
+
+Use the app git commit SHA as the immutable image tag. Build is a multi-stage
+`Dockerfile` (Rocky Linux 9 builder + slim Rocky 9 runtime) that compiles
+ychat entirely inside the container.
+
+```sh
+cd ~/git/ychat/ychat
+podman build -t ychat:dev .
+
+TAG=$(git rev-parse --short HEAD)
+podman tag ychat:$TAG r0.lan.buetow.org:30001/ychat:$TAG
+podman tag ychat:latest r0.lan.buetow.org:30001/ychat:latest
+podman push --tls-verify=false r0.lan.buetow.org:30001/ychat:$TAG
+podman push --tls-verify=false r0.lan.buetow.org:30001/ychat:latest
+```
+
+The registry is the f3s private registry on NodePort `30001` (plain
+HTTP/insecure). In Kubernetes manifests, pods pull the image as:
+
+```text
+registry.lan.buetow.org:30001/ychat:<TAG>
+```
+
+## Deploy (GitOps)
+
+Config lives in the `conf` repo (mirrored on the in-cluster git-server):
+
+- Helm chart: `f3s/ychat/helm-chart`
+- ArgoCD app: `f3s/argocd-apps/services/ychat.yaml`
+
+The Deployment pulls `registry.lan.buetow.org:30001/ychat:<TAG>` (tag matches
+`appVersion` in `Chart.yaml`). The default chat port is **2000**.
+
+## Storage notes
+
+- **Logs** (`/app/log`: `access_log`, `system_log`, `rooms/<room>`) go to an
+  `emptyDir` — ephemeral by design.
+- **SQLite database** (`/app/data/ychat.db`) holds registered accounts and
+  is backed by the `ychat-data-pvc` persistent volume (hostPath-backed NFS
+  share, mounted at `/app/data`), so accounts survive pod rescheduling.
+
+Only registered accounts persist (in SQLite). Sessions/rooms/online-state are
+in-memory, and unregistered `chat.enableguest=true` guest chatters are wiped
+on restart — only the accounts table persists.
+
+## Runtime config notes
+
+- Configuration is `ychat/etc/ychat.conf`, baked into the image at
+  `/app/etc/ychat.conf`. Any config key can be overridden at runtime with
+  `-o <key> <value>` (the image already does this for
+  `chat.session.md5hash=false` and `chat.database.dbname=data/ychat.db`).
+- The `/exec` command module is removed from the image entirely
+  (defense-in-depth against its shell-injection RCE), and operator status
+  via `chat.defaultop` requires a database-authenticated registered account —
+  an unregistered guest can never claim it.
\ No newline at end of file