skills: split f3s into six focused sibling skills (index pattern)

Execute plans/f3s-skill-split-plan.md. Carve the oversized f3s skill into six
f3s-prefixed siblings so each loads on its own triggers, and slim f3s to a hub
(38 reference files -> 10; 119 -> 73 lines):

- f3s-storage       ZFS/zrepl/CARP/NFS-stunnel, nfs-mount-monitor, backups
- f3s-k3s           cluster install, off-LAN access, ingress, etcd, r-node Rex
- f3s-observability Prometheus/Alloy/Loki/Tempo, FreeBSD node_exporter
- f3s-workloads     Immich, Garage, Player, yChat, goprecords/uptimed
- f3s-raspberry-pi  pi0/pi1 NetBSD static site, pi2/pi3 Pi-hole/LAN DNS
- f3s-dtail         dserver deployment/ops (SSH 2222)

f3s hub keeps the master Host-IP table, physical hosts, bhyve, WireGuard mesh,
and off-LAN access as the canonical inward-pointing context, plus a Related
skills block. Applies skill-maintenance best practices: fixes the three inlined
SKILL.md duplications (Pi/webserver, DTail) by moving prose to one canonical home
and keeping each new SKILL.md a slim index that points to its references rather
than re-inlining them. All cross-skill links rewritten to ../../<skill>/references/
form (incl. inbound links from pkgrepo and rocky-vm-setup, and two pre-existing
broken links); verified all 112 relative links resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2 files changed, 382 insertions, 0 deletions

diff --git a/prompts/skills/f3s-raspberry-pi/references/bootstrap-netbsd-pi.md b/prompts/skills/f3s-raspberry-pi/references/bootstrap-netbsd-pi.md
new file mode 100644
index 0000000..44d4b58
--- /dev/null
+++ b/prompts/skills/f3s-raspberry-pi/references/bootstrap-netbsd-pi.md
@@ -0,0 +1,337 @@
+# NetBSD services on pi0/pi1
+
+`pi0` and `pi1` run NetBSD 10.1 (evbarm-aarch64). This documents how their
+services are installed and configured — useful reference for troubleshooting,
+rebuilding a service, or reinstalling either node.
+
+**Do this one node at a time.** Never take down both of `pi0`/`pi1` (the
+static-HTTP pair) simultaneously — one must always keep serving
+`f3s.buetow.org`/`snonux.foo`.
+
+## Base state
+
+- NetBSD 10.1 `GENERIC64` evbarm64 (aarch64)
+- User `paul`, in group `wheel`, SSH key auth
+- Static LAN IP via `rc.conf` (`ifconfig_mue0="inet 192.168.1.12N netmask
+  0xffffff00"`, `defaultroute="192.168.1.1"`)
+- Hostname set (`hostname="piN.lan.buetow.org"`)
+- No `doas`/`sudo`, no pkgsrc/pkgin by default — bootstrapped below
+- A pre-baked root crontab entry for the hourly goprecords upload already
+  points at `/usr/pkg/bin/goprecords-upload-client.sh` with `GOPRECORDS_HOST`
+  set correctly — check `doas crontab -l` before deploying that script
+  manually to a different path.
+
+## Bootstrap pkgin + real doas
+
+```sh
+ssh paul@piN.lan.buetow.org
+su -
+export PKG_PATH=https://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/aarch64/10.1/All/
+pkg_add -v pkgin
+pkgin -y update
+pkgin -y install doas rsync curl
+printf 'permit nopass :wheel\n' > /usr/pkg/etc/doas.conf   # NOT "permit persist" --
+                                                             # that still prompts once
+                                                             # per session, which never
+                                                             # succeeds over a
+                                                             # non-interactive SSH
+                                                             # command (no tty)
+chmod 644 /usr/pkg/etc/doas.conf
+exit   # back to paul
+doas true   # should succeed with no password prompt
+```
+
+**Why real `doas`, not the Rocky pattern**: `pi2`–`pi3` only alias `doas` to
+`sudo` via `/etc/profile.d/doas.sh`, which doesn't expand in the
+non-interactive shell an SSH command runs in — so
+`~/git/dotfiles/scripts/wol-f3s`'s `shutdown-pis`/`shutdown-all` (which runs
+`ssh paul@pi "doas poweroff"`) is silently broken on the Rocky Pis today. A
+real `doas` binary is why it works on `pi0`/`pi1`.
+
+**Gotcha**: commands run via `doas` get a minimal `PATH` that excludes
+`/usr/sbin` and `/usr/pkg/bin` — always use full paths (`doas
+/usr/sbin/chown`, `doas /usr/pkg/bin/wg`) or an explicit `PATH=` for cron.
+
+## WireGuard (userspace — no native `wg(4)` on this platform)
+
+**`wg(4)` doesn't exist on evbarm-aarch64 10.1** — the module is absent from
+all 249 files under `/stand/evbarm/10.1/modules`, so `ifconfig wg0 create`
+fails outright (`clone_command: Invalid argument`), despite `wg(4)` being
+upstream NetBSD since 9.2. Don't waste time on it; `wireguard-go` + `wg`
+(pkgsrc `wireguard-tools`, **no `wg-quick`** in this build) is the working
+path:
+
+```sh
+pkgin -y install wireguard-go wireguard-tools
+```
+
+**`wireguardmeshgenerator` (`~/git/wireguardmeshgenerator`) has full NetBSD
+support** — for a host with `os: NetBSD` in `wireguardmeshgenerator.yaml`,
+`--generate` produces both the stripped `tun0.conf` (no `Address`/`DNS` lines
+— `wg setconf` rejects wg-quick extensions with "Line unrecognized") and the
+`/etc/rc.d/wireguard` script itself, with one `route add`/`delete` pair per
+peer AllowedIPs prefix (wg-quick would normally manage these automatically;
+`wg` only does the crypto/routing decision inside the tunnel). `--install`
+uploads and places both files with the right ownership/permissions and
+restarts the service. Per-host YAML fields that matter for a NetBSD entry:
+
+```yaml
+pi0:
+  os: NetBSD
+  ssh:
+    user: paul
+    conf_dir: /usr/pkg/etc/wireguard
+    sudo_cmd: doas
+    reload_cmd: /etc/rc.d/wireguard restart
+    wg_bin: /usr/pkg/bin/wg   # doas's PATH excludes /usr/pkg/bin
+```
+
+Key facts the generator's implementation encodes:
+
+- The interface **must** be named `tunN` (`wireguard-go` rejects `wg0`:
+  "Interface name must be tun[0-9]*"). The generator always uses `tun0`.
+- **The interface must be addressed before `wireguard-go` starts**, or its
+  read loop dies immediately with `EHOSTDOWN` ("host is down") and does not
+  retry — the generated rc.d script's `wireguard_start` does `ifconfig`
+  before `wireguard-go`.
+- `doas` on NetBSD resets `PATH` to exclude both `/usr/pkg/bin` (hence the
+  `wg_bin` override above) and `/usr/sbin` (hence the generator using a full
+  path for `chown` during install — caught by actually running `--install`
+  against a live host, not by inspection).
+
+To (re)deploy after any topology change: `ruby wireguardmeshgenerator.rb
+--generate --install --hosts=pi0,pi1`.
+
+## Webserver — bozohttpd
+
+Built into NetBSD base, no package or config file. No stock rc.d exists that
+actually uses `httpd_flags` (the shipped `/etc/rc.d/httpd` computes
+`command_args` itself and never references that variable) — write a
+dedicated `/etc/rc.d/bozohttpd`:
+
+```sh
+command="/usr/libexec/httpd"
+pidfile="/var/run/bozohttpd.pid"
+command_args="-b -X -U _httpd -P ${pidfile} -v /var/www/html -V /var/www/html"
+required_dirs="/var/www/html"
+```
+
+- `-v /var/www/html -V /var/www/html`: vhost directory = same tree as the
+  default docroot. A `Host:` header matching a **literally-named**
+  subdirectory (e.g. `snonux.foo/`) is served from there; anything unmatched
+  falls back to the plain docroot via `-V`.
+- `www.snonux.foo` needs to be a symlink to `snonux.foo` (bozohttpd matches
+  the literal Host header as a directory name, not a regex like lighttpd's
+  `$HTTP["host"] =~ "^(www\.)?snonux\.foo$"`).
+- **`-X` (directory indexing) is required**, not optional: bare directories
+  with no `index.html` (e.g. a photo gallery folder under `/fotos/`) 404
+  without it.
+- **Give every real routed hostname its own vhost entry, even the "default"
+  one** — don't rely on `-V` fallback for anything actually reachable from
+  the internet. `f3s.buetow.org` (checked in `relayd.conf` on the frontends:
+  the real routed names are `f3s.buetow.org`, `www.f3s.buetow.org`,
+  `standby.f3s.buetow.org` — `/scifi/` etc. are **paths** under it, not
+  separate subdomains) needs a vhost dir, or it hits `-V`, and bozohttpd's
+  directory-without-trailing-slash redirect in that fallback path uses its
+  own **system hostname**, not the client's `Host:` header (unlike a real
+  vhost match, which correctly echoes back e.g. `snonux.foo`). Since the
+  system hostname (`piN.lan.buetow.org`) doesn't resolve outside the LAN,
+  this produces redirects that hang for external clients. Fix:
+  self-referencing symlinks so these become vhost matches instead of
+  fallbacks — `ln -sf . /var/www/html/f3s.buetow.org` (and the
+  `www.`/`standby.` variants).
+
+Enable with `bozohttpd=YES` in `/etc/rc.conf`.
+
+## Static content sync
+
+`pi0` is the source of truth for `/var/www/html`; `pi1` pulls hourly:
+
+```sh
+#!/bin/sh
+set -e
+STAGE=/tmp/wwwsync-cron
+mkdir -p "$STAGE"
+rsync -a --delete -e "ssh -o StrictHostKeyChecking=accept-new" \
+	paul@pi0.lan.buetow.org:/var/www/html/ "$STAGE/"
+doas rsync -a --delete --exclude=snonux.foo "$STAGE/" /var/www/html/
+doas rsync -a --delete "$STAGE/snonux.foo/" /var/www/html/snonux.foo/
+doas /usr/sbin/chown -R root:wheel /var/www/html/index.html /var/www/html/fotos /var/www/html/scifi
+# snonux.foo is owned by paul, not root: the snonux microblog tool rsyncs
+# directly into it as paul (no doas hop), so it must stay paul-writable.
+doas /usr/sbin/chown -R paul:wheel /var/www/html/snonux.foo
+```
+
+Needs an SSH keypair for `paul` on `pi1`, authorized on `pi0`'s
+`~/.ssh/authorized_keys`, plus a static `/etc/hosts` entry for `pi0` (Pi-to-Pi
+`.lan.buetow.org` resolution isn't reliable — add the IP directly rather
+than debugging DNS).
+
+Install as `paul`'s crontab on `pi1` (not root's — needs the SSH key):
+`47 * * * * /usr/local/bin/sync-from-pi0.sh >$HOME/sync-from-pi0.log 2>&1`
+
+## uptimed (built from source — no prebuilt package)
+
+**No aarch64 binary package exists** in pkgsrc for `uptimed` on any branch
+checked (10.0, 10.1, 11.0, 9.4). Build from upstream instead — small C
+project, NetBSD base already has `gcc`/`make`:
+
+```sh
+pkgin -y install autoconf automake libtool pkg-config
+cd /tmp
+curl -sLO https://github.com/rpodgorny/uptimed/archive/refs/tags/v0.4.7.tar.gz
+tar xzf v0.4.7.tar.gz && cd uptimed-0.4.7
+PATH=/usr/pkg/bin:$PATH ./autogen.sh
+PATH=/usr/pkg/bin:$PATH ./configure --prefix=/usr/pkg --sysconfdir=/etc
+PATH=/usr/pkg/bin:$PATH make
+doas env PATH=/usr/pkg/bin:/usr/bin:/bin:/usr/sbin:/sbin make install
+```
+
+Installs `uptimed` to `/usr/pkg/sbin`, `uprecords` to `/usr/pkg/bin`, and uses
+`/var/spool/uptimed/records` (hardcoded upstream, not an OS convention thing).
+
+**Before first start**, write `/etc/uptimed.conf` with `LOG_MAXIMUM_ENTRIES=0`
+(keep forever) plus milestone lines. If restoring a backed-up uptime history,
+seed **both** `records` and `records.old` with the same content:
+
+```sh
+doas cp <backed-up-records-file> /var/spool/uptimed/records
+doas cp <backed-up-records-file> /var/spool/uptimed/records.old   # both, not just one
+doas /usr/sbin/chown root:wheel /var/spool/uptimed/records /var/spool/uptimed/records.old
+```
+
+**Critical bug to know about**: `read_records()` in `libuptimed/urec.c`
+unconditionally sets `useold = -1` ("no useable database found") if
+`records.old` doesn't exist yet — **regardless of whether the primary
+`records` file is valid**. Seeding only `records` and starting the daemon
+loses the imported history immediately (it gets shunted to a fresh
+`records.old` on the first periodic rewrite, then overwritten again 60s
+later). Seed **both** files with the same content before the first start.
+
+Write a custom `/etc/rc.d/uptimed` (upstream ships a Linux-init `etc/rc.uptimed`,
+not usable directly):
+
+```sh
+# PROVIDE: uptimed
+# REQUIRE: NETWORKING ntpdate
+# KEYWORD: shutdown
+
+command="/usr/pkg/sbin/uptimed"
+pidfile="/var/run/uptimed.pid"
+command_args="-p ${pidfile}"
+```
+
+These Raspberry Pis have no hardware RTC. NetBSD initializes the wall clock
+from the root filesystem timestamp, which can be stale after power-off. Merely
+starting `ntpd` is not a synchronization barrier, so enable the synchronous
+boot-time correction in `/etc/rc.conf`:
+
+```sh
+ntpdate=YES
+```
+
+The explicit `ntpdate` requirement above ensures that uptimed starts only after
+the clock has been set from the network. Verify the boot ordering with:
+
+```sh
+/sbin/rcorder /etc/rc.d/* | egrep '/(NETWORKING|ntpdate|ntpd|uptimed)$'
+```
+
+The expected order is `NETWORKING`, `ntpdate`, `ntpd`, then `uptimed`. This was
+deployed and reboot-tested one node at a time on both pi0 and pi1 on
+**2026-07-16**. `ntpdate` corrected pi0 by 12.29 seconds and pi1 by 12.68
+seconds before uptimed started; their new current records had the correct boot
+times, both webservers returned HTTP 200, and no historical record repair was
+necessary.
+
+Run `uptimed -b` once (creates the boot ID), enable with `uptimed=YES`.
+
+## goprecords upload
+
+```sh
+kubectl exec -n services deployment/goprecords -- \
+	goprecords --create-client-key <host> -stats-dir=/data/stats
+```
+(from a machine with cluster access — this can occasionally 502 if the
+apiserver's exec proxy can't reach whichever k3s node the pod landed on; just
+retry, it's a transient networking issue, not a token problem.)
+
+Deploy `goprecords-upload-client.sh` (from `~/git/goprecords/scripts/`,
+already POSIX/generic and already handles `/var/spool/uptimed/records` and a
+NetBSD `dmesg.boot`/`sysctl` fallback for `os.txt`/`cpuinfo.txt` — no changes
+needed) to **`/usr/pkg/bin/`** to match the pre-baked crontab's path, token at
+`/etc/goprecords-upload.token` (`0600`), `GOPRECORDS_HOST=<host>`.
+
+`curl` and `uprecords` need to be resolvable via whatever `PATH` the cron
+entry sets — test with that exact `PATH` before trusting a manual test run
+under plain `doas` (which won't have it).
+
+## Firewall — npf, not firewalld
+
+```
+$ext_if = "mue0"
+
+group "external" on $ext_if {
+	pass stateful out final all
+	pass stateful in final family inet4 proto tcp to $ext_if port 22
+	pass stateful in final family inet4 proto tcp to $ext_if port 80
+	pass stateful in final family inet4 proto tcp to $ext_if port 2222
+	pass stateful in final family inet4 proto icmp all
+}
+
+group "wireguard" on tun0 {
+	pass stateful out final all
+	pass stateful in final family inet4 all
+	pass stateful in final family inet6 all
+}
+
+group default {
+	pass final on lo0 all
+	block all
+}
+```
+
+Port 2222 is dserver (DTail) — see the `pkgrepo` skill's `dtail-package.md`
+for the install steps.
+
+`family inet4`/`inet6` must be explicit on multi-family interfaces or
+`npfctl validate` fails with "address family mismatch". `proto <name>` must
+be followed by `all` or a `from`/`to` clause, or it's a syntax error — e.g.
+`proto icmp` alone fails, `proto icmp all` doesn't. Don't forget the ICMP
+rule: without it, ping-dependent tooling (e.g. the `snonux` publishing
+tool's reachability pre-check) silently breaks while SSH/HTTP keep working
+fine.
+
+Sequence carefully to avoid locking yourself out over SSH:
+
+```sh
+doas npfctl validate           # syntax-check first
+doas npfctl reload             # loads config, does NOT enable filtering yet
+doas npfctl start               # enables filtering
+# from a FRESH ssh connection (not the one you're already in), confirm:
+#   - ssh still connects
+#   - curl http://localhost/ still works
+doas sh -c 'echo npf=YES >> /etc/rc.conf'   # only after confirming the above
+```
+
+If you get a JIT warning (`error loading the bpfjit module... Operation not
+permitted`) — harmless, just means `kern.securelevel` blocks loading that
+optional performance module; filtering still works, just slightly slower
+packet matching.
+
+## Verification
+
+- `curl -fsI http://<host>.lan.buetow.org/` and the vhost via `Host:` header.
+- `wg show tun0` shows recent handshakes with `blowfish` and `fishfinger` (and
+  `rocky` if that VM happens to be up — it's often not, unrelated to this).
+- goprecords report (`https://goprecords.f3s.buetow.org/report`) picks up the
+  host after the hourly cron fires (won't rank in the "top 20 all-time" table
+  with a short history — that's expected, not a failure).
+- **Redundancy test**: stop the *other* node's webserver entirely, then curl
+  every real page through the **public** domains (not just localhost) — root
+  page, each vhost, and any bare directory paths (e.g. `/fotos/`). Restore
+  the other node's webserver immediately after.
+- `wol-f3s shutdown-pis` (or a targeted `ssh paul@<host> "doas poweroff"`)
+  actually powers the Pi off — confirms `doas` works non-interactively, but
+  there's no WoL for Pis, so only do this when you can physically power it
+  back on.
diff --git a/prompts/skills/f3s-raspberry-pi/references/pihole-pi.md b/prompts/skills/f3s-raspberry-pi/references/pihole-pi.md
new file mode 100644
index 0000000..b2e6de5
--- /dev/null
+++ b/prompts/skills/f3s-raspberry-pi/references/pihole-pi.md
@@ -0,0 +1,45 @@
+# Pi-hole on Raspberry Pi (pi2, pi3)
+
+Pi-hole runs in **Docker** on **`pi2.lan.buetow.org`** and **`pi3.lan.buetow.org`** with **`network_mode: host`** (Rocky Linux 9, firewalld allows 53/tcp, 53/udp, `http`). Compose uses **`cap_add: [NET_ADMIN]`**, bind-mounts **`./etc-pihole:/etc/pihole`** and **`./etc-dnsmasq.d:/etc/dnsmasq.d`**. Secrets live in **`~/pihole/.env`** on each host (**`WEBPASSWORD`** is host-local, not in git).
+
+**Client DNS (LAN):** prefer **`192.168.1.127`** (pi2), then **`192.168.1.128`** (pi3), then router fallback — see **`f3s/pihole/README.md`** in conf for `nmcli` examples.
+
+**Kubernetes:** Pi-hole was moved off the cluster; **`f3s/argocd-apps/services/pihole.yaml`** has sync disabled, but **`dnsmasq.customDnsEntries`** stays aligned with the Pis’ wildcard (`address=/.f3s.lan.buetow.org/192.168.1.138`) if that app is ever re-enabled.
+
+## LAN wildcard DNS
+
+Homelab LAN hostnames under **`*.f3s.lan.buetow.org`** should resolve to the **CARP VIP** **`192.168.1.138`** (FreeBSD **f0/f1** → relayd → k3s Traefik). In **dnsmasq** (Pi-hole):
+
+```text
+address=/.f3s.lan.buetow.org/192.168.1.138
+```
+
+The leading **`.`** matches the apex and all subdomains.
+
+## Tracked files in `conf`
+
+In the **`f3s`** repo (`https://codeberg.org/snonux/conf`):
+
+- **`f3s/pihole/docker-pi/dnsmasq.d/99-f3s-lan-wildcard.conf`** — copy into **`~/pihole/etc-dnsmasq.d/`** on each Pi (bind-mounted to `/etc/dnsmasq.d` in the live compose).
+- **`f3s/pihole/docker-pi/docker-compose.example.yml`** — reference compose including the **`etc-dnsmasq.d`** volume; merge with your live **`docker-compose.yml`**.
+
+After changing dnsmasq config: **`docker compose restart`** in **`~/pihole`**.
+
+**Rollout (from a workstation with SSH):** copy **`99-f3s-lan-wildcard.conf`** to each Pi (e.g. `/tmp`), then `sudo install -o root -g root -m 644 … ~/pihole/etc-dnsmasq.d/`, remove any obsolete apex-only file (e.g. **`02-custom-f3s.conf`**), restart compose. Keep both nodes in sync.
+
+## Verify
+
+```bash
+dig @pi2.lan.buetow.org foo.f3s.lan.buetow.org +short   # expect 192.168.1.138
+dig @pi3.lan.buetow.org f3s.lan.buetow.org +short       # expect 192.168.1.138
+```
+
+Admin UI: **`http://pi2.lan.buetow.org/admin/`** (and pi3).
+
+## Timekeeping on the RTC-less Pis
+
+pi2 and pi3 have no hardware RTC, so their clocks are stale briefly during boot until chronyd synchronizes. `uptimed` must not start in that window: both hosts use a systemd override that waits for `chronyc waitsync`, preventing invalid boot timestamps and missing active markers in goprecords. The canonical override, recovery procedure, and verification commands are in [goprecords-uptimed.md](../../f3s-workloads/references/goprecords-uptimed.md#rocky-pi-uptimed-clock-synchronization).
+
+## Public DNS note
+
+**`frontends/var/nsd/zones/master/buetow.org.zone.tpl`** already has **`*.f3s.lan IN A 192.168.1.138`** for authoritative **`buetow.org`**; Pi-hole on the LAN keeps the same mapping for clients that use pi2/pi3 as resolver.