diff options
| -rw-r--r-- | prompts/skills/f3s/SKILL.md | 27 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/bootstrap-netbsd-pi.md | 356 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/dtail.md | 10 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/goprecords-uptimed.md | 3 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/shelly-plug.md | 11 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/wireguard.md | 29 | ||||
| -rw-r--r-- | prompts/skills/pkgrepo/SKILL.md | 2 | ||||
| -rw-r--r-- | prompts/skills/pkgrepo/references/client-setup.md | 6 | ||||
| -rw-r--r-- | prompts/skills/pkgrepo/references/dtail-package.md | 12 | ||||
| -rw-r--r-- | prompts/skills/pkgrepo/references/packaging-workflow.md | 2 |
10 files changed, 424 insertions, 34 deletions
diff --git a/prompts/skills/f3s/SKILL.md b/prompts/skills/f3s/SKILL.md index ccd2202..fb90871 100644 --- a/prompts/skills/f3s/SKILL.md +++ b/prompts/skills/f3s/SKILL.md @@ -1,6 +1,6 @@ --- name: f3s -description: Reference skill for the f3s homelab—four Beelink S12 Pro hosts (f0/f1/f2/f3) running FreeBSD with Rocky Linux Bhyve VMs and a k3s Kubernetes cluster. f0/f1/f2 run r0/r1/r2 k3s nodes; f3 is standalone bhyve only (not part of k3s) and hosts the plain Rocky Linux VM named rocky. Four Raspberry Pi 3 nodes (pi0–pi3) on Rocky Linux 9; pi2/pi3 run Pi-hole (Docker) and LAN wildcard DNS for *.f3s.lan.buetow.org. Covers DTail/dserver on Pis (arm64) and k3s VMs (amd64). Use when troubleshooting or making configuration decisions for the f3s setup. +description: Reference skill for the f3s homelab—four Beelink S12 Pro hosts (f0/f1/f2/f3) running FreeBSD with Rocky Linux Bhyve VMs and a k3s Kubernetes cluster. f0/f1/f2 run r0/r1/r2 k3s nodes; f3 is standalone bhyve only (not part of k3s) and hosts the plain Rocky Linux VM named rocky. Four Raspberry Pi 3 nodes (pi0–pi3): pi0 and pi1 are NetBSD 10.1 (the static f3s.buetow.org/snonux.foo HTTP pair), pi2/pi3 are still Rocky Linux 9 running Pi-hole (Docker) and LAN wildcard DNS for *.f3s.lan.buetow.org. Covers DTail/dserver on Pis (arm64) and k3s VMs (amd64). Use when troubleshooting or making configuration decisions for the f3s setup. --- # f3s Homelab Reference @@ -24,6 +24,7 @@ Detailed reference documentation is in the `references/` subfolder: - [Rocky Linux VMs](references/rocky-linux-vms.md) — Bhyve, vm-bhyve, VM config, NVMe disk fix; FreeBSD VM on f3 (migrated from f0) - [f3 Rocky VM](references/f3-rocky-vm.md) — Plain Rocky Linux 9 VM on f3 (`rocky`, `192.168.1.123`), autostart policy, root SSH - [Bootstrap Rocky bhyve VM](references/bootstrap-rocky-bhyve.md) — Runbook for creating a new plain Rocky Linux bhyve guest with unattended kickstart +- [Bootstrap NetBSD Pi](references/bootstrap-netbsd-pi.md) — Runbook for converting a Rocky Pi to NetBSD, validated twice now (`pi0` then `pi1`, both reboot-tested): doas/pkgin bootstrap, WireGuard via userspace `wireguard-go` (no native `wg(4)` on this platform), bozohttpd (`-X` for dir-listing parity, vhost symlinks for every real routed hostname from day one), uptimed built from source, npf firewall, content-sync direction. Only `pi2`/`pi3` (Pi-hole) remain Rocky. - [WireGuard Mesh](references/wireguard.md) — Mesh topology, IP assignments, peer configs - [Storage](references/storage.md) — index into `references/storage/`: ZFS (zdata), zrepl, CARP, NFS over stunnel, nfs-mount-monitor, troubleshooting (incl. thermal), backups & local-path - [r-node Deploy (Rex)](references/r-node-deploy.md) — reusable Rex rollout to **r0/r1/r2** (`f3s/r-nodes/Rexfile`, task `nfs_mount_monitor`): root SSH, `parallelism 3`, idempotent `file`/`on_change` reload, verify with `systemctl`/`journalctl` @@ -60,14 +61,14 @@ The plain Rocky Linux VM on f3 (`rocky`, `192.168.1.123`) is documented in the s | earth | Fedora laptop (roaming) | — | 192.168.2.200 | | pixel7pro | Android (roaming) | — | 192.168.2.201 | | f3s-storage-ha | CARP VIP (f0/f1) | 192.168.1.138 | — | -| pi0 | Raspberry Pi 3, Rocky Linux 9, static `f3s.buetow.org` backend | 192.168.1.125 | 192.168.2.203 | -| pi1 | Raspberry Pi 3, Rocky Linux 9, static `f3s.buetow.org` backend | 192.168.1.126 | 192.168.2.204 | +| pi0 | Raspberry Pi 3, **NetBSD 10.1** (evbarm-aarch64), static `f3s.buetow.org` backend | 192.168.1.125 | 192.168.2.203 | +| pi1 | Raspberry Pi 3, **NetBSD 10.1** (evbarm-aarch64), static `f3s.buetow.org` backend | 192.168.1.126 | 192.168.2.204 | | pi2 | Raspberry Pi 3, Rocky Linux 9, Pi-hole (Docker, host net) | 192.168.1.127 | — | | pi3 | Raspberry Pi 3, Rocky Linux 9, Pi-hole (Docker, host net) | 192.168.1.128 | — | ## Raspberry Pi Nodes -Four Raspberry Pi 3 boards running Rocky Linux 9.2 (Blue Onyx) aarch64 from the SIG/AltArch image (`RockyLinuxRpi_9-latest.img.xz`). Each has: +`pi2`/`pi3` run Rocky Linux 9.2 (Blue Onyx) aarch64 from the SIG/AltArch image (`RockyLinuxRpi_9-latest.img.xz`). `pi0` and `pi1` were both reinstalled to **NetBSD 10.1** (evbarm-aarch64) — `pi0` first (2026-07-03), `pi1` the same day once `pi0` was validated. Each Rocky Pi has: - User `paul` with passwordless sudo and SSH key auth - Static IP on eth0 via NetworkManager @@ -77,19 +78,23 @@ Four Raspberry Pi 3 boards running Rocky Linux 9.2 (Blue Onyx) aarch64 from the - No GRUB — boots via Pi's native bootloader (`/boot/cmdline.txt`) - Custom RPi kernel from the `rockyrpi` repo +`pi0`/`pi1` (NetBSD) differ: user `paul` in `wheel`, privilege escalation via a **real `doas`** (pkgsrc `security/doas`, `permit nopass :wheel`) — not the `alias doas=sudo` shell alias `pi2`/`pi3` carry in `/etc/profile.d/doas.sh`, which doesn't expand in the non-interactive shell an SSH command runs in and so silently breaks `wol-f3s shutdown-pis`/`shutdown-all` for the Rocky Pis (`doas poweroff` resolves to nothing) — only the NetBSD nodes actually work with that script today. Config repo home for NetBSD-specific setup: `f3s/pi-netbsd/`. Full conversion runbook: [Bootstrap NetBSD Pi](references/bootstrap-netbsd-pi.md). + Current role split: -- `pi0` and `pi1` serve static `f3s.buetow.org` content behind OpenBSD `relayd` over WireGuard +- `pi0` and `pi1` serve static `f3s.buetow.org`/`snonux.foo` content behind OpenBSD `relayd` over WireGuard. WireGuard peers are `blowfish`, `fishfinger`, **and `rocky`** (not gateway-only to just the two frontends, despite older docs here). Both nodes reboot-tested (2026-07-03): all rc.d services (`wireguard`, `bozohttpd`, `uptimed`, `npf`) and both crontabs come back automatically. - `pi2` and `pi3` run **Pi-hole** in Docker (`network_mode: host`, `~/pihole` on each host). Tracked dnsmasq LAN wildcard: **`f3s/pihole/docker-pi/`** in the conf repo; details in [references/pihole-pi.md](references/pihole-pi.md). -### lighttpd Configuration +### Webserver Configuration + +Both `pi0` and `pi1` run **bozohttpd** (built into NetBSD base, no package/config file) via a custom `/etc/rc.d/bozohttpd` (the stock rc.d/httpd script ignores `httpd_flags` entirely — never references it — so flags had to go directly in this script's own `command_args`), using `-v`/`-V` for vhosting instead of lighttpd's `$HTTP["host"]` regex match — a vhost needs a directory *literally* named after the hostname (e.g. `snonux.foo/`, with `www.snonux.foo` a symlink to it). Also needs **`-X`** (directory indexing) to match lighttpd's old `dir-listing.activate = "enable"` — without it, bare directories with no `index.html` (e.g. `/fotos/<gallery>/` at the top level) 404 instead of showing a listing; caught on `pi0` by a live redundancy test (stopped the other node's webserver, `curl`'d every page through the public domains) and included from the start on `pi1`. -Config file: `/etc/lighttpd/lighttpd.conf` (managed directly on pi0/pi1, not in a config repo) +**bozohttpd `-V` fallback bug** (hit and fixed on `pi0`, avoided on `pi1` since the fix was already baked in): `f3s.buetow.org` (the real routed hostname — relayd forwards `f3s.buetow.org`/`www.f3s.buetow.org`/`standby.f3s.buetow.org`, all Host-matched via `match request header "Host" value ... forward to <f3s_static_proxy>` in `relayd.conf` on the frontends; there is no separate `scifi.f3s.buetow.org` subdomain, `/scifi/` is just a **path** under it) had no dedicated vhost directory, so it hit the `-V` fallback — and bozohttpd's directory-without-trailing-slash redirect in that fallback path uses its own **system hostname** (e.g. `pi0.lan.buetow.org`) instead of the client's `Host:` header, unlike a vhost-*matched* request (which correctly echoes back the matched name, e.g. `snonux.foo`). Since the system hostname doesn't resolve outside the LAN, external clients following that redirect would hang. **Fixed** with self-referencing vhost symlinks so these hostnames become vhost matches instead of fallbacks: `ln -sf . /var/www/html/f3s.buetow.org`, same for `www.f3s.buetow.org` and `standby.f3s.buetow.org`. -- Document root: `/var/www/html` +- Document root: `/var/www/html` (same path both nodes now — the historical `/var/www/html/snonux` vs `/var/www/html/snonux.foo` naming difference no longer applies since `pi1` was reinstalled and now mirrors `pi0`'s tree exactly) - SSH access: `ssh paul@piN.lan.buetow.org -p 22` -- Host-based virtual hosting maps domains to subdirectories: - - `snonux.foo` / `www.snonux.foo` → `/var/www/html/snonux` +- Host-based virtual hosting maps domains to subdirectories: `snonux.foo` / `www.snonux.foo` → `/var/www/html/snonux.foo` +- Content sync direction flipped mid-migration: `pi1` → `pi0` while `pi1` was still the only node with real content (Rocky), then reversed to `pi0` → `pi1` once `pi1` was also reinstalled (`pi0` is now the long-lived source of truth). `pi1`'s hourly pull is `/usr/local/bin/sync-from-pi0.sh` (cron `:47`); `pi0`'s old `sync-from-pi1.sh` (cron `:17`) is a harmless now-pointless leftover pulling from a node with no independent content anymore. **Why Host-based vhosts?** `relayd` on the OpenBSD frontends cannot rewrite URL paths. It forwards requests with the original path intact. To serve a subdirectory as root for a domain, lighttpd must remap the document root based on the `Host` header. @@ -104,7 +109,7 @@ $HTTP["host"] =~ "^(www\.)?snonux\.foo$" { ## DTail (dserver) -Distributed log access over SSH on port **2222** (not sshd’s 22). **pi0–pi3**: cross-build **linux/arm64** + `DTAIL_NO_ZSTD=yes`. **r0–r2** (k3s Rocky VMs): **linux/amd64** only; install as **root** over SSH; **`dtail.json` must list `root` in `Server.Permissions.Users`**; mirror **`/root/.ssh/authorized_keys`** → `/var/run/dserver/cache/root.authorized_keys` because the key-cache script only walks `/home/*`. **firewalld**: open **2222/tcp**. Rebuild clients from current **dtail** `master` if the “trust these hosts” prompt still hangs (stdout pause bug fixed upstream). +Distributed log access over SSH on port **2222** (not sshd’s 22). **pi2–pi3**: cross-build **linux/arm64** + `DTAIL_NO_ZSTD=yes`. **pi0**/**pi1** (NetBSD) do **not** run DTail — deliberately deferred on both; would need an untested `GOOS=netbsd GOARCH=arm64` cross-build and an `rc.d` script in place of the systemd unit. **r0–r2** (k3s Rocky VMs): **linux/amd64** only; install as **root** over SSH; **`dtail.json` must list `root` in `Server.Permissions.Users`**; mirror **`/root/.ssh/authorized_keys`** → `/var/run/dserver/cache/root.authorized_keys` because the key-cache script only walks `/home/*`. **firewalld**: open **2222/tcp**. Rebuild clients from current **dtail** `master` if the “trust these hosts” prompt still hangs (stdout pause bug fixed upstream). Details: [references/dtail.md](references/dtail.md) (section **dserver on r0, r1, r2**). diff --git a/prompts/skills/f3s/references/bootstrap-netbsd-pi.md b/prompts/skills/f3s/references/bootstrap-netbsd-pi.md new file mode 100644 index 0000000..411b059 --- /dev/null +++ b/prompts/skills/f3s/references/bootstrap-netbsd-pi.md @@ -0,0 +1,356 @@ +# Bootstrapping a Raspberry Pi 3 f3s node on NetBSD + +Runbook for bringing a Rocky Pi node up to NetBSD parity, validated twice now: +`pi0` first (2026-07-03), then `pi1` the same day using this exact procedure — +both reboot-tested and confirmed durable. Only `pi2`/`pi3` (Pi-hole) remain +Rocky; this is the reusable procedure for converting either of them next. + +**Do this one node at a time.** Never take down two of `pi0`–`pi3`'s static-HTTP +pair (or the two Pi-hole nodes) simultaneously — one must always keep serving. + +## 0. Before you reinstall: back up state that doesn't come from git + +None of this lives in the conf repo — it only exists on the running Pi, so it +must be pulled off **before** wiping it: + +- **Uptime history**: `scp` the target's `/var/spool/uptimed/records` (and + `.old` if present) somewhere safe (e.g. `f0.lan.buetow.org:/tmp/<host>.records`, + the same pattern used for `pi0`). Restoring it later preserves the uptime + streak instead of resetting to zero — see step 6. +- **WireGuard identity**: not actually needed from the live host — the + private key and PSKs already live in `~/git/wireguardmeshgenerator/keys/` + (`keys/<host>/priv.key`, `keys/psk/<pair>.key`). Reuse those; do not + regenerate, or every peer (`blowfish`, `fishfinger`, `rocky`) needs updating. +- **Static content**: only relevant if this node is currently the source of + truth for `/var/www/html`. Once `pi0` is up and holding the full synced + tree, `pi0` becomes the natural source for converting `pi1` next — no + separate backup needed, just sync FROM `pi0` afterward (see step 4). + +## 1. Flash/image plain NetBSD + +Not yet documented in this repo — `f3s/pi-netbsd/bake` and `flash/` exist as +scaffolding in the conf repo but were empty as of 2026-07-03; the imaging +process used for `pi0` happened outside this repo. Whatever process is used, +the expected end state (confirmed on `pi0`) is: + +- NetBSD 10.1 `GENERIC64` evbarm64 (aarch64) +- User `paul`, in group `wheel`, SSH key auth already working +- Static LAN IP already set via `rc.conf` (`ifconfig_mue0="inet 192.168.1.12N + netmask 0xffffff00"`, `defaultroute="192.168.1.1"` — match the hardware + table's IP for this host) +- Hostname set (`hostname="piN.lan.buetow.org"`) +- **No** `doas`/`sudo`, **no** pkgsrc/pkgin bootstrapped — bare base install +- `paul` has no private SSH key of its own yet (only `authorized_keys`) + +If any of these aren't true after flashing, do them first (they're baked into +the disk image for `pi0`, presumably via sysinst answers or a post-install +script — worth capturing properly in `f3s/pi-netbsd/` once that process is +pinned down). + +**Note found on `pi0`**: the image already carried a **pre-baked root +crontab** entry for the hourly goprecords upload, pointing at +`/usr/pkg/bin/goprecords-upload-client.sh` with `GOPRECORDS_HOST` already set +correctly. Check `doas crontab -l` early — if it's there, deploy the script to +match that path instead of improvising a different location. + +## 2. Bootstrap pkgin + real doas + +```sh +ssh paul@piN.lan.buetow.org +su - # root password from the bake; rotate both root and + # paul's passwords once doas works (step below) — + # not automated, do it by hand +export PKG_PATH=https://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/aarch64/10.1/All/ +pkg_add -v pkgin +pkgin -y update +pkgin -y install doas rsync curl +printf 'permit nopass :wheel\n' > /usr/pkg/etc/doas.conf # NOT "permit persist" -- + # that still prompts once + # per session, which never + # succeeds over a + # non-interactive SSH + # command (no tty) +chmod 644 /usr/pkg/etc/doas.conf +exit # back to paul +doas true # should succeed with no password prompt +``` + +**Why real `doas`, not the Rocky pattern**: `pi2`–`pi3` only alias `doas` to +`sudo` via `/etc/profile.d/doas.sh`, which doesn't expand in the +non-interactive shell an SSH command runs in — so +`~/git/dotfiles/scripts/wol-f3s`'s `shutdown-pis`/`shutdown-all` (which runs +`ssh paul@pi "doas poweroff"`) is silently broken on the Rocky Pis today. A +real `doas` binary fixes it for whichever Pi gets NetBSD. + +**Gotcha**: commands run via `doas` get a minimal `PATH` that excludes +`/usr/sbin` and `/usr/pkg/bin` — always use full paths (`doas +/usr/sbin/chown`, `doas /usr/pkg/bin/wg`) or an explicit `PATH=` for cron. + +## 3. WireGuard (userspace — no native `wg(4)` on this platform) + +**`wg(4)` doesn't exist on evbarm-aarch64 10.1** — confirmed on `pi0`: the +module is absent from all 249 files under `/stand/evbarm/10.1/modules`, so +`ifconfig wg0 create` fails outright (`clone_command: Invalid argument`), +despite `wg(4)` being upstream NetBSD since 9.2. Don't waste time on it; +`wireguard-go` + `wg` (pkgsrc `wireguard-tools`, **no `wg-quick`** in this +build) is the working path: + +```sh +pkgin -y install wireguard-go wireguard-tools +``` + +Pull this host's existing private key and the three PSKs (`blowfish`, +`fishfinger`, `rocky`) from `~/git/wireguardmeshgenerator/keys/` — do not +regenerate. Also grab peer pubkeys/endpoints/AllowedIPs from +`dist/<host>/etc/wireguard/wg0.conf` if the generator has already rendered +them, or from another live node's `/etc/wireguard/wg0.conf` for the endpoint +details (same for every Pi, only `AllowedIPs`/PSK differ per host). + +Key facts that cost time on `pi0`: + +- The interface **must** be named `tunN` (`wireguard-go` rejects `wg0`: + "Interface name must be tun[0-9]*"). Use `tun0`. +- **Address the interface before starting `wireguard-go`**, or its read loop + dies immediately with `EHOSTDOWN` ("host is down") and does not retry: + ```sh + ifconfig tun0 create + ifconfig tun0 inet <wg-ip> <wg-ip> netmask 255.255.255.255 + ifconfig tun0 inet6 <wg-ipv6> + ifconfig tun0 up + wireguard-go tun0 # daemonizes on its own + ``` +- Apply crypto config with the real `wg` CLI (supports `PersistentKeepalive`, + unlike the native `wgconfig` tool which has no keepalive flag at all): + ```sh + wg setconf tun0 /usr/pkg/etc/wireguard/tun0.conf + ``` + `tun0.conf` is the normal `[Interface]`/`[Peer]` format — same content as + what `wireguardmeshgenerator` renders to `dist/<host>/etc/wireguard/wg0.conf`, + just handed to `wg` instead of `wg-quick`. +- **No `wg-quick` means no automatic routes.** Each peer's AllowedIPs needs an + explicit host route via the local tun IP: + ```sh + route add -inet <peer-allowed-ip>/32 <local-tun4-ip> -iface + route add -inet6 <peer-allowed-ipv6>/128 <local-tun6-ip> -iface + ``` + +Wire all of this into a custom `/etc/rc.d/wireguard` (there's no stock rc.d +for this combination — write one, model it on `pi0`'s). Enable with +`wireguard=YES` in `/etc/rc.conf`. + +**Follow-up not yet done for any host**: `wireguardmeshgenerator.rb` only +branches on `os == 'Linux' | 'FreeBSD' | 'OpenBSD'` and always emits +`wg-quick`-style files; the generator's YAML still lists NetBSD hosts as +`os: Linux`. Until it gains NetBSD support, each converted Pi's WireGuard +config is a manually-maintained exception that a future +`--generate`/`--install` regen would clobber. + +## 4. Webserver — bozohttpd + +Built into NetBSD base, matches the "static-only, low-RAM" reasoning already +used to justify lighttpd on the Rocky Pis. No stock rc.d exists that actually +uses `httpd_flags` (the shipped `/etc/rc.d/httpd` computes `command_args` +itself and never references that variable) — write a dedicated +`/etc/rc.d/bozohttpd`: + +```sh +command="/usr/libexec/httpd" +pidfile="/var/run/bozohttpd.pid" +command_args="-b -X -U _httpd -P ${pidfile} -v /var/www/html -V /var/www/html" +required_dirs="/var/www/html" +``` + +- `-v /var/www/html -V /var/www/html`: vhost directory = same tree as the + default docroot. A `Host:` header matching a **literally-named** + subdirectory (e.g. `snonux.foo/`) is served from there; anything unmatched + falls back to the plain docroot via `-V`. +- `www.snonux.foo` needs to be a symlink to `snonux.foo` (bozohttpd matches + the literal Host header as a directory name, not a regex like lighttpd's + `$HTTP["host"] =~ "^(www\.)?snonux\.foo$"`). +- **`-X` (directory indexing) is required**, not optional — caught by a live + redundancy test (stopping the other node's lighttpd and curling every page): + bare directories with no `index.html` (e.g. a photo gallery folder under + `/fotos/`) 404 without it, because lighttpd has + `dir-listing.activate = "enable"` and bozohttpd needs the equivalent flag. +- **Give every real routed hostname its own vhost entry, even the "default" + one** — don't rely on `-V` fallback for anything actually reachable from the + internet. `f3s.buetow.org` (checked in `relayd.conf` on the frontends: the + real routed names are `f3s.buetow.org`, `www.f3s.buetow.org`, + `standby.f3s.buetow.org` — `/scifi/` etc. are **paths** under it, not + separate subdomains) had no vhost dir, so it hit `-V`, and bozohttpd's + directory-without-trailing-slash redirect in that fallback path uses its own + **system hostname**, not the client's `Host:` header (unlike a real vhost + match, which correctly echoes back e.g. `snonux.foo`). Since the system + hostname (`piN.lan.buetow.org`) doesn't resolve outside the LAN, this + produced redirects that hang for external clients. Fix: self-referencing + symlinks so these become vhost matches instead of fallbacks — + `ln -sf . /var/www/html/f3s.buetow.org` (and the `www.`/`standby.` variants). + +Enable with `bozohttpd=YES` in `/etc/rc.conf`. + +## 5. Static content sync + +Direction depends on which Pi is being converted and what's already been +migrated: + +- Converting the **first** Rocky Pi in a same-role pair (this was `pi0`): + pull from the Rocky sibling that's staying up (`pi1` for `pi0`). +- Converting the **second** one (`pi1`, once `pi0` is already NetBSD and + holds the full current tree): reverse it — pull from `pi0` instead. + +One-time bring-up + hourly cron, same script either way (just swap source +host): + +```sh +#!/bin/sh +set -e +STAGE=/tmp/wwwsync-cron +mkdir -p "$STAGE" +rsync -a --delete -e "ssh -o StrictHostKeyChecking=accept-new" \ + paul@<source-host>.lan.buetow.org:/var/www/html/ "$STAGE/" +doas rsync -a --delete "$STAGE/index.html" "$STAGE/fotos" "$STAGE/scifi" /var/www/html/ +doas rsync -a --delete "$STAGE/snonux/" /var/www/html/snonux.foo/ +doas /usr/sbin/chown -R root:wheel /var/www/html +``` + +Needs a fresh SSH keypair for `paul` on this host (none exists on a bare +image), authorized on the source host's `~/.ssh/authorized_keys`, plus a +static `/etc/hosts` entry for the source hostname (Pi-to-Pi `.lan.buetow.org` +resolution failed for both `pi0`→`pi1` and originally `pi0`→`pi1` during the +very first Rocky bring-up — add the IP directly rather than debugging DNS). + +Install as `paul`'s crontab (not root's — needs the new SSH key): +`17 * * * * /usr/local/bin/sync-from-<source>.sh >$HOME/sync-from-<source>.log 2>&1` + +## 6. uptimed (built from source — no prebuilt package) + +**No aarch64 binary package exists** in pkgsrc for `uptimed` on any branch +checked (10.0, 10.1, 11.0, 9.4). Build from upstream instead — small C +project, NetBSD base already has `gcc`/`make`: + +```sh +pkgin -y install autoconf automake libtool pkg-config +cd /tmp +curl -sLO https://github.com/rpodgorny/uptimed/archive/refs/tags/v0.4.7.tar.gz +tar xzf v0.4.7.tar.gz && cd uptimed-0.4.7 +PATH=/usr/pkg/bin:$PATH ./autogen.sh +PATH=/usr/pkg/bin:$PATH ./configure --prefix=/usr/pkg --sysconfdir=/etc +PATH=/usr/pkg/bin:$PATH make +doas env PATH=/usr/pkg/bin:/usr/bin:/bin:/usr/sbin:/sbin make install +``` + +Installs `uptimed` to `/usr/pkg/sbin`, `uprecords` to `/usr/pkg/bin`, and uses +`/var/spool/uptimed/records` (matches the Rocky path — this is hardcoded +upstream, not an OS convention thing). + +**Before first start**, write `/etc/uptimed.conf` with `LOG_MAXIMUM_ENTRIES=0` +(keep forever — copy the Rocky sibling's `/etc/uptimed.conf` verbatim, +milestones included) and restore the backed-up history from step 0: + +```sh +doas cp <backed-up-records-file> /var/spool/uptimed/records +doas cp <backed-up-records-file> /var/spool/uptimed/records.old # both, not just one +doas /usr/sbin/chown root:wheel /var/spool/uptimed/records /var/spool/uptimed/records.old +``` + +**Critical bug to know about**: `read_records()` in `libuptimed/urec.c` +unconditionally sets `useold = -1` ("no useable database found") if +`records.old` doesn't exist yet — **regardless of whether the primary +`records` file is valid**. Seeding only `records` and starting the daemon +loses the imported history immediately (it gets shunted to a fresh +`records.old` on the first periodic rewrite, then overwritten again 60s +later). Seed **both** files with the same content before the first start. + +Write a custom `/etc/rc.d/uptimed` (upstream ships a Linux-init `etc/rc.uptimed`, +not usable directly): + +```sh +command="/usr/pkg/sbin/uptimed" +pidfile="/var/run/uptimed.pid" +command_args="-p ${pidfile}" +``` + +Run `uptimed -b` once (creates the boot ID), enable with `uptimed=YES`. + +## 7. goprecords upload + +```sh +kubectl exec -n services deployment/goprecords -- \ + goprecords --create-client-key <host> -stats-dir=/data/stats +``` +(from a machine with cluster access — this can occasionally 502 if the +apiserver's exec proxy can't reach whichever k3s node the pod landed on; just +retry, it's a transient networking issue, not a token problem.) + +Deploy `goprecords-upload-client.sh` (from `~/git/goprecords/scripts/`, +already POSIX/generic and already handles `/var/spool/uptimed/records` and a +NetBSD `dmesg.boot`/`sysctl` fallback for `os.txt`/`cpuinfo.txt` — no changes +needed) to **`/usr/pkg/bin/`** if the pre-baked crontab expects that path (see +the note in step 1), token at `/etc/goprecords-upload.token` (`0600`), +`GOPRECORDS_HOST=<host>`. + +`curl` and `uprecords` need to be resolvable via whatever `PATH` the cron +entry sets — if it's the pre-baked one with an explicit `PATH=` line +including `/usr/pkg/bin`, no symlinking needed; test with that exact `PATH` +before trusting a manual test run under plain `doas` (which won't have it). + +## 8. Firewall — npf, not firewalld + +``` +$ext_if = "mue0" + +group "external" on $ext_if { + pass stateful out final all + pass stateful in final family inet4 proto tcp to $ext_if port 22 + pass stateful in final family inet4 proto tcp to $ext_if port 80 +} + +group "wireguard" on tun0 { + pass stateful out final all + pass stateful in final family inet4 all + pass stateful in final family inet6 all +} + +group default { + pass final on lo0 all + block all +} +``` + +`family inet4`/`inet6` must be explicit on multi-family interfaces or +`npfctl validate` fails with "address family mismatch". Sequence carefully to +avoid locking yourself out over SSH: + +```sh +doas npfctl validate # syntax-check first +doas npfctl reload # loads config, does NOT enable filtering yet +doas npfctl start # enables filtering +# from a FRESH ssh connection (not the one you're already in), confirm: +# - ssh still connects +# - curl http://localhost/ still works +doas sh -c 'echo npf=YES >> /etc/rc.conf' # only after confirming the above +``` + +If you get a JIT warning (`error loading the bpfjit module... Operation not +permitted`) — harmless, just means `kern.securelevel` blocks loading that +optional performance module; filtering still works, just slightly slower +packet matching. + +## 9. Verification + +- `curl -fsI http://<host>.lan.buetow.org/` and the vhost via `Host:` header. +- `wg show tun0` shows recent handshakes with `blowfish` and `fishfinger` (and + `rocky` if that VM happens to be up — it's often not, unrelated to this + work). +- goprecords report (`https://goprecords.f3s.buetow.org/report`) picks up the + new host after the hourly cron fires (won't rank in the "top 20 all-time" + table with a fresh/short history — that's expected, not a failure). +- **Real redundancy test** (this is what actually caught the missing `-X` + flag on `pi0` — don't skip it): stop the *other* node in the pair's + webserver entirely, then curl every real page through the **public** + domains (not just localhost) — root page, each vhost, and any bare + directory paths (e.g. `/fotos/`). Restore the other node's webserver + immediately after. +- `wol-f3s shutdown-pis` (or a targeted `ssh paul@<host> "doas poweroff"`) + actually powers the Pi off — confirms the real-`doas` fix, but there's no + WoL for Pis, so only do this when you can physically power it back on. diff --git a/prompts/skills/f3s/references/dtail.md b/prompts/skills/f3s/references/dtail.md index 5d6a2a8..63941f1 100644 --- a/prompts/skills/f3s/references/dtail.md +++ b/prompts/skills/f3s/references/dtail.md @@ -8,14 +8,16 @@ Upstream install and examples live in the repo: `doc/installation.md`, `examples | Hosts | OS / arch | dserver binary | Typical SSH user | |-------|-----------|----------------|------------------| -| **pi0–pi3** | Rocky Linux 9 **aarch64** (Raspberry Pi 3) | Cross-build **linux/arm64**, `nozstd` | `paul@piN.lan.buetow.org` | +| **pi2–pi3** | Rocky Linux 9 **aarch64** (Raspberry Pi 3) | Cross-build **linux/arm64**, `nozstd` | `paul@piN.lan.buetow.org` | | **r0–r2** | Rocky Linux 9 **x86_64** (bhyve VMs, k3s nodes) | Cross-build **linux/amd64**, `nozstd` | Often `root@rN.lan.buetow.org` (see [Rocky Linux VMs](rocky-linux-vms.md)); add `root` (and `paul` if present) to **Server.Permissions.Users** in `dtail.json` | | **blowfish, fishfinger** | OpenBSD 7.8 **amd64** | Native OpenBSD package build | `rex@blowfish.buetow.org`, `rex@fishfinger.buetow.org` | +`pi0`/`pi1` are **NetBSD** now (see `bootstrap-netbsd-pi.md`) and deliberately do **not** run DTail — would need an untested `GOOS=netbsd GOARCH=arm64` cross-build and an `rc.d` script. + **Key cache filenames matter:** `examples/update_key_cache.sh.example` only scans `/home/*` and writes `/var/run/dserver/cache/USER.authorized_keys`. In this lab, DTail auth worked only after writing the exact cache filename for the login user: - **r0–r2**: `root.authorized_keys` -- **pi0–pi3**: `paul.authorized_keys` +- **pi2–pi3**: `paul.authorized_keys` - **blowfish, fishfinger**: `rex.authorized_keys` If clients connect as **root**, copy keys once (e.g. after install) and on key changes: @@ -156,7 +158,7 @@ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags nozstd -o dserver-linux-amd ## Installation checklist (each server) -Do this on **each** target (pi0–pi3 and/or r0–r2). Adjust **user** if you are not using `paul` on the node. +Do this on **each** target (pi2–pi3 and/or r0–r2 — not pi0/pi1, which are NetBSD and don't run DTail). Adjust **user** if you are not using `paul` on the node. 1. **Binary**: `/usr/local/bin/dserver`, mode `0755`, owned by root. 2. **OS user**: `dserver` system account (`useradd -r -d /var/lib/dserver -s /sbin/nologin -U dserver`). @@ -184,7 +186,7 @@ Do this on **each** target (pi0–pi3 and/or r0–r2). Adjust **user** if you ar ```bash dcat --plain --noColor --trustAllHosts --user paul \ - --servers pi0.lan.buetow.org,pi1.lan.buetow.org,pi2.lan.buetow.org,pi3.lan.buetow.org \ + --servers pi2.lan.buetow.org,pi3.lan.buetow.org \ --files /etc/fstab dcat --plain --noColor --trustAllHosts --user root \ diff --git a/prompts/skills/f3s/references/goprecords-uptimed.md b/prompts/skills/f3s/references/goprecords-uptimed.md index 439c0fb..bf2c4ee 100644 --- a/prompts/skills/f3s/references/goprecords-uptimed.md +++ b/prompts/skills/f3s/references/goprecords-uptimed.md @@ -35,7 +35,8 @@ Install **`curl`** and **`uptimed`** on every client that uploads. |--------|--------|------------|--------| | OpenBSD frontends | **fishfinger**, **blowfish** | **Rex** **`goprecords_upload`** in **`~/git/conf/frontends`**; **`/etc/daily.local`** runs **`/usr/local/bin/goprecords-upload.sh`** once per **day** | Tokens in **geheim** **`secrets/etc/goprecords/<host>.token`**; template **`scripts/goprecords-upload.sh.tpl`** | | FreeBSD (Beelinks) | **f0**–**f3** (LAN **`192.168.1.130`–`133`**) | Manual **hourly** **root** **`cron`** calling **`goprecords-upload-client.sh`** with **`GOPRECORDS_HOST=f0`** … **`f3`** | **`/var/db/uptimed/records`**; SSH: **`fN.lan.buetow.org`** or **`192.168.1.(130+N)`** for **fN**, **`-p 22`** | -| Raspberry Pi (Rocky) | **pi0**–**pi3** | Manual **hourly** **systemd** **timer** (see README) | **`/var/spool/uptimed/records`**; SSH: **`piN.lan.buetow.org`**, **`-p 22`** | +| Raspberry Pi (Rocky) | **pi2**–**pi3** | Manual **hourly** **systemd** **timer** (see README) | **`/var/spool/uptimed/records`**; SSH: **`piN.lan.buetow.org`**, **`-p 22`** | +| Raspberry Pi (NetBSD) | **pi0**–**pi1** | Manual **hourly** **root** **`cron`** (no systemd) calling **`goprecords-upload-client.sh`** with **`GOPRECORDS_HOST=pi0`**/**`pi1`** | **`/var/spool/uptimed/records`** (uptimed built from source — no prebuilt aarch64 pkgsrc package); SSH: **`piN.lan.buetow.org`**, **`-p 22`** | | Fedora laptop | **earth** | **user** **systemd** **`oneshot` + hourly timer** `goprecords-upload-earth.{service,timer}` | Service sets **`Environment=GOPRECORDS_HOST=earth`** and runs **`~/.local/bin/goprecords-upload-earth.sh`**; token **`~/.config/goprecords-upload-earth/token`** | | Mac (uptimed) → published by earth | **mega-m3-pro** (raw host `MBDVXJ4XKH9C`) | Mac drops records into the **worktime** git repo; **earth** pushes them via a **second `ExecStart`** in `goprecords-upload-earth.service` | See [Mac / mega-m3-pro via earth](#mac--mega-m3-pro-via-earth) below | diff --git a/prompts/skills/f3s/references/shelly-plug.md b/prompts/skills/f3s/references/shelly-plug.md index ebeff7b..6021921 100644 --- a/prompts/skills/f3s/references/shelly-plug.md +++ b/prompts/skills/f3s/references/shelly-plug.md @@ -93,7 +93,16 @@ wake via WoL when this was set up — deploy when it is back online). `wol-f3s` (dotfiles `scripts/wol-f3s`; deployed to `/home/paul/scripts/wol-f3s` on earth and `/usr/local/bin/wol-f3s` on pi0/pi1/pi2) controls the plug as part -of bulk power actions: +of bulk power actions. On `pi0`/`pi1` (NetBSD) this needed: pkgsrc `bash` +(already present as a dependency of other packages) and pkgsrc `wol` installed, +the shebang changed from `#!/bin/bash` to `#!/usr/pkg/bin/bash` on the deployed +copy (dotfiles' own copy for earth/Linux stays as-is), `~/.shelly_plug` copied +over (missing on a fresh image), and `/etc/hosts` entries for `f0`–`f3`/`pi2`–`pi3` +(cross-Pi/host `.lan.buetow.org` resolution isn't reliable — same DNS gap +noted elsewhere in this skill). End-to-end verified from `pi0` (2026-07-03): +woke `f3` via WoL, toggled the shelly plug off/on, shut `f3` back down via +`wol-f3s shutdown-f3` — all worked. Note single-host `wol-f3s f3`/`shutdown-f3` +does **not** touch the shelly plug (only the bulk `all`/`shutdown-all` paths do). - `wol-f3s` / `all` → `shelly_set true` **before** sending WoL packets (fans on). - `wol-f3s shutdown-all` → `shelly_set false` **after** all hosts/Pis are down diff --git a/prompts/skills/f3s/references/wireguard.md b/prompts/skills/f3s/references/wireguard.md index 126b86d..d34079c 100644 --- a/prompts/skills/f3s/references/wireguard.md +++ b/prompts/skills/f3s/references/wireguard.md @@ -9,16 +9,27 @@ Hybrid WireGuard topology connecting the f3s infrastructure mesh, two gateway-on - `r0`, `r1`, `r2` — Rocky Linux Bhyve VMs - `blowfish`, `fishfinger` — OpenBSD internet gateways (OpenBSD Amsterdam and Hetzner) -**Gateway-only peers** (connect only to gateways): -- `pi0` — Rocky Linux 9 on Raspberry Pi 3 (`192.168.2.203`) -- `pi1` — Rocky Linux 9 on Raspberry Pi 3 (`192.168.2.204`) +**Limited-peer nodes** (connect to the gateways, plus `rocky` — not full mesh): +- `pi0` — **NetBSD 10.1** on Raspberry Pi 3 (`192.168.2.203`) +- `pi1` — **NetBSD 10.1** on Raspberry Pi 3 (`192.168.2.204`) — converted after `pi0`, same procedure, both peers up successfully first try on `pi1` since the `pi0`-derived runbook already had all the gotchas baked in **Roaming clients** (connect only to gateways): - `earth` — Fedora laptop (192.168.2.200) - `pixel7pro` — Android phone (192.168.2.201) Even `fN <-> rN` tunnels exist (technically redundant since the VM runs on the host) to keep config uniform. -`pi0` and `pi1` are intentionally not full-mesh peers; they only establish tunnels to `blowfish` and `fishfinger`. +`pi0` and `pi1` are not full-mesh peers; each has exactly 3 peers: `blowfish`, `fishfinger`, and `rocky` (verified against both hosts' live configs — not gateway-only as older notes here claimed). + +### `pi0`/`pi1` (NetBSD): no native `wg(4)`, use `wireguard-go` instead + +The `wg` kernel module documented above does **not** ship in the evbarm-aarch64 10.1 module set (confirmed: absent from all 249 modules under `/stand/evbarm/10.1/modules`, so `ifconfig wg0 create` fails outright) — despite `wg(4)` being upstream NetBSD since 9.2, this platform/release combination just doesn't have it. Fixed with pkgsrc's `wireguard-go` (userspace) + `wireguard-tools` (`wg` CLI only — no `wg-quick` in this package) instead: + +- Interface must be named `tunN` (`wireguard-go` on NetBSD requires this — `wg0` is rejected: "Interface name must be tun[0-9]*"). Used `tun0`. +- Bring the interface up **and address it** (`ifconfig tun0 inet <ip> <ip> netmask 255.255.255.255`) *before* starting `wireguard-go`, or its read loop dies immediately with `EHOSTDOWN` ("host is down") and does not retry. +- `wg setconf tun0 <conf>` takes the normal `[Interface]`/`[Peer]` format (including `PersistentKeepalive`, unlike native `wgconfig` which has no keepalive flag at all) — same keys/PSKs as the `wg-quick`-format file `wireguardmeshgenerator` already renders to `dist/pi0/etc/wireguard/wg0.conf`, just fed to a different tool. +- No `wg-quick` means **no automatic routes**: each peer's AllowedIPs needs an explicit `route add -inet <ip>/32 <local-tun-ip> -iface` (and `-inet6` for the v6 ones) — `wg` only does the crypto/routing decision inside the tunnel, not the OS route table. +- All of this is wired into a custom `/etc/rc.d/wireguard` script (there's no stock rc.d for this) since there's no native `ifconfig.wg0`/wg-quick integration to hook into. +- Follow-up not yet done: `wireguardmeshgenerator.rb` only branches on `os == 'Linux' | 'FreeBSD' | 'OpenBSD'` and always emits `wg-quick`-style files; `wireguardmeshgenerator.yaml`'s `pi0:`/`pi1:` entries still say `os: Linux` with a `systemctl reload wg-quick@wg0.service` `reload_cmd`. Until the generator gains NetBSD support, both nodes' WireGuard configs are manually-maintained exceptions that a future `--generate`/`--install` regen would otherwise clobber. ## WireGuard IP Assignments @@ -33,8 +44,8 @@ Even `fN <-> rN` tunnels exist (technically redundant since the VM runs on the h | r2 | 192.168.2.122 | fd42:beef:cafe:2::122 | Rocky VM (k3s node) | | blowfish | 192.168.2.110 | fd42:beef:cafe:2::110 | OpenBSD internet GW | | fishfinger | 192.168.2.111 | fd42:beef:cafe:2::111 | OpenBSD internet GW | -| pi0 | 192.168.2.203 | fd42:beef:cafe:2::203 | Rocky Linux 9 on Raspberry Pi 3 (gateway-only peer) | -| pi1 | 192.168.2.204 | fd42:beef:cafe:2::204 | Rocky Linux 9 on Raspberry Pi 3 (gateway-only peer) | +| pi0 | 192.168.2.203 | fd42:beef:cafe:2::203 | NetBSD 10.1 on Raspberry Pi 3 (limited-peer: blowfish/fishfinger/rocky) | +| pi1 | 192.168.2.204 | fd42:beef:cafe:2::204 | NetBSD 10.1 on Raspberry Pi 3 (limited-peer: blowfish/fishfinger/rocky) | | earth | 192.168.2.200 | fd42:beef:cafe:2::200 | Fedora laptop (roaming) | | pixel7pro | 192.168.2.201 | fd42:beef:cafe:2::201 | Android phone (roaming) | @@ -54,7 +65,9 @@ doas service wireguard start doas wg show # check public key and listen port ``` -## Rocky Linux Setup (r0, r1, r2, pi0, pi1) +## Rocky Linux Setup (r0, r1, r2) + +(`pi0`/`pi1` used to follow this same setup but are now NetBSD — see "`pi0`/`pi1` (NetBSD): no native `wg(4)`" above instead.) ```sh dnf install -y wireguard-tools @@ -246,7 +259,7 @@ The script generates all configs and can push them via SSH. Current mesh-specific notes: -- `pi0` and `pi1` are defined as Rocky Linux hosts but excluded from all non-gateway peers, so they only tunnel to `blowfish` and `fishfinger` +- `pi0` and `pi1` are defined in the generator's YAML as Rocky Linux hosts (now stale — both are NetBSD; the generator has no NetBSD support yet, see above) and excluded from most non-gateway peers, so they only tunnel to `blowfish`, `fishfinger`, and `rocky` - Installed config ownership must be OS-specific: - Linux: `root:root` - BSD: `root:wheel` diff --git a/prompts/skills/pkgrepo/SKILL.md b/prompts/skills/pkgrepo/SKILL.md index 5155537..fff92e8 100644 --- a/prompts/skills/pkgrepo/SKILL.md +++ b/prompts/skills/pkgrepo/SKILL.md @@ -31,6 +31,6 @@ Use `f3s` alongside this skill when the task depends on broader host-role or clu - `f0` as the FreeBSD NFS/PV host for `/data/nfs/k3svolumes/pkgrepo/` - `fishfinger` and `blowfish` as the OpenBSD frontend hosts - `r0-r2` as Rocky Linux x86_64 bhyve VMs -- `pi0-pi3` as Rocky Linux aarch64 Raspberry Pi nodes +- `pi2-pi3` as Rocky Linux aarch64 Raspberry Pi nodes (`pi0`/`pi1` were reinstalled to NetBSD — see the `f3s` skill's `bootstrap-netbsd-pi.md` — and are no longer Rocky package-repo clients) - `earth` as the Fedora laptop used for package publication and verification - `f0-f3` as FreeBSD hosts diff --git a/prompts/skills/pkgrepo/references/client-setup.md b/prompts/skills/pkgrepo/references/client-setup.md index 5e231ff..67850ab 100644 --- a/prompts/skills/pkgrepo/references/client-setup.md +++ b/prompts/skills/pkgrepo/references/client-setup.md @@ -83,11 +83,13 @@ OpenBSD packages are signed with `signify(1)` via `pkg_sign`: - Signing happens automatically during `make pkg-openbsd` / `make pkg` - `pkg_add` verifies the signature — no `-D unsigned` needed -## Rocky Linux (r0–r2, pi0–pi3) +## Rocky Linux (r0–r2, pi2–pi3) + +`pi0`/`pi1` were reinstalled to NetBSD (see `f3s` skill's `bootstrap-netbsd-pi.md`) and are no longer Rocky clients of this repo. Architecture-specific repo URLs: - `https://pkgrepo.f3s.buetow.org/rockylinux/9/x86_64/` (r0–r2) -- `https://pkgrepo.f3s.buetow.org/rockylinux/9/aarch64/` (pi0–pi3) +- `https://pkgrepo.f3s |