diff options
Diffstat (limited to 'prompts')
| -rw-r--r-- | prompts/skills/f3s/SKILL.md | 18 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/bootstrap-netbsd-pi.md | 252 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/dtail.md | 2 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/shelly-plug.md | 21 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/wireguard.md | 4 | ||||
| -rw-r--r-- | prompts/skills/pkgrepo/SKILL.md | 2 | ||||
| -rw-r--r-- | prompts/skills/pkgrepo/references/client-setup.md | 2 | ||||
| -rw-r--r-- | prompts/skills/pkgrepo/references/dtail-package.md | 6 | ||||
| -rw-r--r-- | prompts/skills/pkgrepo/references/packaging-workflow.md | 2 |
9 files changed, 133 insertions, 176 deletions
diff --git a/prompts/skills/f3s/SKILL.md b/prompts/skills/f3s/SKILL.md index fb90871..83adb3e 100644 --- a/prompts/skills/f3s/SKILL.md +++ b/prompts/skills/f3s/SKILL.md @@ -1,6 +1,6 @@ --- name: f3s -description: Reference skill for the f3s homelab—four Beelink S12 Pro hosts (f0/f1/f2/f3) running FreeBSD with Rocky Linux Bhyve VMs and a k3s Kubernetes cluster. f0/f1/f2 run r0/r1/r2 k3s nodes; f3 is standalone bhyve only (not part of k3s) and hosts the plain Rocky Linux VM named rocky. Four Raspberry Pi 3 nodes (pi0–pi3): pi0 and pi1 are NetBSD 10.1 (the static f3s.buetow.org/snonux.foo HTTP pair), pi2/pi3 are still Rocky Linux 9 running Pi-hole (Docker) and LAN wildcard DNS for *.f3s.lan.buetow.org. Covers DTail/dserver on Pis (arm64) and k3s VMs (amd64). Use when troubleshooting or making configuration decisions for the f3s setup. +description: "Reference skill for the f3s homelab—four Beelink S12 Pro hosts (f0/f1/f2/f3) running FreeBSD with Rocky Linux Bhyve VMs and a k3s Kubernetes cluster. f0/f1/f2 run r0/r1/r2 k3s nodes; f3 is standalone bhyve only (not part of k3s) and hosts the plain Rocky Linux VM named rocky. Four Raspberry Pi 3 nodes (pi0–pi3): pi0 and pi1 run NetBSD 10.1 (the static f3s.buetow.org/snonux.foo HTTP pair), pi2/pi3 run Rocky Linux 9 with Pi-hole (Docker) and LAN wildcard DNS for *.f3s.lan.buetow.org. Covers DTail/dserver on Pis (arm64) and k3s VMs (amd64). Use when troubleshooting or making configuration decisions for the f3s setup." --- # f3s Homelab Reference @@ -24,7 +24,7 @@ Detailed reference documentation is in the `references/` subfolder: - [Rocky Linux VMs](references/rocky-linux-vms.md) — Bhyve, vm-bhyve, VM config, NVMe disk fix; FreeBSD VM on f3 (migrated from f0) - [f3 Rocky VM](references/f3-rocky-vm.md) — Plain Rocky Linux 9 VM on f3 (`rocky`, `192.168.1.123`), autostart policy, root SSH - [Bootstrap Rocky bhyve VM](references/bootstrap-rocky-bhyve.md) — Runbook for creating a new plain Rocky Linux bhyve guest with unattended kickstart -- [Bootstrap NetBSD Pi](references/bootstrap-netbsd-pi.md) — Runbook for converting a Rocky Pi to NetBSD, validated twice now (`pi0` then `pi1`, both reboot-tested): doas/pkgin bootstrap, WireGuard via userspace `wireguard-go` (no native `wg(4)` on this platform), bozohttpd (`-X` for dir-listing parity, vhost symlinks for every real routed hostname from day one), uptimed built from source, npf firewall, content-sync direction. Only `pi2`/`pi3` (Pi-hole) remain Rocky. +- [NetBSD Pi Setup](references/bootstrap-netbsd-pi.md) — How services are installed on `pi0`/`pi1` (NetBSD): doas/pkgin bootstrap, WireGuard via userspace `wireguard-go` (no native `wg(4)` on this platform), bozohttpd (`-X` for dir-listing parity, vhost symlinks for every real routed hostname), uptimed built from source, npf firewall, content-sync setup. - [WireGuard Mesh](references/wireguard.md) — Mesh topology, IP assignments, peer configs - [Storage](references/storage.md) — index into `references/storage/`: ZFS (zdata), zrepl, CARP, NFS over stunnel, nfs-mount-monitor, troubleshooting (incl. thermal), backups & local-path - [r-node Deploy (Rex)](references/r-node-deploy.md) — reusable Rex rollout to **r0/r1/r2** (`f3s/r-nodes/Rexfile`, task `nfs_mount_monitor`): root SSH, `parallelism 3`, idempotent `file`/`on_change` reload, verify with `systemctl`/`journalctl` @@ -68,7 +68,7 @@ The plain Rocky Linux VM on f3 (`rocky`, `192.168.1.123`) is documented in the s ## Raspberry Pi Nodes -`pi2`/`pi3` run Rocky Linux 9.2 (Blue Onyx) aarch64 from the SIG/AltArch image (`RockyLinuxRpi_9-latest.img.xz`). `pi0` and `pi1` were both reinstalled to **NetBSD 10.1** (evbarm-aarch64) — `pi0` first (2026-07-03), `pi1` the same day once `pi0` was validated. Each Rocky Pi has: +`pi2`/`pi3` run Rocky Linux 9.2 (Blue Onyx) aarch64 from the SIG/AltArch image (`RockyLinuxRpi_9-latest.img.xz`). `pi0` and `pi1` run **NetBSD 10.1** (evbarm-aarch64). Each Rocky Pi has: - User `paul` with passwordless sudo and SSH key auth - Static IP on eth0 via NetworkManager @@ -78,23 +78,23 @@ The plain Rocky Linux VM on f3 (`rocky`, `192.168.1.123`) is documented in the s - No GRUB — boots via Pi's native bootloader (`/boot/cmdline.txt`) - Custom RPi kernel from the `rockyrpi` repo -`pi0`/`pi1` (NetBSD) differ: user `paul` in `wheel`, privilege escalation via a **real `doas`** (pkgsrc `security/doas`, `permit nopass :wheel`) — not the `alias doas=sudo` shell alias `pi2`/`pi3` carry in `/etc/profile.d/doas.sh`, which doesn't expand in the non-interactive shell an SSH command runs in and so silently breaks `wol-f3s shutdown-pis`/`shutdown-all` for the Rocky Pis (`doas poweroff` resolves to nothing) — only the NetBSD nodes actually work with that script today. Config repo home for NetBSD-specific setup: `f3s/pi-netbsd/`. Full conversion runbook: [Bootstrap NetBSD Pi](references/bootstrap-netbsd-pi.md). +`pi0`/`pi1` (NetBSD) differ: user `paul` in `wheel`, privilege escalation via a **real `doas`** (pkgsrc `security/doas`, `permit nopass :wheel`) — not the `alias doas=sudo` shell alias `pi2`/`pi3` carry in `/etc/profile.d/doas.sh`, which doesn't expand in the non-interactive shell an SSH command runs in and so silently breaks `wol-f3s shutdown-pis`/`shutdown-all` for the Rocky Pis (`doas poweroff` resolves to nothing) — only the NetBSD nodes actually work with that script today. Config repo home for NetBSD-specific setup: `f3s/pi-netbsd/`. Service setup details: [NetBSD Pi Setup](references/bootstrap-netbsd-pi.md). Current role split: -- `pi0` and `pi1` serve static `f3s.buetow.org`/`snonux.foo` content behind OpenBSD `relayd` over WireGuard. WireGuard peers are `blowfish`, `fishfinger`, **and `rocky`** (not gateway-only to just the two frontends, despite older docs here). Both nodes reboot-tested (2026-07-03): all rc.d services (`wireguard`, `bozohttpd`, `uptimed`, `npf`) and both crontabs come back automatically. +- `pi0` and `pi1` serve static `f3s.buetow.org`/`snonux.foo` content behind OpenBSD `relayd` over WireGuard. WireGuard peers are `blowfish`, `fishfinger`, **and `rocky`** (not gateway-only to just the two frontends, despite older docs here). All rc.d services (`wireguard`, `bozohttpd`, `uptimed`, `npf`) and both crontabs are enabled via `rc.conf` and come back automatically on reboot. - `pi2` and `pi3` run **Pi-hole** in Docker (`network_mode: host`, `~/pihole` on each host). Tracked dnsmasq LAN wildcard: **`f3s/pihole/docker-pi/`** in the conf repo; details in [references/pihole-pi.md](references/pihole-pi.md). ### Webserver Configuration -Both `pi0` and `pi1` run **bozohttpd** (built into NetBSD base, no package/config file) via a custom `/etc/rc.d/bozohttpd` (the stock rc.d/httpd script ignores `httpd_flags` entirely — never references it — so flags had to go directly in this script's own `command_args`), using `-v`/`-V` for vhosting instead of lighttpd's `$HTTP["host"]` regex match — a vhost needs a directory *literally* named after the hostname (e.g. `snonux.foo/`, with `www.snonux.foo` a symlink to it). Also needs **`-X`** (directory indexing) to match lighttpd's old `dir-listing.activate = "enable"` — without it, bare directories with no `index.html` (e.g. `/fotos/<gallery>/` at the top level) 404 instead of showing a listing; caught on `pi0` by a live redundancy test (stopped the other node's webserver, `curl`'d every page through the public domains) and included from the start on `pi1`. +Both `pi0` and `pi1` run **bozohttpd** (built into NetBSD base, no package/config file) via a custom `/etc/rc.d/bozohttpd` (the stock rc.d/httpd script ignores `httpd_flags` entirely — never references it — so flags had to go directly in this script's own `command_args`), using `-v`/`-V` for vhosting instead of lighttpd's `$HTTP["host"]` regex match — a vhost needs a directory *literally* named after the hostname (e.g. `snonux.foo/`, with `www.snonux.foo` a symlink to it). Also needs **`-X`** (directory indexing) — without it, bare directories with no `index.html` (e.g. `/fotos/<gallery>/` at the top level) 404 instead of showing a listing. -**bozohttpd `-V` fallback bug** (hit and fixed on `pi0`, avoided on `pi1` since the fix was already baked in): `f3s.buetow.org` (the real routed hostname — relayd forwards `f3s.buetow.org`/`www.f3s.buetow.org`/`standby.f3s.buetow.org`, all Host-matched via `match request header "Host" value ... forward to <f3s_static_proxy>` in `relayd.conf` on the frontends; there is no separate `scifi.f3s.buetow.org` subdomain, `/scifi/` is just a **path** under it) had no dedicated vhost directory, so it hit the `-V` fallback — and bozohttpd's directory-without-trailing-slash redirect in that fallback path uses its own **system hostname** (e.g. `pi0.lan.buetow.org`) instead of the client's `Host:` header, unlike a vhost-*matched* request (which correctly echoes back the matched name, e.g. `snonux.foo`). Since the system hostname doesn't resolve outside the LAN, external clients following that redirect would hang. **Fixed** with self-referencing vhost symlinks so these hostnames become vhost matches instead of fallbacks: `ln -sf . /var/www/html/f3s.buetow.org`, same for `www.f3s.buetow.org` and `standby.f3s.buetow.org`. +**bozohttpd `-V` fallback note**: `f3s.buetow.org` (the real routed hostname — relayd forwards `f3s.buetow.org`/`www.f3s.buetow.org`/`standby.f3s.buetow.org`, all Host-matched via `match request header "Host" value ... forward to <f3s_static_proxy>` in `relayd.conf` on the frontends; there is no separate `scifi.f3s.buetow.org` subdomain, `/scifi/` is just a **path** under it) needs its own vhost directory — without one it hits the `-V` fallback, and bozohttpd's directory-without-trailing-slash redirect in that fallback path uses its own **system hostname** (e.g. `pi0.lan.buetow.org`) instead of the client's `Host:` header, unlike a vhost-*matched* request (which correctly echoes back the matched name, e.g. `snonux.foo`). Since the system hostname doesn't resolve outside the LAN, external clients following that redirect would hang. Fixed with self-referencing vhost symlinks so these hostnames become vhost matches instead of fallbacks: `ln -sf . /var/www/html/f3s.buetow.org`, same for `www.f3s.buetow.org` and `standby.f3s.buetow.org`. -- Document root: `/var/www/html` (same path both nodes now — the historical `/var/www/html/snonux` vs `/var/www/html/snonux.foo` naming difference no longer applies since `pi1` was reinstalled and now mirrors `pi0`'s tree exactly) +- Document root: `/var/www/html` (same path on both nodes) - SSH access: `ssh paul@piN.lan.buetow.org -p 22` - Host-based virtual hosting maps domains to subdirectories: `snonux.foo` / `www.snonux.foo` → `/var/www/html/snonux.foo` -- Content sync direction flipped mid-migration: `pi1` → `pi0` while `pi1` was still the only node with real content (Rocky), then reversed to `pi0` → `pi1` once `pi1` was also reinstalled (`pi0` is now the long-lived source of truth). `pi1`'s hourly pull is `/usr/local/bin/sync-from-pi0.sh` (cron `:47`); `pi0`'s old `sync-from-pi1.sh` (cron `:17`) is a harmless now-pointless leftover pulling from a node with no independent content anymore. +- `pi1` pulls content hourly from `pi0` (`/usr/local/bin/sync-from-pi0.sh`, cron `:47`) — `pi0` is the source of truth for the docroot. **Why Host-based vhosts?** `relayd` on the OpenBSD frontends cannot rewrite URL paths. It forwards requests with the original path intact. To serve a subdirectory as root for a domain, lighttpd must remap the document root based on the `Host` header. diff --git a/prompts/skills/f3s/references/bootstrap-netbsd-pi.md b/prompts/skills/f3s/references/bootstrap-netbsd-pi.md index 411b059..2d80375 100644 --- a/prompts/skills/f3s/references/bootstrap-netbsd-pi.md +++ b/prompts/skills/f3s/references/bootstrap-netbsd-pi.md @@ -1,65 +1,31 @@ -# Bootstrapping a Raspberry Pi 3 f3s node on NetBSD +# NetBSD services on pi0/pi1 -Runbook for bringing a Rocky Pi node up to NetBSD parity, validated twice now: -`pi0` first (2026-07-03), then `pi1` the same day using this exact procedure — -both reboot-tested and confirmed durable. Only `pi2`/`pi3` (Pi-hole) remain -Rocky; this is the reusable procedure for converting either of them next. +`pi0` and `pi1` run NetBSD 10.1 (evbarm-aarch64). This documents how their +services are installed and configured — useful reference for troubleshooting, +rebuilding a service, or reinstalling either node. -**Do this one node at a time.** Never take down two of `pi0`–`pi3`'s static-HTTP -pair (or the two Pi-hole nodes) simultaneously — one must always keep serving. +**Do this one node at a time.** Never take down both of `pi0`/`pi1` (the +static-HTTP pair) simultaneously — one must always keep serving +`f3s.buetow.org`/`snonux.foo`. -## 0. Before you reinstall: back up state that doesn't come from git - -None of this lives in the conf repo — it only exists on the running Pi, so it -must be pulled off **before** wiping it: - -- **Uptime history**: `scp` the target's `/var/spool/uptimed/records` (and - `.old` if present) somewhere safe (e.g. `f0.lan.buetow.org:/tmp/<host>.records`, - the same pattern used for `pi0`). Restoring it later preserves the uptime - streak instead of resetting to zero — see step 6. -- **WireGuard identity**: not actually needed from the live host — the - private key and PSKs already live in `~/git/wireguardmeshgenerator/keys/` - (`keys/<host>/priv.key`, `keys/psk/<pair>.key`). Reuse those; do not - regenerate, or every peer (`blowfish`, `fishfinger`, `rocky`) needs updating. -- **Static content**: only relevant if this node is currently the source of - truth for `/var/www/html`. Once `pi0` is up and holding the full synced - tree, `pi0` becomes the natural source for converting `pi1` next — no - separate backup needed, just sync FROM `pi0` afterward (see step 4). - -## 1. Flash/image plain NetBSD - -Not yet documented in this repo — `f3s/pi-netbsd/bake` and `flash/` exist as -scaffolding in the conf repo but were empty as of 2026-07-03; the imaging -process used for `pi0` happened outside this repo. Whatever process is used, -the expected end state (confirmed on `pi0`) is: +## Base state - NetBSD 10.1 `GENERIC64` evbarm64 (aarch64) -- User `paul`, in group `wheel`, SSH key auth already working -- Static LAN IP already set via `rc.conf` (`ifconfig_mue0="inet 192.168.1.12N - netmask 0xffffff00"`, `defaultroute="192.168.1.1"` — match the hardware - table's IP for this host) +- User `paul`, in group `wheel`, SSH key auth +- Static LAN IP via `rc.conf` (`ifconfig_mue0="inet 192.168.1.12N netmask + 0xffffff00"`, `defaultroute="192.168.1.1"`) - Hostname set (`hostname="piN.lan.buetow.org"`) -- **No** `doas`/`sudo`, **no** pkgsrc/pkgin bootstrapped — bare base install -- `paul` has no private SSH key of its own yet (only `authorized_keys`) - -If any of these aren't true after flashing, do them first (they're baked into -the disk image for `pi0`, presumably via sysinst answers or a post-install -script — worth capturing properly in `f3s/pi-netbsd/` once that process is -pinned down). +- No `doas`/`sudo`, no pkgsrc/pkgin by default — bootstrapped below +- A pre-baked root crontab entry for the hourly goprecords upload already + points at `/usr/pkg/bin/goprecords-upload-client.sh` with `GOPRECORDS_HOST` + set correctly — check `doas crontab -l` before deploying that script + manually to a different path. -**Note found on `pi0`**: the image already carried a **pre-baked root -crontab** entry for the hourly goprecords upload, pointing at -`/usr/pkg/bin/goprecords-upload-client.sh` with `GOPRECORDS_HOST` already set -correctly. Check `doas crontab -l` early — if it's there, deploy the script to -match that path instead of improvising a different location. - -## 2. Bootstrap pkgin + real doas +## Bootstrap pkgin + real doas ```sh ssh paul@piN.lan.buetow.org -su - # root password from the bake; rotate both root and - # paul's passwords once doas works (step below) — - # not automated, do it by hand +su - export PKG_PATH=https://cdn.NetBSD.org/pub/pkgsrc/packages/NetBSD/aarch64/10.1/All/ pkg_add -v pkgin pkgin -y update @@ -80,33 +46,32 @@ doas true # should succeed with no password prompt non-interactive shell an SSH command runs in — so `~/git/dotfiles/scripts/wol-f3s`'s `shutdown-pis`/`shutdown-all` (which runs `ssh paul@pi "doas poweroff"`) is silently broken on the Rocky Pis today. A -real `doas` binary fixes it for whichever Pi gets NetBSD. +real `doas` binary is why it works on `pi0`/`pi1`. **Gotcha**: commands run via `doas` get a minimal `PATH` that excludes `/usr/sbin` and `/usr/pkg/bin` — always use full paths (`doas /usr/sbin/chown`, `doas /usr/pkg/bin/wg`) or an explicit `PATH=` for cron. -## 3. WireGuard (userspace — no native `wg(4)` on this platform) +## WireGuard (userspace — no native `wg(4)` on this platform) -**`wg(4)` doesn't exist on evbarm-aarch64 10.1** — confirmed on `pi0`: the -module is absent from all 249 files under `/stand/evbarm/10.1/modules`, so -`ifconfig wg0 create` fails outright (`clone_command: Invalid argument`), -despite `wg(4)` being upstream NetBSD since 9.2. Don't waste time on it; -`wireguard-go` + `wg` (pkgsrc `wireguard-tools`, **no `wg-quick`** in this -build) is the working path: +**`wg(4)` doesn't exist on evbarm-aarch64 10.1** — the module is absent from +all 249 files under `/stand/evbarm/10.1/modules`, so `ifconfig wg0 create` +fails outright (`clone_command: Invalid argument`), despite `wg(4)` being +upstream NetBSD since 9.2. Don't waste time on it; `wireguard-go` + `wg` +(pkgsrc `wireguard-tools`, **no `wg-quick`** in this build) is the working +path: ```sh pkgin -y install wireguard-go wireguard-tools ``` -Pull this host's existing private key and the three PSKs (`blowfish`, -`fishfinger`, `rocky`) from `~/git/wireguardmeshgenerator/keys/` — do not -regenerate. Also grab peer pubkeys/endpoints/AllowedIPs from -`dist/<host>/etc/wireguard/wg0.conf` if the generator has already rendered -them, or from another live node's `/etc/wireguard/wg0.conf` for the endpoint -details (same for every Pi, only `AllowedIPs`/PSK differ per host). +Pull this host's private key and the three PSKs (`blowfish`, `fishfinger`, +`rocky`) from `~/git/wireguardmeshgenerator/keys/` — do not regenerate. +Peer pubkeys/endpoints/AllowedIPs come from `dist/<host>/etc/wireguard/wg0.conf` +(generator output) or another live node's `/etc/wireguard/wg0.conf` (same +endpoints for every Pi, only `AllowedIPs`/PSK differ per host). -Key facts that cost time on `pi0`: +Key facts: - The interface **must** be named `tunN` (`wireguard-go` rejects `wg0`: "Interface name must be tun[0-9]*"). Use `tun0`. @@ -135,23 +100,20 @@ Key facts that cost time on `pi0`: ``` Wire all of this into a custom `/etc/rc.d/wireguard` (there's no stock rc.d -for this combination — write one, model it on `pi0`'s). Enable with -`wireguard=YES` in `/etc/rc.conf`. +for this combination). Enable with `wireguard=YES` in `/etc/rc.conf`. -**Follow-up not yet done for any host**: `wireguardmeshgenerator.rb` only -branches on `os == 'Linux' | 'FreeBSD' | 'OpenBSD'` and always emits -`wg-quick`-style files; the generator's YAML still lists NetBSD hosts as -`os: Linux`. Until it gains NetBSD support, each converted Pi's WireGuard -config is a manually-maintained exception that a future -`--generate`/`--install` regen would clobber. +**Known gap**: `wireguardmeshgenerator.rb` only branches on +`os == 'Linux' | 'FreeBSD' | 'OpenBSD'` and always emits `wg-quick`-style +files; the generator's YAML still lists `pi0`/`pi1` as `os: Linux`. Until it +gains NetBSD support, both nodes' WireGuard configs are manually-maintained +exceptions that a future `--generate`/`--install` regen would clobber. -## 4. Webserver — bozohttpd +## Webserver — bozohttpd -Built into NetBSD base, matches the "static-only, low-RAM" reasoning already -used to justify lighttpd on the Rocky Pis. No stock rc.d exists that actually -uses `httpd_flags` (the shipped `/etc/rc.d/httpd` computes `command_args` -itself and never references that variable) — write a dedicated -`/etc/rc.d/bozohttpd`: +Built into NetBSD base, no package or config file. No stock rc.d exists that +actually uses `httpd_flags` (the shipped `/etc/rc.d/httpd` computes +`command_args` itself and never references that variable) — write a +dedicated `/etc/rc.d/bozohttpd`: ```sh command="/usr/libexec/httpd" @@ -167,39 +129,29 @@ required_dirs="/var/www/html" - `www.snonux.foo` needs to be a symlink to `snonux.foo` (bozohttpd matches the literal Host header as a directory name, not a regex like lighttpd's `$HTTP["host"] =~ "^(www\.)?snonux\.foo$"`). -- **`-X` (directory indexing) is required**, not optional — caught by a live - redundancy test (stopping the other node's lighttpd and curling every page): - bare directories with no `index.html` (e.g. a photo gallery folder under - `/fotos/`) 404 without it, because lighttpd has - `dir-listing.activate = "enable"` and bozohttpd needs the equivalent flag. +- **`-X` (directory indexing) is required**, not optional: bare directories + with no `index.html` (e.g. a photo gallery folder under `/fotos/`) 404 + without it. - **Give every real routed hostname its own vhost entry, even the "default" - one** — don't rely on `-V` fallback for anything actually reachable from the - internet. `f3s.buetow.org` (checked in `relayd.conf` on the frontends: the - real routed names are `f3s.buetow.org`, `www.f3s.buetow.org`, + one** — don't rely on `-V` fallback for anything actually reachable from + the internet. `f3s.buetow.org` (checked in `relayd.conf` on the frontends: + the real routed names are `f3s.buetow.org`, `www.f3s.buetow.org`, `standby.f3s.buetow.org` — `/scifi/` etc. are **paths** under it, not - separate subdomains) had no vhost dir, so it hit `-V`, and bozohttpd's - directory-without-trailing-slash redirect in that fallback path uses its own - **system hostname**, not the client's `Host:` header (unlike a real vhost - match, which correctly echoes back e.g. `snonux.foo`). Since the system - hostname (`piN.lan.buetow.org`) doesn't resolve outside the LAN, this - produced redirects that hang for external clients. Fix: self-referencing - symlinks so these become vhost matches instead of fallbacks — - `ln -sf . /var/www/html/f3s.buetow.org` (and the `www.`/`standby.` variants). + separate subdomains) needs a vhost dir, or it hits `-V`, and bozohttpd's + directory-without-trailing-slash redirect in that fallback path uses its + own **system hostname**, not the client's `Host:` header (unlike a real + vhost match, which correctly echoes back e.g. `snonux.foo`). Since the + system hostname (`piN.lan.buetow.org`) doesn't resolve outside the LAN, + this produces redirects that hang for external clients. Fix: + self-referencing symlinks so these become vhost matches instead of + fallbacks — `ln -sf . /var/www/html/f3s.buetow.org` (and the + `www.`/`standby.` variants). Enable with `bozohttpd=YES` in `/etc/rc.conf`. -## 5. Static content sync - -Direction depends on which Pi is being converted and what's already been -migrated: +## Static content sync -- Converting the **first** Rocky Pi in a same-role pair (this was `pi0`): - pull from the Rocky sibling that's staying up (`pi1` for `pi0`). -- Converting the **second** one (`pi1`, once `pi0` is already NetBSD and - holds the full current tree): reverse it — pull from `pi0` instead. - -One-time bring-up + hourly cron, same script either way (just swap source -host): +`pi0` is the source of truth for `/var/www/html`; `pi1` pulls hourly: ```sh #!/bin/sh @@ -207,22 +159,24 @@ set -e STAGE=/tmp/wwwsync-cron mkdir -p "$STAGE" rsync -a --delete -e "ssh -o StrictHostKeyChecking=accept-new" \ - paul@<source-host>.lan.buetow.org:/var/www/html/ "$STAGE/" -doas rsync -a --delete "$STAGE/index.html" "$STAGE/fotos" "$STAGE/scifi" /var/www/html/ -doas rsync -a --delete "$STAGE/snonux/" /var/www/html/snonux.foo/ -doas /usr/sbin/chown -R root:wheel /var/www/html + paul@pi0.lan.buetow.org:/var/www/html/ "$STAGE/" +doas rsync -a --delete --exclude=snonux.foo "$STAGE/" /var/www/html/ +doas rsync -a --delete "$STAGE/snonux.foo/" /var/www/html/snonux.foo/ +doas /usr/sbin/chown -R root:wheel /var/www/html/index.html /var/www/html/fotos /var/www/html/scifi +# snonux.foo is owned by paul, not root: the snonux microblog tool rsyncs +# directly into it as paul (no doas hop), so it must stay paul-writable. +doas /usr/sbin/chown -R paul:wheel /var/www/html/snonux.foo ``` -Needs a fresh SSH keypair for `paul` on this host (none exists on a bare -image), authorized on the source host's `~/.ssh/authorized_keys`, plus a -static `/etc/hosts` entry for the source hostname (Pi-to-Pi `.lan.buetow.org` -resolution failed for both `pi0`→`pi1` and originally `pi0`→`pi1` during the -very first Rocky bring-up — add the IP directly rather than debugging DNS). +Needs an SSH keypair for `paul` on `pi1`, authorized on `pi0`'s +`~/.ssh/authorized_keys`, plus a static `/etc/hosts` entry for `pi0` (Pi-to-Pi +`.lan.buetow.org` resolution isn't reliable — add the IP directly rather +than debugging DNS). -Install as `paul`'s crontab (not root's — needs the new SSH key): -`17 * * * * /usr/local/bin/sync-from-<source>.sh >$HOME/sync-from-<source>.log 2>&1` +Install as `paul`'s crontab on `pi1` (not root's — needs the SSH key): +`47 * * * * /usr/local/bin/sync-from-pi0.sh >$HOME/sync-from-pi0.log 2>&1` -## 6. uptimed (built from source — no prebuilt package) +## uptimed (built from source — no prebuilt package) **No aarch64 binary package exists** in pkgsrc for `uptimed` on any branch checked (10.0, 10.1, 11.0, 9.4). Build from upstream instead — small C @@ -240,12 +194,11 @@ doas env PATH=/usr/pkg/bin:/usr/bin:/bin:/usr/sbin:/sbin make install ``` Installs `uptimed` to `/usr/pkg/sbin`, `uprecords` to `/usr/pkg/bin`, and uses -`/var/spool/uptimed/records` (matches the Rocky path — this is hardcoded -upstream, not an OS convention thing). +`/var/spool/uptimed/records` (hardcoded upstream, not an OS convention thing). **Before first start**, write `/etc/uptimed.conf` with `LOG_MAXIMUM_ENTRIES=0` -(keep forever — copy the Rocky sibling's `/etc/uptimed.conf` verbatim, -milestones included) and restore the backed-up history from step 0: +(keep forever) plus milestone lines. If restoring a backed-up uptime history, +seed **both** `records` and `records.old` with the same content: ```sh doas cp <backed-up-records-file> /var/spool/uptimed/records @@ -272,7 +225,7 @@ command_args="-p ${pidfile}" Run `uptimed -b` once (creates the boot ID), enable with `uptimed=YES`. -## 7. goprecords upload +## goprecords upload ```sh kubectl exec -n services deployment/goprecords -- \ @@ -285,16 +238,14 @@ retry, it's a transient networking issue, not a token problem.) Deploy `goprecords-upload-client.sh` (from `~/git/goprecords/scripts/`, already POSIX/generic and already handles `/var/spool/uptimed/records` and a NetBSD `dmesg.boot`/`sysctl` fallback for `os.txt`/`cpuinfo.txt` — no changes -needed) to **`/usr/pkg/bin/`** if the pre-baked crontab expects that path (see -the note in step 1), token at `/etc/goprecords-upload.token` (`0600`), -`GOPRECORDS_HOST=<host>`. +needed) to **`/usr/pkg/bin/`** to match the pre-baked crontab's path, token at +`/etc/goprecords-upload.token` (`0600`), `GOPRECORDS_HOST=<host>`. `curl` and `uprecords` need to be resolvable via whatever `PATH` the cron -entry sets — if it's the pre-baked one with an explicit `PATH=` line -including `/usr/pkg/bin`, no symlinking needed; test with that exact `PATH` -before trusting a manual test run under plain `doas` (which won't have it). +entry sets — test with that exact `PATH` before trusting a manual test run +under plain `doas` (which won't have it). -## 8. Firewall — npf, not firewalld +## Firewall — npf, not firewalld ``` $ext_if = "mue0" @@ -303,6 +254,7 @@ group "external" on $ext_if { pass stateful out final all pass stateful in final family inet4 proto tcp to $ext_if port 22 pass stateful in final family inet4 proto tcp to $ext_if port 80 + pass stateful in final family inet4 proto icmp all } group "wireguard" on tun0 { @@ -318,13 +270,19 @@ group default { ``` `family inet4`/`inet6` must be explicit on multi-family interfaces or -`npfctl validate` fails with "address family mismatch". Sequence carefully to -avoid locking yourself out over SSH: +`npfctl validate` fails with "address family mismatch". `proto <name>` must +be followed by `all` or a `from`/`to` clause, or it's a syntax error — e.g. +`proto icmp` alone fails, `proto icmp all` doesn't. Don't forget the ICMP +rule: without it, ping-dependent tooling (e.g. the `snonux` publishing +tool's reachability pre-check) silently breaks while SSH/HTTP keep working +fine. + +Sequence carefully to avoid locking yourself out over SSH: ```sh doas npfctl validate # syntax-check first doas npfctl reload # loads config, does NOT enable filtering yet -doas npfctl start # enables filtering +doas npfctl start # enables filtering # from a FRESH ssh connection (not the one you're already in), confirm: # - ssh still connects # - curl http://localhost/ still works @@ -336,21 +294,19 @@ permitted`) — harmless, just means `kern.securelevel` blocks loading that optional performance module; filtering still works, just slightly slower packet matching. -## 9. Verification +## Verification - `curl -fsI http://<host>.lan.buetow.org/` and the vhost via `Host:` header. - `wg show tun0` shows recent handshakes with `blowfish` and `fishfinger` (and - `rocky` if that VM happens to be up — it's often not, unrelated to this - work). + `rocky` if that VM happens to be up — it's often not, unrelated to this). - goprecords report (`https://goprecords.f3s.buetow.org/report`) picks up the - new host after the hourly cron fires (won't rank in the "top 20 all-time" - table with a fresh/short history — that's expected, not a failure). -- **Real redundancy test** (this is what actually caught the missing `-X` - flag on `pi0` — don't skip it): stop the *other* node in the pair's - webserver entirely, then curl every real page through the **public** - domains (not just localhost) — root page, each vhost, and any bare - directory paths (e.g. `/fotos/`). Restore the other node's webserver - immediately after. + host after the hourly cron fires (won't rank in the "top 20 all-time" table + with a short history — that's expected, not a failure). +- **Redundancy test**: stop the *other* node's webserver entirely, then curl + every real page through the **public** domains (not just localhost) — root + page, each vhost, and any bare directory paths (e.g. `/fotos/`). Restore + the other node's webserver immediately after. - `wol-f3s shutdown-pis` (or a targeted `ssh paul@<host> "doas poweroff"`) - actually powers the Pi off — confirms the real-`doas` fix, but there's no - WoL for Pis, so only do this when you can physically power it back on. + actually powers the Pi off — confirms `doas` works non-interactively, but + there's no WoL for Pis, so only do this when you can physically power it + back on. diff --git a/prompts/skills/f3s/references/dtail.md b/prompts/skills/f3s/references/dtail.md index 63941f1..9e77f0b 100644 --- a/prompts/skills/f3s/references/dtail.md +++ b/prompts/skills/f3s/references/dtail.md @@ -12,7 +12,7 @@ Upstream install and examples live in the repo: `doc/installation.md`, `examples | **r0–r2** | Rocky Linux 9 **x86_64** (bhyve VMs, k3s nodes) | Cross-build **linux/amd64**, `nozstd` | Often `root@rN.lan.buetow.org` (see [Rocky Linux VMs](rocky-linux-vms.md)); add `root` (and `paul` if present) to **Server.Permissions.Users** in `dtail.json` | | **blowfish, fishfinger** | OpenBSD 7.8 **amd64** | Native OpenBSD package build | `rex@blowfish.buetow.org`, `rex@fishfinger.buetow.org` | -`pi0`/`pi1` are **NetBSD** now (see `bootstrap-netbsd-pi.md`) and deliberately do **not** run DTail — would need an untested `GOOS=netbsd GOARCH=arm64` cross-build and an `rc.d` script. +`pi0`/`pi1` run **NetBSD** (see `bootstrap-netbsd-pi.md`) and do **not** run DTail — would need an untested `GOOS=netbsd GOARCH=arm64` cross-build and an `rc.d` script. **Key cache filenames matter:** `examples/update_key_cache.sh.example` only scans `/home/*` and writes `/var/run/dserver/cache/USER.authorized_keys`. In this lab, DTail auth worked only after writing the exact cache filename for the login user: diff --git a/prompts/skills/f3s/references/shelly-plug.md b/prompts/skills/f3s/references/shelly-plug.md index 6021921..b91c36d 100644 --- a/prompts/skills/f3s/references/shelly-plug.md +++ b/prompts/skills/f3s/references/shelly-plug.md @@ -93,16 +93,17 @@ wake via WoL when this was set up — deploy when it is back online). `wol-f3s` (dotfiles `scripts/wol-f3s`; deployed to `/home/paul/scripts/wol-f3s` on earth and `/usr/local/bin/wol-f3s` on pi0/pi1/pi2) controls the plug as part -of bulk power actions. On `pi0`/`pi1` (NetBSD) this needed: pkgsrc `bash` -(already present as a dependency of other packages) and pkgsrc `wol` installed, -the shebang changed from `#!/bin/bash` to `#!/usr/pkg/bin/bash` on the deployed -copy (dotfiles' own copy for earth/Linux stays as-is), `~/.shelly_plug` copied -over (missing on a fresh image), and `/etc/hosts` entries for `f0`–`f3`/`pi2`–`pi3` -(cross-Pi/host `.lan.buetow.org` resolution isn't reliable — same DNS gap -noted elsewhere in this skill). End-to-end verified from `pi0` (2026-07-03): -woke `f3` via WoL, toggled the shelly plug off/on, shut `f3` back down via -`wol-f3s shutdown-f3` — all worked. Note single-host `wol-f3s f3`/`shutdown-f3` -does **not** touch the shelly plug (only the bulk `all`/`shutdown-all` paths do). +of bulk power actions. On `pi0`/`pi1` (NetBSD) this needs: pkgsrc `bash` and +pkgsrc `wol` installed, the shebang changed from `#!/bin/bash` to +`#!/usr/pkg/bin/bash` on the deployed copy (dotfiles' own copy for +earth/Linux stays as-is), `~/.shelly_plug` present, and `/etc/hosts` entries +for `f0`–`f3`/`pi2`–`pi3` (cross-Pi/host `.lan.buetow.org` resolution isn't +reliable — same DNS gap noted elsewhere in this skill). SSH trust from +`pi0`/`pi1` to each `fN` host is required for the shutdown path (`ssh +paul@fN "doas poweroff"`) — note the host key must be accepted for both the +hostname **and** the bare IP, since the script connects by IP. Single-host +`wol-f3s <host>`/`shutdown-<host>` does **not** touch the shelly plug — only +the bulk `all`/`shutdown-all` paths do. - `wol-f3s` / `all` → `shelly_set true` **before** sending WoL packets (fans on). - `wol-f3s shutdown-all` → `shelly_set false` **after** all hosts/Pis are down diff --git a/prompts/skills/f3s/references/wireguard.md b/prompts/skills/f3s/references/wireguard.md index d34079c..ec7403e 100644 --- a/prompts/skills/f3s/references/wireguard.md +++ b/prompts/skills/f3s/references/wireguard.md @@ -11,14 +11,14 @@ Hybrid WireGuard topology connecting the f3s infrastructure mesh, two gateway-on **Limited-peer nodes** (connect to the gateways, plus `rocky` — not full mesh): - `pi0` — **NetBSD 10.1** on Raspberry Pi 3 (`192.168.2.203`) -- `pi1` — **NetBSD 10.1** on Raspberry Pi 3 (`192.168.2.204`) — converted after `pi0`, same procedure, both peers up successfully first try on `pi1` since the `pi0`-derived runbook already had all the gotchas baked in +- `pi1` — **NetBSD 10.1** on Raspberry Pi 3 (`192.168.2.204`) **Roaming clients** (connect only to gateways): - `earth` — Fedora laptop (192.168.2.200) - `pixel7pro` — Android phone (192.168.2.201) Even `fN <-> rN` tunnels exist (technically redundant since the VM runs on the host) to keep config uniform. -`pi0` and `pi1` are not full-mesh peers; each has exactly 3 peers: `blowfish`, `fishfinger`, and `rocky` (verified against both hosts' live configs — not gateway-only as older notes here claimed). +`pi0` and `pi1` are not full-mesh peers; each has exactly 3 peers: `blowfish`, `fishfinger`, and `rocky`. ### `pi0`/`pi1` (NetBSD): no native `wg(4)`, use `wireguard-go` instead diff --git a/prompts/skills/pkgrepo/SKILL.md b/prompts/skills/pkgrepo/SKILL.md index fff92e8..aea0294 100644 --- a/prompts/skills/pkgrepo/SKILL.md +++ b/prompts/skills/pkgrepo/SKILL.md @@ -31,6 +31,6 @@ Use `f3s` alongside this skill when the task depends on broader host-role or clu - `f0` as the FreeBSD NFS/PV host for `/data/nfs/k3svolumes/pkgrepo/` - `fishfinger` and `blowfish` as the OpenBSD frontend hosts - `r0-r2` as Rocky Linux x86_64 bhyve VMs -- `pi2-pi3` as Rocky Linux aarch64 Raspberry Pi nodes (`pi0`/`pi1` were reinstalled to NetBSD — see the `f3s` skill's `bootstrap-netbsd-pi.md` — and are no longer Rocky package-repo clients) +- `pi2-pi3` as Rocky Linux aarch64 Raspberry Pi nodes (`pi0`/`pi1` run NetBSD — see the `f3s` skill's `bootstrap-netbsd-pi.md` — and are not Rocky package-repo clients) - `earth` as the Fedora laptop used for package publication and verification - `f0-f3` as FreeBSD hosts diff --git a/prompts/skills/pkgrepo/references/client-setup.md b/prompts/skills/pkgrepo/references/client-setup.md index 67850ab..1a56512 100644 --- a/prompts/skills/pkgrepo/references/client-setup.md +++ b/prompts/skills/pkgrepo/references/client-setup.md @@ -85,7 +85,7 @@ OpenBSD packages are signed with `signify(1)` via `pkg_sign`: ## Rocky Linux (r0–r2, pi2–pi3) -`pi0`/`pi1` were reinstalled to NetBSD (see `f3s` skill's `bootstrap-netbsd-pi.md`) and are no longer Rocky clients of this repo. +`pi0`/`pi1` run NetBSD (see `f3s` skill's `bootstrap-netbsd-pi.md`) and are not Rocky clients of this repo. Architecture-specific repo URLs: - `https://pkgrepo.f3s.buetow.org/rockylinux/9/x86_64/` (r0–r2) diff --git a/prompts/skills/pkgrepo/references/dtail-package.md b/prompts/skills/pkgrepo/references/dtail-package.md index 788319a..468724d 100644 --- a/prompts/skills/pkgrepo/references/dtail-package.md +++ b/prompts/skills/pkgrepo/references/dtail-package.md @@ -35,7 +35,7 @@ make dtail-rocky # Rocky Linux: x86_64 + aarch64 RPMs + repodata ### Rocky Linux (r0–r2 amd64, pi2–pi3 aarch64) -`pi0`/`pi1` were reinstalled to NetBSD (see `f3s` skill's `bootstrap-netbsd-pi.md`) and deliberately do **not** run DTail — see the NetBSD client note below. +`pi0`/`pi1` run NetBSD (see `f3s` skill's `bootstrap-netbsd-pi.md`) and do **not** run DTail — see the NetBSD client note below. | File | |------| @@ -50,7 +50,7 @@ Rocky notes: - Key-cache helper handles both `/root/.ssh/authorized_keys` and `/home/*/.ssh/authorized_keys` — `root` works on r0–r2 without manual cache copy - `dserver.service` includes `RuntimeDirectory=dserver` and `ExecStartPre` to recreate `/var/run/dserver` (tmpfs) on Rocky - Repo is unsigned (`gpgcheck=0`) -- `aarch64` RPM is built on pi2 (was pi0 until it was reinstalled to NetBSD) — Fedora's rpmbuild refuses to emit `aarch64` binary RPMs from an x86_64 host +- `aarch64` RPM is built on pi2 — Fedora's rpmbuild refuses to emit `aarch64` binary RPMs from an x86_64 host ## Install / Update @@ -126,7 +126,7 @@ dcat --plain --noColor --trustAllHosts --user paul \ dcat --plain --noColor --trustAllHosts --user root \ --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org --files /etc/fstab -# Raspberry Pis (pi2–pi3, user paul) -- pi0/pi1 are NetBSD now and don't run DTail +# Raspberry Pis (pi2–pi3, user paul) -- pi0/pi1 run NetBSD and don't run DTail dcat --plain --noColor --trustAllHosts --user paul \ --servers pi2.lan.buetow.org,pi3.lan.buetow.org \ --files /etc/fstab diff --git a/prompts/skills/pkgrepo/references/packaging-workflow.md b/prompts/skills/pkgrepo/references/packaging-workflow.md index bf53bb8..9c7a262 100644 --- a/prompts/skills/pkgrepo/references/packaging-workflow.md +++ b/prompts/skills/pkgrepo/references/packaging-workflow.md @@ -57,7 +57,7 @@ Cross-compilation from Linux fails for CGo (e.g. packages with DataDog/zstd). Us - **OpenBSD**: native build on the local QEMU/KVM build VM (see [openbsd-build-vm.md](openbsd-build-vm.md)) - **FreeBSD**: cross-compile with `CGO_ENABLED=0 -tags nozstd` — disables zstd support but allows static cross-compile -- **Rocky Linux**: built locally on earth (x86_64) and on pi2 (aarch64 via rpmbuild) — was pi0 until it was reinstalled to NetBSD; pi2 is the current designated aarch64 Rocky build host +- **Rocky Linux**: built locally on earth (x86_64) and on pi2 (aarch64 via rpmbuild) ## Manual Packaging Reference |