From fbf9d4815ea25bd85be0f232ac8670bda1a64955 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Sat, 18 Jul 2026 22:03:55 +0300 Subject: skills: split f3s into six focused sibling skills (index pattern) Execute plans/f3s-skill-split-plan.md. Carve the oversized f3s skill into six f3s-prefixed siblings so each loads on its own triggers, and slim f3s to a hub (38 reference files -> 10; 119 -> 73 lines): - f3s-storage ZFS/zrepl/CARP/NFS-stunnel, nfs-mount-monitor, backups - f3s-k3s cluster install, off-LAN access, ingress, etcd, r-node Rex - f3s-observability Prometheus/Alloy/Loki/Tempo, FreeBSD node_exporter - f3s-workloads Immich, Garage, Player, yChat, goprecords/uptimed - f3s-raspberry-pi pi0/pi1 NetBSD static site, pi2/pi3 Pi-hole/LAN DNS - f3s-dtail dserver deployment/ops (SSH 2222) f3s hub keeps the master Host-IP table, physical hosts, bhyve, WireGuard mesh, and off-LAN access as the canonical inward-pointing context, plus a Related skills block. Applies skill-maintenance best practices: fixes the three inlined SKILL.md duplications (Pi/webserver, DTail) by moving prose to one canonical home and keeping each new SKILL.md a slim index that points to its references rather than re-inlining them. All cross-skill links rewritten to ../..//references/ form (incl. inbound links from pkgrepo and rocky-vm-setup, and two pre-existing broken links); verified all 112 relative links resolve. Co-Authored-By: Claude Opus 4.8 --- prompts/skills/f3s-dtail/SKILL.md | 32 ++ prompts/skills/f3s-dtail/references/dserver.d | 7 + prompts/skills/f3s-dtail/references/dtail.md | 233 ++++++++++++++ prompts/skills/f3s-k3s/SKILL.md | 25 ++ prompts/skills/f3s-k3s/references/ingress.md | 120 ++++++++ prompts/skills/f3s-k3s/references/install.md | 169 +++++++++++ prompts/skills/f3s-k3s/references/r-node-deploy.md | 144 +++++++++ prompts/skills/f3s-k3s/references/remote-access.md | 108 +++++++ .../skills/f3s-k3s/references/troubleshooting.md | 49 +++ prompts/skills/f3s-observability/SKILL.md | 46 +++ .../skills/f3s-observability/references/freebsd.md | 112 +++++++ .../skills/f3s-observability/references/stack.md | 158 ++++++++++ prompts/skills/f3s-raspberry-pi/SKILL.md | 54 ++++ .../references/bootstrap-netbsd-pi.md | 337 +++++++++++++++++++++ .../f3s-raspberry-pi/references/pihole-pi.md | 45 +++ prompts/skills/f3s-storage/SKILL.md | 41 +++ prompts/skills/f3s-storage/references/backups.md | 40 +++ prompts/skills/f3s-storage/references/carp.md | 95 ++++++ .../f3s-storage/references/nfs-mount-monitor.md | 107 +++++++ prompts/skills/f3s-storage/references/nfs.md | 204 +++++++++++++ .../f3s-storage/references/troubleshooting.md | 198 ++++++++++++ prompts/skills/f3s-storage/references/usb-keys.md | 116 +++++++ prompts/skills/f3s-storage/references/zfs.md | 90 ++++++ prompts/skills/f3s-storage/references/zrepl.md | 237 +++++++++++++++ prompts/skills/f3s-workloads/SKILL.md | 24 ++ prompts/skills/f3s-workloads/references/garage.md | 158 ++++++++++ .../f3s-workloads/references/goprecords-uptimed.md | 130 ++++++++ prompts/skills/f3s-workloads/references/immich.md | 98 ++++++ prompts/skills/f3s-workloads/references/player.md | 167 ++++++++++ prompts/skills/f3s-workloads/references/ychat.md | 83 +++++ prompts/skills/f3s/SKILL.md | 82 ++--- .../skills/f3s/references/bootstrap-netbsd-pi.md | 337 --------------------- prompts/skills/f3s/references/dserver.d | 7 - prompts/skills/f3s/references/dtail.md | 233 -------------- prompts/skills/f3s/references/garage.md | 158 ---------- .../skills/f3s/references/goprecords-uptimed.md | 130 -------- prompts/skills/f3s/references/immich.md | 98 ------ prompts/skills/f3s/references/k3s-setup.md | 12 - prompts/skills/f3s/references/k3s-setup/ingress.md | 120 -------- prompts/skills/f3s/references/k3s-setup/install.md | 169 ----------- .../f3s/references/k3s-setup/remote-access.md | 108 ------- .../f3s/references/k3s-setup/troubleshooting.md | 49 --- prompts/skills/f3s/references/observability.md | 35 --- .../skills/f3s/references/observability/freebsd.md | 112 ------- .../skills/f3s/references/observability/stack.md | 158 ---------- prompts/skills/f3s/references/pihole-pi.md | 45 --- prompts/skills/f3s/references/player.md | 167 ---------- prompts/skills/f3s/references/r-node-deploy.md | 144 --------- prompts/skills/f3s/references/remote-access.md | 2 +- prompts/skills/f3s/references/rocky-linux-vms.md | 2 +- prompts/skills/f3s/references/storage.md | 29 -- prompts/skills/f3s/references/storage/backups.md | 40 --- prompts/skills/f3s/references/storage/carp.md | 95 ------ .../f3s/references/storage/nfs-mount-monitor.md | 107 ------- prompts/skills/f3s/references/storage/nfs.md | 204 ------------- .../f3s/references/storage/troubleshooting.md | 198 ------------ prompts/skills/f3s/references/storage/usb-keys.md | 116 ------- prompts/skills/f3s/references/storage/zfs.md | 90 ------ prompts/skills/f3s/references/storage/zrepl.md | 237 --------------- prompts/skills/f3s/references/ychat.md | 83 ----- prompts/skills/pkgrepo/SKILL.md | 2 +- prompts/skills/pkgrepo/references/client-setup.md | 2 +- .../skills/rocky-vm-setup/references/overview.md | 2 +- prompts/skills/rocky-vm-setup/references/zrepl.md | 2 +- 64 files changed, 3451 insertions(+), 3351 deletions(-) create mode 100644 prompts/skills/f3s-dtail/SKILL.md create mode 100644 prompts/skills/f3s-dtail/references/dserver.d create mode 100644 prompts/skills/f3s-dtail/references/dtail.md create mode 100644 prompts/skills/f3s-k3s/SKILL.md create mode 100644 prompts/skills/f3s-k3s/references/ingress.md create mode 100644 prompts/skills/f3s-k3s/references/install.md create mode 100644 prompts/skills/f3s-k3s/references/r-node-deploy.md create mode 100644 prompts/skills/f3s-k3s/references/remote-access.md create mode 100644 prompts/skills/f3s-k3s/references/troubleshooting.md create mode 100644 prompts/skills/f3s-observability/SKILL.md create mode 100644 prompts/skills/f3s-observability/references/freebsd.md create mode 100644 prompts/skills/f3s-observability/references/stack.md create mode 100644 prompts/skills/f3s-raspberry-pi/SKILL.md create mode 100644 prompts/skills/f3s-raspberry-pi/references/bootstrap-netbsd-pi.md create mode 100644 prompts/skills/f3s-raspberry-pi/references/pihole-pi.md create mode 100644 prompts/skills/f3s-storage/SKILL.md create mode 100644 prompts/skills/f3s-storage/references/backups.md create mode 100644 prompts/skills/f3s-storage/references/carp.md create mode 100644 prompts/skills/f3s-storage/references/nfs-mount-monitor.md create mode 100644 prompts/skills/f3s-storage/references/nfs.md create mode 100644 prompts/skills/f3s-storage/references/troubleshooting.md create mode 100644 prompts/skills/f3s-storage/references/usb-keys.md create mode 100644 prompts/skills/f3s-storage/references/zfs.md create mode 100644 prompts/skills/f3s-storage/references/zrepl.md create mode 100644 prompts/skills/f3s-workloads/SKILL.md create mode 100644 prompts/skills/f3s-workloads/references/garage.md create mode 100644 prompts/skills/f3s-workloads/references/goprecords-uptimed.md create mode 100644 prompts/skills/f3s-workloads/references/immich.md create mode 100644 prompts/skills/f3s-workloads/references/player.md create mode 100644 prompts/skills/f3s-workloads/references/ychat.md delete mode 100644 prompts/skills/f3s/references/bootstrap-netbsd-pi.md delete mode 100644 prompts/skills/f3s/references/dserver.d delete mode 100644 prompts/skills/f3s/references/dtail.md delete mode 100644 prompts/skills/f3s/references/garage.md delete mode 100644 prompts/skills/f3s/references/goprecords-uptimed.md delete mode 100644 prompts/skills/f3s/references/immich.md delete mode 100644 prompts/skills/f3s/references/k3s-setup.md delete mode 100644 prompts/skills/f3s/references/k3s-setup/ingress.md delete mode 100644 prompts/skills/f3s/references/k3s-setup/install.md delete mode 100644 prompts/skills/f3s/references/k3s-setup/remote-access.md delete mode 100644 prompts/skills/f3s/references/k3s-setup/troubleshooting.md delete mode 100644 prompts/skills/f3s/references/observability.md delete mode 100644 prompts/skills/f3s/references/observability/freebsd.md delete mode 100644 prompts/skills/f3s/references/observability/stack.md delete mode 100644 prompts/skills/f3s/references/pihole-pi.md delete mode 100644 prompts/skills/f3s/references/player.md delete mode 100644 prompts/skills/f3s/references/r-node-deploy.md delete mode 100644 prompts/skills/f3s/references/storage.md delete mode 100644 prompts/skills/f3s/references/storage/backups.md delete mode 100644 prompts/skills/f3s/references/storage/carp.md delete mode 100644 prompts/skills/f3s/references/storage/nfs-mount-monitor.md delete mode 100644 prompts/skills/f3s/references/storage/nfs.md delete mode 100644 prompts/skills/f3s/references/storage/troubleshooting.md delete mode 100644 prompts/skills/f3s/references/storage/usb-keys.md delete mode 100644 prompts/skills/f3s/references/storage/zfs.md delete mode 100644 prompts/skills/f3s/references/storage/zrepl.md delete mode 100644 prompts/skills/f3s/references/ychat.md (limited to 'prompts') diff --git a/prompts/skills/f3s-dtail/SKILL.md b/prompts/skills/f3s-dtail/SKILL.md new file mode 100644 index 0000000..779e1b3 --- /dev/null +++ b/prompts/skills/f3s-dtail/SKILL.md @@ -0,0 +1,32 @@ +--- +name: f3s-dtail +description: Reference skill for DTail/dserver deployment across the f3s fleet, distributed log access over SSH on port 2222 — Pis arm64 (NetBSD + Rocky) vs r0–r2 amd64, r-VM root + root.authorized_keys cache, firewalld/npf 2222 rules, systemd timers. Package building lives in the `pkgrepo` skill. Use when deploying, configuring, or troubleshooting dserver on homelab hosts. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)). +--- + +# f3s DTail / dserver + +Distributed log access (DTail) via `dserver` across the f3s fleet. This skill owns +the **runtime deployment/operations**; package *building/publishing* lives in the +sibling [`pkgrepo`](../pkgrepo/SKILL.md) skill (`dtail-package.md`). + +## When to Use + +- Deploying, configuring, or troubleshooting `dserver` on homelab hosts (Pis, r0–r2) +- SSH-on-2222 access, permissions/key-cache, firewall (firewalld/npf) rules, systemd timers +- For building the `dtail` package (esp. the NetBSD build), use [`pkgrepo`](../pkgrepo/SKILL.md); for the Pi nodes themselves, [`f3s-raspberry-pi`](../f3s-raspberry-pi/SKILL.md); for hosts/IPs, the [`f3s`](../f3s/SKILL.md) hub. + +## Overview + +Distributed log access (`dcat`/`dtail`/`dgrep`/`dmap`) over SSH on port **2222** (not +sshd's 22), by architecture: **pi2/pi3** linux/arm64, **pi0/pi1** netbsd/arm64 (installed +from the `dtail` package in the custom [`pkgrepo`](../pkgrepo/SKILL.md)), **r0–r2** k3s +Rocky VMs linux/amd64. The recurring gotchas — installing as `root`, listing `root` in +`Server.Permissions.Users`, mirroring `/root/.ssh/authorized_keys` into the key cache +(the cache script only walks `/home/*`), and opening 2222 in firewalld/npf — plus the +exact per-host cross-build commands are the canonical detail in +[references/dtail.md](references/dtail.md). + +## Reference Files + +- [DTail / dserver](references/dtail.md) — full deployment detail: Pis **arm64** vs r0–r2 **amd64**, r-VM **root** + `root.authorized_keys` cache, firewalld **2222**, systemd timers (section **dserver on r0, r1, r2**) +- [dserver.d](references/dserver.d) — index: links to the **Rocky r-VM DTail** subsection and full **dtail.md** diff --git a/prompts/skills/f3s-dtail/references/dserver.d b/prompts/skills/f3s-dtail/references/dserver.d new file mode 100644 index 0000000..49d5c54 --- /dev/null +++ b/prompts/skills/f3s-dtail/references/dserver.d @@ -0,0 +1,7 @@ +# dserver on f3s (index) + +- **r0–r2 Rocky bhyve / k3s VMs** — install context and SSH notes: [Rocky Linux VMs – DTail (dserver) on r0–r2](rocky-linux-vms.md#dtail-dserver-on-r0r2) +- **pi0–pi1 NetBSD Pis** — dserver installed from the custom pkgrepo (`dtail-4.3.2ng` package): build with `make dtail-netbsd` in `~/git/conf/packages` (cross-compile netbsd/arm64, package natively on pi0, upload to pkgrepo), deploy via `pkg_add https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-.tgz`. Full build/install/rc.d/npf details and gotchas: `pkgrepo` skill's `dtail-package.md` +- **Full DTail reference** (NetBSD + Rocky Pis, r VMs amd64, firewalld, key cache, clients): [dtail.md](dtail.md) + +Upstream repo: `https://codeberg.org/snonux/dtail` — `doc/installation.md`, `examples/`. diff --git a/prompts/skills/f3s-dtail/references/dtail.md b/prompts/skills/f3s-dtail/references/dtail.md new file mode 100644 index 0000000..840f07d --- /dev/null +++ b/prompts/skills/f3s-dtail/references/dtail.md @@ -0,0 +1,233 @@ +# DTail / dserver on f3s + +[DTail](https://codeberg.org/snonux/dtail) is a distributed log tool (tail/cat/grep/MapReduce) over SSH. The **dserver** daemon listens on **TCP 2222** (not port 22). Clients (`dtail`, `dcat`, `dgrep`, `dmap`, …) use normal SSH keys against dserver. + +Upstream install and examples live in the repo: `doc/installation.md`, `examples/`. + +## Host roles in this lab + +| Hosts | OS / arch | dserver binary | Typical SSH user | +|-------|-----------|----------------|------------------| +| **pi0–pi1** | NetBSD 10.1 **aarch64** (Raspberry Pi 3, see `bootstrap-netbsd-pi.md`) | Cross-build **netbsd/arm64**, `nozstd`; installed via `pkg_add` from the custom pkgrepo | `paul@piN.lan.buetow.org` | +| **pi2–pi3** | Rocky Linux 9 **aarch64** (Raspberry Pi 3) | Cross-build **linux/arm64**, `nozstd` | `paul@piN.lan.buetow.org` | +| **r0–r2** | Rocky Linux 9 **x86_64** (bhyve VMs, k3s nodes) | Cross-build **linux/amd64**, `nozstd` | Often `root@rN.lan.buetow.org` (see [Rocky Linux VMs](../../f3s/references/rocky-linux-vms.md)); add `root` (and `paul` if present) to **Server.Permissions.Users** in `dtail.json` | +| **blowfish, fishfinger** | OpenBSD 7.8 **amd64** | Native OpenBSD package build | `rex@blowfish.buetow.org`, `rex@fishfinger.buetow.org` | + +`pi0`/`pi1` (NetBSD) run dserver since 2026-07-09 from the `dtail` package in the custom pkgrepo — build pipeline (`make dtail-netbsd`), install steps, rc.d/npf details, and gotchas live in the `pkgrepo` skill's `dtail-package.md`. + +**Key cache filenames matter:** `examples/update_key_cache.sh.example` only scans `/home/*` and writes `/var/run/dserver/cache/USER.authorized_keys`. In this lab, DTail auth worked only after writing the exact cache filename for the login user: + +- **r0–r2**: `root.authorized_keys` +- **pi0–pi1**: `paul.authorized_keys` (NetBSD rc.d `start_precmd` repopulates it on every start — `/var/run` is volatile) +- **pi2–pi3**: `paul.authorized_keys` +- **blowfish, fishfinger**: `rex.authorized_keys` + +If clients connect as **root**, copy keys once (e.g. after install) and on key changes: + +```bash +cp /root/.ssh/authorized_keys /var/run/dserver/cache/root.authorized_keys +chown dserver:dserver /var/run/dserver/cache/root.authorized_keys +chmod 600 /var/run/dserver/cache/root.authorized_keys +``` + +For the Pi nodes: + +```bash +cp /home/paul/.ssh/authorized_keys /var/run/dserver/cache/paul.authorized_keys +chown dserver:dserver /var/run/dserver/cache/paul.authorized_keys +chmod 600 /var/run/dserver/cache/paul.authorized_keys +``` + +## dserver on r0, r1, r2 (k3s Rocky VMs, amd64) + +These hosts are the **x86_64** guests on f0/f1/f2. SSH and VM background: [Rocky Linux VMs](../../f3s/references/rocky-linux-vms.md), **DTail subsection** (same content in short form): [DTail (dserver) on r0–r2](../../f3s/references/rocky-linux-vms.md#dtail-dserver-on-r0r2). Shortcut index file: [dserver.d](dserver.d). **Do not** install the Pi **arm64** binary here. + +| Item | Value | +|------|--------| +| Hostnames | `r0.lan.buetow.org`, `r1.lan.buetow.org`, `r2.lan.buetow.org` | +| LAN IPs | `192.168.1.120`–`122` | +| Admin SSH (normal) | `ssh -p 22 root@rN.lan.buetow.org` (key-based; see Rocky VM doc) | +| dserver port | **2222/tcp** (DTail clients); **22** remains sshd | + +### Build the binaries (on earth) + +```bash +cd ~/git/dtail # your checkout of https://codeberg.org/snonux/dtail + +# exact cross-builds for the Rocky VMs +for bin in dserver dtail dcat dgrep dmap dtailhealth; do + CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags nozstd -o "$bin-linux-amd64" ./cmd/$bin/main.go +done +``` + +Use direct `go build -tags nozstd` when you need a deterministic cross-build artifact. Reusing `make dserver` after a local native build can leave an old binary in place because the target name is just `dserver`. + +### Install on each rN (run as root over SSH) + +1. Copy to the VM, e.g. `scp -P 22 dserver root@r0.lan.buetow.org:/tmp/` and the files from `examples/` (see checklist below), or unpack a small staging dir on `/tmp/dtail-install/`. + +2. **System user and dirs** + ```bash + id dserver &>/dev/null || useradd -r -d /var/lib/dserver -s /sbin/nologin -U dserver + mkdir -p /etc/dserver /var/run/dserver/cache /var/run/dserver/log + chown -R dserver:dserver /var/run/dserver + install -m 755 /tmp/dserver /usr/local/bin/dserver + ``` + +3. **`/etc/dserver/dtail.json`** — start from `examples/dtail.json.example` and ensure **`Server.Permissions.Users` includes `"root"`** (with the same `readfiles` rules you need). The stock example lists `paul` / `pbuetow`; without **`root`**, root cannot use dserver even with a valid key. + +4. **systemd units** (from repo `examples/`): + - `dserver.service` → `/etc/systemd/system/dserver.service` + - `update_key_cache.sh.example` → `/var/run/dserver/update_key_cache.sh` (mode `0755`) + - `dserver-update-keycache.service` + **timer** (use a timer unit with `[Install] WantedBy=timers.target` if the raw example omits it) → `/etc/systemd/system/` + - Optional: `prune_dserver_logs.sh.example`, `dserver-prune-logs.service`, `dserver-prune-logs.timer` + + ```bash + systemctl daemon-reload + systemctl enable --now dserver-update-keycache.timer + systemctl enable --now dserver-prune-logs.timer # if installed + systemctl start dserver-update-keycache.service # populate cache once + ``` + + **Important:** the stock unit uses `WorkingDirectory=/var/run/dserver`. On Rocky and Pi hosts, `dserver` failed with `status=200/CHDIR` until the unit also recreated that tmpfs path at service start. Add: + + ```ini + RuntimeDirectory=dserver + RuntimeDirectoryMode=0755 + ExecStartPre=/usr/bin/mkdir -p /var/run/dserver/cache /var/run/dserver/log + ``` + + Then run: + + ```bash + systemctl daemon-reload + systemctl reset-failed dserver + systemctl start dserver + ``` + +5. **Root SSH key cache (required on r VMs)** + The stock `update_key_cache.sh` only copies `/home/*/.ssh/authorized_keys`. **Root’s keys live in `/root`**, so mirror them explicitly (after any change to root’s authorized_keys): + + ```bash + cp /root/.ssh/authorized_keys /var/run/dserver/cache/root.authorized_keys + chown dserver:dserver /var/run/dserver/cache/root.authorized_keys + chmod 600 /var/run/dserver/cache/root.authorized_keys + ``` + +6. **firewalld** (default on Rocky): + + ```bash + firewall-cmd --permanent --add-port=2222/tcp && firewall-cmd --reload + ``` + +7. **Start dserver** (unit is **disabled** by default for boot; start manually or `enable` if desired): + + ```bash + systemctl start dserver + systemctl is-active dserver + ss -tlnp | grep 2222 + ``` + + Expect `dserver` listening on `*:2222` and `ssh_host_key` plus `root.authorized_keys` under `/var/run/dserver/cache/`. + +### Client usage from earth + +DTail uses the **same username** as normal SSH unless overridden. If dserver is set up for **root** only on r VMs, run as root or pass the client user your config expects. Example: + +```bash +dcat --plain --noColor --trustAllHosts --user root \ + --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org \ + --files /etc/fstab +``` + +Add **2222** host keys to `~/.ssh/known_hosts` the first time (interactive trust, or `ssh-keyscan -p 2222`). + +## Cross-compiling dserver (from earth or any dev machine) + +From a clone of the repo: + +```bash +cd ~/git/dtail # or your checkout + +# Raspberry Pi 4× — linux/arm64, static, no CGO zstd +CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -tags nozstd -o dserver-linux-arm64 ./cmd/dserver/main.go + +# k3s VMs r0–r2 — linux/amd64 +CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags nozstd -o dserver-linux-amd64 ./cmd/dserver/main.go +``` + +`DTAIL_NO_ZSTD=yes` sets the `nozstd` build tag so the binary does not link DataDog’s CGO zstd (required for static cross-compiles). **`.zst` log files are not supported** in that binary; gzip still works. + +## Installation checklist (each server) + +Do this on **each** Linux target (pi2–pi3 and/or r0–r2). pi0/pi1 are NetBSD and install the `dtail` package from the custom pkgrepo instead — see the `pkgrepo` skill's `dtail-package.md`. Adjust **user** if you are not using `paul` on the node. + +1. **Binary**: `/usr/local/bin/dserver`, mode `0755`, owned by root. +2. **OS user**: `dserver` system account (`useradd -r -d /var/lib/dserver -s /sbin/nologin -U dserver`). +3. **Directories**: `/etc/dserver`, `/var/run/dserver` (tmpfs — recreated at boot; use systemd `RuntimeDirectory` in the example unit if you adopt it). +4. **Config**: `/etc/dserver/dtail.json` from `examples/dtail.json.example` — ensure **Server.Permissions.Users** includes every login name that will connect (e.g. `paul`, `root`). +5. **systemd**: `examples/dserver.service.example` → `/etc/systemd/system/dserver.service` — unit stays **disabled** by default; start with `systemctl start dserver`. Add `RuntimeDirectory=dserver`, `RuntimeDirectoryMode=0755`, and `ExecStartPre=/usr/bin/mkdir -p /var/run/dserver/cache /var/run/dserver/log` so the tmpfs working directory exists before `dserver` starts. +6. **SSH host keys for clients**: dserver cannot read users’ `~/.ssh/authorized_keys` as user `dserver`. Use `examples/update_key_cache.sh.example` + `dserver-update-keycache.service` / `.timer` to mirror keys into `/var/run/dserver/cache/USER.authorized_keys`. +7. **firewalld**: allow **2222/tcp** (ping may work while TCP is blocked): + + ```bash + sudo firewall-cmd --permanent --add-port=2222/tcp && sudo firewall-cmd --reload + ``` + + Or run `examples/firewalld-dserver-port.sh.example` once. + +8. **Optional**: log pruning — `examples/prune_dserver_logs.sh.example` + `dserver-prune-logs` service/timer (see `doc/installation.md`). + +## SSH from laptops (earth) + +- **Port 2222** is dserver, **port 22** is normal sshd. +- If `~/.ssh/config` has a broad `Host *.buetow.org` with a wrong `Port`, **narrow overrides** for `*.lan.buetow.org` with `Port 22`, or use `ssh -p 22` / `dcat` default port 2222 only for DTail targets. +- First-time **host keys** go to `~/.ssh/known_hosts` for `[hostname]:2222` (and IP lines). A **stdout logger deadlock** when trusting new hosts was fixed upstream (release including commit `28f6319`+); rebuild clients if you still see a hang after “trust these hosts”. + +## Client examples + +```bash +dcat --plain --noColor --trustAllHosts --user paul \ + --servers pi2.lan.buetow.org,pi3.lan.buetow.org \ + --files /etc/fstab + +dcat --plain --noColor --trustAllHosts --user root \ + --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org \ + --files /etc/fstab +``` + +Use hostnames that resolve from where you run the client (often `*.lan.buetow.org`). + +## Package-managed DTail + +For package-repo-backed DTail and other custom package repo tasks, use the sibling `pkgrepo` skill and its reference: + +- [Package Repositories](../../pkgrepo/references/repo-architecture.md) + +That skill now owns: + +- FreeBSD, OpenBSD, and Rocky client repo configuration +- custom repo layout, publication, and verification notes +- Rocky repo client configuration and RPM install flow +- OpenBSD DTail package build, publish, replace, and cache-refresh steps +- DTail package publication details that depend on the repo + +## Verified lab state + +On 2026-04-11 this setup was verified end-to-end with: + +- the full Linux binary set (`dserver`, `dtail`, `dcat`, `dgrep`, `dmap`, `dtailhealth`) installed on `r0`-`r2` and `pi0`-`pi3` +- `dserver` active and listening on `*:2222` on all seven hosts +- successful `dcat /etc/fstab` reads from earth using `--user root` for `r0`-`r2` and `--user paul` for `pi0`-`pi3` + +## k3s / r0–r2 notes + +- **amd64** binaries only; do not deploy the Pi **arm64** build there. +- Same **2222** and **firewalld** rules apply. +- These VMs are heavier than the Pis; defaults in `dtail.json` (`MaxConcurrentCats`, etc.) can be raised if needed. +- Full install sequence for these three nodes: section **dserver on r0, r1, r2** above. + +## Related upstream fixes (f3s-relevant) + +- **Host key file**: first boot uses `RootedPath.Stat` + `errors.Is(…, fs.ErrNotExist)` so missing `cache/ssh_host_key` triggers generation (older `os.IsNotExist` missed wrapped errors). +- **Known-hosts prompt deadlock**: `internal/io/dlog/loggers/stdout.go` must not hold its mutex across the pause/resume wait when the trust prompt runs. diff --git a/prompts/skills/f3s-k3s/SKILL.md b/prompts/skills/f3s-k3s/SKILL.md new file mode 100644 index 0000000..5ae3c47 --- /dev/null +++ b/prompts/skills/f3s-k3s/SKILL.md @@ -0,0 +1,25 @@ +--- +name: f3s-k3s +description: Reference skill for the f3s k3s Kubernetes cluster, 3-node HA install on r0/r1/r2 Rocky VMs (bootstrap, kubeconfig, PVs, ArgoCD), off-LAN access (jump via OpenBSD frontend to root@r0.wg0 to kubectl), ingress (relayd, cert-manager), etcd recovery, and the reusable Rex r-node rollout. Use when installing, accessing, or troubleshooting the k3s cluster or deploying to r0/r1/r2. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)). +--- + +# f3s k3s Cluster + +3-node HA k3s cluster running on the Rocky Linux VMs r0/r1/r2 (one per +FreeBSD bhyve host f0/f1/f2). All control-plane and etcd traffic flows +over WireGuard. + +## When to Use + +- Installing, bootstrapping, or recovering the k3s cluster (etcd, kubeconfig, PVs, ArgoCD) +- Reaching the cluster off-LAN, or deploying to r0/r1/r2 (incl. the Rex r-node rollout) +- Ingress/cert-manager work on the cluster +- For the underlying Rocky VMs, WireGuard mesh, storage/NFS, and host/IP inventory, see the [`f3s`](../f3s/SKILL.md) hub and [`f3s-storage`](../f3s-storage/SKILL.md). + +## Reference Files + +- [Install](references/install.md) — bootstrap, kubeconfig, etcd/controller-manager metrics, built-in components, NFS PV pattern, ArgoCD, node IP summary, useful commands +- [Remote access (off-LAN)](references/remote-access.md) — reaching the cluster while roaming: **preferred** dedicated `wg0` kubectl context talking directly to `r0.wg0.wan.buetow.org:6443` over WireGuard (switch with `kubectl config use-context wg0`); fallback jump via OpenBSD frontend (`ssh -A rex@fishfinger.buetow.org` → `ssh root@r0.wg0` → `kubectl`), one-shot commands, and SSH port-forward tunnel +- [Ingress](references/ingress.md) — OpenBSD `relayd` (internet) and FreeBSD `relayd` on CARP VIP (LAN), cert-manager wildcard, ingress pattern +- [Troubleshooting](references/troubleshooting.md) — etcd Raft log corruption recovery; cluster-wide NFS outage pointer +- [r-node Deploy (Rex)](references/r-node-deploy.md) — reusable Rex rollout to r0/r1/r2 (`f3s/r-nodes/Rexfile`, task `nfs_mount_monitor`): root SSH, `parallelism 3`, idempotent `file`/`on_change` reload diff --git a/prompts/skills/f3s-k3s/references/ingress.md b/prompts/skills/f3s-k3s/references/ingress.md new file mode 100644 index 0000000..6beb4ea --- /dev/null +++ b/prompts/skills/f3s-k3s/references/ingress.md @@ -0,0 +1,120 @@ +# k3s Ingress (Internet + LAN) + +Two ingress paths into the k3s cluster: +- **Internet → OpenBSD relayd** (TLS termination on `blowfish`/`fishfinger`) → WireGuard → Traefik +- **LAN → FreeBSD relayd on CARP VIP** → k3s Traefik + +## External Connectivity: OpenBSD relayd + +Default traffic flow for public k3s-backed services: `Internet → OpenBSD relayd (TLS, Let's Encrypt) → WireGuard → k3s Traefik :80 → Service` + +### relayd.conf on blowfish/fishfinger + +``` +table { + 192.168.2.120 + 192.168.2.121 + 192.168.2.122 +} + +http protocol "https" { + tls keypair f3s.foo.zone + # ... all f3s service TLS keypairs ... + # Non-f3s hosts explicitly forwarded to localhost: + match request header "Host" value "foo.zone" forward to + # f3s hosts have NO match rules — use relay-level failover +} + +relay "https4" { + listen on port 443 tls + protocol "https" + forward to port 80 check tcp # primary + forward to port 8080 # fallback when f3s down +} +``` + +`f3s.buetow.org` is now a special case: it no longer points at the k3s/apache backend and is forwarded by OpenBSD `relayd` to `pi0` (`192.168.2.203`) and `pi1` (`192.168.2.204`) via a dedicated `` backend table. + +When all k3s-backed f3s nodes are down, relayd falls back to `localhost:8080` (OpenBSD httpd serving a "Server turned off" page) for the hosts that still use the shared `` backend. + +## LAN Ingress: FreeBSD relayd on CARP VIP + +For LAN access without going through internet gateways: +`LAN → CARP VIP (192.168.1.138) → FreeBSD relayd → k3s Traefik :443 → Service` + +### FreeBSD relayd config (`/usr/local/etc/relayd.conf`) + +``` +table { 192.168.1.120 192.168.1.121 192.168.1.122 } + +relay "lan_http" { + listen on 192.168.1.138 port 80 + forward to port 80 check tcp +} + +relay "lan_https" { + listen on 192.168.1.138 port 443 + forward to port 443 check tcp +} +``` + +Minimal `/etc/pf.conf` (PF required for relayd): + +``` +set skip on lo0 +pass in quick +pass out quick +``` + +```sh +doas pkg install -y relayd +doas sysrc pf_enable=YES pflog_enable=YES relayd_enable=YES +doas service pf start && doas service pflog start && doas service relayd start +``` + +Run on both f0 and f1. Only CARP MASTER responds to VIP traffic. + +### cert-manager for LAN TLS + +LAN services use `*.f3s.lan.foo.zone` with a self-signed CA managed by cert-manager: + +```sh +cd conf/f3s/cert-manager && just install +# Creates: selfsigned ClusterIssuer, CA cert, wildcard cert (f3s-lan-tls) +``` + +Copy secret to service namespace: +```sh +kubectl get secret f3s-lan-tls -n cert-manager -o yaml | \ + sed 's/namespace: cert-manager/namespace: services/' | \ + kubectl apply -f - +``` + +### LAN ingress pattern + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: ingress-lan + namespace: services + annotations: + spec.ingressClassName: traefik + traefik.ingress.kubernetes.io/router.entrypoints: web,websecure +spec: + tls: + - hosts: + - myservice.f3s.lan.foo.zone + secretName: f3s-lan-tls + rules: + - host: myservice.f3s.lan.foo.zone + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: myservice + port: + number: 8080 +``` diff --git a/prompts/skills/f3s-k3s/references/install.md b/prompts/skills/f3s-k3s/references/install.md new file mode 100644 index 0000000..324895a --- /dev/null +++ b/prompts/skills/f3s-k3s/references/install.md @@ -0,0 +1,169 @@ +# k3s Install + +3-node HA k3s cluster running on Rocky Linux VMs (r0, r1, r2). All nodes act as both control-plane and etcd members (no separate worker nodes). + +- k3s version: **v1.32.6+k3s1** (as of Part 7) +- etcd mode: **embedded HA** (`--cluster-init`) +- All control-plane traffic goes over **WireGuard** (192.168.2.x IPs) + +## Prerequisites + +- All Rocky Linux VMs (r0, r1, r2) updated and running +- WireGuard mesh fully configured (see [wireguard.md](../../f3s/references/wireguard.md)) +- NVMe disk emulation in place (see [rocky-linux-vms.md](../../f3s/references/rocky-linux-vms.md)) — critical for etcd performance + +## Installation + +### Generate shared token + +```sh +# On Fedora laptop +pwgen -n 32 +# Copy output to all r nodes: +echo -n SECRET_TOKEN > ~/.k3s_token # on r0, r1, r2 +``` + +### Bootstrap first node (r0) + +```sh +[root@r0 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \ + sh -s - server --cluster-init \ + --node-ip=192.168.2.120 \ + --advertise-address=192.168.2.120 \ + --tls-san=r0.wg0.wan.buetow.org +``` + +`--node-ip` and `--advertise-address` bind etcd to the WireGuard interface so all control-plane traffic is encrypted. + +### Join remaining nodes (r1, r2) + +```sh +[root@r1 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \ + sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \ + --node-ip=192.168.2.121 \ + --advertise-address=192.168.2.121 \ + --tls-san=r1.wg0.wan.buetow.org + +[root@r2 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \ + sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \ + --node-ip=192.168.2.122 \ + --advertise-address=192.168.2.122 \ + --tls-san=r2.wg0.wan.buetow.org +``` + +### Verify cluster + +```sh +kubectl get nodes +# Expected: r0, r1, r2 all Ready with role control-plane,etcd,master +``` + +## kubeconfig + +```sh +# Copy from any r node to laptop +scp root@r0.lan.buetow.org:/etc/rancher/k3s/k3s.yaml ~/.kube/config +# Edit: replace server address with r0.lan.buetow.org +# (repeat with r1 or r2 if r0 is down) +``` + +## k3s config.yaml — expose etcd and controller-manager metrics + +For Prometheus to scrape etcd and controller-manager metrics, add to `/etc/rancher/k3s/config.yaml` on each r node: + +```sh +cat >> /etc/rancher/k3s/config.yaml << 'EOF' +kube-controller-manager-arg: + - bind-address=0.0.0.0 +etcd-expose-metrics: true +EOF +systemctl restart k3s +``` + +Verify: `curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader` + +## Built-in Components + +| Component | Purpose | +|-----------|---------| +| CoreDNS | DNS for pods | +| Traefik | Ingress controller | +| local-path-provisioner | Local PVC storage | +| metrics-server | Resource metrics | +| svclb-traefik | ServiceLB for Traefik | + +### Scale Traefik to 2 replicas (faster failover) + +```sh +kubectl -n kube-system scale deployment traefik --replicas=2 +``` + +## NFS Persistent Volumes + +Persistent volumes use `hostPath` pointing to NFS-mounted paths: + +``` +/data/nfs/k3svolumes// +``` + +NFS is mounted on all r nodes at `/data/nfs/k3svolumes` via stunnel → CARP VIP → +freeBSD NFS — see [storage/nfs.md](../../f3s-storage/references/nfs.md). The +[`nfs-mount-monitor`](../../f3s-storage/references/nfs-mount-monitor.md) watchdog auto-repairs +hung mounts and force-deletes stuck pods. + +Example PV: + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: example-pv +spec: + capacity: + storage: 1Gi + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + hostPath: + path: /data/nfs/k3svolumes/example-volume + type: Directory +``` + +Create the directory on the NFS share before deploying: `mkdir /data/nfs/k3svolumes//` + +## Deployment: GitOps with ArgoCD + +Config repository: `https://codeberg.org/snonux/conf` (directory: `f3s/`) + +ArgoCD app structure: +``` +argocd-apps/ + monitoring/ # Prometheus, Grafana, Loki, etc. + services/ # User-facing services + infra/ # Infrastructure components + test/ # Test deployments +``` + +**To view pre-ArgoCD state** (how things were in Part 7): +```sh +git clone https://codeberg.org/snonux/conf.git +cd conf && git checkout 15a86f3 # last commit before ArgoCD migration +cd f3s/ +``` + +## Node IP Summary + +| Node | LAN IP | WireGuard IP | k3s API | +|------|--------|-------------|---------| +| r0 | 192.168.1.120 | 192.168.2.120 | r0.wg0.wan.buetow.org:6443 | +| r1 | 192.168.1.121 | 192.168.2.121 | r1.wg0.wan.buetow.org:6443 | +| r2 | 192.168.1.122 | 192.168.2.122 | r2.wg0.wan.buetow.org:6443 | + +## Useful Commands + +```sh +kubectl get nodes # cluster status +kubectl get pods --all-namespaces # all running pods +kubectl get namespaces +kubectl config set-context --current --namespace= +``` diff --git a/prompts/skills/f3s-k3s/references/r-node-deploy.md b/prompts/skills/f3s-k3s/references/r-node-deploy.md new file mode 100644 index 0000000..294958c --- /dev/null +++ b/prompts/skills/f3s-k3s/references/r-node-deploy.md @@ -0,0 +1,144 @@ +# r-node Deploy Mechanism (Rex) + +The reusable way to roll out files and systemd units to the three k3s Rocky +Linux VMs — **r0/r1/r2** — is a [Rex](https://www.rexify.org/) task. The +canonical, fully-worked example is the **`nfs_mount_monitor`** task. Treat it +as the template for *any* r-node rollout; you do not need to re-derive the +flow each time. + +## Hosts: r0/r1/r2 vs f0/f1/f2 + +The r-nodes are Rocky Linux 9 bhyve guests, one per FreeBSD host: + +| VM | LAN IP | Runs on f-host | +|----|--------|----------------| +| r0 | 192.168.1.120 | f0 (192.168.1.130) | +| r1 | 192.168.1.121 | f1 (192.168.1.131) | +| r2 | 192.168.1.122 | f2 (192.168.1.132) | + +f3 is **not** part of this group — it is standalone bhyve and hosts the plain +`rocky` VM, not a k3s node. r-node deploys never touch f3. + +## Where it lives + +In the conf repo (`https://codeberg.org/snonux/conf`, dir `f3s/`): + +``` +f3s/r-nodes/Rexfile # deploy tasks for r0/r1/r2 +f3s/r-nodes/nfs-mount-monitor/ # source files the task pushes + check-nfs-mount.sh # → /usr/local/bin/ + nfs-mount-monitor.default # → /etc/default/ (tunables) + nfs-mount-monitor.service # → /etc/systemd/system/ + nfs-mount-monitor.timer # → /etc/systemd/system/ +``` + +## The deploy command + +Run from the **conf repo root**: + +```sh +rex -f f3s/r-nodes/Rexfile nfs_mount_monitor +``` + +This pushes to all three r-nodes at once. To target a single node, use Rex's +host filter, e.g. `rex -f f3s/r-nodes/Rexfile -H 192.168.1.120 nfs_mount_monitor`. + +## How the pattern works (the reusable parts) + +The Rexfile establishes conventions every r-node task inherits: + +- **`group r_nodes => qw(192.168.1.120 192.168.1.121 192.168.1.122)`** — the + three k3s VMs by LAN IP. Each `task` declares `group => 'r_nodes'`. +- **`user 'root'; sudo FALSE;`** — tasks connect as **root** over SSH. The + `paul` user has no sudo on the r-nodes, and writing to `/usr/local/bin` and + managing systemd both need root. Root SSH is pre-authorized via + `authorized_keys`. +- **`parallelism 3;`** — all three nodes deploy concurrently. Safe because the + tasks are idempotent and independent per node. +- **`$RNODES_DIR`** is resolved with `realpath($::rexfile)` so source-file + paths stay valid regardless of CWD or Rex worker forking. + +### Idempotent rollout flow + +Inside the task, each file is deployed with Rex's `file` resource: + +```perl +file '/usr/local/bin/check-nfs-mount.sh', + source => catfile($monitor_dir, 'check-nfs-mount.sh'), + owner => 'root', group => 'root', mode => '755', + on_change => sub { $changed = 1 }; +``` + +Rex only writes a file when its **content actually differs** from what's on +the node — so re-running the task is a no-op when nothing changed. The +`on_change` handlers set a single `$changed` flag; only if something changed +does the task run: + +```perl +run 'systemctl daemon-reload'; +run 'systemctl restart nfs-mount-monitor.timer'; +``` + +Finally — regardless of whether files changed — the task **converges** the +service state so the timer is always enabled and running: + +```perl +service 'nfs-mount-monitor.timer', ensure => 'started'; +run 'systemctl enable nfs-mount-monitor.timer'; +``` + +This split (reload/restart only on change, enable/start always) is the +pattern to copy: cheap, repeatable, and self-healing if a node drifted. + +The task also pre-creates the state/output directories it needs +(`/var/lib/nfs-mount-monitor` mode 700, +`/var/lib/node_exporter/textfile_collector` mode 755) so the deployed script +has somewhere to write. + +## Verify after deploy + +```sh +# On each r-node (over SSH as root) — confirm the timer is active & enabled: +systemctl status nfs-mount-monitor.timer + +# Watch the service fire and log: +journalctl -u nfs-mount-monitor -f +``` + +What success looks like: + +- `systemctl status …timer` reports **active (waiting)** and **enabled**, with + a `Trigger:` line a few seconds out (the timer fires every 10 s). +- `journalctl -u nfs-mount-monitor` shows a fresh oneshot run roughly every + 10 s with no errors. + +One-shot remote check from a roaming laptop (see +[remote-access.md](remote-access.md)): + +```sh +ssh -A -J rex@fishfinger.buetow.org root@r0.wg0 \ + "systemctl is-active nfs-mount-monitor.timer && systemctl is-enabled nfs-mount-monitor.timer" +``` + +## Gotchas + +- **Run from the conf repo root**, not from `f3s/r-nodes/` — the `-f` path in + every example is repo-relative (`f3s/r-nodes/Rexfile`). +- **Root, no sudo.** If a task errors on permission, it's almost always that + the connection fell back to `paul` (who has no sudo here). The Rexfile sets + `user 'root'` for exactly this reason. +- **`parallelism 3` only works if tasks are idempotent and node-independent.** + Keep new tasks that way, or lower the parallelism. +- **Adding a new file** to a deploy: give it `on_change => sub { $changed = 1 }` + so a reload/restart fires when it changes, and place it before the + `if ($changed)` block. + +## Related + +- The **behavior** of the deployed monitor (probes, fail-counter, cordon & + auto-reboot escalation, alerting) is documented separately in + [storage/nfs-mount-monitor.md](../../f3s-storage/references/nfs-mount-monitor.md). This file + covers only the *deploy mechanism*; that one covers *what gets deployed*. +- There are sibling Rexfiles in the conf repo (`f3s/garage/Rexfile`, + `frontends/Rexfile`) that follow the same group/user/idempotent-`file` + shape against different host groups. diff --git a/prompts/skills/f3s-k3s/references/remote-access.md b/prompts/skills/f3s-k3s/references/remote-access.md new file mode 100644 index 0000000..9837870 --- /dev/null +++ b/prompts/skills/f3s-k3s/references/remote-access.md @@ -0,0 +1,108 @@ +# Accessing the k3s cluster from outside the LAN (roaming) + +The kubeconfig on the laptop points at the **LAN** API endpoint +(`https://r0.lan.buetow.org:6443` → `192.168.1.120`). When **earth** (or any +client) is **not on the f3s LAN**, that address is unreachable and `kubectl` +just times out: + +``` +Unable to connect to the server: dial tcp 192.168.1.120:6443: i/o timeout +``` + +## Preferred method: a dedicated `wg0` kubectl context (direct over WireGuard) + +From **earth** the WireGuard mesh IPs **are** directly reachable +(`192.168.2.120` = `r0.wg0.wan.buetow.org`), so `kubectl` can talk straight to +the API server over `wg0` — no jump host needed. The k3s API cert already has +the `r0.wg0.wan.buetow.org` / `r1.wg0…` / `r2.wg0…` SANs (see install.md +`--tls-san`), so TLS verifies cleanly with no overrides. + +Add a second cluster + context to `~/.kube/config` alongside the LAN `default` +one, reusing the same `certificate-authority-data` and `default` user: + +```yaml +clusters: +- cluster: + certificate-authority-data: + server: https://r0.wg0.wan.buetow.org:6443 # 192.168.2.120 + name: wg0 +contexts: +- context: + cluster: wg0 + namespace: services + user: default + name: wg0 +``` + +Then switch by location: + +```sh +kubectl config use-context default # on the f3s LAN (home) +kubectl config use-context wg0 # on the road (WireGuard) +kubectl --context=wg0 get nodes # one-off without switching +``` + +Notes: + +- `default` → `https://r0.lan.buetow.org:6443` (LAN, `192.168.1.120`); `wg0` → + `https://r0.wg0.wan.buetow.org:6443` (mesh, `192.168.2.120`). +- For failover, point a `wg0` variant at `r1.wg0` / `r2.wg0` (all three are SANs + on the cert) if r0 is down. +- This requires the laptop's WireGuard to actually route to the r-VM mesh IPs. + If a particular client's `wg` does **not** peer to the r-VMs, fall back to the + OpenBSD-frontend jump method below. + +## Fallback method: jump through an OpenBSD frontend + +The OpenBSD internet gateways **fishfinger** and **blowfish** are reachable +from the public internet *and* sit on the WireGuard mesh, so they can reach the +r-VMs over `wg0`. Use one as a jump host, then run `kubectl` as `root` on the +r-VM itself. + +Interactive: + +```sh +ssh -A rex@fishfinger.buetow.org # -A forwards your agent for the next hop +ssh root@r0.wg0 # from fishfinger, over the WireGuard mesh +kubectl get nodes # root on r0 has /etc/rancher/k3s/k3s.yaml +``` + +One-shot (non-interactive) from the laptop — handy for scripts/automation: + +```sh +ssh -A rex@fishfinger.buetow.org "ssh root@r0.wg0 'kubectl get nodes'" +``` + +Notes: + +- `ssh -A` (agent forwarding) is required so the `rex@fishfinger` → `root@r0.wg0` + hop can authenticate. Make sure the right key is loaded (`ssh-add -l`). +- Use **`r0.wg0`** (or `r1.wg0` / `r2.wg0` if r0 is down) — that is the r-VM's + WireGuard hostname as seen *from the frontend*, not `r0.lan…`. +- `blowfish.buetow.org` works the same way as `fishfinger.buetow.org` if one + frontend is unavailable. +- On a fresh laptop, accept host keys with + `-o StrictHostKeyChecking=accept-new` on both hops. +- An OpenSSH **post-quantum key exchange warning** from the `root@r0.wg0` hop is + harmless and can be ignored. + +## Example: issue a goprecords client key while roaming + +```sh +ssh -A rex@fishfinger.buetow.org \ + "ssh root@r0.wg0 'kubectl exec -n services deployment/goprecords -- \ + goprecords --create-client-key mega-m3-pro -stats-dir=/data/stats'" +``` + +## Alternative: serve the API over an SSH tunnel + +If you need a real local `kubectl` (not just one-off commands), forward the API +port through the frontend and point a kubeconfig at `127.0.0.1`: + +```sh +ssh -A -L 6443:r0.wg0:6443 rex@fishfinger.buetow.org +# then, in another shell, with a kubeconfig whose server is +# https://127.0.0.1:6443 (and tls-server-name r0.wg0.wan.buetow.org, since the +# API cert has that SAN — see install.md --tls-san): +kubectl --tls-server-name r0.wg0.wan.buetow.org get nodes +``` diff --git a/prompts/skills/f3s-k3s/references/troubleshooting.md b/prompts/skills/f3s-k3s/references/troubleshooting.md new file mode 100644 index 0000000..188f464 --- /dev/null +++ b/prompts/skills/f3s-k3s/references/troubleshooting.md @@ -0,0 +1,49 @@ +# k3s Troubleshooting + +## Etcd Raft Log Corruption Recovery + +**Symptom**: k3s crashes on startup with panic: +``` +tocommit(XXXXXXX) is out of range [lastIndex(YYYYYYY)] +``` +Caused by `kill -9` on the bhyve process mid-write (corrupts etcd WAL). k3s enters a crash loop and stops after ~2 minutes. + +**Recovery procedure** (example: r1 is corrupt): + +```sh +# 1. Stop k3s on the affected node +ssh root@r1.lan.buetow.org 'systemctl stop k3s' + +# 2. Download etcdctl on a healthy node (not bundled with k3s) +ssh root@r0.lan.buetow.org +curl -sL https://github.com/etcd-io/etcd/releases/download/v3.5.17/etcd-v3.5.17-linux-amd64.tar.gz \ + | tar -xz -C /tmp etcd-v3.5.17-linux-amd64/etcdctl +mv /tmp/etcd-v3.5.17-linux-amd64/etcdctl /tmp/etcdctl + +# 3. Find and remove the corrupt member from the cluster +ETCDCTL_API=3 /tmp/etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \ + --cert=/var/lib/rancher/k3s/server/tls/etcd/client.crt \ + --key=/var/lib/rancher/k3s/server/tls/etcd/client.key \ + member list +# Find the member ID for r1, then: +ETCDCTL_API=3 /tmp/etcdctl ... member remove + +# 4. Delete the corrupted etcd data on the affected node +ssh root@r1.lan.buetow.org 'rm -rf /var/lib/rancher/k3s/server/db/etcd' + +# 5. Restart k3s — it rejoins as a fresh member +ssh root@r1.lan.buetow.org 'systemctl start k3s' + +# 6. Verify +kubectl get nodes # r1 should return to Ready +``` + +> **Prevention**: Always use `doas vm stop rocky` and wait for clean shutdown before stopping the bhyve host. Only use `kill -9` on the bhyve process as a last resort — it can corrupt the etcd WAL. + +## Cluster-wide NFS Outages + +If NFS goes down cluster-wide, the root cause is usually on the FreeBSD NFS +server side (f0/f1). Check CARP state, stunnel, nfsd, and +`vfs.nfsd.nfs_privport` — see [storage/troubleshooting.md](../../f3s-storage/references/troubleshooting.md). diff --git a/prompts/skills/f3s-observability/SKILL.md b/prompts/skills/f3s-observability/SKILL.md new file mode 100644 index 0000000..2e13926 --- /dev/null +++ b/prompts/skills/f3s-observability/SKILL.md @@ -0,0 +1,46 @@ +--- +name: f3s-observability +description: Reference skill for the f3s homelab observability stack, Prometheus, Grafana Alloy, Loki, Tempo, and alerting on the k3s cluster, plus FreeBSD host monitoring (node_exporter + recording rules). Use when working on metrics, logs, traces, dashboards, or alerts for the homelab. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)). +--- + +# f3s Observability Stack + +Observability stack deployed into the `monitoring` namespace of the k3s cluster. + +**Current state (as of 2026-05-16)**: Prometheus + Alloy only. Grafana, Loki, and Tempo are **disabled** — their ArgoCD manifests are renamed to `.disabled` and their pods do not run. + +- Grafana disabled: SQLite-on-NFS is fundamentally unreliable across pod restarts. Grafana's database gets locked when the pod reschedules to a different node. Long-term fix: migrate to local-path PVC (same pattern as navidrome). +- Loki/Tempo disabled: no log aggregation or distributed tracing until Grafana is re-enabled. +- Alloy is running but **only emits its own logs** (`logging { level = "info" }`). Log shipping to Loki and trace forwarding to Tempo are removed from its config. +- Prometheus TSDB was wiped and restarted clean (2026-05-16) after WAL corruption (zero-byte segments from a cluster blip). + +## Components + +| Component | Purpose | State | +|-----------|---------|-------| +| **Prometheus** | Time-series metrics, alerting rules, Alertmanager | **Running** | +| **Alloy** | Telemetry collector (DaemonSet) | **Running** (minimal config only) | +| **Node Exporter** | Host-level metrics (on k3s nodes AND FreeBSD hosts) | **Running** | +| **Grafana** | Visualisation and dashboarding | **Disabled** (SQLite-on-NFS) | +| **Loki** | Log aggregation (single-binary mode) | **Disabled** | +| **Tempo** | Distributed tracing backend | **Disabled** | + +## When to Use + +- Working on metrics, logs, traces, dashboards, or alerts for the homelab +- Prometheus/Alloy config, alerting, TSDB recovery, or FreeBSD host monitoring +- For the k3s cluster this runs on, see [`f3s-k3s`](../f3s-k3s/SKILL.md); for hosts/IPs, the [`f3s`](../f3s/SKILL.md) hub. + +## Reference Files + +- [Stack](references/stack.md) — install Prometheus / Alloy / Loki / Tempo, alerting → Gogios, Prometheus TSDB recovery, LogQL queries, NFS storage paths +- [FreeBSD Monitoring](references/freebsd.md) — `node_exporter` on f-hosts, scrape config, memory & ZFS recording rules + +## Monitoring Scope + +- Kubernetes workloads (pod health, resource usage) +- Node-level metrics (CPU, memory, disk) — both k3s and FreeBSD nodes +- ZFS ARC statistics on FreeBSD hosts +- Application performance metrics +- ~~Log aggregation from all pods (via Alloy → Loki)~~ — disabled +- ~~Distributed traces (via Alloy → Tempo)~~ — disabled diff --git a/prompts/skills/f3s-observability/references/freebsd.md b/prompts/skills/f3s-observability/references/freebsd.md new file mode 100644 index 0000000..d469965 --- /dev/null +++ b/prompts/skills/f3s-observability/references/freebsd.md @@ -0,0 +1,112 @@ +# Monitoring FreeBSD Hosts (f0, f1, f2, f3) + +Scraping the FreeBSD bhyve hosts from in-cluster Prometheus. Includes the +node_exporter setup, the additional scrape config, and the recording rules +that bridge FreeBSD metric names into the Linux-style names Grafana +dashboards expect. + +## Install node_exporter on FreeBSD + +```sh +# On each FreeBSD host +doas pkg install -y node_exporter +doas sysrc node_exporter_enable=YES +# Bind to WireGuard interface (f0=192.168.2.130, f1=192.168.2.131, f2=192.168.2.132, f3=192.168.2.133) +doas sysrc node_exporter_args='--web.listen-address=192.168.2.130:9100' +doas service node_exporter start +``` + +## Prometheus scrape config for FreeBSD + +`additional-scrape-configs.yaml`: + +```yaml +- job_name: 'node-exporter' + static_configs: + - targets: + - '192.168.2.130:9100' # f0 via WireGuard + - '192.168.2.131:9100' # f1 via WireGuard + - '192.168.2.132:9100' # f2 via WireGuard + - '192.168.2.133:9100' # f3 via WireGuard + labels: + os: freebsd +``` + +```sh +kubectl create secret generic additional-scrape-configs \ + --from-file=additional-scrape-configs.yaml -n monitoring +``` + +Add to `persistence-values.yaml`: + +```yaml +prometheus: + prometheusSpec: + additionalScrapeConfigsSecret: + enabled: true + name: additional-scrape-configs + key: additional-scrape-configs.yaml +``` + +## FreeBSD memory compatibility rules + +FreeBSD uses different metric names than Linux. PrometheusRule to create Linux-compatible metrics: + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: freebsd-memory-rules + namespace: monitoring + labels: + release: prometheus +spec: + groups: + - name: freebsd-memory + rules: + - record: node_memory_MemTotal_bytes + expr: node_memory_size_bytes{os="freebsd"} + - record: node_memory_MemAvailable_bytes + expr: | + node_memory_free_bytes{os="freebsd"} + + node_memory_inactive_bytes{os="freebsd"} + + node_memory_cache_bytes{os="freebsd"} + - record: node_memory_MemFree_bytes + expr: node_memory_free_bytes{os="freebsd"} + - record: node_memory_Buffers_bytes + expr: node_memory_buffer_bytes{os="freebsd"} + - record: node_memory_Cached_bytes + expr: node_memory_cache_bytes{os="freebsd"} +``` + +Note: Disk I/O metrics (`node_disk_*`) are not available on FreeBSD — use ZFS-specific dashboards instead. + +## ZFS monitoring recording rules + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: freebsd-zfs-rules + namespace: monitoring + labels: + release: prometheus +spec: + groups: + - name: freebsd-zfs-arc + interval: 30s + rules: + - record: node_zfs_arc_hit_rate_percent + expr: | + 100 * ( + rate(node_zfs_arcstats_hits_total{os="freebsd"}[5m]) / + (rate(node_zfs_arcstats_hits_total{os="freebsd"}[5m]) + + rate(node_zfs_arcstats_misses_total{os="freebsd"}[5m])) + ) + - record: node_zfs_arc_memory_usage_percent + expr: | + 100 * ( + node_zfs_arcstats_size_bytes{os="freebsd"} / + node_zfs_arcstats_c_max_bytes{os="freebsd"} + ) +``` diff --git a/prompts/skills/f3s-observability/references/stack.md b/prompts/skills/f3s-observability/references/stack.md new file mode 100644 index 0000000..bd30874 --- /dev/null +++ b/prompts/skills/f3s-observability/references/stack.md @@ -0,0 +1,158 @@ +# Observability Stack (k3s side) + +Install and operation of the in-cluster observability components: Prometheus, +Alloy, Loki, Tempo, Alertmanager → Gogios. + +## Deployment + +All components deployed via **ArgoCD** (GitOps). Manifests: +``` +https://codeberg.org/snonux/conf/src/branch/master/f3s +argocd-apps/monitoring/ +``` + +Deployment tool: `just` (Justfile in each component directory). + +### Namespaces + +```sh +kubectl create namespace monitoring +``` + +### Disabled component manifests + +These files exist in the repo but are renamed `.disabled` so ArgoCD ignores them: +``` +f3s/argocd-apps/monitoring/loki.yaml.disabled +f3s/argocd-apps/monitoring/tempo.yaml.disabled +f3s/argocd-apps/monitoring/grafana-ingress.yaml.disabled +``` + +To re-enable, rename back to `.yaml` and ensure Grafana is using a non-NFS PVC (local-path). + +## Installing Prometheus + +Uses `kube-prometheus-stack` Helm chart with **Grafana subchart disabled** (`grafana.enabled: false`): + +```sh +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo update + +# Create NFS storage directory first +mkdir -p /data/nfs/k3svolumes/prometheus/data + +cd conf/f3s/prometheus && just install +``` + +### Enable etcd and controller-manager scraping + +Add to `persistence-values.yaml`: + +```yaml +kubeEtcd: + enabled: true + endpoints: [192.168.2.120, 192.168.2.121, 192.168.2.122] + service: + port: 2381 + targetPort: 2381 + +kubeControllerManager: + enabled: true + endpoints: [192.168.2.120, 192.168.2.121, 192.168.2.122] + service: + port: 10257 + targetPort: 10257 + serviceMonitor: + enabled: true + https: true + insecureSkipVerify: true +``` + +Also requires k3s config changes on each r node — see [k3s-setup/install.md](../../f3s-k3s/references/install.md). + +### Grafana credentials + +Default: `admin` / `prom-operator` — change immediately after first login. + +Grafana accessible at `grafana.f3s.foo.zone` via Traefik ingress. + +## Installing Alloy (minimal) + +Alloy is installed as part of the Loki Helm chart but runs with a minimal config (no log shipping): + +```sh +cd conf/f3s/loki && just install +# installs alloy only (loki itself is disabled via loki.yaml.disabled) +``` + +### Current Alloy config (`alloy-values.yaml`) + +Minimal — only emits Alloy's own operational logs: + +``` +logging { + level = "info" +} +``` + +To re-enable log shipping (once Loki is running again), restore the full `discovery.kubernetes` + `loki.source.kubernetes` + `loki.write` pipeline. + +## Installing Loki (disabled) + +```sh +mkdir -p /data/nfs/k3svolumes/loki/data +# Rename loki.yaml.disabled → loki.yaml first, then: +cd conf/f3s/loki && just install +``` + +Loki URL (internal): `http://loki.monitoring.svc.cluster.local:3100` + +## Installing Tempo (disabled) + +```sh +mkdir -p /data/nfs/k3svolumes/tempo/data +# Rename tempo.yaml.disabled → tempo.yaml first, then: +cd conf/f3s/tempo && just install +``` + +## Alerting + +Prometheus → Alertmanager → **Gogios** (custom lightweight monitoring tool running on OpenBSD gateway `blowfish`/`fishfinger`). + +Gogios scrapes Alertmanager at regular intervals and sends email notifications. Reaches Alertmanager via WireGuard mesh. + +## Prometheus TSDB Recovery + +If Prometheus fails to start with `opening storage failed: get segment range: segments are not sequential`, WAL segments are corrupt (can happen after a cluster blip leaving zero-byte WAL files). + +Full TSDB wipe (loses all historical data — confirm first): + +```sh +# On the NFS server (f0 or CARP MASTER) +rm -rf /data/nfs/k3svolumes/prometheus/data/prometheus-db/ +mkdir -p /data/nfs/k3svolumes/prometheus/data/prometheus-db +chown 1000:1000 /data/nfs/k3svolumes/prometheus/data/prometheus-db +# Prometheus will recreate the TSDB on next start +``` + +## Useful LogQL Queries + +``` +# All logs from services namespace +{namespace="services"} + +# Filter by log content +{namespace="services"} |= "error" + +# Parse JSON logs +{namespace="services"} | json | level="error" +``` + +## NFS Storage Paths + +``` +/data/nfs/k3svolumes/prometheus/data # active +/data/nfs/k3svolumes/grafana/data # exists but unused (grafana disabled) +/data/nfs/k3svolumes/loki/data # exists but unused (loki disabled) +/data/nfs/k3svolumes/tempo/data # exists but unused (tempo disabled) +``` diff --git a/prompts/skills/f3s-raspberry-pi/SKILL.md b/prompts/skills/f3s-raspberry-pi/SKILL.md new file mode 100644 index 0000000..0ed3ac1 --- /dev/null +++ b/prompts/skills/f3s-raspberry-pi/SKILL.md @@ -0,0 +1,54 @@ +--- +name: f3s-raspberry-pi +description: Reference skill for the four Raspberry Pi 3 nodes of the f3s homelab, pi0/pi1 run NetBSD 10.1 (aarch64) serving static f3s.buetow.org / snonux.foo via bozohttpd behind OpenBSD relayd over WireGuard; pi2/pi3 run Rocky Linux 9 with Pi-hole in Docker and LAN wildcard DNS (`*.f3s.lan.buetow.org` to 192.168.1.138). Covers doas/pkgin bootstrap, bozohttpd vhosting, npf, uptimed, content sync, and the doas-alias shutdown pitfall. Use when configuring or troubleshooting the Pi nodes, the static site, or Pi-hole/LAN DNS. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)). +--- + +# f3s Raspberry Pi Nodes + +The four Raspberry Pi 3 nodes of the f3s homelab. The master host/IP inventory +(pi0–pi3 rows) lives in the [`f3s`](../f3s/SKILL.md) hub's Host-IP table. + +## When to Use + +- Configuring or troubleshooting pi0–pi3 (NetBSD static site pair, or Rocky Pi-hole pair) +- The static `f3s.buetow.org` / `snonux.foo` site (bozohttpd, relayd forwarding, vhosts) +- Pi-hole and `*.f3s.lan.buetow.org` LAN wildcard DNS +- For the WireGuard mesh these depend on, see the [`f3s`](../f3s/SKILL.md) hub's `wireguard.md`; for DTail/dserver on the Pis, [`f3s-dtail`](../f3s-dtail/SKILL.md); for building the NetBSD dserver package, the [`pkgrepo`](../pkgrepo/SKILL.md) skill. + +## Node roles + +`pi2`/`pi3` run Rocky Linux 9.2 (Blue Onyx) aarch64 from the SIG/AltArch image (`RockyLinuxRpi_9-latest.img.xz`). `pi0` and `pi1` run **NetBSD 10.1** (evbarm-aarch64). Each Rocky Pi has: + +- User `paul` with passwordless sudo and SSH key auth +- Static IP on eth0 via NetworkManager +- Hostname `piN.lan.buetow.org` +- Filesystem expanded with `rootfs-expand` +- Default `rocky` user still present (password: `rockylinux`) +- No GRUB — boots via Pi's native bootloader (`/boot/cmdline.txt`) +- Custom RPi kernel from the `rockyrpi` repo + +`pi0`/`pi1` (NetBSD) differ: user `paul` in `wheel`, privilege escalation via a **real `doas`** (pkgsrc `security/doas`, `permit nopass :wheel`) — not the `alias doas=sudo` shell alias `pi2`/`pi3` carry in `/etc/profile.d/doas.sh`, which doesn't expand in the non-interactive shell an SSH command runs in and so silently breaks `wol-f3s shutdown-pis`/`shutdown-all` for the Rocky Pis (`doas poweroff` resolves to nothing) — only the NetBSD nodes actually work with that script today. Config repo home for NetBSD-specific setup: `f3s/pi-netbsd/`. Service setup details: [NetBSD Pi Setup](references/bootstrap-netbsd-pi.md). + +Current role split: + +- `pi0` and `pi1` serve static `f3s.buetow.org`/`snonux.foo` content behind OpenBSD `relayd` over WireGuard. WireGuard peers are `blowfish`, `fishfinger`, **and `rocky`** (not gateway-only to just the two frontends, despite older docs here). All rc.d services (`wireguard`, `bozohttpd`, `uptimed`, `npf`, `dserver`) and both crontabs are enabled via `rc.conf` and come back automatically on reboot. +- `pi2` and `pi3` run **Pi-hole** in Docker (`network_mode: host`, `~/pihole` on each host). Tracked dnsmasq LAN wildcard: **`f3s/pihole/docker-pi/`** in the conf repo; details in [references/pihole-pi.md](references/pihole-pi.md). + +## Webserver (pi0/pi1 static site) + +`pi0`/`pi1` serve `f3s.buetow.org`/`snonux.foo` with **bozohttpd** (NetBSD base) behind +the OpenBSD `relayd` frontends. Vhosting is directory-based: a vhost needs a directory +*literally* named after the hostname (`snonux.foo/`, with `www.snonux.foo` a symlink), +and `-X` enables directory indexing. Because `relayd` **cannot rewrite URL paths** (it +forwards the original path intact), each domain is mapped to its docroot subdir via the +`Host` header. Docroot `/var/www/html`; `pi1` syncs the docroot hourly from `pi0` (the +source of truth); SSH `paul@piN.lan.buetow.org -p 22`. + +The full bozohttpd setup — the custom `/etc/rc.d/bozohttpd`, the `-V` fallback +system-hostname redirect pitfall, and the self-referencing vhost symlink fix — is the +canonical detail in [references/bootstrap-netbsd-pi.md](