diff options
| author | Paul Buetow <paul@buetow.org> | 2026-03-18 14:04:36 +0200 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2026-03-18 14:04:36 +0200 |
| commit | a34aab167ace5e851e9ee434b8b8425b800b5a73 (patch) | |
| tree | b1ae780a24ffa4e8c1bfd16cbbf29a692663e11e | |
| parent | 96048c65d2be8d6a00af1e14b7e61f095842efc9 (diff) | |
Update
| -rw-r--r-- | fish/conf.d/zoxide.fish | 2 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/freebsd-setup.md | 3 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/k3s-setup.md | 42 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/rocky-linux-vms.md | 9 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/storage.md | 10 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/wireguard.md | 5 |
6 files changed, 68 insertions, 3 deletions
diff --git a/fish/conf.d/zoxide.fish b/fish/conf.d/zoxide.fish index 4005ebb..18b2457 100644 --- a/fish/conf.d/zoxide.fish +++ b/fish/conf.d/zoxide.fish @@ -3,3 +3,5 @@ if type -q zoxide else echo "zoxide not installed?" end + +abbr z zi diff --git a/prompts/skills/f3s/references/freebsd-setup.md b/prompts/skills/f3s/references/freebsd-setup.md index 247fc1d..d2712fc 100644 --- a/prompts/skills/f3s/references/freebsd-setup.md +++ b/prompts/skills/f3s/references/freebsd-setup.md @@ -51,7 +51,8 @@ doas vm start rocky Breaking changes in 15.0 to watch for: - **bhyve PCI BARs**: if VM fails to boot, add `pci.enable_bars='true'` to `/zroot/bhyve/rocky/rocky.conf` -- **NFS privileged ports**: if NFS mounts break on r0/r1/r2, add `resvport` to Rocky Linux mount options or `--no-resvport` to NFS server flags +- **NFS privileged ports**: FreeBSD 15.0 sets `vfs.nfsd.nfs_privport=1` by default, blocking NFS clients connecting via stunnel (unprivileged ports). Fix: add `vfs.nfsd.nfs_privport=0` to `/etc/sysctl.conf` on each f-host, then `doas sysctl vfs.nfsd.nfs_privport=0` to apply immediately, and remount NFS on r-hosts with `mount -a`. +- **WireGuard interface address**: FreeBSD 15.0 requires a prefix length when setting interface addresses. Add `/32` to IPv4 `Address` lines in `/usr/local/etc/wireguard/wg0.conf` (e.g. `Address = 192.168.2.130/32`). Without this, `service wireguard start` fails with "setting interface address without mask is no longer supported". Current version: **FreeBSD 15.0-RELEASE** (as of Part 8, upgraded from 14.3). diff --git a/prompts/skills/f3s/references/k3s-setup.md b/prompts/skills/f3s/references/k3s-setup.md index 3351eb9..ab9eb0d 100644 --- a/prompts/skills/f3s/references/k3s-setup.md +++ b/prompts/skills/f3s/references/k3s-setup.md @@ -271,6 +271,48 @@ spec: number: 8080 ``` +## Etcd Raft Log Corruption Recovery + +**Symptom**: k3s crashes on startup with panic: +``` +tocommit(XXXXXXX) is out of range [lastIndex(YYYYYYY)] +``` +Caused by `kill -9` on the bhyve process mid-write (corrupts etcd WAL). k3s enters a crash loop and stops after ~2 minutes. + +**Recovery procedure** (example: r1 is corrupt): + +```sh +# 1. Stop k3s on the affected node +ssh root@r1.lan.buetow.org 'systemctl stop k3s' + +# 2. Download etcdctl on a healthy node (not bundled with k3s) +ssh root@r0.lan.buetow.org +curl -sL https://github.com/etcd-io/etcd/releases/download/v3.5.17/etcd-v3.5.17-linux-amd64.tar.gz \ + | tar -xz -C /tmp etcd-v3.5.17-linux-amd64/etcdctl +mv /tmp/etcd-v3.5.17-linux-amd64/etcdctl /tmp/etcdctl + +# 3. Find and remove the corrupt member from the cluster +ETCDCTL_API=3 /tmp/etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \ + --cert=/var/lib/rancher/k3s/server/tls/etcd/client.crt \ + --key=/var/lib/rancher/k3s/server/tls/etcd/client.key \ + member list +# Find the member ID for r1, then: +ETCDCTL_API=3 /tmp/etcdctl ... member remove <MEMBER_ID> + +# 4. Delete the corrupted etcd data on the affected node +ssh root@r1.lan.buetow.org 'rm -rf /var/lib/rancher/k3s/server/db/etcd' + +# 5. Restart k3s — it rejoins as a fresh member +ssh root@r1.lan.buetow.org 'systemctl start k3s' + +# 6. Verify +kubectl get nodes # r1 should return to Ready +``` + +> **Prevention**: Always use `doas vm stop rocky` and wait for clean shutdown before stopping the bhyve host. Only use `kill -9` on the bhyve process as a last resort — it can corrupt the etcd WAL. + ## Useful Commands ```sh diff --git a/prompts/skills/f3s/references/rocky-linux-vms.md b/prompts/skills/f3s/references/rocky-linux-vms.md index 2be47c2..e0700f4 100644 --- a/prompts/skills/f3s/references/rocky-linux-vms.md +++ b/prompts/skills/f3s/references/rocky-linux-vms.md @@ -154,7 +154,14 @@ doas vm start rocky ```sh doas vm list # list all VMs and state doas vm start rocky # start VM -doas vm stop rocky # graceful stop +doas vm stop rocky # graceful ACPI stop (can be slow) doas vm reset rocky # force reset doas sockstat -4 | grep 5900 # check VNC port ``` + +> **`vm stop` is ACPI-only** — it sends a shutdown signal but does not wait. If the VM does not shut down within a reasonable time, force-kill the bhyve process: +> ```sh +> doas vm list # note the PID in parentheses, e.g. Running (2086) +> doas kill -9 2086 +> ``` +> **Warning**: Force-killing bhyve with `kill -9` mid-write can corrupt the k3s etcd WAL on the Rocky VM, causing a crash loop on next start. Only use as a last resort, and check etcd health after. See k3s-setup.md for the recovery procedure. diff --git a/prompts/skills/f3s/references/storage.md b/prompts/skills/f3s/references/storage.md index 857645c..77b62e6 100644 --- a/prompts/skills/f3s/references/storage.md +++ b/prompts/skills/f3s/references/storage.md @@ -346,6 +346,16 @@ doas mkdir -p /data/nfs/k3svolumes doas chmod 755 /data/nfs/k3svolumes ``` +> **FreeBSD 15.0 note**: FreeBSD 15.0 changed the default for `vfs.nfsd.nfs_privport` from `0` to `1`, requiring NFS clients to connect from privileged ports (<1024). NFS over stunnel uses unprivileged ports, so this breaks all NFS mounts on the r-hosts. Fix on **each f-host**: +> ```sh +> # Apply immediately +> doas sysctl vfs.nfsd.nfs_privport=0 +> # Persist across reboots +> echo "vfs.nfsd.nfs_privport=0" | doas tee -a /etc/sysctl.conf +> # Remount on each r-host +> mount -a +> ``` + `/etc/exports` (stunnel clients appear as localhost): ``` diff --git a/prompts/skills/f3s/references/wireguard.md b/prompts/skills/f3s/references/wireguard.md index 4cebc86..6ce7a82 100644 --- a/prompts/skills/f3s/references/wireguard.md +++ b/prompts/skills/f3s/references/wireguard.md @@ -94,10 +94,13 @@ Apply with: `doas pfctl -f /etc/pf.conf` ## Example wg0.conf (f0) +> **FreeBSD 15.0 note**: The IPv4 `Address` line **must** include a prefix length (e.g. `/32`). Without it, `service wireguard start` fails: "setting interface address without mask is no longer supported". The IPv6 address already has `/64` so is unaffected. + ``` [Interface] # f0.wg0.wan.buetow.org -Address = 192.168.2.130 +Address = 192.168.2.130/32 +Address = fd42:beef:cafe:2::130/64 PrivateKey = ************************** ListenPort = 56709 |
