summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2026-03-18 14:04:36 +0200
committerPaul Buetow <paul@buetow.org>2026-03-18 14:04:36 +0200
commita34aab167ace5e851e9ee434b8b8425b800b5a73 (patch)
treeb1ae780a24ffa4e8c1bfd16cbbf29a692663e11e
parent96048c65d2be8d6a00af1e14b7e61f095842efc9 (diff)
Update
-rw-r--r--fish/conf.d/zoxide.fish2
-rw-r--r--prompts/skills/f3s/references/freebsd-setup.md3
-rw-r--r--prompts/skills/f3s/references/k3s-setup.md42
-rw-r--r--prompts/skills/f3s/references/rocky-linux-vms.md9
-rw-r--r--prompts/skills/f3s/references/storage.md10
-rw-r--r--prompts/skills/f3s/references/wireguard.md5
6 files changed, 68 insertions, 3 deletions
diff --git a/fish/conf.d/zoxide.fish b/fish/conf.d/zoxide.fish
index 4005ebb..18b2457 100644
--- a/fish/conf.d/zoxide.fish
+++ b/fish/conf.d/zoxide.fish
@@ -3,3 +3,5 @@ if type -q zoxide
else
echo "zoxide not installed?"
end
+
+abbr z zi
diff --git a/prompts/skills/f3s/references/freebsd-setup.md b/prompts/skills/f3s/references/freebsd-setup.md
index 247fc1d..d2712fc 100644
--- a/prompts/skills/f3s/references/freebsd-setup.md
+++ b/prompts/skills/f3s/references/freebsd-setup.md
@@ -51,7 +51,8 @@ doas vm start rocky
Breaking changes in 15.0 to watch for:
- **bhyve PCI BARs**: if VM fails to boot, add `pci.enable_bars='true'` to `/zroot/bhyve/rocky/rocky.conf`
-- **NFS privileged ports**: if NFS mounts break on r0/r1/r2, add `resvport` to Rocky Linux mount options or `--no-resvport` to NFS server flags
+- **NFS privileged ports**: FreeBSD 15.0 sets `vfs.nfsd.nfs_privport=1` by default, blocking NFS clients connecting via stunnel (unprivileged ports). Fix: add `vfs.nfsd.nfs_privport=0` to `/etc/sysctl.conf` on each f-host, then `doas sysctl vfs.nfsd.nfs_privport=0` to apply immediately, and remount NFS on r-hosts with `mount -a`.
+- **WireGuard interface address**: FreeBSD 15.0 requires a prefix length when setting interface addresses. Add `/32` to IPv4 `Address` lines in `/usr/local/etc/wireguard/wg0.conf` (e.g. `Address = 192.168.2.130/32`). Without this, `service wireguard start` fails with "setting interface address without mask is no longer supported".
Current version: **FreeBSD 15.0-RELEASE** (as of Part 8, upgraded from 14.3).
diff --git a/prompts/skills/f3s/references/k3s-setup.md b/prompts/skills/f3s/references/k3s-setup.md
index 3351eb9..ab9eb0d 100644
--- a/prompts/skills/f3s/references/k3s-setup.md
+++ b/prompts/skills/f3s/references/k3s-setup.md
@@ -271,6 +271,48 @@ spec:
number: 8080
```
+## Etcd Raft Log Corruption Recovery
+
+**Symptom**: k3s crashes on startup with panic:
+```
+tocommit(XXXXXXX) is out of range [lastIndex(YYYYYYY)]
+```
+Caused by `kill -9` on the bhyve process mid-write (corrupts etcd WAL). k3s enters a crash loop and stops after ~2 minutes.
+
+**Recovery procedure** (example: r1 is corrupt):
+
+```sh
+# 1. Stop k3s on the affected node
+ssh root@r1.lan.buetow.org 'systemctl stop k3s'
+
+# 2. Download etcdctl on a healthy node (not bundled with k3s)
+ssh root@r0.lan.buetow.org
+curl -sL https://github.com/etcd-io/etcd/releases/download/v3.5.17/etcd-v3.5.17-linux-amd64.tar.gz \
+ | tar -xz -C /tmp etcd-v3.5.17-linux-amd64/etcdctl
+mv /tmp/etcd-v3.5.17-linux-amd64/etcdctl /tmp/etcdctl
+
+# 3. Find and remove the corrupt member from the cluster
+ETCDCTL_API=3 /tmp/etcdctl \
+ --endpoints=https://127.0.0.1:2379 \
+ --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
+ --cert=/var/lib/rancher/k3s/server/tls/etcd/client.crt \
+ --key=/var/lib/rancher/k3s/server/tls/etcd/client.key \
+ member list
+# Find the member ID for r1, then:
+ETCDCTL_API=3 /tmp/etcdctl ... member remove <MEMBER_ID>
+
+# 4. Delete the corrupted etcd data on the affected node
+ssh root@r1.lan.buetow.org 'rm -rf /var/lib/rancher/k3s/server/db/etcd'
+
+# 5. Restart k3s — it rejoins as a fresh member
+ssh root@r1.lan.buetow.org 'systemctl start k3s'
+
+# 6. Verify
+kubectl get nodes # r1 should return to Ready
+```
+
+> **Prevention**: Always use `doas vm stop rocky` and wait for clean shutdown before stopping the bhyve host. Only use `kill -9` on the bhyve process as a last resort — it can corrupt the etcd WAL.
+
## Useful Commands
```sh
diff --git a/prompts/skills/f3s/references/rocky-linux-vms.md b/prompts/skills/f3s/references/rocky-linux-vms.md
index 2be47c2..e0700f4 100644
--- a/prompts/skills/f3s/references/rocky-linux-vms.md
+++ b/prompts/skills/f3s/references/rocky-linux-vms.md
@@ -154,7 +154,14 @@ doas vm start rocky
```sh
doas vm list # list all VMs and state
doas vm start rocky # start VM
-doas vm stop rocky # graceful stop
+doas vm stop rocky # graceful ACPI stop (can be slow)
doas vm reset rocky # force reset
doas sockstat -4 | grep 5900 # check VNC port
```
+
+> **`vm stop` is ACPI-only** — it sends a shutdown signal but does not wait. If the VM does not shut down within a reasonable time, force-kill the bhyve process:
+> ```sh
+> doas vm list # note the PID in parentheses, e.g. Running (2086)
+> doas kill -9 2086
+> ```
+> **Warning**: Force-killing bhyve with `kill -9` mid-write can corrupt the k3s etcd WAL on the Rocky VM, causing a crash loop on next start. Only use as a last resort, and check etcd health after. See k3s-setup.md for the recovery procedure.
diff --git a/prompts/skills/f3s/references/storage.md b/prompts/skills/f3s/references/storage.md
index 857645c..77b62e6 100644
--- a/prompts/skills/f3s/references/storage.md
+++ b/prompts/skills/f3s/references/storage.md
@@ -346,6 +346,16 @@ doas mkdir -p /data/nfs/k3svolumes
doas chmod 755 /data/nfs/k3svolumes
```
+> **FreeBSD 15.0 note**: FreeBSD 15.0 changed the default for `vfs.nfsd.nfs_privport` from `0` to `1`, requiring NFS clients to connect from privileged ports (<1024). NFS over stunnel uses unprivileged ports, so this breaks all NFS mounts on the r-hosts. Fix on **each f-host**:
+> ```sh
+> # Apply immediately
+> doas sysctl vfs.nfsd.nfs_privport=0
+> # Persist across reboots
+> echo "vfs.nfsd.nfs_privport=0" | doas tee -a /etc/sysctl.conf
+> # Remount on each r-host
+> mount -a
+> ```
+
`/etc/exports` (stunnel clients appear as localhost):
```
diff --git a/prompts/skills/f3s/references/wireguard.md b/prompts/skills/f3s/references/wireguard.md
index 4cebc86..6ce7a82 100644
--- a/prompts/skills/f3s/references/wireguard.md
+++ b/prompts/skills/f3s/references/wireguard.md
@@ -94,10 +94,13 @@ Apply with: `doas pfctl -f /etc/pf.conf`
## Example wg0.conf (f0)
+> **FreeBSD 15.0 note**: The IPv4 `Address` line **must** include a prefix length (e.g. `/32`). Without it, `service wireguard start` fails: "setting interface address without mask is no longer supported". The IPv6 address already has `/64` so is unaffected.
+
```
[Interface]
# f0.wg0.wan.buetow.org
-Address = 192.168.2.130
+Address = 192.168.2.130/32
+Address = fd42:beef:cafe:2::130/64
PrivateKey = **************************
ListenPort = 56709