From 53da530eeb5016ec064d231d6be7aba08bd844d7 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Wed, 18 Mar 2026 22:58:42 +0200 Subject: f3: add fourth host, migrate FreeBSD VM from f0, update docs, zrepl, apcupsd, WireGuard --- prompts/skills/f3s/SKILL.md | 7 +-- prompts/skills/f3s/references/freebsd-setup.md | 35 ++++++++++++- prompts/skills/f3s/references/hardware.md | 3 +- prompts/skills/f3s/references/rocky-linux-vms.md | 57 +++++++++++++++++++- prompts/skills/f3s/references/storage.md | 66 ++++++++++++++++++++++-- prompts/skills/f3s/references/ups-power.md | 8 +-- prompts/skills/f3s/references/wireguard.md | 31 ++++++++++- 7 files changed, 189 insertions(+), 18 deletions(-) diff --git a/prompts/skills/f3s/SKILL.md b/prompts/skills/f3s/SKILL.md index 2d5f806..33769c2 100644 --- a/prompts/skills/f3s/SKILL.md +++ b/prompts/skills/f3s/SKILL.md @@ -1,11 +1,11 @@ --- name: f3s -description: Reference skill for the f3s homelab—three Beelink S12 Pro hosts (f0/f1/f2) running FreeBSD with Rocky Linux Bhyve VMs (r0/r1/r2) and a k3s Kubernetes cluster. Use when troubleshooting or making configuration decisions for the f3s setup. +description: Reference skill for the f3s homelab—four Beelink S12 Pro hosts (f0/f1/f2/f3) running FreeBSD with Rocky Linux Bhyve VMs and a k3s Kubernetes cluster. f0/f1/f2 run r0/r1/r2 k3s nodes; f3 is standalone bhyve only (not part of k3s). Use when troubleshooting or making configuration decisions for the f3s setup. --- # f3s Homelab Reference -**f3s** = **f**reeBSD + **k3s**. Three physical Beelink S12 Pro mini-PCs (Intel N100) running FreeBSD as the base OS, each hosting a Rocky Linux 9 bhyve VM, forming a 3-node HA k3s Kubernetes cluster. +**f3s** = **f**reeBSD + **k3s**. Four physical Beelink S12 Pro mini-PCs (Intel N100) running FreeBSD as the base OS. f0/f1/f2 each host a Rocky Linux 9 bhyve VM forming a 3-node HA k3s Kubernetes cluster. f3 is a standalone host for bhyve VMs only — not part of the k3s cluster. ## When to Use @@ -20,7 +20,7 @@ Detailed reference documentation is in the `references/` subfolder: - [Hardware](references/hardware.md) — Beelink S12 Pro specs, network switch, IPs, MAC addresses, Wake-on-LAN - [FreeBSD Setup](references/freebsd-setup.md) — Base OS install, packages, ZFS snapshots, configuration - [UPS & Power](references/ups-power.md) — APC BX750MI, apcupsd config on f0/f1/f2 -- [Rocky Linux VMs](references/rocky-linux-vms.md) — Bhyve, vm-bhyve, VM config, NVMe disk fix +- [Rocky Linux VMs](references/rocky-linux-vms.md) — Bhyve, vm-bhyve, VM config, NVMe disk fix; FreeBSD VM on f3 (migrated from f0) - [WireGuard Mesh](references/wireguard.md) — Mesh topology, IP assignments, peer configs - [Storage](references/storage.md) — ZFS (zdata), CARP, NFS over stunnel, zrepl replication - [k3s Setup](references/k3s-setup.md) — HA k3s cluster, etcd, node IPs, kubeconfig, ArgoCD @@ -33,6 +33,7 @@ Detailed reference documentation is in the `references/` subfolder: | f0 | FreeBSD host | 192.168.1.130 | 192.168.2.130 | | f1 | FreeBSD host | 192.168.1.131 | 192.168.2.131 | | f2 | FreeBSD host | 192.168.1.132 | 192.168.2.132 | +| f3 | FreeBSD host (standalone bhyve, not k3s) | 192.168.1.133 | 192.168.2.133 | | r0 | Rocky Linux VM on f0 | 192.168.1.120 | 192.168.2.120 | | r1 | Rocky Linux VM on f1 | 192.168.1.121 | 192.168.2.121 | | r2 | Rocky Linux VM on f2 | 192.168.1.122 | 192.168.2.122 | diff --git a/prompts/skills/f3s/references/freebsd-setup.md b/prompts/skills/f3s/references/freebsd-setup.md index d2712fc..d55fb8f 100644 --- a/prompts/skills/f3s/references/freebsd-setup.md +++ b/prompts/skills/f3s/references/freebsd-setup.md @@ -49,7 +49,37 @@ doas vm start rocky # kubectl get nodes (from laptop — node should be Ready) ``` -Breaking changes in 15.0 to watch for: +## Slow SSH Login / DNS Troubleshooting + +If SSH logins take ~30 seconds, the cause is reverse DNS lookup timing out. Root cause on bhyve VMs: SLAAC (Router Advertisement RDNSS) injects an unreachable IPv6 nameserver into `/etc/resolv.conf` via `resolvconf`, and it's queried first. + +### Proper fix: `/etc/resolvconf.conf` + +Mark the VM's NIC as `private_interfaces` so SLAAC DNS is not added globally, and pin the IPv4 nameserver: + +```sh +# On bhyve VMs — interface name is typically vtnet0 +cat < doas kill -9 2086 > ``` > **Warning**: Force-killing bhyve with `kill -9` mid-write can corrupt the k3s etcd WAL on the Rocky VM, causing a crash loop on next start. Only use as a last resort, and check etcd health after. See k3s-setup.md for the recovery procedure. + +## FreeBSD Bhyve VM on f3 + +f3 hosts a standalone FreeBSD development VM (not part of k3s). It was migrated from f0 via `zfs send`. + +### VM config (`/zroot/bhyve/freebsd/freebsd.conf`) + +``` +loader="bhyveload" +cpu=4 +memory=14G +network0_type="virtio-net" +network0_switch="public" +disk0_type="nvme" +disk0_name="disk0.img" # 20GB OS disk +disk1_type="nvme" +disk1_name="disk1.img" # 100GB data disk +uuid="" +network0_mac="" +``` + +- Accessible as `freebsd.lan` (hostname inside the VM) +- Auto-starts on f3 boot: `vm_list="freebsd"` in `/etc/rc.conf` +- `zroot/bhyve/freebsd` encrypted with `f3.lan.buetow.org:bhyve.key` +- Replicated to f2 via zrepl (`f3_to_f2_freebsd` job, every 10 min → `zroot/sink/f3/zroot/bhyve/freebsd`) + +### Migration procedure (zfs send) + +```sh +# On source host — snapshot and send +doas zfs snapshot zroot/bhyve/@migrate +ssh sourcehost 'doas zfs send zroot/bhyve/@migrate' | ssh desthost 'doas zfs recv -u zroot/bhyve/' + +# On dest host — mount and start +doas zfs mount zroot/bhyve/ +doas vm start +``` + +Note: Non-raw send re-encrypts under the destination's `zroot/bhyve` key automatically. + +### Slow SSH login inside the VM + +If `ssh freebsd.lan` takes ~30 seconds, SLAAC is injecting an unreachable IPv6 DNS server. Fix via `/etc/resolvconf.conf`: + +```sh +# Inside freebsd.lan: +cat < rN` tunnels exist (technically redundant since the VM runs on the h | f0 | 192.168.2.130 | fd42:beef:cafe:2::130 | FreeBSD host | | f1 | 192.168.2.131 | fd42:beef:cafe:2::131 | FreeBSD host | | f2 | 192.168.2.132 | fd42:beef:cafe:2::132 | FreeBSD host | +| f3 | 192.168.2.133 | fd42:beef:cafe:2::133 | FreeBSD host (standalone bhyve) | | r0 | 192.168.2.120 | fd42:beef:cafe:2::120 | Rocky VM (k3s node) | | r1 | 192.168.2.121 | fd42:beef:cafe:2::121 | Rocky VM (k3s node) | | r2 | 192.168.2.122 | fd42:beef:cafe:2::122 | Rocky VM (k3s node) | @@ -34,7 +35,7 @@ Even `fN <-> rN` tunnels exist (technically redundant since the VM runs on the h WireGuard hostnames: `.wg0.wan.buetow.org` (e.g. `f0.wg0.wan.buetow.org`) -## FreeBSD Setup (f0, f1, f2) +## FreeBSD Setup (f0, f1, f2, f3) ```sh doas pkg install wireguard-tools @@ -171,6 +172,7 @@ Add to `/etc/hosts` on each host (FreeBSD and Rocky Linux): 192.168.2.130 f0.wg0 f0.wg0.wan.buetow.org 192.168.2.131 f1.wg0 f1.wg0.wan.buetow.org 192.168.2.132 f2.wg0 f2.wg0.wan.buetow.org +192.168.2.133 f3.wg0 f3.wg0.wan.buetow.org 192.168.2.120 r0.wg0 r0.wg0.wan.buetow.org 192.168.2.121 r1.wg0 r1.wg0.wan.buetow.org 192.168.2.122 r2.wg0 r2.wg0.wan.buetow.org @@ -179,6 +181,7 @@ Add to `/etc/hosts` on each host (FreeBSD and Rocky Linux): fd42:beef:cafe:2::130 f0.wg0.wan.buetow.org fd42:beef:cafe:2::131 f1.wg0.wan.buetow.org fd42:beef:cafe:2::132 f2.wg0.wan.buetow.org +fd42:beef:cafe:2::133 f3.wg0.wan.buetow.org fd42:beef:cafe:2::120 r0.wg0.wan.buetow.org fd42:beef:cafe:2::121 r1.wg0.wan.buetow.org fd42:beef:cafe:2::122 r2.wg0.wan.buetow.org @@ -186,6 +189,17 @@ fd42:beef:cafe:2::110 blowfish.wg0.wan.buetow.org fd42:beef:cafe:2::111 fishfinger.wg0.wan.buetow.org ``` +## Troubleshooting: `reload` vs `restart` When Adding New Peers + +`service wireguard reload` (used by the mesh generator) updates peer config but **does NOT add routes** for new peers. After adding a new host to the mesh, the other hosts need a full restart to get the new routes: + +```sh +# On each existing host that had a new peer added via reload: +doas service wireguard restart +``` + +**Symptom**: WireGuard handshake succeeds (both sides show `latest handshake`) but TCP/ICMP traffic doesn't flow — confirmed by `netstat -rn | grep 192.168.2.NNN` returning no results. + ## WireGuard Mesh Generator Manually creating 8+ wg0.conf files is error-prone. A Ruby script automates this: @@ -201,6 +215,19 @@ Config file: `wireguardmeshgenerator.yaml` — defines all hosts, their LAN/WG I The script generates all configs and can push them via SSH. +### FreeBSD 15.0 fix applied to generator + +`wireguardmeshgenerator.rb` line 151 was updated from `/24` to `/32` for FreeBSD hosts: + +```ruby +# Before (broken on FreeBSD 15.0 — start fails with "setting interface address without mask"): +ipv4_with_mask = hosts[myself]['os'] == 'FreeBSD' ? "#{ipv4}/24" : ipv4 +# After (correct): +ipv4_with_mask = hosts[myself]['os'] == 'FreeBSD' ? "#{ipv4}/32" : ipv4 +``` + +Note: `reload` only reconfigures peers/PSKs — it does not change the running interface address. A `restart` is needed to pick up the address change if the interface is already running. + ## Traffic Flows | Flow | Purpose | -- cgit v1.2.3