diff options
Diffstat (limited to 'prompts/skills/f3s-k3s')
| -rw-r--r-- | prompts/skills/f3s-k3s/SKILL.md | 25 | ||||
| -rw-r--r-- | prompts/skills/f3s-k3s/references/ingress.md | 120 | ||||
| -rw-r--r-- | prompts/skills/f3s-k3s/references/install.md | 169 | ||||
| -rw-r--r-- | prompts/skills/f3s-k3s/references/r-node-deploy.md | 144 | ||||
| -rw-r--r-- | prompts/skills/f3s-k3s/references/remote-access.md | 108 | ||||
| -rw-r--r-- | prompts/skills/f3s-k3s/references/troubleshooting.md | 49 |
6 files changed, 615 insertions, 0 deletions
diff --git a/prompts/skills/f3s-k3s/SKILL.md b/prompts/skills/f3s-k3s/SKILL.md new file mode 100644 index 0000000..5ae3c47 --- /dev/null +++ b/prompts/skills/f3s-k3s/SKILL.md @@ -0,0 +1,25 @@ +--- +name: f3s-k3s +description: Reference skill for the f3s k3s Kubernetes cluster, 3-node HA install on r0/r1/r2 Rocky VMs (bootstrap, kubeconfig, PVs, ArgoCD), off-LAN access (jump via OpenBSD frontend to root@r0.wg0 to kubectl), ingress (relayd, cert-manager), etcd recovery, and the reusable Rex r-node rollout. Use when installing, accessing, or troubleshooting the k3s cluster or deploying to r0/r1/r2. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)). +--- + +# f3s k3s Cluster + +3-node HA k3s cluster running on the Rocky Linux VMs r0/r1/r2 (one per +FreeBSD bhyve host f0/f1/f2). All control-plane and etcd traffic flows +over WireGuard. + +## When to Use + +- Installing, bootstrapping, or recovering the k3s cluster (etcd, kubeconfig, PVs, ArgoCD) +- Reaching the cluster off-LAN, or deploying to r0/r1/r2 (incl. the Rex r-node rollout) +- Ingress/cert-manager work on the cluster +- For the underlying Rocky VMs, WireGuard mesh, storage/NFS, and host/IP inventory, see the [`f3s`](../f3s/SKILL.md) hub and [`f3s-storage`](../f3s-storage/SKILL.md). + +## Reference Files + +- [Install](references/install.md) — bootstrap, kubeconfig, etcd/controller-manager metrics, built-in components, NFS PV pattern, ArgoCD, node IP summary, useful commands +- [Remote access (off-LAN)](references/remote-access.md) — reaching the cluster while roaming: **preferred** dedicated `wg0` kubectl context talking directly to `r0.wg0.wan.buetow.org:6443` over WireGuard (switch with `kubectl config use-context wg0`); fallback jump via OpenBSD frontend (`ssh -A rex@fishfinger.buetow.org` → `ssh root@r0.wg0` → `kubectl`), one-shot commands, and SSH port-forward tunnel +- [Ingress](references/ingress.md) — OpenBSD `relayd` (internet) and FreeBSD `relayd` on CARP VIP (LAN), cert-manager wildcard, ingress pattern +- [Troubleshooting](references/troubleshooting.md) — etcd Raft log corruption recovery; cluster-wide NFS outage pointer +- [r-node Deploy (Rex)](references/r-node-deploy.md) — reusable Rex rollout to r0/r1/r2 (`f3s/r-nodes/Rexfile`, task `nfs_mount_monitor`): root SSH, `parallelism 3`, idempotent `file`/`on_change` reload diff --git a/prompts/skills/f3s-k3s/references/ingress.md b/prompts/skills/f3s-k3s/references/ingress.md new file mode 100644 index 0000000..6beb4ea --- /dev/null +++ b/prompts/skills/f3s-k3s/references/ingress.md @@ -0,0 +1,120 @@ +# k3s Ingress (Internet + LAN) + +Two ingress paths into the k3s cluster: +- **Internet → OpenBSD relayd** (TLS termination on `blowfish`/`fishfinger`) → WireGuard → Traefik +- **LAN → FreeBSD relayd on CARP VIP** → k3s Traefik + +## External Connectivity: OpenBSD relayd + +Default traffic flow for public k3s-backed services: `Internet → OpenBSD relayd (TLS, Let's Encrypt) → WireGuard → k3s Traefik :80 → Service` + +### relayd.conf on blowfish/fishfinger + +``` +table <f3s> { + 192.168.2.120 + 192.168.2.121 + 192.168.2.122 +} + +http protocol "https" { + tls keypair f3s.foo.zone + # ... all f3s service TLS keypairs ... + # Non-f3s hosts explicitly forwarded to localhost: + match request header "Host" value "foo.zone" forward to <localhost> + # f3s hosts have NO match rules — use relay-level failover +} + +relay "https4" { + listen on <PUBLIC_IP> port 443 tls + protocol "https" + forward to <f3s> port 80 check tcp # primary + forward to <localhost> port 8080 # fallback when f3s down +} +``` + +`f3s.buetow.org` is now a special case: it no longer points at the k3s/apache backend and is forwarded by OpenBSD `relayd` to `pi0` (`192.168.2.203`) and `pi1` (`192.168.2.204`) via a dedicated `<f3s_static>` backend table. + +When all k3s-backed f3s nodes are down, relayd falls back to `localhost:8080` (OpenBSD httpd serving a "Server turned off" page) for the hosts that still use the shared `<f3s>` backend. + +## LAN Ingress: FreeBSD relayd on CARP VIP + +For LAN access without going through internet gateways: +`LAN → CARP VIP (192.168.1.138) → FreeBSD relayd → k3s Traefik :443 → Service` + +### FreeBSD relayd config (`/usr/local/etc/relayd.conf`) + +``` +table <k3s_nodes> { 192.168.1.120 192.168.1.121 192.168.1.122 } + +relay "lan_http" { + listen on 192.168.1.138 port 80 + forward to <k3s_nodes> port 80 check tcp +} + +relay "lan_https" { + listen on 192.168.1.138 port 443 + forward to <k3s_nodes> port 443 check tcp +} +``` + +Minimal `/etc/pf.conf` (PF required for relayd): + +``` +set skip on lo0 +pass in quick +pass out quick +``` + +```sh +doas pkg install -y relayd +doas sysrc pf_enable=YES pflog_enable=YES relayd_enable=YES +doas service pf start && doas service pflog start && doas service relayd start +``` + +Run on both f0 and f1. Only CARP MASTER responds to VIP traffic. + +### cert-manager for LAN TLS + +LAN services use `*.f3s.lan.foo.zone` with a self-signed CA managed by cert-manager: + +```sh +cd conf/f3s/cert-manager && just install +# Creates: selfsigned ClusterIssuer, CA cert, wildcard cert (f3s-lan-tls) +``` + +Copy secret to service namespace: +```sh +kubectl get secret f3s-lan-tls -n cert-manager -o yaml | \ + sed 's/namespace: cert-manager/namespace: services/' | \ + kubectl apply -f - +``` + +### LAN ingress pattern + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: ingress-lan + namespace: services + annotations: + spec.ingressClassName: traefik + traefik.ingress.kubernetes.io/router.entrypoints: web,websecure +spec: + tls: + - hosts: + - myservice.f3s.lan.foo.zone + secretName: f3s-lan-tls + rules: + - host: myservice.f3s.lan.foo.zone + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: myservice + port: + number: 8080 +``` diff --git a/prompts/skills/f3s-k3s/references/install.md b/prompts/skills/f3s-k3s/references/install.md new file mode 100644 index 0000000..324895a --- /dev/null +++ b/prompts/skills/f3s-k3s/references/install.md @@ -0,0 +1,169 @@ +# k3s Install + +3-node HA k3s cluster running on Rocky Linux VMs (r0, r1, r2). All nodes act as both control-plane and etcd members (no separate worker nodes). + +- k3s version: **v1.32.6+k3s1** (as of Part 7) +- etcd mode: **embedded HA** (`--cluster-init`) +- All control-plane traffic goes over **WireGuard** (192.168.2.x IPs) + +## Prerequisites + +- All Rocky Linux VMs (r0, r1, r2) updated and running +- WireGuard mesh fully configured (see [wireguard.md](../../f3s/references/wireguard.md)) +- NVMe disk emulation in place (see [rocky-linux-vms.md](../../f3s/references/rocky-linux-vms.md)) — critical for etcd performance + +## Installation + +### Generate shared token + +```sh +# On Fedora laptop +pwgen -n 32 +# Copy output to all r nodes: +echo -n SECRET_TOKEN > ~/.k3s_token # on r0, r1, r2 +``` + +### Bootstrap first node (r0) + +```sh +[root@r0 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \ + sh -s - server --cluster-init \ + --node-ip=192.168.2.120 \ + --advertise-address=192.168.2.120 \ + --tls-san=r0.wg0.wan.buetow.org +``` + +`--node-ip` and `--advertise-address` bind etcd to the WireGuard interface so all control-plane traffic is encrypted. + +### Join remaining nodes (r1, r2) + +```sh +[root@r1 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \ + sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \ + --node-ip=192.168.2.121 \ + --advertise-address=192.168.2.121 \ + --tls-san=r1.wg0.wan.buetow.org + +[root@r2 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \ + sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \ + --node-ip=192.168.2.122 \ + --advertise-address=192.168.2.122 \ + --tls-san=r2.wg0.wan.buetow.org +``` + +### Verify cluster + +```sh +kubectl get nodes +# Expected: r0, r1, r2 all Ready with role control-plane,etcd,master +``` + +## kubeconfig + +```sh +# Copy from any r node to laptop +scp root@r0.lan.buetow.org:/etc/rancher/k3s/k3s.yaml ~/.kube/config +# Edit: replace server address with r0.lan.buetow.org +# (repeat with r1 or r2 if r0 is down) +``` + +## k3s config.yaml — expose etcd and controller-manager metrics + +For Prometheus to scrape etcd and controller-manager metrics, add to `/etc/rancher/k3s/config.yaml` on each r node: + +```sh +cat >> /etc/rancher/k3s/config.yaml << 'EOF' +kube-controller-manager-arg: + - bind-address=0.0.0.0 +etcd-expose-metrics: true +EOF +systemctl restart k3s +``` + +Verify: `curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader` + +## Built-in Components + +| Component | Purpose | +|-----------|---------| +| CoreDNS | DNS for pods | +| Traefik | Ingress controller | +| local-path-provisioner | Local PVC storage | +| metrics-server | Resource metrics | +| svclb-traefik | ServiceLB for Traefik | + +### Scale Traefik to 2 replicas (faster failover) + +```sh +kubectl -n kube-system scale deployment traefik --replicas=2 +``` + +## NFS Persistent Volumes + +Persistent volumes use `hostPath` pointing to NFS-mounted paths: + +``` +/data/nfs/k3svolumes/<app>/ +``` + +NFS is mounted on all r nodes at `/data/nfs/k3svolumes` via stunnel → CARP VIP → +freeBSD NFS — see [storage/nfs.md](../../f3s-storage/references/nfs.md). The +[`nfs-mount-monitor`](../../f3s-storage/references/nfs-mount-monitor.md) watchdog auto-repairs +hung mounts and force-deletes stuck pods. + +Example PV: + +```yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: example-pv +spec: + capacity: + storage: 1Gi + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + hostPath: + path: /data/nfs/k3svolumes/example-volume + type: Directory +``` + +Create the directory on the NFS share before deploying: `mkdir /data/nfs/k3svolumes/<app>/` + +## Deployment: GitOps with ArgoCD + +Config repository: `https://codeberg.org/snonux/conf` (directory: `f3s/`) + +ArgoCD app structure: +``` +argocd-apps/ + monitoring/ # Prometheus, Grafana, Loki, etc. + services/ # User-facing services + infra/ # Infrastructure components + test/ # Test deployments +``` + +**To view pre-ArgoCD state** (how things were in Part 7): +```sh +git clone https://codeberg.org/snonux/conf.git +cd conf && git checkout 15a86f3 # last commit before ArgoCD migration +cd f3s/ +``` + +## Node IP Summary + +| Node | LAN IP | WireGuard IP | k3s API | +|------|--------|-------------|---------| +| r0 | 192.168.1.120 | 192.168.2.120 | r0.wg0.wan.buetow.org:6443 | +| r1 | 192.168.1.121 | 192.168.2.121 | r1.wg0.wan.buetow.org:6443 | +| r2 | 192.168.1.122 | 192.168.2.122 | r2.wg0.wan.buetow.org:6443 | + +## Useful Commands + +```sh +kubectl get nodes # cluster status +kubectl get pods --all-namespaces # all running pods +kubectl get namespaces +kubectl config set-context --current --namespace=<ns> +``` diff --git a/prompts/skills/f3s-k3s/references/r-node-deploy.md b/prompts/skills/f3s-k3s/references/r-node-deploy.md new file mode 100644 index 0000000..294958c --- /dev/null +++ b/prompts/skills/f3s-k3s/references/r-node-deploy.md @@ -0,0 +1,144 @@ +# r-node Deploy Mechanism (Rex) + +The reusable way to roll out files and systemd units to the three k3s Rocky +Linux VMs — **r0/r1/r2** — is a [Rex](https://www.rexify.org/) task. The +canonical, fully-worked example is the **`nfs_mount_monitor`** task. Treat it +as the template for *any* r-node rollout; you do not need to re-derive the +flow each time. + +## Hosts: r0/r1/r2 vs f0/f1/f2 + +The r-nodes are Rocky Linux 9 bhyve guests, one per FreeBSD host: + +| VM | LAN IP | Runs on f-host | +|----|--------|----------------| +| r0 | 192.168.1.120 | f0 (192.168.1.130) | +| r1 | 192.168.1.121 | f1 (192.168.1.131) | +| r2 | 192.168.1.122 | f2 (192.168.1.132) | + +f3 is **not** part of this group — it is standalone bhyve and hosts the plain +`rocky` VM, not a k3s node. r-node deploys never touch f3. + +## Where it lives + +In the conf repo (`https://codeberg.org/snonux/conf`, dir `f3s/`): + +``` +f3s/r-nodes/Rexfile # deploy tasks for r0/r1/r2 +f3s/r-nodes/nfs-mount-monitor/ # source files the task pushes + check-nfs-mount.sh # → /usr/local/bin/ + nfs-mount-monitor.default # → /etc/default/ (tunables) + nfs-mount-monitor.service # → /etc/systemd/system/ + nfs-mount-monitor.timer # → /etc/systemd/system/ +``` + +## The deploy command + +Run from the **conf repo root**: + +```sh +rex -f f3s/r-nodes/Rexfile nfs_mount_monitor +``` + +This pushes to all three r-nodes at once. To target a single node, use Rex's +host filter, e.g. `rex -f f3s/r-nodes/Rexfile -H 192.168.1.120 nfs_mount_monitor`. + +## How the pattern works (the reusable parts) + +The Rexfile establishes conventions every r-node task inherits: + +- **`group r_nodes => qw(192.168.1.120 192.168.1.121 192.168.1.122)`** — the + three k3s VMs by LAN IP. Each `task` declares `group => 'r_nodes'`. +- **`user 'root'; sudo FALSE;`** — tasks connect as **root** over SSH. The + `paul` user has no sudo on the r-nodes, and writing to `/usr/local/bin` and + managing systemd both need root. Root SSH is pre-authorized via + `authorized_keys`. +- **`parallelism 3;`** — all three nodes deploy concurrently. Safe because the + tasks are idempotent and independent per node. +- **`$RNODES_DIR`** is resolved with `realpath($::rexfile)` so source-file + paths stay valid regardless of CWD or Rex worker forking. + +### Idempotent rollout flow + +Inside the task, each file is deployed with Rex's `file` resource: + +```perl +file '/usr/local/bin/check-nfs-mount.sh', + source => catfile($monitor_dir, 'check-nfs-mount.sh'), + owner => 'root', group => 'root', mode => '755', + on_change => sub { $changed = 1 }; +``` + +Rex only writes a file when its **content actually differs** from what's on +the node — so re-running the task is a no-op when nothing changed. The +`on_change` handlers set a single `$changed` flag; only if something changed +does the task run: + +```perl +run 'systemctl daemon-reload'; +run 'systemctl restart nfs-mount-monitor.timer'; +``` + +Finally — regardless of whether files changed — the task **converges** the +service state so the timer is always enabled and running: + +```perl +service 'nfs-mount-monitor.timer', ensure => 'started'; +run 'systemctl enable nfs-mount-monitor.timer'; +``` + +This split (reload/restart only on change, enable/start always) is the +pattern to copy: cheap, repeatable, and self-healing if a node drifted. + +The task also pre-creates the state/output directories it needs +(`/var/lib/nfs-mount-monitor` mode 700, +`/var/lib/node_exporter/textfile_collector` mode 755) so the deployed script +has somewhere to write. + +## Verify after deploy + +```sh +# On each r-node (over SSH as root) — confirm the timer is active & enabled: +systemctl status nfs-mount-monitor.timer + +# Watch the service fire and log: +journalctl -u nfs-mount-monitor -f +``` + +What success looks like: + +- `systemctl status …timer` reports **active (waiting)** and **enabled**, with + a `Trigger:` line a few seconds out (the timer fires every 10 s). +- `journalctl -u nfs-mount-monitor` shows a fresh oneshot run roughly every + 10 s with no errors. + +One-shot remote check from a roaming laptop (see +[remote-access.md](remote-access.md)): + +```sh +ssh -A -J rex@fishfinger.buetow.org root@r0.wg0 \ + "systemctl is-active nfs-mount-monitor.timer && systemctl is-enabled nfs-mount-monitor.timer" +``` + +## Gotchas + +- **Run from the conf repo root**, not from `f3s/r-nodes/` — the `-f` path in + every example is repo-relative (`f3s/r-nodes/Rexfile`). +- **Root, no sudo.** If a task errors on permission, it's almost always that + the connection fell back to `paul` (who has no sudo here). The Rexfile sets + `user 'root'` for exactly this reason. +- **`parallelism 3` only works if tasks are idempotent and node-independent.** + Keep new tasks that way, or lower the parallelism. +- **Adding a new file** to a deploy: give it `on_change => sub { $changed = 1 }` + so a reload/restart fires when it changes, and place it before the + `if ($changed)` block. + +## Related + +- The **behavior** of the deployed monitor (probes, fail-counter, cordon & + auto-reboot escalation, alerting) is documented separately in + [storage/nfs-mount-monitor.md](../../f3s-storage/references/nfs-mount-monitor.md). This file + covers only the *deploy mechanism*; that one covers *what gets deployed*. +- There are sibling Rexfiles in the conf repo (`f3s/garage/Rexfile`, + `frontends/Rexfile`) that follow the same group/user/idempotent-`file` + shape against different host groups. diff --git a/prompts/skills/f3s-k3s/references/remote-access.md b/prompts/skills/f3s-k3s/references/remote-access.md new file mode 100644 index 0000000..9837870 --- /dev/null +++ b/prompts/skills/f3s-k3s/references/remote-access.md @@ -0,0 +1,108 @@ +# Accessing the k3s cluster from outside the LAN (roaming) + +The kubeconfig on the laptop points at the **LAN** API endpoint +(`https://r0.lan.buetow.org:6443` → `192.168.1.120`). When **earth** (or any +client) is **not on the f3s LAN**, that address is unreachable and `kubectl` +just times out: + +``` +Unable to connect to the server: dial tcp 192.168.1.120:6443: i/o timeout +``` + +## Preferred method: a dedicated `wg0` kubectl context (direct over WireGuard) + +From **earth** the WireGuard mesh IPs **are** directly reachable +(`192.168.2.120` = `r0.wg0.wan.buetow.org`), so `kubectl` can talk straight to +the API server over `wg0` — no jump host needed. The k3s API cert already has +the `r0.wg0.wan.buetow.org` / `r1.wg0…` / `r2.wg0…` SANs (see install.md +`--tls-san`), so TLS verifies cleanly with no overrides. + +Add a second cluster + context to `~/.kube/config` alongside the LAN `default` +one, reusing the same `certificate-authority-data` and `default` user: + +```yaml +clusters: +- cluster: + certificate-authority-data: <same as default> + server: https://r0.wg0.wan.buetow.org:6443 # 192.168.2.120 + name: wg0 +contexts: +- context: + cluster: wg0 + namespace: services + user: default + name: wg0 +``` + +Then switch by location: + +```sh +kubectl config use-context default # on the f3s LAN (home) +kubectl config use-context wg0 # on the road (WireGuard) +kubectl --context=wg0 get nodes # one-off without switching +``` + +Notes: + +- `default` → `https://r0.lan.buetow.org:6443` (LAN, `192.168.1.120`); `wg0` → + `https://r0.wg0.wan.buetow.org:6443` (mesh, `192.168.2.120`). +- For failover, point a `wg0` variant at `r1.wg0` / `r2.wg0` (all three are SANs + on the cert) if r0 is down. +- This requires the laptop's WireGuard to actually route to the r-VM mesh IPs. + If a particular client's `wg` does **not** peer to the r-VMs, fall back to the + OpenBSD-frontend jump method below. + +## Fallback method: jump through an OpenBSD frontend + +The OpenBSD internet gateways **fishfinger** and **blowfish** are reachable +from the public internet *and* sit on the WireGuard mesh, so they can reach the +r-VMs over `wg0`. Use one as a jump host, then run `kubectl` as `root` on the +r-VM itself. + +Interactive: + +```sh +ssh -A rex@fishfinger.buetow.org # -A forwards your agent for the next hop +ssh root@r0.wg0 # from fishfinger, over the WireGuard mesh +kubectl get nodes # root on r0 has /etc/rancher/k3s/k3s.yaml +``` + +One-shot (non-interactive) from the laptop — handy for scripts/automation: + +```sh +ssh -A rex@fishfinger.buetow.org "ssh root@r0.wg0 'kubectl get nodes'" +``` + +Notes: + +- `ssh -A` (agent forwarding) is required so the `rex@fishfinger` → `root@r0.wg0` + hop can authenticate. Make sure the right key is loaded (`ssh-add -l`). +- Use **`r0.wg0`** (or `r1.wg0` / `r2.wg0` if r0 is down) — that is the r-VM's + WireGuard hostname as seen *from the frontend*, not `r0.lan…`. +- `blowfish.buetow.org` works the same way as `fishfinger.buetow.org` if one + frontend is unavailable. +- On a fresh laptop, accept host keys with + `-o StrictHostKeyChecking=accept-new` on both hops. +- An OpenSSH **post-quantum key exchange warning** from the `root@r0.wg0` hop is + harmless and can be ignored. + +## Example: issue a goprecords client key while roaming + +```sh +ssh -A rex@fishfinger.buetow.org \ + "ssh root@r0.wg0 'kubectl exec -n services deployment/goprecords -- \ + goprecords --create-client-key mega-m3-pro -stats-dir=/data/stats'" +``` + +## Alternative: serve the API over an SSH tunnel + +If you need a real local `kubectl` (not just one-off commands), forward the API +port through the frontend and point a kubeconfig at `127.0.0.1`: + +```sh +ssh -A -L 6443:r0.wg0:6443 rex@fishfinger.buetow.org +# then, in another shell, with a kubeconfig whose server is +# https://127.0.0.1:6443 (and tls-server-name r0.wg0.wan.buetow.org, since the +# API cert has that SAN — see install.md --tls-san): +kubectl --tls-server-name r0.wg0.wan.buetow.org get nodes +``` diff --git a/prompts/skills/f3s-k3s/references/troubleshooting.md b/prompts/skills/f3s-k3s/references/troubleshooting.md new file mode 100644 index 0000000..188f464 --- /dev/null +++ b/prompts/skills/f3s-k3s/references/troubleshooting.md @@ -0,0 +1,49 @@ +# k3s Troubleshooting + +## Etcd Raft Log Corruption Recovery + +**Symptom**: k3s crashes on startup with panic: +``` +tocommit(XXXXXXX) is out of range [lastIndex(YYYYYYY)] +``` +Caused by `kill -9` on the bhyve process mid-write (corrupts etcd WAL). k3s enters a crash loop and stops after ~2 minutes. + +**Recovery procedure** (example: r1 is corrupt): + +```sh +# 1. Stop k3s on the affected node +ssh root@r1.lan.buetow.org 'systemctl stop k3s' + +# 2. Download etcdctl on a healthy node (not bundled with k3s) +ssh root@r0.lan.buetow.org +curl -sL https://github.com/etcd-io/etcd/releases/download/v3.5.17/etcd-v3.5.17-linux-amd64.tar.gz \ + | tar -xz -C /tmp etcd-v3.5.17-linux-amd64/etcdctl +mv /tmp/etcd-v3.5.17-linux-amd64/etcdctl /tmp/etcdctl + +# 3. Find and remove the corrupt member from the cluster +ETCDCTL_API=3 /tmp/etcdctl \ + --endpoints=https://127.0.0.1:2379 \ + --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \ + --cert=/var/lib/rancher/k3s/server/tls/etcd/client.crt \ + --key=/var/lib/rancher/k3s/server/tls/etcd/client.key \ + member list +# Find the member ID for r1, then: +ETCDCTL_API=3 /tmp/etcdctl ... member remove <MEMBER_ID> + +# 4. Delete the corrupted etcd data on the affected node +ssh root@r1.lan.buetow.org 'rm -rf /var/lib/rancher/k3s/server/db/etcd' + +# 5. Restart k3s — it rejoins as a fresh member +ssh root@r1.lan.buetow.org 'systemctl start k3s' + +# 6. Verify +kubectl get nodes # r1 should return to Ready +``` + +> **Prevention**: Always use `doas vm stop rocky` and wait for clean shutdown before stopping the bhyve host. Only use `kill -9` on the bhyve process as a last resort — it can corrupt the etcd WAL. + +## Cluster-wide NFS Outages + +If NFS goes down cluster-wide, the root cause is usually on the FreeBSD NFS +server side (f0/f1). Check CARP state, stunnel, nfsd, and +`vfs.nfsd.nfs_privport` — see [storage/troubleshooting.md](../../f3s-storage/references/troubleshooting.md). |