f3s: split storage/k3s-setup/observability references, correct thermal attribution

storage.md (942), k3s-setup.md (342), observability.md (273) split into
per-topic sub-files under references/<topic>/ with short index files at
the original paths.

Drops the SSD TRIM Configuration section added in 8d94f79 and reframes
the Thermal Troubleshooting cascade: the 2026-05-16 f0 incident was
thermal alone, not the multi-cause cascade (autotrim, zrepl interval,
encryption) the previous commit implied. Mitigations applied during
diagnosis are not what fixed it.

Other yesterday additions (zrepl DL-state recovery, CARP-when-ZFS-
suspended, ZFS SUSPENDED runbook, nfs-mount-monitor improvements) are
kept and routed to their respective sub-files.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

16 files changed, 1521 insertions, 1525 deletions

diff --git a/prompts/skills/f3s/SKILL.md b/prompts/skills/f3s/SKILL.md
index a3e7c06..858518a 100644
--- a/prompts/skills/f3s/SKILL.md
+++ b/prompts/skills/f3s/SKILL.md
@@ -24,9 +24,9 @@ Detailed reference documentation is in the `references/` subfolder:
 - [f3 Rocky VM](references/f3-rocky-vm.md) — Plain Rocky Linux 9 VM on f3 (`rocky`, `192.168.1.123`), autostart policy, root SSH
 - [Bootstrap Rocky bhyve VM](references/bootstrap-rocky-bhyve.md) — Runbook for creating a new plain Rocky Linux bhyve guest with unattended kickstart
 - [WireGuard Mesh](references/wireguard.md) — Mesh topology, IP assignments, peer configs
-- [Storage](references/storage.md) — ZFS (zdata), CARP, NFS over stunnel, zrepl replication
-- [k3s Setup](references/k3s-setup.md) — HA k3s cluster, etcd, node IPs, kubeconfig, ArgoCD
-- [Observability](references/observability.md) — Prometheus, Grafana, Loki, Alloy, Tempo
+- [Storage](references/storage.md) — index into `references/storage/`: ZFS (zdata), zrepl, CARP, NFS over stunnel, nfs-mount-monitor, troubleshooting (incl. thermal), backups & local-path
+- [k3s Setup](references/k3s-setup.md) — index into `references/k3s-setup/`: install (bootstrap, kubeconfig, PVs, ArgoCD), ingress (OpenBSD/FreeBSD relayd, cert-manager), troubleshooting (etcd recovery)
+- [Observability](references/observability.md) — index into `references/observability/`: stack (Prometheus/Alloy/Loki/Tempo + alerting), FreeBSD monitoring (node_exporter + recording rules)
 - [Immich](references/immich.md) — Photo server deployment, job queue stats, troubleshooting
 - [Garage](references/garage.md) — Garage cluster, edge domain routing, S3 bucket/key workflow, troubleshooting
 - [DTail / dserver](references/dtail.md) — dserver: Pis **arm64** vs r0–r2 **amd64**, r-VM **root** + `root.authorized_keys` cache, firewalld **2222**, systemd timers
diff --git a/prompts/skills/f3s/references/k3s-setup.md b/prompts/skills/f3s/references/k3s-setup.md
index 7d2239b..0fc7605 100644
--- a/prompts/skills/f3s/references/k3s-setup.md
+++ b/prompts/skills/f3s/references/k3s-setup.md
@@ -1,342 +1,11 @@
 # k3s Setup
 
-## Overview
+3-node HA k3s cluster running on the Rocky Linux VMs r0/r1/r2 (one per
+FreeBSD bhyve host f0/f1/f2). All control-plane and etcd traffic flows
+over WireGuard.
 
-3-node HA k3s cluster running on Rocky Linux VMs (r0, r1, r2). All nodes act as both control-plane and etcd members (no separate worker nodes).
+## Sub-references
 
-- k3s version: **v1.32.6+k3s1** (as of Part 7)
-- etcd mode: **embedded HA** (`--cluster-init`)
-- All control-plane traffic goes over **WireGuard** (192.168.2.x IPs)
-
-## Prerequisites
-
-- All Rocky Linux VMs (r0, r1, r2) updated and running
-- WireGuard mesh fully configured (see wireguard.md)
-- NVMe disk emulation in place (see rocky-linux-vms.md) — critical for etcd performance
-
-## Installation
-
-### Generate shared token
-
-```sh
-# On Fedora laptop
-pwgen -n 32
-# Copy output to all r nodes:
-echo -n SECRET_TOKEN > ~/.k3s_token  # on r0, r1, r2
-```
-
-### Bootstrap first node (r0)
-
-```sh
-[root@r0 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \
-    sh -s - server --cluster-init \
-    --node-ip=192.168.2.120 \
-    --advertise-address=192.168.2.120 \
-    --tls-san=r0.wg0.wan.buetow.org
-```
-
-`--node-ip` and `--advertise-address` bind etcd to the WireGuard interface so all control-plane traffic is encrypted.
-
-### Join remaining nodes (r1, r2)
-
-```sh
-[root@r1 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \
-    sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \
-    --node-ip=192.168.2.121 \
-    --advertise-address=192.168.2.121 \
-    --tls-san=r1.wg0.wan.buetow.org
-
-[root@r2 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \
-    sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \
-    --node-ip=192.168.2.122 \
-    --advertise-address=192.168.2.122 \
-    --tls-san=r2.wg0.wan.buetow.org
-```
-
-### Verify cluster
-
-```sh
-kubectl get nodes
-# Expected: r0, r1, r2 all Ready with role control-plane,etcd,master
-```
-
-## kubeconfig
-
-```sh
-# Copy from any r node to laptop
-scp root@r0.lan.buetow.org:/etc/rancher/k3s/k3s.yaml ~/.kube/config
-# Edit: replace server address with r0.lan.buetow.org
-# (repeat with r1 or r2 if r0 is down)
-```
-
-## k3s config.yaml — expose etcd and controller-manager metrics
-
-For Prometheus to scrape etcd and controller-manager metrics, add to `/etc/rancher/k3s/config.yaml` on each r node:
-
-```sh
-cat >> /etc/rancher/k3s/config.yaml << 'EOF'
-kube-controller-manager-arg:
-  - bind-address=0.0.0.0
-etcd-expose-metrics: true
-EOF
-systemctl restart k3s
-```
-
-Verify: `curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader`
-
-## Built-in Components
-
-| Component | Purpose |
-|-----------|---------|
-| CoreDNS | DNS for pods |
-| Traefik | Ingress controller |
-| local-path-provisioner | Local PVC storage |
-| metrics-server | Resource metrics |
-| svclb-traefik | ServiceLB for Traefik |
-
-### Scale Traefik to 2 replicas (faster failover)
-
-```sh
-kubectl -n kube-system scale deployment traefik --replicas=2
-```
-
-## NFS Persistent Volumes
-
-Persistent volumes use `hostPath` pointing to NFS-mounted paths:
-
-```
-/data/nfs/k3svolumes/<app>/
-```
-
-NFS is mounted on all r nodes at `/data/nfs/k3svolumes` via stunnel → CARP VIP → freeBSD NFS (see storage.md).
-
-Example PV:
-
-```yaml
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: example-pv
-spec:
-  capacity:
-    storage: 1Gi
-  accessModes:
-    - ReadWriteOnce
-  persistentVolumeReclaimPolicy: Retain
-  hostPath:
-    path: /data/nfs/k3svolumes/example-volume
-    type: Directory
-```
-
-Create the directory on the NFS share before deploying: `mkdir /data/nfs/k3svolumes/<app>/`
-
-### NFS Mount Health Monitor (on r0, r1, r2)
-
-Each Rocky Linux node runs `/usr/local/bin/check-nfs-mount.sh` via cron (every minute) to detect and fix stale/missing NFS mounts. After a successful remount, the script also **force-deletes stuck pods** on the local node (status Unknown, Pending, or ContainerCreating) so Kubernetes reschedules them with the healthy mount.
-
-```sh
-# Cron entry (on all r-nodes, as root)
-* * * * * /usr/local/bin/check-nfs-mount.sh >> /var/log/nfs-mount-check.log 2>&1
-```
-
-The script:
-1. Checks if `/data/nfs/k3svolumes` is a mountpoint and responsive (2s timeout)
-2. If stale/missing: force-unmounts + remounts NFS
-3. After successful remount: uses `kubectl` to find and delete stuck pods on this node
-4. Uses a lock file (`/var/run/nfs-mount-check.lock`) to prevent concurrent runs
-
-**Important**: If NFS goes down cluster-wide, the root cause is usually on the FreeBSD NFS server side (f0/f1). Check CARP state, stunnel, nfsd, and `vfs.nfsd.nfs_privport` (see storage.md).
-
-## Deployment: GitOps with ArgoCD
-
-Config repository: `https://codeberg.org/snonux/conf` (directory: `f3s/`)
-
-ArgoCD app structure:
-```
-argocd-apps/
-  monitoring/    # Prometheus, Grafana, Loki, etc.
-  services/      # User-facing services
-  infra/         # Infrastructure components
-  test/          # Test deployments
-```
-
-**To view pre-ArgoCD state** (how things were in Part 7):
-```sh
-git clone https://codeberg.org/snonux/conf.git
-cd conf && git checkout 15a86f3  # last commit before ArgoCD migration
-cd f3s/
-```
-
-## Node IP Summary
-
-| Node | LAN IP | WireGuard IP | k3s API |
-|------|--------|-------------|---------|
-| r0 | 192.168.1.120 | 192.168.2.120 | r0.wg0.wan.buetow.org:6443 |
-| r1 | 192.168.1.121 | 192.168.2.121 | r1.wg0.wan.buetow.org:6443 |
-| r2 | 192.168.1.122 | 192.168.2.122 | r2.wg0.wan.buetow.org:6443 |
-
-## External Connectivity: OpenBSD relayd
-
-Default traffic flow for public k3s-backed services: `Internet → OpenBSD relayd (TLS, Let's Encrypt) → WireGuard → k3s Traefik :80 → Service`
-
-### relayd.conf on blowfish/fishfinger
-
-```
-table <f3s> {
-  192.168.2.120
-  192.168.2.121
-  192.168.2.122
-}
-
-http protocol "https" {
-    tls keypair f3s.foo.zone
-    # ... all f3s service TLS keypairs ...
-    # Non-f3s hosts explicitly forwarded to localhost:
-    match request header "Host" value "foo.zone" forward to <localhost>
-    # f3s hosts have NO match rules — use relay-level failover
-}
-
-relay "https4" {
-    listen on <PUBLIC_IP> port 443 tls
-    protocol "https"
-    forward to <f3s> port 80 check tcp      # primary
-    forward to <localhost> port 8080         # fallback when f3s down
-}
-```
-
-`f3s.buetow.org` is now a special case: it no longer points at the k3s/apache backend and is forwarded by OpenBSD `relayd` to `pi0` (`192.168.2.203`) and `pi1` (`192.168.2.204`) via a dedicated `<f3s_static>` backend table.
-
-When all k3s-backed f3s nodes are down, relayd falls back to `localhost:8080` (OpenBSD httpd serving a "Server turned off" page) for the hosts that still use the shared `<f3s>` backend.
-
-## LAN Ingress: FreeBSD relayd on CARP VIP
-
-For LAN access without going through internet gateways:
-`LAN → CARP VIP (192.168.1.138) → FreeBSD relayd → k3s Traefik :443 → Service`
-
-### FreeBSD relayd config (`/usr/local/etc/relayd.conf`)
-
-```
-table <k3s_nodes> { 192.168.1.120 192.168.1.121 192.168.1.122 }
-
-relay "lan_http" {
-    listen on 192.168.1.138 port 80
-    forward to <k3s_nodes> port 80 check tcp
-}
-
-relay "lan_https" {
-    listen on 192.168.1.138 port 443
-    forward to <k3s_nodes> port 443 check tcp
-}
-```
-
-Minimal `/etc/pf.conf` (PF required for relayd):
-
-```
-set skip on lo0
-pass in quick
-pass out quick
-```
-
-```sh
-doas pkg install -y relayd
-doas sysrc pf_enable=YES pflog_enable=YES relayd_enable=YES
-doas service pf start && doas service pflog start && doas service relayd start
-```
-
-Run on both f0 and f1. Only CARP MASTER responds to VIP traffic.
-
-### cert-manager for LAN TLS
-
-LAN services use `*.f3s.lan.foo.zone` with a self-signed CA managed by cert-manager:
-
-```sh
-cd conf/f3s/cert-manager && just install
-# Creates: selfsigned ClusterIssuer, CA cert, wildcard cert (f3s-lan-tls)
-```
-
-Copy secret to service namespace:
-```sh
-kubectl get secret f3s-lan-tls -n cert-manager -o yaml | \
-    sed 's/namespace: cert-manager/namespace: services/' | \
-    kubectl apply -f -
-```
-
-### LAN ingress pattern
-
-```yaml
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
-  name: ingress-lan
-  namespace: services
-  annotations:
-    spec.ingressClassName: traefik
-    traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
-spec:
-  tls:
-    - hosts:
-        - myservice.f3s.lan.foo.zone
-      secretName: f3s-lan-tls
-  rules:
-    - host: myservice.f3s.lan.foo.zone
-      http:
-        paths:
-          - path: /
-            pathType: Prefix
-            backend:
-              service:
-                name: myservice
-                port:
-                  number: 8080
-```
-
-## Etcd Raft Log Corruption Recovery
-
-**Symptom**: k3s crashes on startup with panic:
-```
-tocommit(XXXXXXX) is out of range [lastIndex(YYYYYYY)]
-```
-Caused by `kill -9` on the bhyve process mid-write (corrupts etcd WAL). k3s enters a crash loop and stops after ~2 minutes.
-
-**Recovery procedure** (example: r1 is corrupt):
-
-```sh
-# 1. Stop k3s on the affected node
-ssh root@r1.lan.buetow.org 'systemctl stop k3s'
-
-# 2. Download etcdctl on a healthy node (not bundled with k3s)
-ssh root@r0.lan.buetow.org
-curl -sL https://github.com/etcd-io/etcd/releases/download/v3.5.17/etcd-v3.5.17-linux-amd64.tar.gz \
-  | tar -xz -C /tmp etcd-v3.5.17-linux-amd64/etcdctl
-mv /tmp/etcd-v3.5.17-linux-amd64/etcdctl /tmp/etcdctl
-
-# 3. Find and remove the corrupt member from the cluster
-ETCDCTL_API=3 /tmp/etcdctl \
-  --endpoints=https://127.0.0.1:2379 \
-  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
-  --cert=/var/lib/rancher/k3s/server/tls/etcd/client.crt \
-  --key=/var/lib/rancher/k3s/server/tls/etcd/client.key \
-  member list
-# Find the member ID for r1, then:
-ETCDCTL_API=3 /tmp/etcdctl ... member remove <MEMBER_ID>
-
-# 4. Delete the corrupted etcd data on the affected node
-ssh root@r1.lan.buetow.org 'rm -rf /var/lib/rancher/k3s/server/db/etcd'
-
-# 5. Restart k3s — it rejoins as a fresh member
-ssh root@r1.lan.buetow.org 'systemctl start k3s'
-
-# 6. Verify
-kubectl get nodes  # r1 should return to Ready
-```
-
-> **Prevention**: Always use `doas vm stop rocky` and wait for clean shutdown before stopping the bhyve host. Only use `kill -9` on the bhyve process as a last resort — it can corrupt the etcd WAL.
-
-## Useful Commands
-
-```sh
-kubectl get nodes                    # cluster status
-kubectl get pods --all-namespaces    # all running pods
-kubectl get namespaces
-kubectl config set-context --current --namespace=<ns>
-```
+- [Install](k3s-setup/install.md) — bootstrap, kubeconfig, etcd/controller-manager metrics, built-in components, NFS PV pattern, ArgoCD, node IP summary, useful commands
+- [Ingress](k3s-setup/ingress.md) — OpenBSD `relayd` (internet) and FreeBSD `relayd` on CARP VIP (LAN), cert-manager wildcard, ingress pattern
+- [Troubleshooting](k3s-setup/troubleshooting.md) — etcd Raft log corruption recovery; cluster-wide NFS outage pointer
diff --git a/prompts/skills/f3s/references/k3s-setup/ingress.md b/prompts/skills/f3s/references/k3s-setup/ingress.md
new file mode 100644
index 0000000..6beb4ea
--- /dev/null
+++ b/prompts/skills/f3s/references/k3s-setup/ingress.md
@@ -0,0 +1,120 @@
+# k3s Ingress (Internet + LAN)
+
+Two ingress paths into the k3s cluster:
+- **Internet → OpenBSD relayd** (TLS termination on `blowfish`/`fishfinger`) → WireGuard → Traefik
+- **LAN → FreeBSD relayd on CARP VIP** → k3s Traefik
+
+## External Connectivity: OpenBSD relayd
+
+Default traffic flow for public k3s-backed services: `Internet → OpenBSD relayd (TLS, Let's Encrypt) → WireGuard → k3s Traefik :80 → Service`
+
+### relayd.conf on blowfish/fishfinger
+
+```
+table <f3s> {
+  192.168.2.120
+  192.168.2.121
+  192.168.2.122
+}
+
+http protocol "https" {
+    tls keypair f3s.foo.zone
+    # ... all f3s service TLS keypairs ...
+    # Non-f3s hosts explicitly forwarded to localhost:
+    match request header "Host" value "foo.zone" forward to <localhost>
+    # f3s hosts have NO match rules — use relay-level failover
+}
+
+relay "https4" {
+    listen on <PUBLIC_IP> port 443 tls
+    protocol "https"
+    forward to <f3s> port 80 check tcp      # primary
+    forward to <localhost> port 8080         # fallback when f3s down
+}
+```
+
+`f3s.buetow.org` is now a special case: it no longer points at the k3s/apache backend and is forwarded by OpenBSD `relayd` to `pi0` (`192.168.2.203`) and `pi1` (`192.168.2.204`) via a dedicated `<f3s_static>` backend table.
+
+When all k3s-backed f3s nodes are down, relayd falls back to `localhost:8080` (OpenBSD httpd serving a "Server turned off" page) for the hosts that still use the shared `<f3s>` backend.
+
+## LAN Ingress: FreeBSD relayd on CARP VIP
+
+For LAN access without going through internet gateways:
+`LAN → CARP VIP (192.168.1.138) → FreeBSD relayd → k3s Traefik :443 → Service`
+
+### FreeBSD relayd config (`/usr/local/etc/relayd.conf`)
+
+```
+table <k3s_nodes> { 192.168.1.120 192.168.1.121 192.168.1.122 }
+
+relay "lan_http" {
+    listen on 192.168.1.138 port 80
+    forward to <k3s_nodes> port 80 check tcp
+}
+
+relay "lan_https" {
+    listen on 192.168.1.138 port 443
+    forward to <k3s_nodes> port 443 check tcp
+}
+```
+
+Minimal `/etc/pf.conf` (PF required for relayd):
+
+```
+set skip on lo0
+pass in quick
+pass out quick
+```
+
+```sh
+doas pkg install -y relayd
+doas sysrc pf_enable=YES pflog_enable=YES relayd_enable=YES
+doas service pf start && doas service pflog start && doas service relayd start
+```
+
+Run on both f0 and f1. Only CARP MASTER responds to VIP traffic.
+
+### cert-manager for LAN TLS
+
+LAN services use `*.f3s.lan.foo.zone` with a self-signed CA managed by cert-manager:
+
+```sh
+cd conf/f3s/cert-manager && just install
+# Creates: selfsigned ClusterIssuer, CA cert, wildcard cert (f3s-lan-tls)
+```
+
+Copy secret to service namespace:
+```sh
+kubectl get secret f3s-lan-tls -n cert-manager -o yaml | \
+    sed 's/namespace: cert-manager/namespace: services/' | \
+    kubectl apply -f -
+```
+
+### LAN ingress pattern
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: ingress-lan
+  namespace: services
+  annotations:
+    spec.ingressClassName: traefik
+    traefik.ingress.kubernetes.io/router.entrypoints: web,websecure
+spec:
+  tls:
+    - hosts:
+        - myservice.f3s.lan.foo.zone
+      secretName: f3s-lan-tls
+  rules:
+    - host: myservice.f3s.lan.foo.zone
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: myservice
+                port:
+                  number: 8080
+```
diff --git a/prompts/skills/f3s/references/k3s-setup/install.md b/prompts/skills/f3s/references/k3s-setup/install.md
new file mode 100644
index 0000000..fae9899
--- /dev/null
+++ b/prompts/skills/f3s/references/k3s-setup/install.md
@@ -0,0 +1,169 @@
+# k3s Install
+
+3-node HA k3s cluster running on Rocky Linux VMs (r0, r1, r2). All nodes act as both control-plane and etcd members (no separate worker nodes).
+
+- k3s version: **v1.32.6+k3s1** (as of Part 7)
+- etcd mode: **embedded HA** (`--cluster-init`)
+- All control-plane traffic goes over **WireGuard** (192.168.2.x IPs)
+
+## Prerequisites
+
+- All Rocky Linux VMs (r0, r1, r2) updated and running
+- WireGuard mesh fully configured (see [wireguard.md](../wireguard.md))
+- NVMe disk emulation in place (see [rocky-linux-vms.md](../rocky-linux-vms.md)) — critical for etcd performance
+
+## Installation
+
+### Generate shared token
+
+```sh
+# On Fedora laptop
+pwgen -n 32
+# Copy output to all r nodes:
+echo -n SECRET_TOKEN > ~/.k3s_token  # on r0, r1, r2
+```
+
+### Bootstrap first node (r0)
+
+```sh
+[root@r0 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \
+    sh -s - server --cluster-init \
+    --node-ip=192.168.2.120 \
+    --advertise-address=192.168.2.120 \
+    --tls-san=r0.wg0.wan.buetow.org
+```
+
+`--node-ip` and `--advertise-address` bind etcd to the WireGuard interface so all control-plane traffic is encrypted.
+
+### Join remaining nodes (r1, r2)
+
+```sh
+[root@r1 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \
+    sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \
+    --node-ip=192.168.2.121 \
+    --advertise-address=192.168.2.121 \
+    --tls-san=r1.wg0.wan.buetow.org
+
+[root@r2 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \
+    sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \
+    --node-ip=192.168.2.122 \
+    --advertise-address=192.168.2.122 \
+    --tls-san=r2.wg0.wan.buetow.org
+```
+
+### Verify cluster
+
+```sh
+kubectl get nodes
+# Expected: r0, r1, r2 all Ready with role control-plane,etcd,master
+```
+
+## kubeconfig
+
+```sh
+# Copy from any r node to laptop
+scp root@r0.lan.buetow.org:/etc/rancher/k3s/k3s.yaml ~/.kube/config
+# Edit: replace server address with r0.lan.buetow.org
+# (repeat with r1 or r2 if r0 is down)
+```
+
+## k3s config.yaml — expose etcd and controller-manager metrics
+
+For Prometheus to scrape etcd and controller-manager metrics, add to `/etc/rancher/k3s/config.yaml` on each r node:
+
+```sh
+cat >> /etc/rancher/k3s/config.yaml << 'EOF'
+kube-controller-manager-arg:
+  - bind-address=0.0.0.0
+etcd-expose-metrics: true
+EOF
+systemctl restart k3s
+```
+
+Verify: `curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader`
+
+## Built-in Components
+
+| Component | Purpose |
+|-----------|---------|
+| CoreDNS | DNS for pods |
+| Traefik | Ingress controller |
+| local-path-provisioner | Local PVC storage |
+| metrics-server | Resource metrics |
+| svclb-traefik | ServiceLB for Traefik |
+
+### Scale Traefik to 2 replicas (faster failover)
+
+```sh
+kubectl -n kube-system scale deployment traefik --replicas=2
+```
+
+## NFS Persistent Volumes
+
+Persistent volumes use `hostPath` pointing to NFS-mounted paths:
+
+```
+/data/nfs/k3svolumes/<app>/
+```
+
+NFS is mounted on all r nodes at `/data/nfs/k3svolumes` via stunnel → CARP VIP →
+freeBSD NFS — see [storage/nfs.md](../storage/nfs.md). The
+[`nfs-mount-monitor`](../storage/nfs-mount-monitor.md) watchdog auto-repairs
+hung mounts and force-deletes stuck pods.
+
+Example PV:
+
+```yaml
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: example-pv
+spec:
+  capacity:
+    storage: 1Gi
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  hostPath:
+    path: /data/nfs/k3svolumes/example-volume
+    type: Directory
+```
+
+Create the directory on the NFS share before deploying: `mkdir /data/nfs/k3svolumes/<app>/`
+
+## Deployment: GitOps with ArgoCD
+
+Config repository: `https://codeberg.org/snonux/conf` (directory: `f3s/`)
+
+ArgoCD app structure:
+```
+argocd-apps/
+  monitoring/    # Prometheus, Grafana, Loki, etc.
+  services/      # User-facing services
+  infra/         # Infrastructure components
+  test/          # Test deployments
+```
+
+**To view pre-ArgoCD state** (how things were in Part 7):
+```sh
+git clone https://codeberg.org/snonux/conf.git
+cd conf && git checkout 15a86f3  # last commit before ArgoCD migration
+cd f3s/
+```
+
+## Node IP Summary
+
+| Node | LAN IP | WireGuard IP | k3s API |
+|------|--------|-------------|---------|
+| r0 | 192.168.1.120 | 192.168.2.120 | r0.wg0.wan.buetow.org:6443 |
+| r1 | 192.168.1.121 | 192.168.2.121 | r1.wg0.wan.buetow.org:6443 |
+| r2 | 192.168.1.122 | 192.168.2.122 | r2.wg0.wan.buetow.org:6443 |
+
+## Useful Commands
+
+```sh
+kubectl get nodes                    # cluster status
+kubectl get pods --all-namespaces    # all running pods
+kubectl get namespaces
+kubectl config set-context --current --namespace=<ns>
+```
diff --git a/prompts/skills/f3s/references/k3s-setup/troubleshooting.md b/prompts/skills/f3s/references/k3s-setup/troubleshooting.md
new file mode 100644
index 0000000..01c6029
--- /dev/null
+++ b/prompts/skills/f3s/references/k3s-setup/troubleshooting.md
@@ -0,0 +1,49 @@
+# k3s Troubleshooting
+
+## Etcd Raft Log Corruption Recovery
+
+**Symptom**: k3s crashes on startup with panic:
+```
+tocommit(XXXXXXX) is out of range [lastIndex(YYYYYYY)]
+```
+Caused by `kill -9` on the bhyve process mid-write (corrupts etcd WAL). k3s enters a crash loop and stops after ~2 minutes.
+
+**Recovery procedure** (example: r1 is corrupt):
+
+```sh
+# 1. Stop k3s on the affected node
+ssh root@r1.lan.buetow.org 'systemctl stop k3s'
+
+# 2. Download etcdctl on a healthy node (not bundled with k3s)
+ssh root@r0.lan.buetow.org
+curl -sL https://github.com/etcd-io/etcd/releases/download/v3.5.17/etcd-v3.5.17-linux-amd64.tar.gz \
+  | tar -xz -C /tmp etcd-v3.5.17-linux-amd64/etcdctl
+mv /tmp/etcd-v3.5.17-linux-amd64/etcdctl /tmp/etcdctl
+
+# 3. Find and remove the corrupt member from the cluster
+ETCDCTL_API=3 /tmp/etcdctl \
+  --endpoints=https://127.0.0.1:2379 \
+  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
+  --cert=/var/lib/rancher/k3s/server/tls/etcd/client.crt \
+  --key=/var/lib/rancher/k3s/server/tls/etcd/client.key \
+  member list
+# Find the member ID for r1, then:
+ETCDCTL_API=3 /tmp/etcdctl ... member remove <MEMBER_ID>
+
+# 4. Delete the corrupted etcd data on the affected node
+ssh root@r1.lan.buetow.org 'rm -rf /var/lib/rancher/k3s/server/db/etcd'
+
+# 5. Restart k3s — it rejoins as a fresh member
+ssh root@r1.lan.buetow.org 'systemctl start k3s'
+
+# 6. Verify
+kubectl get nodes  # r1 should return to Ready
+```
+
+> **Prevention**: Always use `doas vm stop rocky` and wait for clean shutdown before stopping the bhyve host. Only use `kill -9` on the bhyve process as a last resort — it can corrupt the etcd WAL.
+
+## Cluster-wide NFS Outages
+
+If NFS goes down cluster-wide, the root cause is usually on the FreeBSD NFS
+server side (f0/f1). Check CARP state, stunnel, nfsd, and
+`vfs.nfsd.nfs_privport` — see [storage/troubleshooting.md](../storage/troubleshooting.md).
diff --git a/prompts/skills/f3s/references/observability.md b/prompts/skills/f3s/references/observability.md
index 4407b1b..b6d7c35 100644
--- a/prompts/skills/f3s/references/observability.md
+++ b/prompts/skills/f3s/references/observability.md
@@ -1,7 +1,5 @@
 # Observability Stack
 
-## Overview
-
 Observability stack deployed into the `monitoring` namespace of the k3s cluster.
 
 **Current state (as of 2026-05-16)**: Prometheus + Alloy only. Grafana, Loki, and Tempo are **disabled** — their ArgoCD manifests are renamed to `.disabled` and their pods do not run.
@@ -22,230 +20,10 @@ Observability stack deployed into the `monitoring` namespace of the k3s cluster.
 | **Loki** | Log aggregation (single-binary mode) | **Disabled** |
 | **Tempo** | Distributed tracing backend | **Disabled** |
 
-## Deployment
-
-All components deployed via **ArgoCD** (GitOps). Manifests:
-```
-https://codeberg.org/snonux/conf/src/branch/master/f3s
-argocd-apps/monitoring/
-```
-
-Deployment tool: `just` (Justfile in each component directory).
-
-### Namespaces
-
-```sh
-kubectl create namespace monitoring
-```
-
-### Disabled component manifests
-
-These files exist in the repo but are renamed `.disabled` so ArgoCD ignores them:
-```
-f3s/argocd-apps/monitoring/loki.yaml.disabled
-f3s/argocd-apps/monitoring/tempo.yaml.disabled
-f3s/argocd-apps/monitoring/grafana-ingress.yaml.disabled
-```
-
-To re-enable, rename back to `.yaml` and ensure Grafana is using a non-NFS PVC (local-path).
-
-## Installing Prometheus
-
-Uses `kube-prometheus-stack` Helm chart with **Grafana subchart disabled** (`grafana.enabled: false`):
-
-```sh
-helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
-helm repo update
-
-# Create NFS storage directory first
-mkdir -p /data/nfs/k3svolumes/prometheus/data
-
-cd conf/f3s/prometheus && just install
-```
-
-### Enable etcd and controller-manager scraping
-
-Add to `persistence-values.yaml`:
-
-```yaml
-kubeEtcd:
-  enabled: true
-  endpoints: [192.168.2.120, 192.168.2.121, 192.168.2.122]
-  service:
-    port: 2381
-    targetPort: 2381
-
-kubeControllerManager:
-  enabled: true
-  endpoints: [192.168.2.120, 192.168.2.121, 192.168.2.122]
-  service:
-    port: 10257
-    targetPort: 10257
-  serviceMonitor:
-    enabled: true
-    https: true
-    insecureSkipVerify: true
-```
-
-Also requires k3s config changes on each r node — see k3s-setup.md.
-
-### Grafana credentials
-
-Default: `admin` / `prom-operator` — change immediately after first login.
-
-Grafana accessible at `grafana.f3s.foo.zone` via Traefik ingress.
-
-## Installing Alloy (minimal)
-
-Alloy is installed as part of the Loki Helm chart but runs with a minimal config (no log shipping):
-
-```sh
-cd conf/f3s/loki && just install
-# installs alloy only (loki itself is disabled via loki.yaml.disabled)
-```
-
-### Current Alloy config (`alloy-values.yaml`)
-
-Minimal — only emits Alloy's own operational logs:
-
-```
-logging {
-  level = "info"
-}
-```
-
-To re-enable log shipping (once Loki is running again), restore the full `discovery.kubernetes` + `loki.source.kubernetes` + `loki.write` pipeline.
-
-## Installing Loki (disabled)
-
-```sh
-mkdir -p /data/nfs/k3svolumes/loki/data
-# Rename loki.yaml.disabled → loki.yaml first, then:
-cd conf/f3s/loki && just install
-```
+## Sub-references
 
-Loki URL (internal): `http://loki.monitoring.svc.cluster.local:3100`
-
-## Installing Tempo (disabled)
-
-```sh
-mkdir -p /data/nfs/k3svolumes/tempo/data
-# Rename tempo.yaml.disabled → tempo.yaml first, then:
-cd conf/f3s/tempo && just install
-```
-
-## Monitoring FreeBSD Hosts (f0, f1, f2)
-
-### Install node_exporter on FreeBSD
-
-```sh
-# On each FreeBSD host
-doas pkg install -y node_exporter
-doas sysrc node_exporter_enable=YES
-# Bind to WireGuard interface (f0=192.168.2.130, f1=192.168.2.131, f2=192.168.2.132)
-doas sysrc node_exporter_args='--web.listen-address=192.168.2.130:9100'
-doas service node_exporter start
-```
-
-### Prometheus scrape config for FreeBSD
-
-`additional-scrape-configs.yaml`:
-
-```yaml
-- job_name: 'node-exporter'
-  static_configs:
-    - targets:
-      - '192.168.2.130:9100'  # f0 via WireGuard
-      - '192.168.2.131:9100'  # f1 via WireGuard
-      - '192.168.2.132:9100'  # f2 via WireGuard
-      labels:
-        os: freebsd
-```
-
-```sh
-kubectl create secret generic additional-scrape-configs \
-    --from-file=additional-scrape-configs.yaml -n monitoring
-```
-
-Add to `persistence-values.yaml`:
-
-```yaml
-prometheus:
-  prometheusSpec:
-    additionalScrapeConfigsSecret:
-      enabled: true
-      name: additional-scrape-configs
-      key: additional-scrape-configs.yaml
-```
-
-### FreeBSD memory compatibility rules
-
-FreeBSD uses different metric names than Linux. PrometheusRule to create Linux-compatible metrics:
-
-```yaml
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: freebsd-memory-rules
-  namespace: monitoring
-  labels:
-    release: prometheus
-spec:
-  groups:
-    - name: freebsd-memory
-      rules:
-        - record: node_memory_MemTotal_bytes
-          expr: node_memory_size_bytes{os="freebsd"}
-        - record: node_memory_MemAvailable_bytes
-          expr: |
-            node_memory_free_bytes{os="freebsd"}
-              + node_memory_inactive_bytes{os="freebsd"}
-              + node_memory_cache_bytes{os="freebsd"}
-        - record: node_memory_MemFree_bytes
-          expr: node_memory_free_bytes{os="freebsd"}
-        - record: node_memory_Buffers_bytes
-          expr: node_memory_buffer_bytes{os="freebsd"}
-        - record: node_memory_Cached_bytes
-          expr: node_memory_cache_bytes{os="freebsd"}
-```
-
-Note: Disk I/O metrics (`node_disk_*`) are not available on FreeBSD — use ZFS-specific dashboards instead.
-
-### ZFS monitoring recording rules
-
-```yaml
-apiVersion: monitoring.coreos.com/v1
-kind: PrometheusRule
-metadata:
-  name: freebsd-zfs-rules
-  namespace: monitoring
-  labels:
-    release: prometheus
-spec:
-  groups:
-    - name: freebsd-zfs-arc
-      interval: 30s
-      rules:
-        - record: node_zfs_arc_hit_rate_percent
-          expr: |
-            100 * (
-              rate(node_zfs_arcstats_hits_total{os="freebsd"}[5m]) /