sync skills: update f3s USB key docs

4 files changed, 148 insertions, 58 deletions

diff --git a/prompts/skills/f3s/references/storage.md b/prompts/skills/f3s/references/storage.md
index a715358..d6c31b0 100644
--- a/prompts/skills/f3s/references/storage.md
+++ b/prompts/skills/f3s/references/storage.md
@@ -6,7 +6,8 @@ Note: original plan was HAST, replaced by **zrepl** (ZFS send/receive) — more
 
 ## Sub-references
 
-- [ZFS Pools & Encryption](storage/zfs.md) — `zdata` pool, physical disks, USB-stored keys, encrypted datasets, boot-time key loading
+- [ZFS Pools & Encryption](storage/zfs.md) — `zdata` pool, physical disks, USB-stored keys mounted by `f3skeys` (not `/etc/fstab`), encrypted datasets, boot-time key loading
+- [USB Key Mounting](storage/usb-keys.md) — `f3skeys`, `/usr/local/sbin/f3s-mount-keys`, and current `zfskeys_datasets` per f-host
 - [zrepl Replication](storage/zrepl.md) — `f0 → f1` nfsdata, `f3 → f2` freebsd VM, sink configs, troubleshooting, DL-state recovery
 - [CARP HA VIP](storage/carp.md) — VIP `192.168.1.138`, `carpcontrol.sh`, mgmt script, auto-failback, SUSPENDED-pool limitation
 - [NFS over stunnel](storage/nfs.md) — NFS server, mutual-TLS stunnel, Rocky client config, `/etc/fstab`
diff --git a/prompts/skills/f3s/references/storage/carp.md b/prompts/skills/f3s/references/storage/carp.md
index abf0685..3f7b741 100644
--- a/prompts/skills/f3s/references/storage/carp.md
+++ b/prompts/skills/f3s/references/storage/carp.md
@@ -45,45 +45,17 @@ doas service devd restart
 
 ## carpcontrol.sh — start/stop NFS+stunnel on failover
 
+Source of truth: `f3s/freebsd-hosts/carp/carpcontrol.sh`.
+
+Install on f0 and f1:
+
 ```sh
-#!/bin/sh
-HOSTNAME=`hostname`
-
-if [ ! -f /data/nfs/nfs.DO_NOT_REMOVE ]; then
-    logger '/data/nfs not mounted, mounting it now!'
-    if [ "$HOSTNAME" = 'f0.lan.buetow.org' ]; then
-        zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata
-        zfs set mountpoint=/data/nfs zdata/enc/nfsdata
-    else
-        zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/sink/f0/zdata/enc/nfsdata
-        zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata
-        zfs mount zdata/sink/f0/zdata/enc/nfsdata
-        zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
-    fi
-    service nfsd stop 2>&1
-    service mountd stop 2>&1
-fi
-
-case "$2" in
-    MASTER)
-        logger "CARP state changed to MASTER, starting services"
-        service rpcbind start >/dev/null 2>&1
-        service mountd start >/dev/null 2>&1
-        service nfsd start >/dev/null 2>&1
-        service nfsuserd start >/dev/null 2>&1
-        service stunnel restart >/dev/null 2>&1
-        ;;
-    BACKUP)
-        logger "CARP state changed to BACKUP, stopping services"
-        service stunnel stop >/dev/null 2>&1
-        service nfsd stop >/dev/null 2>&1
-        service mountd stop >/dev/null 2>&1
-        service nfsuserd stop >/dev/null 2>&1
-        ;;
-esac
+doas install -o root -g wheel -m 0555 carpcontrol.sh /usr/local/bin/carpcontrol.sh
 ```
 
-Install: `doas chmod +x /usr/local/bin/carpcontrol.sh` (copy to f1 too)
+The script must call `/usr/local/sbin/f3s-mount-keys` before any
+`zfs load-key` operation because `/keys` is not mounted by `/etc/fstab`; see
+[USB Key Mounting](usb-keys.md).
 
 ## CARP management script (`/usr/local/bin/carp`)
 
diff --git a/prompts/skills/f3s/references/storage/usb-keys.md b/prompts/skills/f3s/references/storage/usb-keys.md
new file mode 100644
index 0000000..9871daa
--- /dev/null
+++ b/prompts/skills/f3s/references/storage/usb-keys.md
@@ -0,0 +1,116 @@
+# USB Key Mounting for ZFS Encryption
+
+The f-hosts keep raw ZFS encryption keys on per-host UFS USB sticks mounted at
+`/keys`. All four sticks are labeled `F3S_KEYS` and hold all 8 key files as
+cross-host backups.
+
+Do **not** mount `/keys` from `/etc/fstab`. A missing or corrupt key stick must
+not block the FreeBSD base OS from booting.
+
+## Managed Files
+
+Source files live in the conf repo:
+
+```text
+f3s/freebsd-hosts/keys/
+  f3s-mount-keys
+  f3s-load-zfs-keys
+  f3skeys.rc
+```
+
+Installed paths on each f-host:
+
+```text
+/usr/local/sbin/f3s-mount-keys
+/usr/local/sbin/f3s-load-zfs-keys
+/etc/rc.d/f3skeys
+```
+
+`f3skeys` runs before FreeBSD's built-in `zfskeys` service. If the USB stick is
+missing or `fsck_ufs -p` fails, the helper logs the problem and exits
+successfully so boot continues. Encrypted datasets stay locked until the stick
+is repaired and `/usr/local/sbin/f3s-load-zfs-keys` is run manually.
+
+## Setup
+
+Format a new key stick:
+
+```sh
+doas newfs -L F3S_KEYS /dev/da0
+doas mkdir -p /keys
+doas mount -t ufs -o ro /dev/ufs/F3S_KEYS /keys
+```
+
+Label an existing stick without rebuilding it:
+
+```sh
+doas umount /keys
+doas tunefs -L F3S_KEYS /dev/da0
+```
+
+Keep the old `/etc/fstab` line commented on all f-hosts:
+
+```fstab
+# /dev/da0 /keys ufs rw 0 2
+```
+
+Enable boot loading:
+
+```sh
+doas sysrc f3skeys_enable=YES
+doas sysrc zfskeys_enable=YES
+```
+
+Current `zfskeys_datasets` values:
+
+```sh
+# f0
+doas sysrc zfskeys_datasets="zdata/enc zdata/enc/nfsdata zroot/bhyve zroot/garage"
+
+# f1
+doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zroot/garage zdata/sink/f0/zdata/enc/nfsdata"
+
+# f2
+doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zroot/garage zroot/sink/f3/zroot/bhyve/freebsd"
+
+# f3
+doas sysrc zfskeys_datasets="zroot/bhyve"
+```
+
+Replicated sinks with raw encryption need explicit file keylocations:
+
+```sh
+# f1
+doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key \
+  zdata/sink/f0/zdata/enc/nfsdata
+
+# f2
+doas zfs set keylocation=file:///keys/f3.lan.buetow.org:bhyve.key \
+  zroot/sink/f3/zroot/bhyve/freebsd
+```
+
+Manual recovery after boot:
+
+```sh
+doas /usr/local/sbin/f3s-mount-keys --strict
+doas /usr/local/sbin/f3s-load-zfs-keys
+```
+
+## Verification
+
+```sh
+mount | grep ' /keys '
+sysrc -n f3skeys_enable
+sysrc -n zfskeys_enable
+sysrc -n zfskeys_datasets
+doas /usr/local/sbin/f3s-load-zfs-keys
+zfs list -H -o name,encryption,keylocation,keystatus,mounted |
+  awk '$2 != "off" { print }'
+```
+
+Full reboot validation was run on f0, f1, f2, and f3 on 2026-05-30 after this
+change.
+
+Note: `zroot/sink/f3/zroot/bhyve/freebsd` on f2 has `mountpoint=none`; the
+reboot check expects its key to be `available`, but it is not mounted because it
+has no filesystem mountpoint.
diff --git a/prompts/skills/f3s/references/storage/zfs.md b/prompts/skills/f3s/references/storage/zfs.md
index 12f354c..b30208c 100644
--- a/prompts/skills/f3s/references/storage/zfs.md
+++ b/prompts/skills/f3s/references/storage/zfs.md
@@ -21,16 +21,21 @@ doas zpool create zdata ada1   # ada1 = second SSD
 
 ## Encryption Keys (USB Key Storage)
 
-Encryption keys are stored on USB flash drives (UFS-formatted, mounted at `/keys`).
-All four hosts (f0/f1/f2/f3) have USB keys at `/dev/da0` mounted at `/keys`, each holding
-all 8 key files as cross-host backups.
+Encryption keys are stored on USB flash drives (UFS-formatted, mounted at
+`/keys`). All four hosts (f0/f1/f2/f3) have USB keys with UFS label
+`F3S_KEYS`, mounted at `/keys`, each holding all 8 key files as cross-host
+backups.
+
+Do **not** mount `/keys` from `/etc/fstab`. A missing or corrupt key stick must
+not block the FreeBSD base OS from booting. See [USB Key Mounting](usb-keys.md)
+for the `f3skeys` boot helper, install paths, current `zfskeys_datasets`, and
+reboot validation.
 
 ```sh
 # Format and mount USB key (on each node)
-doas newfs /dev/da0
-echo '/dev/da0 /keys ufs rw 0 2' | doas tee -a /etc/fstab
+doas newfs -L F3S_KEYS /dev/da0
 doas mkdir /keys
-doas mount /keys
+doas mount -t ufs -o ro /dev/ufs/F3S_KEYS /keys
 
 # Generate keys (on f0, then copy to f1, f2, f3)
 doas openssl rand -out /keys/f0.lan.buetow.org:bhyve.key 32
@@ -45,6 +50,14 @@ doas chown root /keys/* && doas chmod 400 /keys/*
 # Copy to f1, f2, f3 via tarball
 ```
 
+If an existing stick has no label, unmount it and label it without rebuilding
+the filesystem:
+
+```sh
+doas umount /keys
+doas tunefs -L F3S_KEYS /dev/da0
+```
+
 ## Encryption Setup
 
 ```sh
@@ -72,18 +85,6 @@ doas zfs destroy -R zroot/bhyve_old
 
 ### Auto-load encryption keys on boot
 
-```sh
-# On f0
-doas sysrc zfskeys_enable=YES
-doas sysrc zfskeys_datasets="zdata/enc zdata/enc/nfsdata zroot/bhyve"
-
-# On f1
-doas sysrc zfskeys_enable=YES
-doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata"
-
-# On f3 (bhyve VMs only, no zdata pool yet)
-doas sysrc zfskeys_enable=YES
-doas sysrc zfskeys_datasets="zroot/bhyve"
-doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key \
-  zdata/sink/f0/zdata/enc/nfsdata
-```
+Boot-time key loading is managed by `f3skeys` plus FreeBSD's `zfskeys`. Keep
+the per-host dataset list in [USB Key Mounting](usb-keys.md) up to date when
+adding encrypted ZFS roots.