diff options
| -rw-r--r-- | prompts/skills/f3s/references/storage.md | 3 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/storage/carp.md | 44 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/storage/usb-keys.md | 116 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/storage/zfs.md | 43 |
4 files changed, 148 insertions, 58 deletions
diff --git a/prompts/skills/f3s/references/storage.md b/prompts/skills/f3s/references/storage.md index a715358..d6c31b0 100644 --- a/prompts/skills/f3s/references/storage.md +++ b/prompts/skills/f3s/references/storage.md @@ -6,7 +6,8 @@ Note: original plan was HAST, replaced by **zrepl** (ZFS send/receive) — more ## Sub-references -- [ZFS Pools & Encryption](storage/zfs.md) — `zdata` pool, physical disks, USB-stored keys, encrypted datasets, boot-time key loading +- [ZFS Pools & Encryption](storage/zfs.md) — `zdata` pool, physical disks, USB-stored keys mounted by `f3skeys` (not `/etc/fstab`), encrypted datasets, boot-time key loading +- [USB Key Mounting](storage/usb-keys.md) — `f3skeys`, `/usr/local/sbin/f3s-mount-keys`, and current `zfskeys_datasets` per f-host - [zrepl Replication](storage/zrepl.md) — `f0 → f1` nfsdata, `f3 → f2` freebsd VM, sink configs, troubleshooting, DL-state recovery - [CARP HA VIP](storage/carp.md) — VIP `192.168.1.138`, `carpcontrol.sh`, mgmt script, auto-failback, SUSPENDED-pool limitation - [NFS over stunnel](storage/nfs.md) — NFS server, mutual-TLS stunnel, Rocky client config, `/etc/fstab` diff --git a/prompts/skills/f3s/references/storage/carp.md b/prompts/skills/f3s/references/storage/carp.md index abf0685..3f7b741 100644 --- a/prompts/skills/f3s/references/storage/carp.md +++ b/prompts/skills/f3s/references/storage/carp.md @@ -45,45 +45,17 @@ doas service devd restart ## carpcontrol.sh — start/stop NFS+stunnel on failover +Source of truth: `f3s/freebsd-hosts/carp/carpcontrol.sh`. + +Install on f0 and f1: + ```sh -#!/bin/sh -HOSTNAME=`hostname` - -if [ ! -f /data/nfs/nfs.DO_NOT_REMOVE ]; then - logger '/data/nfs not mounted, mounting it now!' - if [ "$HOSTNAME" = 'f0.lan.buetow.org' ]; then - zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata - zfs set mountpoint=/data/nfs zdata/enc/nfsdata - else - zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/sink/f0/zdata/enc/nfsdata - zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata - zfs mount zdata/sink/f0/zdata/enc/nfsdata - zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata - fi - service nfsd stop 2>&1 - service mountd stop 2>&1 -fi - -case "$2" in - MASTER) - logger "CARP state changed to MASTER, starting services" - service rpcbind start >/dev/null 2>&1 - service mountd start >/dev/null 2>&1 - service nfsd start >/dev/null 2>&1 - service nfsuserd start >/dev/null 2>&1 - service stunnel restart >/dev/null 2>&1 - ;; - BACKUP) - logger "CARP state changed to BACKUP, stopping services" - service stunnel stop >/dev/null 2>&1 - service nfsd stop >/dev/null 2>&1 - service mountd stop >/dev/null 2>&1 - service nfsuserd stop >/dev/null 2>&1 - ;; -esac +doas install -o root -g wheel -m 0555 carpcontrol.sh /usr/local/bin/carpcontrol.sh ``` -Install: `doas chmod +x /usr/local/bin/carpcontrol.sh` (copy to f1 too) +The script must call `/usr/local/sbin/f3s-mount-keys` before any +`zfs load-key` operation because `/keys` is not mounted by `/etc/fstab`; see +[USB Key Mounting](usb-keys.md). ## CARP management script (`/usr/local/bin/carp`) diff --git a/prompts/skills/f3s/references/storage/usb-keys.md b/prompts/skills/f3s/references/storage/usb-keys.md new file mode 100644 index 0000000..9871daa --- /dev/null +++ b/prompts/skills/f3s/references/storage/usb-keys.md @@ -0,0 +1,116 @@ +# USB Key Mounting for ZFS Encryption + +The f-hosts keep raw ZFS encryption keys on per-host UFS USB sticks mounted at +`/keys`. All four sticks are labeled `F3S_KEYS` and hold all 8 key files as +cross-host backups. + +Do **not** mount `/keys` from `/etc/fstab`. A missing or corrupt key stick must +not block the FreeBSD base OS from booting. + +## Managed Files + +Source files live in the conf repo: + +```text +f3s/freebsd-hosts/keys/ + f3s-mount-keys + f3s-load-zfs-keys + f3skeys.rc +``` + +Installed paths on each f-host: + +```text +/usr/local/sbin/f3s-mount-keys +/usr/local/sbin/f3s-load-zfs-keys +/etc/rc.d/f3skeys +``` + +`f3skeys` runs before FreeBSD's built-in `zfskeys` service. If the USB stick is +missing or `fsck_ufs -p` fails, the helper logs the problem and exits +successfully so boot continues. Encrypted datasets stay locked until the stick +is repaired and `/usr/local/sbin/f3s-load-zfs-keys` is run manually. + +## Setup + +Format a new key stick: + +```sh +doas newfs -L F3S_KEYS /dev/da0 +doas mkdir -p /keys +doas mount -t ufs -o ro /dev/ufs/F3S_KEYS /keys +``` + +Label an existing stick without rebuilding it: + +```sh +doas umount /keys +doas tunefs -L F3S_KEYS /dev/da0 +``` + +Keep the old `/etc/fstab` line commented on all f-hosts: + +```fstab +# /dev/da0 /keys ufs rw 0 2 +``` + +Enable boot loading: + +```sh +doas sysrc f3skeys_enable=YES +doas sysrc zfskeys_enable=YES +``` + +Current `zfskeys_datasets` values: + +```sh +# f0 +doas sysrc zfskeys_datasets="zdata/enc zdata/enc/nfsdata zroot/bhyve zroot/garage" + +# f1 +doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zroot/garage zdata/sink/f0/zdata/enc/nfsdata" + +# f2 +doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zroot/garage zroot/sink/f3/zroot/bhyve/freebsd" + +# f3 +doas sysrc zfskeys_datasets="zroot/bhyve" +``` + +Replicated sinks with raw encryption need explicit file keylocations: + +```sh +# f1 +doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key \ + zdata/sink/f0/zdata/enc/nfsdata + +# f2 +doas zfs set keylocation=file:///keys/f3.lan.buetow.org:bhyve.key \ + zroot/sink/f3/zroot/bhyve/freebsd +``` + +Manual recovery after boot: + +```sh +doas /usr/local/sbin/f3s-mount-keys --strict +doas /usr/local/sbin/f3s-load-zfs-keys +``` + +## Verification + +```sh +mount | grep ' /keys ' +sysrc -n f3skeys_enable +sysrc -n zfskeys_enable +sysrc -n zfskeys_datasets +doas /usr/local/sbin/f3s-load-zfs-keys +zfs list -H -o name,encryption,keylocation,keystatus,mounted | + awk '$2 != "off" { print }' +``` + +Full reboot validation was run on f0, f1, f2, and f3 on 2026-05-30 after this +change. + +Note: `zroot/sink/f3/zroot/bhyve/freebsd` on f2 has `mountpoint=none`; the +reboot check expects its key to be `available`, but it is not mounted because it +has no filesystem mountpoint. diff --git a/prompts/skills/f3s/references/storage/zfs.md b/prompts/skills/f3s/references/storage/zfs.md index 12f354c..b30208c 100644 --- a/prompts/skills/f3s/references/storage/zfs.md +++ b/prompts/skills/f3s/references/storage/zfs.md @@ -21,16 +21,21 @@ doas zpool create zdata ada1 # ada1 = second SSD ## Encryption Keys (USB Key Storage) -Encryption keys are stored on USB flash drives (UFS-formatted, mounted at `/keys`). -All four hosts (f0/f1/f2/f3) have USB keys at `/dev/da0` mounted at `/keys`, each holding -all 8 key files as cross-host backups. +Encryption keys are stored on USB flash drives (UFS-formatted, mounted at +`/keys`). All four hosts (f0/f1/f2/f3) have USB keys with UFS label +`F3S_KEYS`, mounted at `/keys`, each holding all 8 key files as cross-host +backups. + +Do **not** mount `/keys` from `/etc/fstab`. A missing or corrupt key stick must +not block the FreeBSD base OS from booting. See [USB Key Mounting](usb-keys.md) +for the `f3skeys` boot helper, install paths, current `zfskeys_datasets`, and +reboot validation. ```sh # Format and mount USB key (on each node) -doas newfs /dev/da0 -echo '/dev/da0 /keys ufs rw 0 2' | doas tee -a /etc/fstab +doas newfs -L F3S_KEYS /dev/da0 doas mkdir /keys -doas mount /keys +doas mount -t ufs -o ro /dev/ufs/F3S_KEYS /keys # Generate keys (on f0, then copy to f1, f2, f3) doas openssl rand -out /keys/f0.lan.buetow.org:bhyve.key 32 @@ -45,6 +50,14 @@ doas chown root /keys/* && doas chmod 400 /keys/* # Copy to f1, f2, f3 via tarball ``` +If an existing stick has no label, unmount it and label it without rebuilding +the filesystem: + +```sh +doas umount /keys +doas tunefs -L F3S_KEYS /dev/da0 +``` + ## Encryption Setup ```sh @@ -72,18 +85,6 @@ doas zfs destroy -R zroot/bhyve_old ### Auto-load encryption keys on boot -```sh -# On f0 -doas sysrc zfskeys_enable=YES -doas sysrc zfskeys_datasets="zdata/enc zdata/enc/nfsdata zroot/bhyve" - -# On f1 -doas sysrc zfskeys_enable=YES -doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata" - -# On f3 (bhyve VMs only, no zdata pool yet) -doas sysrc zfskeys_enable=YES -doas sysrc zfskeys_datasets="zroot/bhyve" -doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key \ - zdata/sink/f0/zdata/enc/nfsdata -``` +Boot-time key loading is managed by `f3skeys` plus FreeBSD's `zfskeys`. Keep +the per-host dataset list in [USB Key Mounting](usb-keys.md) up to date when +adding encrypted ZFS roots. |