1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
|
# USB Key Mounting for ZFS Encryption
The f-hosts keep raw ZFS encryption keys on per-host UFS USB sticks mounted at
`/keys`. All four sticks are labeled `F3S_KEYS` and hold all 8 key files as
cross-host backups.
Do **not** mount `/keys` from `/etc/fstab`. A missing or corrupt key stick must
not block the FreeBSD base OS from booting.
## Managed Files
Source files live in the conf repo:
```text
f3s/freebsd-hosts/keys/
f3s-mount-keys
f3s-load-zfs-keys
f3skeys.rc
```
Installed paths on each f-host:
```text
/usr/local/sbin/f3s-mount-keys
/usr/local/sbin/f3s-load-zfs-keys
/etc/rc.d/f3skeys
```
`f3skeys` runs before FreeBSD's built-in `zfskeys` service. If the USB stick is
missing or `fsck_ufs -p` fails, the helper logs the problem and exits
successfully so boot continues. Encrypted datasets stay locked until the stick
is repaired and `/usr/local/sbin/f3s-load-zfs-keys` is run manually.
## Setup
Format a new key stick:
```sh
doas newfs -L F3S_KEYS /dev/da0
doas mkdir -p /keys
doas mount -t ufs -o ro /dev/ufs/F3S_KEYS /keys
```
Label an existing stick without rebuilding it:
```sh
doas umount /keys
doas tunefs -L F3S_KEYS /dev/da0
```
Keep the old `/etc/fstab` line commented on all f-hosts:
```fstab
# /dev/da0 /keys ufs rw 0 2
```
Enable boot loading:
```sh
doas sysrc f3skeys_enable=YES
doas sysrc zfskeys_enable=YES
```
Current `zfskeys_datasets` values:
```sh
# f0
doas sysrc zfskeys_datasets="zdata/enc zdata/enc/nfsdata zroot/bhyve zroot/garage"
# f1
doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zroot/garage zdata/sink/f0/zdata/enc/nfsdata"
# f2
doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zroot/garage zroot/sink/f3/zroot/bhyve/freebsd"
# f3
doas sysrc zfskeys_datasets="zroot/bhyve"
```
Replicated sinks with raw encryption need explicit file keylocations:
```sh
# f1
doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key \
zdata/sink/f0/zdata/enc/nfsdata
# f2
doas zfs set keylocation=file:///keys/f3.lan.buetow.org:bhyve.key \
zroot/sink/f3/zroot/bhyve/freebsd
```
Manual recovery after boot:
```sh
doas /usr/local/sbin/f3s-mount-keys --strict
doas /usr/local/sbin/f3s-load-zfs-keys
```
## Verification
```sh
mount | grep ' /keys '
sysrc -n f3skeys_enable
sysrc -n zfskeys_enable
sysrc -n zfskeys_datasets
doas /usr/local/sbin/f3s-load-zfs-keys
zfs list -H -o name,encryption,keylocation,keystatus,mounted |
awk '$2 != "off" { print }'
```
Full reboot validation was run on f0, f1, f2, and f3 on 2026-05-30 after this
change.
Note: `zroot/sink/f3/zroot/bhyve/freebsd` on f2 has `mountpoint=none`; the
reboot check expects its key to be `available`, but it is not mounted because it
has no filesystem mountpoint.
## Removable backup pool (`zusb`)
`zusb` is a **4-disk raidz2 ZFS pool on 1.8 TB USB-SATA disks** (ASMT ASM235CM
bridges) used as the **offline backup storage device** — plugged in and loaded
**roughly once per quarter** to back up data, then exported and unplugged. It
lives on whichever f-host it is currently plugged into (f1 as of 2026-07-20).
It is **not** auto-imported or auto-mounted at boot and is **not** in any
host's `zfskeys_datasets` (removable disks must never block boot). It is loaded
manually with `/usr/local/bin/zusb-load` and exported with
`/usr/local/bin/zusb-unload`.
The encryption root is `zusb/data/enc`, rekeyed to the same raw-key-on-stick
scheme as the other f-host secrets: `keyformat=raw`,
`keylocation=file:///keys/zusb.key`. The 32-byte key file `/keys/zusb.key` is
placed on **all four** `F3S_KEYS` sticks, and the load/unload scripts are
deployed to **all four** f-hosts, so the disk stack can be re-plugged to any
f-host and loaded there with no per-host setup.
`zusb/data/enc` was migrated from t450, where it was unlocked via a
passphrase-protected `zroot/secret` keystore (`/zroot/secret/zroot.enc.key`);
that old passphrase key is now obsolete for `zusb`.
Scripts and deployment docs live in the conf repo at
`f3s/freebsd-hosts/zusb/` (`zusb-load`, `zusb-unload`, `README.md`). The raw
key itself is **not** in git — it is copied stick-to-stick like the other keys.
|
