1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
|
# zrepl: Continuous ZFS Replication
Continuous ZFS replication for the encrypted NFS dataset (f0 → f1) and the
standalone FreeBSD dev VM (f3 → f2). Original plan was HAST, replaced by
zrepl (`zfs send/recv`) — more reliable and avoids the HAST-induced ZFS
corruption that hit us during failover testing.
Install on the participating hosts:
```sh
doas pkg install -y zrepl
```
## f0 configuration (`/usr/local/etc/zrepl/zrepl.yml`)
```yaml
global:
logging:
- type: stdout
level: info
format: human
jobs:
- name: f0_to_f1_nfsdata
type: push
connect:
type: tcp
address: "192.168.2.131:8888" # f1 WireGuard IP
filesystems:
"zdata/enc/nfsdata": true
send:
encrypted: true
snapshotting:
type: periodic
prefix: zrepl_
interval: 1m # every minute
pruning:
keep_sender:
- type: last_n
count: 10
- type: grid
grid: 24x1h | 14x1d | 6x30d
regex: "^zrepl_.*"
keep_receiver:
- type: last_n
count: 10
- type: grid
grid: 24x1h | 14x1d | 6x30d
regex: "^zrepl_.*"
# Note: f0_to_f1_freebsd job removed — the FreeBSD VM was migrated to f3.
# It is now replicated from f3 → f2 (see f3 zrepl config below).
```
## f3 configuration (push: VMs → f2)
```yaml
global:
logging:
- type: stdout
level: info
format: human
jobs:
- name: f3_to_f2_freebsd
type: push
connect:
type: tcp
address: "192.168.2.132:8888" # f2 WireGuard IP
filesystems:
"zroot/bhyve/freebsd": true # development FreeBSD VM
"zroot/bhyve/rocky": true # plain Rocky Linux VM
send:
encrypted: true
snapshotting:
type: periodic
prefix: zrepl_
interval: 10m
pruning:
keep_sender:
- type: last_n
count: 10
- type: grid
grid: 24x1h | 14x1d
regex: "^zrepl_.*"
keep_receiver:
- type: last_n
count: 10
- type: grid
grid: 24x1h | 14x1d
regex: "^zrepl_.*"
```
## f2 configuration (sink for f3's freebsd VM)
f2 has no second drive so the sink lives in `zroot/sink`:
```sh
doas zfs create zroot/sink
```
`/usr/local/etc/zrepl/zrepl.yml`:
```yaml
global:
logging:
- type: stdout
level: info
format: human
jobs:
- name: sink
type: sink
serve:
type: tcp
listen: "192.168.2.132:8888" # f2 WireGuard IP
clients:
"192.168.2.133": "f3"
recv:
placeholder:
encryption: inherit
root_fs: "zroot/sink"
```
Replicated path: `zroot/bhyve/freebsd` → `zroot/sink/f3/zroot/bhyve/freebsd`
Important: do not let `zfs-periodic` snapshot zrepl-managed sender or receiver
datasets. Snapshot creation should be owned by zrepl. On f2,
`/etc/periodic.conf` disables `zfs-periodic` snapshot creation:
```sh
daily_zfs_snapshot_enable="NO"
weekly_zfs_snapshot_enable="NO"
monthly_zfs_snapshot_enable="NO"
```
The local zrepl `snap` job on f3 also excludes both VM datasets so they are only snapshotted by the push job:
```yaml
- name: local_zfs_snapshots
type: snap
filesystems:
"zroot<": true
"zroot/bhyve/freebsd": false
"zroot/bhyve/rocky": false
...
```
The local zrepl `snap` job on f2 also explicitly excludes `zroot/sink<`.
## f1 configuration (sink)
```sh
doas zfs create zdata/sink # receive dataset
```
`/usr/local/etc/zrepl/zrepl.yml`:
```yaml
global:
logging:
- type: stdout
level: info
format: human
jobs:
- name: sink
type: sink
serve:
type: tcp
listen: "192.168.2.131:8888"
clients:
"192.168.2.130": "f0"
recv:
placeholder:
encryption: inherit
root_fs: "zdata/sink"
```
## Enable and start
```sh
doas sysrc zrepl_enable=YES
doas service zrepl start
doas zrepl status # monitor replication
```
Replicated paths: `zdata/enc/nfsdata` → `zdata/sink/f0/zdata/enc/nfsdata`
## Mount replica on f1 (read-only standby)
```sh
doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \
zdata/sink/f0/zdata/enc/nfsdata
doas mkdir -p /data/nfs
doas zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata
doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata # prevent replication breakage
```
## Replication liveness check
A small cron job on f0 writes a canary file into the replicated dataset so its
presence and mtime on f1's read-only sink reveals how fresh the zrepl
replication is.
On **f0** (root crontab), every 10 minutes, gated on a sentinel file so the
canary only runs when the dataset is actually in use:
```
*/10 * * * * test -f /data/nfs/nfs.DO_NOT_REMOVE && /usr/bin/touch /data/nfs/nfs.LIVE_CHECK
```
`/data/nfs` on f0 is `zdata/enc/nfsdata` — the dataset the `f0_to_f1_nfsdata`
job pushes to f1 every minute. zrepl snapshots + sends, so the touched
`nfs.LIVE_CHECK` (and its mtime) appear on f1's read-only sink at
`/data/nfs/nfs.LIVE_CHECK` within roughly one replication interval.
On **f1**, read the replica to check freshness (the sink is read-only):
```sh
stat -f '%Sm' /data/nfs/nfs.LIVE_CHECK # f0's last touch time, as replicated
date '+%Sm' # compare to now
```
If the mtime lags more than ~10–11 minutes behind real time, zrepl replication
is stalled (see Troubleshooting). The `/data/nfs/nfs.DO_NOT_REMOVE` sentinel
also doubles as the zrepl-replication canary guard. Installed on f0 2026-07-20.
## Failover design: intentionally read-only replica
The standby replica is read-only by design. Manual failover (not automatic) to prevent split-brain. To fix broken replication after accidental writes: `doas zfs rollback <snapshot>`.
## Troubleshooting
```sh
# Signal manual replication
doas zrepl signal wakeup f0_to_f1_nfsdata
# Fix "no common snapshot" — destroy and re-replicate
doas zfs destroy -r zdata/sink/f0/zdata/enc/nfsdata
# Test network connectivity
nc -zv 192.168.2.131 8888
# Monitor progress
doas zrepl status --mode raw | grep BytesReplicated
```
**zrepl DL-state on f1 after mid-replication f0 reboot**: if f0 reboots while zrepl is
actively replicating, f1's `[zfskern]` thread can enter **DL state** (disk + locked).
Symptoms: `zpool list`, `zfs list`, `ls /data/nfs/` all hang indefinitely; `zfs set
readonly=off` may return immediately (the kernel path differs). To recover on f1:
```sh
# Stop zrepl to release the replication lock
doas service zrepl stop
# Wait ~30–60 s for the kernel state to drain; then verify
doas zpool list
doas zfs list
doas service zrepl start
```
If ZFS commands still hang after stopping zrepl, a reboot of f1 is required.
The NFS data is still available on f0 so k3s is unaffected during f1 recovery.
|
