diff options
Diffstat (limited to 'prompts/skills/f3s-storage')
| -rw-r--r-- | prompts/skills/f3s-storage/SKILL.md | 2 | ||||
| -rw-r--r-- | prompts/skills/f3s-storage/references/zrepl.md | 29 |
2 files changed, 30 insertions, 1 deletions
diff --git a/prompts/skills/f3s-storage/SKILL.md b/prompts/skills/f3s-storage/SKILL.md index 9c49968..e865f78 100644 --- a/prompts/skills/f3s-storage/SKILL.md +++ b/prompts/skills/f3s-storage/SKILL.md @@ -21,7 +21,7 @@ Note: original plan was HAST, replaced by **zrepl** (ZFS send/receive) — more - [ZFS Pools & Encryption](references/zfs.md) — `zdata` pool, physical disks, USB-stored keys mounted by `f3skeys` (not `/etc/fstab`), encrypted datasets, boot-time key loading - [USB Key Mounting](references/usb-keys.md) — `f3skeys`, `/usr/local/sbin/f3s-mount-keys`, current `zfskeys_datasets` per f-host, and the removable quarterly `zusb` backup pool (raw key on `/keys/zusb.key`, manual `zusb-load`/`zusb-unload` on all f-hosts) -- [zrepl Replication](references/zrepl.md) — `f0 → f1` nfsdata, `f3 → f2` freebsd VM, sink configs, troubleshooting, DL-state recovery +- [zrepl Replication](references/zrepl.md) — `f0 → f1` nfsdata, `f3 → f2` freebsd VM, sink configs, replication liveness check (f0 cron canary on `/data/nfs/nfs.LIVE_CHECK`), troubleshooting, DL-state recovery - [CARP HA VIP](references/carp.md) — VIP `192.168.1.138`, `carpcontrol.sh`, mgmt script, auto-failback, SUSPENDED-pool limitation - [NFS over stunnel](references/nfs.md) — NFS server, mutual-TLS stunnel, Rocky client config, `/etc/fstab` - [nfs-mount-monitor](references/nfs-mount-monitor.md) — systemd watchdog on r-nodes (mount/stat/write probes, fail counter, cordon-and-reboot escalation) diff --git a/prompts/skills/f3s-storage/references/zrepl.md b/prompts/skills/f3s-storage/references/zrepl.md index dc90676..ea31644 100644 --- a/prompts/skills/f3s-storage/references/zrepl.md +++ b/prompts/skills/f3s-storage/references/zrepl.md @@ -198,6 +198,35 @@ doas zfs mount zdata/sink/f0/zdata/enc/nfsdata doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata # prevent replication breakage ``` +## Replication liveness check + +A small cron job on f0 writes a canary file into the replicated dataset so its +presence and mtime on f1's read-only sink reveals how fresh the zrepl +replication is. + +On **f0** (root crontab), every 10 minutes, gated on a sentinel file so the +canary only runs when the dataset is actually in use: + +``` +*/10 * * * * test -f /data/nfs/nfs.DO_NOT_REMOVE && /usr/bin/touch /data/nfs/nfs.LIVE_CHECK +``` + +`/data/nfs` on f0 is `zdata/enc/nfsdata` — the dataset the `f0_to_f1_nfsdata` +job pushes to f1 every minute. zrepl snapshots + sends, so the touched +`nfs.LIVE_CHECK` (and its mtime) appear on f1's read-only sink at +`/data/nfs/nfs.LIVE_CHECK` within roughly one replication interval. + +On **f1**, read the replica to check freshness (the sink is read-only): + +```sh +stat -f '%Sm' /data/nfs/nfs.LIVE_CHECK # f0's last touch time, as replicated +date '+%Sm' # compare to now +``` + +If the mtime lags more than ~10–11 minutes behind real time, zrepl replication +is stalled (see Troubleshooting). The `/data/nfs/nfs.DO_NOT_REMOVE` sentinel +also doubles as the zrepl-replication canary guard. Installed on f0 2026-07-20. + ## Failover design: intentionally read-only replica The standby replica is read-only by design. Manual failover (not automatic) to prevent split-brain. To fix broken replication after accidental writes: `doas zfs rollback <snapshot>`. |