f3s-storage skill: document zrepl replication liveness check (f0 cron canary)

f0 root crontab touches /data/nfs/nfs.LIVE_CHECK every 10 min, gated on
/data/nfs/nfs.DO_NOT_REMOVE. zrepl replicates zdata/enc/nfsdata to f1's
read-only sink every minute, so the file's mtime on f1 reveals replication
freshness (stalled if >~10-11 min behind). Added to zrepl.md and the hub
SKILL.md reference bullet.

2 files changed, 30 insertions, 1 deletions

diff --git a/prompts/skills/f3s-storage/SKILL.md b/prompts/skills/f3s-storage/SKILL.md
index 9c49968..e865f78 100644
--- a/prompts/skills/f3s-storage/SKILL.md
+++ b/prompts/skills/f3s-storage/SKILL.md
@@ -21,7 +21,7 @@ Note: original plan was HAST, replaced by **zrepl** (ZFS send/receive) — more
 
 - [ZFS Pools & Encryption](references/zfs.md) — `zdata` pool, physical disks, USB-stored keys mounted by `f3skeys` (not `/etc/fstab`), encrypted datasets, boot-time key loading
 - [USB Key Mounting](references/usb-keys.md) — `f3skeys`, `/usr/local/sbin/f3s-mount-keys`, current `zfskeys_datasets` per f-host, and the removable quarterly `zusb` backup pool (raw key on `/keys/zusb.key`, manual `zusb-load`/`zusb-unload` on all f-hosts)
-- [zrepl Replication](references/zrepl.md) — `f0 → f1` nfsdata, `f3 → f2` freebsd VM, sink configs, troubleshooting, DL-state recovery
+- [zrepl Replication](references/zrepl.md) — `f0 → f1` nfsdata, `f3 → f2` freebsd VM, sink configs, replication liveness check (f0 cron canary on `/data/nfs/nfs.LIVE_CHECK`), troubleshooting, DL-state recovery
 - [CARP HA VIP](references/carp.md) — VIP `192.168.1.138`, `carpcontrol.sh`, mgmt script, auto-failback, SUSPENDED-pool limitation
 - [NFS over stunnel](references/nfs.md) — NFS server, mutual-TLS stunnel, Rocky client config, `/etc/fstab`
 - [nfs-mount-monitor](references/nfs-mount-monitor.md) — systemd watchdog on r-nodes (mount/stat/write probes, fail counter, cordon-and-reboot escalation)
diff --git a/prompts/skills/f3s-storage/references/zrepl.md b/prompts/skills/f3s-storage/references/zrepl.md
index dc90676..ea31644 100644
--- a/prompts/skills/f3s-storage/references/zrepl.md
+++ b/prompts/skills/f3s-storage/references/zrepl.md
@@ -198,6 +198,35 @@ doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
 doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata   # prevent replication breakage
 ```
 
+## Replication liveness check
+
+A small cron job on f0 writes a canary file into the replicated dataset so its
+presence and mtime on f1's read-only sink reveals how fresh the zrepl
+replication is.
+
+On **f0** (root crontab), every 10 minutes, gated on a sentinel file so the
+canary only runs when the dataset is actually in use:
+
+```
+*/10 * * * * test -f /data/nfs/nfs.DO_NOT_REMOVE && /usr/bin/touch /data/nfs/nfs.LIVE_CHECK
+```
+
+`/data/nfs` on f0 is `zdata/enc/nfsdata` — the dataset the `f0_to_f1_nfsdata`
+job pushes to f1 every minute. zrepl snapshots + sends, so the touched
+`nfs.LIVE_CHECK` (and its mtime) appear on f1's read-only sink at
+`/data/nfs/nfs.LIVE_CHECK` within roughly one replication interval.
+
+On **f1**, read the replica to check freshness (the sink is read-only):
+
+```sh
+stat -f '%Sm' /data/nfs/nfs.LIVE_CHECK     # f0's last touch time, as replicated
+date '+%Sm'                                  # compare to now
+```
+
+If the mtime lags more than ~10–11 minutes behind real time, zrepl replication
+is stalled (see Troubleshooting). The `/data/nfs/nfs.DO_NOT_REMOVE` sentinel
+also doubles as the zrepl-replication canary guard. Installed on f0 2026-07-20.
+
 ## Failover design: intentionally read-only replica
 
 The standby replica is read-only by design. Manual failover (not automatic) to prevent split-brain. To fix broken replication after accidental writes: `doas zfs rollback <snapshot>`.