1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
|
# r-node Deploy Mechanism (Rex)
The reusable way to roll out files and systemd units to the three k3s Rocky
Linux VMs — **r0/r1/r2** — is a [Rex](https://www.rexify.org/) task. The
canonical, fully-worked example is the **`nfs_mount_monitor`** task. Treat it
as the template for *any* r-node rollout; you do not need to re-derive the
flow each time.
## Hosts: r0/r1/r2 vs f0/f1/f2
The r-nodes are Rocky Linux 9 bhyve guests, one per FreeBSD host:
| VM | LAN IP | Runs on f-host |
|----|--------|----------------|
| r0 | 192.168.1.120 | f0 (192.168.1.130) |
| r1 | 192.168.1.121 | f1 (192.168.1.131) |
| r2 | 192.168.1.122 | f2 (192.168.1.132) |
f3 is **not** part of this group — it is standalone bhyve and hosts the plain
`rocky` VM, not a k3s node. r-node deploys never touch f3.
## Where it lives
In the conf repo (`https://codeberg.org/snonux/conf`, dir `f3s/`):
```
f3s/r-nodes/Rexfile # deploy tasks for r0/r1/r2
f3s/r-nodes/nfs-mount-monitor/ # source files the task pushes
check-nfs-mount.sh # → /usr/local/bin/
nfs-mount-monitor.default # → /etc/default/ (tunables)
nfs-mount-monitor.service # → /etc/systemd/system/
nfs-mount-monitor.timer # → /etc/systemd/system/
```
## The deploy command
Run from the **conf repo root**:
```sh
rex -f f3s/r-nodes/Rexfile nfs_mount_monitor
```
This pushes to all three r-nodes at once. To target a single node, use Rex's
host filter, e.g. `rex -f f3s/r-nodes/Rexfile -H 192.168.1.120 nfs_mount_monitor`.
## How the pattern works (the reusable parts)
The Rexfile establishes conventions every r-node task inherits:
- **`group r_nodes => qw(192.168.1.120 192.168.1.121 192.168.1.122)`** — the
three k3s VMs by LAN IP. Each `task` declares `group => 'r_nodes'`.
- **`user 'root'; sudo FALSE;`** — tasks connect as **root** over SSH. The
`paul` user has no sudo on the r-nodes, and writing to `/usr/local/bin` and
managing systemd both need root. Root SSH is pre-authorized via
`authorized_keys`.
- **`parallelism 3;`** — all three nodes deploy concurrently. Safe because the
tasks are idempotent and independent per node.
- **`$RNODES_DIR`** is resolved with `realpath($::rexfile)` so source-file
paths stay valid regardless of CWD or Rex worker forking.
### Idempotent rollout flow
Inside the task, each file is deployed with Rex's `file` resource:
```perl
file '/usr/local/bin/check-nfs-mount.sh',
source => catfile($monitor_dir, 'check-nfs-mount.sh'),
owner => 'root', group => 'root', mode => '755',
on_change => sub { $changed = 1 };
```
Rex only writes a file when its **content actually differs** from what's on
the node — so re-running the task is a no-op when nothing changed. The
`on_change` handlers set a single `$changed` flag; only if something changed
does the task run:
```perl
run 'systemctl daemon-reload';
run 'systemctl restart nfs-mount-monitor.timer';
```
Finally — regardless of whether files changed — the task **converges** the
service state so the timer is always enabled and running:
```perl
service 'nfs-mount-monitor.timer', ensure => 'started';
run 'systemctl enable nfs-mount-monitor.timer';
```
This split (reload/restart only on change, enable/start always) is the
pattern to copy: cheap, repeatable, and self-healing if a node drifted.
The task also pre-creates the state/output directories it needs
(`/var/lib/nfs-mount-monitor` mode 700,
`/var/lib/node_exporter/textfile_collector` mode 755) so the deployed script
has somewhere to write.
## Verify after deploy
```sh
# On each r-node (over SSH as root) — confirm the timer is active & enabled:
systemctl status nfs-mount-monitor.timer
# Watch the service fire and log:
journalctl -u nfs-mount-monitor -f
```
What success looks like:
- `systemctl status …timer` reports **active (waiting)** and **enabled**, with
a `Trigger:` line a few seconds out (the timer fires every 10 s).
- `journalctl -u nfs-mount-monitor` shows a fresh oneshot run roughly every
10 s with no errors.
One-shot remote check from a roaming laptop (see
[remote-access.md](remote-access.md)):
```sh
ssh -A -J rex@fishfinger.buetow.org root@r0.wg0 \
"systemctl is-active nfs-mount-monitor.timer && systemctl is-enabled nfs-mount-monitor.timer"
```
## Gotchas
- **Run from the conf repo root**, not from `f3s/r-nodes/` — the `-f` path in
every example is repo-relative (`f3s/r-nodes/Rexfile`).
- **Root, no sudo.** If a task errors on permission, it's almost always that
the connection fell back to `paul` (who has no sudo here). The Rexfile sets
`user 'root'` for exactly this reason.
- **`parallelism 3` only works if tasks are idempotent and node-independent.**
Keep new tasks that way, or lower the parallelism.
- **Adding a new file** to a deploy: give it `on_change => sub { $changed = 1 }`
so a reload/restart fires when it changes, and place it before the
`if ($changed)` block.
## Related
- The **behavior** of the deployed monitor (probes, fail-counter, cordon &
auto-reboot escalation, alerting) is documented separately in
[storage/nfs-mount-monitor.md](../../f3s-storage/references/nfs-mount-monitor.md). This file
covers only the *deploy mechanism*; that one covers *what gets deployed*.
- There are sibling Rexfiles in the conf repo (`f3s/garage/Rexfile`,
`frontends/Rexfile`) that follow the same group/user/idempotent-`file`
shape against different host groups.
|
