1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
|
# NFS Sentinel InitContainer Pattern
This pattern makes f3s NFS-backed `hostPath` PersistentVolumes fail loudly when
a k3s node has lost its `/data/nfs/k3svolumes` mount.
## Problem
Most f3s Helm charts use static PVs backed by `hostPath` directories under
`/data/nfs/k3svolumes`. Many use one child directory per volume:
```text
/data/nfs/k3svolumes/<app>/<volume>
```
Some charts use the app directory itself as the PV root:
```text
/data/nfs/k3svolumes/<app>
```
That path is an NFS mount on r0, r1, and r2. If the NFS mount disappears on a
node, kubelet can still bind-mount the local XFS directory at the same path into
a pod. The workload then starts successfully and writes state to the local
shadow directory. When NFS later returns, the running pod continues to see the
wrong backing directory and state appears empty or lost.
This happened to Wallabag on 2026-05-17. The desired behavior is for the pod to
refuse startup when the bind mount resolves to local XFS instead of NFS.
## Pattern
For every NFS-backed app data directory, create a sentinel file on the NFS
server and require each workload to prove the sentinel exists through the same
PVC it will later use.
The sentinel file:
```text
<actual-pv-root>/.nfs-sentinel
```
Rules:
- The file is empty.
- Mode is `0644`.
- It exists only at the actual NFS-backed PV root.
- It must not be created on the local XFS fallback directory on r0, r1, or r2.
- Add one initContainer per NFS-backed workload volume.
- Mount the same PVC in the initContainer at `/mnt`, read-only.
- The initContainer exits non-zero if `/mnt/.nfs-sentinel` is missing.
Place the sentinel at the directory named by the PV's `hostPath.path`, not at a
derived path. For example, if the PV root is
`/data/nfs/k3svolumes/registry`, the sentinel is
`/data/nfs/k3svolumes/registry/.nfs-sentinel`. Do not add a child volume
directory unless the PV actually points at one.
Example:
```yaml
initContainers:
- name: nfs-check-data
image: busybox:stable
command:
- sh
- -c
- |
test -f /mnt/.nfs-sentinel || (
echo "ERROR: NFS sentinel missing at /mnt/.nfs-sentinel"
echo "refusing to start; node likely has NFS unmounted"
echo "pod would otherwise bind-mount the local-XFS shadow"
exit 1
)
volumeMounts:
- name: data
mountPath: /mnt
readOnly: true
```
For a chart with multiple NFS-backed PVCs, repeat the initContainer with a
unique name and the matching volume name:
```yaml
initContainers:
- name: nfs-check-data
image: busybox:stable
command:
- sh
- -c
- |
test -f /mnt/.nfs-sentinel || (
echo "ERROR: NFS sentinel missing at /mnt/.nfs-sentinel"
echo "refusing to start; node likely has NFS unmounted"
echo "pod would otherwise bind-mount the local-XFS shadow"
exit 1
)
volumeMounts:
- name: data
mountPath: /mnt
readOnly: true
- name: nfs-check-media
image: busybox:stable
command:
- sh
- -c
- |
test -f /mnt/.nfs-sentinel || (
echo "ERROR: NFS sentinel missing at /mnt/.nfs-sentinel"
echo "refusing to start; node likely has NFS unmounted"
echo "pod would otherwise bind-mount the local-XFS shadow"
exit 1
)
volumeMounts:
- name: media
mountPath: /mnt
readOnly: true
```
## Why This Works
Kubernetes mounts the PVC into the initContainer before starting the main
container. If the node's `hostPath` resolves to the local fallback directory, the
sentinel file is absent and the initContainer fails. The pod stays in
`Init:CrashLoopBackOff`, which is visible in normal `kubectl get pods` output.
After NFS is repaired, delete the failed pod so the replacement pod re-runs the
initContainer against the restored mount:
```sh
kubectl delete pod -n <namespace> <pod-name>
```
The existing `nfs-mount-monitor` repairs stale or missing node mounts and
already force-deletes pods stuck in `Unknown`, `Pending`, or
`ContainerCreating`. A later task can extend it to clean up sentinel-blocked
`Init:CrashLoopBackOff` pods after the mount is healthy again.
## Deployment Checklist
For each NFS-backed chart:
1. Identify every `hostPath` PV under `/data/nfs/k3svolumes` and record the
exact `hostPath.path` value. This exact path is the PV root.
2. Create a sentinel on f0 for every PV root:
```sh
ssh root@f0 '
touch <actual-pv-root>/.nfs-sentinel &&
chmod 644 <actual-pv-root>/.nfs-sentinel
'
```
3. Add one `nfs-check-<volume>` initContainer per mounted NFS PVC in the owned
Deployment or StatefulSet manifest.
4. Commit and push the chart change; ArgoCD will sync it.
5. Identify the workload's actual selector labels from the manifest, usually
`spec.selector.matchLabels` for Deployments and StatefulSets. Do not assume
the selector is `app=<chart>`; examples in this repository include
`app=docker-registry`, `app=jellyfin-server`, and
`app=koreader-sync-server`, and some charts have multiple workloads.
6. Verify the workload with those actual selector labels:
```sh
kubectl get pod -n <namespace> -l '<selector-labels>'
kubectl describe pod -n <namespace> <pod-name>
kubectl logs -n <namespace> <pod-name> -c nfs-check-<volume>
```
Expected result: the workload is `Running`, and each `nfs-check-*`
initContainer is `Completed`.
## Scope
Apply the pattern only where this repository owns the workload manifest. Do not
patch upstream Helm chart internals through ArgoCD values unless a separate task
explicitly scopes that work.
Feasible charts:
| Chart | Notes |
| --- | --- |
| `anki-sync-server` | Owned workload manifest. |
| `apache` | Owned workload manifest. |
| `audiobookshelf` | Owned workload manifest. |
| `filebrowser` | Owned workload manifest. |
| `git-server` | Owned workload manifest. |
| `goprecords` | Owned workload manifest. |
| `immich` | Only `helm-chart/templates/postgres.yaml` is owned here. |
| `jellyfin` | Owned workload manifest. |
| `keybr` | Owned workload manifest. |
| `kobo-sync-server` | Owned workload manifest. |
| `miniflux` | Owned workload manifest. |
| `navidrome` | Owned workload manifest. |
| `opodsync` | Owned workload manifest. |
| `pkgrepo` | Owned workload manifest. |
| `player` | Owned workload manifest. |
| `radicale` | Owned workload manifest. |
| `registry` | Owned workload manifest. |
| `syncthing` | Owned workload manifest. |
| `wallabag` | Owned workload manifest. |
| `xplayer` | Owned workload manifest. |
Not feasible in this pattern pass:
| Chart | Reason |
| --- | --- |
| `prometheus` | Upstream chart wrapped by ArgoCD. |
| `loki` | Upstream chart wrapped by ArgoCD. |
| `tempo` | Upstream chart wrapped by ArgoCD. |
| `argocd` | Upstream chart wrapped by ArgoCD. |
| `pihole` | Upstream or historical chart usage; current state is on pi2/pi3. |
## Notes For Existing InitContainers
Older `wait-for-nfs` style checks that mount `/data/nfs/k3svolumes` and wait for
a global marker prove only that the top-level mount path exists. They do not
prove that the specific PVC backing directory mounted into the workload is the
NFS directory rather than a local shadow. Replace those checks with the
per-volume sentinel pattern when migrating a chart.
`strategy.type: Recreate` is still useful for singleton stateful workloads
because it avoids overlapping pod instances on the same `hostPath` data
directory. It does not replace the sentinel check; the two protections cover
different failure modes.
|
