| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
|
|
|
|
fixes)
|
|
|
|
|
|
|
|
Add shellyfans rc.d service (boot-time auto-on for the rack fans on the
Shelly Plug M Gen 3 at 192.168.1.28) and a standalone shelly-plug.sh
helper to query/control the plug over its HTTP RPC API.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|
relayd on the CARP VIP no longer terminates TLS — it forwards raw TLS to
Traefik, which terminates via the f3s-lan-tls ingress secret. The old
"export cert to relayd" steps were stale and misleading. Add the
restart-not-reload pitfall (a stale relayd process serving an expired
cached keypair caused the 2026-06-30 LAN-cert outage).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|
|
|
|
|
|
|
In-memory guest chat, no DB. Deployment pulls
registry.lan.buetow.org:30001/ychat:f387bac, port 2000, non-root (UID 1000),
readOnlyRootFilesystem with emptyDir on /app/log. TCP-socket probes (ychat
speaks HTTP/0.9-style responses). Ingress: ychat.f3s.buetow.org +
ychat.f3s.lan.buetow.org.
|
|
After a CARP failover or f-host reboot the node host remounts NFS
cleanly but already-running pods keep their old stale bind-mount,
causing writes to return ESTALE (errno 116). Three changes to auto-heal:
1. immich-valkey: replace ping-only liveness with a write-probe
(valkey-cli ping + echo to /data/.nfs-write-probe), so a stale
/data mount fails the probe and kubelet restarts the container with
a fresh bind-mount.
2. immich-server: add label nfs.stale-restart/enabled=true so the
node-level nfs-mount-monitor can exec a write-probe into the pod
and force-delete it if writes fail. Cannot do this via Helm values —
the upstream chart hardcodes an httpGet liveness and its Sprig merge
refills httpGet even when set to null, producing an invalid
dual-handler probe rejected by the k8s API.
3. nfs-mount-monitor/check-nfs-mount.sh: add reap_stale_nfs_pods()
function (throttled to every 30 s) that handles two cases invisible
to the existing fix_mount probe — Case 1: pods stuck in
CreateContainerConfigError with a stale subPath message (git-server,
prometheus hit this); Case 2: opt-in Running+Ready pods whose
in-container write to the NFS volume fails twice in a row (the
immich-server pattern). Deployed to r0/r1/r2 via Rex.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
|
|
|
|
All hostPath PVs are backed by NFS (127.0.0.1:/k3svolumes via stunnel).
Existing NFS sentinel init containers only protect against pod startup when
NFS is unmounted. If NFS goes stale mid-flight (e.g. after an NFS server
restart during FreeBSD upgrades), running pods hold stale file descriptors
and serve errors indefinitely without being restarted.
Fix: add livenessProbe to every NFS-backed container:
- Non-postgres services: test -f /mount-path/.nfs-sentinel (ESTALE fails this)
- Postgres (immich, miniflux): pg_isready AND test -f on pg_filenode.map so
the probe exercises both the TCP listener and the NFS data directory
- webdav: also adds missing nfs-check-data init container (was the only
service without sentinel protection at startup)
- git-server (two containers): test -d /repos (sentinel is in PVC root,
not the repos subPath, so directory existence check is used instead)
All probes: initialDelaySeconds 30 (60 for jellyfin/postgres), periodSeconds
30, failureThreshold 3 — triggers restart after ~90 s of NFS unavailability.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
After a CARP failover where f1 serves as MASTER, any NFS writes go to
the sink dataset (zdata/sink/f0/zdata/enc/nfsdata). When f1 returns to
BACKUP, zfs receive from f0 fails with "destination has been modified
since most recent snapshot" because the filesystem state is ahead of the
last received snapshot.
Fix: on BACKUP transition, roll the sink back to the last zrepl snapshot
before setting readonly=on. Writes during the MASTER window (health
checks, test writes) are intentionally discarded — f0 is authoritative
and replication resumes cleanly from the common snapshot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
When f1 (BACKUP) becomes MASTER, its zrepl sink dataset
(zdata/sink/f0/zdata/enc/nfsdata) was kept readonly=on. NFS clients
could mount and read but all writes failed with EROFS. carpcontrol.sh
now sets readonly=off before starting NFS services on MASTER takeover,
and restores readonly=on before stopping them on BACKUP transition so
zrepl can resume receiving snapshots from f0.
Without this, every controlled CARP failover to f1 left NFS read-only
until manually fixed with `zfs set readonly=off`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
The faketty pipe approach from the shenxn image caused CrashLoopBackOff:
writing to the FIFO via echo sends EOF which terminates cat and bridge.
Instead: override entrypoint to run bridge --noninteractive directly (the
bridge binary at /usr/lib/protonmail/bridge/bridge). Bridge v3 embeds a
gRPC server; the separate 'bridge --cli' exec connects to it as a client.
Login flow: kubectl exec -it <pod> -- /usr/lib/protonmail/bridge/bridge --cli
socat port forwarding (25→1025, 143→1143) preserved in the entrypoint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Root causes of prior CrashLoopBackOff:
1. %no-passphrase unsupported; %no-protection needed instead
2. GPG agent couldn't use pinentry without a TTY
3. Container runs as root (HOME=/root) so mounting at /home/protonmail was wrong
Fix: dedicated setup-pass init container (shares image, has gpg+pass)
that configures allow-loopback-pinentry in gpg-agent.conf + gpg.conf,
then generates a passphrase-free key and inits the pass store. All state
is written to /bridge-data (PVC mount), with GNUPGHOME and
PASSWORD_STORE_DIR env vars so the main container finds the store.
Main container no longer overrides the default entrypoint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Bridge v3 requires 'pass' as keychain store. The startup script now
initializes a passphrase-free GPG key and pass store on first run (all
persisted on the NFS PVC so restarts reconnect without re-login). Mounts
the full /home/protonmail instead of just .config to cover .gnupg and
.password-store. First-time ProtonMail login via bridge --cli still
required; see deployment comment for instructions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
Deploys Proton Bridge (shenxn/protonmail-bridge) as a headless pod in the
services namespace. Config persisted on NFS-backed PVC; nfs-sentinel
initContainer guards against local-XFS shadow on NFS mount failure.
Exposes IMAP (STARTTLS) on NodePort 30143 and SMTP (STARTTLS) on NodePort
30025 for LAN access. Initial ProtonMail authentication is done once via:
kubectl exec -it <pod> -n services -- bridge --cli
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
f3 was previously excluded from FreeBSD host monitoring. Now that
node_exporter is installed and running there, include it in the
node-exporter job so CPU temperature and other host metrics are
collected alongside f0/f1/f2. Also update the temperature alert
comment to reflect that f3 is now covered.
|
|
|
|
|
|
|
|
|
|
|
|
Adds the standard nfs-check initContainer to verify the sentinel file
exists before the main Apache container starts. Prevents silent fall-back
to local XFS when NFS is unmounted on the node.
|
|
|
|
|
|
|
|
|
|
immich-postgres for u5
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
