summaryrefslogtreecommitdiff
path: root/f3s
AgeCommit message (Collapse)Author
2026-05-10nfs-monitor: make textfile .prom world-readable for node_exporterPaul Buetow
node_exporter runs as uid 65534 (nobody); mktemp creates files with mode 600 (root-only). Add chmod 644 before the atomic mv so the node_exporter process can read nfs_mount_monitor.prom on its scrape. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-monitor: fix node_exporter textfile_collector Helm chart keyPaul Buetow
Use extraHostVolumeMounts (prometheus-node-exporter sub-chart key for host path mounts) instead of extraVolumes/extraVolumeMounts, which are for general volumes. This correctly wires /var/lib/node_exporter/ textfile_collector into the container so the textfile arg takes effect. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-monitor: add Prometheus alerts for NFS auto-repair failuresPaul Buetow
- check-nfs-mount.sh: write nfs_mount_monitor_consecutive_failures gauge to /var/lib/node_exporter/textfile_collector/nfs_mount_monitor.prom on every run (via write_textfile_metric helper, called from write_fail_count and directly on healthy runs); atomic tmp+mv write prevents partial reads - Rexfile: create /var/lib/node_exporter/textfile_collector dir on r-nodes - prometheus.yaml (ArgoCD app): enable textfile_collector in node_exporter DaemonSet via extraArgs/extraVolumes/extraVolumeMounts; mount host path /var/lib/node_exporter/textfile_collector into container - persistence-values.yaml: sync node_exporter textfile_collector config - nfs-mount-monitor-alerts.yaml: PrometheusRule with two alerts: NfsMountAutoRepairWarning (>= 3 consecutive failures, severity: warning) NfsMountAutoRepairCritical (>= 5 consecutive failures, severity: critical) wired into new 'nfs-alerts' Alertmanager receiver with 30m repeat_interval Tested: rex deploy succeeded, .prom files present on r0/r1/r2, timer clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-mount-monitor: escalate to reboot after N consecutive fix_mount failuresPaul Buetow
Persist a consecutive-failure counter to /var/lib/nfs-mount-monitor/fail-count. Increment on every fix_mount failure; reset to 0 on any successful repair or when all three probes pass cleanly. After NFS_FAIL_THRESHOLD (default 5, ~50s) consecutive failures the node is cordoned via kubectl and rebooted with 'systemctl reboot' so the cluster stops routing pods to a silently broken node. NFS_FAIL_THRESHOLD is configurable via /etc/default/nfs-mount-monitor (deployed as EnvironmentFile in the .service unit) without touching the script. Also fix Rexfile path resolution: __FILE__ inside a Rex task resolves to the internal Rex loader path, not the Rexfile itself; use realpath($::rexfile) instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-mount-monitor: strengthen fix_mount recovery sequencePaul Buetow
Add lazy umount fallback, D-state process killer, stunnel restart, and 60-second hard deadline to prevent fix_mount from looping forever when processes are stuck in D state on a stale NFSv4-over-stunnel mount. Recovery sequence is now: 1. mount -o remount -f (cheap, no disruption) 2. kill_pinning_processes (SIGKILL D-state procs with nfs_ wchan) 3. umount -f (force unmount) 4. umount -l (lazy detach VFS node if -f failed) 5. systemctl restart stunnel + 2s sleep (refresh TLS transport) 6. mount (fresh mount) The 60s deadline uses bash $SECONDS so fix_mount can never outlast its own 10-second timer interval by an unbounded amount. Deployed to all three r-nodes (r0/r1/r2) via rex nfs_mount_monitor. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-mount-monitor: add write-probe to detect 'reads OK, writes hang' statePaul Buetow
Stunnel-wrapped NFSv4 can enter a half-broken state where mountpoint(1) returns true and stat(1) completes from cache, but ALL writes hang indefinitely. This was observed on r2 on 2026-05-10 causing navidrome to be unschedulable. The existing two probes passed while writes were dead. Add a third probe (write-probe) after the stat probe: write the shell's PID to a per-host .healthcheck.<hostname> file and immediately remove it, wrapped in a 5-second timeout. The per-host filename prevents r0/r1/r2 from racing on the same file. 5s gives one full NFS retransmit window (timeo=10 deciseconds = 1s, retrans=2) plus margin without making the 10-second timer run too long. Deployed to r0/r1/r2 via rex nfs_mount_monitor; all three nodes confirmed running the new script (journalctl shows clean exits). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10f3s/r-nodes: track NFS auto-repair script and systemd units in conf repoPaul Buetow
Pull check-nfs-mount.sh, nfs-mount-monitor.service, and nfs-mount-monitor.timer from r0/r1/r2 (confirmed identical on all three nodes) into f3s/r-nodes/nfs-mount-monitor/. Add f3s/r-nodes/Rexfile with an idempotent nfs_mount_monitor task that pushes the files to all three r-nodes as root and reloads systemd when content changes. Wire the new Rexfile into the repo root Rexfile. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10navidrome: use Recreate strategyPaul Buetow
SQLite on RWO PVC can only have one writer. RollingUpdate keeps the old pod alive until the new pod is Ready, but the new pod blocks indefinitely on SQLite open because the old pod still holds the db-shm/db-wal lock. Recreate kills the old pod first. Amp-Thread-ID: https://ampcode.com/threads/T-019e109f-fa43-7467-bb0b-1a4a2d3d0b9e Co-authored-by: Amp <amp@ampcode.com>
2026-05-10navidrome: add startup/readiness/liveness probesPaul Buetow
Prevents Traefik from routing traffic to navidrome before it has finished its ~1m14s cold start (NFS cache warm-up + initial scan), which was causing 501s and 'context canceled' errors right after each cluster boot. Amp-Thread-ID: https://ampcode.com/threads/T-019e109f-fa43-7467-bb0b-1a4a2d3d0b9e Co-authored-by: Amp <amp@ampcode.com>
2026-05-09add xplayerPaul Buetow
2026-05-03Update player image tagPaul Buetow
2026-05-03Add player f3s deploymentPaul Buetow
2026-05-03add player.f3s.buetow.orgPaul Buetow
2026-04-17f3s/anki-sync-server: add debug NodePort service on 30800Paul Buetow
Exposes anki-sync-server directly on all k3s nodes at NodePort 30800, bypassing Traefik. Used to isolate whether the HTTP 303 stream failures (Anki client maps zstd body read errors to SEE_OTHER) originate in the Traefik HTTP proxy layer or in the pod itself.
2026-04-16goprecords: restore for:1h after alert testPaul Buetow
2026-04-16goprecords: temp set for:1m for alert testPaul Buetow
2026-04-16goprecords: add Prometheus scraping and stale-host alert rulePaul Buetow
- service.yaml: add 'metrics' port (8080) so kubernetes SD auto-discovers the /metrics endpoint alongside the existing http port (80) - prometheus/manifests/goprecords-alerts.yaml: GoprecordsHostNotReporting fires (warning) when a non-excluded host last reported >5 months ago Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16goprecords: bump to 0.5.1Paul Buetow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16goprecords: bump to 0.5.0Paul Buetow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14goprecords: deploy image 0.4.1 (HTML without top nav links).Paul Buetow
Made-with: Cursor
2026-04-14goprecords: deploy image 0.4.0 (foo.zone-style HTML and stats order).Paul Buetow
Made-with: Cursor
2026-04-14goprecords: deploy image 0.3.1 (HTML dashboard at /).Paul Buetow
Bump Helm and docker-image tags for the new goprecords release. Made-with: Cursor
2026-04-14Adjust goprecords runtime permissions for cluster storage.Paul Buetow
Run the goprecords pod as root and keep the hostPath PV type aligned with the existing immutable volume configuration so ArgoCD can sync cleanly while the service can create and open its auth database on the shared stats path. Made-with: Cursor
2026-04-14Add goprecords service deployment for f3s.Paul Buetow
Introduce Docker build/push workflow, Helm manifests, and ArgoCD application wiring for goprecords so the cluster can deploy the new daemon API service from the private registry. Made-with: Cursor
2026-04-13always AlwyasPaul Buetow
2026-04-13add goprecords.f3s.buetow.orgPaul Buetow
2026-04-11pihole: docker-pi dnsmasq wildcard, README for pi2/pi3, ArgoCD parityPaul Buetow
Add dnsmasq.d wildcard for *.f3s.lan.buetow.org → 192.168.1.138 and example compose for Pis; refresh README (DNS on pi2/pi3, etc-dnsmasq.d). Align dormant ArgoCD Helm customDnsEntries with the same wildcard. Made-with: Cursor
2026-04-10add emaPaul Buetow
2026-04-08Deactivate Apache ArgoCD applicationPaul Buetow
Amp-Thread-ID: https://ampcode.com/threads/T-019d6da8-3a08-7079-bb2a-eb072c0bf17f Co-authored-by: Amp <amp@ampcode.com>
2026-04-08h0: document PI phase 3.2 role splitPaul Buetow
2026-04-08g0: add PI Phase 3.1 verification notesPaul Buetow
2026-04-08f0: document Pi-hole phase 2.2 deploymentPaul Buetow
2026-04-08d0: document PI phase 1.2 static content syncPaul Buetow
2026-04-08Document PI phase 2.1 Docker CE for e0Paul Buetow
2026-04-08c0: document pi0/pi1 lighttpd phase 1.1Paul Buetow
2026-04-08b0: document PI phase 0.2 hostname verificationPaul Buetow
2026-04-08a0: record PI Phase 0.1 baselinePaul Buetow
2026-04-08feat(f3s): deploy Trivy Operator for image CVE scanning (task h)Paul Buetow
- ArgoCD app: aquasecurity/trivy-operator in monitoring with ServiceMonitor - PrometheusRule for Critical/High trivy_image_vulnerabilities alerts - Alertmanager route/receiver for component=trivy (UI; webhook TBD) Made-with: Cursor
2026-04-08garage: bind S3 and admin endpoints on IPv4Paul Buetow
Ensure Garage listens on WireGuard IPv4 addresses so relay hosts can reach node S3/admin ports reliably. Made-with: Cursor
2026-04-08f3s/prometheus: add Garage admin scrape targets (task f)Paul Buetow
Add job_name garage for 192.168.2.130-132:3903 with os=freebsd label. Mirror config in additional-scrape-configs-secret for kube apply/ArgoCD. Made-with: Cursor
2026-04-08garage: Garage 2.2 TOML schema and deploy permissionsPaul Buetow
Align etc/garage.f*.toml with garage-2.2.0 (metadata_dir, data_dir, rpc_secret, rpc_bind_addr, rpc_public_addr per host, s3_api/admin, replication_factor). Bind RPC on 0.0.0.0:3901 so IPv4 LAN peers can reach nodes on FreeBSD. Install config as root:garage 640 so the rc.d garage user can read garage.toml. Made-with: Cursor
2026-04-08task b: add f3s/garage deploy layout (Rex, Just, host TOMLs)Paul Buetow
- Per-host garage.fN.toml with __RPC_SECRET__ placeholder and 23G data cap - Rexfile: paul@f0–f2, inject secret from secrets/rpc_secret, install to /usr/local/etc/garage.toml, restart garage - Justfile: deploy, init-secrets, status, layout, stats, bucket-create - Root Rexfile: require f3s/garage/Rexfile (not matched by */Rexfile glob) - Gitignore f3s/garage/secrets/ Made-with: Cursor
2026-04-06immich: fix chart value structure - image tag under server/ml controllers, ↵Paul Buetow
remove duplicate controllers.server Amp-Thread-ID: https://ampcode.com/threads/T-019d6154-8fdf-74fe-b865-f796d8a4214a Co-authored-by: Amp <amp@ampcode.com>
2026-04-06immich: fix ML config to use chart's machine-learning.controllers.main structurePaul Buetow
Amp-Thread-ID: https://ampcode.com/threads/T-019d6154-8fdf-74fe-b865-f796d8a4214a Co-authored-by: Amp <amp@ampcode.com>
2026-04-06immich: tune ML throughput - add postgres anti-affinity, increase intra-op ↵Paul Buetow
threads, increase worker timeout Amp-Thread-ID: https://ampcode.com/threads/T-019d6154-8fdf-74fe-b865-f796d8a4214a Co-authored-by: Amp <amp@ampcode.com>
2026-04-05immich: relax postgres probes and add resource limitsPaul Buetow
- Increase liveness probe tolerance (60s delay, 30s period, 10s timeout, 6 failures) - Increase readiness probe tolerance (15s delay, 10s period, 5s timeout, 6 failures) - Add resource requests (100m CPU, 512Mi RAM) and limits (2Gi RAM) - Fixes crash loop caused by probe killing postgres during recovery Amp-Thread-ID: https://ampcode.com/threads/T-019d5f54-27f2-740c-ac41-0f980e7aecd3 Co-authored-by: Amp <amp@ampcode.com>
2026-04-04fix(immich): use dual-style values for resources and affinity to ensure they ↵Paul Buetow
apply
2026-04-04fix(immich): use correctly nested controllers structure for affinity and 4Gi ↵Paul Buetow
resources
2026-04-04fix(immich): increase memory limits to 4Gi to avoid OOMKilled for MLPaul Buetow
2026-04-04feat(immich): add preferred anti-affinity and resources to balance loadPaul Buetow
generated by cgit v1.2.3 (git 2.25.1) at 2026-08-10 19:29:02 +0000