summaryrefslogtreecommitdiff
path: root/f3s
AgeCommit message (Collapse)Author
2026-05-13f3s/beets-art: schedule at noon Europe/Sofia instead of 03:30 UTCPaul Buetow
Use the CronJob timeZone field (GA since k8s 1.27, supported by the k3s 1.32 cluster) so cron interprets the schedule in local time directly, avoiding manual UTC conversion. Amp-Thread-ID: https://ampcode.com/threads/T-019e223a-d137-705e-879b-84130c0e78ea Co-authored-by: Amp <amp@ampcode.com>
2026-05-13f3s/beets-art: fix fetchart silently disabled by string-not-list configPaul Buetow
beets 2.x parses `sources: coverart itunes amazon albumart` as a single key whose value is "*", rejects it with UnknownPairError, and the entire fetchart plugin fails to load. Net effect: every job ran "successfully" but fetched zero cover art (verified: 0/195 albums had artpath set; all existing cover.jpg files predated the deployment). Convert sources and cover_names to proper YAML lists so the plugin loads. Confirmed network egress is fine (CAA + iTunes return HTTP 200). Amp-Thread-ID: https://ampcode.com/threads/T-019e223a-d137-705e-879b-84130c0e78ea Co-authored-by: Amp <amp@ampcode.com>
2026-05-13f3s/beets-art: pipe yes into embedart to bypass interactive promptPaul Buetow
beet embedart (no -f) hard-codes a "Modify artwork for N albums (Y/n)?" confirmation with no flag to suppress it. The CronJob has no stdin, so the command exits with "stdin stream ended while input required" and embedart never runs. Pipe `yes` into the command; safety still relies on embedart.ifempty:no and embedart.compare_threshold:50 from config. Amp-Thread-ID: https://ampcode.com/threads/T-019e223a-d137-705e-879b-84130c0e78ea Co-authored-by: Amp <amp@ampcode.com>
2026-05-13f3s/beets-art: fix BEETSDIR being read-only (state.pickle write failure)Paul Buetow
The ConfigMap mount at /etc/beets is kernel-enforced read-only, so beets could not write its incremental import state file (state.pickle), which broke incremental: yes — every nightly run would re-walk the entire library. Fix: point BEETSDIR at the writable state PVC (/state) and pass -c /etc/beets/config.yaml on every beet invocation so the ConfigMap is still the single source of truth for config. Also fix the Justfile run-now recipe to use a bash shebang so $() works. Amp-Thread-ID: https://ampcode.com/threads/T-019e223a-d137-705e-879b-84130c0e78ea Co-authored-by: Amp <amp@ampcode.com>
2026-05-13f3s/beets-art: nightly k3s CronJob to fetch+embed cover art for NavidromePaul Buetow
Adds a beets-based CronJob that runs every night on r1 (where the Navidrome music PVC lives), fetching external cover.jpg into each album folder and embedding art into audio files. Idempotent on re-runs: - import.incremental skips already-known album folders - fetchart skips albums that already have cover art - embedart with ifempty:no + compare_threshold:50 only fills missing embeds and refuses risky overwrites Navidrome picks new art up via its existing 1h scan; no Navidrome change required. Reuses navidrome-music-pvc directly (RWO is fine because both pods pin to r1 via nodeSelector). State (library.db, logs) lives on a small local-path PVC, regenerable by deleting the PVC. Files: f3s/beets-art/helm-chart/{Chart.yaml,README.md,templates/*.yaml} f3s/beets-art/Justfile (status, logs, run-now, suspend, resume, shell) f3s/argocd-apps/services/beets-art.yaml Amp-Thread-ID: https://ampcode.com/threads/T-019e223a-d137-705e-879b-84130c0e78ea Co-authored-by: Amp <amp@ampcode.com>
2026-05-10navidrome: pin image to 0.61.2 and set imagePullPolicy: IfNotPresentPaul Buetow
Pinning to a specific version avoids silent breaking upgrades and makes kubectl rollout undo meaningful. IfNotPresent skips unnecessary re-pulls on pod restarts. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10navidrome: move /data PVC from NFS to local-path on r1Paul Buetow
SQLite over NFS causes two problems: file-lock races on rolling restarts (fixed with Recreate strategy but underlying fragility remains), and 19s image-cache init at startup due to stunnel TLS latency on every cache read. Replace navidrome-data-pv/pvc (static hostPath over NFS at /data/nfs/k3svolumes/navidrome/data) with a dynamic local-path PVC provisioned on r1 (/var/lib/rancher/k3s/storage). Pin the deployment to r1 via nodeSelector so the local PV is always reachable. Existing DB and cache migrated: navidrome.db (23 MB), image/background/ plugin caches (~118 MB) copied via a migration pod before first start. Result: startupTime=41ms (was ~20s), Image cache init=29ms (was ~19s). Music PVC stays on NFS (200 GB library, unchanged). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-monitor: make textfile .prom world-readable for node_exporterPaul Buetow
node_exporter runs as uid 65534 (nobody); mktemp creates files with mode 600 (root-only). Add chmod 644 before the atomic mv so the node_exporter process can read nfs_mount_monitor.prom on its scrape. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-monitor: fix node_exporter textfile_collector Helm chart keyPaul Buetow
Use extraHostVolumeMounts (prometheus-node-exporter sub-chart key for host path mounts) instead of extraVolumes/extraVolumeMounts, which are for general volumes. This correctly wires /var/lib/node_exporter/ textfile_collector into the container so the textfile arg takes effect. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-monitor: add Prometheus alerts for NFS auto-repair failuresPaul Buetow
- check-nfs-mount.sh: write nfs_mount_monitor_consecutive_failures gauge to /var/lib/node_exporter/textfile_collector/nfs_mount_monitor.prom on every run (via write_textfile_metric helper, called from write_fail_count and directly on healthy runs); atomic tmp+mv write prevents partial reads - Rexfile: create /var/lib/node_exporter/textfile_collector dir on r-nodes - prometheus.yaml (ArgoCD app): enable textfile_collector in node_exporter DaemonSet via extraArgs/extraVolumes/extraVolumeMounts; mount host path /var/lib/node_exporter/textfile_collector into container - persistence-values.yaml: sync node_exporter textfile_collector config - nfs-mount-monitor-alerts.yaml: PrometheusRule with two alerts: NfsMountAutoRepairWarning (>= 3 consecutive failures, severity: warning) NfsMountAutoRepairCritical (>= 5 consecutive failures, severity: critical) wired into new 'nfs-alerts' Alertmanager receiver with 30m repeat_interval Tested: rex deploy succeeded, .prom files present on r0/r1/r2, timer clean. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-mount-monitor: escalate to reboot after N consecutive fix_mount failuresPaul Buetow
Persist a consecutive-failure counter to /var/lib/nfs-mount-monitor/fail-count. Increment on every fix_mount failure; reset to 0 on any successful repair or when all three probes pass cleanly. After NFS_FAIL_THRESHOLD (default 5, ~50s) consecutive failures the node is cordoned via kubectl and rebooted with 'systemctl reboot' so the cluster stops routing pods to a silently broken node. NFS_FAIL_THRESHOLD is configurable via /etc/default/nfs-mount-monitor (deployed as EnvironmentFile in the .service unit) without touching the script. Also fix Rexfile path resolution: __FILE__ inside a Rex task resolves to the internal Rex loader path, not the Rexfile itself; use realpath($::rexfile) instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-mount-monitor: strengthen fix_mount recovery sequencePaul Buetow
Add lazy umount fallback, D-state process killer, stunnel restart, and 60-second hard deadline to prevent fix_mount from looping forever when processes are stuck in D state on a stale NFSv4-over-stunnel mount. Recovery sequence is now: 1. mount -o remount -f (cheap, no disruption) 2. kill_pinning_processes (SIGKILL D-state procs with nfs_ wchan) 3. umount -f (force unmount) 4. umount -l (lazy detach VFS node if -f failed) 5. systemctl restart stunnel + 2s sleep (refresh TLS transport) 6. mount (fresh mount) The 60s deadline uses bash $SECONDS so fix_mount can never outlast its own 10-second timer interval by an unbounded amount. Deployed to all three r-nodes (r0/r1/r2) via rex nfs_mount_monitor. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10nfs-mount-monitor: add write-probe to detect 'reads OK, writes hang' statePaul Buetow
Stunnel-wrapped NFSv4 can enter a half-broken state where mountpoint(1) returns true and stat(1) completes from cache, but ALL writes hang indefinitely. This was observed on r2 on 2026-05-10 causing navidrome to be unschedulable. The existing two probes passed while writes were dead. Add a third probe (write-probe) after the stat probe: write the shell's PID to a per-host .healthcheck.<hostname> file and immediately remove it, wrapped in a 5-second timeout. The per-host filename prevents r0/r1/r2 from racing on the same file. 5s gives one full NFS retransmit window (timeo=10 deciseconds = 1s, retrans=2) plus margin without making the 10-second timer run too long. Deployed to r0/r1/r2 via rex nfs_mount_monitor; all three nodes confirmed running the new script (journalctl shows clean exits). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10f3s/r-nodes: track NFS auto-repair script and systemd units in conf repoPaul Buetow
Pull check-nfs-mount.sh, nfs-mount-monitor.service, and nfs-mount-monitor.timer from r0/r1/r2 (confirmed identical on all three nodes) into f3s/r-nodes/nfs-mount-monitor/. Add f3s/r-nodes/Rexfile with an idempotent nfs_mount_monitor task that pushes the files to all three r-nodes as root and reloads systemd when content changes. Wire the new Rexfile into the repo root Rexfile. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-10navidrome: use Recreate strategyPaul Buetow
SQLite on RWO PVC can only have one writer. RollingUpdate keeps the old pod alive until the new pod is Ready, but the new pod blocks indefinitely on SQLite open because the old pod still holds the db-shm/db-wal lock. Recreate kills the old pod first. Amp-Thread-ID: https://ampcode.com/threads/T-019e109f-fa43-7467-bb0b-1a4a2d3d0b9e Co-authored-by: Amp <amp@ampcode.com>
2026-05-10navidrome: add startup/readiness/liveness probesPaul Buetow
Prevents Traefik from routing traffic to navidrome before it has finished its ~1m14s cold start (NFS cache warm-up + initial scan), which was causing 501s and 'context canceled' errors right after each cluster boot. Amp-Thread-ID: https://ampcode.com/threads/T-019e109f-fa43-7467-bb0b-1a4a2d3d0b9e Co-authored-by: Amp <amp@ampcode.com>
2026-05-09add xplayerPaul Buetow
2026-05-03Update player image tagPaul Buetow
2026-05-03Add player f3s deploymentPaul Buetow
2026-05-03add player.f3s.buetow.orgPaul Buetow
2026-04-17f3s/anki-sync-server: add debug NodePort service on 30800Paul Buetow
Exposes anki-sync-server directly on all k3s nodes at NodePort 30800, bypassing Traefik. Used to isolate whether the HTTP 303 stream failures (Anki client maps zstd body read errors to SEE_OTHER) originate in the Traefik HTTP proxy layer or in the pod itself.
2026-04-16goprecords: restore for:1h after alert testPaul Buetow
2026-04-16goprecords: temp set for:1m for alert testPaul Buetow
2026-04-16goprecords: add Prometheus scraping and stale-host alert rulePaul Buetow
- service.yaml: add 'metrics' port (8080) so kubernetes SD auto-discovers the /metrics endpoint alongside the existing http port (80) - prometheus/manifests/goprecords-alerts.yaml: GoprecordsHostNotReporting fires (warning) when a non-excluded host last reported >5 months ago Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16goprecords: bump to 0.5.1Paul Buetow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-16goprecords: bump to 0.5.0Paul Buetow
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-14goprecords: deploy image 0.4.1 (HTML without top nav links).Paul Buetow
Made-with: Cursor
2026-04-14goprecords: deploy image 0.4.0 (foo.zone-style HTML and stats order).Paul Buetow
Made-with: Cursor
2026-04-14goprecords: deploy image 0.3.1 (HTML dashboard at /).Paul Buetow
Bump Helm and docker-image tags for the new goprecords release. Made-with: Cursor
2026-04-14Adjust goprecords runtime permissions for cluster storage.Paul Buetow
Run the goprecords pod as root and keep the hostPath PV type aligned with the existing immutable volume configuration so ArgoCD can sync cleanly while the service can create and open its auth database on the shared stats path. Made-with: Cursor
2026-04-14Add goprecords service deployment for f3s.Paul Buetow
Introduce Docker build/push workflow, Helm manifests, and ArgoCD application wiring for goprecords so the cluster can deploy the new daemon API service from the private registry. Made-with: Cursor
2026-04-13always AlwyasPaul Buetow
2026-04-13add goprecords.f3s.buetow.orgPaul Buetow
2026-04-11pihole: docker-pi dnsmasq wildcard, README for pi2/pi3, ArgoCD parityPaul Buetow
Add dnsmasq.d wildcard for *.f3s.lan.buetow.org → 192.168.1.138 and example compose for Pis; refresh README (DNS on pi2/pi3, etc-dnsmasq.d). Align dormant ArgoCD Helm customDnsEntries with the same wildcard. Made-with: Cursor
2026-04-10add emaPaul Buetow
2026-04-08Deactivate Apache ArgoCD applicationPaul Buetow
Amp-Thread-ID: https://ampcode.com/threads/T-019d6da8-3a08-7079-bb2a-eb072c0bf17f Co-authored-by: Amp <amp@ampcode.com>
2026-04-08h0: document PI phase 3.2 role splitPaul Buetow
2026-04-08g0: add PI Phase 3.1 verification notesPaul Buetow
2026-04-08f0: document Pi-hole phase 2.2 deploymentPaul Buetow
2026-04-08d0: document PI phase 1.2 static content syncPaul Buetow
2026-04-08Document PI phase 2.1 Docker CE for e0Paul Buetow
2026-04-08c0: document pi0/pi1 lighttpd phase 1.1Paul Buetow
2026-04-08b0: document PI phase 0.2 hostname verificationPaul Buetow
2026-04-08a0: record PI Phase 0.1 baselinePaul Buetow
2026-04-08feat(f3s): deploy Trivy Operator for image CVE scanning (task h)Paul Buetow
- ArgoCD app: aquasecurity/trivy-operator in monitoring with ServiceMonitor - PrometheusRule for Critical/High trivy_image_vulnerabilities alerts - Alertmanager route/receiver for component=trivy (UI; webhook TBD) Made-with: Cursor
2026-04-08garage: bind S3 and admin endpoints on IPv4Paul Buetow
Ensure Garage listens on WireGuard IPv4 addresses so relay hosts can reach node S3/admin ports reliably. Made-with: Cursor
2026-04-08f3s/prometheus: add Garage admin scrape targets (task f)Paul Buetow
Add job_name garage for 192.168.2.130-132:3903 with os=freebsd label. Mirror config in additional-scrape-configs-secret for kube apply/ArgoCD. Made-with: Cursor
2026-04-08garage: Garage 2.2 TOML schema and deploy permissionsPaul Buetow
Align etc/garage.f*.toml with garage-2.2.0 (metadata_dir, data_dir, rpc_secret, rpc_bind_addr, rpc_public_addr per host, s3_api/admin, replication_factor). Bind RPC on 0.0.0.0:3901 so IPv4 LAN peers can reach nodes on FreeBSD. Install config as root:garage 640 so the rc.d garage user can read garage.toml. Made-with: Cursor
2026-04-08task b: add f3s/garage deploy layout (Rex, Just, host TOMLs)Paul Buetow
- Per-host garage.fN.toml with __RPC_SECRET__ placeholder and 23G data cap - Rexfile: paul@f0–f2, inject secret from secrets/rpc_secret, install to /usr/local/etc/garage.toml, restart garage - Justfile: deploy, init-secrets, status, layout, stats, bucket-create - Root Rexfile: require f3s/garage/Rexfile (not matched by */Rexfile glob) - Gitignore f3s/garage/secrets/ Made-with: Cursor
2026-04-06immich: fix chart value structure - image tag under server/ml controllers, ↵Paul Buetow
remove duplicate controllers.server Amp-Thread-ID: https://ampcode.com/threads/T-019d6154-8fdf-74fe-b865-f796d8a4214a Co-authored-by: Amp <amp@ampcode.com>
generated by cgit v1.2.3 (git 2.25.1) at 2026-08-17 07:24:14 +0000