| Age | Commit message (Collapse) | Author |
|---|
|
The quarterly backup script travels on the pool; its S3 sync needs
awscli + /root/.aws/credentials symlink to /opt/snonux/secrets/aws.credentials
on the hosting f-host. Snapshot export still works without it; only the
offsite S3 sync is skipped.
|
|
|
|
add startupProbe
- Run bridge via 'protonmail-bridge --cli' with stdin/tty enabled so the
CLI is attachable (kubectl attach) for first-time account login instead
of a separate --cli process against the gRPC daemon.
- Install libfido2-1 at container start (runtime dependency).
- Persist /root/.local (Bridge's self-updated binary) on the data PVC.
- Add a startupProbe (tcp 1143, 30x10s) so slow first starts aren't killed
by the livenessProbe; update the login/setup comments accordingly.
|
|
4-disk raidz2 USB-SATA pool used as offline backup storage, loaded ~once
per quarter. Migrated from t450 to the f3s USB-key scheme: zusb/data/enc
rekeyed from passphrase to a raw key on the F3S_KEYS sticks at
/keys/zusb.key. Scripts + key deployed to all f-hosts (f0/f1/f2/f3) so
the disk stack can be re-plugged to any host. Not auto-mounted.
|
|
Bump image.tag v2.7.5 -> v3.0.3 for both immich-server and immich-machine-learning.
Stage A already migrated Postgres to VectorChord (vchord 0.4.3), which 3.x requires
(pgvecto.rs dropped); the postgres bundle image preloads vchord.so + vectors.so per
the 3.0.1 requirement. Pre-upgrade ZFS snapshot: zdata/enc/nfsdata@immich-pre-3x-20260718 (f0).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|
Swap the hand-rolled immich-postgres image from tensorchord/pgvecto-rs:pg16-v0.3.0
to the official Immich bundle ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvector0.8.0-pgvectors0.3.0.
This makes vchord.so available (still preloading vectors.so) so Immich can migrate
smart-search/face vectors off the deprecated pgvecto.rs extension while still on the
2.x server, ahead of the 3.x bump (which drops pgvecto.rs entirely). Tag pins pgvecto.rs
0.3.0 to match the installed 'vectors' 0.3.0 catalog. Pre-upgrade ZFS snapshot taken:
zdata/enc/nfsdata@immich-pre-3x-20260718 on f0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|
|
|
The beets-art CronJob was hard-pinned to r1 via nodeSelector, and its
CPU limit was 2 (half the 4-core passively-cooled Beelink N100 on f1).
The daily noon fetchart/embedart sweep sustained ~35% CPU for ~28 min
and pushed f1 to ~95 C, tripping the FreebsdCpuTemperatureHigh alert
(>=80 C for 5m).
Two changes:
1. Cap the container CPU limit from 2 to 1. The run takes ~2x longer but
stays well under the 6h activeDeadlineSeconds and roughly halves the
sustained thermal load.
2. Unpin the CronJob from r1 so the scheduler can spread it. This is a
manifest/config change, not a storage migration: the music library is
already shared NFS (mounted at the same path on all three r-nodes).
The only thing forcing r1 was the music PV/PVC being declared RWO
(single-node mount) while shared with Navidrome, which is itself pinned
to r1 by its local-path SQLite data PVC. Flip navidrome-music PV/PVC
to ReadWriteMany so beets-art can mount it on any r-node alongside the
read-only Navidrome pod.
Also move beets-art-state from local-path (pinned to r1) to a static
NFS-backed RWX PV/PVC so the job's SQLite state is reachable from any
node. SQLite-over-NFS is acceptable here because concurrencyPolicy:
Forbid guarantees a single writer (no lock contention), and the state
is regenerable.
Application requires recreating the bound music PVC (access-mode changes
are immutable on a bound PVC) and the state PVC; see deploy notes.
|
|
- Makefile: new check-dtail-version target guards dtail-openbsd,
dtail-freebsd, and the build-VM stamp recipe — a wrong DTAIL_SRC now
aborts with a clear error before any VM start, /tmp cleanup, or
remote work runs with an empty version in the file names (dtail-netbsd
already had an equivalent inline guard)
- Makefile: dtail-openbsd pre-cleans its stale staging tgz like the
netbsd target, so a failed run can never silently upload an old
package
- pkg-dtail-openbsd.sh / pkg-dtail-freebsd.sh: reject a missing version
argument like pkg-dtail-netbsd.sh (would otherwise build dtail-.tgz)
Verified: make -n renders the guard first for both targets;
make dtail-{openbsd,freebsd} DTAIL_SRC=/nonexistent fails fast with the
version error before any rm/remote step. Guards only change failure
paths — no package rebuild needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
- dtail-freebsd.json.tpl: move HostKeyFile from volatile
/var/run/dserver/cache to persistent /var/db/dserver/ssh_host_key —
FreeBSD's cleanvar purges /var/run at boot, so the host key was
regenerated on every reboot (mirrors the NetBSD/OpenBSD templates)
- dserver-freebsd.tpl rc.d: start_precmd now creates /var/db/dserver
(0700, dserver-owned) and re-runs dserver-update-key-cache.sh on
every start, so the volatile key cache repopulates right after a
reboot or restart instead of waiting for the daily periodic job
- packages/Makefile: .SHELLFLAGS gains -o pipefail — the
"git archive | ssh ... tar" pipeline could mask a git archive
failure as long as tar succeeded on the truncated stream (the only
recipe-level pipeline; $(shell ...) calls are unaffected)
- packages/Makefile: OS-suffixed /tmp staging names for the OpenBSD
and NetBSD dtail tgz (dtail-openbsd-*/dtail-netbsd-*) — a dash-less
version would have made both targets stage to the same
/tmp/dtail-<version>.tgz locally and on f0; the final repo copy
keeps the canonical dtail-<version>.tgz name
- dserver-update-key-cache.sh.tpl (OpenBSD): port the NetBSD
hardening — quote all variable expansions, derive the user via
basename suffix stripping instead of cut -d. -f1 (dotted usernames
broke), fix the obsolete-cachefile echo that used single quotes and
never interpolated $cachefile, add a header comment documenting the
rc_pre and /etc/daily.local call sites
- dserver.tpl (OpenBSD rc.d): replace 'rc_cmd $1 &' with rc_bg=YES and
a plain rc_cmd "$1" — the daemon needs backgrounding because it does
not daemonize, but backgrounding the whole rc framework made rc_pre
failures and the start result invisible to rcctl; rc_bg is rc.subr's
supported way to background only the daemon
- dserver-update-key-cache-freebsd.sh.tpl: header now documents the
new rc.d start_precmd call site
Deployed: FreeBSD pkg republished, f0/f1/f2 upgraded with host key
migrated to /var/db/dserver first (sha256 verified identical); f3
unreachable, still on the old package. OpenBSD pkg republished,
fishfinger reinstalled + restarted, host key unchanged, dcat verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
install -d applies -o/-m only to the final directory, so the previous
single 'install -d -o _dserver /var/run/dserver/cache' left the parent
/var/run/dserver (the _dserver home dir) implicitly created as
root:wheel after a reboot. Create the parent explicitly with the right
ownership, matching the NetBSD template (dserver-netbsd.tpl).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
The OpenBSD dserver template used relative CacheDir "cache" and
HostKeyFile "cache/ssh_host_key", which only worked because rc.d starts
the daemon via su -l _dserver (CWD = /var/run/dserver). Any start from
another directory broke public key lookup — same bug class dtail commit
fec2f9d fixed on the server side. Switch to absolute paths like the
FreeBSD/NetBSD templates.
OpenBSD /etc/rc wipes /var/run/* at boot, so move the SSH host key to
persistent /var/db/dserver/ssh_host_key (mirrors the NetBSD template) —
a regenerated host key would break clients' known_hosts. The rc.d
rc_pre now also creates /var/db/dserver and re-runs the key-cache
helper on every service start, so the volatile cache is repopulated
right after a reboot instead of waiting for the daily cron job.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
- Makefile: .SHELLFLAGS := -ec — with .ONESHELL a failing intermediate
recipe line (remote pkg_create, scp, PV upload) previously did not
stop the recipe and make exited 0; also remove stale /tmp package
artifacts at target start so a failed build can never silently
re-upload an old tgz, guard against an empty extracted version, and
quote the tr operand
- dtail-netbsd.json.tpl + rc.d: move the SSH host key from volatile
/var/run/dserver/cache to persistent /var/db/dserver/ssh_host_key so
it survives reboots (clients re-accept the key once after upgrade)
- pkg-dtail-netbsd.sh: reject a missing version argument; write the
pkg_summary via a temp file so a pkg_info failure cannot publish a
truncated pkg_summary.gz; document that the dserver user/group is a
deliberate manual install step
- key-cache template: comment now states the daily cron entry is added
manually at install time (documented in the pkgrepo skill)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
- Makefile target dtail-netbsd: cross-compile netbsd/arm64 (CGO_ENABLED=0,
nozstd), package natively on pi0 with pkg_create, upload package plus
pkg_summary.gz to the PV at netbsd/10.1/packages/aarch64/ via f0
- scripts/pkg-dtail-netbsd.sh: NetBSD pkg_create packaging (prefix /,
root-owned files via @owner/@group plist directives)
- frontends: NetBSD dserver rc.d script, dtail.json (absolute cache
paths), and key-cache helper templates
- pkgrepo nginx: serve /netbsd/ with autoindex
- Fix stale ROCKY_ARM_BUILD_HOST: pi0 was re-imaged to NetBSD, aarch64
RPMs are built on pi2
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
|
|
|
|
|
|
|
|
Fixes the crash a vulnerability scanner triggered twice against the
live pod (std::out_of_range from a broken string::npos overflow check
in the GET/POST request-line parser).
|
|
|
|
|
|
Replaces the frameset-based chat layout with iframes + flexbox and adds
viewport meta tags across templates, so the chat is usable on phones.
|
|
Image 67babb2 used PRAGMA journal_mode=WAL, which silently lost writes
on the NFS-backed hostPath PV (WAL needs shared-memory mmap of a -shm
file that network filesystems don't support reliably) -- a registered
user could not log back in because the INSERT never actually landed.
b8faffb switches to the default rollback journal, which only needs
ordinary byte-range locks that NFS handles correctly. The stale,
partially-corrupted ychat.db/-wal/-shm files on the PV were wiped
before this rollout so the pod starts from a clean, empty database.
|
|
ychat now requires SQLite (no more no-DB build), so registered accounts
need to survive pod restarts. Adds a hostPath-backed PV/PVC for
/app/data, an NFS sentinel initContainer (same pattern as other f3s
charts), and points the image at the SQLite-mandatory build.
|
|
|
|
Adds ensure-wizard-completed initContainer after nfs-check-config. If
jellyfin.db already exists but IsStartupWizardCompleted flipped to false
(interrupted wizard run or partial config restore), restore it to true so a
working instance never drops back into the setup wizard. Fresh installs (no
jellyfin.db) keep their legitimate first-run wizard. Runs after the sentinel
check so config is only rewritten on the real NFS volume.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
fixes)
|
|
|
|
|
|
|
|
Add shellyfans rc.d service (boot-time auto-on for the rack fans on the
Shelly Plug M Gen 3 at 192.168.1.28) and a standalone shelly-plug.sh
helper to query/control the plug over its HTTP RPC API.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|
relayd on the CARP VIP no longer terminates TLS — it forwards raw TLS to
Traefik, which terminates via the f3s-lan-tls ingress secret. The old
"export cert to relayd" steps were stale and misleading. Add the
restart-not-reload pitfall (a stale relayd process serving an expired
cached keypair caused the 2026-06-30 LAN-cert outage).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|
|
|
|
|
|
|
In-memory guest chat, no DB. Deployment pulls
registry.lan.buetow.org:30001/ychat:f387bac, port 2000, non-root (UID 1000),
readOnlyRootFilesystem with emptyDir on /app/log. TCP-socket probes (ychat
speaks HTTP/0.9-style responses). Ingress: ychat.f3s.buetow.org +
ychat.f3s.lan.buetow.org.
|
|
After a CARP failover or f-host reboot the node host remounts NFS
cleanly but already-running pods keep their old stale bind-mount,
causing writes to return ESTALE (errno 116). Three changes to auto-heal:
1. immich-valkey: replace ping-only liveness with a write-probe
(valkey-cli ping + echo to /data/.nfs-write-probe), so a stale
/data mount fails the probe and kubelet restarts the container with
a fresh bind-mount.
2. immich-server: add label nfs.stale-restart/enabled=true so the
node-level nfs-mount-monitor can exec a write-probe into the pod
and force-delete it if writes fail. Cannot do this via Helm values —
the upstream chart hardcodes an httpGet liveness and its Sprig merge
refills httpGet even when set to null, producing an invalid
dual-handler probe rejected by the k8s API.
3. nfs-mount-monitor/check-nfs-mount.sh: add reap_stale_nfs_pods()
function (throttled to every 30 s) that handles two cases invisible
to the existing fix_mount probe — Case 1: pods stuck in
CreateContainerConfigError with a stale subPath message (git-server,
prometheus hit this); Case 2: opt-in Running+Ready pods whose
in-container write to the NFS volume fails twice in a row (the
immich-server pattern). Deployed to r0/r1/r2 via Rex.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
