| Age | Commit message (Collapse) | Author |
|---|
|
Amp-Thread-ID: https://ampcode.com/threads/T-019fc408-6e41-73fe-826d-d47b0408981b
Co-authored-by: Amp <amp@ampcode.com>
|
|
Amp-Thread-ID: https://ampcode.com/threads/T-019fc408-6e41-73fe-826d-d47b0408981b
Co-authored-by: Amp <amp@ampcode.com>
|
|
f3s hosts don't get a distinct standby certificate/keypair; alias the
primary cert's files under the standby name (relayd selects a keypair
by SNI, so it needs matching filenames) and add standby.<host> as a
SAN on the primary cert. Every host now also gets www.<host> as a SAN
unconditionally, since public DNS publishes www for all of them.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Colliding with the nightly generation CronJob's flock at the top of
some hours caused an intermittent EBADF failure. Move sync to two
fixed daytime ticks (10:00/18:00 Europe/Sofia) clear of the 04:00
generation run, and replace the mtime-based "did a generation
complete" check with a per-site content diff (image_count/
total_size_bytes vs a backed-up status.json from the last publish),
so a tick with no real content change is a no-op instead of an
unconditional publish.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
|
Amp-Thread-ID: https://ampcode.com/threads/T-019fc2f1-f70e-74aa-8efe-650e1a1c3097
Co-authored-by: Amp <amp@ampcode.com>
|
|
Amp-Thread-ID: https://ampcode.com/threads/T-019fc2a1-15bb-7179-939e-39dd3bb10597
Co-authored-by: Amp <amp@ampcode.com>
|
|
Amp-Thread-ID: https://ampcode.com/threads/T-019fc2a1-15bb-7179-939e-39dd3bb10597
Co-authored-by: Amp <amp@ampcode.com>
|
|
|
|
Track the shuriken.sh 0.14.0 release: Chart appVersion, generation +
sync CronJob image, docker-image/Justfile TAG, and README references.
|
|
|
|
|
|
Amp-Thread-ID: https://ampcode.com/threads/T-019fc2a1-15bb-7179-939e-39dd3bb10597
Co-authored-by: Amp <amp@ampcode.com>
|
|
internal)
|
|
Rex's <%# ... %> only comments the first line; the multi-line comment
was parsed as Perl (bareword 'generation'/'1 week').
|
|
Add a custom gogios check that monitors shuriken album freshness via
dist/status.json's unix_epoch, read locally on each frontend (so it
covers the whole generation+sync pipeline). warn at 1 week, crit at 2
weeks; CRIT if status.json is missing (album not published). DependsOn
the site's HTTP checks.
The check_shuriken_age plugin (Perl, Nagios API) lives in the
shuriken.sh repo (contrib/); the frontends Rexfile gogios task installs
it to /usr/local/bin/ from the sibling shuriken.sh checkout (single
source of truth, no vendored copy).
|
|
The generation and sync CronJobs are separate, so each one's
concurrencyPolicy:Forbid only blocks itself, not the other. Add a
shared flock on /data/shuriken.sh/.lock (the NFS volume, auto-released
on pod death so a crash never leaves a stale lock):
- generation: blocking-acquire the lock around the whole multi-site
entrypoint run (waits for any in-progress sync, which is short).
- sync: non-blocking acquire; skip the tick if generation holds it.
flock is already in the image (util-linux), so no rebuild needed.
|
|
Per feedback: the shuriken-sync CronJob now uses the rsync daemon
protocol (rsync://) instead of SSH -- no key/Secret needed. The
frontends run rsyncd via inetd with hosts allow *.wg0; k3s pods run on
r-nodes with .wg0 (WireGuard) connectivity, so they're authorized to
push over the mesh. Add writable irregular-ninja and alt-irregular-ninja
modules to frontends/etc/rsyncd.conf.tpl (deploy with rex rsync).
It only publishes when a generation has COMPLETED since the last sync:
shuriken now deletes dist/status.json at the start of a run and writes
it last on success, so status.json presence+freshness vs a .last-sync
marker on NFS is the completed/not-yet-published signal. Most ticks
skip; a publish fires once after each daily generation. hostAliases pin
the .wg0 frontend names to mesh IPs (cluster DNS doesn't resolve *.wg0).
The nfs-check initContainer stays so rsync --delete can't wipe the live
site if NFS is down. shuriken --sync over SSH stays as a manual option.
|
|
Without an nfs-check initContainer, a run scheduled while NFS is
unmounted on the node would see an empty/stale /data source and
rsync --delete would wipe the live public site on fishfinger/blowfish.
Mirror the generation CronJob's sentinel guard (refuse to start if
/mnt/shuriken.sh/.nfs-sentinel is missing). Also run the publish
script under /bin/bash for reliable pipefail (ash's is version-gated).
|
|
Per request, decouple publishing from generation: add a second
shuriken-sync CronJob that rsyncs /data/shuriken.sh/<site>/dist to
admin@fishfinger.buetow.org and admin@blowfish.buetow.org (with
--delete) every 4h -- far more often than the daily generate. rsync
is incremental, so post-generation publishes are cheap and a failed
publish retries on the next tick without re-running generation.
Reuses the shuriken image (now with openssh-client). Requires a
shuriken-rsync-ssh-key Secret holding the admin key authorized on
fishfinger/blowfish; until it exists the publish pods fail to mount
the key and publish nothing (safe by design -- no live publish
without the key). README documents the one-time Secret creation and
the image rebuild/push for openssh.
|
|
The first run OOM-killed (exit 137) on irregular.ninja: Alpine's
ImageMagick is Q16-HDRI (32 bytes/pixel), so a 37 MP AI-upscaled PNG
decodes to ~1.2 GiB per buffer and -auto-orient doubles it -- well
over the 1 Gi pod limit. The image now ships an ImageMagick policy
(memory/map 512 MiB, disk 8 GiB) that spills oversized decodes to a
disk cache instead of OOMing. Match it here: bump the pod memory
limit to 2 Gi for the working set, pin OMP_NUM_THREADS=1 (the build
is OpenMP-enabled; extra resample threads only multiply HDRI memory
pressure), and add a disk-backed /tmp emptyDir (sizeLimit 8 Gi) for
the pixel-cache spill.
|
|
ArgoCD-managed CronJob that regenerates the irregular.ninja and
alt.irregular.ninja photo albums with the shuriken.sh Docker image,
writing them to /data/nfs/k3svolumes/shuriken.sh/<site>/dist on the
shared NFS export. Single image job by default (passively cooled N100
hosts). 04:00 Europe/Sofia daily, Forbid stacking, 6h deadline.
|
|
The NodePort is LAN-only, so the advertised clone URL
(ssh://git@r0.lan.buetow.org:30222/...) did not work from outside. Add a plain
TCP relay on the gateways forwarding 2022 to NodePort 30222, and have Forgejo
advertise code.f3s.buetow.org:2022.
Port 2022 rather than 22: the gateways run their own sshd on port 2, so 22 was
in fact free, but leaving the forge off the default port keeps it clear of the
mass scanning that port attracts. That is noise reduction and not security --
the real protection is that Forgejo's SSH does key-only auth for git operations
and offers no shell. 2222 would have been the conventional choice but dserver
(DTail) already holds it on the gateways; verified 2022 free on both.
The relay carries no "protocol" line, so relayd forwards the stream untouched.
TLS must not be involved: SSH secures its own transport and the client verifies
Forgejo's host key at the far end. A dedicated <forgejo_ssh> table keeps the
health check on the SSH port, since the web UI can be up while SSH is not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
Review of the initial chart turned up one blocker and several correctness gaps:
- `just create-admin` could never work. `admin user create` does not prompt; it
exits with "must set either password or random-password flag". With the
installer locked and registration disabled that recipe was the only way into
the instance, so first login was impossible. Use --random-password, which
prints the generated password once, rather than passing one on the command
line where it would reach shell history and ps.
- SSH_DOMAIN advertised code.f3s.buetow.org, but that name resolves to the
OpenBSD gateways (443 only) and *.f3s.lan points at the CARP storage VIP, not
a k3s node. Every clone URL Forgejo rendered would have been unreachable.
Advertise r0.lan.buetow.org, which actually answers on the NodePort.
- No REVERSE_PROXY_TRUSTED_PROXIES, so behind relayd -> Traefik every request
was attributed to the Traefik pod IP: real client IPs absent from the audit
trail and per-IP rate limiting defeated, on an internet-facing instance.
- Pin both PVCs with volumeName. The 1Gi config claim also matched the 20Gi
data PV, leaving the choice to the binder's smallest-sufficient heuristic.
README: add the gogios task to the frontends deploy (the new host otherwise
gets no TLS/HTTP checks); drop the false claim that pushing alone deploys the
Application, since nothing watches f3s/argocd-apps/; keep .nfs-sentinel at 0644
instead of sweeping it into a recursive 0750; note that the real certificate is
only issued on the gateway holding the DNS master IP.
Still not activated. Re-verified: helm template renders, and all 9 objects pass
kubectl apply --dry-run=server.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
Deliberately independent of the cgit git-server: separate namespace (services),
separate NFS volumes, separate SSH NodePort (30222 vs 30022), and no shared
storage. cgit keeps serving the existing 80 bare repos at c-git.f3s.buetow.org
and is not touched. Forgejo starts empty; repos get migrated by hand later.
ArgoCD deliberately keeps reading conf.git from the existing git-server, so
Forgejo has no consumers and cannot take cluster deploys down with it.
SQLite rather than a PostgreSQL pod: single writer (replicas 1 + Recreate) and
NFSv4.2 does real byte-range locking, so the usual SQLite-on-NFS failure mode
does not apply. Uses the -rootless image so the pod runs wholly as UID 1000
with all capabilities dropped, and both volumes carry the .nfs-sentinel guard.
The installer is locked and registration disabled because the instance is
reachable from the internet; the admin account is created via the CLI.
code.f3s.buetow.org added to @f3s_hosts, which drives the DNS zone, the relayd
route, the ACME cert and the gogios checks.
Not yet activated: the ArgoCD Application still needs applying, the NFS
directories creating, and the frontends deploying. See f3s/forgejo/README.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
The three cache-control response rules were unscoped, so relayd rewrote every
response passing through the https relay -- not just the fallback pages the
comment described. Everything the k3s cluster serves was forced to
"no-cache, no-store, must-revalidate", overriding whatever the backend set.
cgit, for example, marks its CSS and logo "expires 30d", but browsers
re-fetched them on every page view.
relayd cannot filter a response by the backend table that produced it, and a
"header set" cannot be combined with a header match in the same rule. So match
the Server header -- the local httpd is the only backend answering
"OpenBSD httpd" -- and carry that across with a sticky tag.
Verified on blowfish before rollout with an isolated relayd instance: a
response from httpd:8080 gets the tagged headers, one from the cluster's nginx
does not. Fallback behaviour retested after deploy: c-git.f3s.buetow.org
served from httpd:8080 still returns the "Server turned off" page on any path,
with all three headers intact. cgit.css now passes through max-age=2592000.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
cgit ran with cache-size=0 against scan-path over ~80 repos on NFS, so every
request reopened all of them to resolve the index age column. Measured 376ms
per index render, of which only ~50ms was CPU -- the rest was NFS wait. With
the cache enabled the same render takes ~19-29ms.
cache-root points at the existing node-local cgit-runtime emptyDir rather than
the NFS-backed /repos; caching onto NFS is what the old "permission issues"
comment was actually about. The startup script creates the directory since
cgit does not.
fcgiwrap ran as a single worker, so the 26 ArgoCD apps polling conf.git through
git-http-backend serialized against web UI renders; 4 concurrent requests
managed only ~1.5x the throughput of one. Preforking 4 workers separates them.
The cgit container also idled at 105m against a 250m limit and had accumulated
~2.9h of throttled time (nr_throttled 96979), so raise the ceiling and set the
request from measured idle usage.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
|
|
|
|
|
|
The quarterly backup script travels on the pool; its S3 sync needs
awscli + /root/.aws/credentials symlink to /opt/snonux/secrets/aws.credentials
on the hosting f-host. Snapshot export still works without it; only the
offsite S3 sync is skipped.
|
|
|
|
add startupProbe
- Run bridge via 'protonmail-bridge --cli' with stdin/tty enabled so the
CLI is attachable (kubectl attach) for first-time account login instead
of a separate --cli process against the gRPC daemon.
- Install libfido2-1 at container start (runtime dependency).
- Persist /root/.local (Bridge's self-updated binary) on the data PVC.
- Add a startupProbe (tcp 1143, 30x10s) so slow first starts aren't killed
by the livenessProbe; update the login/setup comments accordingly.
|
|
4-disk raidz2 USB-SATA pool used as offline backup storage, loaded ~once
per quarter. Migrated from t450 to the f3s USB-key scheme: zusb/data/enc
rekeyed from passphrase to a raw key on the F3S_KEYS sticks at
/keys/zusb.key. Scripts + key deployed to all f-hosts (f0/f1/f2/f3) so
the disk stack can be re-plugged to any host. Not auto-mounted.
|
|
Bump image.tag v2.7.5 -> v3.0.3 for both immich-server and immich-machine-learning.
Stage A already migrated Postgres to VectorChord (vchord 0.4.3), which 3.x requires
(pgvecto.rs dropped); the postgres bundle image preloads vchord.so + vectors.so per
the 3.0.1 requirement. Pre-upgrade ZFS snapshot: zdata/enc/nfsdata@immich-pre-3x-20260718 (f0).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|
Swap the hand-rolled immich-postgres image from tensorchord/pgvecto-rs:pg16-v0.3.0
to the official Immich bundle ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvector0.8.0-pgvectors0.3.0.
This makes vchord.so available (still preloading vectors.so) so Immich can migrate
smart-search/face vectors off the deprecated pgvecto.rs extension while still on the
2.x server, ahead of the 3.x bump (which drops pgvecto.rs entirely). Tag pins pgvecto.rs
0.3.0 to match the installed 'vectors' 0.3.0 catalog. Pre-upgrade ZFS snapshot taken:
zdata/enc/nfsdata@immich-pre-3x-20260718 on f0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
|
|
|
The beets-art CronJob was hard-pinned to r1 via nodeSelector, and its
CPU limit was 2 (half the 4-core passively-cooled Beelink N100 on f1).
The daily noon fetchart/embedart sweep sustained ~35% CPU for ~28 min
and pushed f1 to ~95 C, tripping the FreebsdCpuTemperatureHigh alert
(>=80 C for 5m).
Two changes:
1. Cap the container CPU limit from 2 to 1. The run takes ~2x longer but
stays well under the 6h activeDeadlineSeconds and roughly halves the
sustained thermal load.
2. Unpin the CronJob from r1 so the scheduler can spread it. This is a
manifest/config change, not a storage migration: the music library is
already shared NFS (mounted at the same path on all three r-nodes).
The only thing forcing r1 was the music PV/PVC being declared RWO
(single-node mount) while shared with Navidrome, which is itself pinned
to r1 by its local-path SQLite data PVC. Flip navidrome-music PV/PVC
to ReadWriteMany so beets-art can mount it on any r-node alongside the
read-only Navidrome pod.
Also move beets-art-state from local-path (pinned to r1) to a static
NFS-backed RWX PV/PVC so the job's SQLite state is reachable from any
node. SQLite-over-NFS is acceptable here because concurrencyPolicy:
Forbid guarantees a single writer (no lock contention), and the state
is regenerable.
Application requires recreating the bound music PVC (access-mode changes
are immutable on a bound PVC) and the state PVC; see deploy notes.
|
|
- Makefile: new check-dtail-version target guards dtail-openbsd,
dtail-freebsd, and the build-VM stamp recipe — a wrong DTAIL_SRC now
aborts with a clear error before any VM start, /tmp cleanup, or
remote work runs with an empty version in the file names (dtail-netbsd
already had an equivalent inline guard)
- Makefile: dtail-openbsd pre-cleans its stale staging tgz like the
netbsd target, so a failed run can never silently upload an old
package
- pkg-dtail-openbsd.sh / pkg-dtail-freebsd.sh: reject a missing version
argument like pkg-dtail-netbsd.sh (would otherwise build dtail-.tgz)
Verified: make -n renders the guard first for both targets;
make dtail-{openbsd,freebsd} DTAIL_SRC=/nonexistent fails fast with the
version error before any rm/remote step. Guards only change failure
paths — no package rebuild needed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
- dtail-freebsd.json.tpl: move HostKeyFile from volatile
/var/run/dserver/cache to persistent /var/db/dserver/ssh_host_key —
FreeBSD's cleanvar purges /var/run at boot, so the host key was
regenerated on every reboot (mirrors the NetBSD/OpenBSD templates)
- dserver-freebsd.tpl rc.d: start_precmd now creates /var/db/dserver
(0700, dserver-owned) and re-runs dserver-update-key-cache.sh on
every start, so the volatile key cache repopulates right after a
reboot or restart instead of waiting for the daily periodic job
- packages/Makefile: .SHELLFLAGS gains -o pipefail — the
"git archive | ssh ... tar" pipeline could mask a git archive
failure as long as tar succeeded on the truncated stream (the only
recipe-level pipeline; $(shell ...) calls are unaffected)
- packages/Makefile: OS-suffixed /tmp staging names for the OpenBSD
and NetBSD dtail tgz (dtail-openbsd-*/dtail-netbsd-*) — a dash-less
version would have made both targets stage to the same
/tmp/dtail-<version>.tgz locally and on f0; the final repo copy
keeps the canonical dtail-<version>.tgz name
- dserver-update-key-cache.sh.tpl (OpenBSD): port the NetBSD
hardening — quote all variable expansions, derive the user via
basename suffix stripping instead of cut -d. -f1 (dotted usernames
broke), fix the obsolete-cachefile echo that used single quotes and
never interpolated $cachefile, add a header comment documenting the
rc_pre and /etc/daily.local call sites
- dserver.tpl (OpenBSD rc.d): replace 'rc_cmd $1 &' with rc_bg=YES and
a plain rc_cmd "$1" — the daemon needs backgrounding because it does
not daemonize, but backgrounding the whole rc framework made rc_pre
failures and the start result invisible to rcctl; rc_bg is rc.subr's
supported way to background only the daemon
- dserver-update-key-cache-freebsd.sh.tpl: header now documents the
new rc.d start_precmd call site
Deployed: FreeBSD pkg republished, f0/f1/f2 upgraded with host key
migrated to /var/db/dserver first (sha256 verified identical); f3
unreachable, still on the old package. OpenBSD pkg republished,
fishfinger reinstalled + restarted, host key unchanged, dcat verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
install -d applies -o/-m only to the final directory, so the previous
single 'install -d -o _dserver /var/run/dserver/cache' left the parent
/var/run/dserver (the _dserver home dir) implicitly created as
root:wheel after a reboot. Create the parent explicitly with the right
ownership, matching the NetBSD template (dserver-netbsd.tpl).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
The OpenBSD dserver template used relative CacheDir "cache" and
HostKeyFile "cache/ssh_host_key", which only worked because rc.d starts
the daemon via su -l _dserver (CWD = /var/run/dserver). Any start from
another directory broke public key lookup — same bug class dtail commit
fec2f9d fixed on the server side. Switch to absolute paths like the
FreeBSD/NetBSD templates.
OpenBSD /etc/rc wipes /var/run/* at boot, so move the SSH host key to
persistent /var/db/dserver/ssh_host_key (mirrors the NetBSD template) —
a regenerated host key would break clients' known_hosts. The rc.d
rc_pre now also creates /var/db/dserver and re-runs the key-cache
helper on every service start, so the volatile cache is repopulated
right after a reboot instead of waiting for the daily cron job.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
- Makefile: .SHELLFLAGS := -ec — with .ONESHELL a failing intermediate
recipe line (remote pkg_create, scp, PV upload) previously did not
stop the recipe and make exited 0; also remove stale /tmp package
artifacts at target start so a failed build can never silently
re-upload an old tgz, guard against an empty extracted version, and
quote the tr operand
- dtail-netbsd.json.tpl + rc.d: move the SSH host key from volatile
/var/run/dserver/cache to persistent /var/db/dserver/ssh_host_key so
it survives reboots (clients re-accept the key once after upgrade)
- pkg-dtail-netbsd.sh: reject a missing version argument; write the
pkg_summary via a temp file so a pkg_info failure cannot publish a
truncated pkg_summary.gz; document that the dserver user/group is a
deliberate manual install step
- key-cache template: comment now states the daily cron entry is added
manually at install time (documented in the pkgrepo skill)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
- Makefile target dtail-netbsd: cross-compile netbsd/arm64 (CGO_ENABLED=0,
nozstd), package natively on pi0 with pkg_create, upload package plus
pkg_summary.gz to the PV at netbsd/10.1/packages/aarch64/ via f0
- scripts/pkg-dtail-netbsd.sh: NetBSD pkg_create packaging (prefix /,
root-owned files via @owner/@group plist directives)
- frontends: NetBSD dserver rc.d script, dtail.json (absolute cache
paths), and key-cache helper templates
- pkgrepo nginx: serve /netbsd/ with autoindex
- Fix stale ROCKY_ARM_BUILD_HOST: pi0 was re-imaged to NetBSD, aarch64
RPMs are built on pi2
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
|
|
|
|
|
|
|
|
|
Fixes the crash a vulnerability scanner triggered twice against the
live pod (std::out_of_range from a broken string::npos overflow check
in the GET/POST request-line parser).
|
|
|
|
|
