summaryrefslogtreecommitdiff
path: root/prompts/skills/f3s-pkgrepo/references
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2026-07-18 22:08:10 +0300
committerPaul Buetow <paul@buetow.org>2026-07-18 22:08:10 +0300
commitefbb7876347bc667bdb3633f72a22a51b3bfbc49 (patch)
tree0a2d1d9367f4bf549584aaa71b0b82337cab2226 /prompts/skills/f3s-pkgrepo/references
parentfbf9d4815ea25bd85be0f232ac8670bda1a64955 (diff)
skills: rename pkgrepo -> f3s-pkgrepo and rocky-vm-setup -> f3s-rocky-vm-setup
Extend the f3s- prefix to the two remaining f3s-scoped siblings so the whole homelab family groups together. Both are f3s-specific: pkgrepo IS pkgrepo.f3s.buetow.org, and rocky-vm-setup documents the 'rocky' VM on the f3s host f3. - git mv both skill dirs (history preserved); update name: frontmatter; add the 'Part of the f3s homelab skill family' note to both descriptions. - Rewrite every reference to the *skills*: backtick mentions (`pkgrepo` -> `f3s-pkgrepo`, `rocky-vm-setup` -> `f3s-rocky-vm-setup`), markdown cross-links (../pkgrepo/ -> ../f3s-pkgrepo/, etc.), the f3s hub Related-skills block and description, and skill-maintenance's index-pattern example. - Left references to the package REPOSITORY (pkgrepo.f3s.buetow.org, 'the custom pkgrepo') unchanged — those name a service, not the skill. - Verified all 114 relative links resolve. Plan doc updated to record the rename. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diffstat (limited to 'prompts/skills/f3s-pkgrepo/references')
-rw-r--r--prompts/skills/f3s-pkgrepo/references/client-setup.md141
-rw-r--r--prompts/skills/f3s-pkgrepo/references/dtail-package.md219
-rw-r--r--prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md42
-rw-r--r--prompts/skills/f3s-pkgrepo/references/packaging-workflow.md120
-rw-r--r--prompts/skills/f3s-pkgrepo/references/repo-architecture.md80
5 files changed, 602 insertions, 0 deletions
diff --git a/prompts/skills/f3s-pkgrepo/references/client-setup.md b/prompts/skills/f3s-pkgrepo/references/client-setup.md
new file mode 100644
index 0000000..1e29c4e
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/references/client-setup.md
@@ -0,0 +1,141 @@
+# Client Setup
+
+How to configure each OS to install packages from `pkgrepo.f3s.buetow.org`.
+
+## FreeBSD (f0–f3)
+
+Custom repo is configured alongside the official FreeBSD repos.
+
+File: `/usr/local/etc/pkg/repos/custom.conf`
+
+```
+custom: {
+ url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
+ mirror_type: "NONE",
+ signature_type: "NONE",
+ enabled: yes
+}
+```
+
+### Using packages
+
+```sh
+doas pkg update
+doas pkg install <package-name>
+doas pkg upgrade # upgrade all, including custom packages
+doas pkg install -fy <package> # force reinstall (same version)
+```
+
+### Setting up a new FreeBSD host
+
+Pipe via stdin to avoid csh quoting issues:
+
+```sh
+cat <<'REPO' | ssh -p 22 <host>.lan.buetow.org 'doas mkdir -p /usr/local/etc/pkg/repos && doas tee /usr/local/etc/pkg/repos/custom.conf > /dev/null'
+custom: {
+ url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
+ mirror_type: "NONE",
+ signature_type: "NONE",
+ enabled: yes
+}
+REPO
+```
+
+## OpenBSD (blowfish, fishfinger)
+
+Custom repo is configured via `PKG_PATH` in `/root/.profile`, deployed by `rex pkgrepo_setup`.
+Official OpenBSD packages still install normally via `/etc/installurl`.
+
+```sh
+export PKG_PATH="https://pkgrepo.f3s.buetow.org/openbsd/7.8/packages/amd64/"
+```
+
+### Using packages
+
+```sh
+doas pkg_add <package> # installs from custom repo (signed with signify)
+doas pkg_add -u <package> # update to latest version
+```
+
+### Setting up a new OpenBSD host
+
+Two things needed:
+
+1. **Signify public key** — copy from an existing host:
+ ```sh
+ scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub /tmp/
+ scp /tmp/custom-pkg.pub rex@<newhost>:/tmp/
+ ssh rex@<newhost> "doas cp /tmp/custom-pkg.pub /etc/signify/custom-pkg.pub"
+ ```
+
+2. **PKG_PATH** — run the Rex task from `~/git/conf/frontends`:
+ ```sh
+ rex pkgrepo_setup # adds PKG_PATH to /root/.profile on all frontends
+ ```
+
+Update `PKG_PATH` whenever the OpenBSD version changes (currently 7.8).
+
+### Package signing
+
+OpenBSD packages are signed with `signify(1)` via `pkg_sign`:
+- **Private key**: `/etc/signify/custom-pkg.sec` on fishfinger (build host)
+- **Public key**: `/etc/signify/custom-pkg.pub` on all OpenBSD clients
+- Signing happens automatically during `make pkg-openbsd` / `make pkg`
+- `pkg_add` verifies the signature — no `-D unsigned` needed
+
+## NetBSD (pi0, pi1)
+
+No persistent repo config needed — install/update straight from the package URL (repo is unsigned):
+
+```sh
+export PATH=/usr/sbin:$PATH # pkg_* live in /usr/sbin, not in the default non-interactive PATH
+doas pkg_add https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz # first install
+doas pkg_add -u https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz # update to newer version
+```
+
+Notes:
+- Same-version reinstall: `pkg_add -u` skips it — `doas pkg_delete dtail` first, then `pkg_add` again
+- `pkg_summary.gz` is published alongside the packages, so the repo URL could also be added to `/usr/pkg/etc/pkgin/repositories.conf` for pkgin; today plain `pkg_add` is used
+- Version in the URL path must match the host OS release (currently 10.1) and the arch is `aarch64` (`uname -p`)
+- pkgsrc packages from cdn.netbsd.org still install normally via pkgin — the custom repo lives under a different prefix (`/` with files in `/usr/local`, `/etc`) and doesn't conflict
+
+## Rocky Linux (r0–r2, pi2–pi3)
+
+`pi0`/`pi1` run NetBSD (see above and the `f3s-raspberry-pi` skill's `bootstrap-netbsd-pi.md`) and are not Rocky clients of this repo.
+
+Architecture-specific repo URLs:
+- `https://pkgrepo.f3s.buetow.org/rockylinux/9/x86_64/` (r0–r2)
+- `https://pkgrepo.f3s.buetow.org/rockylinux/9/aarch64/` (pi2–pi3)
+
+### Persistent repo file
+
+Create `/etc/yum.repos.d/f3s-dtail.repo`:
+
+```ini
+[f3s-dtail]
+name=f3s DTail
+baseurl=https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/
+enabled=1
+gpgcheck=0
+repo_gpgcheck=0
+```
+
+Then:
+
+```sh
+sudo dnf makecache
+sudo dnf install dtail
+sudo dnf upgrade dtail
+```
+
+### Temporary one-off usage (no persistent repo file needed)
+
+```sh
+sudo dnf repoquery \
+ --disablerepo='*' \
+ --repofrompath=f3s-dtail,https://pkgrepo.f3s.buetow.org/rockylinux/9/$(uname -m)/ \
+ --enablerepo=f3s-dtail \
+ dtail
+```
+
+Do **not** pass `--repofrompath` when the persistent repo file already exists — dnf errors with "listed more than once".
diff --git a/prompts/skills/f3s-pkgrepo/references/dtail-package.md b/prompts/skills/f3s-pkgrepo/references/dtail-package.md
new file mode 100644
index 0000000..fbdd8d6
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/references/dtail-package.md
@@ -0,0 +1,219 @@
+# DTail Package
+
+DTail is a multi-binary package (6 binaries + config + service script). There are separate targets for each OS.
+
+## Build Commands
+
+```sh
+cd ~/git/conf/packages
+make dtail-openbsd # OpenBSD: native build on QEMU/KVM VM (CGo/zstd supported)
+make dtail-freebsd # FreeBSD: cross-compiled on Linux (CGO_ENABLED=0, nozstd — .zst logs unsupported)
+make dtail-netbsd # NetBSD/aarch64: cross-compiled on Linux (CGO_ENABLED=0, nozstd), packaged natively on pi0
+make dtail-rocky # Rocky Linux: x86_64 + aarch64 RPMs + repodata
+```
+
+## Package Contents by OS
+
+### OpenBSD (blowfish, fishfinger)
+
+| File | Source template |
+|------|----------------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | built natively on build VM |
+| `/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail.json.tpl` (absolute `CacheDir: /var/run/dserver/cache`; `HostKeyFile` in persistent `/var/db/dserver/ssh_host_key`) |
+| `/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver.tpl` |
+| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache.sh.tpl` (ksh) |
+
+OpenBSD notes:
+- Packages before 2026-07-10 used relative `CacheDir: "cache"` / `HostKeyFile: "cache/ssh_host_key"` — that only worked because rc.d starts the daemon via `su -l _dserver` (CWD = `/var/run/dserver`, the `_dserver` home dir); a manual start from any other directory broke public key lookup — the same bug class dtail commit `fec2f9d` fixed on the server side (making absolute `CacheDir` paths resolve independently of dserver's CWD)
+- OpenBSD's `/etc/rc` wipes `/var/run/*` at boot, so the SSH host key now lives in persistent `/var/db/dserver/ssh_host_key` (mirrors NetBSD). The rc.d `rc_pre` recreates `/var/run/dserver/cache` and `/var/db/dserver` and re-runs `dserver-update-key-cache.sh` on every start; the daily cron entry keeps it fresh afterwards
+- When upgrading a host from a pre-2026-07-10 package, follow this order — neither `pkg_add` nor Rex restarts a running daemon, and once the host reboots `/etc/rc` has already wiped the old host key, making migration impossible:
+ 1. Migrate the host key FIRST to preserve the host identity: `doas install -d -o _dserver -m 0700 /var/db/dserver && doas cp -p /var/run/dserver/cache/ssh_host_key /var/db/dserver/` — otherwise dserver generates a new host key and clients without `--trustAllHosts` must re-accept it
+ 2. Reinstall the package (same-version: `pkg_add -u` is a no-op — `doas pkg_delete dtail` then `doas env PKG_PATH=... pkg_add dtail`)
+ 3. `doas rcctl restart dserver` — without this the running daemon keeps the old relative-path config until reboot
+ (done on fishfinger 2026-07-10; **blowfish still runs the pre-2026-07-10 package**)
+- Since 2026-07-10 the rc.d script sets `rc_bg=YES` and calls `rc_cmd "$1"` normally — dserver does not daemonize itself, and rc.subr's `rc_bg` is the proper way to background it. Earlier packages backgrounded the whole framework (`rc_cmd $1 &`), which hid `rc_pre` failures and the start result from `rcctl`
+- From the WireGuard VPN, `f0.lan.buetow.org` may not route — run the Makefile with `make dtail-openbsd FREEBSD_HOST=f0.wg0` (same override applies to the other `dtail-*` targets, which upload via f0)
+
+### FreeBSD (f0–f3)
+
+| File | Source template |
+|------|----------------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | cross-compiled `GOOS=freebsd CGO_ENABLED=0 -tags nozstd` |
+| `/usr/local/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail-freebsd.json.tpl` |
+| `/usr/local/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver-freebsd.tpl` |
+| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache-freebsd.sh.tpl` (sh) |
+
+**FreeBSD config note:** `dtail-freebsd.json.tpl` uses **absolute paths** for `CacheDir` and `HostKeyFile`. FreeBSD's `daemon(8)` resets CWD to `/`, so the relative `"cache"` in the standard template resolves to `/cache` — silently breaking key lookup. (Since dtail commit `fec2f9d`, absolute `CacheDir` paths also resolve independently of the CWD dserver was started from — before that fix, a manual service restart from a home directory broke public key auth.)
+
+FreeBSD notes:
+- FreeBSD's `cleanvar` purges `/var/run` at boot, so the SSH host key lives in persistent `/var/db/dserver/ssh_host_key` (mirrors NetBSD/OpenBSD). The rc.d `start_precmd` recreates `/var/run/dserver/cache` and `/var/db/dserver` and re-runs `dserver-update-key-cache.sh` on every start; the daily periodic job keeps it fresh afterwards
+- Packages before 2026-07-10 kept the host key in volatile `/var/run/dserver/cache/ssh_host_key` — it regenerated on every reboot. When upgrading a host from such a package, migrate the key FIRST (before restarting dserver) to preserve the host identity:
+ 1. `doas install -d -o dserver -m 0700 /var/db/dserver && doas cp -p /var/run/dserver/cache/ssh_host_key /var/db/dserver/ssh_host_key`
+ 2. `doas pkg update -f && doas pkg install -fy dtail` (same-version force reinstall)
+ 3. `doas service dserver restart`
+ (done on f0/f1/f2 2026-07-10 — sha256 of the migrated key verified identical; **f3 was unreachable and still runs the pre-2026-07-10 package**)
+
+### NetBSD (pi0, pi1 — aarch64)
+
+Package name is `dtail-4.3.2ng` — NetBSD versions must not contain dashes, so `-ng` becomes `ng`.
+
+| File | Source template |
+|------|----------------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | cross-compiled `GOOS=netbsd GOARCH=arm64 CGO_ENABLED=0 -tags nozstd` |
+| `/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail-netbsd.json.tpl` (absolute `CacheDir` like FreeBSD; `HostKeyFile` in persistent `/var/db/dserver/ssh_host_key`) |
+| `/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver-netbsd.tpl` |
+| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache-netbsd.sh.tpl` (sh) |
+
+NetBSD notes:
+- `pkg_create` runs natively on pi0 (Makefile ships binaries + templates there via SSH and runs `packages/scripts/pkg-dtail-netbsd.sh`); `pkg_summary.gz` for pkgin is generated and uploaded alongside
+- NetBSD has no `daemon(8)` and dserver doesn't daemonize — the rc.d script backgrounds it via `command_args="... &"` and runs it as user `dserver` (`dserver_user`)
+- `/var/run` is volatile — the rc.d `start_precmd` recreates `/var/run/dserver/cache` and re-runs the key-cache helper on every start; a daily root cron entry (`dserver-update-key-cache.sh`) keeps it fresh
+- The SSH host key lives in persistent `/var/db/dserver/ssh_host_key` (created by the precmd) so it survives reboots — since 2026-07-10 FreeBSD uses the same layout. Packages before 2026-07-09 used the volatile path; after upgrading, DTail clients without `--trustAllHosts` must re-accept the host key once
+- npf firewall needs `pass stateful in final family inet4 proto tcp to $ext_if port 2222` in the `"external"` group of `/etc/npf.conf`
+
+### Rocky Linux (r0–r2 amd64, pi2–pi3 aarch64)
+
+| File |
+|------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` |
+| `/usr/local/bin/dserver-update-key-cache.sh` |
+| `/etc/dserver/dtail.json` |
+| `/usr/lib/systemd/system/dserver.service` |
+| `/usr/lib/systemd/system/dserver-update-keycache.service` |
+| `/usr/lib/systemd/system/dserver-update-keycache.timer` |
+
+Rocky notes:
+- Key-cache helper handles both `/root/.ssh/authorized_keys` and `/home/*/.ssh/authorized_keys` — `root` works on r0–r2 without manual cache copy
+- `dserver.service` includes `RuntimeDirectory=dserver` and `ExecStartPre` to recreate `/var/run/dserver` (tmpfs) on Rocky
+- Repo is unsigned (`gpgcheck=0`)
+- `aarch64` RPM is built on pi2 — Fedora's rpmbuild refuses to emit `aarch64` binary RPMs from an x86_64 host
+
+## Install / Update
+
+### OpenBSD (via Rex)
+
+```sh
+cd ~/git/conf/frontends
+rex dtail_install # install or update from custom repo
+rex dtail # full setup: install + _dserver user + daily cron + service start
+```
+
+### FreeBSD (manual, f0–f3)
+
+```sh
+# Custom repo must already be configured (see client-setup.md)
+doas pkg install dtail # first install
+doas pkg install -fy dtail # force reinstall (same version)
+
+# Create service user (once per host)
+doas pw useradd dserver -d /var/run/dserver -s /usr/sbin/nologin
+
+# Enable and start dserver
+doas sysrc dserver_enable=YES
+doas service dserver start
+
+# Key cache is populated by the rc.d start_precmd on every service start;
+# run manually only when authorized_keys changed without a restart:
+doas /usr/local/bin/dserver-update-key-cache.sh
+
+# Register daily key cache refresh
+doas mkdir -p /usr/local/etc/periodic/daily
+printf '%s\n%s\n' '#!/bin/sh' '/usr/local/bin/dserver-update-key-cache.sh' | \
+ doas tee /usr/local/etc/periodic/daily/200.dserver-update-key-cache > /dev/null
+doas chmod 755 /usr/local/etc/periodic/daily/200.dserver-update-key-cache
+```
+
+**FreeBSD gotchas:**
+- Since the 2026-07-10 package the rc.d `start_precmd` re-runs `dserver-update-key-cache.sh` on every start, so the key cache repopulates automatically after restart or reboot (older packages required a manual re-run after any restart)
+- `pkg install -fy` replaces `/usr/local/etc/dserver/dtail.json` with the package version; local customisations are lost
+- Avoid inline one-liners with `||`, `!`, or multi-quote strings over SSH to FreeBSD (csh) — pipe a script to `doas /bin/sh` instead or use separate SSH commands
+
+### NetBSD (manual, pi0–pi1)
+
+```sh
+# All as root via doas; pkg_* tools live in /usr/sbin (not in non-interactive SSH PATH)
+export PATH=/usr/sbin:$PATH
+
+# Service group + user (once per host)
+doas groupadd dserver
+doas useradd -g dserver -d /var/run/dserver -s /sbin/nologin -c "DTail server" dserver
+
+# Install / update from the custom repo
+doas pkg_add https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz
+doas pkg_add -u https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz # newer version
+# Same-version reinstall: pkg_delete dtail first, then pkg_add
+
+# Enable and start (rc.d script ships in the package)
+doas sh -c 'echo dserver=YES >> /etc/rc.conf' # once per host
+doas /etc/rc.d/dserver start
+
+# Open port 2222 (once per host): add to the "external" group in /etc/npf.conf:
+# pass stateful in final family inet4 proto tcp to $ext_if port 2222
+# then: doas npfctl validate && doas npfctl reload
+
+# Daily key-cache refresh (once per host; rc.d start also refreshes it)
+# root crontab entry: 30 4 * * * /usr/local/bin/dserver-update-key-cache.sh >/dev/null 2>&1
+```
+
+**NetBSD gotchas:**
+- The package deliberately does not create the `dserver` user/group (matching the FreeBSD package) — run the `groupadd`/`useradd` step above before the first service start or the rc.d precmd fails
+- The key cache lives in volatile `/var/run` but the rc.d `start_precmd` recreates and repopulates it on every start — no manual re-run needed after restart or reboot
+- `dserver -version` panics when run as root (`Not allowed to run as UID 0`) — check with `su -m dserver -c '/usr/local/bin/dserver -version'` or as a normal user
+
+### Rocky Linux (dnf)
+
+```sh
+# On r0–r2 (root) or pi2–pi3 (paul with sudo):
+sudo dnf upgrade dtail
+sudo systemctl restart dserver
+sudo systemctl start dserver-update-keycache.service # repopulate after restart
+```
+
+**Rocky gotcha:** A legacy `/etc/systemd/system/dserver-update-keycache.service` left from manual pre-RPM setup points to the old path `ExecStart=/var/run/dserver/update_key_cache.sh` and shadows the RPM-installed unit. Remove it on any host where the timer fails:
+
+```sh
+sudo rm -f /etc/systemd/system/dserver-update-keycache.service
+sudo systemctl daemon-reload
+sudo systemctl start dserver-update-keycache.service
+```
+
+This was cleaned up on all r0–r2 and pi0–pi3 on 2026-04-19.
+
+## Client Usage from earth
+
+```sh
+# OpenBSD frontends
+dcat --plain --noColor --trustAllHosts --user rex \
+ --servers blowfish.buetow.org,fishfinger.buetow.org --files /etc/fstab
+
+# FreeBSD hosts
+dcat --plain --noColor --trustAllHosts --user paul \
+ --servers f0.lan.buetow.org,f1.lan.buetow.org,f2.lan.buetow.org,f3.lan.buetow.org \
+ --files /etc/fstab
+
+# Rocky VMs (r0–r2, user root)
+dcat --plain --noColor --trustAllHosts --user root \
+ --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org --files /etc/fstab
+
+# Raspberry Pis Rocky (pi2–pi3, user paul)
+dcat --plain --noColor --trustAllHosts --user paul \
+ --servers pi2.lan.buetow.org,pi3.lan.buetow.org \
+ --files /etc/fstab
+
+# Raspberry Pis NetBSD (pi0–pi1, user paul)
+dcat --plain --noColor --trustAllHosts --user paul \
+ --servers pi0.lan.buetow.org,pi1.lan.buetow.org \
+ --files /etc/fstab
+```
+
+## Verification State
+
+| Date | Platform | Result |
+|------|----------|--------|
+| 2026-04-19 | FreeBSD f0–f3 | `dtail-4.3.2-ng` installed, dserver running under `daemon(8)`, `dcat /etc/fstab` ✓ (`--user paul`) |
+| 2026-04-19 | OpenBSD blowfish, fishfinger | `dtail-4.3.2-ng` current, `dcat /etc/fstab` ✓ (`--user rex`) |
+| 2026-04-19 | Rocky r0–r2 | `dtail-4.3.2-ng` current, dserver running, `dcat /etc/fstab` ✓ (`--user root`) |
+| 2026-04-19 | Rocky pi0–pi3 | `dtail-4.3.2-ng` current, dserver running, `dcat /etc/fstab` ✓ (`--user paul`) — pi0/pi1 since re-imaged to NetBSD |
+| 2026-07-09 | NetBSD pi0–pi1 | `dtail-4.3.2ng` installed (first NetBSD deployment), dserver running as `dserver` on 2222, `dcat /etc/fstab` ✓ (`--user paul`) |
+| 2026-07-10 | FreeBSD f0–f2 | `dtail-4.3.2-ng` reinstalled with persistent host key (`/var/db/dserver`) — key migrated first, sha256 identical, mtime preserved; rc.d precmd repopulates key cache; `dcat /etc/fstab` ✓ (`--user paul`). **f3 unreachable (no route via wg0) — still on the pre-2026-07-10 package, migrate + upgrade when back** |
+| 2026-07-10 | OpenBSD fishfinger | `dtail-4.3.2-ng` reinstalled (rc_bg rc.d + hardened key-cache script); host key sha256/mtime unchanged, `rcctl restart` reports ok, `dcat /etc/fstab` ✓ (`--user rex`). blowfish still on the older package |
diff --git a/prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md b/prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md
new file mode 100644
index 0000000..2c2831f
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md
@@ -0,0 +1,42 @@
+# OpenBSD Build VM
+
+A minimal OpenBSD QEMU/KVM VM on earth for native compilation of CGo packages (e.g. dtail with DataDog/zstd). Cross-compiling CGo from Linux to OpenBSD needs a C cross-compiler; native build sidesteps that entirely.
+
+Scripts in `~/git/conf/packages/buildvm/`.
+
+## Initial Setup (once)
+
+```sh
+cd ~/git/conf/packages/buildvm
+
+# Fetch signify keys from fishfinger
+scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.sec .
+scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub .
+
+./setup.sh # downloads ISO, runs fully automated install (~5 min)
+./provision.sh # installs Go, git, gmake, signify keys, SSH keys
+```
+
+`setup.sh` is fully automated via `install-expect.exp` — drives the OpenBSD serial console installer without manual interaction.
+
+## Day-to-Day Use
+
+```sh
+make buildvm-start # boot VM (~15s)
+make dtail-openbsd # auto-starts VM if needed, then builds
+make buildvm-stop # shut down when done
+```
+
+VM specs: headless, SSH on `localhost:2222`, 1 GB RAM, 2 CPUs, 4 GB disk.
+Username: `pbuild` (password: `build123`). SSH key installed by `provision.sh`.
+
+Source is synced via `git archive HEAD | ssh ... tar -x` — not `scp -r` — to avoid filling `/tmp` with build artifacts and test data (full repo with benchmarks exceeds the 4 GB disk).
+
+## Installer Notes (install-expect.exp)
+
+- Expect script is a **separate file** to avoid bash/expect quoting interactions — embedding it in a bash heredoc breaks password sends
+- Serial console activated via `set tty com0` at the OpenBSD boot prompt
+- Password prompts need `sleep 2` before `send` — `sleep 1` is not enough for the serial console
+- OpenBSD 7.8 added an "Encrypt the root disk?" prompt before the partition layout
+- After CONGRATULATIONS, choose shell (`s`) not reboot — the CD is still attached; configure wheel group via `chroot /mnt`
+- Username `build` is rejected ("not a usable loginname") — use `pbuild`
diff --git a/prompts/skills/f3s-pkgrepo/references/packaging-workflow.md b/prompts/skills/f3s-pkgrepo/references/packaging-workflow.md
new file mode 100644
index 0000000..d2b7388
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/references/packaging-workflow.md
@@ -0,0 +1,120 @@
+# Packaging Workflow
+
+Build scripts live in `~/git/conf/packages/`. The Makefile cross-compiles Go binaries on Linux, ships them to target hosts for native packaging, and uploads to the PV.
+
+## Single-Binary Go Packages
+
+For pure Go packages (no CGo), cross-compilation from Linux works for both FreeBSD and OpenBSD.
+
+### Build and upload
+
+```sh
+cd ~/git/conf/packages
+
+# Both FreeBSD and OpenBSD
+make pkg NAME=gogios SRC=/home/paul/git/gogios \
+ COMMENT="Monitoring tool with email alerts and HTML status page" \
+ DESC="Gogios is a lightweight monitoring tool written in Go."
+
+# Single OS
+make pkg-freebsd NAME=gogios SRC=/home/paul/git/gogios
+make pkg-openbsd NAME=gogios SRC=/home/paul/git/gogios
+```
+
+### How it works
+
+1. Cross-compiles on Linux (`GOOS=freebsd/openbsd GOARCH=amd64`)
+2. SCPs binary + packaging script to target host (f0 for FreeBSD, fishfinger for OpenBSD)
+3. Runs the packaging script via SSH (`pkg create` / `pkg_create`)
+4. OpenBSD packages are signed with signify automatically
+5. FreeBSD repo metadata is regenerated with `pkg repo`
+6. Packages are copied to the PV at `/data/nfs/k3svolumes/pkgrepo/`
+
+### Required Makefile variables
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `NAME` | Package name | `gogios` |
+| `SRC` | Go project root (must have `cmd/<NAME>/main.go` and `internal/version.go`) | `/home/paul/git/gogios` |
+
+### Optional Makefile variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `COMMENT` | `$(NAME)` | One-line package description |
+| `DESC` | `$(NAME)` | Longer description |
+| `MAINTAINER` | `paul@buetow.org` | Maintainer email |
+| `WWW` | `https://buetow.org` | Project URL |
+| `ENTRY` | `cmd/$(NAME)/main.go` | Go main package path relative to SRC |
+
+### Version detection
+
+Version is read automatically from `$(SRC)/internal/version.go` — expects a `Version` constant like `const Version = "v1.4.1"`. The `v` prefix is stripped.
+
+## CGo Packages
+
+Cross-compilation from Linux fails for CGo (e.g. packages with DataDog/zstd). Use native builds instead:
+
+- **OpenBSD**: native build on the local QEMU/KVM build VM (see [openbsd-build-vm.md](openbsd-build-vm.md))
+- **FreeBSD**: cross-compile with `CGO_ENABLED=0 -tags nozstd` — disables zstd support but allows static cross-compile
+- **NetBSD**: cross-compile with `CGO_ENABLED=0 GOOS=netbsd GOARCH=arm64 -tags nozstd`; packaging (`pkg_create`) runs natively on pi0 because `+BUILD_INFO` must match the target host
+- **Rocky Linux**: built locally on earth (x86_64) and on pi2 (aarch64 via rpmbuild)
+
+## Manual Packaging Reference
+
+### FreeBSD (on f0)
+
+```sh
+pkg create -M +MANIFEST -p plist -r stagedir -o output/All
+pkg repo output/ # regenerates repo metadata
+doas cp -Rf output/* /data/nfs/k3svolumes/pkgrepo/freebsd/FreeBSD:15:amd64/latest/
+```
+
+### OpenBSD (on fishfinger)
+
+```sh
+pkg_create \
+ -D COMMENT="Package description" \
+ -d descfile \
+ -f packing-list \
+ -B stagedir \
+ -p / \
+ output/package-name-1.0.tgz
+# Copy to PV via f0
+scp package.tgz f0.lan.buetow.org:/tmp/
+ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package.tgz /data/nfs/k3svolumes/pkgrepo/openbsd/7.8/packages/amd64/"
+```
+
+### NetBSD (on pi0)
+
+```sh
+# pkg_* tools are in /usr/sbin (not in the non-interactive SSH PATH)
+# Files staged under stagedir (-p), installed relative to / (-I);
+# @owner root / @group wheel in the packing list keeps installed files root-owned.
+/usr/sbin/pkg_create \
+ -B build-info \
+ -c commentfile \
+ -d descfile \
+ -f packing-list \
+ -I / \
+ -p stagedir \
+ output/package-name-1.0.tgz
+# pkg_summary.gz enables pkgin (pkg_add alone doesn't need it)
+/usr/sbin/pkg_info -X output/*.tgz | gzip -9 > output/pkg_summary.gz
+# Copy to PV via f0
+scp -P 22 output/* f0.lan.buetow.org:/tmp/
+ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package-name-1.0.tgz /tmp/pkg_summary.gz /data/nfs/k3svolumes/pkgrepo/netbsd/10.1/packages/aarch64/"
+```
+
+NetBSD package versions must not contain dashes (the last dash separates the
+package name from the version), so DTail's `4.3.2-ng` becomes `4.3.2ng`.
+
+## Install/Update on Frontends via Rex
+
+```sh
+cd ~/git/conf/frontends
+rex gogios_install # installs or updates gogios on blowfish + fishfinger (OpenBSD) and f0-f3 (FreeBSD)
+rex gogios # full setup: gogios_install + config + cron
+```
+
+The `gogios_install` Rex task auto-detects the OS and uses `pkg install` (FreeBSD) or `pkg_add` (OpenBSD).
diff --git a/prompts/skills/f3s-pkgrepo/references/repo-architecture.md b/prompts/skills/f3s-pkgrepo/references/repo-architecture.md
new file mode 100644
index 0000000..ae42e1f
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/references/repo-architecture.md
@@ -0,0 +1,80 @@
+# Repo Architecture
+
+Custom FreeBSD, OpenBSD, NetBSD, and Rocky Linux package repository served from k3s.
+
+## Overview
+
+- **nginx pod** in k3s `infra` namespace serves static files from a PV
+- URL: `https://pkgrepo.f3s.buetow.org`
+- Path prefixes: `/freebsd/`, `/openbsd/`, `/netbsd/`, `/rockylinux/`
+- TLS terminated by OpenBSD relayd on the internet gateways (not in the pod)
+- DNS, ACME certs, httpd fallback, and relayd routing auto-generated from `@f3s_hosts` in `frontends/Rexfile`
+- HTTP always redirected to HTTPS by the OpenBSD gateways — client URLs must use `https://`
+
+## PV Directory Structure
+
+```
+/data/nfs/k3svolumes/pkgrepo/ ← NFS-backed PV, physically on f0
+ freebsd/
+ FreeBSD:15:amd64/
+ latest/
+ All/ # .pkg files
+ packagesite.pkg # repo metadata (generated by `pkg repo`)
+ meta.conf / meta
+ data.pkg
+ openbsd/
+ 7.8/
+ packages/
+ amd64/ # .tgz files (signify-signed)
+ netbsd/
+ 10.1/
+ packages/
+ aarch64/ # .tgz files (unsigned) + pkg_summary.gz for pkgin
+ rockylinux/
+ 9/
+ x86_64/ # .rpm files + repodata/
+ aarch64/ # .rpm files + repodata/
+```
+
+## Key Config Files
+
+| File | Purpose |
+|------|---------|
+| `f3s/pkgrepo/helm-chart/` | Helm chart (nginx deployment, service, ingress, PV, configmap) |
+| `f3s/argocd-apps/infra/pkgrepo.yaml` | ArgoCD Application manifest |
+| `f3s/pkgrepo/test-artifacts/build-test-packages.sh` | Builds hello-test packages for FreeBSD/OpenBSD |
+| `frontends/Rexfile` (`@f3s_hosts`) | DNS + routing entry for `pkgrepo.f3s.buetow.org` |
+| `frontends/Rexfile` (`pkgrepo_setup`) | Adds `PKG_PATH` to root's `.profile` on OpenBSD frontends |
+| `packages/Makefile` | Cross-compiles, packages, and uploads for all OSes |
+| `packages/scripts/pkg-freebsd.sh` | Runs on f0 via SSH: `pkg create` + `pkg repo` + copy to PV |
+| `packages/scripts/pkg-openbsd.sh` | Runs on fishfinger via SSH: `pkg_create` + signify signing |
+| `packages/scripts/pkg-dtail-openbsd.sh` | DTail multi-binary OpenBSD packaging |
+| `packages/scripts/pkg-dtail-freebsd.sh` | DTail multi-binary FreeBSD packaging |
+| `packages/scripts/pkg-dtail-netbsd.sh` | DTail multi-binary NetBSD packaging (runs on pi0 via SSH: `pkg_create` + `pkg_summary.gz`) |
+| `packages/scripts/pkg-dtail-rpm.sh` | Builds DTail RPMs from prebuilt or locally built payloads |
+
+## SSH Access for Package Tasks
+
+| Host | Access | Root | Notes |
+|------|--------|------|-------|
+| f0 (FreeBSD, NFS) | `ssh -p 22 f0.lan.buetow.org` | `doas` | PV is local; default shell is **csh** |
+| fishfinger (OpenBSD) | `ssh rex@fishfinger.buetow.org` | `doas` | OpenBSD packages built here, then copied to f0 PV |
+| blowfish (OpenBSD) | `ssh rex@blowfish.buetow.org` | `doas` | Same setup as fishfinger |
+| pi0 (NetBSD) | `ssh -p 22 paul@pi0.lan.buetow.org` | `doas` | NetBSD packages assembled here (`pkg_create`), then copied to f0 PV; non-interactive SSH PATH lacks `/usr/sbin` (`pkg_*` live there) |
+
+## Nginx Pod Gotcha — Stale NFS Handle
+
+After writing new files to the PV on f0, the nginx pod can get a stale NFS file handle (error 116) and return HTTP 500. Fix with a pod restart:
+
+```sh
+kubectl -n infra rollout restart deployment/pkgrepo
+kubectl -n infra rollout status deployment/pkgrepo
+```
+
+## Per-OS Repo Notes
+
+- **FreeBSD**: version follows ABI naming (`FreeBSD:15:amd64`); packages unsigned (`signature_type: "NONE"`); always regenerate metadata with `pkg repo` after adding/removing packages
+- **OpenBSD**: no repo index needed — `pkg_add` fetches by name; packages signed with signify; version in path must match host OS (currently 7.8)
+- **NetBSD**: packages unsigned; `pkg_add` works with a direct package URL, `pkg_summary.gz` alongside the packages enables pkgin; package versions must not contain dashes (`4.3.2-ng` → `4.3.2ng`); `+BUILD_INFO` (`MACHINE_ARCH`/`OS_VERSION`) is checked by `pkg_add` on install, which is why packages are assembled natively on pi0
+- **Rocky Linux**: standard DNF layout; unsigned repo (`gpgcheck=0`); architecture-specific paths (`x86_64` / `aarch64`)
+- **FreeBSD csh**: default shell is csh — avoid inline one-liners with `||`, `&&`, `!`, or multi-line quoting over SSH; use piped `/bin/sh` or separate SSH invocations