Update package repo references, add new reference files for client setup, dtail package, OpenBSD build VM, and repo architecture. Remove obsolete package-repos.md.

7 files changed, 479 insertions, 436 deletions

diff --git a/prompts/skills/pkgrepo/SKILL.md b/prompts/skills/pkgrepo/SKILL.md
index 77de23e..5155537 100644
--- a/prompts/skills/pkgrepo/SKILL.md
+++ b/prompts/skills/pkgrepo/SKILL.md
@@ -16,7 +16,11 @@ Use this skill when the task is specifically about the custom package repositori
 
 ## Reference Files
 
-- [Package Repositories](references/package-repos.md) — repo layout, PV paths, nginx exposure, client config, and DTail package workflows
+- [Repo Architecture](references/repo-architecture.md) — nginx/k3s setup, PV directory structure, SSH access, stale NFS handle fix, per-OS repo notes
+- [Client Setup](references/client-setup.md) — per-OS client repo configuration (FreeBSD, OpenBSD, Rocky Linux), new-host setup, package signing
+- [Packaging Workflow](references/packaging-workflow.md) — Makefile workflow for single-binary Go packages, CGo packages, manual packaging reference
+- [DTail Package](references/dtail-package.md) — multi-binary DTail package for all platforms, install/update steps, gotchas, client usage, verification
+- [OpenBSD Build VM](references/openbsd-build-vm.md) — QEMU/KVM build VM for native CGo compilation, day-to-day use, installer notes
 
 ## Scope
 
@@ -29,3 +33,4 @@ Use `f3s` alongside this skill when the task depends on broader host-role or clu
 - `r0-r2` as Rocky Linux x86_64 bhyve VMs
 - `pi0-pi3` as Rocky Linux aarch64 Raspberry Pi nodes
 - `earth` as the Fedora laptop used for package publication and verification
+- `f0-f3` as FreeBSD hosts
diff --git a/prompts/skills/pkgrepo/references/client-setup.md b/prompts/skills/pkgrepo/references/client-setup.md
new file mode 100644
index 0000000..5e231ff
--- /dev/null
+++ b/prompts/skills/pkgrepo/references/client-setup.md
@@ -0,0 +1,123 @@
+# Client Setup
+
+How to configure each OS to install packages from `pkgrepo.f3s.buetow.org`.
+
+## FreeBSD (f0–f3)
+
+Custom repo is configured alongside the official FreeBSD repos.
+
+File: `/usr/local/etc/pkg/repos/custom.conf`
+
+```
+custom: {
+  url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
+  mirror_type: "NONE",
+  signature_type: "NONE",
+  enabled: yes
+}
+```
+
+### Using packages
+
+```sh
+doas pkg update
+doas pkg install <package-name>
+doas pkg upgrade                  # upgrade all, including custom packages
+doas pkg install -fy <package>    # force reinstall (same version)
+```
+
+### Setting up a new FreeBSD host
+
+Pipe via stdin to avoid csh quoting issues:
+
+```sh
+cat <<'REPO' | ssh -p 22 <host>.lan.buetow.org 'doas mkdir -p /usr/local/etc/pkg/repos && doas tee /usr/local/etc/pkg/repos/custom.conf > /dev/null'
+custom: {
+  url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
+  mirror_type: "NONE",
+  signature_type: "NONE",
+  enabled: yes
+}
+REPO
+```
+
+## OpenBSD (blowfish, fishfinger)
+
+Custom repo is configured via `PKG_PATH` in `/root/.profile`, deployed by `rex pkgrepo_setup`.
+Official OpenBSD packages still install normally via `/etc/installurl`.
+
+```sh
+export PKG_PATH="https://pkgrepo.f3s.buetow.org/openbsd/7.8/packages/amd64/"
+```
+
+### Using packages
+
+```sh
+doas pkg_add <package>       # installs from custom repo (signed with signify)
+doas pkg_add -u <package>    # update to latest version
+```
+
+### Setting up a new OpenBSD host
+
+Two things needed:
+
+1. **Signify public key** — copy from an existing host:
+   ```sh
+   scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub /tmp/
+   scp /tmp/custom-pkg.pub rex@<newhost>:/tmp/
+   ssh rex@<newhost> "doas cp /tmp/custom-pkg.pub /etc/signify/custom-pkg.pub"
+   ```
+
+2. **PKG_PATH** — run the Rex task from `~/git/conf/frontends`:
+   ```sh
+   rex pkgrepo_setup   # adds PKG_PATH to /root/.profile on all frontends
+   ```
+
+Update `PKG_PATH` whenever the OpenBSD version changes (currently 7.8).
+
+### Package signing
+
+OpenBSD packages are signed with `signify(1)` via `pkg_sign`:
+- **Private key**: `/etc/signify/custom-pkg.sec` on fishfinger (build host)
+- **Public key**: `/etc/signify/custom-pkg.pub` on all OpenBSD clients
+- Signing happens automatically during `make pkg-openbsd` / `make pkg`
+- `pkg_add` verifies the signature — no `-D unsigned` needed
+
+## Rocky Linux (r0–r2, pi0–pi3)
+
+Architecture-specific repo URLs:
+- `https://pkgrepo.f3s.buetow.org/rockylinux/9/x86_64/`  (r0–r2)
+- `https://pkgrepo.f3s.buetow.org/rockylinux/9/aarch64/` (pi0–pi3)
+
+### Persistent repo file
+
+Create `/etc/yum.repos.d/f3s-dtail.repo`:
+
+```ini
+[f3s-dtail]
+name=f3s DTail
+baseurl=https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/
+enabled=1
+gpgcheck=0
+repo_gpgcheck=0
+```
+
+Then:
+
+```sh
+sudo dnf makecache
+sudo dnf install dtail
+sudo dnf upgrade dtail
+```
+
+### Temporary one-off usage (no persistent repo file needed)
+
+```sh
+sudo dnf repoquery \
+  --disablerepo='*' \
+  --repofrompath=f3s-dtail,https://pkgrepo.f3s.buetow.org/rockylinux/9/$(uname -m)/ \
+  --enablerepo=f3s-dtail \
+  dtail
+```
+
+Do **not** pass `--repofrompath` when the persistent repo file already exists — dnf errors with "listed more than once".
diff --git a/prompts/skills/pkgrepo/references/dtail-package.md b/prompts/skills/pkgrepo/references/dtail-package.md
new file mode 100644
index 0000000..ca49f32
--- /dev/null
+++ b/prompts/skills/pkgrepo/references/dtail-package.md
@@ -0,0 +1,140 @@
+# DTail Package
+
+DTail is a multi-binary package (6 binaries + config + service script). There are separate targets for each OS.
+
+## Build Commands
+
+```sh
+cd ~/git/conf/packages
+make dtail-openbsd   # OpenBSD: native build on QEMU/KVM VM (CGo/zstd supported)
+make dtail-freebsd   # FreeBSD: cross-compiled on Linux (CGO_ENABLED=0, nozstd — .zst logs unsupported)
+make dtail-rocky     # Rocky Linux: x86_64 + aarch64 RPMs + repodata
+```
+
+## Package Contents by OS
+
+### OpenBSD (blowfish, fishfinger)
+
+| File | Source template |
+|------|----------------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | built natively on build VM |
+| `/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail.json.tpl` |
+| `/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver.tpl` |
+| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache.sh.tpl` (ksh) |
+
+### FreeBSD (f0–f3)
+
+| File | Source template |
+|------|----------------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | cross-compiled `GOOS=freebsd CGO_ENABLED=0 -tags nozstd` |
+| `/usr/local/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail-freebsd.json.tpl` |
+| `/usr/local/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver-freebsd.tpl` |
+| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache-freebsd.sh.tpl` (sh) |
+
+**FreeBSD config note:** `dtail-freebsd.json.tpl` uses **absolute paths** for `CacheDir` and `HostKeyFile` (`/var/run/dserver/cache/...`). FreeBSD's `daemon(8)` resets CWD to `/`, so the relative `"cache"` in the standard template resolves to `/cache` — silently breaking key lookup.
+
+### Rocky Linux (r0–r2 amd64, pi0–pi3 aarch64)
+
+| File |
+|------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` |
+| `/usr/local/bin/dserver-update-key-cache.sh` |
+| `/etc/dserver/dtail.json` |
+| `/usr/lib/systemd/system/dserver.service` |
+| `/usr/lib/systemd/system/dserver-update-keycache.service` |
+| `/usr/lib/systemd/system/dserver-update-keycache.timer` |
+
+Rocky notes:
+- Key-cache helper handles both `/root/.ssh/authorized_keys` and `/home/*/.ssh/authorized_keys` — `root` works on r0–r2 without manual cache copy
+- `dserver.service` includes `RuntimeDirectory=dserver` and `ExecStartPre` to recreate `/var/run/dserver` (tmpfs) on Rocky
+- Repo is unsigned (`gpgcheck=0`)
+- `aarch64` RPM is built on pi0 — Fedora's rpmbuild refuses to emit `aarch64` binary RPMs from an x86_64 host
+
+## Install / Update
+
+### OpenBSD (via Rex)
+
+```sh
+cd ~/git/conf/frontends
+rex dtail_install   # install or update from custom repo
+rex dtail           # full setup: install + _dserver user + daily cron + service start
+```
+
+### FreeBSD (manual, f0–f3)
+
+```sh
+# Custom repo must already be configured (see client-setup.md)
+doas pkg install dtail          # first install
+doas pkg install -fy dtail      # force reinstall (same version)
+
+# Create service user (once per host)
+doas pw useradd dserver -d /var/run/dserver -s /usr/sbin/nologin
+
+# Enable and start dserver
+doas sysrc dserver_enable=YES
+doas service dserver start
+
+# Populate key cache immediately (also runs daily via periodic)
+doas /usr/local/bin/dserver-update-key-cache.sh
+
+# Register daily key cache refresh
+doas mkdir -p /usr/local/etc/periodic/daily
+printf '%s\n%s\n' '#!/bin/sh' '/usr/local/bin/dserver-update-key-cache.sh' | \
+  doas tee /usr/local/etc/periodic/daily/200.dserver-update-key-cache > /dev/null
+doas chmod 755 /usr/local/etc/periodic/daily/200.dserver-update-key-cache
+```
+
+**FreeBSD gotchas:**
+- `service dserver restart` clears `/var/run/dserver` (it's recreated by rc.d `start_precmd`, but the key cache files are gone) — always re-run `dserver-update-key-cache.sh` after any restart
+- `pkg install -fy` replaces `/usr/local/etc/dserver/dtail.json` with the package version; local customisations are lost
+- Avoid inline one-liners with `||`, `!`, or multi-quote strings over SSH to FreeBSD (csh) — pipe a script to `doas /bin/sh` instead or use separate SSH commands
+
+### Rocky Linux (dnf)
+
+```sh
+# On r0–r2 (root) or pi0–pi3 (paul with sudo):
+sudo dnf upgrade dtail
+sudo systemctl restart dserver
+sudo systemctl start dserver-update-keycache.service   # repopulate after restart
+```
+
+**Rocky gotcha:** A legacy `/etc/systemd/system/dserver-update-keycache.service` left from manual pre-RPM setup points to the old path `ExecStart=/var/run/dserver/update_key_cache.sh` and shadows the RPM-installed unit. Remove it on any host where the timer fails:
+
+```sh
+sudo rm -f /etc/systemd/system/dserver-update-keycache.service
+sudo systemctl daemon-reload
+sudo systemctl start dserver-update-keycache.service
+```
+
+This was cleaned up on all r0–r2 and pi0–pi3 on 2026-04-19.
+
+## Client Usage from earth
+
+```sh
+# OpenBSD frontends
+dcat --plain --noColor --trustAllHosts --user rex \
+  --servers blowfish.buetow.org,fishfinger.buetow.org --files /etc/fstab
+
+# FreeBSD hosts
+dcat --plain --noColor --trustAllHosts --user paul \
+  --servers f0.lan.buetow.org,f1.lan.buetow.org,f2.lan.buetow.org,f3.lan.buetow.org \
+  --files /etc/fstab
+
+# Rocky VMs (r0–r2, user root)
+dcat --plain --noColor --trustAllHosts --user root \
+  --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org --files /etc/fstab
+
+# Raspberry Pis (pi0–pi3, user paul)
+dcat --plain --noColor --trustAllHosts --user paul \
+  --servers pi0.lan.buetow.org,pi1.lan.buetow.org,pi2.lan.buetow.org,pi3.lan.buetow.org \
+  --files /etc/fstab
+```
+
+## Verification State
+
+| Date | Platform | Result |
+|------|----------|--------|
+| 2026-04-19 | FreeBSD f0–f3 | `dtail-4.3.2-ng` installed, dserver running, `dcat /etc/fstab` ✓ (`--user paul`) |
+| 2026-04-19 | OpenBSD blowfish, fishfinger | `dtail-4.3.2-ng` current, `dcat /etc/fstab` ✓ (`--user rex`) |
+| 2026-04-19 | Rocky r0–r2 | `dtail-4.3.2-ng` current, dserver running, `dcat /etc/fstab` ✓ (`--user root`) |
+| 2026-04-19 | Rocky pi0–pi3 | `dtail-4.3.2-ng` current, dserver running, `dcat /etc/fstab` ✓ (`--user paul`) |
diff --git a/prompts/skills/pkgrepo/references/openbsd-build-vm.md b/prompts/skills/pkgrepo/references/openbsd-build-vm.md
new file mode 100644
index 0000000..2c2831f
--- /dev/null
+++ b/prompts/skills/pkgrepo/references/openbsd-build-vm.md
@@ -0,0 +1,42 @@
+# OpenBSD Build VM
+
+A minimal OpenBSD QEMU/KVM VM on earth for native compilation of CGo packages (e.g. dtail with DataDog/zstd). Cross-compiling CGo from Linux to OpenBSD needs a C cross-compiler; native build sidesteps that entirely.
+
+Scripts in `~/git/conf/packages/buildvm/`.
+
+## Initial Setup (once)
+
+```sh
+cd ~/git/conf/packages/buildvm
+
+# Fetch signify keys from fishfinger
+scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.sec .
+scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub .
+
+./setup.sh       # downloads ISO, runs fully automated install (~5 min)
+./provision.sh   # installs Go, git, gmake, signify keys, SSH keys
+```
+
+`setup.sh` is fully automated via `install-expect.exp` — drives the OpenBSD serial console installer without manual interaction.
+
+## Day-to-Day Use
+
+```sh
+make buildvm-start   # boot VM (~15s)
+make dtail-openbsd   # auto-starts VM if needed, then builds
+make buildvm-stop    # shut down when done
+```
+
+VM specs: headless, SSH on `localhost:2222`, 1 GB RAM, 2 CPUs, 4 GB disk.
+Username: `pbuild` (password: `build123`). SSH key installed by `provision.sh`.
+
+Source is synced via `git archive HEAD | ssh ... tar -x` — not `scp -r` — to avoid filling `/tmp` with build artifacts and test data (full repo with benchmarks exceeds the 4 GB disk).
+
+## Installer Notes (install-expect.exp)
+
+- Expect script is a **separate file** to avoid bash/expect quoting interactions — embedding it in a bash heredoc breaks password sends
+- Serial console activated via `set tty com0` at the OpenBSD boot prompt
+- Password prompts need `sleep 2` before `send` — `sleep 1` is not enough for the serial console
+- OpenBSD 7.8 added an "Encrypt the root disk?" prompt before the partition layout
+- After CONGRATULATIONS, choose shell (`s`) not reboot — the CD is still attached; configure wheel group via `chroot /mnt`
+- Username `build` is rejected ("not a usable loginname") — use `pbuild`
diff --git a/prompts/skills/pkgrepo/references/package-repos.md b/prompts/skills/pkgrepo/references/package-repos.md
deleted file mode 100644
index d6c45c4..0000000
--- a/prompts/skills/pkgrepo/references/package-repos.md
+++ /dev/null
@@ -1,435 +0,0 @@
-# Package Repositories
-
-Custom FreeBSD, OpenBSD, and Rocky Linux package repository served from k3s, allowing native package tools (`pkg`, `pkg_add`, and `dnf`) to install custom-built packages.
-
-## Architecture
-
-- **nginx pod** in k3s `infra` namespace serves static files from a PV
-- Path prefixes: `/freebsd/`, `/openbsd/`, and `/rockylinux/`
-- Accessible at `https://pkgrepo.f3s.buetow.org`
-- TLS terminated by OpenBSD relayd on the internet gateways (not in the pod)
-- DNS, ACME certs, httpd fallback, and relayd routing auto-generated from `@f3s_hosts` in `frontends/Rexfile`
-
-## Required host context
-
-This reference is package-repo focused, but it still uses a few f3s host names directly:
-
-- `f0` = FreeBSD host that carries the NFS-backed PV at `/data/nfs/k3svolumes/pkgrepo/`
-- `fishfinger` and `blowfish` = OpenBSD frontend hosts used for OpenBSD package install/build tasks
-- `r0-r2` = Rocky Linux 9 x86_64 bhyve VMs
-- `pi0-pi3` = Rocky Linux 9 aarch64 Raspberry Pi nodes
-- `earth` = Fedora laptop used to build, publish, and verify packages
-
-If you need the broader homelab topology, network roles, or non-package service context, load the sibling `f3s` skill as well.
-
-## Key Files
-
-| File | Purpose |
-|------|---------|
-| `f3s/pkgrepo/helm-chart/` | Helm chart (nginx deployment, service, ingress, PV, configmap) |
-| `f3s/argocd-apps/infra/pkgrepo.yaml` | ArgoCD Application manifest |
-| `f3s/pkgrepo/test-artifacts/build-test-packages.sh` | Builds hello-test packages for FreeBSD/OpenBSD |
-| `frontends/Rexfile` (`@f3s_hosts`) | DNS + routing entry for `pkgrepo.f3s.buetow.org` |
-| `frontends/Rexfile` (`pkgrepo_setup`) | Adds `PKG_PATH` to root's `.profile` on OpenBSD frontends |
-| `frontends/Rexfile` (`gogios_install`) | Installs/updates gogios from the custom repo |
-| `packages/Makefile` | `make pkg` cross-compiles, packages, and uploads for both OSes; `make dtail-rocky` builds and publishes the Rocky repo |
-| `packages/scripts/pkg-freebsd.sh` | Runs on f0 via SSH: pkg create + pkg repo + copy to PV |
-| `packages/scripts/pkg-openbsd.sh` | Runs on fishfinger via SSH: pkg_create + signify signing |
-| `packages/scripts/pkg-dtail-rpm.sh` | Builds DTail RPMs from prebuilt or locally built payloads |
-
-## PV Directory Structure
-
-```
-/data/nfs/k3svolumes/pkgrepo/
-  freebsd/
-    FreeBSD:15:amd64/
-      latest/
-        All/              # .pkg files
-        packagesite.pkg   # repo metadata (generated by `pkg repo`)
-        meta / meta.conf
-        data.pkg
-  openbsd/
-    7.8/
-      packages/
-        amd64/            # .tgz files
-  rockylinux/
-    9/
-      x86_64/             # .rpm files + repodata/
-      aarch64/            # .rpm files + repodata/
-```
-
-Hosted on NFS at f0 (`/data/nfs/k3svolumes/pkgrepo/`).
-
-## FreeBSD Client Setup (f0–f3)
-
-Custom repo is pre-configured on f0, f1, f2 (f3 offline at time of setup).
-Official FreeBSD repos remain active alongside the custom repo.
-
-File: `/usr/local/etc/pkg/repos/custom.conf`
-
-```
-custom: {
-  url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
-  mirror_type: "NONE",
-  signature_type: "NONE",
-  enabled: yes
-}
-```
-
-### Using packages
-
-```sh
-doas pkg update
-doas pkg install <package-name>   # works for both official and custom packages
-```
-
-### Setting up a new FreeBSD host
-
-Pipe the config via stdin to avoid csh quoting issues (FreeBSD default shell is csh):
-
-```sh
-cat <<'REPO' | ssh -p 22 <host>.lan.buetow.org 'doas mkdir -p /usr/local/etc/pkg/repos && doas tee /usr/local/etc/pkg/repos/custom.conf > /dev/null'
-custom: {
-  url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
-  mirror_type: "NONE",
-  signature_type: "NONE",
-  enabled: yes
-}
-REPO
-```
-
-## OpenBSD Client Setup (blowfish/fishfinger)
-
-Custom repo is configured via `PKG_PATH` in `/root/.profile`, deployed by `rex pkgrepo_setup`.
-Official OpenBSD packages still install normally via `/etc/installurl`.
-
-```sh
-export PKG_PATH="https://pkgrepo.f3s.buetow.org/openbsd/7.8/packages/amd64/"
-```
-
-### Using packages
-
-```sh
-doas pkg_add <package>               # from custom repo (signed with signify)
-doas pkg_add <official-package>      # from official repo (still works)
-```
-
-### Setting up new OpenBSD hosts
-
-Two things needed:
-
-1. **Signify public key** — copy from an existing host to `/etc/signify/custom-pkg.pub`:
-   ```sh
-   scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub /tmp/
-   scp /tmp/custom-pkg.pub rex@<newhost>:/tmp/
-   ssh rex@<newhost> "doas cp /tmp/custom-pkg.pub /etc/signify/custom-pkg.pub"
-   ```
-
-2. **PKG_PATH** — run the Rex task from `~/git/conf/frontends`:
-   ```sh
-   rex pkgrepo_setup   # adds PKG_PATH to /root/.profile on all frontends
-   ```
-
-The PKG_PATH must be updated when the OpenBSD version changes (currently 7.8).
-
-## Rocky Linux Client Setup (r0-r2, pi0-pi3)
-
-The Rocky repo is architecture-specific and follows the standard DNF layout:
-
-- `https://pkgrepo.f3s.buetow.org/rockylinux/9/x86_64/`
-- `https://pkgrepo.f3s.buetow.org/rockylinux/9/aarch64/`
-
-### Temporary one-off usage
-
-```sh
-sudo dnf repoquery \
-  --disablerepo='*' \
-  --repofrompath=f3s-dtail,https://pkgrepo.f3s.buetow.org/rockylinux/9/$(uname -m)/ \
-  --enablerepo=f3s-dtail \
-  dtail
-```
-
-### Persistent repo file
-
-Create `/etc/yum.repos.d/f3s-dtail.repo`:
-
-```ini
-[f3s-dtail]
-name=f3s DTail
-baseurl=https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/
-enabled=1
-gpgcheck=0
-repo_gpgcheck=0
-```
-
-Then:
-
-```sh
-sudo dnf makecache
-sudo dnf install dtail
-```
-
-### Package signing
-
-OpenBSD packages are signed with `signify(1)` via `pkg_sign`:
-- **Private key**: `/etc/signify/custom-pkg.sec` on fishfinger (build host)
-- **Public key**: `/etc/signify/custom-pkg.pub` on all OpenBSD clients
-- Signing happens automatically during `make pkg-openbsd` / `make pkg`
-- `pkg_add` verifies the signature against the public key — no `-D unsigned` needed
-
-## Packaging Workflow (Makefile)
-
-Build scripts live in `~/git/conf/packages/`, separate from individual project repos. The Makefile cross-compiles Go binaries on Linux, ships them to target hosts for native packaging, and uploads to the PV.
-
-### Build and upload
-
-```sh
-cd ~/git/conf/packages
-
-# Both FreeBSD and OpenBSD
-make pkg NAME=gogios SRC=/home/paul/git/gogios \
-    COMMENT="Monitoring tool with email alerts and HTML status page" \
-    DESC="Gogios is a lightweight monitoring tool written in Go."
-
-# Single OS
-make pkg-freebsd NAME=gogios SRC=/home/paul/git/gogios
-make pkg-openbsd NAME=gogios SRC=/home/paul/git/gogios
-
-# DTail Rocky Linux repo (x86_64 + aarch64 RPMs + repodata)
-make dtail-rocky
-```
-
-### How it works
-
-**Single-binary packages (pure Go, cross-compilable):**
-1. Cross-compiles on Linux (`GOOS=freebsd/openbsd GOARCH=amd64`)
-2. SCPs binary + packaging script to target host (f0 for FreeBSD, fishfinger for OpenBSD)
-3. Runs the script via SSH to package natively (`pkg create` / `pkg_create`)
-4. OpenBSD packages are signed with signify automatically
-5. FreeBSD repo metadata is regenerated with `pkg repo`
-6. Packages are copied to the PV at `/data/nfs/k3svolumes/pkgrepo/`
-
-**CGo packages (e.g. dtail with DataDog/zstd):**
-Cross-compilation from Linux to OpenBSD fails for CGo — needs a C cross-compiler.
-Solution: native build on the local OpenBSD VM. Source is sent via `git archive HEAD | ssh ... tar -x`
-(not `scp -r`) to avoid filling /tmp with build artifacts and test data.
-
-**Rocky Linux RPMs (dtail only, current implementation):**
-- `x86_64` RPM is built locally on earth
-- `aarch64` RPM is packaged on `pi0` because Fedora's x86_64 `rpmbuild` refuses to emit an `aarch64` binary RPM even when the payload is already prebuilt
-- repo metadata is generated locally in a `rockylinux:9` container with `createrepo_c`
-- the finished repo tree is copied to `/data/nfs/k3svolumes/pkgrepo/rockylinux/9/`
-
-### Required variables
-
-| Variable | Description | Example |
-|----------|-------------|---------|
-| `NAME` | Package name | `gogios` |
-| `SRC` | Go project root (must have `cmd/<NAME>/main.go` and `internal/version.go`) | `/home/paul/git/gogios` |
-
-### Optional variables
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `COMMENT` | `$(NAME)` | One-line description |
-| `DESC` | `$(NAME)` | Longer description |
-| `MAINTAINER` | `paul@buetow.org` | Maintainer email |
-| `WWW` | `https://buetow.org` | Project URL |
-| `ENTRY` | `cmd/$(NAME)/main.go` | Go main package path relative to SRC |
-
-### Version detection
-
-Version is read automatically from `$(SRC)/internal/version.go` — expects a `Version` constant like `const Version = "v1.4.1"`. The `v` prefix is stripped.
-
-### Install/update on frontends via Rex
-
-```sh
-cd ~/git/conf/frontends
-rex gogios_install   # installs or updates gogios on blowfish + fishfinger
-rex gogios           # full setup (includes gogios_install + config + cron)
-```
-
-The `gogios_install` Rex task auto-detects the OS and uses `pkg install` (FreeBSD) or `pkg_add` (OpenBSD).
-
-## DTail Package (multi-binary, OpenBSD only)
-
-DTail is a distributed log tailing tool with 6 binaries, config files, and an rc script.
-Unlike single-binary packages, it has a dedicated Makefile target and packaging script.
-
-### Build and upload
-
-```sh
-cd ~/git/conf/packages
-make dtail-openbsd
-```
-
-### What the package includes
-
-| File | Purpose |
-|------|---------|
-| `/usr/local/bin/dserver` | DTail server daemon |
-| `/usr/local/bin/dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | Client tools |
-| `/etc/dserver/dtail.json` | Server config (from `frontends/etc/dserver/dtail.json.tpl`) |
-| `/etc/rc.d/dserver` | RC script (from `frontends/etc/rc.d/dserver.tpl`) |
-| `/usr/local/bin/dserver-update-key-cache.sh` | SSH key cache helper for daily cron |
-
-### Rex integration
-
-```sh
-cd ~/git/conf/frontends
-rex dtail_install   # install/update dtail package from custom repo
-rex dtail           # full setup (install + service user + daily cron + start)
-```
-
-The `dtail` Rex task installs the package, creates the `_dserver` service user,
-adds the key cache script to daily cron, and ensures dserver is running.
-
-## DTail Rocky Linux RPM Repo
-
-DTail for Rocky is published as a multi-binary RPM for both lab architectures:
-
-- `dtail-4.3.2-1.ng.x86_64.rpm`
-- `dtail-4.3.2-1.ng.aarch64.rpm`
-
-The RPM currently installs:
-
-- `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth`
-- `/usr/local/bin/dserver-update-key-cache.sh`
-- `/etc/dserver/dtail.json`
-- `/usr/lib/systemd/system/dserver.service`
-- `/usr/lib/systemd/system/dserver-update-keycache.service`
-- `/usr/lib/systemd/system/dserver-update-keycache.timer`
-
-### Build and publish
-
-```sh
-cd ~/git/conf/packages
-make dtail-rocky
-```
-
-### Notes
-
-- The packaged key-cache helper handles both `/root/.ssh/authorized_keys` and `/home/*/.ssh/authorized_keys`, so `root` works on `r0-r2` without a separate manual cache copy.
-- The packaged `dserver.service` includes the `RuntimeDirectory` and `ExecStartPre` fix required on Rocky so `/var/run/dserver` exists before startup.
-- The repo is unsigned for now, so the example repo file uses `gpgcheck=0` and `repo_gpgcheck=0`.
-
-### Verified DTail package state
-
-On 2026-04-11 this package repo path was verified with:
-
-- `dtail-4.3.2-1.ng.x86_64.rpm` and `dtail-4.3.2-1.ng.aarch64.rpm` published at `https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/`
-- successful `dnf repoquery` for `dtail` from `r0` (`x86_64`) and `pi0` (`aarch64`)
-- successful `dnf install dtail` on `r0-r2` and `pi0-pi3`
-- `dserver` active after package install, timer enable, restart, and key-cache refresh on all seven Rocky hosts
-- successful `dcat /etc/fstab` reads from earth using `--user root` for `r0-r2` and `--user paul` for `pi0-pi3`
-
-### OpenBSD DTail package verification
-
-Also verified on 2026-04-11:
-
-- `dtail-4.3.2-ng` rebuilt and published to the OpenBSD repo path
-- package replacement, key-cache refresh, and successful `dcat /etc/fstab` reads as `--user rex` on `blowfish.buetow.org` and `fishfinger.buetow.org`
-
-### Why a build VM?
-
-DTail depends on DataDog/zstd which requires CGo. Cross-compiling CGo from Linux
-to OpenBSD needs a C cross-compiler. Instead, a local OpenBSD QEMU/KVM VM builds
-natively — this also works for any future packages with C dependencies.
-
-## OpenBSD Build VM
-
-A minimal OpenBSD QEMU/KVM VM on the laptop (earth) for native compilation.
-Scripts in `packages/buildvm/`.
-
-### Initial setup (once)
-
-```sh
-cd ~/git/conf/packages/buildvm
-
-# Fetch signify keys from fishfinger
-scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.sec .
-scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub .
-
-./setup.sh       # downloads ISO, runs fully automated install (~5 min)
-./provision.sh   # installs Go, git, gmake, signify keys, SSH keys
-```
-
-`setup.sh` is fully automated — it calls `install-expect.exp` which drives the
-OpenBSD serial console installer via expect. No manual interaction needed.
-
-### Day-to-day use
-
-```sh
-make buildvm-start   # boot VM (~15s)
-make dtail-openbsd   # auto-starts VM if needed
-make buildvm-stop    # shut down when done
-```
-
-The VM is headless, SSH on `localhost:2222`, 1GB RAM, 2 CPUs, 4GB disk.
-Username: `pbuild` (password: `build123`). SSH key installed by `provision.sh`.
-
-### Automated installer notes (install-expect.exp)
-
-- The expect script is in a **separate file** (`install-expect.exp`) to avoid bash/expect quoting interactions — embedding expect in a bash heredoc causes password send failures
-- Serial console is activated via `set tty com0` at the OpenBSD boot prompt
-- Password prompts require `sleep 2` before `send` — `sleep 1` is not enough for the serial console to be ready
-- OpenBSD 7.8 added a new "Encrypt the root disk?" prompt before the partition layout
-- After CONGRATULATIONS, choose shell (`s`) instead of reboot — the CD is still attached and reboot re-boots the installer; configure wheel group via `chroot /mnt`
-- Username `build` is rejected by the installer ("not a usable loginname") — use `pbuild` instead
-- Source code is synced to the VM via `git archive HEAD | ssh ... tar -x` to avoid filling /tmp (full repo with benchmarks exceeds available space on 4GB disk)
-
-## Building Custom Packages
-
-### FreeBSD
-
-On any FreeBSD host, or via the Mage SCP-and-remote-execute pattern:
-
-```sh
-pkg create -M +MANIFEST -p plist -r stagedir -o output/All
-pkg repo output/   # regenerates repo metadata (unsigned)
-```
-
-Copy output to PV:
-```sh
-doas cp -Rf output/* /data/nfs/k3svolumes/pkgrepo/freebsd/FreeBSD:15:amd64/latest/
-```
-
-### OpenBSD
-
-On an OpenBSD host:
-
-```sh
-pkg_create \
-    -D COMMENT="Package description" \
-    -d descfile \
-    -f packing-list \
-    -B stagedir \
-    -p / \
-    output/package-name-1.0.tgz
-```
-
-Copy to PV via f0:
-```sh
-scp package.tgz f0.lan.buetow.org:/tmp/
-ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package.tgz /data/nfs/k3svolumes/pkgrepo/openbsd/7.8/packages/amd64/"
-```
-
-## SSH Access for Package Tasks
-
-| Host | Access | Root | Notes |
-|------|--------|------|-------|
-| f0 (FreeBSD, NFS) | `ssh -p 22 f0.lan.buetow.org` | `doas` | PV is local, default shell is csh |
-| fishfinger (OpenBSD) | `ssh rex@fishfinger.buetow.org` | `doas` | Packages built here, then copied to f0 PV |
-| blowfish (OpenBSD) | `ssh rex@blowfish.buetow.org` | `doas` | Same setup as fishfinger |
-
-## Notes
-
-- HTTP is always redirected to HTTPS by the OpenBSD gateways — client URLs must use `https://`
-- OpenBSD version in the path must match the host OS version (currently 7.8)
-- FreeBSD version follows the ABI naming convention (`FreeBSD:15:amd64`)
-- OpenBSD packages are signed with signify (key pair at `/etc/signify/custom-pkg.*`)
-- FreeBSD packages are unsigned (`signature_type: "NONE"`) — signing with RSA is possible but not yet configured
-- After adding/removing FreeBSD packages, always regenerate repo metadata with `pkg repo`
-- OpenBSD does not need a repo index — `pkg_add` fetches packages directly by name
-- FreeBSD hosts use csh as default shell — remote scripts must use `/bin/sh` explicitly
-- f3 is currently offline; configure its custom repo when it comes back online
diff --git a/prompts/skills/pkgrepo/references/packaging-workflow.md b/prompts/skills/pkgrepo/references/packaging-workflow.md
new file mode 100644
index 0000000..d180be3
--- /dev/null
+++ b/prompts/skills/pkgrepo/references/packaging-workflow.md
@@ -0,0 +1,95 @@
+# Packaging Workflow
+
+Build scripts live in `~/git/conf/packages/`. The Makefile cross-compiles Go binaries on Linux, ships them to target hosts for native packaging, and uploads to the PV.
+
+## Single-Binary Go Packages
+
+For pure Go packages (no CGo), cross-compilation from Linux works for both FreeBSD and OpenBSD.
+
+### Build and upload
+
+```sh
+cd ~/git/conf/packages
+
+# Both FreeBSD and OpenBSD
+make pkg NAME=gogios SRC=/home/paul/git/gogios \
+    COMMENT="Monitoring tool with email alerts and HTML status page" \
+    DESC="Gogios is a lightweight monitoring tool written in Go."
+
+# Single OS
+make pkg-freebsd NAME=gogios SRC=/home/paul/git/gogios
+make pkg-openbsd NAME=gogios SRC=/home/paul/git/gogios
+```
+
+### How it works
+
+1. Cross-compiles on Linux (`GOOS=freebsd/openbsd GOARCH=amd64`)
+2. SCPs binary + packaging script to target host (f0 for FreeBSD, fishfinger for OpenBSD)
+3. Runs the packaging script via SSH (`pkg create` / `pkg_create`)
+4. OpenBSD packages are signed with signify automatically
+5. FreeBSD repo metadata is regenerated with `pkg repo`
+6. Packages are copied to the PV at `/data/nfs/k3svolumes/pkgrepo/`
+
+### Required Makefile variables
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `NAME` | Package name | `gogios` |
+| `SRC` | Go project root (must have `cmd/<NAME>/main.go` and `internal/version.go`) | `/home/paul/git/gogios` |
+
+### Optional Makefile variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `COMMENT` | `$(NAME)` | One-line package description |
+| `DESC` | `$(NAME)` | Longer description |
+| `MAINTAINER` | `paul@buetow.org` | Maintainer email |
+| `WWW` | `https://buetow.org` | Project URL |
+| `ENTRY` | `cmd/$(NAME)/main.go` | Go main package path relative to SRC |
+
+### Version detection
+
+Version is read automatically from `$(SRC)/internal/version.go` — expects a `Version` constant like `const Version = "v1.4.1"`. The `v` prefix is stripped.
+
+## CGo Packages
+
+Cross-compilation from Linux fails for CGo (e.g. packages with DataDog/zstd). Use native builds instead:
+
+- **OpenBSD**: native build on the local QEMU/KVM build VM (see [openbsd-build-vm.md](openbsd-build-vm.md))
+- **FreeBSD**: cross-compile with `CGO_ENABLED=0 -tags nozstd` — disables zstd support but allows static cross-compile
+- **Rocky Linux**: built locally on earth (x86_64) and on pi0 (aarch64 via rpmbuild)
+
+## Manual Packaging Reference
+
+### FreeBSD (on f0)
+
+```sh
+pkg create -M +MANIFEST -p plist -r stagedir -o output/All
+pkg repo output/   # regenerates repo metadata
+doas cp -Rf output/* /data/nfs/k3svolumes/pkgrepo/freebsd/FreeBSD:15:amd64/latest/
+```
+
+### OpenBSD (on fishfinger)
+
+```sh
+pkg_create \
+    -D COMMENT="Package description" \
+    -d descfile \
+    -f packing-list \
+    -B stagedir \
+    -p / \
+    output/package-name-1.0.tgz
+# Copy to PV via f0
+scp package.tgz f0.lan.buetow.org:/tmp/
+ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package.tgz /data/nfs/k3svolumes/pkgrepo/openbsd/7.8/packages/amd64/"
+```
+
+## Install/Update on Frontends via Rex
+
+```sh
+cd ~/git/conf/frontends
+rex gogios_install   # installs or updates gogios on blowfish + fishfinger (OpenBSD) and f0-f3 (FreeBSD)
+rex gogios           # full setup: gogios_install + config + cron
+```
+
+The `gogios_install` Rex task auto-detects the OS and uses `pkg install` (FreeBSD) or `pkg_add` (OpenBSD).
diff --git a/prompts/skills/pkgrepo/references/repo-architecture.md b/prompts/skills/pkgrepo/references/repo-architecture.md
new file mode 100644
index 0000000..0d36fa6
--- /dev/null
+++ b/prompts/skills/pkgrepo/references/repo-architecture.md
@@ -0,0 +1,73 @@
+# Repo Architecture
+
+Custom FreeBSD, OpenBSD, and Rocky Linux package repository served from k3s.
+
+## Overview
+
+- **nginx pod** in k3s `infra` namespace serves static files from a PV
+- URL: `https://pkgrepo.f3s.buetow.org`
+- Path prefixes: `/freebsd/`, `/openbsd/`, `/rockylinux/`
+- TLS terminated by OpenBSD relayd on the internet gateways (not in the pod)
+- DNS, ACME certs, httpd fallback, and relayd routing auto-generated from `@f3s_hosts` in `frontends/Rexfile`
+- HTTP always redirected to HTTPS by the OpenBSD gateways — client URLs must use `https://`
+
+## PV Directory Structure
+
+```
+/data/nfs/k3svolumes/pkgrepo/       ← NFS-backed PV, physically on f0
+  freebsd/
+    FreeBSD:15:amd64/
+      latest/
+        All/              # .pkg files
+        packagesite.pkg   # repo metadata (generated by `pkg repo`)
+        meta.conf / meta
+        data.pkg
+  openbsd/
+    7.8/
+      packages/
+        amd64/            # .tgz files (signify-signed)
+  rockylinux/
+    9/
+      x86_64/             # .rpm files + repodata/
+      aarch64/            # .rpm files + repodata/
+```
+
+## Key Config Files
+
+| File | Purpose |
+|------|---------|
+| `f3s/pkgrepo/helm-chart/` | Helm chart (nginx deployment, service, ingress, PV, configmap) |
+| `f3s/argocd-apps/infra/pkgrepo.yaml` | ArgoCD Application manifest |
+| `f3s/pkgrepo/test-artifacts/build-test-packages.sh` | Builds hello-test packages for FreeBSD/OpenBSD |
+| `frontends/Rexfile` (`@f3s_hosts`) | DNS + routing entry for `pkgrepo.f3s.buetow.org` |
+| `frontends/Rexfile` (`pkgrepo_setup`) | Adds `PKG_PATH` to root's `.profile` on OpenBSD frontends |
+| `packages/Makefile` | Cross-compiles, packages, and uploads for all OSes |
+| `packages/scripts/pkg-freebsd.sh` | Runs on f0 via SSH: `pkg create` + `pkg repo` + copy to PV |
+| `packages/scripts/pkg-openbsd.sh` | Runs on fishfinger via SSH: `pkg_create` + signify signing |
+| `packages/scripts/pkg-dtail-openbsd.sh` | DTail multi-binary OpenBSD packaging |
+| `packages/scripts/pkg-dtail-freebsd.sh` | DTail multi-binary FreeBSD packaging |
+| `packages/scripts/pkg-dtail-rpm.sh` | Builds DTail RPMs from prebuilt or locally built payloads |
+
+## SSH Access for Package Tasks
+
+| Host | Access | Root | Notes |
+|------|--------|------|-------|
+| f0 (FreeBSD, NFS) | `ssh -p 22 f0.lan.buetow.org` | `doas` | PV is local; default shell is **csh** |
+| fishfinger (OpenBSD) | `ssh rex@fishfinger.buetow.org` | `doas` | OpenBSD packages built here, then copied to f0 PV |
+| blowfish (OpenBSD) | `ssh rex@blowfish.buetow.org` | `doas` | Same setup as fishfinger |
+
+## Nginx Pod Gotcha — Stale NFS Handle
+
+After writing new files to the PV on f0, the nginx pod can get a stale NFS file handle (error 116) and return HTTP 500. Fix with a pod restart:
+
+```sh
+kubectl -n infra rollout restart deployment/pkgrepo
+kubectl -n infra rollout status deployment/pkgrepo
+```
+
+## Per-OS Repo Notes
+
+- **FreeBSD**: version follows ABI naming (`FreeBSD:15:amd64`); packages unsigned (`signature_type: "NONE"`); always regenerate metadata with `pkg repo` after adding/removing packages
+- **OpenBSD**: no repo index needed — `pkg_add` fetches by name; packages signed with signify; version in path must match host OS (currently 7.8)
+- **Rocky Linux**: standard DNF layout; unsigned repo (`gpgcheck=0`); architecture-specific paths (`x86_64` / `aarch64`)
+- **FreeBSD csh**: default shell is csh — avoid inline one-liners with `||`, `&&`, `!`, or multi-line quoting over SSH; use piped `/bin/sh` or separate SSH invocations