skills: rename pkgrepo -> f3s-pkgrepo and rocky-vm-setup -> f3s-rocky-vm-setup

Extend the f3s- prefix to the two remaining f3s-scoped siblings so the whole
homelab family groups together. Both are f3s-specific: pkgrepo IS
pkgrepo.f3s.buetow.org, and rocky-vm-setup documents the 'rocky' VM on the f3s
host f3.

- git mv both skill dirs (history preserved); update name: frontmatter; add the
  'Part of the f3s homelab skill family' note to both descriptions.
- Rewrite every reference to the *skills*: backtick mentions (`pkgrepo` ->
  `f3s-pkgrepo`, `rocky-vm-setup` -> `f3s-rocky-vm-setup`), markdown cross-links
  (../pkgrepo/ -> ../f3s-pkgrepo/, etc.), the f3s hub Related-skills block and
  description, and skill-maintenance's index-pattern example.
- Left references to the package REPOSITORY (pkgrepo.f3s.buetow.org, 'the custom
  pkgrepo') unchanged — those name a service, not the skill.
- Verified all 114 relative links resolve. Plan doc updated to record the rename.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

6 files changed, 639 insertions, 0 deletions

diff --git a/prompts/skills/f3s-pkgrepo/SKILL.md b/prompts/skills/f3s-pkgrepo/SKILL.md
new file mode 100644
index 0000000..262a6e7
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/SKILL.md
@@ -0,0 +1,37 @@
+---
+name: f3s-pkgrepo
+description: Reference skill for the homelab package repositories behind pkgrepo.f3s.buetow.org. Covers FreeBSD pkg, OpenBSD pkg_add, NetBSD pkg_add, Rocky Linux dnf repos, PV layout, nginx/ArgoCD wiring, DTail package publishing, and client repo configuration. Use when publishing packages, troubleshooting repository access or metadata, or deciding how packages should be installed from the custom repo. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)).
+---
+
+# Package Repo Reference
+
+Use this skill when the task is specifically about the custom package repositories rather than the broader f3s homelab.
+
+## When to Use
+
+- Publishing or updating packages in `pkgrepo.f3s.buetow.org`
+- Troubleshooting repo layout, metadata, or HTTP exposure
+- Configuring FreeBSD, OpenBSD, NetBSD, or Rocky Linux clients to install from the custom repo
+- Working on DTail package publishing and installation through the repo
+
+## Reference Files
+
+- [Repo Architecture](references/repo-architecture.md) — nginx/k3s setup, PV directory structure, SSH access, stale NFS handle fix, per-OS repo notes
+- [Client Setup](references/client-setup.md) — per-OS client repo configuration (FreeBSD, OpenBSD, NetBSD, Rocky Linux), new-host setup, package signing
+- [Packaging Workflow](references/packaging-workflow.md) — Makefile workflow for single-binary Go packages, CGo packages, manual packaging reference
+- [DTail Package](references/dtail-package.md) — multi-binary DTail package for all platforms, install/update steps, gotchas, client usage, verification
+- [OpenBSD Build VM](references/openbsd-build-vm.md) — QEMU/KVM build VM for native CGo compilation, day-to-day use, installer notes
+
+## Scope
+
+This skill owns package repository details that used to live under `f3s`.
+
+Use `f3s` alongside this skill when the task depends on broader host-role or cluster context, especially:
+
+- `f0` as the FreeBSD NFS/PV host for `/data/nfs/k3svolumes/pkgrepo/`
+- `fishfinger` and `blowfish` as the OpenBSD frontend hosts
+- `r0-r2` as Rocky Linux x86_64 bhyve VMs
+- `pi2-pi3` as Rocky Linux aarch64 Raspberry Pi nodes
+- `pi0`/`pi1` as NetBSD aarch64 Raspberry Pi nodes (see the `f3s-raspberry-pi` skill's `bootstrap-netbsd-pi.md`) — NetBSD repo clients and the native NetBSD package build host (`pi0`)
+- `earth` as the Fedora laptop used for package publication and verification
+- `f0-f3` as FreeBSD hosts
diff --git a/prompts/skills/f3s-pkgrepo/references/client-setup.md b/prompts/skills/f3s-pkgrepo/references/client-setup.md
new file mode 100644
index 0000000..1e29c4e
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/references/client-setup.md
@@ -0,0 +1,141 @@
+# Client Setup
+
+How to configure each OS to install packages from `pkgrepo.f3s.buetow.org`.
+
+## FreeBSD (f0–f3)
+
+Custom repo is configured alongside the official FreeBSD repos.
+
+File: `/usr/local/etc/pkg/repos/custom.conf`
+
+```
+custom: {
+  url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
+  mirror_type: "NONE",
+  signature_type: "NONE",
+  enabled: yes
+}
+```
+
+### Using packages
+
+```sh
+doas pkg update
+doas pkg install <package-name>
+doas pkg upgrade                  # upgrade all, including custom packages
+doas pkg install -fy <package>    # force reinstall (same version)
+```
+
+### Setting up a new FreeBSD host
+
+Pipe via stdin to avoid csh quoting issues:
+
+```sh
+cat <<'REPO' | ssh -p 22 <host>.lan.buetow.org 'doas mkdir -p /usr/local/etc/pkg/repos && doas tee /usr/local/etc/pkg/repos/custom.conf > /dev/null'
+custom: {
+  url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
+  mirror_type: "NONE",
+  signature_type: "NONE",
+  enabled: yes
+}
+REPO
+```
+
+## OpenBSD (blowfish, fishfinger)
+
+Custom repo is configured via `PKG_PATH` in `/root/.profile`, deployed by `rex pkgrepo_setup`.
+Official OpenBSD packages still install normally via `/etc/installurl`.
+
+```sh
+export PKG_PATH="https://pkgrepo.f3s.buetow.org/openbsd/7.8/packages/amd64/"
+```
+
+### Using packages
+
+```sh
+doas pkg_add <package>       # installs from custom repo (signed with signify)
+doas pkg_add -u <package>    # update to latest version
+```
+
+### Setting up a new OpenBSD host
+
+Two things needed:
+
+1. **Signify public key** — copy from an existing host:
+   ```sh
+   scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub /tmp/
+   scp /tmp/custom-pkg.pub rex@<newhost>:/tmp/
+   ssh rex@<newhost> "doas cp /tmp/custom-pkg.pub /etc/signify/custom-pkg.pub"
+   ```
+
+2. **PKG_PATH** — run the Rex task from `~/git/conf/frontends`:
+   ```sh
+   rex pkgrepo_setup   # adds PKG_PATH to /root/.profile on all frontends
+   ```
+
+Update `PKG_PATH` whenever the OpenBSD version changes (currently 7.8).
+
+### Package signing
+
+OpenBSD packages are signed with `signify(1)` via `pkg_sign`:
+- **Private key**: `/etc/signify/custom-pkg.sec` on fishfinger (build host)
+- **Public key**: `/etc/signify/custom-pkg.pub` on all OpenBSD clients
+- Signing happens automatically during `make pkg-openbsd` / `make pkg`
+- `pkg_add` verifies the signature — no `-D unsigned` needed
+
+## NetBSD (pi0, pi1)
+
+No persistent repo config needed — install/update straight from the package URL (repo is unsigned):
+
+```sh
+export PATH=/usr/sbin:$PATH   # pkg_* live in /usr/sbin, not in the default non-interactive PATH
+doas pkg_add https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz    # first install
+doas pkg_add -u https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz # update to newer version
+```
+
+Notes:
+- Same-version reinstall: `pkg_add -u` skips it — `doas pkg_delete dtail` first, then `pkg_add` again
+- `pkg_summary.gz` is published alongside the packages, so the repo URL could also be added to `/usr/pkg/etc/pkgin/repositories.conf` for pkgin; today plain `pkg_add` is used
+- Version in the URL path must match the host OS release (currently 10.1) and the arch is `aarch64` (`uname -p`)
+- pkgsrc packages from cdn.netbsd.org still install normally via pkgin — the custom repo lives under a different prefix (`/` with files in `/usr/local`, `/etc`) and doesn't conflict
+
+## Rocky Linux (r0–r2, pi2–pi3)
+
+`pi0`/`pi1` run NetBSD (see above and the `f3s-raspberry-pi` skill's `bootstrap-netbsd-pi.md`) and are not Rocky clients of this repo.
+
+Architecture-specific repo URLs:
+- `https://pkgrepo.f3s.buetow.org/rockylinux/9/x86_64/`  (r0–r2)
+- `https://pkgrepo.f3s.buetow.org/rockylinux/9/aarch64/` (pi2–pi3)
+
+### Persistent repo file
+
+Create `/etc/yum.repos.d/f3s-dtail.repo`:
+
+```ini
+[f3s-dtail]
+name=f3s DTail
+baseurl=https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/
+enabled=1
+gpgcheck=0
+repo_gpgcheck=0
+```
+
+Then:
+
+```sh
+sudo dnf makecache
+sudo dnf install dtail
+sudo dnf upgrade dtail
+```
+
+### Temporary one-off usage (no persistent repo file needed)
+
+```sh
+sudo dnf repoquery \
+  --disablerepo='*' \
+  --repofrompath=f3s-dtail,https://pkgrepo.f3s.buetow.org/rockylinux/9/$(uname -m)/ \
+  --enablerepo=f3s-dtail \
+  dtail
+```
+
+Do **not** pass `--repofrompath` when the persistent repo file already exists — dnf errors with "listed more than once".
diff --git a/prompts/skills/f3s-pkgrepo/references/dtail-package.md b/prompts/skills/f3s-pkgrepo/references/dtail-package.md
new file mode 100644
index 0000000..fbdd8d6
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/references/dtail-package.md
@@ -0,0 +1,219 @@
+# DTail Package
+
+DTail is a multi-binary package (6 binaries + config + service script). There are separate targets for each OS.
+
+## Build Commands
+
+```sh
+cd ~/git/conf/packages
+make dtail-openbsd   # OpenBSD: native build on QEMU/KVM VM (CGo/zstd supported)
+make dtail-freebsd   # FreeBSD: cross-compiled on Linux (CGO_ENABLED=0, nozstd — .zst logs unsupported)
+make dtail-netbsd    # NetBSD/aarch64: cross-compiled on Linux (CGO_ENABLED=0, nozstd), packaged natively on pi0
+make dtail-rocky     # Rocky Linux: x86_64 + aarch64 RPMs + repodata
+```
+
+## Package Contents by OS
+
+### OpenBSD (blowfish, fishfinger)
+
+| File | Source template |
+|------|----------------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | built natively on build VM |
+| `/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail.json.tpl` (absolute `CacheDir: /var/run/dserver/cache`; `HostKeyFile` in persistent `/var/db/dserver/ssh_host_key`) |
+| `/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver.tpl` |
+| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache.sh.tpl` (ksh) |
+
+OpenBSD notes:
+- Packages before 2026-07-10 used relative `CacheDir: "cache"` / `HostKeyFile: "cache/ssh_host_key"` — that only worked because rc.d starts the daemon via `su -l _dserver` (CWD = `/var/run/dserver`, the `_dserver` home dir); a manual start from any other directory broke public key lookup — the same bug class dtail commit `fec2f9d` fixed on the server side (making absolute `CacheDir` paths resolve independently of dserver's CWD)
+- OpenBSD's `/etc/rc` wipes `/var/run/*` at boot, so the SSH host key now lives in persistent `/var/db/dserver/ssh_host_key` (mirrors NetBSD). The rc.d `rc_pre` recreates `/var/run/dserver/cache` and `/var/db/dserver` and re-runs `dserver-update-key-cache.sh` on every start; the daily cron entry keeps it fresh afterwards
+- When upgrading a host from a pre-2026-07-10 package, follow this order — neither `pkg_add` nor Rex restarts a running daemon, and once the host reboots `/etc/rc` has already wiped the old host key, making migration impossible:
+  1. Migrate the host key FIRST to preserve the host identity: `doas install -d -o _dserver -m 0700 /var/db/dserver && doas cp -p /var/run/dserver/cache/ssh_host_key /var/db/dserver/` — otherwise dserver generates a new host key and clients without `--trustAllHosts` must re-accept it
+  2. Reinstall the package (same-version: `pkg_add -u` is a no-op — `doas pkg_delete dtail` then `doas env PKG_PATH=... pkg_add dtail`)
+  3. `doas rcctl restart dserver` — without this the running daemon keeps the old relative-path config until reboot
+  (done on fishfinger 2026-07-10; **blowfish still runs the pre-2026-07-10 package**)
+- Since 2026-07-10 the rc.d script sets `rc_bg=YES` and calls `rc_cmd "$1"` normally — dserver does not daemonize itself, and rc.subr's `rc_bg` is the proper way to background it. Earlier packages backgrounded the whole framework (`rc_cmd $1 &`), which hid `rc_pre` failures and the start result from `rcctl`
+- From the WireGuard VPN, `f0.lan.buetow.org` may not route — run the Makefile with `make dtail-openbsd FREEBSD_HOST=f0.wg0` (same override applies to the other `dtail-*` targets, which upload via f0)
+
+### FreeBSD (f0–f3)
+
+| File | Source template |
+|------|----------------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | cross-compiled `GOOS=freebsd CGO_ENABLED=0 -tags nozstd` |
+| `/usr/local/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail-freebsd.json.tpl` |
+| `/usr/local/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver-freebsd.tpl` |
+| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache-freebsd.sh.tpl` (sh) |
+
+**FreeBSD config note:** `dtail-freebsd.json.tpl` uses **absolute paths** for `CacheDir` and `HostKeyFile`. FreeBSD's `daemon(8)` resets CWD to `/`, so the relative `"cache"` in the standard template resolves to `/cache` — silently breaking key lookup. (Since dtail commit `fec2f9d`, absolute `CacheDir` paths also resolve independently of the CWD dserver was started from — before that fix, a manual service restart from a home directory broke public key auth.)
+
+FreeBSD notes:
+- FreeBSD's `cleanvar` purges `/var/run` at boot, so the SSH host key lives in persistent `/var/db/dserver/ssh_host_key` (mirrors NetBSD/OpenBSD). The rc.d `start_precmd` recreates `/var/run/dserver/cache` and `/var/db/dserver` and re-runs `dserver-update-key-cache.sh` on every start; the daily periodic job keeps it fresh afterwards
+- Packages before 2026-07-10 kept the host key in volatile `/var/run/dserver/cache/ssh_host_key` — it regenerated on every reboot. When upgrading a host from such a package, migrate the key FIRST (before restarting dserver) to preserve the host identity:
+  1. `doas install -d -o dserver -m 0700 /var/db/dserver && doas cp -p /var/run/dserver/cache/ssh_host_key /var/db/dserver/ssh_host_key`
+  2. `doas pkg update -f && doas pkg install -fy dtail` (same-version force reinstall)
+  3. `doas service dserver restart`
+  (done on f0/f1/f2 2026-07-10 — sha256 of the migrated key verified identical; **f3 was unreachable and still runs the pre-2026-07-10 package**)
+
+### NetBSD (pi0, pi1 — aarch64)
+
+Package name is `dtail-4.3.2ng` — NetBSD versions must not contain dashes, so `-ng` becomes `ng`.
+
+| File | Source template |
+|------|----------------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | cross-compiled `GOOS=netbsd GOARCH=arm64 CGO_ENABLED=0 -tags nozstd` |
+| `/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail-netbsd.json.tpl` (absolute `CacheDir` like FreeBSD; `HostKeyFile` in persistent `/var/db/dserver/ssh_host_key`) |
+| `/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver-netbsd.tpl` |
+| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache-netbsd.sh.tpl` (sh) |
+
+NetBSD notes:
+- `pkg_create` runs natively on pi0 (Makefile ships binaries + templates there via SSH and runs `packages/scripts/pkg-dtail-netbsd.sh`); `pkg_summary.gz` for pkgin is generated and uploaded alongside
+- NetBSD has no `daemon(8)` and dserver doesn't daemonize — the rc.d script backgrounds it via `command_args="... &"` and runs it as user `dserver` (`dserver_user`)
+- `/var/run` is volatile — the rc.d `start_precmd` recreates `/var/run/dserver/cache` and re-runs the key-cache helper on every start; a daily root cron entry (`dserver-update-key-cache.sh`) keeps it fresh
+- The SSH host key lives in persistent `/var/db/dserver/ssh_host_key` (created by the precmd) so it survives reboots — since 2026-07-10 FreeBSD uses the same layout. Packages before 2026-07-09 used the volatile path; after upgrading, DTail clients without `--trustAllHosts` must re-accept the host key once
+- npf firewall needs `pass stateful in final family inet4 proto tcp to $ext_if port 2222` in the `"external"` group of `/etc/npf.conf`
+
+### Rocky Linux (r0–r2 amd64, pi2–pi3 aarch64)
+
+| File |
+|------|
+| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` |
+| `/usr/local/bin/dserver-update-key-cache.sh` |
+| `/etc/dserver/dtail.json` |
+| `/usr/lib/systemd/system/dserver.service` |
+| `/usr/lib/systemd/system/dserver-update-keycache.service` |
+| `/usr/lib/systemd/system/dserver-update-keycache.timer` |
+
+Rocky notes:
+- Key-cache helper handles both `/root/.ssh/authorized_keys` and `/home/*/.ssh/authorized_keys` — `root` works on r0–r2 without manual cache copy
+- `dserver.service` includes `RuntimeDirectory=dserver` and `ExecStartPre` to recreate `/var/run/dserver` (tmpfs) on Rocky
+- Repo is unsigned (`gpgcheck=0`)
+- `aarch64` RPM is built on pi2 — Fedora's rpmbuild refuses to emit `aarch64` binary RPMs from an x86_64 host
+
+## Install / Update
+
+### OpenBSD (via Rex)
+
+```sh
+cd ~/git/conf/frontends
+rex dtail_install   # install or update from custom repo
+rex dtail           # full setup: install + _dserver user + daily cron + service start
+```
+
+### FreeBSD (manual, f0–f3)
+
+```sh
+# Custom repo must already be configured (see client-setup.md)
+doas pkg install dtail          # first install
+doas pkg install -fy dtail      # force reinstall (same version)
+
+# Create service user (once per host)
+doas pw useradd dserver -d /var/run/dserver -s /usr/sbin/nologin
+
+# Enable and start dserver
+doas sysrc dserver_enable=YES
+doas service dserver start
+
+# Key cache is populated by the rc.d start_precmd on every service start;
+# run manually only when authorized_keys changed without a restart:
+doas /usr/local/bin/dserver-update-key-cache.sh
+
+# Register daily key cache refresh
+doas mkdir -p /usr/local/etc/periodic/daily
+printf '%s\n%s\n' '#!/bin/sh' '/usr/local/bin/dserver-update-key-cache.sh' | \
+  doas tee /usr/local/etc/periodic/daily/200.dserver-update-key-cache > /dev/null
+doas chmod 755 /usr/local/etc/periodic/daily/200.dserver-update-key-cache
+```
+
+**FreeBSD gotchas:**
+- Since the 2026-07-10 package the rc.d `start_precmd` re-runs `dserver-update-key-cache.sh` on every start, so the key cache repopulates automatically after restart or reboot (older packages required a manual re-run after any restart)
+- `pkg install -fy` replaces `/usr/local/etc/dserver/dtail.json` with the package version; local customisations are lost
+- Avoid inline one-liners with `||`, `!`, or multi-quote strings over SSH to FreeBSD (csh) — pipe a script to `doas /bin/sh` instead or use separate SSH commands
+
+### NetBSD (manual, pi0–pi1)
+
+```sh
+# All as root via doas; pkg_* tools live in /usr/sbin (not in non-interactive SSH PATH)
+export PATH=/usr/sbin:$PATH
+
+# Service group + user (once per host)
+doas groupadd dserver
+doas useradd -g dserver -d /var/run/dserver -s /sbin/nologin -c "DTail server" dserver
+
+# Install / update from the custom repo
+doas pkg_add https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz
+doas pkg_add -u https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz  # newer version
+# Same-version reinstall: pkg_delete dtail first, then pkg_add
+
+# Enable and start (rc.d script ships in the package)
+doas sh -c 'echo dserver=YES >> /etc/rc.conf'   # once per host
+doas /etc/rc.d/dserver start
+
+# Open port 2222 (once per host): add to the "external" group in /etc/npf.conf:
+#   pass stateful in final family inet4 proto tcp to $ext_if port 2222
+# then: doas npfctl validate && doas npfctl reload
+
+# Daily key-cache refresh (once per host; rc.d start also refreshes it)
+# root crontab entry: 30 4 * * * /usr/local/bin/dserver-update-key-cache.sh >/dev/null 2>&1
+```
+
+**NetBSD gotchas:**
+- The package deliberately does not create the `dserver` user/group (matching the FreeBSD package) — run the `groupadd`/`useradd` step above before the first service start or the rc.d precmd fails
+- The key cache lives in volatile `/var/run` but the rc.d `start_precmd` recreates and repopulates it on every start — no manual re-run needed after restart or reboot
+- `dserver -version` panics when run as root (`Not allowed to run as UID 0`) — check with `su -m dserver -c '/usr/local/bin/dserver -version'` or as a normal user
+
+### Rocky Linux (dnf)
+
+```sh
+# On r0–r2 (root) or pi2–pi3 (paul with sudo):
+sudo dnf upgrade dtail
+sudo systemctl restart dserver
+sudo systemctl start dserver-update-keycache.service   # repopulate after restart
+```
+
+**Rocky gotcha:** A legacy `/etc/systemd/system/dserver-update-keycache.service` left from manual pre-RPM setup points to the old path `ExecStart=/var/run/dserver/update_key_cache.sh` and shadows the RPM-installed unit. Remove it on any host where the timer fails:
+
+```sh
+sudo rm -f /etc/systemd/system/dserver-update-keycache.service
+sudo systemctl daemon-reload
+sudo systemctl start dserver-update-keycache.service
+```
+
+This was cleaned up on all r0–r2 and pi0–pi3 on 2026-04-19.
+
+## Client Usage from earth
+
+```sh
+# OpenBSD frontends
+dcat --plain --noColor --trustAllHosts --user rex \
+  --servers blowfish.buetow.org,fishfinger.buetow.org --files /etc/fstab
+
+# FreeBSD hosts
+dcat --plain --noColor --trustAllHosts --user paul \
+  --servers f0.lan.buetow.org,f1.lan.buetow.org,f2.lan.buetow.org,f3.lan.buetow.org \
+  --files /etc/fstab
+
+# Rocky VMs (r0–r2, user root)
+dcat --plain --noColor --trustAllHosts --user root \
+  --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org --files /etc/fstab
+
+# Raspberry Pis Rocky (pi2–pi3, user paul)
+dcat --plain --noColor --trustAllHosts --user paul \
+  --servers pi2.lan.buetow.org,pi3.lan.buetow.org \
+  --files /etc/fstab
+
+# Raspberry Pis NetBSD (pi0–pi1, user paul)
+dcat --plain --noColor --trustAllHosts --user paul \
+  --servers pi0.lan.buetow.org,pi1.lan.buetow.org \
+  --files /etc/fstab
+```
+
+## Verification State
+
+| Date | Platform | Result |
+|------|----------|--------|
+| 2026-04-19 | FreeBSD f0–f3 | `dtail-4.3.2-ng` installed, dserver running under `daemon(8)`, `dcat /etc/fstab` ✓ (`--user paul`) |
+| 2026-04-19 | OpenBSD blowfish, fishfinger | `dtail-4.3.2-ng` current, `dcat /etc/fstab` ✓ (`--user rex`) |
+| 2026-04-19 | Rocky r0–r2 | `dtail-4.3.2-ng` current, dserver running, `dcat /etc/fstab` ✓ (`--user root`) |
+| 2026-04-19 | Rocky pi0–pi3 | `dtail-4.3.2-ng` current, dserver running, `dcat /etc/fstab` ✓ (`--user paul`) — pi0/pi1 since re-imaged to NetBSD |
+| 2026-07-09 | NetBSD pi0–pi1 | `dtail-4.3.2ng` installed (first NetBSD deployment), dserver running as `dserver` on 2222, `dcat /etc/fstab` ✓ (`--user paul`) |
+| 2026-07-10 | FreeBSD f0–f2 | `dtail-4.3.2-ng` reinstalled with persistent host key (`/var/db/dserver`) — key migrated first, sha256 identical, mtime preserved; rc.d precmd repopulates key cache; `dcat /etc/fstab` ✓ (`--user paul`). **f3 unreachable (no route via wg0) — still on the pre-2026-07-10 package, migrate + upgrade when back** |
+| 2026-07-10 | OpenBSD fishfinger | `dtail-4.3.2-ng` reinstalled (rc_bg rc.d + hardened key-cache script); host key sha256/mtime unchanged, `rcctl restart` reports ok, `dcat /etc/fstab` ✓ (`--user rex`). blowfish still on the older package |
diff --git a/prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md b/prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md
new file mode 100644
index 0000000..2c2831f
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md
@@ -0,0 +1,42 @@
+# OpenBSD Build VM
+
+A minimal OpenBSD QEMU/KVM VM on earth for native compilation of CGo packages (e.g. dtail with DataDog/zstd). Cross-compiling CGo from Linux to OpenBSD needs a C cross-compiler; native build sidesteps that entirely.
+
+Scripts in `~/git/conf/packages/buildvm/`.
+
+## Initial Setup (once)
+
+```sh
+cd ~/git/conf/packages/buildvm
+
+# Fetch signify keys from fishfinger
+scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.sec .
+scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub .
+
+./setup.sh       # downloads ISO, runs fully automated install (~5 min)
+./provision.sh   # installs Go, git, gmake, signify keys, SSH keys
+```
+
+`setup.sh` is fully automated via `install-expect.exp` — drives the OpenBSD serial console installer without manual interaction.
+
+## Day-to-Day Use
+
+```sh
+make buildvm-start   # boot VM (~15s)
+make dtail-openbsd   # auto-starts VM if needed, then builds
+make buildvm-stop    # shut down when done
+```
+
+VM specs: headless, SSH on `localhost:2222`, 1 GB RAM, 2 CPUs, 4 GB disk.
+Username: `pbuild` (password: `build123`). SSH key installed by `provision.sh`.
+
+Source is synced via `git archive HEAD | ssh ... tar -x` — not `scp -r` — to avoid filling `/tmp` with build artifacts and test data (full repo with benchmarks exceeds the 4 GB disk).
+
+## Installer Notes (install-expect.exp)
+
+- Expect script is a **separate file** to avoid bash/expect quoting interactions — embedding it in a bash heredoc breaks password sends
+- Serial console activated via `set tty com0` at the OpenBSD boot prompt
+- Password prompts need `sleep 2` before `send` — `sleep 1` is not enough for the serial console
+- OpenBSD 7.8 added an "Encrypt the root disk?" prompt before the partition layout
+- After CONGRATULATIONS, choose shell (`s`) not reboot — the CD is still attached; configure wheel group via `chroot /mnt`
+- Username `build` is rejected ("not a usable loginname") — use `pbuild`
diff --git a/prompts/skills/f3s-pkgrepo/references/packaging-workflow.md b/prompts/skills/f3s-pkgrepo/references/packaging-workflow.md
new file mode 100644
index 0000000..d2b7388
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/references/packaging-workflow.md
@@ -0,0 +1,120 @@
+# Packaging Workflow
+
+Build scripts live in `~/git/conf/packages/`. The Makefile cross-compiles Go binaries on Linux, ships them to target hosts for native packaging, and uploads to the PV.
+
+## Single-Binary Go Packages
+
+For pure Go packages (no CGo), cross-compilation from Linux works for both FreeBSD and OpenBSD.
+
+### Build and upload
+
+```sh
+cd ~/git/conf/packages
+
+# Both FreeBSD and OpenBSD
+make pkg NAME=gogios SRC=/home/paul/git/gogios \
+    COMMENT="Monitoring tool with email alerts and HTML status page" \
+    DESC="Gogios is a lightweight monitoring tool written in Go."
+
+# Single OS
+make pkg-freebsd NAME=gogios SRC=/home/paul/git/gogios
+make pkg-openbsd NAME=gogios SRC=/home/paul/git/gogios
+```
+
+### How it works
+
+1. Cross-compiles on Linux (`GOOS=freebsd/openbsd GOARCH=amd64`)
+2. SCPs binary + packaging script to target host (f0 for FreeBSD, fishfinger for OpenBSD)
+3. Runs the packaging script via SSH (`pkg create` / `pkg_create`)
+4. OpenBSD packages are signed with signify automatically
+5. FreeBSD repo metadata is regenerated with `pkg repo`
+6. Packages are copied to the PV at `/data/nfs/k3svolumes/pkgrepo/`
+
+### Required Makefile variables
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `NAME` | Package name | `gogios` |
+| `SRC` | Go project root (must have `cmd/<NAME>/main.go` and `internal/version.go`) | `/home/paul/git/gogios` |
+
+### Optional Makefile variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `COMMENT` | `$(NAME)` | One-line package description |
+| `DESC` | `$(NAME)` | Longer description |
+| `MAINTAINER` | `paul@buetow.org` | Maintainer email |
+| `WWW` | `https://buetow.org` | Project URL |
+| `ENTRY` | `cmd/$(NAME)/main.go` | Go main package path relative to SRC |
+
+### Version detection
+
+Version is read automatically from `$(SRC)/internal/version.go` — expects a `Version` constant like `const Version = "v1.4.1"`. The `v` prefix is stripped.
+
+## CGo Packages
+
+Cross-compilation from Linux fails for CGo (e.g. packages with DataDog/zstd). Use native builds instead:
+
+- **OpenBSD**: native build on the local QEMU/KVM build VM (see [openbsd-build-vm.md](openbsd-build-vm.md))
+- **FreeBSD**: cross-compile with `CGO_ENABLED=0 -tags nozstd` — disables zstd support but allows static cross-compile
+- **NetBSD**: cross-compile with `CGO_ENABLED=0 GOOS=netbsd GOARCH=arm64 -tags nozstd`; packaging (`pkg_create`) runs natively on pi0 because `+BUILD_INFO` must match the target host
+- **Rocky Linux**: built locally on earth (x86_64) and on pi2 (aarch64 via rpmbuild)
+
+## Manual Packaging Reference
+
+### FreeBSD (on f0)
+
+```sh
+pkg create -M +MANIFEST -p plist -r stagedir -o output/All
+pkg repo output/   # regenerates repo metadata
+doas cp -Rf output/* /data/nfs/k3svolumes/pkgrepo/freebsd/FreeBSD:15:amd64/latest/
+```
+
+### OpenBSD (on fishfinger)
+
+```sh
+pkg_create \
+    -D COMMENT="Package description" \
+    -d descfile \
+    -f packing-list \
+    -B stagedir \
+    -p / \
+    output/package-name-1.0.tgz
+# Copy to PV via f0
+scp package.tgz f0.lan.buetow.org:/tmp/
+ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package.tgz /data/nfs/k3svolumes/pkgrepo/openbsd/7.8/packages/amd64/"
+```
+
+### NetBSD (on pi0)
+
+```sh
+# pkg_* tools are in /usr/sbin (not in the non-interactive SSH PATH)
+# Files staged under stagedir (-p), installed relative to / (-I);
+# @owner root / @group wheel in the packing list keeps installed files root-owned.
+/usr/sbin/pkg_create \
+    -B build-info \
+    -c commentfile \
+    -d descfile \
+    -f packing-list \
+    -I / \
+    -p stagedir \
+    output/package-name-1.0.tgz
+# pkg_summary.gz enables pkgin (pkg_add alone doesn't need it)
+/usr/sbin/pkg_info -X output/*.tgz | gzip -9 > output/pkg_summary.gz
+# Copy to PV via f0
+scp -P 22 output/* f0.lan.buetow.org:/tmp/
+ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package-name-1.0.tgz /tmp/pkg_summary.gz /data/nfs/k3svolumes/pkgrepo/netbsd/10.1/packages/aarch64/"
+```
+
+NetBSD package versions must not contain dashes (the last dash separates the
+package name from the version), so DTail's `4.3.2-ng` becomes `4.3.2ng`.
+
+## Install/Update on Frontends via Rex
+
+```sh
+cd ~/git/conf/frontends
+rex gogios_install   # installs or updates gogios on blowfish + fishfinger (OpenBSD) and f0-f3 (FreeBSD)
+rex gogios           # full setup: gogios_install + config + cron
+```
+
+The `gogios_install` Rex task auto-detects the OS and uses `pkg install` (FreeBSD) or `pkg_add` (OpenBSD).
diff --git a/prompts/skills/f3s-pkgrepo/references/repo-architecture.md b/prompts/skills/f3s-pkgrepo/references/repo-architecture.md
new file mode 100644
index 0000000..ae42e1f
--- /dev/null
+++ b/prompts/skills/f3s-pkgrepo/references/repo-architecture.md
@@ -0,0 +1,80 @@
+# Repo Architecture
+
+Custom FreeBSD, OpenBSD, NetBSD, and Rocky Linux package repository served from k3s.
+
+## Overview
+
+- **nginx pod** in k3s `infra` namespace serves static files from a PV
+- URL: `https://pkgrepo.f3s.buetow.org`
+- Path prefixes: `/freebsd/`, `/openbsd/`, `/netbsd/`, `/rockylinux/`
+- TLS terminated by OpenBSD relayd on the internet gateways (not in the pod)
+- DNS, ACME certs, httpd fallback, and relayd routing auto-generated from `@f3s_hosts` in `frontends/Rexfile`
+- HTTP always redirected to HTTPS by the OpenBSD gateways — client URLs must use `https://`
+
+## PV Directory Structure
+
+```
+/data/nfs/k3svolumes/pkgrepo/       ← NFS-backed PV, physically on f0
+  freebsd/
+    FreeBSD:15:amd64/
+      latest/
+        All/              # .pkg files
+        packagesite.pkg   # repo metadata (generated by `pkg repo`)
+        meta.conf / meta
+        data.pkg
+  openbsd/
+    7.8/
+      packages/
+        amd64/            # .tgz files (signify-signed)
+  netbsd/
+    10.1/
+      packages/
+        aarch64/          # .tgz files (unsigned) + pkg_summary.gz for pkgin
+  rockylinux/
+    9/
+      x86_64/             # .rpm files + repodata/
+      aarch64/            # .rpm files + repodata/
+```
+
+## Key Config Files
+
+| File | Purpose |
+|------|---------|
+| `f3s/pkgrepo/helm-chart/` | Helm chart (nginx deployment, service, ingress, PV, configmap) |
+| `f3s/argocd-apps/infra/pkgrepo.yaml` | ArgoCD Application manifest |
+| `f3s/pkgrepo/test-artifacts/build-test-packages.sh` | Builds hello-test packages for FreeBSD/OpenBSD |
+| `frontends/Rexfile` (`@f3s_hosts`) | DNS + routing entry for `pkgrepo.f3s.buetow.org` |
+| `frontends/Rexfile` (`pkgrepo_setup`) | Adds `PKG_PATH` to root's `.profile` on OpenBSD frontends |
+| `packages/Makefile` | Cross-compiles, packages, and uploads for all OSes |
+| `packages/scripts/pkg-freebsd.sh` | Runs on f0 via SSH: `pkg create` + `pkg repo` + copy to PV |
+| `packages/scripts/pkg-openbsd.sh` | Runs on fishfinger via SSH: `pkg_create` + signify signing |
+| `packages/scripts/pkg-dtail-openbsd.sh` | DTail multi-binary OpenBSD packaging |
+| `packages/scripts/pkg-dtail-freebsd.sh` | DTail multi-binary FreeBSD packaging |
+| `packages/scripts/pkg-dtail-netbsd.sh` | DTail multi-binary NetBSD packaging (runs on pi0 via SSH: `pkg_create` + `pkg_summary.gz`) |
+| `packages/scripts/pkg-dtail-rpm.sh` | Builds DTail RPMs from prebuilt or locally built payloads |
+
+## SSH Access for Package Tasks
+
+| Host | Access | Root | Notes |
+|------|--------|------|-------|
+| f0 (FreeBSD, NFS) | `ssh -p 22 f0.lan.buetow.org` | `doas` | PV is local; default shell is **csh** |
+| fishfinger (OpenBSD) | `ssh rex@fishfinger.buetow.org` | `doas` | OpenBSD packages built here, then copied to f0 PV |
+| blowfish (OpenBSD) | `ssh rex@blowfish.buetow.org` | `doas` | Same setup as fishfinger |
+| pi0 (NetBSD) | `ssh -p 22 paul@pi0.lan.buetow.org` | `doas` | NetBSD packages assembled here (`pkg_create`), then copied to f0 PV; non-interactive SSH PATH lacks `/usr/sbin` (`pkg_*` live there) |
+
+## Nginx Pod Gotcha — Stale NFS Handle
+
+After writing new files to the PV on f0, the nginx pod can get a stale NFS file handle (error 116) and return HTTP 500. Fix with a pod restart:
+
+```sh
+kubectl -n infra rollout restart deployment/pkgrepo
+kubectl -n infra rollout status deployment/pkgrepo
+```
+
+## Per-OS Repo Notes
+
+- **FreeBSD**: version follows ABI naming (`FreeBSD:15:amd64`); packages unsigned (`signature_type: "NONE"`); always regenerate metadata with `pkg repo` after adding/removing packages
+- **OpenBSD**: no repo index needed — `pkg_add` fetches by name; packages signed with signify; version in path must match host OS (currently 7.8)
+- **NetBSD**: packages unsigned; `pkg_add` works with a direct package URL, `pkg_summary.gz` alongside the packages enables pkgin; package versions must not contain dashes (`4.3.2-ng` → `4.3.2ng`); `+BUILD_INFO` (`MACHINE_ARCH`/`OS_VERSION`) is checked by `pkg_add` on install, which is why packages are assembled natively on pi0
+- **Rocky Linux**: standard DNF layout; unsigned repo (`gpgcheck=0`); architecture-specific paths (`x86_64` / `aarch64`)
+- **FreeBSD csh**: default shell is csh — avoid inline one-liners with `||`, `&&`, `!`, or multi-line quoting over SSH; use piped `/bin/sh` or separate SSH invocations