skills: split package repo docs out of f3s

2 files changed, 466 insertions, 0 deletions

diff --git a/prompts/skills/pkgrepo/SKILL.md b/prompts/skills/pkgrepo/SKILL.md
new file mode 100644
index 0000000..77de23e
--- /dev/null
+++ b/prompts/skills/pkgrepo/SKILL.md
@@ -0,0 +1,31 @@
+---
+name: pkgrepo
+description: Reference skill for the homelab package repositories behind pkgrepo.f3s.buetow.org. Covers FreeBSD pkg, OpenBSD pkg_add, Rocky Linux dnf repos, PV layout, nginx/ArgoCD wiring, DTail package publishing, and client repo configuration. Use when publishing packages, troubleshooting repository access or metadata, or deciding how packages should be installed from the custom repo.
+---
+
+# Package Repo Reference
+
+Use this skill when the task is specifically about the custom package repositories rather than the broader f3s homelab.
+
+## When to Use
+
+- Publishing or updating packages in `pkgrepo.f3s.buetow.org`
+- Troubleshooting repo layout, metadata, or HTTP exposure
+- Configuring FreeBSD, OpenBSD, or Rocky Linux clients to install from the custom repo
+- Working on DTail package publishing and installation through the repo
+
+## Reference Files
+
+- [Package Repositories](references/package-repos.md) — repo layout, PV paths, nginx exposure, client config, and DTail package workflows
+
+## Scope
+
+This skill owns package repository details that used to live under `f3s`.
+
+Use `f3s` alongside this skill when the task depends on broader host-role or cluster context, especially:
+
+- `f0` as the FreeBSD NFS/PV host for `/data/nfs/k3svolumes/pkgrepo/`
+- `fishfinger` and `blowfish` as the OpenBSD frontend hosts
+- `r0-r2` as Rocky Linux x86_64 bhyve VMs
+- `pi0-pi3` as Rocky Linux aarch64 Raspberry Pi nodes
+- `earth` as the Fedora laptop used for package publication and verification
diff --git a/prompts/skills/pkgrepo/references/package-repos.md b/prompts/skills/pkgrepo/references/package-repos.md
new file mode 100644
index 0000000..d6c45c4
--- /dev/null
+++ b/prompts/skills/pkgrepo/references/package-repos.md
@@ -0,0 +1,435 @@
+# Package Repositories
+
+Custom FreeBSD, OpenBSD, and Rocky Linux package repository served from k3s, allowing native package tools (`pkg`, `pkg_add`, and `dnf`) to install custom-built packages.
+
+## Architecture
+
+- **nginx pod** in k3s `infra` namespace serves static files from a PV
+- Path prefixes: `/freebsd/`, `/openbsd/`, and `/rockylinux/`
+- Accessible at `https://pkgrepo.f3s.buetow.org`
+- TLS terminated by OpenBSD relayd on the internet gateways (not in the pod)
+- DNS, ACME certs, httpd fallback, and relayd routing auto-generated from `@f3s_hosts` in `frontends/Rexfile`
+
+## Required host context
+
+This reference is package-repo focused, but it still uses a few f3s host names directly:
+
+- `f0` = FreeBSD host that carries the NFS-backed PV at `/data/nfs/k3svolumes/pkgrepo/`
+- `fishfinger` and `blowfish` = OpenBSD frontend hosts used for OpenBSD package install/build tasks
+- `r0-r2` = Rocky Linux 9 x86_64 bhyve VMs
+- `pi0-pi3` = Rocky Linux 9 aarch64 Raspberry Pi nodes
+- `earth` = Fedora laptop used to build, publish, and verify packages
+
+If you need the broader homelab topology, network roles, or non-package service context, load the sibling `f3s` skill as well.
+
+## Key Files
+
+| File | Purpose |
+|------|---------|
+| `f3s/pkgrepo/helm-chart/` | Helm chart (nginx deployment, service, ingress, PV, configmap) |
+| `f3s/argocd-apps/infra/pkgrepo.yaml` | ArgoCD Application manifest |
+| `f3s/pkgrepo/test-artifacts/build-test-packages.sh` | Builds hello-test packages for FreeBSD/OpenBSD |
+| `frontends/Rexfile` (`@f3s_hosts`) | DNS + routing entry for `pkgrepo.f3s.buetow.org` |
+| `frontends/Rexfile` (`pkgrepo_setup`) | Adds `PKG_PATH` to root's `.profile` on OpenBSD frontends |
+| `frontends/Rexfile` (`gogios_install`) | Installs/updates gogios from the custom repo |
+| `packages/Makefile` | `make pkg` cross-compiles, packages, and uploads for both OSes; `make dtail-rocky` builds and publishes the Rocky repo |
+| `packages/scripts/pkg-freebsd.sh` | Runs on f0 via SSH: pkg create + pkg repo + copy to PV |
+| `packages/scripts/pkg-openbsd.sh` | Runs on fishfinger via SSH: pkg_create + signify signing |
+| `packages/scripts/pkg-dtail-rpm.sh` | Builds DTail RPMs from prebuilt or locally built payloads |
+
+## PV Directory Structure
+
+```
+/data/nfs/k3svolumes/pkgrepo/
+  freebsd/
+    FreeBSD:15:amd64/
+      latest/
+        All/              # .pkg files
+        packagesite.pkg   # repo metadata (generated by `pkg repo`)
+        meta / meta.conf
+        data.pkg
+  openbsd/
+    7.8/
+      packages/
+        amd64/            # .tgz files
+  rockylinux/
+    9/
+      x86_64/             # .rpm files + repodata/
+      aarch64/            # .rpm files + repodata/
+```
+
+Hosted on NFS at f0 (`/data/nfs/k3svolumes/pkgrepo/`).
+
+## FreeBSD Client Setup (f0–f3)
+
+Custom repo is pre-configured on f0, f1, f2 (f3 offline at time of setup).
+Official FreeBSD repos remain active alongside the custom repo.
+
+File: `/usr/local/etc/pkg/repos/custom.conf`
+
+```
+custom: {
+  url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
+  mirror_type: "NONE",
+  signature_type: "NONE",
+  enabled: yes
+}
+```
+
+### Using packages
+
+```sh
+doas pkg update
+doas pkg install <package-name>   # works for both official and custom packages
+```
+
+### Setting up a new FreeBSD host
+
+Pipe the config via stdin to avoid csh quoting issues (FreeBSD default shell is csh):
+
+```sh
+cat <<'REPO' | ssh -p 22 <host>.lan.buetow.org 'doas mkdir -p /usr/local/etc/pkg/repos && doas tee /usr/local/etc/pkg/repos/custom.conf > /dev/null'
+custom: {
+  url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest",
+  mirror_type: "NONE",
+  signature_type: "NONE",
+  enabled: yes
+}
+REPO
+```
+
+## OpenBSD Client Setup (blowfish/fishfinger)
+
+Custom repo is configured via `PKG_PATH` in `/root/.profile`, deployed by `rex pkgrepo_setup`.
+Official OpenBSD packages still install normally via `/etc/installurl`.
+
+```sh
+export PKG_PATH="https://pkgrepo.f3s.buetow.org/openbsd/7.8/packages/amd64/"
+```
+
+### Using packages
+
+```sh
+doas pkg_add <package>               # from custom repo (signed with signify)
+doas pkg_add <official-package>      # from official repo (still works)
+```
+
+### Setting up new OpenBSD hosts
+
+Two things needed:
+
+1. **Signify public key** — copy from an existing host to `/etc/signify/custom-pkg.pub`:
+   ```sh
+   scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub /tmp/
+   scp /tmp/custom-pkg.pub rex@<newhost>:/tmp/
+   ssh rex@<newhost> "doas cp /tmp/custom-pkg.pub /etc/signify/custom-pkg.pub"
+   ```
+
+2. **PKG_PATH** — run the Rex task from `~/git/conf/frontends`:
+   ```sh
+   rex pkgrepo_setup   # adds PKG_PATH to /root/.profile on all frontends
+   ```
+
+The PKG_PATH must be updated when the OpenBSD version changes (currently 7.8).
+
+## Rocky Linux Client Setup (r0-r2, pi0-pi3)
+
+The Rocky repo is architecture-specific and follows the standard DNF layout:
+
+- `https://pkgrepo.f3s.buetow.org/rockylinux/9/x86_64/`
+- `https://pkgrepo.f3s.buetow.org/rockylinux/9/aarch64/`
+
+### Temporary one-off usage
+
+```sh
+sudo dnf repoquery \
+  --disablerepo='*' \
+  --repofrompath=f3s-dtail,https://pkgrepo.f3s.buetow.org/rockylinux/9/$(uname -m)/ \
+  --enablerepo=f3s-dtail \
+  dtail
+```
+
+### Persistent repo file
+
+Create `/etc/yum.repos.d/f3s-dtail.repo`:
+
+```ini
+[f3s-dtail]
+name=f3s DTail
+baseurl=https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/
+enabled=1
+gpgcheck=0
+repo_gpgcheck=0
+```
+
+Then:
+
+```sh
+sudo dnf makecache
+sudo dnf install dtail
+```
+
+### Package signing
+
+OpenBSD packages are signed with `signify(1)` via `pkg_sign`:
+- **Private key**: `/etc/signify/custom-pkg.sec` on fishfinger (build host)
+- **Public key**: `/etc/signify/custom-pkg.pub` on all OpenBSD clients
+- Signing happens automatically during `make pkg-openbsd` / `make pkg`
+- `pkg_add` verifies the signature against the public key — no `-D unsigned` needed
+
+## Packaging Workflow (Makefile)
+
+Build scripts live in `~/git/conf/packages/`, separate from individual project repos. The Makefile cross-compiles Go binaries on Linux, ships them to target hosts for native packaging, and uploads to the PV.
+
+### Build and upload
+
+```sh
+cd ~/git/conf/packages
+
+# Both FreeBSD and OpenBSD
+make pkg NAME=gogios SRC=/home/paul/git/gogios \
+    COMMENT="Monitoring tool with email alerts and HTML status page" \
+    DESC="Gogios is a lightweight monitoring tool written in Go."
+
+# Single OS
+make pkg-freebsd NAME=gogios SRC=/home/paul/git/gogios
+make pkg-openbsd NAME=gogios SRC=/home/paul/git/gogios
+
+# DTail Rocky Linux repo (x86_64 + aarch64 RPMs + repodata)
+make dtail-rocky
+```
+
+### How it works
+
+**Single-binary packages (pure Go, cross-compilable):**
+1. Cross-compiles on Linux (`GOOS=freebsd/openbsd GOARCH=amd64`)
+2. SCPs binary + packaging script to target host (f0 for FreeBSD, fishfinger for OpenBSD)
+3. Runs the script via SSH to package natively (`pkg create` / `pkg_create`)
+4. OpenBSD packages are signed with signify automatically
+5. FreeBSD repo metadata is regenerated with `pkg repo`
+6. Packages are copied to the PV at `/data/nfs/k3svolumes/pkgrepo/`
+
+**CGo packages (e.g. dtail with DataDog/zstd):**
+Cross-compilation from Linux to OpenBSD fails for CGo — needs a C cross-compiler.
+Solution: native build on the local OpenBSD VM. Source is sent via `git archive HEAD | ssh ... tar -x`
+(not `scp -r`) to avoid filling /tmp with build artifacts and test data.
+
+**Rocky Linux RPMs (dtail only, current implementation):**
+- `x86_64` RPM is built locally on earth
+- `aarch64` RPM is packaged on `pi0` because Fedora's x86_64 `rpmbuild` refuses to emit an `aarch64` binary RPM even when the payload is already prebuilt
+- repo metadata is generated locally in a `rockylinux:9` container with `createrepo_c`
+- the finished repo tree is copied to `/data/nfs/k3svolumes/pkgrepo/rockylinux/9/`
+
+### Required variables
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `NAME` | Package name | `gogios` |
+| `SRC` | Go project root (must have `cmd/<NAME>/main.go` and `internal/version.go`) | `/home/paul/git/gogios` |
+
+### Optional variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `COMMENT` | `$(NAME)` | One-line description |
+| `DESC` | `$(NAME)` | Longer description |
+| `MAINTAINER` | `paul@buetow.org` | Maintainer email |
+| `WWW` | `https://buetow.org` | Project URL |
+| `ENTRY` | `cmd/$(NAME)/main.go` | Go main package path relative to SRC |
+
+### Version detection
+
+Version is read automatically from `$(SRC)/internal/version.go` — expects a `Version` constant like `const Version = "v1.4.1"`. The `v` prefix is stripped.
+
+### Install/update on frontends via Rex
+
+```sh
+cd ~/git/conf/frontends
+rex gogios_install   # installs or updates gogios on blowfish + fishfinger
+rex gogios           # full setup (includes gogios_install + config + cron)
+```
+
+The `gogios_install` Rex task auto-detects the OS and uses `pkg install` (FreeBSD) or `pkg_add` (OpenBSD).
+
+## DTail Package (multi-binary, OpenBSD only)
+
+DTail is a distributed log tailing tool with 6 binaries, config files, and an rc script.
+Unlike single-binary packages, it has a dedicated Makefile target and packaging script.
+
+### Build and upload
+
+```sh
+cd ~/git/conf/packages
+make dtail-openbsd
+```
+
+### What the package includes
+
+| File | Purpose |
+|------|---------|
+| `/usr/local/bin/dserver` | DTail server daemon |
+| `/usr/local/bin/dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | Client tools |
+| `/etc/dserver/dtail.json` | Server config (from `frontends/etc/dserver/dtail.json.tpl`) |
+| `/etc/rc.d/dserver` | RC script (from `frontends/etc/rc.d/dserver.tpl`) |
+| `/usr/local/bin/dserver-update-key-cache.sh` | SSH key cache helper for daily cron |
+
+### Rex integration
+
+```sh
+cd ~/git/conf/frontends
+rex dtail_install   # install/update dtail package from custom repo
+rex dtail           # full setup (install + service user + daily cron + start)
+```
+
+The `dtail` Rex task installs the package, creates the `_dserver` service user,
+adds the key cache script to daily cron, and ensures dserver is running.
+
+## DTail Rocky Linux RPM Repo
+
+DTail for Rocky is published as a multi-binary RPM for both lab architectures:
+
+- `dtail-4.3.2-1.ng.x86_64.rpm`
+- `dtail-4.3.2-1.ng.aarch64.rpm`
+
+The RPM currently installs:
+
+- `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth`
+- `/usr/local/bin/dserver-update-key-cache.sh`
+- `/etc/dserver/dtail.json`
+- `/usr/lib/systemd/system/dserver.service`
+- `/usr/lib/systemd/system/dserver-update-keycache.service`
+- `/usr/lib/systemd/system/dserver-update-keycache.timer`
+
+### Build and publish
+
+```sh
+cd ~/git/conf/packages
+make dtail-rocky
+```
+
+### Notes
+
+- The packaged key-cache helper handles both `/root/.ssh/authorized_keys` and `/home/*/.ssh/authorized_keys`, so `root` works on `r0-r2` without a separate manual cache copy.
+- The packaged `dserver.service` includes the `RuntimeDirectory` and `ExecStartPre` fix required on Rocky so `/var/run/dserver` exists before startup.
+- The repo is unsigned for now, so the example repo file uses `gpgcheck=0` and `repo_gpgcheck=0`.
+
+### Verified DTail package state
+
+On 2026-04-11 this package repo path was verified with:
+
+- `dtail-4.3.2-1.ng.x86_64.rpm` and `dtail-4.3.2-1.ng.aarch64.rpm` published at `https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/`
+- successful `dnf repoquery` for `dtail` from `r0` (`x86_64`) and `pi0` (`aarch64`)
+- successful `dnf install dtail` on `r0-r2` and `pi0-pi3`
+- `dserver` active after package install, timer enable, restart, and key-cache refresh on all seven Rocky hosts
+- successful `dcat /etc/fstab` reads from earth using `--user root` for `r0-r2` and `--user paul` for `pi0-pi3`
+
+### OpenBSD DTail package verification
+
+Also verified on 2026-04-11:
+
+- `dtail-4.3.2-ng` rebuilt and published to the OpenBSD repo path
+- package replacement, key-cache refresh, and successful `dcat /etc/fstab` reads as `--user rex` on `blowfish.buetow.org` and `fishfinger.buetow.org`
+
+### Why a build VM?
+
+DTail depends on DataDog/zstd which requires CGo. Cross-compiling CGo from Linux
+to OpenBSD needs a C cross-compiler. Instead, a local OpenBSD QEMU/KVM VM builds
+natively — this also works for any future packages with C dependencies.
+
+## OpenBSD Build VM
+
+A minimal OpenBSD QEMU/KVM VM on the laptop (earth) for native compilation.
+Scripts in `packages/buildvm/`.
+
+### Initial setup (once)
+
+```sh
+cd ~/git/conf/packages/buildvm
+
+# Fetch signify keys from fishfinger
+scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.sec .
+scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub .
+
+./setup.sh       # downloads ISO, runs fully automated install (~5 min)
+./provision.sh   # installs Go, git, gmake, signify keys, SSH keys
+```
+
+`setup.sh` is fully automated — it calls `install-expect.exp` which drives the
+OpenBSD serial console installer via expect. No manual interaction needed.
+
+### Day-to-day use
+
+```sh
+make buildvm-start   # boot VM (~15s)
+make dtail-openbsd   # auto-starts VM if needed
+make buildvm-stop    # shut down when done
+```
+
+The VM is headless, SSH on `localhost:2222`, 1GB RAM, 2 CPUs, 4GB disk.
+Username: `pbuild` (password: `build123`). SSH key installed by `provision.sh`.
+
+### Automated installer notes (install-expect.exp)
+
+- The expect script is in a **separate file** (`install-expect.exp`) to avoid bash/expect quoting interactions — embedding expect in a bash heredoc causes password send failures
+- Serial console is activated via `set tty com0` at the OpenBSD boot prompt
+- Password prompts require `sleep 2` before `send` — `sleep 1` is not enough for the serial console to be ready
+- OpenBSD 7.8 added a new "Encrypt the root disk?" prompt before the partition layout
+- After CONGRATULATIONS, choose shell (`s`) instead of reboot — the CD is still attached and reboot re-boots the installer; configure wheel group via `chroot /mnt`
+- Username `build` is rejected by the installer ("not a usable loginname") — use `pbuild` instead
+- Source code is synced to the VM via `git archive HEAD | ssh ... tar -x` to avoid filling /tmp (full repo with benchmarks exceeds available space on 4GB disk)
+
+## Building Custom Packages
+
+### FreeBSD
+
+On any FreeBSD host, or via the Mage SCP-and-remote-execute pattern:
+
+```sh
+pkg create -M +MANIFEST -p plist -r stagedir -o output/All
+pkg repo output/   # regenerates repo metadata (unsigned)
+```
+
+Copy output to PV:
+```sh
+doas cp -Rf output/* /data/nfs/k3svolumes/pkgrepo/freebsd/FreeBSD:15:amd64/latest/
+```
+
+### OpenBSD
+
+On an OpenBSD host:
+
+```sh
+pkg_create \
+    -D COMMENT="Package description" \
+    -d descfile \
+    -f packing-list \
+    -B stagedir \
+    -p / \
+    output/package-name-1.0.tgz
+```
+
+Copy to PV via f0:
+```sh
+scp package.tgz f0.lan.buetow.org:/tmp/
+ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package.tgz /data/nfs/k3svolumes/pkgrepo/openbsd/7.8/packages/amd64/"
+```
+
+## SSH Access for Package Tasks
+
+| Host | Access | Root | Notes |
+|------|--------|------|-------|
+| f0 (FreeBSD, NFS) | `ssh -p 22 f0.lan.buetow.org` | `doas` | PV is local, default shell is csh |
+| fishfinger (OpenBSD) | `ssh rex@fishfinger.buetow.org` | `doas` | Packages built here, then copied to f0 PV |
+| blowfish (OpenBSD) | `ssh rex@blowfish.buetow.org` | `doas` | Same setup as fishfinger |
+
+## Notes
+
+- HTTP is always redirected to HTTPS by the OpenBSD gateways — client URLs must use `https://`
+- OpenBSD version in the path must match the host OS version (currently 7.8)
+- FreeBSD version follows the ABI naming convention (`FreeBSD:15:amd64`)
+- OpenBSD packages are signed with signify (key pair at `/etc/signify/custom-pkg.*`)
+- FreeBSD packages are unsigned (`signature_type: "NONE"`) — signing with RSA is possible but not yet configured
+- After adding/removing FreeBSD packages, always regenerate repo metadata with `pkg repo`
+- OpenBSD does not need a repo index — `pkg_add` fetches packages directly by name
+- FreeBSD hosts use csh as default shell — remote scripts must use `/bin/sh` explicitly
+- f3 is currently offline; configure its custom repo when it comes back online