From efbb7876347bc667bdb3633f72a22a51b3bfbc49 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Sat, 18 Jul 2026 22:08:10 +0300 Subject: skills: rename pkgrepo -> f3s-pkgrepo and rocky-vm-setup -> f3s-rocky-vm-setup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Extend the f3s- prefix to the two remaining f3s-scoped siblings so the whole homelab family groups together. Both are f3s-specific: pkgrepo IS pkgrepo.f3s.buetow.org, and rocky-vm-setup documents the 'rocky' VM on the f3s host f3. - git mv both skill dirs (history preserved); update name: frontmatter; add the 'Part of the f3s homelab skill family' note to both descriptions. - Rewrite every reference to the *skills*: backtick mentions (`pkgrepo` -> `f3s-pkgrepo`, `rocky-vm-setup` -> `f3s-rocky-vm-setup`), markdown cross-links (../pkgrepo/ -> ../f3s-pkgrepo/, etc.), the f3s hub Related-skills block and description, and skill-maintenance's index-pattern example. - Left references to the package REPOSITORY (pkgrepo.f3s.buetow.org, 'the custom pkgrepo') unchanged — those name a service, not the skill. - Verified all 114 relative links resolve. Plan doc updated to record the rename. Co-Authored-By: Claude Opus 4.8 --- prompts/skills/f3s-dtail/SKILL.md | 8 +- prompts/skills/f3s-dtail/references/dserver.d | 2 +- prompts/skills/f3s-dtail/references/dtail.md | 8 +- prompts/skills/f3s-pkgrepo/SKILL.md | 37 ++++ .../skills/f3s-pkgrepo/references/client-setup.md | 141 +++++++++++++ .../skills/f3s-pkgrepo/references/dtail-package.md | 219 +++++++++++++++++++++ .../f3s-pkgrepo/references/openbsd-build-vm.md | 42 ++++ .../f3s-pkgrepo/references/packaging-workflow.md | 120 +++++++++++ .../f3s-pkgrepo/references/repo-architecture.md | 80 ++++++++ prompts/skills/f3s-raspberry-pi/SKILL.md | 4 +- .../references/bootstrap-netbsd-pi.md | 2 +- prompts/skills/f3s-rocky-vm-setup/SKILL.md | 39 ++++ .../f3s-rocky-vm-setup/references/git-remotes.md | 13 ++ .../skills/f3s-rocky-vm-setup/references/notes.md | 5 + .../f3s-rocky-vm-setup/references/overview.md | 42 ++++ .../f3s-rocky-vm-setup/references/privileges.md | 20 ++ .../skills/f3s-rocky-vm-setup/references/rex.md | 22 +++ .../f3s-rocky-vm-setup/references/scripts.md | 20 ++ .../skills/f3s-rocky-vm-setup/references/tmux.md | 72 +++++++ .../skills/f3s-rocky-vm-setup/references/tools.md | 69 +++++++ .../skills/f3s-rocky-vm-setup/references/zrepl.md | 12 ++ prompts/skills/f3s/SKILL.md | 6 +- prompts/skills/pkgrepo/SKILL.md | 37 ---- prompts/skills/pkgrepo/references/client-setup.md | 141 ------------- prompts/skills/pkgrepo/references/dtail-package.md | 219 --------------------- .../skills/pkgrepo/references/openbsd-build-vm.md | 42 ---- .../pkgrepo/references/packaging-workflow.md | 120 ----------- .../skills/pkgrepo/references/repo-architecture.md | 80 -------- prompts/skills/rocky-vm-setup/SKILL.md | 39 ---- .../rocky-vm-setup/references/git-remotes.md | 13 -- prompts/skills/rocky-vm-setup/references/notes.md | 5 - .../skills/rocky-vm-setup/references/overview.md | 42 ---- .../skills/rocky-vm-setup/references/privileges.md | 20 -- prompts/skills/rocky-vm-setup/references/rex.md | 22 --- .../skills/rocky-vm-setup/references/scripts.md | 20 -- prompts/skills/rocky-vm-setup/references/tmux.md | 72 ------- prompts/skills/rocky-vm-setup/references/tools.md | 69 ------- prompts/skills/rocky-vm-setup/references/zrepl.md | 12 -- prompts/skills/skill-maintenance/SKILL.md | 4 +- .../skill-maintenance/references/sub-division.md | 10 +- 40 files changed, 975 insertions(+), 975 deletions(-) create mode 100644 prompts/skills/f3s-pkgrepo/SKILL.md create mode 100644 prompts/skills/f3s-pkgrepo/references/client-setup.md create mode 100644 prompts/skills/f3s-pkgrepo/references/dtail-package.md create mode 100644 prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md create mode 100644 prompts/skills/f3s-pkgrepo/references/packaging-workflow.md create mode 100644 prompts/skills/f3s-pkgrepo/references/repo-architecture.md create mode 100644 prompts/skills/f3s-rocky-vm-setup/SKILL.md create mode 100644 prompts/skills/f3s-rocky-vm-setup/references/git-remotes.md create mode 100644 prompts/skills/f3s-rocky-vm-setup/references/notes.md create mode 100644 prompts/skills/f3s-rocky-vm-setup/references/overview.md create mode 100644 prompts/skills/f3s-rocky-vm-setup/references/privileges.md create mode 100644 prompts/skills/f3s-rocky-vm-setup/references/rex.md create mode 100644 prompts/skills/f3s-rocky-vm-setup/references/scripts.md create mode 100644 prompts/skills/f3s-rocky-vm-setup/references/tmux.md create mode 100644 prompts/skills/f3s-rocky-vm-setup/references/tools.md create mode 100644 prompts/skills/f3s-rocky-vm-setup/references/zrepl.md delete mode 100644 prompts/skills/pkgrepo/SKILL.md delete mode 100644 prompts/skills/pkgrepo/references/client-setup.md delete mode 100644 prompts/skills/pkgrepo/references/dtail-package.md delete mode 100644 prompts/skills/pkgrepo/references/openbsd-build-vm.md delete mode 100644 prompts/skills/pkgrepo/references/packaging-workflow.md delete mode 100644 prompts/skills/pkgrepo/references/repo-architecture.md delete mode 100644 prompts/skills/rocky-vm-setup/SKILL.md delete mode 100644 prompts/skills/rocky-vm-setup/references/git-remotes.md delete mode 100644 prompts/skills/rocky-vm-setup/references/notes.md delete mode 100644 prompts/skills/rocky-vm-setup/references/overview.md delete mode 100644 prompts/skills/rocky-vm-setup/references/privileges.md delete mode 100644 prompts/skills/rocky-vm-setup/references/rex.md delete mode 100644 prompts/skills/rocky-vm-setup/references/scripts.md delete mode 100644 prompts/skills/rocky-vm-setup/references/tmux.md delete mode 100644 prompts/skills/rocky-vm-setup/references/tools.md delete mode 100644 prompts/skills/rocky-vm-setup/references/zrepl.md (limited to 'prompts') diff --git a/prompts/skills/f3s-dtail/SKILL.md b/prompts/skills/f3s-dtail/SKILL.md index 779e1b3..127b708 100644 --- a/prompts/skills/f3s-dtail/SKILL.md +++ b/prompts/skills/f3s-dtail/SKILL.md @@ -1,25 +1,25 @@ --- name: f3s-dtail -description: Reference skill for DTail/dserver deployment across the f3s fleet, distributed log access over SSH on port 2222 — Pis arm64 (NetBSD + Rocky) vs r0–r2 amd64, r-VM root + root.authorized_keys cache, firewalld/npf 2222 rules, systemd timers. Package building lives in the `pkgrepo` skill. Use when deploying, configuring, or troubleshooting dserver on homelab hosts. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)). +description: Reference skill for DTail/dserver deployment across the f3s fleet, distributed log access over SSH on port 2222 — Pis arm64 (NetBSD + Rocky) vs r0–r2 amd64, r-VM root + root.authorized_keys cache, firewalld/npf 2222 rules, systemd timers. Package building lives in the `f3s-pkgrepo` skill. Use when deploying, configuring, or troubleshooting dserver on homelab hosts. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)). --- # f3s DTail / dserver Distributed log access (DTail) via `dserver` across the f3s fleet. This skill owns the **runtime deployment/operations**; package *building/publishing* lives in the -sibling [`pkgrepo`](../pkgrepo/SKILL.md) skill (`dtail-package.md`). +sibling [`f3s-pkgrepo`](../f3s-pkgrepo/SKILL.md) skill (`dtail-package.md`). ## When to Use - Deploying, configuring, or troubleshooting `dserver` on homelab hosts (Pis, r0–r2) - SSH-on-2222 access, permissions/key-cache, firewall (firewalld/npf) rules, systemd timers -- For building the `dtail` package (esp. the NetBSD build), use [`pkgrepo`](../pkgrepo/SKILL.md); for the Pi nodes themselves, [`f3s-raspberry-pi`](../f3s-raspberry-pi/SKILL.md); for hosts/IPs, the [`f3s`](../f3s/SKILL.md) hub. +- For building the `dtail` package (esp. the NetBSD build), use [`f3s-pkgrepo`](../f3s-pkgrepo/SKILL.md); for the Pi nodes themselves, [`f3s-raspberry-pi`](../f3s-raspberry-pi/SKILL.md); for hosts/IPs, the [`f3s`](../f3s/SKILL.md) hub. ## Overview Distributed log access (`dcat`/`dtail`/`dgrep`/`dmap`) over SSH on port **2222** (not sshd's 22), by architecture: **pi2/pi3** linux/arm64, **pi0/pi1** netbsd/arm64 (installed -from the `dtail` package in the custom [`pkgrepo`](../pkgrepo/SKILL.md)), **r0–r2** k3s +from the `dtail` package in the custom [`f3s-pkgrepo`](../f3s-pkgrepo/SKILL.md)), **r0–r2** k3s Rocky VMs linux/amd64. The recurring gotchas — installing as `root`, listing `root` in `Server.Permissions.Users`, mirroring `/root/.ssh/authorized_keys` into the key cache (the cache script only walks `/home/*`), and opening 2222 in firewalld/npf — plus the diff --git a/prompts/skills/f3s-dtail/references/dserver.d b/prompts/skills/f3s-dtail/references/dserver.d index 49d5c54..5d4e7be 100644 --- a/prompts/skills/f3s-dtail/references/dserver.d +++ b/prompts/skills/f3s-dtail/references/dserver.d @@ -1,7 +1,7 @@ # dserver on f3s (index) - **r0–r2 Rocky bhyve / k3s VMs** — install context and SSH notes: [Rocky Linux VMs – DTail (dserver) on r0–r2](rocky-linux-vms.md#dtail-dserver-on-r0r2) -- **pi0–pi1 NetBSD Pis** — dserver installed from the custom pkgrepo (`dtail-4.3.2ng` package): build with `make dtail-netbsd` in `~/git/conf/packages` (cross-compile netbsd/arm64, package natively on pi0, upload to pkgrepo), deploy via `pkg_add https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-.tgz`. Full build/install/rc.d/npf details and gotchas: `pkgrepo` skill's `dtail-package.md` +- **pi0–pi1 NetBSD Pis** — dserver installed from the custom pkgrepo (`dtail-4.3.2ng` package): build with `make dtail-netbsd` in `~/git/conf/packages` (cross-compile netbsd/arm64, package natively on pi0, upload to pkgrepo), deploy via `pkg_add https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-.tgz`. Full build/install/rc.d/npf details and gotchas: `f3s-pkgrepo` skill's `dtail-package.md` - **Full DTail reference** (NetBSD + Rocky Pis, r VMs amd64, firewalld, key cache, clients): [dtail.md](dtail.md) Upstream repo: `https://codeberg.org/snonux/dtail` — `doc/installation.md`, `examples/`. diff --git a/prompts/skills/f3s-dtail/references/dtail.md b/prompts/skills/f3s-dtail/references/dtail.md index 840f07d..4d94c29 100644 --- a/prompts/skills/f3s-dtail/references/dtail.md +++ b/prompts/skills/f3s-dtail/references/dtail.md @@ -13,7 +13,7 @@ Upstream install and examples live in the repo: `doc/installation.md`, `examples | **r0–r2** | Rocky Linux 9 **x86_64** (bhyve VMs, k3s nodes) | Cross-build **linux/amd64**, `nozstd` | Often `root@rN.lan.buetow.org` (see [Rocky Linux VMs](../../f3s/references/rocky-linux-vms.md)); add `root` (and `paul` if present) to **Server.Permissions.Users** in `dtail.json` | | **blowfish, fishfinger** | OpenBSD 7.8 **amd64** | Native OpenBSD package build | `rex@blowfish.buetow.org`, `rex@fishfinger.buetow.org` | -`pi0`/`pi1` (NetBSD) run dserver since 2026-07-09 from the `dtail` package in the custom pkgrepo — build pipeline (`make dtail-netbsd`), install steps, rc.d/npf details, and gotchas live in the `pkgrepo` skill's `dtail-package.md`. +`pi0`/`pi1` (NetBSD) run dserver since 2026-07-09 from the `dtail` package in the custom pkgrepo — build pipeline (`make dtail-netbsd`), install steps, rc.d/npf details, and gotchas live in the `f3s-pkgrepo` skill's `dtail-package.md`. **Key cache filenames matter:** `examples/update_key_cache.sh.example` only scans `/home/*` and writes `/var/run/dserver/cache/USER.authorized_keys`. In this lab, DTail auth worked only after writing the exact cache filename for the login user: @@ -160,7 +160,7 @@ CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -tags nozstd -o dserver-linux-amd ## Installation checklist (each server) -Do this on **each** Linux target (pi2–pi3 and/or r0–r2). pi0/pi1 are NetBSD and install the `dtail` package from the custom pkgrepo instead — see the `pkgrepo` skill's `dtail-package.md`. Adjust **user** if you are not using `paul` on the node. +Do this on **each** Linux target (pi2–pi3 and/or r0–r2). pi0/pi1 are NetBSD and install the `dtail` package from the custom pkgrepo instead — see the `f3s-pkgrepo` skill's `dtail-package.md`. Adjust **user** if you are not using `paul` on the node. 1. **Binary**: `/usr/local/bin/dserver`, mode `0755`, owned by root. 2. **OS user**: `dserver` system account (`useradd -r -d /var/lib/dserver -s /sbin/nologin -U dserver`). @@ -200,9 +200,9 @@ Use hostnames that resolve from where you run the client (often `*.lan.buetow.or ## Package-managed DTail -For package-repo-backed DTail and other custom package repo tasks, use the sibling `pkgrepo` skill and its reference: +For package-repo-backed DTail and other custom package repo tasks, use the sibling `f3s-pkgrepo` skill and its reference: -- [Package Repositories](../../pkgrepo/references/repo-architecture.md) +- [Package Repositories](../../f3s-pkgrepo/references/repo-architecture.md) That skill now owns: diff --git a/prompts/skills/f3s-pkgrepo/SKILL.md b/prompts/skills/f3s-pkgrepo/SKILL.md new file mode 100644 index 0000000..262a6e7 --- /dev/null +++ b/prompts/skills/f3s-pkgrepo/SKILL.md @@ -0,0 +1,37 @@ +--- +name: f3s-pkgrepo +description: Reference skill for the homelab package repositories behind pkgrepo.f3s.buetow.org. Covers FreeBSD pkg, OpenBSD pkg_add, NetBSD pkg_add, Rocky Linux dnf repos, PV layout, nginx/ArgoCD wiring, DTail package publishing, and client repo configuration. Use when publishing packages, troubleshooting repository access or metadata, or deciding how packages should be installed from the custom repo. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)). +--- + +# Package Repo Reference + +Use this skill when the task is specifically about the custom package repositories rather than the broader f3s homelab. + +## When to Use + +- Publishing or updating packages in `pkgrepo.f3s.buetow.org` +- Troubleshooting repo layout, metadata, or HTTP exposure +- Configuring FreeBSD, OpenBSD, NetBSD, or Rocky Linux clients to install from the custom repo +- Working on DTail package publishing and installation through the repo + +## Reference Files + +- [Repo Architecture](references/repo-architecture.md) — nginx/k3s setup, PV directory structure, SSH access, stale NFS handle fix, per-OS repo notes +- [Client Setup](references/client-setup.md) — per-OS client repo configuration (FreeBSD, OpenBSD, NetBSD, Rocky Linux), new-host setup, package signing +- [Packaging Workflow](references/packaging-workflow.md) — Makefile workflow for single-binary Go packages, CGo packages, manual packaging reference +- [DTail Package](references/dtail-package.md) — multi-binary DTail package for all platforms, install/update steps, gotchas, client usage, verification +- [OpenBSD Build VM](references/openbsd-build-vm.md) — QEMU/KVM build VM for native CGo compilation, day-to-day use, installer notes + +## Scope + +This skill owns package repository details that used to live under `f3s`. + +Use `f3s` alongside this skill when the task depends on broader host-role or cluster context, especially: + +- `f0` as the FreeBSD NFS/PV host for `/data/nfs/k3svolumes/pkgrepo/` +- `fishfinger` and `blowfish` as the OpenBSD frontend hosts +- `r0-r2` as Rocky Linux x86_64 bhyve VMs +- `pi2-pi3` as Rocky Linux aarch64 Raspberry Pi nodes +- `pi0`/`pi1` as NetBSD aarch64 Raspberry Pi nodes (see the `f3s-raspberry-pi` skill's `bootstrap-netbsd-pi.md`) — NetBSD repo clients and the native NetBSD package build host (`pi0`) +- `earth` as the Fedora laptop used for package publication and verification +- `f0-f3` as FreeBSD hosts diff --git a/prompts/skills/f3s-pkgrepo/references/client-setup.md b/prompts/skills/f3s-pkgrepo/references/client-setup.md new file mode 100644 index 0000000..1e29c4e --- /dev/null +++ b/prompts/skills/f3s-pkgrepo/references/client-setup.md @@ -0,0 +1,141 @@ +# Client Setup + +How to configure each OS to install packages from `pkgrepo.f3s.buetow.org`. + +## FreeBSD (f0–f3) + +Custom repo is configured alongside the official FreeBSD repos. + +File: `/usr/local/etc/pkg/repos/custom.conf` + +``` +custom: { + url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest", + mirror_type: "NONE", + signature_type: "NONE", + enabled: yes +} +``` + +### Using packages + +```sh +doas pkg update +doas pkg install +doas pkg upgrade # upgrade all, including custom packages +doas pkg install -fy # force reinstall (same version) +``` + +### Setting up a new FreeBSD host + +Pipe via stdin to avoid csh quoting issues: + +```sh +cat <<'REPO' | ssh -p 22 .lan.buetow.org 'doas mkdir -p /usr/local/etc/pkg/repos && doas tee /usr/local/etc/pkg/repos/custom.conf > /dev/null' +custom: { + url: "https://pkgrepo.f3s.buetow.org/freebsd/FreeBSD:15:amd64/latest", + mirror_type: "NONE", + signature_type: "NONE", + enabled: yes +} +REPO +``` + +## OpenBSD (blowfish, fishfinger) + +Custom repo is configured via `PKG_PATH` in `/root/.profile`, deployed by `rex pkgrepo_setup`. +Official OpenBSD packages still install normally via `/etc/installurl`. + +```sh +export PKG_PATH="https://pkgrepo.f3s.buetow.org/openbsd/7.8/packages/amd64/" +``` + +### Using packages + +```sh +doas pkg_add # installs from custom repo (signed with signify) +doas pkg_add -u # update to latest version +``` + +### Setting up a new OpenBSD host + +Two things needed: + +1. **Signify public key** — copy from an existing host: + ```sh + scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub /tmp/ + scp /tmp/custom-pkg.pub rex@:/tmp/ + ssh rex@ "doas cp /tmp/custom-pkg.pub /etc/signify/custom-pkg.pub" + ``` + +2. **PKG_PATH** — run the Rex task from `~/git/conf/frontends`: + ```sh + rex pkgrepo_setup # adds PKG_PATH to /root/.profile on all frontends + ``` + +Update `PKG_PATH` whenever the OpenBSD version changes (currently 7.8). + +### Package signing + +OpenBSD packages are signed with `signify(1)` via `pkg_sign`: +- **Private key**: `/etc/signify/custom-pkg.sec` on fishfinger (build host) +- **Public key**: `/etc/signify/custom-pkg.pub` on all OpenBSD clients +- Signing happens automatically during `make pkg-openbsd` / `make pkg` +- `pkg_add` verifies the signature — no `-D unsigned` needed + +## NetBSD (pi0, pi1) + +No persistent repo config needed — install/update straight from the package URL (repo is unsigned): + +```sh +export PATH=/usr/sbin:$PATH # pkg_* live in /usr/sbin, not in the default non-interactive PATH +doas pkg_add https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz # first install +doas pkg_add -u https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz # update to newer version +``` + +Notes: +- Same-version reinstall: `pkg_add -u` skips it — `doas pkg_delete dtail` first, then `pkg_add` again +- `pkg_summary.gz` is published alongside the packages, so the repo URL could also be added to `/usr/pkg/etc/pkgin/repositories.conf` for pkgin; today plain `pkg_add` is used +- Version in the URL path must match the host OS release (currently 10.1) and the arch is `aarch64` (`uname -p`) +- pkgsrc packages from cdn.netbsd.org still install normally via pkgin — the custom repo lives under a different prefix (`/` with files in `/usr/local`, `/etc`) and doesn't conflict + +## Rocky Linux (r0–r2, pi2–pi3) + +`pi0`/`pi1` run NetBSD (see above and the `f3s-raspberry-pi` skill's `bootstrap-netbsd-pi.md`) and are not Rocky clients of this repo. + +Architecture-specific repo URLs: +- `https://pkgrepo.f3s.buetow.org/rockylinux/9/x86_64/` (r0–r2) +- `https://pkgrepo.f3s.buetow.org/rockylinux/9/aarch64/` (pi2–pi3) + +### Persistent repo file + +Create `/etc/yum.repos.d/f3s-dtail.repo`: + +```ini +[f3s-dtail] +name=f3s DTail +baseurl=https://pkgrepo.f3s.buetow.org/rockylinux/9/$basearch/ +enabled=1 +gpgcheck=0 +repo_gpgcheck=0 +``` + +Then: + +```sh +sudo dnf makecache +sudo dnf install dtail +sudo dnf upgrade dtail +``` + +### Temporary one-off usage (no persistent repo file needed) + +```sh +sudo dnf repoquery \ + --disablerepo='*' \ + --repofrompath=f3s-dtail,https://pkgrepo.f3s.buetow.org/rockylinux/9/$(uname -m)/ \ + --enablerepo=f3s-dtail \ + dtail +``` + +Do **not** pass `--repofrompath` when the persistent repo file already exists — dnf errors with "listed more than once". diff --git a/prompts/skills/f3s-pkgrepo/references/dtail-package.md b/prompts/skills/f3s-pkgrepo/references/dtail-package.md new file mode 100644 index 0000000..fbdd8d6 --- /dev/null +++ b/prompts/skills/f3s-pkgrepo/references/dtail-package.md @@ -0,0 +1,219 @@ +# DTail Package + +DTail is a multi-binary package (6 binaries + config + service script). There are separate targets for each OS. + +## Build Commands + +```sh +cd ~/git/conf/packages +make dtail-openbsd # OpenBSD: native build on QEMU/KVM VM (CGo/zstd supported) +make dtail-freebsd # FreeBSD: cross-compiled on Linux (CGO_ENABLED=0, nozstd — .zst logs unsupported) +make dtail-netbsd # NetBSD/aarch64: cross-compiled on Linux (CGO_ENABLED=0, nozstd), packaged natively on pi0 +make dtail-rocky # Rocky Linux: x86_64 + aarch64 RPMs + repodata +``` + +## Package Contents by OS + +### OpenBSD (blowfish, fishfinger) + +| File | Source template | +|------|----------------| +| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | built natively on build VM | +| `/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail.json.tpl` (absolute `CacheDir: /var/run/dserver/cache`; `HostKeyFile` in persistent `/var/db/dserver/ssh_host_key`) | +| `/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver.tpl` | +| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache.sh.tpl` (ksh) | + +OpenBSD notes: +- Packages before 2026-07-10 used relative `CacheDir: "cache"` / `HostKeyFile: "cache/ssh_host_key"` — that only worked because rc.d starts the daemon via `su -l _dserver` (CWD = `/var/run/dserver`, the `_dserver` home dir); a manual start from any other directory broke public key lookup — the same bug class dtail commit `fec2f9d` fixed on the server side (making absolute `CacheDir` paths resolve independently of dserver's CWD) +- OpenBSD's `/etc/rc` wipes `/var/run/*` at boot, so the SSH host key now lives in persistent `/var/db/dserver/ssh_host_key` (mirrors NetBSD). The rc.d `rc_pre` recreates `/var/run/dserver/cache` and `/var/db/dserver` and re-runs `dserver-update-key-cache.sh` on every start; the daily cron entry keeps it fresh afterwards +- When upgrading a host from a pre-2026-07-10 package, follow this order — neither `pkg_add` nor Rex restarts a running daemon, and once the host reboots `/etc/rc` has already wiped the old host key, making migration impossible: + 1. Migrate the host key FIRST to preserve the host identity: `doas install -d -o _dserver -m 0700 /var/db/dserver && doas cp -p /var/run/dserver/cache/ssh_host_key /var/db/dserver/` — otherwise dserver generates a new host key and clients without `--trustAllHosts` must re-accept it + 2. Reinstall the package (same-version: `pkg_add -u` is a no-op — `doas pkg_delete dtail` then `doas env PKG_PATH=... pkg_add dtail`) + 3. `doas rcctl restart dserver` — without this the running daemon keeps the old relative-path config until reboot + (done on fishfinger 2026-07-10; **blowfish still runs the pre-2026-07-10 package**) +- Since 2026-07-10 the rc.d script sets `rc_bg=YES` and calls `rc_cmd "$1"` normally — dserver does not daemonize itself, and rc.subr's `rc_bg` is the proper way to background it. Earlier packages backgrounded the whole framework (`rc_cmd $1 &`), which hid `rc_pre` failures and the start result from `rcctl` +- From the WireGuard VPN, `f0.lan.buetow.org` may not route — run the Makefile with `make dtail-openbsd FREEBSD_HOST=f0.wg0` (same override applies to the other `dtail-*` targets, which upload via f0) + +### FreeBSD (f0–f3) + +| File | Source template | +|------|----------------| +| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | cross-compiled `GOOS=freebsd CGO_ENABLED=0 -tags nozstd` | +| `/usr/local/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail-freebsd.json.tpl` | +| `/usr/local/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver-freebsd.tpl` | +| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache-freebsd.sh.tpl` (sh) | + +**FreeBSD config note:** `dtail-freebsd.json.tpl` uses **absolute paths** for `CacheDir` and `HostKeyFile`. FreeBSD's `daemon(8)` resets CWD to `/`, so the relative `"cache"` in the standard template resolves to `/cache` — silently breaking key lookup. (Since dtail commit `fec2f9d`, absolute `CacheDir` paths also resolve independently of the CWD dserver was started from — before that fix, a manual service restart from a home directory broke public key auth.) + +FreeBSD notes: +- FreeBSD's `cleanvar` purges `/var/run` at boot, so the SSH host key lives in persistent `/var/db/dserver/ssh_host_key` (mirrors NetBSD/OpenBSD). The rc.d `start_precmd` recreates `/var/run/dserver/cache` and `/var/db/dserver` and re-runs `dserver-update-key-cache.sh` on every start; the daily periodic job keeps it fresh afterwards +- Packages before 2026-07-10 kept the host key in volatile `/var/run/dserver/cache/ssh_host_key` — it regenerated on every reboot. When upgrading a host from such a package, migrate the key FIRST (before restarting dserver) to preserve the host identity: + 1. `doas install -d -o dserver -m 0700 /var/db/dserver && doas cp -p /var/run/dserver/cache/ssh_host_key /var/db/dserver/ssh_host_key` + 2. `doas pkg update -f && doas pkg install -fy dtail` (same-version force reinstall) + 3. `doas service dserver restart` + (done on f0/f1/f2 2026-07-10 — sha256 of the migrated key verified identical; **f3 was unreachable and still runs the pre-2026-07-10 package**) + +### NetBSD (pi0, pi1 — aarch64) + +Package name is `dtail-4.3.2ng` — NetBSD versions must not contain dashes, so `-ng` becomes `ng`. + +| File | Source template | +|------|----------------| +| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | cross-compiled `GOOS=netbsd GOARCH=arm64 CGO_ENABLED=0 -tags nozstd` | +| `/etc/dserver/dtail.json` | `frontends/etc/dserver/dtail-netbsd.json.tpl` (absolute `CacheDir` like FreeBSD; `HostKeyFile` in persistent `/var/db/dserver/ssh_host_key`) | +| `/etc/rc.d/dserver` | `frontends/etc/rc.d/dserver-netbsd.tpl` | +| `/usr/local/bin/dserver-update-key-cache.sh` | `frontends/scripts/dserver-update-key-cache-netbsd.sh.tpl` (sh) | + +NetBSD notes: +- `pkg_create` runs natively on pi0 (Makefile ships binaries + templates there via SSH and runs `packages/scripts/pkg-dtail-netbsd.sh`); `pkg_summary.gz` for pkgin is generated and uploaded alongside +- NetBSD has no `daemon(8)` and dserver doesn't daemonize — the rc.d script backgrounds it via `command_args="... &"` and runs it as user `dserver` (`dserver_user`) +- `/var/run` is volatile — the rc.d `start_precmd` recreates `/var/run/dserver/cache` and re-runs the key-cache helper on every start; a daily root cron entry (`dserver-update-key-cache.sh`) keeps it fresh +- The SSH host key lives in persistent `/var/db/dserver/ssh_host_key` (created by the precmd) so it survives reboots — since 2026-07-10 FreeBSD uses the same layout. Packages before 2026-07-09 used the volatile path; after upgrading, DTail clients without `--trustAllHosts` must re-accept the host key once +- npf firewall needs `pass stateful in final family inet4 proto tcp to $ext_if port 2222` in the `"external"` group of `/etc/npf.conf` + +### Rocky Linux (r0–r2 amd64, pi2–pi3 aarch64) + +| File | +|------| +| `/usr/local/bin/dserver`, `dcat`, `dgrep`, `dmap`, `dtail`, `dtailhealth` | +| `/usr/local/bin/dserver-update-key-cache.sh` | +| `/etc/dserver/dtail.json` | +| `/usr/lib/systemd/system/dserver.service` | +| `/usr/lib/systemd/system/dserver-update-keycache.service` | +| `/usr/lib/systemd/system/dserver-update-keycache.timer` | + +Rocky notes: +- Key-cache helper handles both `/root/.ssh/authorized_keys` and `/home/*/.ssh/authorized_keys` — `root` works on r0–r2 without manual cache copy +- `dserver.service` includes `RuntimeDirectory=dserver` and `ExecStartPre` to recreate `/var/run/dserver` (tmpfs) on Rocky +- Repo is unsigned (`gpgcheck=0`) +- `aarch64` RPM is built on pi2 — Fedora's rpmbuild refuses to emit `aarch64` binary RPMs from an x86_64 host + +## Install / Update + +### OpenBSD (via Rex) + +```sh +cd ~/git/conf/frontends +rex dtail_install # install or update from custom repo +rex dtail # full setup: install + _dserver user + daily cron + service start +``` + +### FreeBSD (manual, f0–f3) + +```sh +# Custom repo must already be configured (see client-setup.md) +doas pkg install dtail # first install +doas pkg install -fy dtail # force reinstall (same version) + +# Create service user (once per host) +doas pw useradd dserver -d /var/run/dserver -s /usr/sbin/nologin + +# Enable and start dserver +doas sysrc dserver_enable=YES +doas service dserver start + +# Key cache is populated by the rc.d start_precmd on every service start; +# run manually only when authorized_keys changed without a restart: +doas /usr/local/bin/dserver-update-key-cache.sh + +# Register daily key cache refresh +doas mkdir -p /usr/local/etc/periodic/daily +printf '%s\n%s\n' '#!/bin/sh' '/usr/local/bin/dserver-update-key-cache.sh' | \ + doas tee /usr/local/etc/periodic/daily/200.dserver-update-key-cache > /dev/null +doas chmod 755 /usr/local/etc/periodic/daily/200.dserver-update-key-cache +``` + +**FreeBSD gotchas:** +- Since the 2026-07-10 package the rc.d `start_precmd` re-runs `dserver-update-key-cache.sh` on every start, so the key cache repopulates automatically after restart or reboot (older packages required a manual re-run after any restart) +- `pkg install -fy` replaces `/usr/local/etc/dserver/dtail.json` with the package version; local customisations are lost +- Avoid inline one-liners with `||`, `!`, or multi-quote strings over SSH to FreeBSD (csh) — pipe a script to `doas /bin/sh` instead or use separate SSH commands + +### NetBSD (manual, pi0–pi1) + +```sh +# All as root via doas; pkg_* tools live in /usr/sbin (not in non-interactive SSH PATH) +export PATH=/usr/sbin:$PATH + +# Service group + user (once per host) +doas groupadd dserver +doas useradd -g dserver -d /var/run/dserver -s /sbin/nologin -c "DTail server" dserver + +# Install / update from the custom repo +doas pkg_add https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz +doas pkg_add -u https://pkgrepo.f3s.buetow.org/netbsd/10.1/packages/aarch64/dtail-4.3.2ng.tgz # newer version +# Same-version reinstall: pkg_delete dtail first, then pkg_add + +# Enable and start (rc.d script ships in the package) +doas sh -c 'echo dserver=YES >> /etc/rc.conf' # once per host +doas /etc/rc.d/dserver start + +# Open port 2222 (once per host): add to the "external" group in /etc/npf.conf: +# pass stateful in final family inet4 proto tcp to $ext_if port 2222 +# then: doas npfctl validate && doas npfctl reload + +# Daily key-cache refresh (once per host; rc.d start also refreshes it) +# root crontab entry: 30 4 * * * /usr/local/bin/dserver-update-key-cache.sh >/dev/null 2>&1 +``` + +**NetBSD gotchas:** +- The package deliberately does not create the `dserver` user/group (matching the FreeBSD package) — run the `groupadd`/`useradd` step above before the first service start or the rc.d precmd fails +- The key cache lives in volatile `/var/run` but the rc.d `start_precmd` recreates and repopulates it on every start — no manual re-run needed after restart or reboot +- `dserver -version` panics when run as root (`Not allowed to run as UID 0`) — check with `su -m dserver -c '/usr/local/bin/dserver -version'` or as a normal user + +### Rocky Linux (dnf) + +```sh +# On r0–r2 (root) or pi2–pi3 (paul with sudo): +sudo dnf upgrade dtail +sudo systemctl restart dserver +sudo systemctl start dserver-update-keycache.service # repopulate after restart +``` + +**Rocky gotcha:** A legacy `/etc/systemd/system/dserver-update-keycache.service` left from manual pre-RPM setup points to the old path `ExecStart=/var/run/dserver/update_key_cache.sh` and shadows the RPM-installed unit. Remove it on any host where the timer fails: + +```sh +sudo rm -f /etc/systemd/system/dserver-update-keycache.service +sudo systemctl daemon-reload +sudo systemctl start dserver-update-keycache.service +``` + +This was cleaned up on all r0–r2 and pi0–pi3 on 2026-04-19. + +## Client Usage from earth + +```sh +# OpenBSD frontends +dcat --plain --noColor --trustAllHosts --user rex \ + --servers blowfish.buetow.org,fishfinger.buetow.org --files /etc/fstab + +# FreeBSD hosts +dcat --plain --noColor --trustAllHosts --user paul \ + --servers f0.lan.buetow.org,f1.lan.buetow.org,f2.lan.buetow.org,f3.lan.buetow.org \ + --files /etc/fstab + +# Rocky VMs (r0–r2, user root) +dcat --plain --noColor --trustAllHosts --user root \ + --servers r0.lan.buetow.org,r1.lan.buetow.org,r2.lan.buetow.org --files /etc/fstab + +# Raspberry Pis Rocky (pi2–pi3, user paul) +dcat --plain --noColor --trustAllHosts --user paul \ + --servers pi2.lan.buetow.org,pi3.lan.buetow.org \ + --files /etc/fstab + +# Raspberry Pis NetBSD (pi0–pi1, user paul) +dcat --plain --noColor --trustAllHosts --user paul \ + --servers pi0.lan.buetow.org,pi1.lan.buetow.org \ + --files /etc/fstab +``` + +## Verification State + +| Date | Platform | Result | +|------|----------|--------| +| 2026-04-19 | FreeBSD f0–f3 | `dtail-4.3.2-ng` installed, dserver running under `daemon(8)`, `dcat /etc/fstab` ✓ (`--user paul`) | +| 2026-04-19 | OpenBSD blowfish, fishfinger | `dtail-4.3.2-ng` current, `dcat /etc/fstab` ✓ (`--user rex`) | +| 2026-04-19 | Rocky r0–r2 | `dtail-4.3.2-ng` current, dserver running, `dcat /etc/fstab` ✓ (`--user root`) | +| 2026-04-19 | Rocky pi0–pi3 | `dtail-4.3.2-ng` current, dserver running, `dcat /etc/fstab` ✓ (`--user paul`) — pi0/pi1 since re-imaged to NetBSD | +| 2026-07-09 | NetBSD pi0–pi1 | `dtail-4.3.2ng` installed (first NetBSD deployment), dserver running as `dserver` on 2222, `dcat /etc/fstab` ✓ (`--user paul`) | +| 2026-07-10 | FreeBSD f0–f2 | `dtail-4.3.2-ng` reinstalled with persistent host key (`/var/db/dserver`) — key migrated first, sha256 identical, mtime preserved; rc.d precmd repopulates key cache; `dcat /etc/fstab` ✓ (`--user paul`). **f3 unreachable (no route via wg0) — still on the pre-2026-07-10 package, migrate + upgrade when back** | +| 2026-07-10 | OpenBSD fishfinger | `dtail-4.3.2-ng` reinstalled (rc_bg rc.d + hardened key-cache script); host key sha256/mtime unchanged, `rcctl restart` reports ok, `dcat /etc/fstab` ✓ (`--user rex`). blowfish still on the older package | diff --git a/prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md b/prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md new file mode 100644 index 0000000..2c2831f --- /dev/null +++ b/prompts/skills/f3s-pkgrepo/references/openbsd-build-vm.md @@ -0,0 +1,42 @@ +# OpenBSD Build VM + +A minimal OpenBSD QEMU/KVM VM on earth for native compilation of CGo packages (e.g. dtail with DataDog/zstd). Cross-compiling CGo from Linux to OpenBSD needs a C cross-compiler; native build sidesteps that entirely. + +Scripts in `~/git/conf/packages/buildvm/`. + +## Initial Setup (once) + +```sh +cd ~/git/conf/packages/buildvm + +# Fetch signify keys from fishfinger +scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.sec . +scp rex@fishfinger.buetow.org:/etc/signify/custom-pkg.pub . + +./setup.sh # downloads ISO, runs fully automated install (~5 min) +./provision.sh # installs Go, git, gmake, signify keys, SSH keys +``` + +`setup.sh` is fully automated via `install-expect.exp` — drives the OpenBSD serial console installer without manual interaction. + +## Day-to-Day Use + +```sh +make buildvm-start # boot VM (~15s) +make dtail-openbsd # auto-starts VM if needed, then builds +make buildvm-stop # shut down when done +``` + +VM specs: headless, SSH on `localhost:2222`, 1 GB RAM, 2 CPUs, 4 GB disk. +Username: `pbuild` (password: `build123`). SSH key installed by `provision.sh`. + +Source is synced via `git archive HEAD | ssh ... tar -x` — not `scp -r` — to avoid filling `/tmp` with build artifacts and test data (full repo with benchmarks exceeds the 4 GB disk). + +## Installer Notes (install-expect.exp) + +- Expect script is a **separate file** to avoid bash/expect quoting interactions — embedding it in a bash heredoc breaks password sends +- Serial console activated via `set tty com0` at the OpenBSD boot prompt +- Password prompts need `sleep 2` before `send` — `sleep 1` is not enough for the serial console +- OpenBSD 7.8 added an "Encrypt the root disk?" prompt before the partition layout +- After CONGRATULATIONS, choose shell (`s`) not reboot — the CD is still attached; configure wheel group via `chroot /mnt` +- Username `build` is rejected ("not a usable loginname") — use `pbuild` diff --git a/prompts/skills/f3s-pkgrepo/references/packaging-workflow.md b/prompts/skills/f3s-pkgrepo/references/packaging-workflow.md new file mode 100644 index 0000000..d2b7388 --- /dev/null +++ b/prompts/skills/f3s-pkgrepo/references/packaging-workflow.md @@ -0,0 +1,120 @@ +# Packaging Workflow + +Build scripts live in `~/git/conf/packages/`. The Makefile cross-compiles Go binaries on Linux, ships them to target hosts for native packaging, and uploads to the PV. + +## Single-Binary Go Packages + +For pure Go packages (no CGo), cross-compilation from Linux works for both FreeBSD and OpenBSD. + +### Build and upload + +```sh +cd ~/git/conf/packages + +# Both FreeBSD and OpenBSD +make pkg NAME=gogios SRC=/home/paul/git/gogios \ + COMMENT="Monitoring tool with email alerts and HTML status page" \ + DESC="Gogios is a lightweight monitoring tool written in Go." + +# Single OS +make pkg-freebsd NAME=gogios SRC=/home/paul/git/gogios +make pkg-openbsd NAME=gogios SRC=/home/paul/git/gogios +``` + +### How it works + +1. Cross-compiles on Linux (`GOOS=freebsd/openbsd GOARCH=amd64`) +2. SCPs binary + packaging script to target host (f0 for FreeBSD, fishfinger for OpenBSD) +3. Runs the packaging script via SSH (`pkg create` / `pkg_create`) +4. OpenBSD packages are signed with signify automatically +5. FreeBSD repo metadata is regenerated with `pkg repo` +6. Packages are copied to the PV at `/data/nfs/k3svolumes/pkgrepo/` + +### Required Makefile variables + +| Variable | Description | Example | +|----------|-------------|---------| +| `NAME` | Package name | `gogios` | +| `SRC` | Go project root (must have `cmd//main.go` and `internal/version.go`) | `/home/paul/git/gogios` | + +### Optional Makefile variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `COMMENT` | `$(NAME)` | One-line package description | +| `DESC` | `$(NAME)` | Longer description | +| `MAINTAINER` | `paul@buetow.org` | Maintainer email | +| `WWW` | `https://buetow.org` | Project URL | +| `ENTRY` | `cmd/$(NAME)/main.go` | Go main package path relative to SRC | + +### Version detection + +Version is read automatically from `$(SRC)/internal/version.go` — expects a `Version` constant like `const Version = "v1.4.1"`. The `v` prefix is stripped. + +## CGo Packages + +Cross-compilation from Linux fails for CGo (e.g. packages with DataDog/zstd). Use native builds instead: + +- **OpenBSD**: native build on the local QEMU/KVM build VM (see [openbsd-build-vm.md](openbsd-build-vm.md)) +- **FreeBSD**: cross-compile with `CGO_ENABLED=0 -tags nozstd` — disables zstd support but allows static cross-compile +- **NetBSD**: cross-compile with `CGO_ENABLED=0 GOOS=netbsd GOARCH=arm64 -tags nozstd`; packaging (`pkg_create`) runs natively on pi0 because `+BUILD_INFO` must match the target host +- **Rocky Linux**: built locally on earth (x86_64) and on pi2 (aarch64 via rpmbuild) + +## Manual Packaging Reference + +### FreeBSD (on f0) + +```sh +pkg create -M +MANIFEST -p plist -r stagedir -o output/All +pkg repo output/ # regenerates repo metadata +doas cp -Rf output/* /data/nfs/k3svolumes/pkgrepo/freebsd/FreeBSD:15:amd64/latest/ +``` + +### OpenBSD (on fishfinger) + +```sh +pkg_create \ + -D COMMENT="Package description" \ + -d descfile \ + -f packing-list \ + -B stagedir \ + -p / \ + output/package-name-1.0.tgz +# Copy to PV via f0 +scp package.tgz f0.lan.buetow.org:/tmp/ +ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package.tgz /data/nfs/k3svolumes/pkgrepo/openbsd/7.8/packages/amd64/" +``` + +### NetBSD (on pi0) + +```sh +# pkg_* tools are in /usr/sbin (not in the non-interactive SSH PATH) +# Files staged under stagedir (-p), installed relative to / (-I); +# @owner root / @group wheel in the packing list keeps installed files root-owned. +/usr/sbin/pkg_create \ + -B build-info \ + -c commentfile \ + -d descfile \ + -f packing-list \ + -I / \ + -p stagedir \ + output/package-name-1.0.tgz +# pkg_summary.gz enables pkgin (pkg_add alone doesn't need it) +/usr/sbin/pkg_info -X output/*.tgz | gzip -9 > output/pkg_summary.gz +# Copy to PV via f0 +scp -P 22 output/* f0.lan.buetow.org:/tmp/ +ssh -p 22 f0.lan.buetow.org "doas cp /tmp/package-name-1.0.tgz /tmp/pkg_summary.gz /data/nfs/k3svolumes/pkgrepo/netbsd/10.1/packages/aarch64/" +``` + +NetBSD package versions must not contain dashes (the last dash separates the +package name from the version), so DTail's `4.3.2-ng` becomes `4.3.2ng`. + +## Install/Update on Frontends via Rex + +```sh +cd ~/git/conf/frontends +rex gogios_install # installs or updates gogios on blowfish + fishfinger (OpenBSD) and f0-f3 (FreeBSD) +rex gogios # full setup: gogios_install + config + cron +``` + +The `gogios_install` Rex task auto-detects the OS and uses `pkg install` (FreeBSD) or `pkg_add` (OpenBSD). diff --git a/prompts/skills/f3s-pkgrepo/references/repo-architecture.md b/prompts/skills/f3s-pkgrepo/references/repo-architecture.md new file mode 100644 index 0000000..ae42e1f --- /dev/null +++ b/prompts/skills/f3s-pkgrepo/references/repo-architecture.md @@ -0,0 +1,80 @@ +# Repo Architecture + +Custom FreeBSD, OpenBSD, NetBSD, and Rocky Linux package repository served from k3s. + +## Overview + +- **nginx pod** in k3s `infra` namespace serves static files from a PV +- URL: `https://pkgrepo.f3s.buetow.org` +- Path prefixes: `/freebsd/`, `/openbsd/`, `/netbsd/`, `/rockylinux/` +- TLS terminated by OpenBSD relayd on the internet gateways (not in the pod) +- DNS, ACME certs, httpd fallback, and relayd routing auto-generated from `@f3s_hosts` in `frontends/Rexfile` +- HTTP always redirected to HTTPS by the OpenBSD gateways — client URLs must use `https://` + +## PV Directory Structure + +``` +/data/nfs/k3svolumes/pkgrepo/ ← NFS-backed PV, physically on f0 + freebsd/ + FreeBSD:15:amd64/ + latest/ + All/ # .pkg files + packagesite.pkg # repo metadata (generated by `pkg repo`) + meta.conf / meta + data.pkg + openbsd/ + 7.8/ + packages/ + amd64/ # .tgz files (signify-signed) + netbsd/ + 10.1/ + packages/ + aarch64/ # .tgz files (unsigned) + pkg_summary.gz for pkgin + rockylinux/ + 9/ + x86_64/ # .rpm files + repodata/ + aarch64/ # .rpm files + repodata/ +``` + +## Key Config Files + +| File | Purpose | +|------|---------| +| `f3s/pkgrepo/helm-chart/` | Helm chart (nginx deployment, service, ingress, PV, configmap) | +| `f3s/argocd-apps/infra/pkgrepo.yaml` | ArgoCD Application manifest | +| `f3s/pkgrepo/test-artifacts/build-test-packages.sh` | Builds hello-test packages for FreeBSD/OpenBSD | +| `frontends/Rexfile` (`@f3s_hosts`) | DNS + routing entry for `pkgrepo.f3s.buetow.org` | +| `frontends/Rexfile` (`pkgrepo_setup`) | Adds `PKG_PATH` to root's `.profile` on OpenBSD frontends | +| `packages/Makefile` | Cross-compiles, packages, and uploads for all OSes | +| `packages/scripts/pkg-freebsd.sh` | Runs on f0 via SSH: `pkg create` + `pkg repo` + copy to PV | +| `packages/scripts/pkg-openbsd.sh` | Runs on fishfinger via SSH: `pkg_create` + signify signing | +| `packages/scripts/pkg-dtail-openbsd.sh` | DTail multi-binary OpenBSD packaging | +| `packages/scripts/pkg-dtail-freebsd.sh` | DTail multi-binary FreeBSD packaging | +| `packages/scripts/pkg-dtail-netbsd.sh` | DTail multi-binary NetBSD packaging (runs on pi0 via SSH: `pkg_create` + `pkg_summary.gz`) | +| `packages/scripts/pkg-dtail-rpm.sh` | Builds DTail RPMs from prebuilt or locally built payloads | + +## SSH Access for Package Tasks + +| Host | Access | Root | Notes | +|------|--------|------|-------| +| f0 (FreeBSD, NFS) | `ssh -p 22 f0.lan.buetow.org` | `doas` | PV is local; default shell is **csh** | +| fishfinger (OpenBSD) | `ssh rex@fishfinger.buetow.org` | `doas` | OpenBSD packages built here, then copied to f0 PV | +| blowfish (OpenBSD) | `ssh rex@blowfish.buetow.org` | `doas` | Same setup as fishfinger | +| pi0 (NetBSD) | `ssh -p 22 paul@pi0.lan.buetow.org` | `doas` | NetBSD packages assembled here (`pkg_create`), then copied to f0 PV; non-interactive SSH PATH lacks `/usr/sbin` (`pkg_*` live there) | + +## Nginx Pod Gotcha — Stale NFS Handle + +After writing new files to the PV on f0, the nginx pod can get a stale NFS file handle (error 116) and return HTTP 500. Fix with a pod restart: + +```sh +kubectl -n infra rollout restart deployment/pkgrepo +kubectl -n infra rollout status deployment/pkgrepo +``` + +## Per-OS Repo Notes + +- **FreeBSD**: version follows ABI naming (`FreeBSD:15:amd64`); packages unsigned (`signature_type: "NONE"`); always regenerate metadata with `pkg repo` after adding/removing packages +- **OpenBSD**: no repo index needed — `pkg_add` fetches by name; packages signed with signify; version in path must match host OS (currently 7.8) +- **NetBSD**: packages unsigned; `pkg_add` works with a direct package URL, `pkg_summary.gz` alongside the packages enables pkgin; package versions must not contain dashes (`4.3.2-ng` → `4.3.2ng`); `+BUILD_INFO` (`MACHINE_ARCH`/`OS_VERSION`) is checked by `pkg_add` on install, which is why packages are assembled natively on pi0 +- **Rocky Linux**: standard DNF layout; unsigned repo (`gpgcheck=0`); architecture-specific paths (`x86_64` / `aarch64`) +- **FreeBSD csh**: default shell is csh — avoid inline one-liners with `||`, `&&`, `!`, or multi-line quoting over SSH; use piped `/bin/sh` or separate SSH invocations diff --git a/prompts/skills/f3s-raspberry-pi/SKILL.md b/prompts/skills/f3s-raspberry-pi/SKILL.md index 0ed3ac1..2ede716 100644 --- a/prompts/skills/f3s-raspberry-pi/SKILL.md +++ b/prompts/skills/f3s-raspberry-pi/SKILL.md @@ -13,7 +13,7 @@ The four Raspberry Pi 3 nodes of the f3s homelab. The master host/IP inventory - Configuring or troubleshooting pi0–pi3 (NetBSD static site pair, or Rocky Pi-hole pair) - The static `f3s.buetow.org` / `snonux.foo` site (bozohttpd, relayd forwarding, vhosts) - Pi-hole and `*.f3s.lan.buetow.org` LAN wildcard DNS -- For the WireGuard mesh these depend on, see the [`f3s`](../f3s/SKILL.md) hub's `wireguard.md`; for DTail/dserver on the Pis, [`f3s-dtail`](../f3s-dtail/SKILL.md); for building the NetBSD dserver package, the [`pkgrepo`](../pkgrepo/SKILL.md) skill. +- For the WireGuard mesh these depend on, see the [`f3s`](../f3s/SKILL.md) hub's `wireguard.md`; for DTail/dserver on the Pis, [`f3s-dtail`](../f3s-dtail/SKILL.md); for building the NetBSD dserver package, the [`f3s-pkgrepo`](../f3s-pkgrepo/SKILL.md) skill. ## Node roles @@ -50,5 +50,5 @@ canonical detail in [references/bootstrap-netbsd-pi.md](references/bootstrap-net ## Reference Files -- [NetBSD Pi Setup](references/bootstrap-netbsd-pi.md) — how services are installed on `pi0`/`pi1` (NetBSD): doas/pkgin bootstrap, WireGuard via userspace `wireguard-go` (no native `wg(4)`), bozohttpd (`-X` dir-listing, vhost symlinks), uptimed from source, npf firewall, content-sync. dserver (DTail) is installed from the custom pkgrepo — see the [`pkgrepo`](../pkgrepo/SKILL.md) skill's `dtail-package.md`. +- [NetBSD Pi Setup](references/bootstrap-netbsd-pi.md) — how services are installed on `pi0`/`pi1` (NetBSD): doas/pkgin bootstrap, WireGuard via userspace `wireguard-go` (no native `wg(4)`), bozohttpd (`-X` dir-listing, vhost symlinks), uptimed from source, npf firewall, content-sync. dserver (DTail) is installed from the custom pkgrepo — see the [`f3s-pkgrepo`](../f3s-pkgrepo/SKILL.md) skill's `dtail-package.md`. - [Pi-hole on Pis](references/pihole-pi.md) — **pi2/pi3** Docker Pi-hole, **`~/pihole`**, **`*.f3s.lan.buetow.org` → 192.168.1.138**, paths under **`f3s/pihole/docker-pi/`** diff --git a/prompts/skills/f3s-raspberry-pi/references/bootstrap-netbsd-pi.md b/prompts/skills/f3s-raspberry-pi/references/bootstrap-netbsd-pi.md index 44d4b58..99c29a2 100644 --- a/prompts/skills/f3s-raspberry-pi/references/bootstrap-netbsd-pi.md +++ b/prompts/skills/f3s-raspberry-pi/references/bootstrap-netbsd-pi.md @@ -291,7 +291,7 @@ group default { } ``` -Port 2222 is dserver (DTail) — see the `pkgrepo` skill's `dtail-package.md` +Port 2222 is dserver (DTail) — see the `f3s-pkgrepo` skill's `dtail-package.md` for the install steps. `family inet4`/`inet6` must be explicit on multi-family interfaces or diff --git a/prompts/skills/f3s-rocky-vm-setup/SKILL.md b/prompts/skills/f3s-rocky-vm-setup/SKILL.md new file mode 100644 index 0000000..bab9de0 --- /dev/null +++ b/prompts/skills/f3s-rocky-vm-setup/SKILL.md @@ -0,0 +1,39 @@ +--- +name: f3s-rocky-vm-setup +description: Reference for the plain Rocky Linux 9 bhyve VM (host `rocky`, 192.168.1.123) running on f3. Covers SSH keys, local git server remotes, tooling (tmux, fish, amp, claude-code, pi, taskwarrior, Rex), zrepl replication, and restricted user privileges. Use when working on or replicating the rocky VM configuration. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)). +--- + +# Rocky VM Setup Reference + +The `rocky` VM is a plain Rocky Linux 9 bhyve guest on **f3** (LAN IP `192.168.1.123`, WireGuard `192.168.2.123`). It is **not** part of the k3s cluster and serves as a general-purpose build / dev / git client VM. + +Parent infrastructure: see the [`f3s`](../f3s/SKILL.md) skill (f3 host, zrepl, bhyve, git server). + +## When to Use + +- Working on or replicating the `rocky` VM configuration +- SSH keys, git remotes, installed tooling, tmux/fish first-run setup +- User privileges and sudoers, Rex deployment, zrepl replication + +## Reference Files + +Detailed reference documentation is in the `references/` subfolder — load the one that matches the task: + +- [Overview](references/overview.md) — VM role, SSH keys, `/etc/hosts` LAN aliases for all f3s hosts +- [Installed Tools](references/tools.md) — tool/version/install table, building taskwarrior 2.6.2 from source, first-run fish + fisher + Go tooling setup, tmux 3.2a compatibility note +- [Nested tmux](references/tmux.md) — `C-g` prefix on rocky vs `C-b` on earth, red/orange color scheme, source ordering, 256-color and truecolor (`COLORTERM`) passthrough +- [Git Remotes](references/git-remotes.md) — `r0`/`r1`/`r2` remotes replacing codeberg, `git@r{N}:30022`, repos pushed, authorized-keys secret +- [User and Privileges](references/privileges.md) — `root` full access; `paul` removed from `wheel`, NOPASSWD only for `update-coding-agents`, sudoers config +- [Scripts](references/scripts.md) — `/home/paul/scripts/update-coding-agents` (updates claude-code + pi) +- [Rex Usage](references/rex.md) — `rex pkg_rocky` (root) / `rex home` (paul), Rocky-specific `home_tmux_rocky` task +- [ZFS Snapshot / Replication](references/zrepl.md) — `zroot/bhyve/rocky` via zrepl on f3 → f2, retention; full config in `f3s` skill +- [Notes](references/notes.md) — `claude` wrapper must be a symlink not a shell script (fork bomb), Node.js 22 module, `amp` non-TTY panic + +## Quick Reference + +- Host: `rocky` / `192.168.1.123` (LAN), `192.168.2.123` (WireGuard) +- Parent: f3 (see `f3s` skill) +- Git remotes: `ssh://git@r0:30022/repos/REPO.git` (and r1, r2) +- tmux prefix: `C-g` (rocky inner) over `C-b` (earth outer) +- paul sudo: only `/home/paul/scripts/update-coding-agents` +- Replication: zrepl f3 → f2, every 10 min \ No newline at end of file diff --git a/prompts/skills/f3s-rocky-vm-setup/references/git-remotes.md b/prompts/skills/f3s-rocky-vm-setup/references/git-remotes.md new file mode 100644 index 0000000..bddc6c3 --- /dev/null +++ b/prompts/skills/f3s-rocky-vm-setup/references/git-remotes.md @@ -0,0 +1,13 @@ +# Git Remotes on Rocky + +All repos available on the local git server have `r0`, `r1`, `r2` remotes replacing any codeberg ones: + +``` +url = ssh://git@r0:30022/repos/REPO.git +url = ssh://git@r1:30022/repos/REPO.git +url = ssh://git@r2:30022/repos/REPO.git +``` + +Repos pushed: conf, dotfiles, gemtexter, gitsyncer, goprecords, gt, hexai, hypr, ior, photoalbum, rcm, snonux, tasksamurai, wireguardmeshgenerator + +The public keys of both `root` and `paul` on rocky are in the k3s `git-server-authorized-keys` secret (namespace `cicd`). diff --git a/prompts/skills/f3s-rocky-vm-setup/references/notes.md b/prompts/skills/f3s-rocky-vm-setup/references/notes.md new file mode 100644 index 0000000..124c345 --- /dev/null +++ b/prompts/skills/f3s-rocky-vm-setup/references/notes.md @@ -0,0 +1,5 @@ +# Notes + +- The `claude` wrapper must **not** be a shell script calling the JS wrapper — that caused a fork bomb because `cli-wrapper.cjs` tried to exec the `claude` binary but found the script instead. Use a direct symlink or the npm-installed binary. +- Node.js is installed via `dnf module install nodejs:22/common` (pi 0.75.0+ requires Node >= 22.19.0). +- `amp` panics in non-TTY environments — that's expected for a TUI editor. diff --git a/prompts/skills/f3s-rocky-vm-setup/references/overview.md b/prompts/skills/f3s-rocky-vm-setup/references/overview.md new file mode 100644 index 0000000..e606f37 --- /dev/null +++ b/prompts/skills/f3s-rocky-vm-setup/references/overview.md @@ -0,0 +1,42 @@ +# Rocky VM Overview + +The `rocky` VM is a plain Rocky Linux 9 bhyve guest on **f3** (LAN IP `192.168.1.123`, WireGuard `192.168.2.123`). It is **not** part of the k3s cluster and serves as a general-purpose build / dev / git client VM. + +Parent infrastructure: see the [`f3s`](../../f3s/SKILL.md) skill (f3 host, zrepl, bhyve, git server). + +## SSH Keys + +| Key | Path | Purpose | +|-----|------|---------| +| Root VM key | `/root/.ssh/id_ed25519` | Git server SSH auth | +| Paul VM key | `/home/paul/.ssh/id_ed25519` | Git server SSH auth, local remotes | + +The public keys are added to the k3s `git-server-authorized-keys` secret (namespace `cicd`) so both `root` and `paul` can push/pull via `git@r{N}:30022`. + +```sh +# Regenerate if needed +ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519 -C 'root@rocky.f3s.lan.buetow.org' +``` + +## /etc/hosts + +Short LAN aliases for all f3s hosts (short, `.lan`, and `.lan.buetow.org` variants): + +``` +# f3s k3s node LAN aliases +192.168.1.120 r0 r0.lan r0.lan.buetow.org +192.168.1.121 r1 r1.lan r1.lan.buetow.org +192.168.1.122 r2 r2.lan r2.lan.buetow.org + +# f3s FreeBSD host LAN aliases +192.168.1.130 f0 f0.lan f0.lan.buetow.org +192.168.1.131 f1 f1.lan f1.lan.buetow.org +192.168.1.132 f2 f2.lan f2.lan.buetow.org +192.168.1.133 f3 f3.lan f3.lan.buetow.org + +# f3s Raspberry Pi LAN aliases +192.168.1.125 pi0 pi0.lan pi0.lan.buetow.org +192.168.1.126 pi1 pi1.lan pi1.lan.buetow.org +192.168.1.127 pi2 pi2.lan pi2.lan.buetow.org +192.168.1.128 pi3 pi3.lan pi3.lan.buetow.org +``` diff --git a/prompts/skills/f3s-rocky-vm-setup/references/privileges.md b/prompts/skills/f3s-rocky-vm-setup/references/privileges.md new file mode 100644 index 0000000..d8f25ba --- /dev/null +++ b/prompts/skills/f3s-rocky-vm-setup/references/privileges.md @@ -0,0 +1,20 @@ +# User and Privileges + +**`root`** — full root, used for package installs and Rex tasks. + +**`paul`** +- **Removed from `wheel`** group. No general `sudo` access. +- **Only** allowed to run without password: + ``` + /home/paul/scripts/update-coding-agents + ``` +- Home: `/home/paul` +- Git repos: `~/git/` (cloned via local `r0`/`r1`/`r2` remotes) + +## Sudoers config + +``` +paul ALL=(root) NOPASSWD: /home/paul/scripts/update-coding-agents +``` + +No other sudo privileges. The `wheel-nopasswd` file was removed and paul was removed from the `wheel` group. diff --git a/prompts/skills/f3s-rocky-vm-setup/references/rex.md b/prompts/skills/f3s-rocky-vm-setup/references/rex.md new file mode 100644 index 0000000..1eba83a --- /dev/null +++ b/prompts/skills/f3s-rocky-vm-setup/references/rex.md @@ -0,0 +1,22 @@ +# Rex Usage + +The dotfiles repo (`~/git/dotfiles`) contains the main `Rexfile`. + +```sh +# Install packages (as root) +rex pkg_rocky + +# Deploy dotfiles (as paul) +rex home +``` + +`pkg_rocky` uses Rex's `pkg` directive which requires root — no `sudo` wrappers. + +## Rocky-specific Rex tasks + +| Task | Runs when | What it does | +|------|-----------|--------------| +| `home_tmux_rocky` | `hostname =~ /rocky/` | Sources `tmux.rocky.conf` at the **end** of `tmux.conf` so red/orange colors win | +| `pkg_rocky` | Manually (as root) | Installs packages: tmux, fish, helix, zoxide, fzf, golang, nodejs, etc. | + +The `home_tmux_rocky` task also cleans stale references from `tmux.local.conf` and strips the `extended-keys-format` line for tmux 3.2a compatibility. diff --git a/prompts/skills/f3s-rocky-vm-setup/references/scripts.md b/prompts/skills/f3s-rocky-vm-setup/references/scripts.md new file mode 100644 index 0000000..14fa7e0 --- /dev/null +++ b/prompts/skills/f3s-rocky-vm-setup/references/scripts.md @@ -0,0 +1,20 @@ +# Scripts + +## `/home/paul/scripts/update-coding-agents` + +```sh +#!/bin/sh +set -e +if [ "$(id -u)" -ne 0 ]; then + exec sudo "$0" "$@" +fi +echo "Updating Claude Code..." +npm update -g @anthropic-ai/claude-code @anthropic-ai/claude-code-linux-x64 +echo "Updating pi coding agent..." +npm update -g @earendil-works/pi-coding-agent +echo "All coding agents updated." +``` + +Run as paul: `$ /home/paul/scripts/update-coding-agents` + +Sudoers allows this specific script without a password. No other root access for paul. diff --git a/prompts/skills/f3s-rocky-vm-setup/references/tmux.md b/prompts/skills/f3s-rocky-vm-setup/references/tmux.md new file mode 100644 index 0000000..8c1a112 --- /dev/null +++ b/prompts/skills/f3s-rocky-vm-setup/references/tmux.md @@ -0,0 +1,72 @@ +# Nested tmux on Rocky + +Earth (outer tmux) uses the default **C-b** prefix. Rocky (inner tmux) uses **C-g** so you can control both layers. + +## Visual distinction + +| Layer | Prefix | Active Border | Status Bar | Pane Indicators | +|-------|--------|---------------|------------|-----------------| +| **Earth (outer)** | `C-b` | **Magenta** | White-on-purple | Default blue | +| **Rocky (inner)** | `C-g` | **Bright Red** | **Black-on-orange** (`colour208`) with `[ROCKY]` label | **Red/orange** pane numbers, border labels | + +## Workflow + +| Key | Action | +|-----|--------| +| `C-b c` | Create window in outer tmux (earth) | +| `C-g c` | Create window in inner tmux (rocky) | +| `C-b b` | Send `C-b` through to inner tmux | +| `C-g g` | Send `C-g` through to inner-inner tmux | + +## Config + +Rocky config is in `~/.config/tmux/tmux.rocky.conf` and sourced from `tmux.local.conf`: + +```sh +# ~/.config/tmux/tmux.rocky.conf +unbind C-b +set -g prefix C-g +bind C-g send-prefix + +# Drastic RED/ORANGE color scheme +set -g pane-active-border-style 'fg=brightred,bold' +set -g status-style 'bg=colour208,fg=black,bold' +set -g status-left ' [ROCKY] #[bg=brightred,fg=white] #S ' +set -g window-status-current-style 'bg=brightred,fg=white,bold' +set -g window-status-style 'bg=colour208,fg=black' + +# Active pane indicators +set -g display-panes-colour colour208 # prefix+q pane numbers +set -g display-panes-active-colour brightred # active pane number +set -g pane-border-status top # show pane info on borders +set -g pane-border-format '#[fg=colour208] #{pane_index} #[fg=brightred]#{pane_title} ' +set -g window-status-current-format ' #I*#[bg=brightred,fg=white] #W ' +``` + +## Important: source ordering + +The rocky config must be sourced **at the end of** `~/.config/tmux/tmux.conf` so its color overrides win over the shared config. The `home_tmux_rocky` Rex task handles this by: + +1. Cleaning any stale reference from `tmux.local.conf` +2. Appending `source-file ~/.config/tmux/tmux.rocky.conf` to the **end** of `tmux.conf` + +## Terminal / color support + +The inner tmux must advertise 256 colors (or helix/fzf/etc fall back to 16): + +```sh +set -g default-terminal 'tmux-256color' +set -ga terminal-overrides ',xterm-256color:Tc,*-256color:Tc' +``` + +Without `default-terminal`, tmux defaults to `TERM=screen` (8 colors). + +### Truecolor (24-bit / 16777216 colors) + +For truecolor themes in helix, `COLORTERM=truecolor` must also be present inside tmux. tmux strips it by default via `update-environment`. Pass it through: + +```sh +set -g update-environment 'DISPLAY SSH_ASKPASS SSH_AGENT_LAUNCHER SSH_AUTH_SOCK SSH_CONNECTION SSH_TTY WINDOWID XAUTHORITY TERM COLORTERM' +``` + +Without this, helix sees `COLORTERM=` and falls back to 256 colors only, breaking truecolor themes. diff --git a/prompts/skills/f3s-rocky-vm-setup/references/tools.md b/prompts/skills/f3s-rocky-vm-setup/references/tools.md new file mode 100644 index 0000000..b3307ef --- /dev/null +++ b/prompts/skills/f3s-rocky-vm-setup/references/tools.md @@ -0,0 +1,69 @@ +# Installed Tools + +| Tool | Version | How Installed | +|------|---------|---------------| +| tmux | 3.2a | `dnf install -y tmux` | +| tmux prefix | C-g | **Rocky override** — nested tmux (see [tmux.md](tmux.md)) | +| fish | 3.7.1 | `dnf install -y fish` (EPEL) | +| helix | 25.07.1 | `dnf install -y helix helix-themes` (EPEL) | +| amp | 0.7.1 | Downloaded binary from GitHub releases | +| claude-code | 2.1.169 | `npm install -g @anthropic-ai/claude-code` | +| pi coding agent | 0.79.0 | `npm install -g @earendil-works/pi-coding-agent` | +| taskwarrior | 2.6.2 | **Built from source** (see below) | +| Rex | 1.16.1 | `cpanm Rex` (requires expat-devel, perl-LWP-Protocol-https) | +| zoxide | 0.9.8 | `dnf install -y zoxide` (EPEL) | +| fzf | 0.58.0 | `dnf install -y fzf` (EPEL) | +| fzf fish plugin | — | **fisher install PatrickF1/fzf.fish** | +| ask, hexai*, gt, gitsyncer, etc. | — | `go install codeberg.org/snonux/...` (see update::tools) | + +## Building taskwarrior from source + +Rocky 9 does not ship `task`/`taskwarrior`. v3.x requires Rust; v2.6.2 compiles cleanly. + +```sh +# deps +dnf install -y cmake gcc-c++ make libuuid-devel gnutls-devel libssh2-devel + +# build +git clone --depth 1 --branch v2.6.2 \ + https://github.com/GothenburgBitFactory/taskwarrior.git /tmp/tw-build +cd /tmp/tw-build +cmake -S . -B build -DCMAKE_BUILD_TYPE=Release +cmake --build build -j$(nproc) +cmake --install build + +# verify +/usr/local/bin/task --version # 2.6.2 +``` + +## First-run fish setup + +After the dotfiles `home` task deploys fish config, some plugins and binaries are expected but not yet present: + +```sh +# 1. Install fisher (fish plugin manager) +curl -sL https://raw.githubusercontent.com/jorgebucaran/fisher/main/functions/fisher.fish | source +fisher install jorgebucaran/fisher + +# 2. Install the fzf.fish plugin (provides fzf_configure_bindings) +fisher install PatrickF1/fzf.fish + +# 3. First-run taskwarrior creates ~/.taskrc +yes | task >/dev/null 2>&1 + +# 4. Install Go tooling binaries (run as paul) +for prog in ask hexai hexai-lsp-server hexai-tmux-action hexai-mcp-server; do + go install codeberg.org/snonux/hexai/cmd/$prog@latest +done +for prog in tasksamurai timesamurai gt; do + go install codeberg.org/snonux/$prog/cmd/$prog@latest +done +for prog in gitsyncer gos snonux; do + go install codeberg.org/snonux/$prog/cmd/$prog@latest +done +# (foostore, loadbars, totalrecall, goprecords may need X11/GL deps for GUI — skip on headless) +``` + +## tmux 3.2a compatibility note + +The dotfiles `tmux.conf` includes `set -g extended-keys-format csi-u` (tmux 3.3+). On rocky this line is automatically stripped by the `home_tmux_rocky` Rex task.