4 files changed, 281 insertions, 0 deletions
diff --git a/prompts/skills/f3s/SKILL.md b/prompts/skills/f3s/SKILL.md
index 150ee6d..aef73d9 100644
--- a/prompts/skills/f3s/SKILL.md
+++ b/prompts/skills/f3s/SKILL.md
@@ -37,6 +37,8 @@ Detailed reference documentation is in the `references/` subfolder:
 
 Package repository details were split into the sibling `pkgrepo` skill. Use `pkgrepo` for `pkgrepo.f3s.buetow.org`, repo layout, package publication, and client repo configuration.
 
+The plain Rocky Linux VM on f3 (`rocky`, `192.168.1.123`) is documented in the sibling [`rocky-vm-setup`](skills/rocky-vm-setup) skill. Use that for SSH keys, git remotes, tooling (tmux/fish/claude-code/pi/taskwarrior/Rex), zrepl replication, and user privileges.
+
 ## Quick Reference: Host IPs
 
 | Host | Role | LAN IP | WireGuard IP |
diff --git a/prompts/skills/f3s/references/storage/nfs.md b/prompts/skills/f3s/references/storage/nfs.md
index 5aa0743..2c21114 100644
--- a/prompts/skills/f3s/references/storage/nfs.md
+++ b/prompts/skills/f3s/references/storage/nfs.md
@@ -160,3 +160,45 @@ NFS path structure on k3s nodes: `/data/nfs/k3svolumes/<app>/`
 
 The `nfs-mount-monitor` watchdog on each r-node detects and repairs stale or
 hung mounts automatically — see [nfs-mount-monitor.md](nfs-mount-monitor.md).
+
+## NFS Client Configuration (earth, roaming laptop)
+
+`earth` mounts two exports (`/earthdata`, `/k3svolumes`) the same way as the
+r-nodes — local stunnel client on `127.0.0.1:2323` → CARP VIP `192.168.1.138:2323`.
+Certs live in `/etc/stunnel/` (`earth-stunnel.pem`, `ca-cert.pem`). The mounts are
+`noauto` and mounted on demand.
+
+**Mount options differ from the r-nodes on purpose.** The r-nodes are on a fast,
+stable LAN and use `hard,timeo=600`. `earth` roams on WiFi, so it uses `soft` so it
+fails instead of hanging forever when off-network. The working `/etc/fstab` lines:
+
+```
+127.0.0.1:/earthdata   /data/nfs/earthdata   nfs4 port=2323,_netdev,soft,timeo=150,retrans=3,nofail,noauto 0 0
+127.0.0.1:/k3svolumes  /data/nfs/k3svolumes  nfs4 port=2323,_netdev,soft,timeo=150,retrans=3,nofail,noauto 0 0
+```
+
+> **Critical: do NOT use `timeo=10` on earth.** `timeo` is in **deciseconds**, so
+> `timeo=10` = **1.0s per RPC**. A large write is split into many `wsize` (128KB)
+> RPCs plus a final `COMMIT`; with `soft,timeo=10,retrans=2`, any single RPC or the
+> COMMIT that can't be acked within ~1s over jittery WiFi makes the soft mount abort
+> mid-transfer with EIO. That both **corrupts the partial file** and leaves a wedged
+> NFS session that retransmits constantly, saturating 2.4GHz WiFi airtime (slows ALL
+> WiFi traffic, e.g. unrelated scp to/from earth). `timeo=150` (15s) gives each RPC
+> and the COMMIT enough slack to ride out WiFi jitter while still failing eventually
+> if truly disconnected. Drop the deprecated `intr` option (no-op on modern Linux).
+
+**Superblock caching gotcha:** Linux NFS shares one superblock per `server:export`,
+so remounting with new options is ignored while any process still references the old
+mount (e.g. a shell with its cwd inside it). Either release all references first, or
+remount with `-o nosharecache` to force a fresh superblock with the new options.
+
+**The real fix is the mount option, not the transfer tool.** With `timeo=150` the
+mount never wedges, so transfers just complete and plain `cp`/`mv` are perfectly
+safe — they check return codes and report errors normally. The one time we saw a
+half-finalized destination file, it was *not* a `cp`/`mv` flaw: the in-flight `mv`
+was killed mid-finalize by a forced f0 reboot + stunnel restart that we only did
+*because* the mount had wedged (the `timeo=10` bug). Fix the hang and that whole
+chain disappears. `rsync` (temp-file-then-rename, `--remove-source-files` only after
+a verified copy) is worth using as **interruption insurance** on a flaky WiFi link —
+it can't leave a bad file under the real name if the link dies mid-transfer — but it
+is not a substitute for the `timeo` fix and `cp`/`mv` do not "cause corruption."
diff --git a/prompts/skills/f3s/references/storage/troubleshooting.md b/prompts/skills/f3s/references/storage/troubleshooting.md
index 7f7adfd..7baaa63 100644
--- a/prompts/skills/f3s/references/storage/troubleshooting.md
+++ b/prompts/skills/f3s/references/storage/troubleshooting.md
@@ -19,6 +19,38 @@ After NFS is restored on the server side, the `nfs-mount-monitor` systemd timer
 
 **Note:** The monitor catches three failure modes: missing mountpoint, stat hang (reads unresponsive), and **silent write hang** (reads OK but writes block — the hardest case, e.g. stunnel-wrapped NFSv4 after a CARP failover). Watch the consecutive-failure counter via Prometheus (`nfs_mount_monitor_consecutive_failures`) — warning fires at ≥3, critical at ≥5. At 5 consecutive failures the node cordons itself and reboots.
 
+### Large file transfers from earth hang / WiFi slows to a crawl
+
+**Symptom**: copying a large file into earth's NFS mount hangs; kernel logs show
+`nfs: server 127.0.0.1 not responding, timed out` and `NFSv4: state recovery
+failed ... error = -116` (ESTALE). Unrelated WiFi traffic on earth (e.g. scp
+from f1) also slows dramatically at the same time.
+
+**Root cause**: earth's mount used `soft,timeo=10` (1.0s per RPC). Large writes
+abort mid-transfer, leaving a wedged NFS session that retransmits constantly and
+saturates 2.4GHz WiFi airtime — which throttles all of earth's WiFi traffic. This
+is a **client-side** problem; f0 itself is healthy (verified: local write to
+`zdata` ~1.6 GB/s, pool ONLINE, disk_wait ~1ms, cores 71–79°C).
+
+**Fix** (no f0 reboot needed — rebooting f0 only "worked" as a side effect of
+tearing down the wedged session, and it disrupts the k3s cluster via CARP failover):
+
+```sh
+# On earth: clear the wedged session
+sudo systemctl restart stunnel
+
+# Fix the mount options (see nfs.md → earth client config): use timeo=150, soft
+# Force new options past the cached superblock if a process still holds the mount:
+sudo umount -l /data/nfs/earthdata
+sudo mount -t nfs4 -o port=2323,_netdev,soft,timeo=150,retrans=3,nosharecache \
+  127.0.0.1:/earthdata /data/nfs/earthdata
+```
+
+Then redo the transfer with `rsync --remove-source-files` (not `mv`), which avoids
+corrupt partials. Diagnostics that confirmed the transport (not f0) was the limit:
+raw ssh earth→f0 ~3 MB/s, NFS ~5 MB/s, while the radio negotiated 97/206 Mbit/s —
+the ceiling is single-stream TCP over jittery 2.4GHz WiFi + stunnel TLS, not f0 I/O.
+
 ### Checklist for NFS outage on CARP MASTER (f0 or f1)
 
 ```sh
diff --git a/prompts/skills/rocky-vm-setup/SKILL.md b/prompts/skills/rocky-vm-setup/SKILL.md
new file mode 100644
index 0000000..ddb78e8
--- /dev/null
+++ b/prompts/skills/rocky-vm-setup/SKILL.md
@@ -0,0 +1,205 @@
+---
+name: rocky-vm-setup
+description: Reference for the plain Rocky Linux 9 bhyve VM (host `rocky`, 192.168.1.123) running on f3. Covers SSH keys, local git server remotes, tooling (tmux, fish, amp, claude-code, pi, taskwarrior, Rex), zrepl replication, and restricted user privileges. Use when working on or replicating the rocky VM configuration.
+---
+
+# Rocky VM Setup Reference
+
+The `rocky` VM is a plain Rocky Linux 9 bhyve guest on **f3** (LAN IP `192.168.1.123`, WireGuard `192.168.2.123`). It is **not** part of the k3s cluster and serves as a general-purpose build / dev / git client VM.
+
+Parent infrastructure: see the [`f3s`](skills/f3s) skill (f3 host, zrepl, bhyve, git server).
+
+---
+
+## SSH Keys
+
+| Key | Path | Purpose |
+|-----|------|---------|
+| Root VM key | `/root/.ssh/id_ed25519` | Git server SSH auth |
+| Paul VM key | `/home/paul/.ssh/id_ed25519` | Git server SSH auth, local remotes |
+
+The public keys are added to the k3s `git-server-authorized-keys` secret (namespace `cicd`) so both `root` and `paul` can push/pull via `git@r{N}:30022`.
+
+```sh
+# Regenerate if needed
+ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519 -C 'root@rocky.f3s.lan.buetow.org'
+```
+
+---
+
+## /etc/hosts
+
+Short LAN aliases for all f3s hosts (short, `.lan`, and `.lan.buetow.org` variants):
+
+```
+# f3s k3s node LAN aliases
+192.168.1.120 r0 r0.lan r0.lan.buetow.org
+192.168.1.121 r1 r1.lan r1.lan.buetow.org
+192.168.1.122 r2 r2.lan r2.lan.buetow.org
+
+# f3s FreeBSD host LAN aliases
+192.168.1.130 f0 f0.lan f0.lan.buetow.org
+192.168.1.131 f1 f1.lan f1.lan.buetow.org
+192.168.1.132 f2 f2.lan f2.lan.buetow.org
+192.168.1.133 f3 f3.lan f3.lan.buetow.org
+
+# f3s Raspberry Pi LAN aliases
+192.168.1.125 pi0 pi0.lan pi0.lan.buetow.org
+192.168.1.126 pi1 pi1.lan pi1.lan.buetow.org
+192.168.1.127 pi2 pi2.lan pi2.lan.buetow.org
+192.168.1.128 pi3 pi3.lan pi3.lan.buetow.org
+```
+
+---
+
+## Installed Tools
+
+| Tool | Version | How Installed |
+|------|---------|---------------|
+| tmux | 3.2a | `dnf install -y tmux` |
+| fish | 3.7.1 | `dnf install -y fish` (EPEL) |
+| amp | 0.7.1 | Downloaded binary from GitHub releases |
+| claude-code | 2.1.169 | `npm install -g @anthropic-ai/claude-code` |
+| pi coding agent | 0.74.2 | `npm install -g @earendil-works/pi-coding-agent` |
+| taskwarrior | 2.6.2 | **Built from source** (see below) |
+| Rex | 1.16.1 | `cpanm Rex` (requires expat-devel, perl-LWP-Protocol-https) |
+| zoxide | 0.9.8 | `dnf install -y zoxide` (EPEL) |
+| fzf | 0.58.0 | `dnf install -y fzf` (EPEL) |
+| fzf fish plugin | — | **fisher install PatrickF1/fzf.fish** |
+| ask, hexai*, gt, gitsyncer, etc. | — | `go install codeberg.org/snonux/...` (see update::tools) |
+
+### Building taskwarrior from source
+
+Rocky 9 does not ship `task`/`taskwarrior`. v3.x requires Rust; v2.6.2 compiles cleanly.
+
+```sh
+# deps
+dnf install -y cmake gcc-c++ make libuuid-devel gnutls-devel libssh2-devel
+
+# build
+git clone --depth 1 --branch v2.6.2 \
+  https://github.com/GothenburgBitFactory/taskwarrior.git /tmp/tw-build
+cd /tmp/tw-build
+cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
+cmake --build build -j$(nproc)
+cmake --install build
+
+# verify
+/usr/local/bin/task --version   # 2.6.2
+```
+
+### First-run fish setup
+
+After the dotfiles `home` task deploys fish config, some plugins and binaries are expected but not yet present:
+
+```sh
+# 1. Install fisher (fish plugin manager)
+curl -sL https://raw.githubusercontent.com/jorgebucaran/fisher/main/functions/fisher.fish | source
+fisher install jorgebucaran/fisher
+
+# 2. Install the fzf.fish plugin (provides fzf_configure_bindings)
+fisher install PatrickF1/fzf.fish
+
+# 3. First-run taskwarrior creates ~/.taskrc
+yes | task >/dev/null 2>&1
+
+# 4. Install Go tooling binaries (run as paul)
+for prog in ask hexai hexai-lsp-server hexai-tmux-action hexai-tmux-edit hexai-mcp-server; do
+    go install codeberg.org/snonux/hexai/cmd/$prog@latest
+done
+for prog in tasksamurai timesamurai gt; do
+    go install codeberg.org/snonux/$prog/cmd/$prog@latest
+done
+for prog in gitsyncer gos snonux; do
+    go install codeberg.org/snonux/$prog/cmd/$prog@latest
+done
+# (foostore, loadbars, totalrecall, goprecords may need X11/GL deps for GUI — skip on headless)
+```
+
+---
+
+## User and Privileges
+
+**`root`** — full root, used for package installs and Rex tasks.
+
+**`paul`**
+- **Removed from `wheel`** group. No general `sudo` access.
+- **Only** allowed to run without password:
+  ```
+  /home/paul/scripts/update-coding-agents
+  ```
+- Home: `/home/paul`
+- Git repos: `~/git/` (cloned via local `r0`/`r1`/`r2` remotes)
+
+---
+
+## Git Remotes
+
+All repos available on the local git server have `r0`, `r1`, `r2` remotes replacing any codeberg ones:
+
+```
+url = ssh://git@r0:30022/repos/REPO.git
+url = ssh://git@r1:30022/repos/REPO.git
+url = ssh://git@r2:30022/repos/REPO.git
+```
+
+Repos pushed: conf, dotfiles, gemtexter, gitsyncer, goprecords, gt, hexai, hypr, ior, photoalbum, rcm, snonux, tasksamurai, wireguardmeshgenerator
+
+---
+
+## Scripts
+
+`/home/paul/scripts/update-coding-agents`
+```sh
+#!/bin/sh
+set -e
+if [ "$(id -u)" -ne 0 ]; then
+    exec sudo "$0" "$@"
+fi
+echo "Updating Claude Code..."
+npm update -g @anthropic-ai/claude-code @anthropic-ai/claude-code-linux-x64
+echo "Updating pi coding agent..."
+npm update -g @earendil-works/pi-coding-agent
+echo "All coding agents updated."
+```
+
+Run as paul: `$ /home/paul/scripts/update-coding-agents`
+
+---
+
+## Rex Usage
+
+The dotfiles repo (`~/git/dotfiles`) contains the main `Rexfile`.
+
+```sh
+# Install packages (as root)
+rex pkg_rocky
+
+# Deploy dotfiles (as paul)
+rex home
+```
+
+`pkg_rocky` uses Rex's `pkg` directive which requires root — no `sudo` wrappers.
+
+---
+
+## ZFS Snapshot / Replication
+
+The rocky VM dataset `zroot/bhyve/rocky` is managed by **zrepl** on the FreeBSD host f3. It is **not** included in local `zfs-periodic` snapshots.
+
+| Property | Value |
+|------------|-------|
+| Snapshots | Every 10 minutes via zrepl (`zrepl_` prefix) |
+| Replication | f3 → f2 (`zroot/sink/f3/zroot/bhyve/rocky`) |
+| Retention | 10 immediate + 24 hourly + 14 daily |
+| Local snap job | `zroot/bhyve/rocky` excluded from `local_zfs_snapshots` |
+
+See [`f3s` skill zrepl.md](skills/f3s/references/storage/zrepl.md) for full config.
+
+---
+
+## Notes
+
+- The `claude` wrapper must **not** be a shell script calling the JS wrapper — that caused a fork bomb because `cli-wrapper.cjs` tried to exec the `claude` binary but found the script instead. Use a direct symlink or the npm-installed binary.
+- Node.js v20 is installed via `dnf module install nodejs:20/common`.
+- `amp` panics in non-TTY environments — that's expected for a TUI editor.