39 files changed, 228 insertions, 128 deletions
diff --git a/prompts/skills/f3s-dtail/SKILL.md b/prompts/skills/f3s-dtail/SKILL.md
new file mode 100644
index 0000000..779e1b3
--- /dev/null
+++ b/prompts/skills/f3s-dtail/SKILL.md
@@ -0,0 +1,32 @@
+---
+name: f3s-dtail
+description: Reference skill for DTail/dserver deployment across the f3s fleet, distributed log access over SSH on port 2222 — Pis arm64 (NetBSD + Rocky) vs r0–r2 amd64, r-VM root + root.authorized_keys cache, firewalld/npf 2222 rules, systemd timers. Package building lives in the `pkgrepo` skill. Use when deploying, configuring, or troubleshooting dserver on homelab hosts. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)).
+---
+
+# f3s DTail / dserver
+
+Distributed log access (DTail) via `dserver` across the f3s fleet. This skill owns
+the **runtime deployment/operations**; package *building/publishing* lives in the
+sibling [`pkgrepo`](../pkgrepo/SKILL.md) skill (`dtail-package.md`).
+
+## When to Use
+
+- Deploying, configuring, or troubleshooting `dserver` on homelab hosts (Pis, r0–r2)
+- SSH-on-2222 access, permissions/key-cache, firewall (firewalld/npf) rules, systemd timers
+- For building the `dtail` package (esp. the NetBSD build), use [`pkgrepo`](../pkgrepo/SKILL.md); for the Pi nodes themselves, [`f3s-raspberry-pi`](../f3s-raspberry-pi/SKILL.md); for hosts/IPs, the [`f3s`](../f3s/SKILL.md) hub.
+
+## Overview
+
+Distributed log access (`dcat`/`dtail`/`dgrep`/`dmap`) over SSH on port **2222** (not
+sshd's 22), by architecture: **pi2/pi3** linux/arm64, **pi0/pi1** netbsd/arm64 (installed
+from the `dtail` package in the custom [`pkgrepo`](../pkgrepo/SKILL.md)), **r0–r2** k3s
+Rocky VMs linux/amd64. The recurring gotchas — installing as `root`, listing `root` in
+`Server.Permissions.Users`, mirroring `/root/.ssh/authorized_keys` into the key cache
+(the cache script only walks `/home/*`), and opening 2222 in firewalld/npf — plus the
+exact per-host cross-build commands are the canonical detail in
+[references/dtail.md](references/dtail.md).
+
+## Reference Files
+
+- [DTail / dserver](references/dtail.md) — full deployment detail: Pis **arm64** vs r0–r2 **amd64**, r-VM **root** + `root.authorized_keys` cache, firewalld **2222**, systemd timers (section **dserver on r0, r1, r2**)
+- [dserver.d](references/dserver.d) — index: links to the **Rocky r-VM DTail** subsection and full **dtail.md**
diff --git a/prompts/skills/f3s/references/dserver.d b/prompts/skills/f3s-dtail/references/dserver.d
index 49d5c54..49d5c54 100644
--- a/prompts/skills/f3s/references/dserver.d
+++ b/prompts/skills/f3s-dtail/references/dserver.d
diff --git a/prompts/skills/f3s/references/dtail.md b/prompts/skills/f3s-dtail/references/dtail.md
index c7963a5..840f07d 100644
--- a/prompts/skills/f3s/references/dtail.md
+++ b/prompts/skills/f3s-dtail/references/dtail.md
@@ -10,7 +10,7 @@ Upstream install and examples live in the repo: `doc/installation.md`, `examples
 |-------|-----------|----------------|------------------|
 | **pi0–pi1** | NetBSD 10.1 **aarch64** (Raspberry Pi 3, see `bootstrap-netbsd-pi.md`) | Cross-build **netbsd/arm64**, `nozstd`; installed via `pkg_add` from the custom pkgrepo | `paul@piN.lan.buetow.org` |
 | **pi2–pi3** | Rocky Linux 9 **aarch64** (Raspberry Pi 3) | Cross-build **linux/arm64**, `nozstd` | `paul@piN.lan.buetow.org` |
-| **r0–r2** | Rocky Linux 9 **x86_64** (bhyve VMs, k3s nodes) | Cross-build **linux/amd64**, `nozstd` | Often `root@rN.lan.buetow.org` (see [Rocky Linux VMs](rocky-linux-vms.md)); add `root` (and `paul` if present) to **Server.Permissions.Users** in `dtail.json` |
+| **r0–r2** | Rocky Linux 9 **x86_64** (bhyve VMs, k3s nodes) | Cross-build **linux/amd64**, `nozstd` | Often `root@rN.lan.buetow.org` (see [Rocky Linux VMs](../../f3s/references/rocky-linux-vms.md)); add `root` (and `paul` if present) to **Server.Permissions.Users** in `dtail.json` |
 | **blowfish, fishfinger** | OpenBSD 7.8 **amd64** | Native OpenBSD package build | `rex@blowfish.buetow.org`, `rex@fishfinger.buetow.org` |
 
 `pi0`/`pi1` (NetBSD) run dserver since 2026-07-09 from the `dtail` package in the custom pkgrepo — build pipeline (`make dtail-netbsd`), install steps, rc.d/npf details, and gotchas live in the `pkgrepo` skill's `dtail-package.md`.
@@ -40,7 +40,7 @@ chmod 600 /var/run/dserver/cache/paul.authorized_keys
 
 ## dserver on r0, r1, r2 (k3s Rocky VMs, amd64)
 
-These hosts are the **x86_64** guests on f0/f1/f2. SSH and VM background: [Rocky Linux VMs](rocky-linux-vms.md), **DTail subsection** (same content in short form): [DTail (dserver) on r0–r2](rocky-linux-vms.md#dtail-dserver-on-r0r2). Shortcut index file: [dserver.d](dserver.d). **Do not** install the Pi **arm64** binary here.
+These hosts are the **x86_64** guests on f0/f1/f2. SSH and VM background: [Rocky Linux VMs](../../f3s/references/rocky-linux-vms.md), **DTail subsection** (same content in short form): [DTail (dserver) on r0–r2](../../f3s/references/rocky-linux-vms.md#dtail-dserver-on-r0r2). Shortcut index file: [dserver.d](dserver.d). **Do not** install the Pi **arm64** binary here.
 
 | Item | Value |
 |------|--------|
@@ -202,7 +202,7 @@ Use hostnames that resolve from where you run the client (often `*.lan.buetow.or
 
 For package-repo-backed DTail and other custom package repo tasks, use the sibling `pkgrepo` skill and its reference:
 
-- [Package Repositories](../../pkgrepo/references/package-repos.md)
+- [Package Repositories](../../pkgrepo/references/repo-architecture.md)
 
 That skill now owns:
 
diff --git a/prompts/skills/f3s-k3s/SKILL.md b/prompts/skills/f3s-k3s/SKILL.md
new file mode 100644
index 0000000..5ae3c47
--- /dev/null
+++ b/prompts/skills/f3s-k3s/SKILL.md
@@ -0,0 +1,25 @@
+---
+name: f3s-k3s
+description: Reference skill for the f3s k3s Kubernetes cluster, 3-node HA install on r0/r1/r2 Rocky VMs (bootstrap, kubeconfig, PVs, ArgoCD), off-LAN access (jump via OpenBSD frontend to root@r0.wg0 to kubectl), ingress (relayd, cert-manager), etcd recovery, and the reusable Rex r-node rollout. Use when installing, accessing, or troubleshooting the k3s cluster or deploying to r0/r1/r2. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)).
+---
+
+# f3s k3s Cluster
+
+3-node HA k3s cluster running on the Rocky Linux VMs r0/r1/r2 (one per
+FreeBSD bhyve host f0/f1/f2). All control-plane and etcd traffic flows
+over WireGuard.
+
+## When to Use
+
+- Installing, bootstrapping, or recovering the k3s cluster (etcd, kubeconfig, PVs, ArgoCD)
+- Reaching the cluster off-LAN, or deploying to r0/r1/r2 (incl. the Rex r-node rollout)
+- Ingress/cert-manager work on the cluster
+- For the underlying Rocky VMs, WireGuard mesh, storage/NFS, and host/IP inventory, see the [`f3s`](../f3s/SKILL.md) hub and [`f3s-storage`](../f3s-storage/SKILL.md).
+
+## Reference Files
+
+- [Install](references/install.md) — bootstrap, kubeconfig, etcd/controller-manager metrics, built-in components, NFS PV pattern, ArgoCD, node IP summary, useful commands
+- [Remote access (off-LAN)](references/remote-access.md) — reaching the cluster while roaming: **preferred** dedicated `wg0` kubectl context talking directly to `r0.wg0.wan.buetow.org:6443` over WireGuard (switch with `kubectl config use-context wg0`); fallback jump via OpenBSD frontend (`ssh -A rex@fishfinger.buetow.org` → `ssh root@r0.wg0` → `kubectl`), one-shot commands, and SSH port-forward tunnel
+- [Ingress](references/ingress.md) — OpenBSD `relayd` (internet) and FreeBSD `relayd` on CARP VIP (LAN), cert-manager wildcard, ingress pattern
+- [Troubleshooting](references/troubleshooting.md) — etcd Raft log corruption recovery; cluster-wide NFS outage pointer
+- [r-node Deploy (Rex)](references/r-node-deploy.md) — reusable Rex rollout to r0/r1/r2 (`f3s/r-nodes/Rexfile`, task `nfs_mount_monitor`): root SSH, `parallelism 3`, idempotent `file`/`on_change` reload
diff --git a/prompts/skills/f3s/references/k3s-setup/ingress.md b/prompts/skills/f3s-k3s/references/ingress.md
index 6beb4ea..6beb4ea 100644
--- a/prompts/skills/f3s/references/k3s-setup/ingress.md
+++ b/prompts/skills/f3s-k3s/references/ingress.md
diff --git a/prompts/skills/f3s/references/k3s-setup/install.md b/prompts/skills/f3s-k3s/references/install.md
index fae9899..324895a 100644
--- a/prompts/skills/f3s/references/k3s-setup/install.md
+++ b/prompts/skills/f3s-k3s/references/install.md
@@ -9,8 +9,8 @@
 ## Prerequisites
 
 - All Rocky Linux VMs (r0, r1, r2) updated and running
-- WireGuard mesh fully configured (see [wireguard.md](../wireguard.md))
-- NVMe disk emulation in place (see [rocky-linux-vms.md](../rocky-linux-vms.md)) — critical for etcd performance
+- WireGuard mesh fully configured (see [wireguard.md](../../f3s/references/wireguard.md))
+- NVMe disk emulation in place (see [rocky-linux-vms.md](../../f3s/references/rocky-linux-vms.md)) — critical for etcd performance
 
 ## Installation
 
@@ -107,8 +107,8 @@ Persistent volumes use `hostPath` pointing to NFS-mounted paths:
 ```
 
 NFS is mounted on all r nodes at `/data/nfs/k3svolumes` via stunnel → CARP VIP →
-freeBSD NFS — see [storage/nfs.md](../storage/nfs.md). The
-[`nfs-mount-monitor`](../storage/nfs-mount-monitor.md) watchdog auto-repairs
+freeBSD NFS — see [storage/nfs.md](../../f3s-storage/references/nfs.md). The
+[`nfs-mount-monitor`](../../f3s-storage/references/nfs-mount-monitor.md) watchdog auto-repairs
 hung mounts and force-deletes stuck pods.
 
 Example PV:
diff --git a/prompts/skills/f3s/references/r-node-deploy.md b/prompts/skills/f3s-k3s/references/r-node-deploy.md
index e020b21..294958c 100644
--- a/prompts/skills/f3s/references/r-node-deploy.md
+++ b/prompts/skills/f3s-k3s/references/r-node-deploy.md
@@ -137,7 +137,7 @@ ssh -A -J rex@fishfinger.buetow.org root@r0.wg0 \
 
 - The **behavior** of the deployed monitor (probes, fail-counter, cordon &
   auto-reboot escalation, alerting) is documented separately in
-  [storage/nfs-mount-monitor.md](storage/nfs-mount-monitor.md). This file
+  [storage/nfs-mount-monitor.md](../../f3s-storage/references/nfs-mount-monitor.md). This file
   covers only the *deploy mechanism*; that one covers *what gets deployed*.
 - There are sibling Rexfiles in the conf repo (`f3s/garage/Rexfile`,
   `frontends/Rexfile`) that follow the same group/user/idempotent-`file`
diff --git a/prompts/skills/f3s/references/k3s-setup/remote-access.md b/prompts/skills/f3s-k3s/references/remote-access.md
index 9837870..9837870 100644
--- a/prompts/skills/f3s/references/k3s-setup/remote-access.md
+++ b/prompts/skills/f3s-k3s/references/remote-access.md
diff --git a/prompts/skills/f3s/references/k3s-setup/troubleshooting.md b/prompts/skills/f3s-k3s/references/troubleshooting.md
index 01c6029..188f464 100644
--- a/prompts/skills/f3s/references/k3s-setup/troubleshooting.md
+++ b/prompts/skills/f3s-k3s/references/troubleshooting.md
@@ -46,4 +46,4 @@ kubectl get nodes  # r1 should return to Ready
 
 If NFS goes down cluster-wide, the root cause is usually on the FreeBSD NFS
 server side (f0/f1). Check CARP state, stunnel, nfsd, and
-`vfs.nfsd.nfs_privport` — see [storage/troubleshooting.md](../storage/troubleshooting.md).
+`vfs.nfsd.nfs_privport` — see [storage/troubleshooting.md](../../f3s-storage/references/troubleshooting.md).
diff --git a/prompts/skills/f3s/references/observability.md b/prompts/skills/f3s-observability/SKILL.md
index b6d7c35..2e13926 100644
--- a/prompts/skills/f3s/references/observability.md
+++ b/prompts/skills/f3s-observability/SKILL.md
@@ -1,4 +1,9 @@
-# Observability Stack
+---
+name: f3s-observability
+description: Reference skill for the f3s homelab observability stack, Prometheus, Grafana Alloy, Loki, Tempo, and alerting on the k3s cluster, plus FreeBSD host monitoring (node_exporter + recording rules). Use when working on metrics, logs, traces, dashboards, or alerts for the homelab. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)).
+---
+
+# f3s Observability Stack
 
 Observability stack deployed into the `monitoring` namespace of the k3s cluster.
 
@@ -20,10 +25,16 @@ Observability stack deployed into the `monitoring` namespace of the k3s cluster.
 | **Loki** | Log aggregation (single-binary mode) | **Disabled** |
 | **Tempo** | Distributed tracing backend | **Disabled** |
 
-## Sub-references
+## When to Use
+
+- Working on metrics, logs, traces, dashboards, or alerts for the homelab
+- Prometheus/Alloy config, alerting, TSDB recovery, or FreeBSD host monitoring
+- For the k3s cluster this runs on, see [`f3s-k3s`](../f3s-k3s/SKILL.md); for hosts/IPs, the [`f3s`](../f3s/SKILL.md) hub.
+
+## Reference Files
 
-- [Stack](observability/stack.md) — install Prometheus / Alloy / Loki / Tempo, alerting → Gogios, Prometheus TSDB recovery, LogQL queries, NFS storage paths
-- [FreeBSD Monitoring](observability/freebsd.md) — `node_exporter` on f-hosts, scrape config, memory & ZFS recording rules
+- [Stack](references/stack.md) — install Prometheus / Alloy / Loki / Tempo, alerting → Gogios, Prometheus TSDB recovery, LogQL queries, NFS storage paths
+- [FreeBSD Monitoring](references/freebsd.md) — `node_exporter` on f-hosts, scrape config, memory & ZFS recording rules
 
 ## Monitoring Scope
 
diff --git a/prompts/skills/f3s/references/observability/freebsd.md b/prompts/skills/f3s-observability/references/freebsd.md
index d469965..d469965 100644
--- a/prompts/skills/f3s/references/observability/freebsd.md
+++ b/prompts/skills/f3s-observability/references/freebsd.md
diff --git a/prompts/skills/f3s/references/observability/stack.md b/prompts/skills/f3s-observability/references/stack.md
index e752419..bd30874 100644
--- a/prompts/skills/f3s/references/observability/stack.md
+++ b/prompts/skills/f3s-observability/references/stack.md
@@ -68,7 +68,7 @@ kubeControllerManager:
     insecureSkipVerify: true
 ```
 
-Also requires k3s config changes on each r node — see [k3s-setup/install.md](../k3s-setup/install.md).
+Also requires k3s config changes on each r node — see [k3s-setup/install.md](../../f3s-k3s/references/install.md).
 
 ### Grafana credentials
 
diff --git a/prompts/skills/f3s-raspberry-pi/SKILL.md b/prompts/skills/f3s-raspberry-pi/SKILL.md
new file mode 100644
index 0000000..0ed3ac1
--- /dev/null
+++ b/prompts/skills/f3s-raspberry-pi/SKILL.md
@@ -0,0 +1,54 @@
+---
+name: f3s-raspberry-pi
+description: Reference skill for the four Raspberry Pi 3 nodes of the f3s homelab, pi0/pi1 run NetBSD 10.1 (aarch64) serving static f3s.buetow.org / snonux.foo via bozohttpd behind OpenBSD relayd over WireGuard; pi2/pi3 run Rocky Linux 9 with Pi-hole in Docker and LAN wildcard DNS (`*.f3s.lan.buetow.org` to 192.168.1.138). Covers doas/pkgin bootstrap, bozohttpd vhosting, npf, uptimed, content sync, and the doas-alias shutdown pitfall. Use when configuring or troubleshooting the Pi nodes, the static site, or Pi-hole/LAN DNS. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)).
+---
+
+# f3s Raspberry Pi Nodes
+
+The four Raspberry Pi 3 nodes of the f3s homelab. The master host/IP inventory
+(pi0–pi3 rows) lives in the [`f3s`](../f3s/SKILL.md) hub's Host-IP table.
+
+## When to Use
+
+- Configuring or troubleshooting pi0–pi3 (NetBSD static site pair, or Rocky Pi-hole pair)
+- The static `f3s.buetow.org` / `snonux.foo` site (bozohttpd, relayd forwarding, vhosts)
+- Pi-hole and `*.f3s.lan.buetow.org` LAN wildcard DNS
+- For the WireGuard mesh these depend on, see the [`f3s`](../f3s/SKILL.md) hub's `wireguard.md`; for DTail/dserver on the Pis, [`f3s-dtail`](../f3s-dtail/SKILL.md); for building the NetBSD dserver package, the [`pkgrepo`](../pkgrepo/SKILL.md) skill.
+
+## Node roles
+
+`pi2`/`pi3` run Rocky Linux 9.2 (Blue Onyx) aarch64 from the SIG/AltArch image (`RockyLinuxRpi_9-latest.img.xz`). `pi0` and `pi1` run **NetBSD 10.1** (evbarm-aarch64). Each Rocky Pi has:
+
+- User `paul` with passwordless sudo and SSH key auth
+- Static IP on eth0 via NetworkManager
+- Hostname `piN.lan.buetow.org`
+- Filesystem expanded with `rootfs-expand`
+- Default `rocky` user still present (password: `rockylinux`)
+- No GRUB — boots via Pi's native bootloader (`/boot/cmdline.txt`)
+- Custom RPi kernel from the `rockyrpi` repo
+
+`pi0`/`pi1` (NetBSD) differ: user `paul` in `wheel`, privilege escalation via a **real `doas`** (pkgsrc `security/doas`, `permit nopass :wheel`) — not the `alias doas=sudo` shell alias `pi2`/`pi3` carry in `/etc/profile.d/doas.sh`, which doesn't expand in the non-interactive shell an SSH command runs in and so silently breaks `wol-f3s shutdown-pis`/`shutdown-all` for the Rocky Pis (`doas poweroff` resolves to nothing) — only the NetBSD nodes actually work with that script today. Config repo home for NetBSD-specific setup: `f3s/pi-netbsd/`. Service setup details: [NetBSD Pi Setup](references/bootstrap-netbsd-pi.md).
+
+Current role split:
+
+- `pi0` and `pi1` serve static `f3s.buetow.org`/`snonux.foo` content behind OpenBSD `relayd` over WireGuard. WireGuard peers are `blowfish`, `fishfinger`, **and `rocky`** (not gateway-only to just the two frontends, despite older docs here). All rc.d services (`wireguard`, `bozohttpd`, `uptimed`, `npf`, `dserver`) and both crontabs are enabled via `rc.conf` and come back automatically on reboot.
+- `pi2` and `pi3` run **Pi-hole** in Docker (`network_mode: host`, `~/pihole` on each host). Tracked dnsmasq LAN wildcard: **`f3s/pihole/docker-pi/`** in the conf repo; details in [references/pihole-pi.md](references/pihole-pi.md).
+
+## Webserver (pi0/pi1 static site)
+
+`pi0`/`pi1` serve `f3s.buetow.org`/`snonux.foo` with **bozohttpd** (NetBSD base) behind
+the OpenBSD `relayd` frontends. Vhosting is directory-based: a vhost needs a directory
+*literally* named after the hostname (`snonux.foo/`, with `www.snonux.foo` a symlink),
+and `-X` enables directory indexing. Because `relayd` **cannot rewrite URL paths** (it
+forwards the original path intact), each domain is mapped to its docroot subdir via the
+`Host` header. Docroot `/var/www/html`; `pi1` syncs the docroot hourly from `pi0` (the
+source of truth); SSH `paul@piN.lan.buetow.org -p 22`.
+
+The full bozohttpd setup — the custom `/etc/rc.d/bozohttpd`, the `-V` fallback
+system-hostname redirect pitfall, and the self-referencing vhost symlink fix — is the
+canonical detail in [references/bootstrap-netbsd-pi.md](references/bootstrap-netbsd-pi.md#webserver--bozohttpd).
+
+## Reference Files
+
+- [NetBSD Pi Setup](references/bootstrap-netbsd-pi.md) — how services are installed on `pi0`/`pi1` (NetBSD): doas/pkgin bootstrap, WireGuard via userspace `wireguard-go` (no native `wg(4)`), bozohttpd (`-X` dir-listing, vhost symlinks), uptimed from source, npf firewall, content-sync. dserver (DTail) is installed from the custom pkgrepo — see the [`pkgrepo`](../pkgrepo/SKILL.md) skill's `dtail-package.md`.
+- [Pi-hole on Pis](references/pihole-pi.md) — **pi2/pi3** Docker Pi-hole, **`~/pihole`**, **`*.f3s.lan.buetow.org` → 192.168.1.138**, paths under **`f3s/pihole/docker-pi/`**
diff --git a/prompts/skills/f3s/references/bootstrap-netbsd-pi.md b/prompts/skills/f3s-raspberry-pi/references/bootstrap-netbsd-pi.md
index 44d4b58..44d4b58 100644
--- a/prompts/skills/f3s/references/bootstrap-netbsd-pi.md
+++ b/prompts/skills/f3s-raspberry-pi/references/bootstrap-netbsd-pi.md
diff --git a/prompts/skills/f3s/references/pihole-pi.md b/prompts/skills/f3s-raspberry-pi/references/pihole-pi.md
index 3039926..b2e6de5 100644
--- a/prompts/skills/f3s/references/pihole-pi.md
+++ b/prompts/skills/f3s-raspberry-pi/references/pihole-pi.md
@@ -38,7 +38,7 @@ Admin UI: **`http://pi2.lan.buetow.org/admin/`** (and pi3).
 
 ## Timekeeping on the RTC-less Pis
 
-pi2 and pi3 have no hardware RTC, so their clocks are stale briefly during boot until chronyd synchronizes. `uptimed` must not start in that window: both hosts use a systemd override that waits for `chronyc waitsync`, preventing invalid boot timestamps and missing active markers in goprecords. The canonical override, recovery procedure, and verification commands are in [goprecords-uptimed.md](goprecords-uptimed.md#rocky-pi-uptimed-clock-synchronization).
+pi2 and pi3 have no hardware RTC, so their clocks are stale briefly during boot until chronyd synchronizes. `uptimed` must not start in that window: both hosts use a systemd override that waits for `chronyc waitsync`, preventing invalid boot timestamps and missing active markers in goprecords. The canonical override, recovery procedure, and verification commands are in [goprecords-uptimed.md](../../f3s-workloads/references/goprecords-uptimed.md#rocky-pi-uptimed-clock-synchronization).
 
 ## Public DNS note
 
diff --git a/prompts/skills/f3s-storage/SKILL.md b/prompts/skills/f3s-storage/SKILL.md
new file mode 100644
index 0000000..82c0a5e
--- /dev/null
+++ b/prompts/skills/f3s-storage/SKILL.md
@@ -0,0 +1,41 @@
+---
+name: f3s-storage
+description: Reference skill for the f3s homelab storage layer, ZFS (`zdata`), zrepl replication, CARP storage VIP (f0/f1, `f3s-storage-ha` 192.168.1.138), NFS over stunnel, the nfs-mount-monitor watchdog, USB key material, local-path/backups, and storage troubleshooting (incl. thermal). Use when working on homelab storage, ZFS/zrepl, NFS mounts, CARP failover, or disk issues. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)).
+---
+
+# f3s Storage
+
+Persistent storage for k3s is served via **NFS over stunnel** from the FreeBSD hosts, backed by **ZFS** (`zdata` pool) with **CARP** for high availability and **zrepl** for continuous replication.
+
+Note: original plan was HAST, replaced by **zrepl** (ZFS send/receive) — more reliable, avoids the ZFS corruption during failover that HAST caused.
+
+## When to Use
+
+- Working on the homelab storage layer: ZFS pools/datasets, encryption, USB keys
+- zrepl replication (f0→f1 nfsdata, f3→f2 VM), CARP failover, NFS-over-stunnel
+- Diagnosing NFS mount problems, SUSPENDED pools, or thermal issues
+- For the physical hosts, WireGuard mesh, and host/IP inventory this depends on, see the [`f3s`](../f3s/SKILL.md) hub skill.
+
+## Reference Files
+
+- [ZFS Pools & Encryption](references/zfs.md) — `zdata` pool, physical disks, USB-stored keys mounted by `f3skeys` (not `/etc/fstab`), encrypted datasets, boot-time key loading
+- [USB Key Mounting](references/usb-keys.md) — `f3skeys`, `/usr/local/sbin/f3s-mount-keys`, and current `zfskeys_datasets` per f-host
+- [zrepl Replication](references/zrepl.md) — `f0 → f1` nfsdata, `f3 → f2` freebsd VM, sink configs, troubleshooting, DL-state recovery
+- [CARP HA VIP](references/carp.md) — VIP `192.168.1.138`, `carpcontrol.sh`, mgmt script, auto-failback, SUSPENDED-pool limitation
+- [NFS over stunnel](references/nfs.md) — NFS server, mutual-TLS stunnel, Rocky client config, `/etc/fstab`
+- [nfs-mount-monitor](references/nfs-mount-monitor.md) — systemd watchdog on r-nodes (mount/stat/write probes, fail counter, cordon-and-reboot escalation)
+- [Troubleshooting](references/troubleshooting.md) — NFS issues, ZFS pool SUSPENDED recovery, **thermal** troubleshooting (Beelink S12 Pro)
+- [Backups & Local-Path](references/backups.md) — S3 Glacier Deep Archive, when to use `local-path` instead of NFS
+
+## Storage Summary
+
+| Layer | Technology | Role |
+|-------|-----------|------|
+| Block | M.2+2.5" SSD (f0/f1) | Physical storage |
+| Filesystem | ZFS (`zdata/enc`) | Data integrity, AES-256-GCM encryption |
+| Replication | `zrepl` | Continuous ZFS replication f0→f1 (1min NFS, 10min VM) |
+| HA | CARP VIP 192.168.1.138 | Automatic failover for NFS/stunnel |
+| Network | NFS over stunnel | Encrypted shared storage, mutual TLS auth |
+| Local-path | k3s local-path provisioner | Node-local storage for SQLite/cache workloads |
+| LAN access | FreeBSD relayd on CARP VIP | TCP forwarding to k3s :80/:443 |
+| Backup | S3 Glacier Deep Archive | Off-site encrypted backup |
diff --git a/prompts/skills/f3s/references/storage/backups.md b/prompts/skills/f3s-storage/references/backups.md
index 99fd425..99fd425 100644
--- a/prompts/skills/f3s/references/storage/backups.md
+++ b/prompts/skills/f3s-storage/references/backups.md
diff --git a/prompts/skills/f3s/references/storage/carp.md b/prompts/skills/f3s-storage/references/carp.md
index 3f7b741..3f7b741 100644
--- a/prompts/skills/f3s/references/storage/carp.md
+++ b/prompts/skills/f3s-storage/references/carp.md
diff --git a/prompts/skills/f3s/references/storage/nfs-mount-monitor.md b/prompts/skills/f3s-storage/references/nfs-mount-monitor.md
index a9b71e7..a9b71e7 100644
--- a/prompts/skills/f3s/references/storage/nfs-mount-monitor.md
+++ b/prompts/skills/f3s-storage/references/nfs-mount-monitor.md
diff --git a/prompts/skills/f3s/references/storage/nfs.md b/prompts/skills/f3s-storage/references/nfs.md
index 2c21114..2c21114 100644
--- a/prompts/skills/f3s/references/storage/nfs.md
+++ b/prompts/skills/f3s-storage/references/nfs.md
diff --git a/prompts/skills/f3s/references/storage/troubleshooting.md b/prompts/skills/f3s-storage/references/troubleshooting.md
index 7baaa63..7baaa63 100644
--- a/prompts/skills/f3s/references/storage/troubleshooting.md
+++ b/prompts/skills/f3s-storage/references/troubleshooting.md
diff --git a/prompts/skills/f3s/references/storage/usb-keys.md b/prompts/skills/f3s-storage/references/usb-keys.md
index 9871daa..9871daa 100644
--- a/prompts/skills/f3s/references/storage/usb-keys.md
+++ b/prompts/skills/f3s-storage/references/usb-keys.md
diff --git a/prompts/skills/f3s/references/storage/zfs.md b/prompts/skills/f3s-storage/references/zfs.md
index b30208c..b30208c 100644
--- a/prompts/skills/f3s/references/storage/zfs.md
+++ b/prompts/skills/f3s-storage/references/zfs.md
diff --git a/prompts/skills/f3s/references/storage/zrepl.md b/prompts/skills/f3s-storage/references/zrepl.md
index dc90676..dc90676 100644
--- a/prompts/skills/f3s/references/storage/zrepl.md
+++ b/prompts/skills/f3s-storage/references/zrepl.md
diff --git a/prompts/skills/f3s-workloads/SKILL.md b/prompts/skills/f3s-workloads/SKILL.md
new file mode 100644
index 0000000..4d946c5
--- /dev/null
+++ b/prompts/skills/f3s-workloads/SKILL.md
@@ -0,0 +1,24 @@
+---
+name: f3s-workloads
+description: Reference skill for the application workloads running on the f3s homelab, Immich (photos), Garage (S3), the Player service, yChat (legacy C++ chat), and goprecords/uptimed uploads. Covers image build/push, Helm charts, ArgoCD sync, NFS PV/PVC wiring, edge domain routing, and per-app troubleshooting. Use when deploying, updating, or debugging a specific homelab application. Part of the f3s homelab skill family (hub, [`f3s`](../f3s/SKILL.md)).
+---
+
+# f3s Workloads
+
+The application workloads hosted on the f3s k3s cluster. Each app has its own
+reference with image build/push steps, Helm chart path, ArgoCD wiring, and NFS
+PV/PVC notes.
+
+## When to Use
+
+- Deploying, updating, or debugging one of the hosted homelab applications below
+- Questions about a specific app's image build/push, Helm chart, ArgoCD sync, or storage wiring
+- For the cluster these run on, see [`f3s-k3s`](../f3s-k3s/SKILL.md); for the NFS/PV storage layer, [`f3s-storage`](../f3s-storage/SKILL.md); for hosts/IPs, the [`f3s`](../f3s/SKILL.md) hub.
+
+## Reference Files
+
+- [Immich](references/immich.md) — photo server deployment, job queue stats, troubleshooting
+- [Garage](references/garage.md) — Garage cluster, edge domain routing, S3 bucket/key workflow, troubleshooting
+- [Player](references/player.md) — `player.f3s.buetow.org`, image build/push workflow, Helm chart path, ArgoCD sync, NFS PV/PVC notes
+- [yChat](references/ychat.md) — `ychat.f3s.lan.buetow.org`, legacy C++ chat server, image build/push, Helm chart + ArgoCD; the single home for f3s yChat deployment details
+- [goprecords / uptimed uploads](references/goprecords-uptimed.md) — `https://goprecords.f3s.buetow.org`, `PUT /upload`, OpenBSD Rex daily vs FreeBSD/Pi manual hourly cron/systemd, upload client, geheim tokens
diff --git a/prompts/skills/f3s/references/garage.md b/prompts/skills/f3s-workloads/references/garage.md
index 52120df..52120df 100644
--- a/prompts/skills/f3s/references/garage.md
+++ b/prompts/skills/f3s-workloads/references/garage.md
diff --git a/prompts/skills/f3s/references/goprecords-uptimed.md b/prompts/skills/f3s-workloads/references/goprecords-uptimed.md
index 01d609c..4ae4297 100644
--- a/prompts/skills/f3s/references/goprecords-uptimed.md
+++ b/prompts/skills/f3s-workloads/references/goprecords-uptimed.md
@@ -36,7 +36,7 @@ Install **`curl`** and **`uptimed`** on every client that uploads.
 | OpenBSD frontends | **fishfinger**, **blowfish** | **Rex** **`goprecords_upload`** in **`~/git/conf/frontends`**