From 76307f4826efbf995b0be72392282fccd62daf00 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Mon, 22 Jun 2026 08:49:48 +0300 Subject: update prompts --- prompts/skills/f3s/references/k3s-setup.md | 2 +- .../f3s/references/k3s-setup/remote-access.md | 47 ++++++++++++++++++++-- prompts/skills/f3s/references/remote-access.md | 20 +++++++++ prompts/skills/f3s/references/wireguard.md | 36 ++++++++++++++++- 4 files changed, 99 insertions(+), 6 deletions(-) diff --git a/prompts/skills/f3s/references/k3s-setup.md b/prompts/skills/f3s/references/k3s-setup.md index 952a4bf..3b94b88 100644 --- a/prompts/skills/f3s/references/k3s-setup.md +++ b/prompts/skills/f3s/references/k3s-setup.md @@ -7,6 +7,6 @@ over WireGuard. ## Sub-references - [Install](k3s-setup/install.md) — bootstrap, kubeconfig, etcd/controller-manager metrics, built-in components, NFS PV pattern, ArgoCD, node IP summary, useful commands -- [Remote access (off-LAN)](k3s-setup/remote-access.md) — reaching the cluster while roaming: jump via OpenBSD frontend (`ssh -A rex@fishfinger.buetow.org` → `ssh root@r0.wg0` → `kubectl`), one-shot commands, and SSH port-forward tunnel +- [Remote access (off-LAN)](k3s-setup/remote-access.md) — reaching the cluster while roaming: **preferred** dedicated `wg0` kubectl context talking directly to `r0.wg0.wan.buetow.org:6443` over WireGuard (switch with `kubectl config use-context wg0`); fallback jump via OpenBSD frontend (`ssh -A rex@fishfinger.buetow.org` → `ssh root@r0.wg0` → `kubectl`), one-shot commands, and SSH port-forward tunnel - [Ingress](k3s-setup/ingress.md) — OpenBSD `relayd` (internet) and FreeBSD `relayd` on CARP VIP (LAN), cert-manager wildcard, ingress pattern - [Troubleshooting](k3s-setup/troubleshooting.md) — etcd Raft log corruption recovery; cluster-wide NFS outage pointer diff --git a/prompts/skills/f3s/references/k3s-setup/remote-access.md b/prompts/skills/f3s/references/k3s-setup/remote-access.md index 5da8c84..9837870 100644 --- a/prompts/skills/f3s/references/k3s-setup/remote-access.md +++ b/prompts/skills/f3s/references/k3s-setup/remote-access.md @@ -9,11 +9,50 @@ just times out: Unable to connect to the server: dial tcp 192.168.1.120:6443: i/o timeout ``` -The WireGuard mesh IPs (`192.168.2.120` etc.) are also not directly reachable -from a roaming laptop, because the laptop's `wg1` interface does not peer -straight to the r-VMs. +## Preferred method: a dedicated `wg0` kubectl context (direct over WireGuard) + +From **earth** the WireGuard mesh IPs **are** directly reachable +(`192.168.2.120` = `r0.wg0.wan.buetow.org`), so `kubectl` can talk straight to +the API server over `wg0` — no jump host needed. The k3s API cert already has +the `r0.wg0.wan.buetow.org` / `r1.wg0…` / `r2.wg0…` SANs (see install.md +`--tls-san`), so TLS verifies cleanly with no overrides. + +Add a second cluster + context to `~/.kube/config` alongside the LAN `default` +one, reusing the same `certificate-authority-data` and `default` user: + +```yaml +clusters: +- cluster: + certificate-authority-data: + server: https://r0.wg0.wan.buetow.org:6443 # 192.168.2.120 + name: wg0 +contexts: +- context: + cluster: wg0 + namespace: services + user: default + name: wg0 +``` + +Then switch by location: + +```sh +kubectl config use-context default # on the f3s LAN (home) +kubectl config use-context wg0 # on the road (WireGuard) +kubectl --context=wg0 get nodes # one-off without switching +``` + +Notes: + +- `default` → `https://r0.lan.buetow.org:6443` (LAN, `192.168.1.120`); `wg0` → + `https://r0.wg0.wan.buetow.org:6443` (mesh, `192.168.2.120`). +- For failover, point a `wg0` variant at `r1.wg0` / `r2.wg0` (all three are SANs + on the cert) if r0 is down. +- This requires the laptop's WireGuard to actually route to the r-VM mesh IPs. + If a particular client's `wg` does **not** peer to the r-VMs, fall back to the + OpenBSD-frontend jump method below. -## Working method: jump through an OpenBSD frontend +## Fallback method: jump through an OpenBSD frontend The OpenBSD internet gateways **fishfinger** and **blowfish** are reachable from the public internet *and* sit on the WireGuard mesh, so they can reach the diff --git a/prompts/skills/f3s/references/remote-access.md b/prompts/skills/f3s/references/remote-access.md index c507671..984d5ce 100644 --- a/prompts/skills/f3s/references/remote-access.md +++ b/prompts/skills/f3s/references/remote-access.md @@ -5,6 +5,26 @@ unreachable directly. The f-hosts (`f0`–`f3`) and VMs have LAN IPs (`192.168.1.x`) and WireGuard IPs (`192.168.2.x`), but a roaming laptop's `wg0` only peers to the OpenBSD gateways — not directly to the homelab mesh. +## Direct SSH from `earth` (when on the VPN) + +`earth` has been given a return route to the mesh: `192.168.2.200/32, fd42:beef:cafe:2::200/128` +was added to the **fishfinger** peer's `AllowedIPs` on `f0`, `f1`, `f2`, `r0`, `r1`, +`r2`, and `rocky`. So when earth's `wg0` is up, you can SSH directly — no ProxyJump +needed: + +```sh +ssh paul@f0.wg0 # also f1.wg0, f2.wg0 +ssh root@r0.wg0 # also r1.wg0, r2.wg0 +ssh root@rocky.wg0 +``` + +This is earth-specific (other roaming clients like `pixel7pro` still need ProxyJump), +returns go via fishfinger only, and it is currently a **manual, non-durable** change +(reverted by a `wireguardmeshgenerator` regen; durability tracked as the generator +`+reachableRoaming` task). See `wireguard.md` → "Direct SSH from a roaming client +to mesh hosts" for details and constraints. The ProxyJump method below remains the +general fallback and the way to reach hosts not yet patched (e.g. `pi0`/`pi1`, `f3`). + ## Working method: jump through an OpenBSD gateway **fishfinger** and **blowfish** are reachable from the public internet *and* diff --git a/prompts/skills/f3s/references/wireguard.md b/prompts/skills/f3s/references/wireguard.md index 2246578..126b86d 100644 --- a/prompts/skills/f3s/references/wireguard.md +++ b/prompts/skills/f3s/references/wireguard.md @@ -171,7 +171,23 @@ Endpoint = 46.23.94.99:56709 PersistentKeepalive = 25 ``` -Roaming clients route all traffic (`0.0.0.0/0`) through gateways, only connect to blowfish/fishfinger, and cannot be directly reached by LAN hosts. +Roaming clients route all traffic (`0.0.0.0/0`) through gateways and only **peer** to blowfish/fishfinger (they are not full-mesh members). By default they cannot be directly reached by mesh hosts, because no mesh host carries the roaming client's wg0 IP in any peer's `AllowedIPs` — so the mesh host has no return route and sends replies out its LAN interface, where they are lost. + +### Direct SSH from a roaming client to mesh hosts (earth enhancement) + +`earth` is an exception: its wg0 IP (`192.168.2.200/32`, `fd42:beef:cafe:2::200/128`) has been added to the **fishfinger** peer's `AllowedIPs` on `f0`, `f1`, `f2`, `r0`, `r1`, `r2`, and `rocky`, so those hosts route `192.168.2.200` back through `wg0` to fishfinger, which forwards to earth. This lets you SSH directly from earth over the VPN with no ProxyJump: + +```sh +ssh paul@f0.wg0 # also f1.wg0, f2.wg0 +ssh root@r0.wg0 # also r1.wg0, r2.wg0 +ssh root@rocky.wg0 +``` + +Constraints / notes: +- earth still only **peers** to the gateways — reachability is via gateway forwarding plus a return route on the mesh side, not a direct peer relationship. +- Returns go via **fishfinger only**, not blowfish. earth's `blowfish` peer ends up with `allowed ips: (none)` in the running config because both gateway peers are configured with `0.0.0.0/0, ::/0` and wg-quick can only install one default route (see the dual-`0.0.0.0/0` troubleshooting note below). Returns via blowfish would be dropped by earth. +- `pi0`/`pi1` are not yet reachable directly from earth; they need the same `.200` addition to their fishfinger peer if desired. +- This is currently a **manual, non-durable** change — it is reverted by any `wireguardmeshgenerator` regen because earth is in every infra host's `exclude_peers`. The durable version is tracked as generator `ask` tasks (`+reachableRoaming`); see the `wireguardmeshgenerator` project task list. ## /etc/hosts Entries for WireGuard @@ -249,6 +265,23 @@ ipv4_with_mask = hosts[myself]['os'] == 'FreeBSD' ? "#{ipv4}/32" : ipv4 Note: `reload` only reconfigures peers/PSKs — it does not change the running interface address. A `restart` is needed to pick up the address change if the interface is already running. +## Troubleshooting: Regenerated Keypair Breaks the Tunnel + +If a host's wg0 PrivateKey (and PSKs) are regenerated out-of-band (e.g. OS reinstall) but the peer configs on the other side are not updated, handshakes silently fail: `wg show` shows `0 B received` on the client and `endpoint: (none), rx=0` for that peer on the gateway, even though packets leave the client (confirmed with `tcpdump` on the wifi interface). WireGuard drops initiations it cannot authenticate and emits nothing, so the symptom is one-way traffic with no replies. + +All three must match on both sides: +- the client's **PublicKey** as configured in the gateway's peer block, +- the gateway's **PublicKey** as configured in the client's peer block (must equal `wg show wg0 public-key` on the gateway), +- the per-pair **PresharedKey** (must be identical on both sides). + +When fixing this by hand, also update the `wireguardmeshgenerator` `keys/` directory (`keys//priv.key`, `pub.key`, `keys/psk/.key`) so a regen reproduces the live configs — otherwise the next `--generate`/`--install` reverts the fix. This is tracked as the generator `+credentials` task. + +## Troubleshooting: Dual `0.0.0.0/0` on Roaming Clients + +A roaming client config that gives `AllowedIPs = 0.0.0.0/0, ::/0` to **both** gateway peers (as the generator currently does for `gateway: true`) is only partially functional: wg-quick can install only one default route, so the second peer silently ends up with `allowed ips: (none)` in the running config and is **not** a real failover. On `earth`, `sudo wg show` shows fishfinger with `0.0.0.0/0, ::/0` and blowfish with `(none)`. + +Consequence: all return traffic to earth must go via fishfinger (the peer earth actually accepts traffic from). This is why the direct-SSH return route is added to the fishfinger peer only. A proper fix (single primary gateway with failover, or `Table = off` with policy routing) is tracked as the generator `+roamingFailover` task. + ## Traffic Flows | Flow | Purpose | @@ -260,3 +293,4 @@ Note: `reload` only reconfigures peers/PSKs — it does not change the running i | rN ↔ rM | k3s intra-cluster traffic | | fN ↔ fM | zrepl storage replication | | earth/pixel7pro ↔ gateways | Remote access (all traffic routed through VPN) | +| earth ↔ fN/rN/rocky (via fishfinger) | Direct SSH from the VPN to mesh hosts (earth's IP added to the fishfinger peer AllowedIPs on those hosts; no ProxyJump needed) | -- cgit v1.2.3