diff options
| -rw-r--r-- | fish/conf.d/worktime.fish | 43 | ||||
| -rw-r--r-- | prompts/skills/f3s/SKILL.md | 2 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/goprecords-uptimed.md | 29 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/k3s-setup.md | 1 | ||||
| -rw-r--r-- | prompts/skills/f3s/references/k3s-setup/remote-access.md | 69 |
5 files changed, 140 insertions, 4 deletions
diff --git a/fish/conf.d/worktime.fish b/fish/conf.d/worktime.fish index 973645c..941a4a8 100644 --- a/fish/conf.d/worktime.fish +++ b/fish/conf.d/worktime.fish @@ -48,7 +48,7 @@ function worktime::supersync_sync cd - end -function worktime::darwin_uprecords +function worktime::uprecords::darwin::collect if test (uname) != Darwin return end @@ -80,10 +80,49 @@ function worktime::darwin_uprecords end end +function worktime::uprecords::darwin::import + if test (hostname) != earth + return + end + + if not test -d $WORKTIME_DIR + return + end + + set -l src_host MBDVXJ4XKH9C + set -l goprecords_host mega-m3-pro + set -l base_url https://goprecords.f3s.buetow.org + + set -l config_dir (test -n "$XDG_CONFIG_HOME"; and echo $XDG_CONFIG_HOME; or echo ~/.config) + set -l token_file $config_dir/goprecords-upload-$goprecords_host/token + if not test -r $token_file + echo "worktime::uprecords::darwin::import: cannot read $token_file (create the goprecords key for $goprecords_host)" >&2 + return 0 + end + set -l token (string trim <$token_file) + + # Map the file kind (extension) collected in the repo to the goprecords + # upload kind. + set -l records_file $WORKTIME_DIR/uprecords-$src_host.records + set -l txt_file $WORKTIME_DIR/uprecords-$src_host.txt + + for pair in records:$records_file txt:$txt_file + set -l kind (string split -m1 ':' $pair)[1] + set -l file (string split -m1 ':' $pair)[2] + if not test -f $file + continue + end + curl -fsS -X PUT --data-binary "@$file" \ + -H "Authorization: Bearer $token" \ + "$base_url/upload/$goprecords_host/$kind" + end +end + function worktime::supersync worktime::supersync_sync sync_quotes taskwarrior::invoke - worktime::darwin_uprecords + worktime::uprecords::darwin::collect + worktime::uprecords::darwin::import worktime::supersync_sync no_sync_quotes end diff --git a/prompts/skills/f3s/SKILL.md b/prompts/skills/f3s/SKILL.md index aef73d9..fef0e49 100644 --- a/prompts/skills/f3s/SKILL.md +++ b/prompts/skills/f3s/SKILL.md @@ -25,7 +25,7 @@ Detailed reference documentation is in the `references/` subfolder: - [Bootstrap Rocky bhyve VM](references/bootstrap-rocky-bhyve.md) — Runbook for creating a new plain Rocky Linux bhyve guest with unattended kickstart - [WireGuard Mesh](references/wireguard.md) — Mesh topology, IP assignments, peer configs - [Storage](references/storage.md) — index into `references/storage/`: ZFS (zdata), zrepl, CARP, NFS over stunnel, nfs-mount-monitor, troubleshooting (incl. thermal), backups & local-path -- [k3s Setup](references/k3s-setup.md) — index into `references/k3s-setup/`: install (bootstrap, kubeconfig, PVs, ArgoCD), ingress (OpenBSD/FreeBSD relayd, cert-manager), troubleshooting (etcd recovery) +- [k3s Setup](references/k3s-setup.md) — index into `references/k3s-setup/`: install (bootstrap, kubeconfig, PVs, ArgoCD), remote access off-LAN (jump via OpenBSD frontend → `root@r0.wg0` → kubectl), ingress (OpenBSD/FreeBSD relayd, cert-manager), troubleshooting (etcd recovery) - [Observability](references/observability.md) — index into `references/observability/`: stack (Prometheus/Alloy/Loki/Tempo + alerting), FreeBSD monitoring (node_exporter + recording rules) - [Immich](references/immich.md) — Photo server deployment, job queue stats, troubleshooting - [Garage](references/garage.md) — Garage cluster, edge domain routing, S3 bucket/key workflow, troubleshooting diff --git a/prompts/skills/f3s/references/goprecords-uptimed.md b/prompts/skills/f3s/references/goprecords-uptimed.md index 444f421..4ae999a 100644 --- a/prompts/skills/f3s/references/goprecords-uptimed.md +++ b/prompts/skills/f3s/references/goprecords-uptimed.md @@ -36,7 +36,8 @@ Install **`curl`** and **`uptimed`** on every client that uploads. | OpenBSD frontends | **fishfinger**, **blowfish** | **Rex** **`goprecords_upload`** in **`~/git/conf/frontends`**; **`/etc/daily.local`** runs **`/usr/local/bin/goprecords-upload.sh`** once per **day** | Tokens in **geheim** **`secrets/etc/goprecords/<host>.token`**; template **`scripts/goprecords-upload.sh.tpl`** | | FreeBSD (Beelinks) | **f0**–**f3** (LAN **`192.168.1.130`–`133`**) | Manual **hourly** **root** **`cron`** calling **`goprecords-upload-client.sh`** with **`GOPRECORDS_HOST=f0`** … **`f3`** | **`/var/db/uptimed/records`**; SSH: **`fN.lan.buetow.org`** or **`192.168.1.(130+N)`** for **fN**, **`-p 22`** | | Raspberry Pi (Rocky) | **pi0**–**pi3** | Manual **hourly** **systemd** **timer** (see README) | **`/var/spool/uptimed/records`**; SSH: **`piN.lan.buetow.org`**, **`-p 22`** | -| Fedora laptop | **earth** | Optional **user** **systemd** timer or manual | Example: **`~/.config/goprecords-upload-earth/`** | +| Fedora laptop | **earth** | **user** **systemd** **`oneshot` + hourly timer** `goprecords-upload-earth.{service,timer}` | Service sets **`Environment=GOPRECORDS_HOST=earth`** and runs **`~/.local/bin/goprecords-upload-earth.sh`**; token **`~/.config/goprecords-upload-earth/token`** | +| Mac (uptimed) → published by earth | **mega-m3-pro** (raw host `MBDVXJ4XKH9C`) | Mac drops records into the **worktime** git repo; **earth** pushes them via a **second `ExecStart`** in `goprecords-upload-earth.service` | See [Mac / mega-m3-pro via earth](#mac--mega-m3-pro-via-earth) below | ## OpenBSD frontends (Rex) @@ -59,6 +60,32 @@ The canonical unified script is **`scripts/goprecords-upload-client.sh`** (also Copy to **`/usr/local/bin/`** (system) or **`~/.local/bin/`** (user), set **`GOPRECORDS_HOST`** per machine (**cron** **`env`** or **`systemd`** **`Environment`**/**`EnvironmentFile`**). Full snippets: **goprecords** **`README.md`**. +> **earth gotcha:** `goprecords-upload-earth.sh` is a *copy of the generic* `goprecords-upload-client.sh`, so it aborts with `set GOPRECORDS_HOST` unless the var is provided. The service therefore **must** carry `Environment=GOPRECORDS_HOST=earth`. (A missing env var silently broke earth uploads for ~a month — symptom: stale timestamp on the report, service `status=1/FAILURE` with `set GOPRECORDS_HOST` in `journalctl --user -u goprecords-upload-earth`.) + +## Mac / mega-m3-pro via earth + +The Mac (Apple Silicon, **`Darwin`**) is **not** a direct upload client. Instead: + +1. On the Mac, a fish function **`worktime::uprecords::darwin::collect`** copies the local **uptimed** records into the **worktime** git repo as + **`uprecords-MBDVXJ4XKH9C.records`** and **`uprecords-MBDVXJ4XKH9C.txt`**, and they get synced via `git` (part of **`worktime::supersync`**). +2. On **earth** (the only host that publishes), the fish function **`worktime::uprecords::darwin::import`** reads those repo files and **`PUT`**s them to goprecords, **re-labelling** the raw host `MBDVXJ4XKH9C` → **`mega-m3-pro`**: + - `PUT /upload/mega-m3-pro/records` and `PUT /upload/mega-m3-pro/txt` + - token at **`~/.config/goprecords-upload-mega-m3-pro/token`** (`0600`). + - Both functions live in **`dotfiles/fish/conf.d/worktime.fish`**. +3. Automation: `goprecords-upload-earth.service` has a **second `ExecStart`** that runs the import hourly alongside earth's own upload: + ```ini + ExecStart=%h/.local/bin/goprecords-upload-earth.sh + ExecStart=/usr/local/sbin/fish -c worktime::uprecords::darwin::import + ``` + The import is a **no-op off earth** (guards on `hostname = earth`) and exits cleanly with a warning if the `mega-m3-pro` token is absent, so it never fails the service. + +**Token note:** each token is bound to its host name server-side — the **earth** token returns **403** for `mega-m3-pro`. Issue a dedicated key (see *Daemon and keys* above): +```sh +kubectl exec -n services deployment/goprecords -- \ + goprecords --create-client-key mega-m3-pro -stats-dir=/data/stats +``` +Re-issuing **replaces** any previous `mega-m3-pro` token (the Mac used to upload directly until it switched to the repo route). When roaming, reach the cluster via the OpenBSD frontend jump (see [k3s remote access](k3s-setup/remote-access.md)), then store the printed token in `~/.config/goprecords-upload-mega-m3-pro/token` on earth. + ## Related conf repo paths - Kubernetes Helm: **`conf/f3s/goprecords/`** (image, PVC, ingress **`goprecords.f3s.buetow.org`**) diff --git a/prompts/skills/f3s/references/k3s-setup.md b/prompts/skills/f3s/references/k3s-setup.md index 0fc7605..952a4bf 100644 --- a/prompts/skills/f3s/references/k3s-setup.md +++ b/prompts/skills/f3s/references/k3s-setup.md @@ -7,5 +7,6 @@ over WireGuard. ## Sub-references - [Install](k3s-setup/install.md) — bootstrap, kubeconfig, etcd/controller-manager metrics, built-in components, NFS PV pattern, ArgoCD, node IP summary, useful commands +- [Remote access (off-LAN)](k3s-setup/remote-access.md) — reaching the cluster while roaming: jump via OpenBSD frontend (`ssh -A rex@fishfinger.buetow.org` → `ssh root@r0.wg0` → `kubectl`), one-shot commands, and SSH port-forward tunnel - [Ingress](k3s-setup/ingress.md) — OpenBSD `relayd` (internet) and FreeBSD `relayd` on CARP VIP (LAN), cert-manager wildcard, ingress pattern - [Troubleshooting](k3s-setup/troubleshooting.md) — etcd Raft log corruption recovery; cluster-wide NFS outage pointer diff --git a/prompts/skills/f3s/references/k3s-setup/remote-access.md b/prompts/skills/f3s/references/k3s-setup/remote-access.md new file mode 100644 index 0000000..5da8c84 --- /dev/null +++ b/prompts/skills/f3s/references/k3s-setup/remote-access.md @@ -0,0 +1,69 @@ +# Accessing the k3s cluster from outside the LAN (roaming) + +The kubeconfig on the laptop points at the **LAN** API endpoint +(`https://r0.lan.buetow.org:6443` → `192.168.1.120`). When **earth** (or any +client) is **not on the f3s LAN**, that address is unreachable and `kubectl` +just times out: + +``` +Unable to connect to the server: dial tcp 192.168.1.120:6443: i/o timeout +``` + +The WireGuard mesh IPs (`192.168.2.120` etc.) are also not directly reachable +from a roaming laptop, because the laptop's `wg1` interface does not peer +straight to the r-VMs. + +## Working method: jump through an OpenBSD frontend + +The OpenBSD internet gateways **fishfinger** and **blowfish** are reachable +from the public internet *and* sit on the WireGuard mesh, so they can reach the +r-VMs over `wg0`. Use one as a jump host, then run `kubectl` as `root` on the +r-VM itself. + +Interactive: + +```sh +ssh -A rex@fishfinger.buetow.org # -A forwards your agent for the next hop +ssh root@r0.wg0 # from fishfinger, over the WireGuard mesh +kubectl get nodes # root on r0 has /etc/rancher/k3s/k3s.yaml +``` + +One-shot (non-interactive) from the laptop — handy for scripts/automation: + +```sh +ssh -A rex@fishfinger.buetow.org "ssh root@r0.wg0 'kubectl get nodes'" +``` + +Notes: + +- `ssh -A` (agent forwarding) is required so the `rex@fishfinger` → `root@r0.wg0` + hop can authenticate. Make sure the right key is loaded (`ssh-add -l`). +- Use **`r0.wg0`** (or `r1.wg0` / `r2.wg0` if r0 is down) — that is the r-VM's + WireGuard hostname as seen *from the frontend*, not `r0.lan…`. +- `blowfish.buetow.org` works the same way as `fishfinger.buetow.org` if one + frontend is unavailable. +- On a fresh laptop, accept host keys with + `-o StrictHostKeyChecking=accept-new` on both hops. +- An OpenSSH **post-quantum key exchange warning** from the `root@r0.wg0` hop is + harmless and can be ignored. + +## Example: issue a goprecords client key while roaming + +```sh +ssh -A rex@fishfinger.buetow.org \ + "ssh root@r0.wg0 'kubectl exec -n services deployment/goprecords -- \ + goprecords --create-client-key mega-m3-pro -stats-dir=/data/stats'" +``` + +## Alternative: serve the API over an SSH tunnel + +If you need a real local `kubectl` (not just one-off commands), forward the API +port through the frontend and point a kubeconfig at `127.0.0.1`: + +```sh +ssh -A -L 6443:r0.wg0:6443 rex@fishfinger.buetow.org +# then, in another shell, with a kubeconfig whose server is +# https://127.0.0.1:6443 (and tls-server-name r0.wg0.wan.buetow.org, since the +# API cert has that SAN — see install.md --tls-san): +kubectl --tls-server-name r0.wg0.wan.buetow.org get nodes +``` |