f3s: split storage/k3s-setup/observability references, correct thermal attribution

storage.md (942), k3s-setup.md (342), observability.md (273) split into
per-topic sub-files under references/<topic>/ with short index files at
the original paths.

Drops the SSD TRIM Configuration section added in 8d94f79 and reframes
the Thermal Troubleshooting cascade: the 2026-05-16 f0 incident was
thermal alone, not the multi-cause cascade (autotrim, zrepl interval,
encryption) the previous commit implied. Mitigations applied during
diagnosis are not what fixed it.

Other yesterday additions (zrepl DL-state recovery, CARP-when-ZFS-
suspended, ZFS SUSPENDED runbook, nfs-mount-monitor improvements) are
kept and routed to their respective sub-files.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

2 files changed, 269 insertions, 0 deletions

diff --git a/prompts/skills/f3s/references/observability/freebsd.md b/prompts/skills/f3s/references/observability/freebsd.md
new file mode 100644
index 0000000..3005355
--- /dev/null
+++ b/prompts/skills/f3s/references/observability/freebsd.md
@@ -0,0 +1,111 @@
+# Monitoring FreeBSD Hosts (f0, f1, f2)
+
+Scraping the FreeBSD bhyve hosts from in-cluster Prometheus. Includes the
+node_exporter setup, the additional scrape config, and the recording rules
+that bridge FreeBSD metric names into the Linux-style names Grafana
+dashboards expect.
+
+## Install node_exporter on FreeBSD
+
+```sh
+# On each FreeBSD host
+doas pkg install -y node_exporter
+doas sysrc node_exporter_enable=YES
+# Bind to WireGuard interface (f0=192.168.2.130, f1=192.168.2.131, f2=192.168.2.132)
+doas sysrc node_exporter_args='--web.listen-address=192.168.2.130:9100'
+doas service node_exporter start
+```
+
+## Prometheus scrape config for FreeBSD
+
+`additional-scrape-configs.yaml`:
+
+```yaml
+- job_name: 'node-exporter'
+  static_configs:
+    - targets:
+      - '192.168.2.130:9100'  # f0 via WireGuard
+      - '192.168.2.131:9100'  # f1 via WireGuard
+      - '192.168.2.132:9100'  # f2 via WireGuard
+      labels:
+        os: freebsd
+```
+
+```sh
+kubectl create secret generic additional-scrape-configs \
+    --from-file=additional-scrape-configs.yaml -n monitoring
+```
+
+Add to `persistence-values.yaml`:
+
+```yaml
+prometheus:
+  prometheusSpec:
+    additionalScrapeConfigsSecret:
+      enabled: true
+      name: additional-scrape-configs
+      key: additional-scrape-configs.yaml
+```
+
+## FreeBSD memory compatibility rules
+
+FreeBSD uses different metric names than Linux. PrometheusRule to create Linux-compatible metrics:
+
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: freebsd-memory-rules
+  namespace: monitoring
+  labels:
+    release: prometheus
+spec:
+  groups:
+    - name: freebsd-memory
+      rules:
+        - record: node_memory_MemTotal_bytes
+          expr: node_memory_size_bytes{os="freebsd"}
+        - record: node_memory_MemAvailable_bytes
+          expr: |
+            node_memory_free_bytes{os="freebsd"}
+              + node_memory_inactive_bytes{os="freebsd"}
+              + node_memory_cache_bytes{os="freebsd"}
+        - record: node_memory_MemFree_bytes
+          expr: node_memory_free_bytes{os="freebsd"}
+        - record: node_memory_Buffers_bytes
+          expr: node_memory_buffer_bytes{os="freebsd"}
+        - record: node_memory_Cached_bytes
+          expr: node_memory_cache_bytes{os="freebsd"}
+```
+
+Note: Disk I/O metrics (`node_disk_*`) are not available on FreeBSD — use ZFS-specific dashboards instead.
+
+## ZFS monitoring recording rules
+
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: freebsd-zfs-rules
+  namespace: monitoring
+  labels:
+    release: prometheus
+spec:
+  groups:
+    - name: freebsd-zfs-arc
+      interval: 30s
+      rules:
+        - record: node_zfs_arc_hit_rate_percent
+          expr: |
+            100 * (
+              rate(node_zfs_arcstats_hits_total{os="freebsd"}[5m]) /
+              (rate(node_zfs_arcstats_hits_total{os="freebsd"}[5m]) +
+               rate(node_zfs_arcstats_misses_total{os="freebsd"}[5m]))
+            )
+        - record: node_zfs_arc_memory_usage_percent
+          expr: |
+            100 * (
+              node_zfs_arcstats_size_bytes{os="freebsd"} /
+              node_zfs_arcstats_c_max_bytes{os="freebsd"}
+            )
+```
diff --git a/prompts/skills/f3s/references/observability/stack.md b/prompts/skills/f3s/references/observability/stack.md
new file mode 100644
index 0000000..e752419
--- /dev/null
+++ b/prompts/skills/f3s/references/observability/stack.md
@@ -0,0 +1,158 @@
+# Observability Stack (k3s side)
+
+Install and operation of the in-cluster observability components: Prometheus,
+Alloy, Loki, Tempo, Alertmanager → Gogios.
+
+## Deployment
+
+All components deployed via **ArgoCD** (GitOps). Manifests:
+```
+https://codeberg.org/snonux/conf/src/branch/master/f3s
+argocd-apps/monitoring/
+```
+
+Deployment tool: `just` (Justfile in each component directory).
+
+### Namespaces
+
+```sh
+kubectl create namespace monitoring
+```
+
+### Disabled component manifests
+
+These files exist in the repo but are renamed `.disabled` so ArgoCD ignores them:
+```
+f3s/argocd-apps/monitoring/loki.yaml.disabled
+f3s/argocd-apps/monitoring/tempo.yaml.disabled
+f3s/argocd-apps/monitoring/grafana-ingress.yaml.disabled
+```
+
+To re-enable, rename back to `.yaml` and ensure Grafana is using a non-NFS PVC (local-path).
+
+## Installing Prometheus
+
+Uses `kube-prometheus-stack` Helm chart with **Grafana subchart disabled** (`grafana.enabled: false`):
+
+```sh
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+
+# Create NFS storage directory first
+mkdir -p /data/nfs/k3svolumes/prometheus/data
+
+cd conf/f3s/prometheus && just install
+```
+
+### Enable etcd and controller-manager scraping
+
+Add to `persistence-values.yaml`:
+
+```yaml
+kubeEtcd:
+  enabled: true
+  endpoints: [192.168.2.120, 192.168.2.121, 192.168.2.122]
+  service:
+    port: 2381
+    targetPort: 2381
+
+kubeControllerManager:
+  enabled: true
+  endpoints: [192.168.2.120, 192.168.2.121, 192.168.2.122]
+  service:
+    port: 10257
+    targetPort: 10257
+  serviceMonitor:
+    enabled: true
+    https: true
+    insecureSkipVerify: true
+```
+
+Also requires k3s config changes on each r node — see [k3s-setup/install.md](../k3s-setup/install.md).
+
+### Grafana credentials
+
+Default: `admin` / `prom-operator` — change immediately after first login.
+
+Grafana accessible at `grafana.f3s.foo.zone` via Traefik ingress.
+
+## Installing Alloy (minimal)
+
+Alloy is installed as part of the Loki Helm chart but runs with a minimal config (no log shipping):
+
+```sh
+cd conf/f3s/loki && just install
+# installs alloy only (loki itself is disabled via loki.yaml.disabled)
+```
+
+### Current Alloy config (`alloy-values.yaml`)
+
+Minimal — only emits Alloy's own operational logs:
+
+```
+logging {
+  level = "info"
+}
+```
+
+To re-enable log shipping (once Loki is running again), restore the full `discovery.kubernetes` + `loki.source.kubernetes` + `loki.write` pipeline.
+
+## Installing Loki (disabled)
+
+```sh
+mkdir -p /data/nfs/k3svolumes/loki/data
+# Rename loki.yaml.disabled → loki.yaml first, then:
+cd conf/f3s/loki && just install
+```
+
+Loki URL (internal): `http://loki.monitoring.svc.cluster.local:3100`
+
+## Installing Tempo (disabled)
+
+```sh
+mkdir -p /data/nfs/k3svolumes/tempo/data
+# Rename tempo.yaml.disabled → tempo.yaml first, then:
+cd conf/f3s/tempo && just install
+```
+
+## Alerting
+
+Prometheus → Alertmanager → **Gogios** (custom lightweight monitoring tool running on OpenBSD gateway `blowfish`/`fishfinger`).
+
+Gogios scrapes Alertmanager at regular intervals and sends email notifications. Reaches Alertmanager via WireGuard mesh.
+
+## Prometheus TSDB Recovery
+
+If Prometheus fails to start with `opening storage failed: get segment range: segments are not sequential`, WAL segments are corrupt (can happen after a cluster blip leaving zero-byte WAL files).
+
+Full TSDB wipe (loses all historical data — confirm first):
+
+```sh
+# On the NFS server (f0 or CARP MASTER)
+rm -rf /data/nfs/k3svolumes/prometheus/data/prometheus-db/
+mkdir -p /data/nfs/k3svolumes/prometheus/data/prometheus-db
+chown 1000:1000 /data/nfs/k3svolumes/prometheus/data/prometheus-db
+# Prometheus will recreate the TSDB on next start
+```
+
+## Useful LogQL Queries
+
+```
+# All logs from services namespace
+{namespace="services"}
+
+# Filter by log content
+{namespace="services"} |= "error"
+
+# Parse JSON logs
+{namespace="services"} | json | level="error"
+```
+
+## NFS Storage Paths
+
+```
+/data/nfs/k3svolumes/prometheus/data   # active
+/data/nfs/k3svolumes/grafana/data      # exists but unused (grafana disabled)
+/data/nfs/k3svolumes/loki/data         # exists but unused (loki disabled)
+/data/nfs/k3svolumes/tempo/data        # exists but unused (tempo disabled)
+```