diff options
Diffstat (limited to 'prompts/skills/f3s/references/observability/stack.md')
| -rw-r--r-- | prompts/skills/f3s/references/observability/stack.md | 158 |
1 files changed, 0 insertions, 158 deletions
diff --git a/prompts/skills/f3s/references/observability/stack.md b/prompts/skills/f3s/references/observability/stack.md deleted file mode 100644 index e752419..0000000 --- a/prompts/skills/f3s/references/observability/stack.md +++ /dev/null @@ -1,158 +0,0 @@ -# Observability Stack (k3s side) - -Install and operation of the in-cluster observability components: Prometheus, -Alloy, Loki, Tempo, Alertmanager → Gogios. - -## Deployment - -All components deployed via **ArgoCD** (GitOps). Manifests: -``` -https://codeberg.org/snonux/conf/src/branch/master/f3s -argocd-apps/monitoring/ -``` - -Deployment tool: `just` (Justfile in each component directory). - -### Namespaces - -```sh -kubectl create namespace monitoring -``` - -### Disabled component manifests - -These files exist in the repo but are renamed `.disabled` so ArgoCD ignores them: -``` -f3s/argocd-apps/monitoring/loki.yaml.disabled -f3s/argocd-apps/monitoring/tempo.yaml.disabled -f3s/argocd-apps/monitoring/grafana-ingress.yaml.disabled -``` - -To re-enable, rename back to `.yaml` and ensure Grafana is using a non-NFS PVC (local-path). - -## Installing Prometheus - -Uses `kube-prometheus-stack` Helm chart with **Grafana subchart disabled** (`grafana.enabled: false`): - -```sh -helm repo add prometheus-community https://prometheus-community.github.io/helm-charts -helm repo update - -# Create NFS storage directory first -mkdir -p /data/nfs/k3svolumes/prometheus/data - -cd conf/f3s/prometheus && just install -``` - -### Enable etcd and controller-manager scraping - -Add to `persistence-values.yaml`: - -```yaml -kubeEtcd: - enabled: true - endpoints: [192.168.2.120, 192.168.2.121, 192.168.2.122] - service: - port: 2381 - targetPort: 2381 - -kubeControllerManager: - enabled: true - endpoints: [192.168.2.120, 192.168.2.121, 192.168.2.122] - service: - port: 10257 - targetPort: 10257 - serviceMonitor: - enabled: true - https: true - insecureSkipVerify: true -``` - -Also requires k3s config changes on each r node — see [k3s-setup/install.md](../k3s-setup/install.md). - -### Grafana credentials - -Default: `admin` / `prom-operator` — change immediately after first login. - -Grafana accessible at `grafana.f3s.foo.zone` via Traefik ingress. - -## Installing Alloy (minimal) - -Alloy is installed as part of the Loki Helm chart but runs with a minimal config (no log shipping): - -```sh -cd conf/f3s/loki && just install -# installs alloy only (loki itself is disabled via loki.yaml.disabled) -``` - -### Current Alloy config (`alloy-values.yaml`) - -Minimal — only emits Alloy's own operational logs: - -``` -logging { - level = "info" -} -``` - -To re-enable log shipping (once Loki is running again), restore the full `discovery.kubernetes` + `loki.source.kubernetes` + `loki.write` pipeline. - -## Installing Loki (disabled) - -```sh -mkdir -p /data/nfs/k3svolumes/loki/data -# Rename loki.yaml.disabled → loki.yaml first, then: -cd conf/f3s/loki && just install -``` - -Loki URL (internal): `http://loki.monitoring.svc.cluster.local:3100` - -## Installing Tempo (disabled) - -```sh -mkdir -p /data/nfs/k3svolumes/tempo/data -# Rename tempo.yaml.disabled → tempo.yaml first, then: -cd conf/f3s/tempo && just install -``` - -## Alerting - -Prometheus → Alertmanager → **Gogios** (custom lightweight monitoring tool running on OpenBSD gateway `blowfish`/`fishfinger`). - -Gogios scrapes Alertmanager at regular intervals and sends email notifications. Reaches Alertmanager via WireGuard mesh. - -## Prometheus TSDB Recovery - -If Prometheus fails to start with `opening storage failed: get segment range: segments are not sequential`, WAL segments are corrupt (can happen after a cluster blip leaving zero-byte WAL files). - -Full TSDB wipe (loses all historical data — confirm first): - -```sh -# On the NFS server (f0 or CARP MASTER) -rm -rf /data/nfs/k3svolumes/prometheus/data/prometheus-db/ -mkdir -p /data/nfs/k3svolumes/prometheus/data/prometheus-db -chown 1000:1000 /data/nfs/k3svolumes/prometheus/data/prometheus-db -# Prometheus will recreate the TSDB on next start -``` - -## Useful LogQL Queries - -``` -# All logs from services namespace -{namespace="services"} - -# Filter by log content -{namespace="services"} |= "error" - -# Parse JSON logs -{namespace="services"} | json | level="error" -``` - -## NFS Storage Paths - -``` -/data/nfs/k3svolumes/prometheus/data # active -/data/nfs/k3svolumes/grafana/data # exists but unused (grafana disabled) -/data/nfs/k3svolumes/loki/data # exists but unused (loki disabled) -/data/nfs/k3svolumes/tempo/data # exists but unused (tempo disabled) -``` |