1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
|
# Observability Stack (k3s side)
Install and operation of the in-cluster observability components: Prometheus,
Alloy, Loki, Tempo, Alertmanager → Gogios.
## Deployment
All components deployed via **ArgoCD** (GitOps). Manifests:
```
https://codeberg.org/snonux/conf/src/branch/master/f3s
argocd-apps/monitoring/
```
Deployment tool: `just` (Justfile in each component directory).
### Namespaces
```sh
kubectl create namespace monitoring
```
### Disabled component manifests
These files exist in the repo but are renamed `.disabled` so ArgoCD ignores them:
```
f3s/argocd-apps/monitoring/loki.yaml.disabled
f3s/argocd-apps/monitoring/tempo.yaml.disabled
f3s/argocd-apps/monitoring/grafana-ingress.yaml.disabled
```
To re-enable, rename back to `.yaml` and ensure Grafana is using a non-NFS PVC (local-path).
## Installing Prometheus
Uses `kube-prometheus-stack` Helm chart with **Grafana subchart disabled** (`grafana.enabled: false`):
```sh
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
# Create NFS storage directory first
mkdir -p /data/nfs/k3svolumes/prometheus/data
cd conf/f3s/prometheus && just install
```
### Enable etcd and controller-manager scraping
Add to `persistence-values.yaml`:
```yaml
kubeEtcd:
enabled: true
endpoints: [192.168.2.120, 192.168.2.121, 192.168.2.122]
service:
port: 2381
targetPort: 2381
kubeControllerManager:
enabled: true
endpoints: [192.168.2.120, 192.168.2.121, 192.168.2.122]
service:
port: 10257
targetPort: 10257
serviceMonitor:
enabled: true
https: true
insecureSkipVerify: true
```
Also requires k3s config changes on each r node — see [k3s-setup/install.md](../../f3s-k3s/references/install.md).
### Grafana credentials
Default: `admin` / `prom-operator` — change immediately after first login.
Grafana accessible at `grafana.f3s.foo.zone` via Traefik ingress.
## Installing Alloy (minimal)
Alloy is installed as part of the Loki Helm chart but runs with a minimal config (no log shipping):
```sh
cd conf/f3s/loki && just install
# installs alloy only (loki itself is disabled via loki.yaml.disabled)
```
### Current Alloy config (`alloy-values.yaml`)
Minimal — only emits Alloy's own operational logs:
```
logging {
level = "info"
}
```
To re-enable log shipping (once Loki is running again), restore the full `discovery.kubernetes` + `loki.source.kubernetes` + `loki.write` pipeline.
## Installing Loki (disabled)
```sh
mkdir -p /data/nfs/k3svolumes/loki/data
# Rename loki.yaml.disabled → loki.yaml first, then:
cd conf/f3s/loki && just install
```
Loki URL (internal): `http://loki.monitoring.svc.cluster.local:3100`
## Installing Tempo (disabled)
```sh
mkdir -p /data/nfs/k3svolumes/tempo/data
# Rename tempo.yaml.disabled → tempo.yaml first, then:
cd conf/f3s/tempo && just install
```
## Alerting
Prometheus → Alertmanager → **Gogios** (custom lightweight monitoring tool running on OpenBSD gateway `blowfish`/`fishfinger`).
Gogios scrapes Alertmanager at regular intervals and sends email notifications. Reaches Alertmanager via WireGuard mesh.
## Prometheus TSDB Recovery
If Prometheus fails to start with `opening storage failed: get segment range: segments are not sequential`, WAL segments are corrupt (can happen after a cluster blip leaving zero-byte WAL files).
Full TSDB wipe (loses all historical data — confirm first):
```sh
# On the NFS server (f0 or CARP MASTER)
rm -rf /data/nfs/k3svolumes/prometheus/data/prometheus-db/
mkdir -p /data/nfs/k3svolumes/prometheus/data/prometheus-db
chown 1000:1000 /data/nfs/k3svolumes/prometheus/data/prometheus-db
# Prometheus will recreate the TSDB on next start
```
## Useful LogQL Queries
```
# All logs from services namespace
{namespace="services"}
# Filter by log content
{namespace="services"} |= "error"
# Parse JSON logs
{namespace="services"} | json | level="error"
```
## NFS Storage Paths
```
/data/nfs/k3svolumes/prometheus/data # active
/data/nfs/k3svolumes/grafana/data # exists but unused (grafana disabled)
/data/nfs/k3svolumes/loki/data # exists but unused (loki disabled)
/data/nfs/k3svolumes/tempo/data # exists but unused (tempo disabled)
```
|
