1 files changed, 82 insertions, 0 deletions
diff --git a/docs/operations/setup-prometheus.md b/docs/operations/setup-prometheus.md
new file mode 100644
index 0000000..294ce20
--- /dev/null
+++ b/docs/operations/setup-prometheus.md
@@ -0,0 +1,82 @@
+# Setup: Prometheus
+
+To use historic mode, backfill mode, auto mode with old data, or watch mode with `-prometheus`, you must enable the Prometheus Remote Write receiver. Without it, Epimetheus can only push realtime data via Pushgateway.
+
+## 1. Enable Remote Write Receiver and Admin API
+
+Example configuration (Prometheus 3.x style). Adjust paths and stack to match your environment (e.g. [conf repository](https://codeberg.org/snonux/conf) at `f3s/prometheus/persistence-values.yaml`):
+
+```yaml
+prometheus:
+  prometheusSpec:
+    additionalArgs:
+      - name: web.enable-remote-write-receiver
+        value: ""
+      - name: web.enable-admin-api
+        value: ""
+
+    enableFeatures:
+      - exemplar-storage
+      - otlp-write-receiver
+
+    tsdb:
+      outOfOrderTimeWindow: 744h  # 31 days for backfilling
+```
+
+This provides:
+
+- **Remote Write API** at `/api/v1/write` for ingesting metrics with custom timestamps.
+- **Admin API** at `/api/v1/admin/tsdb/*` for deleting series and cleaning tombstones.
+- **Out-of-order ingestion** so older points can be written for existing series (within the time window).
+
+After changing config, upgrade Prometheus (e.g. `helm upgrade` or your usual apply).
+
+### Verify
+
+```bash
+# Remote Write receiver
+kubectl get pod -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
+  -o jsonpath='{.spec.containers[0].args}' | grep -o "web.enable-remote-write-receiver"
+
+# Out-of-order window
+kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus \
+  -o jsonpath='{.spec.tsdb.outOfOrderTimeWindow}'
+
+# Admin API
+kubectl get pod -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
+  -o jsonpath='{.spec.containers[0].args}' | grep -o "web.enable-admin-api"
+```
+
+**Note:** In Prometheus 3.x use `additionalArgs` for `web.enable-remote-write-receiver`; the older `enableFeatures: [remote-write-receiver]` is deprecated.
+
+## 2. Scrape Config for Pushgateway
+
+For realtime mode, Prometheus must scrape Pushgateway. Example:
+
+```yaml
+# additional-scrape-configs.yaml
+- job_name: 'pushgateway'
+  honor_labels: true
+  static_configs:
+    - targets:
+      - 'pushgateway.monitoring.svc.cluster.local:9091'
+```
+
+Apply as a Secret (example):
+
+```bash
+kubectl create secret generic additional-scrape-configs \
+  --from-file=additional-scrape-configs.yaml \
+  --dry-run=client -o yaml -n monitoring | kubectl apply -f -
+```
+
+## 3. Retention
+
+Check retention so you know how far back Epimetheus can write:
+
+```bash
+kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus \
+  -o jsonpath='{.spec.retention}'
+```
+
+For very old data, increase retention or use a dedicated dev/test Prometheus. Enabling out-of-order ingestion and a large `outOfOrderTimeWindow` has memory and I/O trade-offs; see [Prometheus backend](../backends/prometheus.md) and keep production config conservative.