docs: restructure documentation and move scripts to scripts/

- Add docs/ hierarchy: guides, backends, operations, reference, design
- Slim root README; add documentation index and links to docs/
- Add missing docs: csv-format-flexibility, dns-resolution, dtail-metrics-example, magefile
- Document Prometheus/VictoriaMetrics and ClickHouse backends
- Move all helper shell scripts to scripts/; update Magefile and doc references
- Add ASCII diagrams for watch mode (CSV watcher), auto mode, and ingestion paths
- Add .gitignore

Co-authored-by: Cursor <cursoragent@cursor.com>

1 files changed, 130 insertions, 0 deletions

diff --git a/docs/guides/modes.md b/docs/guides/modes.md
new file mode 100644
index 0000000..bcbbb6b
--- /dev/null
+++ b/docs/guides/modes.md
@@ -0,0 +1,130 @@
+# Operating Modes
+
+Epimetheus has five modes. Backend support:
+
+| Mode      | Prometheus (Pushgateway) | Prometheus (Remote Write) | ClickHouse |
+|-----------|--------------------------|---------------------------|------------|
+| Realtime  | Yes                      | No                        | No         |
+| Historic  | No                       | Yes                       | No         |
+| Backfill  | No                       | Yes                       | No         |
+| Auto      | Yes (samples < 5 min)  | Yes (samples ≥ 5 min)     | No         |
+| Watch     | Optional                 | Optional                  | Optional   |
+
+At least one of Prometheus or ClickHouse must be configured for watch mode.
+
+---
+
+## Watch Mode
+
+Monitor CSV files and push metrics using file modification time as the timestamp. Works with any tabular CSV; numeric columns become metrics, string columns become labels.
+
+### Watch mode data flow
+
+```
+┌─────────────────┐     poll (1s)      ┌─────────────────────────────────────┐
+│   CSV file(s)   │ ─────────────────▶ │  Epimetheus (watch mode)           │
+│                 │                    │  • Parse tabular CSV                │
+│  File mtime =   │                    │  • Numeric columns → metrics        │
+│  sample time    │                    │  • String columns → labels         │
+└─────────────────┘                    │  • Optional DNS resolution (IPs)  │
+                                       └─────────────────────────────────────┘
+                                                        │
+                                    ┌────────────────────┼────────────────────┐
+                                    │                    │                    │
+                                    ▼                    ▼                    │
+                            ┌───────────────┐    ┌───────────────┐            │
+                            │  Prometheus   │    │  ClickHouse   │            │
+                            │  (optional)   │    │  (optional)   │            │
+                            │               │    │               │            │
+                            │  Remote Write │    │  HTTP insert  │            │
+                            │  /api/v1/write│    │  (batched)    │            │
+                            └───────────────┘    └───────────────┘            │
+                                    │                    │                    │
+                                    └────────────────────┴────────────────────┘
+                                    At least one of -prometheus or -clickhouse
+```
+
+```bash
+./epimetheus -mode=watch -file=mydata.csv -metric-name=myapp \
+  -prometheus=http://localhost:9090/api/v1/write
+```
+
+**Options:** `-file`, `-metric-name`, `-prometheus`, `-clickhouse`, `-clickhouse-table`, `-job`, `-resolve-ip-labels`. See [CLI Reference](../reference/cli.md).
+
+**Features:** Format-agnostic CSV, automatic numeric/string detection, label name sanitization, optional DNS resolution for IP labels, timestamp from file mtime, continuous polling (1s), Remote Write (and optionally ClickHouse). See [CSV Format Flexibility](csv-format-flexibility.md) and [DNS Resolution](dns-resolution.md).
+
+---
+
+## Realtime Mode (default)
+
+Push current metrics to Pushgateway with "now" timestamp.
+
+```bash
+./epimetheus -mode=realtime -continuous
+```
+
+**Options:** `-pushgateway` (default `http://localhost:9091`), `-job`, `-continuous`. Pushes every 15 seconds when `-continuous` is set.
+
+---
+
+## Historic Mode
+
+Push a single historic datapoint via Remote Write.
+
+```bash
+./epimetheus -mode=historic -hours-ago=24
+```
+
+**Options:** `-prometheus` (default `http://localhost:9090/api/v1/write`), `-hours-ago` (default 24). Requires Remote Write receiver. See [Backends: Prometheus](../backends/prometheus.md).
+
+---
+
+## Backfill Mode
+
+Import a range of historic data points.
+
+```bash
+./epimetheus -mode=backfill -start-hours=48 -end-hours=0 -interval=1
+./epimetheus -mode=backfill -start-hours=168 -end-hours=0 -interval=6
+```
+
+**Options:** `-start-hours`, `-end-hours` (0 = now), `-interval` (hours between points). Requires Remote Write receiver.
+
+---
+
+## Auto Mode
+
+Route samples by timestamp age: < 5 minutes → Pushgateway; ≥ 5 minutes → Remote Write. Use for mixed or unknown-age data.
+
+### Auto mode data flow
+
+```
+┌─────────────────┐                    ┌─────────────────────────────────────┐
+│  CSV/JSON file  │ ─────────────────▶ │  Epimetheus (auto mode)             │
+│  (per-sample    │                    │  • Parse file (csv or json)          │
+│   timestamps)   │                    │  • Route by sample age:               │
+└─────────────────┘                    │    < 5 min → Pushgateway            │
+                                       │    ≥ 5 min → Remote Write           │
+                                       └─────────────────────────────────────┘
+                                                        │
+                                    ┌────────────────────┴────────────────────┐
+                                    ▼                     ▼                    │
+                            ┌───────────────┐     ┌───────────────┐            │
+                            │  Pushgateway  │     │  Prometheus   │            │
+                            │  (realtime    │     │  Remote Write │            │
+                            │   samples)    │     │  (historic    │            │
+                            └───────┬───────┘     │   samples)    │            │
+                                    │             └───────────────┘            │
+                                    │  Scraped by Prometheus                   │
+                                    ▼                                           │
+                            ┌───────────────┐                                   │
+                            │  Prometheus   │◀──────────────────────────────────┘
+                            └───────────────┘
+```
+
+```bash
+./scripts/generate-test-data.sh
+./epimetheus -mode=auto -file=test-all-ages.csv
+```
+
+**Options:** `-file`, `-format` (csv or json), `-pushgateway`, `-prometheus`. See [Data Formats](data-formats.md).