docs: restructure documentation and move scripts to scripts/

- Add docs/ hierarchy: guides, backends, operations, reference, design
- Slim root README; add documentation index and links to docs/
- Add missing docs: csv-format-flexibility, dns-resolution, dtail-metrics-example, magefile
- Document Prometheus/VictoriaMetrics and ClickHouse backends
- Move all helper shell scripts to scripts/; update Magefile and doc references
- Add ASCII diagrams for watch mode (CSV watcher), auto mode, and ingestion paths
- Add .gitignore

Co-authored-by: Cursor <cursoragent@cursor.com>

1 files changed, 101 insertions, 0 deletions

diff --git a/docs/design/architecture.md b/docs/design/architecture.md
new file mode 100644
index 0000000..2a01e09
--- /dev/null
+++ b/docs/design/architecture.md
@@ -0,0 +1,101 @@
+# Architecture
+
+High-level data flow and when to use each ingestion path or backend.
+
+## Data flow
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                            Epimetheus                                   │
+│                     (Metrics Ingestion Tool)                            │
+│                                                                         │
+│  Modes:                                                                 │
+│  • Realtime  - Current metrics (< 5 min old)                            │
+│  • Historic  - Historic metrics (≥ 5 min old)                          │
+│  • Backfill  - Range of historic data                                  │
+│  • Auto      - Automatic routing based on timestamp age                 │
+│  • Watch     - CSV file monitoring (Prometheus and/or ClickHouse)       │
+└─────────────────────────────────────────────────────────────────────────┘
+         │                                           │
+         │ Realtime Data                            │ Historic Data
+         │ (via HTTP POST)                          │ (via Remote Write API)
+         │ Uses "now" timestamp                     │ Preserves timestamps
+         ▼                                           ▼
+┌─────────────────────┐                    ┌─────────────────────┐
+│   Pushgateway       │                    │ Prometheus /        │
+│   (Port 9091)       │                    │ VictoriaMetrics     │
+│                     │                    │ (Remote Write)      │
+│ • Buffers metrics   │                    │                     │
+│ • Scraped by        │──── Scraped ─────▶ │ /api/v1/write       │
+│   Prometheus        │    every 15-30s    │                     │
+└─────────────────────┘                    └─────────────────────┘
+                                                      │
+                                                      │ Query API
+                                                      ▼
+                                           ┌─────────────────────┐
+                                           │     Grafana         │
+                                           │   Dashboards        │
+                                           └─────────────────────┘
+```
+
+**Watch mode** can also write to **ClickHouse** (separate path; see [ClickHouse backend](../backends/clickhouse.md)).
+
+## Watch mode (CSV file watcher)
+
+Watch mode polls CSV file(s), uses file modification time as the sample timestamp, and can push to Prometheus (Remote Write) and/or ClickHouse.
+
+```
+┌─────────────────┐     poll (1s)      ┌─────────────────────────────────────┐
+│   CSV file(s)   │ ─────────────────▶ │  Epimetheus (watch mode)           │
+│                 │                    │  • Parse tabular CSV                │
+│  File mtime =   │                    │  • Numeric columns → metrics       │
+│  sample time    │                    │  • String columns → labels         │
+└─────────────────┘                    │  • Optional DNS resolution (IPs)  │
+                                       └─────────────────────────────────────┘
+                                                        │
+                                    ┌────────────────────┼────────────────────┐
+                                    │                    │                    │
+                                    ▼                    ▼                    │
+                            ┌───────────────┐    ┌───────────────┐            │
+                            │  Prometheus  │    │  ClickHouse   │            │
+                            │  (optional)  │    │  (optional)   │            │
+                            │  Remote Write │    │  HTTP insert  │            │
+                            │  /api/v1/write│    │  (batched)    │            │
+                            └───────────────┘    └───────────────┘            │
+```
+
+At least one of `-prometheus` or `-clickhouse` must be set. See [Operating Modes](../guides/modes.md) and [ClickHouse backend](../backends/clickhouse.md).
+
+## When to use Pushgateway vs Remote Write
+
+**Use Pushgateway (realtime mode):**
+
+- Short-lived batch jobs
+- Service-level metrics
+- Jobs behind firewalls
+- Current/recent data (< 5 minutes old)
+
+**Use Remote Write (historic, backfill, watch, or auto with old data):**
+
+- Historic data import
+- Backfilling gaps
+- Data migration
+- Data older than 5 minutes
+- Watch mode (to preserve file mtime as timestamp)
+
+**Use Auto mode:**
+
+- Mixed current and historic data in one file
+- Unknown timestamp ages
+- General-purpose file import
+
+## When to use which backend
+
+- **Prometheus or VictoriaMetrics:** Set `-prometheus=` to the backend’s Remote Write URL. Use for realtime (via Pushgateway scraped by Prometheus/VM), historic, backfill, auto, and watch.
+- **ClickHouse:** Set `-clickhouse=` in watch mode for analytics/long-term storage. Can be used together with Prometheus or alone (with `-prometheus=` empty).
+
+## Metric design (best practices)
+
+- **Types:** Counter for cumulative values (requests, errors); Gauge for point-in-time (temperature, connections); Histogram for distributions (latency).
+- **Labels:** Meaningful labels; avoid high cardinality (user IDs, raw timestamps); keep combinations reasonable (< 1000 per metric).
+- **Naming:** Descriptive names; units in gauge names (e.g. `_celsius`, `_bytes`); `_total` suffix for counters.