docs: restructure documentation and move scripts to scripts/

- Add docs/ hierarchy: guides, backends, operations, reference, design
- Slim root README; add documentation index and links to docs/
- Add missing docs: csv-format-flexibility, dns-resolution, dtail-metrics-example, magefile
- Document Prometheus/VictoriaMetrics and ClickHouse backends
- Move all helper shell scripts to scripts/; update Magefile and doc references
- Add ASCII diagrams for watch mode (CSV watcher), auto mode, and ingestion paths
- Add .gitignore

Co-authored-by: Cursor <cursoragent@cursor.com>

22 files changed, 1581 insertions, 0 deletions

diff --git a/docs/DOCS-RESTRUCTURE-PLAN.md b/docs/DOCS-RESTRUCTURE-PLAN.md
new file mode 100644
index 0000000..c688993
--- /dev/null
+++ b/docs/DOCS-RESTRUCTURE-PLAN.md
@@ -0,0 +1,235 @@
+# Documentation Restructure Plan
+
+This plan addresses the current documentation sprawl and clarifies the **multiple ingestion backends** (Prometheus, ClickHouse, and future backends such as VictoriaMetrics) and **modes** (realtime, historic, backfill, auto, watch).
+
+---
+
+## 1. Current State Summary
+
+### 1.1 Existing Markdown Files
+
+| File        | Purpose                          | Issues |
+|------------|-----------------------------------|--------|
+| `README.md` | Single ~995-line doc: intro, modes, backends, setup, troubleshooting, macOS, cleanup | Too long; mixes audiences and backends; hard to maintain |
+| `AGENT.md`  | Agent rules (Grafana dashboard guidelines + ref to `~/git/conf/snippets/go/go-projects.md`) | Fine as-is; not user docs |
+| `CLAUDE.md` | One-line pointer to AGENT.md      | Fine as-is |
+
+### 1.2 Broken or Missing References in README
+
+- `CSV-FORMAT-FLEXIBILITY.md` – linked, **does not exist**
+- `DNS-RESOLUTION-FEATURE.md` – linked, **does not exist**
+- `DTAIL-METRICS-EXAMPLE.md` – linked, **does not exist**
+- `MAGEFILE.md` – linked, **does not exist** (build logic lives in `Magefile.go`)
+
+### 1.3 Ingestion Backends (from codebase)
+
+| Backend    | Modes                    | Notes |
+|-----------|---------------------------|--------|
+| **Prometheus** | realtime (Pushgateway), historic/backfill/auto (Remote Write), watch (Remote Write) | Primary; Remote Write requires feature flag |
+| **ClickHouse** | watch only               | Optional; can run with Prometheus or alone |
+
+*VictoriaDB / VictoriaMetrics:* Not present in code today. Plan leaves room for a dedicated backend doc when added.
+
+---
+
+## 2. Goals
+
+1. **Separate by ingestion backend** so Prometheus vs ClickHouse (and future backends) have clear, non-redundant docs.
+2. **Split by audience and topic**: quick start vs reference vs operations (setup, troubleshooting, cleanup).
+3. **Fix broken links**: either add the missing docs or replace links with in-README sections / new doc paths.
+4. **Single source of truth** for each concept (e.g. “how watch mode works” and “how to configure Prometheus” in one place each).
+5. **Easier maintenance**: smaller, focused files; clear naming; one `docs/` tree.
+
+---
+
+## 3. Proposed Directory Layout
+
+```
+epimetheus/
+├── README.md                    # Short overview + quick start + doc index (slimmed)
+├── AGENT.md                     # Unchanged
+├── CLAUDE.md                    # Unchanged
+├── docs/
+│   ├── README.md                # Documentation index (nav + short descriptions)
+│   │
+│   ├── guides/                  # How-to and concepts
+│   │   ├── quickstart.md        # Minimal path to first push (Prometheus or ClickHouse)
+│   │   ├── modes.md             # All modes: realtime, historic, backfill, auto, watch
+│   │   ├── data-formats.md      # CSV (epimetheus + tabular) and JSON
+│   │   ├── csv-format-flexibility.md   # “Any CSV” + examples (replaces missing file)
+│   │   ├── dns-resolution.md    # IP → hostname resolution (replaces missing file)
+│   │   └── dtail-metrics-example.md    # Optional: dtail.csv walkthrough (replaces missing file)
+│   │
+│   ├── backends/                # One doc per ingestion backend
+│   │   ├── prometheus.md        # Pushgateway + Remote Write, config, limits
+│   │   ├── clickhouse.md        # Watch-only; schema; verify script
+│   │   └── (future) victoriametrics.md  # When/if added
+│   │
+│   ├── operations/              # Setup, runbooks, platform-specific
+│   │   ├── setup-prometheus.md  # Remote Write receiver, scrape config, retention
+│   │   ├── setup-clickhouse.md  # Table creation, verify-clickhouse.sh
+│   │   ├── troubleshooting.md   # Connection issues, “no metrics”, out-of-order, etc.
+│   │   ├── cleanup.md           # Benchmark cleanup, Pushgateway delete, port-forwards
+│   │   ├── macos-setup.md       # Brew, Prometheus args, Remote Write on macOS
+│   │   └── kubernetes.md        # Port-forwards, Helm, ConfigMaps (from current README)
+│   │
+│   ├── reference/               # Reference material
+│   │   ├── cli.md               # All flags by mode
+│   │   ├── test-metrics.md      # epimetheus_test_* metrics and types
+│   │   ├── grafana-dashboard.md # Panels, deploy options, datasource
+│   │   ├── example-queries.md   # PromQL and curl examples
+│   │   └── magefile.md          # Mage targets (replaces missing MAGEFILE.md)
+│   │
+│   └── design/                  # Optional, for contributors
+│       └── architecture.md     # High-level data flow (current ASCII diagrams)
+```
+
+---
+
+## 4. File-by-File Plan
+
+### 4.1 Root `README.md` (slimmed)
+
+- **Keep:** Project name, tagline, “Why Epimetheus”, **one** high-level architecture diagram (simplified).
+- **Keep:** Very short “Overview” (1 paragraph) and **Quick Start** (3–5 steps pointing at `docs/guides/quickstart.md` for details).
+- **Add:** **Documentation index** – bullet list with links to:
+  - `docs/README.md`
+  - `docs/guides/quickstart.md`, `docs/guides/modes.md`
+  - `docs/backends/prometheus.md`, `docs/backends/clickhouse.md`
+  - `docs/operations/setup-prometheus.md`, `docs/operations/troubleshooting.md`
+  - `docs/reference/cli.md`, `docs/reference/magefile.md`
+- **Move out of README into `docs/`:**
+  - All mode details → `docs/guides/modes.md`
+  - Backend-specific behaviour → `docs/backends/*.md`
+  - Setup (Prometheus, ClickHouse, k8s, macOS) → `docs/operations/*.md`
+  - Data formats → `docs/guides/data-formats.md` (+ csv-format-flexibility, dns-resolution, dtail example)
+  - Test metrics, Grafana, example queries → `docs/reference/*.md`
+  - Troubleshooting, cleanup → `docs/operations/*.md`
+  - Time range / retention → `docs/backends/prometheus.md` and `docs/operations/setup-prometheus.md`
+- **Fix links:** Remove links to `CSV-FORMAT-FLEXIBILITY.md`, `DNS-RESOLUTION-FEATURE.md`, `DTAIL-METRICS-EXAMPLE.md`, `MAGEFILE.md` from README; point to `docs/guides/...` and `docs/reference/magefile.md` instead.
+
+**Target:** README under ~150–200 lines.
+
+---
+
+### 4.2 `docs/README.md` (new)
+
+- Title: “Epimetheus Documentation”.
+- Short intro (2–3 sentences).
+- **Structured index** with sections:
+  - **Guides:** quickstart, modes, data formats, CSV flexibility, DNS resolution, dtail example.
+  - **Ingestion backends:** Prometheus, ClickHouse (and placeholder for Victoria* if desired).
+  - **Operations:** setup (Prometheus, ClickHouse), troubleshooting, cleanup, macOS, Kubernetes.
+  - **Reference:** CLI, test metrics, Grafana, example queries, Mage.
+- Each entry: link + one-line description.
+
+---
+
+### 4.3 Guides
+
+| Doc | Content | Source |
+|-----|--------|--------|
+| `guides/quickstart.md` | Minimal steps: build/run, push to Prometheus or ClickHouse, view (Prometheus UI or verify-clickhouse.sh). | Current README “Quick Start” + “Run in Realtime Mode” + one watch example. |
+| `guides/modes.md` | Table: mode name, purpose, which backends, main flags. Then one subsection per mode (realtime, historic, backfill, auto, watch) with short description and example command. | Current README “Operating Modes”. |
+| `guides/data-formats.md` | Epimetheus CSV (metric_name, labels, value, timestamp_ms), JSON format, optional timestamp. Link to csv-format-flexibility for tabular CSV. | Current README “Data Formats”. |
+| `guides/csv-format-flexibility.md` | “Works with any CSV”: headers → metric names/labels, numeric vs string columns, sanitization, examples (web, food). | New content; replaces missing `CSV-FORMAT-FLEXIBILITY.md`. |
+| `guides/dns-resolution.md` | Default `ip` resolution; `-resolve-ip-labels`; behaviour on failure. | New content; replaces missing `DNS-RESOLUTION-FEATURE.md`. |
+| `guides/dtail-metrics-example.md` | Optional: step-by-step dtail.csv example. | New content; replaces missing `DTAIL-METRICS-EXAMPLE.md`; can be short. |
+
+---
+
+### 4.4 Backends
+
+| Doc | Content | Source |
+|-----|--------|--------|
+| `backends/prometheus.md` | Pushgateway (realtime) vs Remote Write (historic/watch); URLs; time range and retention limits; out-of-order; link to setup-prometheus. | README Prometheus bits + “Time Range Limitations” + “Setup Requirements” (Remote Write). |
+| `backends/clickhouse.md` | Watch-only; `-clickhouse`, `-clickhouse-table`; table schema (from code/comments); `verify-clickhouse.sh`; Prometheus + ClickHouse together. | README “ClickHouse Support” + verify-clickhouse.sh + internal/ingester/clickhouse.go. |
+
+---
+
+### 4.5 Operations
+
+| Doc | Content | Source |
+|-----|--------|--------|
+| `operations/setup-prometheus.md` | Enable Remote Write receiver (and Admin API); scrape config for Pushgateway; retention; Prometheus 3.x syntax; verify commands. | Current README “Setup Requirements” (Prometheus). |
+| `operations/setup-clickhouse.md` | Ensure table exists (e.g. from ingester); run verify script; optional Docker/systemd. | From README + scripts + code. |
+| `operations/troubleshooting.md` | Pushgateway connection; metrics not in Prometheus; “Remote write receiver not enabled”; out-of-order errors; dashboard not in Grafana; ClickHouse connection. | Current README “Troubleshooting”. |
+| `operations/cleanup.md` | Cleanup benchmark data script; manual Prometheus delete/tombstones; Pushgateway delete; stop port-forwards; uninstall Pushgateway. | Current README “Cleanup”. |
+| `operations/macos-setup.md` | Brew install; prometheus.args (Remote Write, Admin API); verify; optional “temporary” run. | Current README “MacOS Setup”. |
+| `operations/kubernetes.md` | Port-forwards (Pushgateway, Prometheus, Grafana); Helm/ConfigMap for dashboard; namespace. | Extracted from README examples. |
+
+---
+
+### 4.6 Reference
+
+| Doc | Content | Source |
+|-----|--------|--------|
+| `reference/cli.md` | Table or list of all flags by mode (realtime, historic, backfill, auto, watch); default values. | From README + `cmd/epimetheus/main.go`. |
+| `reference/test-metrics.md` | Each `epimetheus_test_*` metric: type, description, labels, use case. | Current README “Test Metrics”. |
+| `reference/grafana-dashboard.md` | Panels list; deploy (ConfigMap, manual import, script); datasource; link to AGENT.md for panel guidelines. | Current README “Grafana Dashboard”. |
+| `reference/example-queries.md` | PromQL and curl examples (basic, histogram, labeled counter). | Current README “Example Queries”. |
+| `reference/magefile.md` | List of Mage targets (build, test, run, RunWatchClickHouse, cleanup, etc.) with one-line description and example. | From `Magefile.go`; replaces missing `MAGEFILE.md`. |
+
+---
+
+### 4.7 Design (optional)
+
+| Doc | Content | Source |
+|-----|--------|--------|
+| `design/architecture.md` | High-level data flow; ASCII diagrams (current README); “when to use Pushgateway vs Remote Write” and “when to use which backend”. | Current README “Architecture” and “Best Practices”. |
+
+---
+
+## 5. Implementation Order
+
+1. **Create `docs/` and index**
+   - Create `docs/README.md` with the full index (links can target paths that don’t exist yet).
+2. **Fix broken links and add missing content**
+   - Add `docs/guides/csv-format-flexibility.md`, `docs/guides/dns-resolution.md`, `docs/guides/dtail-metrics-example.md`, `docs/reference/magefile.md` so all current README links resolve.
+3. **Backend-centric docs**
+   - Add `docs/backends/prometheus.md` and `docs/backends/clickhouse.md`; move/duplicate content from README.
+4. **Operations**
+   - Add `docs/operations/setup-prometheus.md`, `setup-clickhouse.md`, `troubleshooting.md`, `cleanup.md`, `macos-setup.md`, `kubernetes.md`; move content from README.
+5. **Guides**
+   - Add `docs/guides/quickstart.md`, `modes.md`, `data-formats.md`; move content from README.
+6. **Reference**
+   - Add `docs/reference/cli.md`, `test-metrics.md`, `grafana-dashboard.md`, `example-queries.md`; move content from README.
+7. **Slim README**
+   - Cut README down to overview, quick start, and doc index; replace old links with `docs/...` links.
+8. **Optional**
+   - Add `docs/design/architecture.md` and link from `docs/README.md`.
+
+---
+
+## 6. Cross-Cutting Conventions
+
+- **Links:** Prefer relative links from repo root (e.g. `[Modes](docs/guides/modes.md)`) or from `docs/` (e.g. `[Prometheus](backends/prometheus.md)` inside docs).
+- **Backend mentions:** In mode/CLI docs, use a short table or sentence: “Supported backends: Prometheus (all modes), ClickHouse (watch only).”
+- **One diagram:** Keep one high-level diagram in README or `design/architecture.md`; avoid duplicating large ASCII art in multiple files.
+- **CLI and defaults:** Single source of truth in `reference/cli.md`; guides and backend docs can quote the relevant subset.
+- **Version/legal:** Keep “Version” and “License” in root README (or CONTRIBUTING.md if you add one).
+
+---
+
+## 7. Future: VictoriaMetrics / VictoriaDB
+
+When adding a new backend (e.g. VictoriaMetrics, which speaks Prometheus Remote Write):
+
+- Add `docs/backends/victoriametrics.md` (or `victoriadb.md`) with URL format, any extra flags, and differences from Prometheus.
+- In `docs/README.md` and root README, add one line to the “Ingestion backends” section.
+- In `docs/guides/modes.md` and `reference/cli.md`, extend the “which backends support which mode” table and flags.
+- No need to duplicate full setup/troubleshooting if it matches Prometheus; link to `backends/prometheus.md` and note compatibility where relevant.
+
+---
+
+## 8. Checklist Before Calling Done
+
+- [ ] All current README links resolve (no 404s).
+- [ ] README is under ~200 lines and ends with doc index.
+- [ ] `docs/README.md` lists every new doc with link and one-line description.
+- [ ] Prometheus vs ClickHouse (and modes) are clearly separated in backends and guides.
+- [ ] Setup, troubleshooting, and cleanup live under `docs/operations/`.
+- [ ] Mage is documented in `docs/reference/magefile.md` and linked from root README.
+- [ ] Optional: `docs/design/architecture.md` exists and is linked from index.
+
+This plan gives you a single place to extend when you add VictoriaDB/VictoriaMetrics or another backend, and keeps the root README short while all detailed docs live under `docs/` with a clear structure by topic and backend.
diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 0000000..5f944d4
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,66 @@
+# Epimetheus Documentation
+
+Documentation for Epimetheus: a Go tool for pushing metrics to Prometheus (and Prometheus-compatible backends) and ClickHouse.
+
+## Index
+
+### Guides
+
+| Document | Description |
+|----------|-------------|
+| [Quick Start](guides/quickstart.md) | Minimal path to first push: build, run, view in Prometheus or ClickHouse |
+| [Operating Modes](guides/modes.md) | Realtime, historic, backfill, auto, and watch modes with examples |
+| [Data Formats](guides/data-formats.md) | Epimetheus CSV and JSON input formats |
+| [CSV Format Flexibility](guides/csv-format-flexibility.md) | Use any tabular CSV; numeric vs string columns; sanitization and examples |
+| [DNS Resolution](guides/dns-resolution.md) | IP-to-hostname resolution for watch mode labels |
+| [Dtail Metrics Example](guides/dtail-metrics-example.md) | Walkthrough using dtail.csv |
+
+### Ingestion Backends
+
+| Document | Description |
+|----------|-------------|
+| [Prometheus (and Prometheus-compatible)](backends/prometheus.md) | Pushgateway, Remote Write, time ranges; VictoriaMetrics via same URL |
+| [ClickHouse](backends/clickhouse.md) | Watch-mode ingestion; table schema; verify script |
+
+### Operations
+
+| Document | Description |
+|----------|-------------|
+| [Setup: Prometheus](operations/setup-prometheus.md) | Enable Remote Write receiver, scrape config, retention |
+| [Setup: ClickHouse](operations/setup-clickhouse.md) | Table creation, verification |
+| [Troubleshooting](operations/troubleshooting.md) | Connection issues, metrics not appearing, out-of-order errors |
+| [Cleanup](operations/cleanup.md) | Benchmark cleanup, Pushgateway delete, port-forwards |
+| [macOS Setup](operations/macos-setup.md) | Homebrew, Prometheus args, Remote Write on macOS |
+| [Kubernetes](operations/kubernetes.md) | Port-forwards, Helm, ConfigMaps |
+
+### Reference
+
+| Document | Description |
+|----------|-------------|
+| [CLI Reference](reference/cli.md) | All flags by mode with defaults |
+| [Test Metrics](reference/test-metrics.md) | epimetheus_test_* metrics and types |
+| [Grafana Dashboard](reference/grafana-dashboard.md) | Panels, deployment options, datasource |
+| [Example Queries](reference/example-queries.md) | PromQL and curl examples |
+| [Magefile](reference/magefile.md) | Mage build and run targets |
+
+### Design
+
+| Document | Description |
+|----------|-------------|
+| [Architecture](design/architecture.md) | Data flow, when to use Pushgateway vs Remote Write, backend choice |
+
+### Helper scripts
+
+Helper shell scripts live in **`scripts/`** at the repo root. Run them from the repo root (e.g. `./scripts/verify-clickhouse.sh`).
+
+| Script | Purpose |
+|--------|---------|
+| `verify-clickhouse.sh` | Verify ClickHouse ingestion (row count, sample data) |
+| `generate-test-data.sh` | Generate `test-all-ages.csv` for auto mode |
+| `cleanup-benchmark-data.sh` | Delete benchmark metrics from Prometheus (Admin API) |
+| `cleanup-benchmark-metrics.sh` | Same + starts port-forward, then cleans up |
+| `benchmark-100mb.sh` | 100MB ingestion benchmark |
+| `benchmark-1gb.sh` | 1GB ingestion benchmark |
+| `backfill-historic-data.sh` | Backfill 7 days of historic data to Prometheus |
+| `run.sh` | Port-forward Pushgateway and run epimetheus in realtime mode |
+| `deploy-dashboard.sh` | Deploy Grafana dashboard via API (if present) |
diff --git a/docs/backends/clickhouse.md b/docs/backends/clickhouse.md
new file mode 100644
index 0000000..ad1b5f0
--- /dev/null
+++ b/docs/backends/clickhouse.md
@@ -0,0 +1,92 @@
+# ClickHouse
+
+Epimetheus can ingest metrics into ClickHouse in **watch mode** only. ClickHouse is optional: you can use it in addition to Prometheus or as the only backend (by setting `-prometheus=` to disable Prometheus ingestion).
+
+## Data flow (watch mode only)
+
+```
+┌─────────────────┐     poll (1s)      ┌─────────────────────────────────────┐
+│   CSV file(s)   │ ─────────────────▶ │  Epimetheus (watch mode)           │
+│   (mtime =      │                    │  • Parse tabular CSV                 │
+│   timestamp)    │                    │  • -metric-name + columns → metrics  │
+└─────────────────┘                    └─────────────────────────────────────┘
+                                                        │
+                                    ┌────────────────────┼────────────────────┐
+                                    │                    │                    │
+                                    ▼                    ▼                    │
+                            ┌───────────────┐    ┌───────────────┐            │
+                            │  Prometheus   │    │  ClickHouse   │            │
+                            │  (optional)   │    │  (optional)   │            │
+                            │  -prometheus= │    │  -clickhouse= │            │
+                            │  Remote Write │    │  HTTP insert  │            │
+                            └───────────────┘    └───────────────┘            │
+                                                                              │
+                                    At least one of -prometheus or -clickhouse │
+```
+
+## When It's Used
+
+- **Mode:** Watch only. Other modes (realtime, historic, backfill, auto) do not write to ClickHouse.
+- **Flags:**
+  - `-clickhouse` – ClickHouse HTTP URL (e.g. `http://localhost:8123`). If empty, no ClickHouse ingestion.
+  - `-clickhouse-table` – Table name (default: `epimetheus_metrics`).
+
+At least one of `-prometheus` or `-clickhouse` must be set for watch mode.
+
+## Table Schema
+
+Epimetheus creates the table if it does not exist. Schema:
+
+```sql
+CREATE TABLE IF NOT EXISTS epimetheus_metrics (
+    metric String,
+    labels Map(String, String),
+    value Float64,
+    timestamp DateTime64(3)
+) ENGINE = MergeTree()
+ORDER BY (metric, timestamp)
+```
+
+- `metric` – metric name (e.g. from `-metric-name` and column headers in tabular CSV).
+- `labels` – key-value map of label names and values.
+- `value` – sample value.
+- `timestamp` – sample time (millisecond precision).
+
+## Examples
+
+**Prometheus and ClickHouse:**
+
+```bash
+./epimetheus -mode=watch -file=data.csv -metric-name=myapp \
+  -prometheus=http://localhost:9090/api/v1/write \
+  -clickhouse=http://localhost:8123
+```
+
+**ClickHouse only:**
+
+```bash
+./epimetheus -mode=watch -file=test-data/watch-clickhouse-test.csv \
+  -metric-name=watch_test \
+  -clickhouse=http://localhost:8123 \
+  -prometheus=
+```
+
+**Custom table:**
+
+```bash
+./epimetheus -mode=watch -file=data.csv -metric-name=myapp \
+  -clickhouse=http://localhost:8123 \
+  -clickhouse-table=my_metrics
+```
+
+## Verification
+
+Use the provided script to check that data landed in ClickHouse:
+
+```bash
+./scripts/verify-clickhouse.sh
+# Or with custom URL/table:
+./scripts/verify-clickhouse.sh http://localhost:8123 epimetheus_metrics
+```
+
+The script checks connectivity, row count, distinct metrics, sample rows, and rows per metric. See [Setup: ClickHouse](../operations/setup-clickhouse.md) for getting ClickHouse running.
diff --git a/docs/backends/prometheus.md b/docs/backends/prometheus.md
new file mode 100644
index 0000000..f8d2a9b
--- /dev/null
+++ b/docs/backends/prometheus.md
@@ -0,0 +1,76 @@
+# Prometheus (and Prometheus-Compatible Backends)
+
+Epimetheus can ingest metrics into Prometheus via two paths. Any backend that exposes the Prometheus Remote Write API (including **VictoriaMetrics**) is supported by pointing `-prometheus=` at that backend's write URL (e.g. `http://victoriametrics:8428/api/v1/write`).
+
+## Ingestion paths (overview)
+
+```
+                    Epimetheus
+                         │
+         ┌───────────────┼───────────────┐
+         │               │               │
+         ▼               ▼               ▼
+   Realtime mode    Historic/Backfill   Watch mode
+   (current data)   (old data)          (CSV file mtime)
+         │               │               │
+         ▼               │               │
+   ┌───────────┐         │               │
+   │Pushgateway │         │               │
+   │ (HTTP POST)│         │               │
+   └─────┬─────┘         │               │
+         │ Scrape         │               │
+         │ (15–30s)       │               │
+         ▼               ▼               ▼
+   ┌─────────────────────────────────────────────┐
+   │  Prometheus / VictoriaMetrics               │
+   │  Remote Write API: /api/v1/write            │
+   │  (realtime: via Pushgateway scrape;         │
+   │   historic/watch: direct POST)              │
+   └─────────────────────────────────────────────┘
+```
+
+## Ingestion Paths
+
+### Realtime: Pushgateway
+
+- **Used by:** realtime mode, and auto mode for samples < 5 minutes old.
+- **Flow:** Epimetheus pushes to Pushgateway (HTTP POST); Prometheus scrapes Pushgateway on its schedule. Timestamps become "now" at scrape time.
+- **Flags:** `-pushgateway` (default `http://localhost:9091`), `-job`, `-continuous`.
+
+### Historic: Remote Write API
+
+- **Used by:** historic mode, backfill mode, auto mode for samples ≥ 5 minutes old, and watch mode (when `-prometheus` is set).
+- **Flow:** Epimetheus sends samples to the Remote Write endpoint (e.g. `/api/v1/write`). Timestamps from the data are preserved.
+- **Flags:** `-prometheus` (default `http://localhost:9090/api/v1/write`).
+
+The Remote Write receiver must be enabled on Prometheus for historic/watch/backfill/auto with old data. See [Setup: Prometheus](../operations/setup-prometheus.md).
+
+## Prometheus-Compatible Backends (e.g. VictoriaMetrics)
+
+Backends that implement the [Prometheus Remote Write](https://prometheus.io/docs/concepts/remote_write_spec/) API work with Epimetheus without any code changes. Use their write endpoint as the `-prometheus=` URL.
+
+**Example (VictoriaMetrics):**
+
+```bash
+./epimetheus -mode=watch -file=data.csv -metric-name=myapp \
+  -prometheus=http://victoriametrics:8428/api/v1/write
+```
+
+Replace host/port with your VictoriaMetrics (or other compatible) write URL. Realtime mode still uses Pushgateway (scraped by your Prometheus or VictoriaMetrics); for watch/historic/backfill/auto, only the `-prometheus=` target changes.
+
+## Time Ranges
+
+| Time range | Status | Method |
+|------------|--------|--------|
+| Current (< 5 min) | Supported | Pushgateway |
+| 1 hour old | Supported | Remote Write |
+| 1 day to 1 month old | Supported | Remote Write |
+| 6+ months | May be rejected (retention) | Remote Write |
+| Years old | Likely rejected; use `promtool tsdb create-blocks-from` | — |
+| Future (> 5 min ahead) | Rejected | — |
+
+Out-of-order samples (older than existing data for the same series) require out-of-order ingestion to be enabled on the backend, or use different labels. See [Troubleshooting](../operations/troubleshooting.md).
+
+## Retention and Configuration
+
+Check your backend's retention (e.g. Prometheus `retention`, VictoriaMetrics settings). For very old data you may need to increase retention or enable out-of-order ingestion. See [Setup: Prometheus](../operations/setup-prometheus.md) for Prometheus-specific options.
diff --git a/docs/design/architecture.md b/docs/design/architecture.md
new file mode 100644
index 0000000..2a01e09
--- /dev/null
+++ b/docs/design/architecture.md
@@ -0,0 +1,101 @@
+# Architecture
+
+High-level data flow and when to use each ingestion path or backend.
+
+## Data flow
+
+```
+┌─────────────────────────────────────────────────────────────────────────┐
+│                            Epimetheus                                   │
+│                     (Metrics Ingestion Tool)                            │
+│                                                                         │
+│  Modes:                                                                 │
+│  • Realtime  - Current metrics (< 5 min old)                            │
+│  • Historic  - Historic metrics (≥ 5 min old)                          │
+│  • Backfill  - Range of historic data                                  │
+│  • Auto      - Automatic routing based on timestamp age                 │
+│  • Watch     - CSV file monitoring (Prometheus and/or ClickHouse)       │
+└─────────────────────────────────────────────────────────────────────────┘
+         │                                           │
+         │ Realtime Data                            │ Historic Data
+         │ (via HTTP POST)                          │ (via Remote Write API)
+         │ Uses "now" timestamp                     │ Preserves timestamps
+         ▼                                           ▼
+┌─────────────────────┐                    ┌─────────────────────┐
+│   Pushgateway       │                    │ Prometheus /        │
+│   (Port 9091)       │                    │ VictoriaMetrics     │
+│                     │                    │ (Remote Write)      │
+│ • Buffers metrics   │                    │                     │
+│ • Scraped by        │──── Scraped ─────▶ │ /api/v1/write       │
+│   Prometheus        │    every 15-30s    │                     │
+└─────────────────────┘                    └─────────────────────┘
+                                                      │
+                                                      │ Query API
+                                                      ▼
+                                           ┌─────────────────────┐
+                                           │     Grafana         │
+                                           │   Dashboards        │
+                                           └─────────────────────┘
+```
+
+**Watch mode** can also write to **ClickHouse** (separate path; see [ClickHouse backend](../backends/clickhouse.md)).
+
+## Watch mode (CSV file watcher)
+
+Watch mode polls CSV file(s), uses file modification time as the sample timestamp, and can push to Prometheus (Remote Write) and/or ClickHouse.
+
+```
+┌─────────────────┐     poll (1s)      ┌─────────────────────────────────────┐
+│   CSV file(s)   │ ─────────────────▶ │  Epimetheus (watch mode)           │
+│                 │                    │  • Parse tabular CSV                │
+│  File mtime =   │                    │  • Numeric columns → metrics       │
+│  sample time    │                    │  • String columns → labels         │
+└─────────────────┘                    │  • Optional DNS resolution (IPs)  │
+                                       └─────────────────────────────────────┘
+                                                        │
+                                    ┌────────────────────┼────────────────────┐
+                                    │                    │                    │
+                                    ▼                    ▼                    │
+                            ┌───────────────┐    ┌───────────────┐            │
+                            │  Prometheus  │    │  ClickHouse   │            │
+                            │  (optional)  │    │  (optional)   │            │
+                            │  Remote Write │    │  HTTP insert  │            │
+                            │  /api/v1/write│    │  (batched)    │            │
+                            └───────────────┘    └───────────────┘            │
+```
+
+At least one of `-prometheus` or `-clickhouse` must be set. See [Operating Modes](../guides/modes.md) and [ClickHouse backend](../backends/clickhouse.md).
+
+## When to use Pushgateway vs Remote Write
+
+**Use Pushgateway (realtime mode):**
+
+- Short-lived batch jobs
+- Service-level metrics
+- Jobs behind firewalls
+- Current/recent data (< 5 minutes old)
+
+**Use Remote Write (historic, backfill, watch, or auto with old data):**
+
+- Historic data import
+- Backfilling gaps
+- Data migration
+- Data older than 5 minutes
+- Watch mode (to preserve file mtime as timestamp)
+
+**Use Auto mode:**
+
+- Mixed current and historic data in one file
+- Unknown timestamp ages
+- General-purpose file import
+
+## When to use which backend
+
+- **Prometheus or VictoriaMetrics:** Set `-prometheus=` to the backend’s Remote Write URL. Use for realtime (via Pushgateway scraped by Prometheus/VM), historic, backfill, auto, and watch.
+- **ClickHouse:** Set `-clickhouse=` in watch mode for analytics/long-term storage. Can be used together with Prometheus or alone (with `-prometheus=` empty).
+
+## Metric design (best practices)
+
+- **Types:** Counter for cumulative values (requests, errors); Gauge for point-in-time (temperature, connections); Histogram for distributions (latency).
+- **Labels:** Meaningful labels; avoid high cardinality (user IDs, raw timestamps); keep combinations reasonable (< 1000 per metric).
+- **Naming:** Descriptive names; units in gauge names (e.g. `_celsius`, `_bytes`); `_total` suffix for counters.
diff --git a/docs/guides/csv-format-flexibility.md b/docs/guides/csv-format-flexibility.md
new file mode 100644
index 0000000..180dc28
--- /dev/null
+++ b/docs/guides/csv-format-flexibility.md
@@ -0,0 +1,52 @@
+# CSV Format Flexibility
+
+Watch mode works with **any tabular CSV**. You do not need a fixed schema; Epimetheus infers metric names and labels from column headers and value types.
+
+## How It Works
+
+- **First row:** Column headers (automatically sanitized for Prometheus label/metric names).
+- **Numeric columns:** Treated as metric values. Each gets a metric name derived from the base metric name and the column header.
+- **String columns:** Treated as labels. Each row’s value becomes the label value for that series.
+- **Metric name:** Set with `-metric-name` (e.g. `web`, `food`, `network`). It is used as a prefix for all numeric columns.
+
+Column names can contain characters that are invalid in Prometheus (e.g. parentheses, spaces). They are sanitized: for example `min(potatoes)` becomes a valid metric suffix like `min_potatoes`.
+
+## Examples
+
+### Web metrics
+
+```csv
+avg(response_time),p99(latency),endpoint,method
+45.2,120.5,/api/users,GET
+52.1,135.8,/api/orders,POST
+```
+
+With `-metric-name=web` this produces series such as:
+
+- `web_avg_response_time{endpoint="/api/users",method="GET"} 45.2`
+- `web_p99_latency{endpoint="/api/users",m