Consolidate all documentation into single comprehensive README.md

- Merged content from 10 separate .md files into README.md
- Removed: ANSWER.md, AUTO-MODE.md, DASHBOARD.md, HISTORIC.md, LIMITATIONS.md,
  QUERY_EXAMPLES.md, QUICK-START.md, SETUP-COMPLETE.md, SUMMARY.md, USAGE.md
- README.md now includes:
  * Quick start guide
  * All operating modes (realtime, historic, backfill, auto)
  * Data formats (CSV, JSON)
  * Test metrics documentation
  * Grafana dashboard setup
  * Example queries and curl commands
  * Time range limitations
  * Troubleshooting guide
  * Architecture diagram
  * Best practices

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

11 files changed, 504 insertions, 2387 deletions

diff --git a/f3s/prometheus-pusher/ANSWER.md b/f3s/prometheus-pusher/ANSWER.md
deleted file mode 100644
index a6339cb..0000000
--- a/f3s/prometheus-pusher/ANSWER.md
+++ /dev/null
@@ -1,201 +0,0 @@
-# Can the Tool Import All These Time Ranges?
-
-## Question
-Can you import current data, data 1 hour old, data 1 day old, data 1 week old, and data 1 month old with the tool?
-
-## Answer: YES! ✅
-
-The tool can now import data from **ALL** these time ranges automatically!
-
-## How It Works
-
-### Before (Manual Mode)
-You had to:
-1. Calculate how old your data is
-2. Choose the right mode manually
-3. Specify `-hours-ago` for each time range
-4. Run the tool multiple times for different ages
-
-### After (AUTO Mode) 🤖
-You just:
-1. Provide data with timestamps
-2. Run: `./prometheus-pusher -mode=auto -file=yourdata.csv`
-3. **Done!** The tool automatically detects ages and routes correctly
-
-## Demonstration
-
-I've created test data for you with **all 5 time ranges**:
-
-```bash
-cd /home/paul/git/conf/f3s/prometheus-pusher
-
-# View the generated test data
-cat test-all-ages.csv
-```
-
-**Contents** (generated with actual timestamps):
-```csv
-# CURRENT data (< 5min old)
-app_requests_total,instance=current;env=prod,100,1767125148000
-app_temperature_celsius,instance=current;zone=us-east,22.5,1767125148000
-app_active_connections,instance=current;env=prod,50,1767125148000
-
-# 1 HOUR OLD data
-app_requests_total,instance=1h_ago;env=prod,95,1767121548000
-app_active_connections,instance=1h_ago;env=prod,45,1767121548000
-app_temperature_celsius,instance=1h_ago;zone=us-east,21.8,1767121548000
-
-# 1 DAY OLD data
-app_requests_total,instance=1d_ago;env=prod,150,1767038748000
-app_temperature_celsius,instance=1d_ago;zone=eu-west,18.3,1767038748000
-app_active_connections,instance=1d_ago;env=prod,60,1767038748000
-
-# 1 WEEK OLD data
-app_requests_total,instance=1w_ago;env=prod,200,1766520348000
-app_jobs_processed_total,instance=1w_ago;env=prod;job_type=email;status=success,75,1766520348000
-app_temperature_celsius,instance=1w_ago;zone=asia,25.2,1766520348000
-
-# 1 MONTH OLD data
-app_requests_total,instance=1m_ago;env=prod,180,1764533148000
-app_active_connections,instance=1m_ago;env=prod,30,1764533148000
-app_temperature_celsius,instance=1m_ago;zone=africa,28.7,1764533148000
-```
-
-## Test It Yourself
-
-Once Prometheus is configured with remote write receiver:
-
-```bash
-# 1. Port-forward services
-kubectl port-forward -n monitoring svc/pushgateway 9091:9091 &
-kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &
-
-# 2. Import ALL time ranges in one command!
-./prometheus-pusher \
-  -mode=auto \
-  -file=test-all-ages.csv \
-  -pushgateway=http://localhost:9091 \
-  -prometheus=http://localhost:9090/api/v1/write
-```
-
-**Expected Output**:
-```
-🤖 AUTO mode: Automatically detecting timestamp age and choosing ingestion method
-
-📁 Reading metrics from: test-all-ages.csv (format: csv)
-📊 Auto-ingest summary:
-  Total samples: 15
-  Realtime samples (< 5min old): 3
-  Historic samples (> 5min old): 12
-
-🔄 Ingesting 3 REALTIME samples via Pushgateway...
-  Note: Pushgateway ingestion uses current timestamp
-✅ Successfully ingested 3 realtime samples
-
-⏰ Ingesting 12 HISTORIC samples via Remote Write...
-  [1/12] app_requests_total (age: 1.0 hours)
-  [2/12] app_active_connections (age: 1.0 hours)
-  [3/12] app_temperature_celsius (age: 1.0 hours)
-  [4/12] app_requests_total (age: 1.0 days)
-  [5/12] app_temperature_celsius (age: 1.0 days)
-  [6/12] app_active_connections (age: 1.0 days)
-  [7/12] app_requests_total (age: 7.0 days)
-  [8/12] app_jobs_processed_total (age: 7.0 days)
-  [9/12] app_temperature_celsius (age: 7.0 days)
-  [10/12] app_requests_total (age: 30.0 days)
-  [11/12] app_active_connections (age: 30.0 days)
-  [12/12] app_temperature_celsius (age: 30.0 days)
-✅ Successfully ingested 12 historic samples
-
-🎉 Auto-ingest complete!
-```
-
-## Verification
-
-After import, query the data in Prometheus:
-
-```bash
-# Query current data
-curl 'http://localhost:9090/api/v1/query?query={instance="current"}'
-
-# Query 1 hour old data
-curl 'http://localhost:9090/api/v1/query?query={instance="1h_ago"}'
-
-# Query 1 day old data
-curl 'http://localhost:9090/api/v1/query?query={instance="1d_ago"}'
-
-# Query 1 week old data
-curl 'http://localhost:9090/api/v1/query?query={instance="1w_ago"}'
-
-# Query 1 month old data
-curl 'http://localhost:9090/api/v1/query?query={instance="1m_ago"}'
-
-# See all imported data
-curl 'http://localhost:9090/api/v1/query?query={env="prod"}'
-```
-
-## Summary Table
-
-| Time Range | Status | Ingestion Method | Notes |
-|------------|--------|------------------|-------|
-| **Current** (now) | ✅ YES | Pushgateway | Uses "now" timestamp |
-| **1 hour old** | ✅ YES | Remote Write | Preserves original timestamp |
-| **1 day old** | ✅ YES | Remote Write | Preserves original timestamp |
-| **1 week old** | ✅ YES | Remote Write | Preserves original timestamp |
-| **1 month old** | ✅ YES | Remote Write | Preserves original timestamp |
-
-## Key Features
-
-✅ **Automatic Detection** - Tool detects age, you don't calculate
-✅ **Smart Routing** - Chooses Pushgateway or Remote Write automatically
-✅ **Clear Logging** - See exactly what's happening for each metric
-✅ **Batch Import** - Import all ages in one go
-✅ **Format Support** - CSV and JSON formats
-✅ **No Manual Work** - Just provide timestamps, tool handles the rest
-
-## Pending Setup
-
-To use the historic data features (1h, 1d, 1w, 1m old):
-
-1. **Enable Remote Write Receiver** in Prometheus:
-   ```bash
-   cd /home/paul/git/conf/f3s/prometheus
-   helm upgrade prometheus prometheus-community/kube-prometheus-stack \
-     -n monitoring -f persistence-values.yaml
-   ```
-
-2. **Wait for Prometheus** to restart with the new flag enabled
-
-3. **Run the test** as shown above
-
-## Documentation
-
-- **AUTO-MODE.md** - Complete guide to auto mode
-- **HISTORIC.md** - Guide to historic data ingestion
-- **SETUP-COMPLETE.md** - Setup instructions
-- **test-all-ages.csv** - Ready-to-use test data
-
-## Conclusion
-
-**YES**, the tool can import data from:
-- ✅ Current time
-- ✅ 1 hour ago
-- ✅ 1 day ago
-- ✅ 1 week ago
-- ✅ 1 month ago
-
-And it does this **automatically** - you don't need to think about it! 🎉
-
-Just run:
-```bash
-./prometheus-pusher -mode=auto -file=your-data.csv
-```
-
-The tool will:
-1. Read your timestamps
-2. Calculate age for each metric
-3. Route to appropriate ingestion method
-4. Log what it's doing
-5. Complete the import
-
-**All changes committed and pushed to git!**
diff --git a/f3s/prometheus-pusher/AUTO-MODE.md b/f3s/prometheus-pusher/AUTO-MODE.md
deleted file mode 100644
index 03db0b5..0000000
--- a/f3s/prometheus-pusher/AUTO-MODE.md
+++ /dev/null
@@ -1,297 +0,0 @@
-# Auto Mode - Automatic Timestamp Detection
-
-## Overview
-
-The **AUTO mode** is a smart ingestion mode that:
-1. **Reads metrics** with timestamps from a file or stdin
-2. **Automatically detects** how old each metric is
-3. **Chooses the right ingestion method**:
-   - Realtime data (< 5 minutes old) → Pushgateway
-   - Historic data (> 5 minutes old) → Remote Write API
-4. **Logs what it's doing** so you can see which method is used
-
-**No manual timestamp calculation needed!** Just provide data with timestamps.
-
-## Why Use Auto Mode?
-
-### Problem
-Previously, you had to:
-- Manually calculate how old your data is
-- Choose between `--mode=realtime` or `--mode=historic`
-- Specify `-hours-ago` for each datapoint
-
-### Solution
-Now you can:
-- Provide data with timestamps in any format (CSV or JSON)
-- The tool automatically detects age and chooses ingestion method
-- Batch import mixed data (some current, some old)
-
-## Usage
-
-### From File
-
-```bash
-# CSV format
-./prometheus-pusher -mode=auto -file=metrics.csv -format=csv
-
-# JSON format
-./prometheus-pusher -mode=auto -file=metrics.json -format=json
-```
-
-### From Stdin
-
-```bash
-# Pipe CSV data
-cat metrics.csv | ./prometheus-pusher -mode=auto -format=csv
-
-# Interactive input
-./prometheus-pusher -mode=auto -format=csv
-# (then paste data and press Ctrl+D)
-```
-
-## Input Formats
-
-### CSV Format
-
-```
-# Format: metric_name,labels,value,timestamp_ms
-# Labels: key1=value1;key2=value2
-
-app_requests_total,instance=web1;env=prod,100,1767125148000
-app_temperature_celsius,instance=web2;zone=us,22.5,1767038748000
-```
-
-**Fields**:
-1. `metric_name`: Prometheus metric name
-2. `labels`: Semicolon-separated label pairs (optional)
-3. `value`: Metric value (float)
-4. `timestamp_ms`: Unix timestamp in milliseconds (optional, defaults to now)
-
-**Example**:
-```csv
-# Current data (no timestamp = uses now)
-app_requests_total,instance=web1,100,
-
-# 1 hour ago
-app_requests_total,instance=web2,95,1767121548000
-
-# 1 day ago
-app_requests_total,instance=web3,150,1767038748000
-```
-
-### JSON Format
-
-```json
-[
-  {
-    "metric": "app_requests_total",
-    "labels": {"instance": "web1", "env": "prod"},
-    "value": 100,
-    "timestamp_ms": 1767125148000
-  },
-  {
-    "metric": "app_temperature_celsius",
-    "labels": {"instance": "web2", "zone": "us"},
-    "value": 22.5,
-    "timestamp_ms": 1767038748000
-  }
-]
-```
-
-**Fields**:
-- `metric`: Metric name (required)
-- `labels`: Object with label key-value pairs (optional)
-- `value`: Metric value (required)
-- `timestamp_ms`: Unix timestamp in milliseconds (optional)
-
-## Generating Test Data
-
-Use the provided script to generate test data for all time ranges:
-
-```bash
-./generate-test-data.sh
-```
-
-This creates `test-all-ages.csv` with:
-- Current data (< 5 min old)
-- 1 hour old data
-- 1 day old data
-- 1 week old data
-- 1 month old data
-
-## Example: Import All Time Ranges
-
-```bash
-# 1. Generate test data
-./generate-test-data.sh
-
-# 2. Port-forward Prometheus (for historic data)
-kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &
-
-# 3. Port-forward Pushgateway (for current data)
-kubectl port-forward -n monitoring svc/pushgateway 9091:9091 &
-
-# 4. Auto-import all data
-./prometheus-pusher \
-  -mode=auto \
-  -file=test-all-ages.csv \
-  -format=csv \
-  -pushgateway=http://localhost:9091 \
-  -prometheus=http://localhost:9090/api/v1/write
-```
-
-**Expected Output**:
-```
-🤖 AUTO mode: Automatically detecting timestamp age and choosing ingestion method
-
-📁 Reading metrics from: test-all-ages.csv (format: csv)
-📊 Auto-ingest summary:
-  Total samples: 15
-  Realtime samples (< 5min old): 3
-  Historic samples (> 5min old): 12
-
-🔄 Ingesting 3 REALTIME samples via Pushgateway...
-  Note: Pushgateway ingestion uses current timestamp
-✅ Successfully ingested 3 realtime samples
-
-⏰ Ingesting 12 HISTORIC samples via Remote Write...
-  [1/12] app_requests_total (age: 1.0 hours)
-  [2/12] app_active_connections (age: 1.0 hours)
-  [3/12] app_temperature_celsius (age: 1.0 hours)
-  [4/12] app_requests_total (age: 1.0 days)
-  [5/12] app_temperature_celsius (age: 1.0 days)
-  [6/12] app_active_connections (age: 1.0 days)
-  [7/12] app_requests_total (age: 7.0 days)
-  [8/12] app_jobs_processed_total (age: 7.0 days)
-  [9/12] app_temperature_celsius (age: 7.0 days)
-  [10/12] app_requests_total (age: 30.0 days)
-  [11/12] app_active_connections (age: 30.0 days)
-  [12/12] app_temperature_celsius (age: 30.0 days)
-✅ Successfully ingested 12 historic samples
-
-🎉 Auto-ingest complete!
-```
-
-## Detection Logic
-
-The tool uses a **5-minute threshold**:
-
-| Data Age | Ingestion Method | Reason |
-|----------|------------------|---------|
-| < 5 minutes | Pushgateway (realtime) | Recent enough to use "now" timestamp |
-| ≥ 5 minutes | Remote Write (historic) | Too old, needs preserved timestamp |
-
-**Why 5 minutes?**
-- Allows for clock skew and processing delays
-- Prometheus scrapes Pushgateway every 15-30s
-- Gives buffer for network delays
-
-## Query Imported Data
-
-After import, query in Prometheus:
-
-```promql
-# View current data (from Pushgateway)
-{instance="current"}
-
-# View 1 hour old data
-{instance="1h_ago"}
-
-# View 1 day old data
-{instance="1d_ago"}
-
-# View 1 week old data
-{instance="1w_ago"}
-
-# View 1 month old data
-{instance="1m_ago"}
-
-# All imported data
-{env="prod"}
-```
-
-## Flags
-
-```
--mode=auto          Enable auto mode
--file=<path>        Input file (CSV or JSON)
--format=<fmt>       Format: csv or json (default: csv)
--pushgateway=<url>  Pushgateway URL (default: http://localhost:9091)
--prometheus=<url>   Prometheus remote write URL (default: http://localhost:9090/api/v1/write)
--job=<name>         Job name for metrics (default: example_metrics_pusher)
-```
-
-## Supported Time Ranges
-
-✅ **Current data** (< 5 min): Works perfectly
-✅ **1 hour old**: Works via Remote Write
-✅ **1 day old**: Works via Remote Write
-✅ **1 week old**: Works via Remote Write
-✅ **1 month old**: Works via Remote Write
-⚠️ **Very old data** (months/years): May hit Prometheus limits
-
-For very old data (> few months), consider:
-- Using `promtool tsdb create-blocks-from` instead
-- Increasing Prometheus retention settings
-- Using long-term storage solutions
-
-## Benefits
-
-1. **No timestamp math** - Tool calculates age automatically
-2. **Mixed data** - Import both current and historic data in one go
-3. **Visual feedback** - See exactly which ingestion method is used
-4. **Batch import** - Process large CSV/JSON files easily
-5. **Error handling** - Clear messages if ingestion fails
-
-## Comparison with Other Modes
-
-| Mode | Use Case | Timestamp Handling |
-|------|----------|-------------------|
-| `realtime` | Live monitoring | Always uses "now" |
-| `historic` | Single old datapoint | Manually specify `-hours-ago` |
-| `backfill` | Range of datapoints | Manually specify range |
-| `auto` | **Any mix of data** | **Automatic detection** |
-
-## Advanced Example: Import from Multiple Sources
-
-```bash
-# Generate various test data
-./generate-test-data.sh
-
-# Import yesterday's backup
-./prometheus-pusher -mode=auto -file=backup_yesterday.csv
-
-# Import last week's logs
-./prometheus-pusher -mode=auto -file=logs_lastweek.json -format=json
-
-# Import current metrics
-./prometheus-pusher -mode=auto -file=current_metrics.csv
-```
-
-All data is automatically routed to the correct ingestion method!
-
-## Troubleshooting
-
-### "No valid samples found"
-- Check CSV/JSON format
-- Ensure timestamps are in milliseconds
-- Check for syntax errors in labels
-
-### "Remote write receiver not enabled"
-- Ensure Prometheus has `--web.enable-remote-write-receiver` flag
-- Check prometheus/persistence-values.yaml configuration
-
-### "Pushgateway connection refused"
-- Verify port-forward: `kubectl port-forward -n monitoring svc/pushgateway 9091:9091`
-- Check Pushgateway is running: `kubectl get pods -n monitoring | grep pushgateway`
-
-## Summary
-
-Auto mode makes importing data effortless:
-- 📥 Read data from file or stdin
-- 🔍 Automatically detect timestamp age
-- 🎯 Choose optimal ingestion method
-- 📊 Clear logging of what's happening
-- ✅ Support for all time ranges (current → 1 month old)
-
-No more manual timestamp calculations - just provide your data!
diff --git a/f3s/prometheus-pusher/DASHBOARD.md b/f3s/prometheus-pusher/DASHBOARD.md
deleted file mode 100644
index 47888ff..0000000
--- a/f3s/prometheus-pusher/DASHBOARD.md
+++ /dev/null
@@ -1,151 +0,0 @@
-# Grafana Dashboard for Prometheus Pusher Test Metrics
-
-This document describes the Grafana dashboard for visualizing metrics generated by prometheus-pusher.
-
-## Dashboard Overview
-
-The dashboard displays all test metrics with the `prometheus_pusher_test_` prefix, making it clear they are generated by the prometheus-pusher testing/demo functionality.
-
-## Metrics Displayed
-
-### 1. Request Rate
-- **Type**: Line graph
-- **Metric**: `rate(prometheus_pusher_test_requests_total[5m])`
-- **Description**: Shows the rate of requests per second over the last 5 minutes
-- **Use**: Monitor request throughput
-
-### 2. Total Requests
-- **Type**: Stat panel
-- **Metric**: `prometheus_pusher_test_requests_total`
-- **Description**: Counter showing total number of requests processed
-- **Display**: Large number with area graph background
-
-### 3. Active Connections
-- **Type**: Gauge
-- **Metric**: `prometheus_pusher_test_active_connections`
-- **Description**: Current number of active connections (0-100)
-- **Thresholds**:
-  - Green: 0-50
-  - Yellow: 50-80
-  - Red: 80-100
-
-### 4. Temperature
-- **Type**: Gauge
-- **Metric**: `prometheus_pusher_test_temperature_celsius`
-- **Description**: Current temperature in Celsius (0-50°C)
-- **Thresholds**:
-  - Blue: 0-20°C
-  - Green: 20-30°C
-  - Yellow: 30-35°C
-  - Red: 35-50°C
-
-### 5. Request Duration Histogram
-- **Type**: Line graph
-- **Metrics**:
-  - `histogram_quantile(0.50, rate(prometheus_pusher_test_request_duration_seconds_bucket[5m]))` - p50
-  - `histogram_quantile(0.90, rate(prometheus_pusher_test_request_duration_seconds_bucket[5m]))` - p90
-  - `histogram_quantile(0.99, rate(prometheus_pusher_test_request_duration_seconds_bucket[5m]))` - p99
-- **Description**: Shows request duration percentiles over time
-- **Use**: Identify latency trends and outliers
-
-### 6. Average Request Duration
-- **Type**: Stat panel
-- **Metric**: `rate(prometheus_pusher_test_request_duration_seconds_sum[5m]) / rate(prometheus_pusher_test_request_duration_seconds_count[5m])`
-- **Description**: Average request duration in seconds
-- **Display**: Number with area graph, 3 decimal places
-
-### 7. Jobs Processed by Type
-- **Type**: Bar gauge
-- **Metric**: `sum by (job_type) (prometheus_pusher_test_jobs_processed_total)`
-- **Description**: Total jobs processed grouped by job type (email, report, backup)
-- **Display**: Horizontal gradient bars
-
-### 8. Jobs Status Breakdown
-- **Type**: Table
-- **Metric**: `prometheus_pusher_test_jobs_processed_total`
-- **Description**: Detailed breakdown showing job type, status, and count
-- **Columns**: Job Type, Status, Count
-
-## Deployment
-
-### Prerequisites
-- Grafana instance running and accessible
-- Prometheus as a data source in Grafana
-- Metrics being pushed to Prometheus via prometheus-pusher
-
-### Deploy via Script
-```bash
-# Port-forward to Grafana (if running in Kubernetes)
-kubectl port-forward -n monitoring svc/prometheus-grafana 3000:80
-
-# Deploy dashboard (in another terminal)
-./deploy-dashboard.sh
-```
-
-### Deploy Manually
-1. Open Grafana UI
-2. Go to Dashboards → Import
-3. Upload `grafana-dashboard.json`
-4. Select Prometheus data source
-5. Click Import
-
-### Custom Deployment
-```bash
-# With custom Grafana URL and credentials
-GRAFANA_URL="http://grafana.example.com" \
-GRAFANA_USER="admin" \
-GRAFANA_PASSWORD="secret" \
-./deploy-dashboard.sh
-```
-
-## Dashboard Features
-
-- **Auto-refresh**: Updates every 10 seconds
-- **Time range**: Last 15 minutes by default
-- **Refresh intervals**: 5s, 10s, 30s, 1m, 5m
-- **Shared tooltips**: Hover over graphs to see all series values
-- **Dark theme**: Optimized for dark mode viewing
-
-## Metric Naming Convention
-
-All metrics use the `prometheus_pusher_test_` prefix to:
-- Clearly identify them as test/demo metrics
-- Distinguish from production application metrics
-- Make cleanup and filtering easier
-- Prevent confusion with real application data
-
-## Testing the Dashboard
-
-1. **Push test metrics**:
-   ```bash
-   ./prometheus-pusher -mode realtime -pushgateway http://localhost:9091
-   ```
-
-2. **Continuous updates** (for live dashboard testing):
-   ```bash
-   ./prometheus-pusher -mode realtime -pushgateway http://localhost:9091 -continuous
-   ```
-
-3. **View in Grafana**:
-   - Navigate to the "Prometheus Pusher Test Metrics" dashboard
-   - Watch metrics update in real-time
-   - Interact with time ranges and refresh intervals
-
-## Cleanup
-
-To remove old test metrics from Pushgateway:
-```bash
-# Delete all metrics for the example_metrics_pusher job
-curl -X DELETE http://localhost:9091/metrics/job/example_metrics_pusher
-```
-
-## Customization
-
-The dashboard JSON file can be customized:
-- Adjust panel sizes via `gridPos`
-- Change colors and thresholds
-- Add new panels for additional metrics
-- Modify queries and aggregations
-- Update refresh intervals
-
-After making changes, redeploy using the deployment script.
diff --git a/f3s/prometheus-pusher/HISTORIC.md b/f3s/prometheus-pusher/HISTORIC.md
deleted file mode 100644
index 22c397d..0000000
--- a/f3s/prometheus-pusher/HISTORIC.md
+++ /dev/null
@@ -1,231 +0,0 @@
-# Historic Data Ingestion
-
-This document explains how to ingest historic data into Prometheus using the prometheus-pusher tool.
-
-## Problem
-
-The standard Pushgateway approach has a limitation: it doesn't support custom timestamps. When you push metrics to Pushgateway, Prometheus scrapes them with the current timestamp. This means you cannot backfill historic data (e.g., data from yesterday or last week).
-
-## Solution
-
-Prometheus supports the **Remote Write API** which accepts timestamped samples. By enabling the `remote-write-receiver` feature flag, Prometheus can accept historic data with custom timestamps via HTTP POST.
-
-### Limitations
-
-- **Out-of-order samples**: By default, Prometheus rejects samples that are older than the most recent sample for that time series
-- **Time window**: Prometheus typically accepts data within a certain time window (default: up to 1 hour in the past for new series)
-- **Feature flag required**: The remote write receiver must be enabled with `--enable-feature=remote-write-receiver`
-
-## Setup
-
-### 1. Enable Remote Write Receiver
-
-The Prometheus instance needs to be configured with the remote write receiver feature:
-
-```yaml
-# In prometheus/persistence-values.yaml
-prometheus:
-  prometheusSpec:
-    enableFeatures:
-      - remote-write-receiver
-```
-
-This has been configured and applied to the monitoring namespace Prometheus instance.
-
-### 2. Verify Feature is Enabled
-
-```bash
-kubectl logs -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 | grep "remote-write-receiver"
-```
-
-You should see: `msg="Experimental features enabled" features=[remote-write-receiver]`
-
-## Usage
-
-The `prometheus-pusher` binary supports three modes:
-
-### Mode 1: Realtime (Default)
-
-Push current metrics to Pushgateway (same as before):
-
-```bash
-./prometheus-pusher -mode=realtime -continuous
-```
-
-Options:
-- `-pushgateway`: Pushgateway URL (default: http://localhost:9091)
-- `-job`: Job name (default: example_metrics_pusher)
-- `-continuous`: Keep pushing every 15 seconds
-
-### Mode 2: Historic (Single Datapoint)
-
-Push a single datapoint from X hours ago:
-
-```bash
-# Port-forward Prometheus
-kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &
-
-# Push data from 24 hours ago
-./prometheus-pusher -mode=historic -hours-ago=24
-
-# Push data from 3 hours ago
-./prometheus-pusher -mode=historic -hours-ago=3
-
-# Push data from yesterday (48 hours ago)
-./prometheus-pusher -mode=historic -hours-ago=48
-```
-
-Options:
-- `-prometheus`: Prometheus remote write URL (default: http://localhost:9090/api/v1/write)
-- `-hours-ago`: How many hours in the past (default: 24)
-
-### Mode 3: Backfill (Multiple Datapoints)
-
-Backfill a range of historic data:
-
-```bash
-# Backfill last 48 hours with 1-hour intervals
-./prometheus-pusher -mode=backfill -start-hours=48 -end-hours=0 -interval=1
-
-# Backfill last week with 6-hour intervals
-./prometheus-pusher -mode=backfill -start-hours=168 -end-hours=0 -interval=6
-
-# Backfill specific range (24h ago to 12h ago, every 2 hours)
-./prometheus-pusher -mode=backfill -start-hours=24 -end-hours=12 -interval=2
-```
-
-Options:
-- `-start-hours`: Start time in hours ago (e.g., 48 = 2 days ago)
-- `-end-hours`: End time in hours ago (e.g., 0 = now)
-- `-interval`: Interval between datapoints in hours
-
-## Data Format
-
-Historic data is sent using the Prometheus Remote Write protocol (Protobuf):
-
-1. **Protocol**: HTTP POST with Protobuf payload
-2. **Encoding**: Snappy compression
-3. **Headers**:
-   - Content-Type: application/x-protobuf
-   - Content-Encoding: snappy
-   - X-Prometheus-Remote-Write-Version: 0.1.0
-
-4. **Payload**: TimeSeries with custom timestamps
-
-Example time series:
-```protobuf
-TimeSeries {
-  Labels: [
-    {Name: "__name__", Value: "app_requests_total"},
-    {Name: "instance", Value: "example-app"},
-    {Name: "job", Value: "historic_data"}
-  ],
-  Samples: [
-    {Value: 42, Timestamp: 1735516800000}  // milliseconds since epoch
-  ]
-}
-```
-
-## Example: Backfill Last 24 Hours
-
-```bash
-#!/bin/bash
-
-# 1. Port-forward Prometheus
-kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &
-PF_PID=$!
-sleep 2
-
-# 2. Backfill data for every hour in the last 24 hours
-cd /home/paul/git/conf/f3s/prometheus-pusher
-./prometheus-pusher \
-  -mode=backfill \
-  -prometheus=http://localhost:9090/api/v1/write \
-  -start-hours=24 \
-  -end-hours=0 \
-  -interval=1
-
-# 3. Clean up
-kill $PF_PID
-```
-
-## Querying Historic Data
-
-Once backfilled, the historic data is queryable in Prometheus:
-
-```promql
-# View all historic data
-{job="historic_data"}
-
-# View specific metric from historic data
-app_requests_total{job="historic_data"}
-
-# View data from a specific time range
-app_temperature_celsius{job="historic_data"}[24h]
-
-# Compare realtime vs historic data
-app_requests_total{job="example_metrics_pusher"}  # realtime
-app_requests_total{job="historic_data"}           # historic
-```
-
-## Troubleshooting
-
-### Error: "remote write receiver not enabled"
-
-```
-Error: remote write failed with status 404: remote write receiver not enabled
-```
-
-Solution: Ensure Prometheus has the `remote-write-receiver` feature enabled and has restarted.
-
-### Error: "out of order sample"
-
-```
-Error: sample timestamp out of order
-```
-
-This occurs when trying to insert data older than existing data for the same time series. Solutions:
-1. Use a different job label for historic data (already done: `job="historic_data"`)
-2. Enable out-of-order ingestion in Prometheus (experimental)
-3. Ensure backfill starts from oldest to newest
-
-### Error: "sample too old"
-
-```
-Error: sample is too old
-```
-
-Prometheus has limits on how old data can be. By default:
-- For existing series: can't be older than the oldest block
-- For new series: typically accepts data up to 1 hour old
-
-Solution: For very old data (weeks/months), use `promtool tsdb create-blocks-from` instead.
-
-## Best Practices
-
-1. **Use different job labels**: Historic data uses `job="historic_data"`, realtime uses `job="example_metrics_pusher"`
-2. **Backfill in order**: Always backfill from oldest to newest to avoid out-of-order rejections
-3. **Small batches**: Don't overwhelm Prometheus - the tool includes 100ms delays between datapoints
-4. **Verify first**: Test with a single datapoint before running large backfills
-5. **Monitor errors**: Check Prometheus logs if ingestion fails
-
-## Limitations
-
-- **Very old data**: For data older than a few days, consider using `promtool` for TSDB block creation
-- **High cardinality**: Be careful with label combinations - they create separate time series
-- **Performance**: Large backfills can impact Prometheus performance
-- **Out-of-order**: By default, Prometheus rejects out-of-order samples
-
-## Alternative: Using Promtool
-
-For very large historic datasets, you can use `promtool` to create TSDB blocks:
-
-```bash
-# 1. Generate OpenMetrics format file
-./prometheus-pusher -mode=export -output=metrics.txt
-
-# 2. Create blocks from the file
-promtool tsdb create-blocks-from openmetrics metrics.txt /path/to/prometheus/data
-```
-
-This method bypasses the API and writes directly to the TSDB, but requires filesystem access.
diff --git a/f3s/prometheus-pusher/LIMITATIONS.md b/f3s/prometheus-pusher/LIMITATIONS.md
deleted file mode 100644
index e3fe11a..0000000
--- a/f3s/prometheus-pusher/LIMITATIONS.md
+++ /dev/null
@@ -1,267 +0,0 @@
-# Prometheus Ingestion Limitations
-
-## Time Range Limits
-
-### ✅ What Works (Tested)
-
-| Time Range | Status | Method |
-|------------|--------|--------|
-| Current (now) | ✅ Works | Pushgateway |
-| 1 hour old | ✅ Works | Remote Write |
-| 1 day old | ✅ Works | Remote Write |
-| 1 week old | ✅ Works | Remote Write |
-| 1 month old | ✅ Works | Remote Write |
-
-### ⚠️ Potential Issues
-
-#### 1. **Future Data** (timestamps in the future)
-
-**Limit**: Prometheus rejects samples too far in the future
-
-```bash
-# Default: ~5 minutes into the future is allowed
-# Controlled by: --storage.tsdb.allow-out-of-order-time-window
-```
-
-**Example that might fail**:
-```csv
-# 1 hour in the future - WILL BE REJECTED
-app_requests_total,instance=test,100,TIMESTAMP_1H_FUTURE
-```
-
-**Error**: `sample is too far in the future`
-
-#### 2. **Very Old Data** (months/years old)
-
-**Limits depend on**:
-- Prometheus retention period
-- TSDB block structure
-- `--storage.tsdb.min-block-duration` setting
-
-**Typical limits**:
-- **Few months old**: Usually works
-- **6+ months old**: May be rejected
-- **Years old**: Likely rejected unless using promtool
-
-**Example that might fail**:
-```csv
-# 6 months old - MIGHT BE REJECTED
-app_requests_total,instance=test,100,TIMESTAMP_6M_AGO
-
-# 1 year old - LIKELY REJECTED
-app_requests_total,instance=test,100,TIMESTAMP_1Y_AGO
-```
-
-**Error**: `sample is too old`
-
-#### 3. **Out-of-Order Samples** (for existing time series)
-
-**Problem**: If a time series already has recent data, you can't insert older data
-
-**Example**:
-```bash
-# Step 1: Push current data
-echo "app_requests_total,instance=test,100,$NOW" | ./prometheus-pusher -mode=auto
-
-# Step 2: Try to push older data for SAME time series - WILL BE REJECTED
-echo "app_requests_total,instance=test,95,$ONE_HOUR_AGO" | ./prometheus-pusher -mode=auto
-```
-
-**Error**: `out of order sample`
-
-**Workaround**: Use different labels (different time series)
-
-#### 4. **Pushgateway Timestamp Limitations**
-
-**Problem**: Pushgateway NEVER preserves timestamps
-
-- All data uses "now" when Prometheus scrapes
-- Cannot backfill old data via Pushgateway
-- Only suitable for current/recent data
-
-**Example**:
-```csv
-# Even with old timestamp, Pushgateway uses "now"
-app_requests_total,instance=current,100,TIMESTAMP_1D_AGO
-# ↑ This timestamp is IGNORED by Pushgateway
-```
-
-## Testing Edge Cases
-
-Let me create a test for various edge cases:
-
-```bash
-#!/bin/bash
-# test-limits.sh
-
-NOW=$(date +%s)000
-
-# Test cases
-cat > test-limits.csv << EOF
-# Edge case tests
-
-# 1. CURRENT - should work
-app_test_current,instance=test1,100,$NOW
-
-# 2. 5 minutes in future - might work
-app_test_future_5m,instance=test2,100,$((NOW + 300000))
-
-# 3. 1 hour in future - will likely be rejected
-app_test_future_1h,instance=test3,100,$((NOW + 3600000))
-
-# 4. 2 months old - might work
-app_test_2m_old,instance=test4,100,$((NOW - 5184000000))
-
-# 5. 6 months old - might be rejected
-app_test_6m_old,instance=test5,100,$((NOW - 15552000000))
-
-# 6. 1 year old - likely rejected
-app_test_1y_old,instance=test6,100,$((NOW - 31536000000))
-
-# 7. 2 years old - very likely rejected
-app_test_2y_old,instance=test7,100,$((NOW - 63072000000))
-EOF
-
-echo "Testing edge cases..."
-./prometheus