Add documentation and tests for ingestion time range limits

2 files changed, 397 insertions, 0 deletions

diff --git a/f3s/prometheus-pusher/LIMITATIONS.md b/f3s/prometheus-pusher/LIMITATIONS.md
new file mode 100644
index 0000000..e3fe11a
--- /dev/null
+++ b/f3s/prometheus-pusher/LIMITATIONS.md
@@ -0,0 +1,267 @@
+# Prometheus Ingestion Limitations
+
+## Time Range Limits
+
+### ✅ What Works (Tested)
+
+| Time Range | Status | Method |
+|------------|--------|--------|
+| Current (now) | ✅ Works | Pushgateway |
+| 1 hour old | ✅ Works | Remote Write |
+| 1 day old | ✅ Works | Remote Write |
+| 1 week old | ✅ Works | Remote Write |
+| 1 month old | ✅ Works | Remote Write |
+
+### ⚠️ Potential Issues
+
+#### 1. **Future Data** (timestamps in the future)
+
+**Limit**: Prometheus rejects samples too far in the future
+
+```bash
+# Default: ~5 minutes into the future is allowed
+# Controlled by: --storage.tsdb.allow-out-of-order-time-window
+```
+
+**Example that might fail**:
+```csv
+# 1 hour in the future - WILL BE REJECTED
+app_requests_total,instance=test,100,TIMESTAMP_1H_FUTURE
+```
+
+**Error**: `sample is too far in the future`
+
+#### 2. **Very Old Data** (months/years old)
+
+**Limits depend on**:
+- Prometheus retention period
+- TSDB block structure
+- `--storage.tsdb.min-block-duration` setting
+
+**Typical limits**:
+- **Few months old**: Usually works
+- **6+ months old**: May be rejected
+- **Years old**: Likely rejected unless using promtool
+
+**Example that might fail**:
+```csv
+# 6 months old - MIGHT BE REJECTED
+app_requests_total,instance=test,100,TIMESTAMP_6M_AGO
+
+# 1 year old - LIKELY REJECTED
+app_requests_total,instance=test,100,TIMESTAMP_1Y_AGO
+```
+
+**Error**: `sample is too old`
+
+#### 3. **Out-of-Order Samples** (for existing time series)
+
+**Problem**: If a time series already has recent data, you can't insert older data
+
+**Example**:
+```bash
+# Step 1: Push current data
+echo "app_requests_total,instance=test,100,$NOW" | ./prometheus-pusher -mode=auto
+
+# Step 2: Try to push older data for SAME time series - WILL BE REJECTED
+echo "app_requests_total,instance=test,95,$ONE_HOUR_AGO" | ./prometheus-pusher -mode=auto
+```
+
+**Error**: `out of order sample`
+
+**Workaround**: Use different labels (different time series)
+
+#### 4. **Pushgateway Timestamp Limitations**
+
+**Problem**: Pushgateway NEVER preserves timestamps
+
+- All data uses "now" when Prometheus scrapes
+- Cannot backfill old data via Pushgateway
+- Only suitable for current/recent data
+
+**Example**:
+```csv
+# Even with old timestamp, Pushgateway uses "now"
+app_requests_total,instance=current,100,TIMESTAMP_1D_AGO
+# ↑ This timestamp is IGNORED by Pushgateway
+```
+
+## Testing Edge Cases
+
+Let me create a test for various edge cases:
+
+```bash
+#!/bin/bash
+# test-limits.sh
+
+NOW=$(date +%s)000
+
+# Test cases
+cat > test-limits.csv << EOF
+# Edge case tests
+
+# 1. CURRENT - should work
+app_test_current,instance=test1,100,$NOW
+
+# 2. 5 minutes in future - might work
+app_test_future_5m,instance=test2,100,$((NOW + 300000))
+
+# 3. 1 hour in future - will likely be rejected
+app_test_future_1h,instance=test3,100,$((NOW + 3600000))
+
+# 4. 2 months old - might work
+app_test_2m_old,instance=test4,100,$((NOW - 5184000000))
+
+# 5. 6 months old - might be rejected
+app_test_6m_old,instance=test5,100,$((NOW - 15552000000))
+
+# 6. 1 year old - likely rejected
+app_test_1y_old,instance=test6,100,$((NOW - 31536000000))
+
+# 7. 2 years old - very likely rejected
+app_test_2y_old,instance=test7,100,$((NOW - 63072000000))
+EOF
+
+echo "Testing edge cases..."
+./prometheus-pusher -mode=auto -file=test-limits.csv
+```
+
+## Prometheus Configuration Limits
+
+### Default Settings
+
+```yaml
+# Prometheus default limits
+--storage.tsdb.retention.time=15d       # Data older than 15 days is deleted
+--storage.tsdb.min-block-duration=2h    # Minimum block size
+--web.enable-remote-write-receiver      # Must be enabled for historic data
+```
+
+### What These Mean for Ingestion
+
+1. **Retention Time** (`--storage.tsdb.retention.time`)
+   - Default: 15 days
+   - Can't ingest data older than retention period
+   - Check your Prometheus config
+
+2. **Min Block Duration** (`--storage.tsdb.min-block-duration`)
+   - Affects how old data can be written
+   - Default: 2 hours
+   - Older data needs to align with block boundaries
+
+3. **Out-of-Order Time Window**
+   - Default: disabled
+   - Can be enabled with `--enable-feature=out-of-order-ingestion`
+   - Allows writing old data to existing series
+
+## Checking Your Limits
+
+```bash
+# Check Prometheus retention
+kubectl get prometheus -n monitoring prometheus-kube-prometheus-prometheus \
+  -o jsonpath='{.spec.retention}'
+
+# Check Prometheus logs for limits
+kubectl logs -n monitoring prometheus-prometheus-kube-prometheus-prometheus-0 \
+  | grep -i "retention\|block\|sample.*old\|sample.*future"
+```
+
+## Summary Table
+
+| Time Range | Ingestion | Notes |
+|------------|-----------|-------|
+| 1 hour future | ❌ Rejected | "Too far in future" |
+| 5 min future | ⚠️ Maybe | Depends on config |
+| Current | ✅ Works | Both methods |
+| 1 hour old | ✅ Works | Remote Write |
+| 1 day old | ✅ Works | Remote Write |
+| 1 week old | ✅ Works | Remote Write |
+| 1 month old | ✅ Works | Remote Write |
+| 2 months old | ⚠️ Maybe | Depends on retention |
+| 6 months old | ⚠️ Maybe | Likely rejected |
+| 1 year old | ❌ Rejected | Too old |
+| 2+ years old | ❌ Rejected | Way too old |
+
+## Solutions for Very Old Data
+
+If you need to ingest data older than a few months:
+
+### Option 1: Use promtool (Recommended for very old data)
+
+```bash
+# 1. Export data in OpenMetrics format
+cat > old-metrics.txt << EOF
+# HELP app_requests_total Total requests
+# TYPE app_requests_total counter
+app_requests_total{instance="old"} 100 TIMESTAMP_1Y_AGO
+EOF
+
+# 2. Create TSDB blocks directly
+promtool tsdb create-blocks-from openmetrics old-metrics.txt /prometheus/data
+
+# 3. Restart Prometheus to load new blocks
+kubectl rollout restart statefulset/prometheus-prometheus-kube-prometheus-prometheus -n monitoring
+```
+
+### Option 2: Adjust Prometheus Retention
+
+```yaml
+# Increase retention to accept older data
+prometheus:
+  prometheusSpec:
+    retention: 90d  # Keep data for 90 days
+    retentionSize: 50GB
+```
+
+### Option 3: Enable Out-of-Order Ingestion
+
+```yaml
+# Allow out-of-order samples
+prometheus:
+  prometheusSpec:
+    enableFeatures:
+      - out-of-order-ingestion
+    additionalArgs:
+      - --storage.tsdb.out-of-order-time-window=30d
+```
+
+## Best Practices
+
+1. ✅ **Current to 1 month**: Use prometheus-pusher auto mode
+2. ⚠️ **1-3 months old**: Test first, may need config changes
+3. ❌ **6+ months old**: Use promtool instead
+4. ❌ **Years old**: Definitely use promtool
+
+## Testing Your Limits
+
+To find your exact limits:
+
+```bash
+# Generate test data for various ages
+./generate-test-data.sh
+
+# Try importing and watch for errors
+./prometheus-pusher -mode=auto -file=test-limits.csv 2>&1 | tee import.log
+
+# Check what failed
+grep -i "error\|rejected\|failed" import.log
+```
+
+## Error Messages Guide
+
+| Error Message | Meaning | Solution |
+|---------------|---------|----------|
+| `sample is too old` | Beyond retention | Use promtool or increase retention |
+| `sample is too far in the future` | Timestamp in future | Check your clock/timestamps |
+| `out of order sample` | Older than existing data | Use different labels or enable OOO |
+| `remote write receiver not enabled` | Feature not enabled | Enable --web.enable-remote-write-receiver |
+| `sample timestamp out of order` | Wrong order in batch | Sort by timestamp |
+
+## Conclusion
+
+**Practical Limits for prometheus-pusher**:
+- ✅ **Safe range**: Current to 1 month old
+- ⚠️ **Test first**: 1-3 months old
+- ❌ **Use promtool**: 3+ months old
+
+The 1 month limit we tested (and works!) is a safe, practical upper bound for most use cases.
diff --git a/f3s/prometheus-pusher/test-edge-cases.sh b/f3s/prometheus-pusher/test-edge-cases.sh
new file mode 100755
index 0000000..07477f0
--- /dev/null
+++ b/f3s/prometheus-pusher/test-edge-cases.sh
@@ -0,0 +1,130 @@
+#!/bin/bash
+
+# Test edge cases for Prometheus ingestion limits
+
+set +e  # Don't exit on errors - we expect some to fail
+
+NOW=$(date +%s)000
+
+echo "=================================================="
+echo "Testing Prometheus Ingestion Edge Cases"
+echo "=================================================="
+echo ""
+echo "Current time: $(date -d @$((NOW/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo ""
+
+# Generate test data for various edge cases
+cat > test-edge-cases.csv << EOF
+# Edge case tests for Prometheus ingestion
+
+# RECENT/CURRENT - Should all work
+app_edge_now,instance=now,100,$NOW
+app_edge_1min_ago,instance=1min_ago,100,$((NOW - 60000))
+app_edge_5min_ago,instance=5min_ago,100,$((NOW - 300000))
+
+# FUTURE - Will likely be rejected
+app_edge_1min_future,instance=1min_future,100,$((NOW + 60000))
+app_edge_10min_future,instance=10min_future,100,$((NOW + 600000))
+app_edge_1h_future,instance=1h_future,100,$((NOW + 3600000))
+
+# PAST - Testing various ages
+app_edge_1h_old,instance=1h_old,100,$((NOW - 3600000))
+app_edge_1d_old,instance=1d_old,100,$((NOW - 86400000))
+app_edge_1w_old,instance=1w_old,100,$((NOW - 604800000))
+app_edge_1m_old,instance=1m_old,100,$((NOW - 2592000000))
+app_edge_2m_old,instance=2m_old,100,$((NOW - 5184000000))
+app_edge_3m_old,instance=3m_old,100,$((NOW - 7776000000))
+app_edge_6m_old,instance=6m_old,100,$((NOW - 15552000000))
+app_edge_1y_old,instance=1y_old,100,$((NOW - 31536000000))
+app_edge_2y_old,instance=2y_old,100,$((NOW - 63072000000))
+EOF
+
+echo "Generated test data with following timestamps:"
+echo "  Now:        $(date -d @$((NOW/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  1min future: $(date -d @$(((NOW + 60000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  1h future:  $(date -d @$(((NOW + 3600000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  1h ago:     $(date -d @$(((NOW - 3600000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  1d ago:     $(date -d @$(((NOW - 86400000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  1w ago:     $(date -d @$(((NOW - 604800000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  1m ago:     $(date -d @$(((NOW - 2592000000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  2m ago:     $(date -d @$(((NOW - 5184000000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  3m ago:     $(date -d @$(((NOW - 7776000000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  6m ago:     $(date -d @$(((NOW - 15552000000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  1y ago:     $(date -d @$(((NOW - 31536000000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo "  2y ago:     $(date -d @$(((NOW - 63072000000)/1000)) '+%Y-%m-%d %H:%M:%S')"
+echo ""
+
+echo "=================================================="
+echo "IMPORTANT: Port-forward Prometheus before running this test:"
+echo "  kubectl port-forward -n monitoring svc/prometheus-kube-prometheus-prometheus 9090:9090 &"
+echo "=================================================="
+echo ""
+
+read -p "Press Enter to start test (or Ctrl+C to cancel)..."
+
+echo ""
+echo "Running ingestion test..."
+echo ""
+
+./prometheus-pusher \
+  -mode=auto \
+  -file=test-edge-cases.csv \
+  -prometheus=http://localhost:9090/api/v1/write \
+  2>&1 | tee test-edge-cases.log
+
+echo ""
+echo "=================================================="
+echo "Test Results Summary"
+echo "=================================================="
+echo ""
+
+# Analyze results
+if grep -q "Successfully ingested" test-edge-cases.log; then
+    echo "✅ Some samples were successfully ingested"
+    SUCCESS=$(grep -o "Successfully ingested [0-9]* historic samples" test-edge-cases.log | grep -o "[0-9]*")
+    echo "   Success count: $SUCCESS samples"
+else
+    echo "❌ No samples were successfully ingested"
+fi
+
+echo ""
+
+if grep -qi "error\|failed\|rejected" test-edge-cases.log; then
+    echo "❌ Some samples were rejected:"
+    grep -i "error\|failed\|rejected" test-edge-cases.log | head -10
+else
+    echo "✅ No errors detected"
+fi
+
+echo ""
+echo "Full log saved to: test-edge-cases.log"
+echo ""
+
+# Check what data made it into Prometheus
+echo "=================================================="
+echo "Querying Prometheus for Successfully Imported Data"
+echo "=================================================="
+echo ""
+
+sleep 2  # Give Prometheus time to process
+
+for age in "now" "1min_ago" "5min_ago" "1min_future" "10min_future" "1h_future" \
+           "1h_old" "1d_old" "1w_old" "1m_old" "2m_old" "3m_old" "6m_old" "1y_old" "2y_old"; do
+    result=$(curl -s "http://localhost:9090/api/v1/query?query=app_edge_$age" | \
+             python3 -c "import sys, json; d=json.load(sys.stdin); print('✅ Found' if d['data']['result'] else '❌ Not found')" 2>/dev/null || echo "⚠️  Query failed")
+    printf "  %-15s : %s\n" "$age" "$result"
+done
+
+echo ""
+echo "=================================================="
+echo "Conclusion"
+echo "=================================================="
+echo ""
+echo "Check the results above to see which time ranges work."
+echo "Generally:"
+echo "  ✅ Current to 1 month:  Should work"
+echo "  ⚠️  2-3 months:         Depends on retention settings"
+echo "  ❌ 6+ months:          Likely rejected"
+echo "  ❌ Years old:          Definitely rejected"
+echo "  ❌ Future timestamps:  Rejected (except maybe 1-2 min)"
+echo ""