| Age | Commit message (Collapse) | Author |
|---|
|
This commit configures Prometheus to accept historic data via the Remote
Write API, enabling backfilling of test metrics for development and
troubleshooting purposes.
Changes:
- Enable Remote Write receiver (--web.enable-remote-write-receiver)
- Enable out-of-order ingestion with 30-day window (720h)
- Enable exemplar-storage and otlp-write-receiver features
- Add Epimetheus dashboard ConfigMap for Grafana provisioning
- Remove old prometheus-pusher directory (moved to separate repo)
- Document configuration, use cases, and performance considerations
Configuration allows backfilling data up to 30 days in the past, supporting
tools like Epimetheus for generating synthetic historic metrics.
Performance note: This is optimized for ad-hoc troubleshooting, not
production use. Out-of-order ingestion increases memory usage, TSDB overhead,
and may impact query performance.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
- Merged content from 10 separate .md files into README.md
- Removed: ANSWER.md, AUTO-MODE.md, DASHBOARD.md, HISTORIC.md, LIMITATIONS.md,
QUERY_EXAMPLES.md, QUICK-START.md, SETUP-COMPLETE.md, SUMMARY.md, USAGE.md
- README.md now includes:
* Quick start guide
* All operating modes (realtime, historic, backfill, auto)
* Data formats (CSV, JSON)
* Test metrics documentation
* Grafana dashboard setup
* Example queries and curl commands
* Time range limitations
* Troubleshooting guide
* Architecture diagram
* Best practices
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
Renamed all test metrics with "prometheus_pusher_test_" prefix to clearly
indicate they are generated by the prometheus-pusher testing/demo functionality.
Metric renaming:
- app_requests_total → prometheus_pusher_test_requests_total
- app_active_connections → prometheus_pusher_test_active_connections
- app_temperature_celsius → prometheus_pusher_test_temperature_celsius
- app_request_duration_seconds → prometheus_pusher_test_request_duration_seconds
- app_jobs_processed_total → prometheus_pusher_test_jobs_processed_total
Grafana Dashboard:
- Created comprehensive dashboard with 8 panels
- Request rate and total requests visualization
- Active connections gauge (0-100 with thresholds)
- Temperature gauge (0-50°C with thresholds)
- Request duration percentiles (p50, p90, p99)
- Average request duration stat
- Jobs processed by type (bar gauge)
- Jobs status breakdown table
- Auto-refresh every 10s, 15-minute default time range
Files added:
- grafana-dashboard.json: Dashboard definition
- deploy-dashboard.sh: Automated deployment script
- DASHBOARD.md: Complete dashboard documentation
Updated:
- internal/metrics/generator.go: Renamed metric names
- internal/ingester/remotewrite.go: Updated historic metric names
- internal/ingester/remotewrite_test.go: Updated test expectations
Tests updated and passing with 63.9% coverage maintained.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
Document demonstrates actual curl commands and their outputs for all
metric types ingested by prometheus-pusher:
- Counter metrics (app_requests_total)
- Gauge metrics (app_temperature_celsius, app_active_connections)
- Histogram metrics (app_request_duration_seconds with buckets, sum, count)
- Labeled counter metrics (app_jobs_processed_total with multiple label combinations)
Includes:
- Complete curl commands
- Actual JSON responses from Prometheus API
- Explanations of each metric type
- Additional query examples (filters, ranges, aggregations)
Verifies data ingestion works correctly with real query results.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
Implemented unit tests across all internal packages to achieve
63.9% test coverage, exceeding the 60% target.
Test coverage by package:
- internal/config: 100.0% (config validation, constants)
- internal/metrics: 100.0% (Sample methods, Collectors, Simulate)
- internal/parser: 92.3% (CSV/JSON parsing, format detection)
- internal/ingester: 44.9% (auto routing, time series conversion)
New test files:
- internal/config/config_test.go: Config creation and constants
- internal/metrics/sample_test.go: Sample type methods (Age, IsRecent)
- internal/metrics/generator_test.go: Collectors and simulation
- internal/parser/csv_test.go: CSV parsing with various inputs
- internal/parser/json_test.go: JSON parsing and validation
- internal/parser/parser_test.go: Parser factory and format handling
- internal/ingester/auto_test.go: Auto mode routing logic
- internal/ingester/remotewrite_test.go: Time series conversion
- internal/ingester/pushgateway_test.go: Pushgateway ingester
Tests cover:
- Happy path and error cases
- Context cancellation support
- Edge cases (empty input, invalid formats)
- Label parsing and timestamp handling
- Metric type generation (counter, gauge, histogram)
- Table-driven tests for comprehensive coverage
All 50+ tests passing ✅
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
Major refactoring to improve code organization and follow Go conventions:
- Moved main entry point to cmd/prometheus-pusher/main.go
- Organized code into internal packages (config, metrics, parser, ingester, version)
- Implemented proper dependency injection (no package-level variables)
- Added context.Context to all blocking operations
- Used value semantics where feasible (Sample, Config, Ingesters)
- Proper error wrapping with %w throughout
- All functions under 50 lines, focused and single-purpose
- Consistent ordering: constants, types, constructors, public, private
- Added -version flag to display version from internal/version package
Package structure:
- cmd/prometheus-pusher: Main entry point with flag parsing and mode routing
- internal/config: Configuration types and constants
- internal/version: Version constant (0.0.0)
- internal/metrics: Sample type and Collectors for metric generation
- internal/parser: CSV/JSON parsers with context support
- internal/ingester: Pushgateway, RemoteWrite, and Auto ingesters
All modes tested and working: realtime, historic, backfill, auto
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
|
|
|
|
Unified all functionality into one binary instead of multiple variants.
## Changes
- Removed: prometheus-pusher-auto, prometheus-pusher-historic
- Single binary: prometheus-pusher (supports all modes)
- Updated all documentation to reference single binary
- Updated run.sh to use unified binary
## Usage
One binary, four modes:
```bash
# Realtime mode (default)
./prometheus-pusher -mode=realtime -continuous
# Historic mode (single datapoint)
./prometheus-pusher -mode=historic -hours-ago=24
# Backfill mode (range of datapoints)
./prometheus-pusher -mode=backfill -start-hours=48 -end-hours=0 -interval=1
# Auto mode (automatic timestamp detection)
./prometheus-pusher -mode=auto -file=data.csv
```
All features accessible from one unified tool!
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
|
|
This commit adds intelligent auto-detection of data age, automatically
choosing the appropriate ingestion method without user intervention.
## New Features
1. **AUTO Mode** (-mode=auto)
- Automatically detects timestamp age from input data
- Routes realtime data (< 5min) → Pushgateway
- Routes historic data (> 5min) → Remote Write API
- No manual timestamp calculation needed!
2. **Input Format Support**
- CSV format: metric_name,labels,value,timestamp_ms
- JSON format: [{metric, labels, value, timestamp_ms}]
- Read from file (-file=path) or stdin
- Comments supported in CSV (#)
3. **Smart Routing Logic**
- 5-minute threshold determines ingestion method
- Handles mixed data (current + historic) in single import
- Clear logging shows which method is used for each sample
4. **Test Data Generation**
- generate-test-data.sh creates samples for all time ranges
- Demonstrates: current, 1h, 1d, 1w, 1m old data
- Actual timestamps generated dynamically
## Files Added
- auto-ingest.go: Core auto-detection logic
- AUTO-MODE.md: Complete documentation
- generate-test-data.sh: Test data generator
- test-data.csv: Example data template
- test-all-ages.csv: Generated test data (all ages)
## Example Usage
```bash
# Generate test data
./generate-test-data.sh
# Auto-import (detects ages automatically)
./prometheus-pusher-auto \
-mode=auto \
-file=test-all-ages.csv \
-pushgateway=http://localhost:9091 \
-prometheus=http://localhost:9090/api/v1/write
```
## Output Example
```
📊 Auto-ingest summary:
Total samples: 15
Realtime samples (< 5min old): 3
Historic samples (> 5min old): 12
🔄 Ingesting 3 REALTIME samples via Pushgateway...
⏰ Ingesting 12 HISTORIC samples via Remote Write...
[1/12] app_requests_total (age: 1.0 hours)
[2/12] app_temperature_celsius (age: 1.0 days)
...
🎉 Auto-ingest complete!
```
## Supported Time Ranges
✅ Current data (< 5min)
✅ 1 hour old data
✅ 1 day old data
✅ 1 week old data
✅ 1 month old data
All ages are automatically detected and routed correctly!
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
This commit extends prometheus-pusher to support ingesting historic data
with custom timestamps via Prometheus Remote Write API.
## Key Changes
1. New Historic Data Module (historic.go)
- GenerateHistoricMetrics: Creates metrics for specific past timestamps
- PushHistoricData: Sends single datapoint via Remote Write API
- BackfillHistoricData: Backfills range of historic data
- Uses Protobuf + Snappy encoding per Prometheus spec
2. Enhanced Main Binary (main.go, realtime.go)
- Refactored to support multiple modes
- Mode 1: realtime - Push to Pushgateway (original behavior)
- Mode 2: historic - Push single historic datapoint
- Mode 3: backfill - Backfill range of historic data
- Command-line flags for configuration
3. Prometheus Configuration (persistence-values.yaml)
- Added web.enable-remote-write-receiver flag
- Enables Prometheus to accept timestamped samples via Remote Write API
- Required for historic data ingestion
4. Documentation (HISTORIC.md)
- Complete guide for historic data ingestion
- Explains limitations and best practices
- Examples for all three modes
- Troubleshooting guide
## Technical Details
**Problem**: Pushgateway doesn't support custom timestamps - Prometheus
always uses "now" when scraping. This prevents backfilling historic data.
**Solution**: Use Prometheus Remote Write API which accepts timestamped
samples. Requires enabling --web.enable-remote-write-receiver flag.
**Data Format**: Protobuf (prompb.WriteRequest) with Snappy compression
**Use Cases**:
- Backfill missing data (e.g., during outage)
- Import historic data from other systems
- Testing with specific timestamps
- Data migration scenarios
## Usage Examples
```bash
# Realtime mode (original behavior)
./prometheus-pusher-historic -mode=realtime -continuous
# Push data from 24 hours ago
./prometheus-pusher-historic -mode=historic -hours-ago=24
# Backfill last 48 hours with 1-hour intervals
./prometheus-pusher-historic -mode=backfill -start-hours=48 -end-hours=0 -interval=1
```
## Dependencies Added
- github.com/prometheus/prometheus (for prompb package)
- github.com/golang/snappy (for compression)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
This commit adds a complete Prometheus data ingestion solution:
1. Pushgateway Helm Chart (f3s/pushgateway/)
- Standalone helm chart for Prometheus Pushgateway
- Deployed to monitoring namespace
- Receives pushed metrics via HTTP POST on port 9091
2. Prometheus Pusher (f3s/prometheus-pusher/)
- Standalone Go binary (12MB) for pushing metrics to Pushgateway
- Demonstrates all Prometheus metric types:
* Counter (app_requests_total)
* Gauge (app_active_connections, app_temperature_celsius)
* Histogram (app_request_duration_seconds)
* Labeled Counter (app_jobs_processed_total)
- Pushes metrics every 15 seconds
- Includes comprehensive documentation and examples
3. Prometheus Configuration
- Updated additional-scrape-configs.yaml to scrape Pushgateway
- Uses honor_labels to preserve pushed metric labels
Architecture:
Go Binary → Pushgateway → Prometheus → Grafana
The pusher binary generates realistic example metrics and pushes them
to Pushgateway in Prometheus text format. Prometheus then scrapes the
Pushgateway and makes the metrics available for querying and alerting.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
