| Age | Commit message (Collapse) | Author |
|---|
|
Updated Prometheus documentation to reflect current configuration:
- Added web.enable-admin-api flag documentation
- Updated outOfOrderTimeWindow from 720h to 744h (31 days)
- Added Data Deletion section with cleanup script usage
- Documented manual deletion via Admin API endpoints
Provides complete guide for data cleanup after benchmark testing.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
This commit configures Prometheus to accept historic data via the Remote
Write API, enabling backfilling of test metrics for development and
troubleshooting purposes.
Changes:
- Enable Remote Write receiver (--web.enable-remote-write-receiver)
- Enable out-of-order ingestion with 30-day window (720h)
- Enable exemplar-storage and otlp-write-receiver features
- Add Epimetheus dashboard ConfigMap for Grafana provisioning
- Remove old prometheus-pusher directory (moved to separate repo)
- Document configuration, use cases, and performance considerations
Configuration allows backfilling data up to 30 days in the past, supporting
tools like Epimetheus for generating synthetic historic metrics.
Performance note: This is optimized for ad-hoc troubleshooting, not
production use. Out-of-order ingestion increases memory usage, TSDB overhead,
and may impact query performance.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
After extensive debugging (documented in problem.md), resolved the issue where Tempo
and Loki datasources would not appear in Grafana despite correct configuration.
Root Cause:
- Sidecar-based provisioning with label discovery was not triggering the provisioner module
- Multi-step indirection (sidecar → watch → write → reload) had silent failures
Solution (following x-rag pattern):
- Disabled sidecar datasource provisioning
- Created unified grafana-datasources-all.yaml with all datasources
- Mount ConfigMap directly to /etc/grafana/provisioning/datasources/
- Grafana now reads datasources on startup via built-in provisioning
Changes:
- NEW: grafana-datasources-all.yaml - Unified datasource configuration (Prometheus, Alertmanager, Loki, Tempo)
- MODIFIED: persistence-values.yaml - Disabled sidecar, added extraVolumes/extraVolumeMounts
- MODIFIED: Justfile - Updated to use unified ConfigMap, removed patch script
- MODIFIED: README.md - Documented new provisioning approach
- NEW: problem.md - Complete debugging journey with 16 attempts documented
- DEPRECATED: loki-datasource.yaml, tempo-datasource.yaml, patch-datasources.sh (kept for history)
Result:
✅ All datasources now successfully provision on Grafana startup
✅ Tempo datasource (uid=tempo) appears in Grafana with traces-to-logs correlation
✅ Loki datasource (uid=loki) appears in Grafana
✅ Simple, maintainable approach without sidecar complexity
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
|
|
|
