summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2026-01-08Update CLAUDE.md instruction formatPaul Buetow
2026-01-08Add Grafana dashboard for ArgoCD applications monitoringPaul Buetow
Created comprehensive Grafana dashboard showing: - Total applications count - Healthy vs unhealthy applications - Out-of-sync status - Detailed table with all applications and their status - Health status timeline graph - Sync operations rate - Active ArgoCD-related alerts Dashboard will auto-load in Grafana via ConfigMap with label grafana_dashboard='1' Access at: https://grafana.f3s.buetow.org → Dashboards → ArgoCD Applications Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-08Add comprehensive ArgoCD application monitoring and alertsPaul Buetow
This implements monitoring for ALL services deployed via ArgoCD by leveraging ArgoCD's native Prometheus metrics instead of scraping individual services. Changes: - Created ArgoCD application alerts for health and sync status monitoring - Alert when applications are unhealthy (Degraded, Missing, Unknown, Suspended) - Alert when applications are out of sync for >10 minutes - Alert when sync operations are failing repeatedly - Alert when applications are stuck in Progressing state - Added recording rules for unhealthy/out-of-sync application counts - Added radicale health monitoring via scrape config - Added radicale to additional-scrape-configs for direct health checks - Monitors radicale web interface availability Benefits: - Single monitoring solution for all 21 ArgoCD-managed applications - Automatic monitoring for new applications added to ArgoCD - Early detection of configuration drift and deployment issues - Centralized alerting with actionable remediation steps Monitored applications include: radicale, registry, alloy, grafana, loki, prometheus, tempo, anki-sync-server, audiobookshelf, filebrowser, immich, keybr, kobo-sync-server, miniflux, opodsync, and more. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Reorganize argocd-apps by namespace for better structurePaul Buetow
- Create subdirectories: monitoring/, services/, infra/, test/ - Move 6 monitoring apps to monitoring/ - Move 13 service apps to services/ - Move 1 infra app to infra/ - Move 1 test app to test/ - Add README.md documenting the structure and usage This organization: - Makes it easier to understand which apps belong to which namespace - Allows applying apps by namespace: kubectl apply -f argocd-apps/monitoring/ - Supports namespace-scoped app-of-apps patterns - Provides better clarity when browsing the repository All 21 applications remain functional and validated with kubectl --dry-run.
2026-01-07Migrate Grafana-Ingress to ArgoCD GitOpsPaul Buetow
- Created ArgoCD Application for grafana-ingress - Simple custom Helm chart exposing Grafana via Traefik - Updated Justfile with ArgoCD commands - Status: Synced and Healthy - Ingress working at https://grafana.f3s.buetow.org 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate Prometheus to ArgoCD GitOpsPaul Buetow
- Successfully migrated kube-prometheus-stack to ArgoCD - Multi-source Application: upstream chart + manifests directory - PostSync hook automatically restarts Grafana to reload datasources - All recording rules applied (FreeBSD, OpenBSD, ZFS) - All dashboards provisioned - Grafana datasources configured (Prometheus, Loki, Tempo, Alertmanager) - Updated Justfile with ArgoCD commands - Status: Synced and Healthy - Grafana restarted successfully by PostSync hook 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Prepare Prometheus for ArgoCD GitOps migrationPaul Buetow
- Created manifests/ directory with all additional resources - Added sync wave annotations for proper ordering - Created PostSync hook for Grafana pod restart - Converted additional-scrape-configs to Kubernetes Secret - Organized: PVs (wave 0), Secrets/ConfigMaps (wave 1), PrometheusRules (wave 3), Dashboards (wave 4), Hook (wave 10) - Created multi-source ArgoCD Application (upstream chart + manifests) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate Loki and Alloy to ArgoCD GitOpsPaul Buetow
- Created two ArgoCD Application manifests (loki and alloy) - Updated Justfile with ArgoCD commands for both apps - Loki: log aggregation (SingleBinary mode, 10Gi storage) - Alloy: log collection DaemonSet + OTLP receiver for traces - Both apps are Synced and Healthy - Alloy forwards logs to Loki and traces to Tempo 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate Tempo to ArgoCD GitOpsPaul Buetow
- Created ArgoCD Application manifest for Tempo - Updated Justfile with ArgoCD commands (sync, argocd-status, restart) - Tested delete/re-deploy workflow - Verified Tempo is Synced and Healthy - OTLP receivers enabled on ports 4317 (gRPC) and 4318 (HTTP) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate remaining 4 apps: example-apache-volume-claim, registry, ↵Paul Buetow
pushgateway, immich Apps migrated in this commit: - example-apache-volume-claim (test namespace, 2 replicas, 1 PVC) - registry (infra namespace, Docker registry, 1 PVC) - pushgateway (monitoring namespace, Prometheus metrics) - immich (multi-component: server, postgres, valkey, ML) Also: - Deleted unused example-apache directory - Updated all Justfiles with ArgoCD commands - All apps synced and healthy Progress: 16/22 active apps (73%) Remaining apps (all in monitoring namespace): - prometheus (kube-prometheus-stack) - loki (umbrella chart) - tempo - grafana-ingress 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate 8 remaining simple service apps to ArgoCDPaul Buetow
Apps migrated in this commit: - anki-sync-server (custom images, 1 PVC) - syncthing (file sync, 2 PVCs) - audiobookshelf (3 PVCs) - radicale (CalDAV/CardDAV) - opodsync (podcast sync, 2-container pod) - kobo-sync-server (eReader sync) - filebrowser (3 PVCs) - webdav (WebDAV server) All apps: - Created ArgoCD Application manifests - Updated Justfiles with ArgoCD commands - All synced successfully and healthy - Zero downtime migrations Also includes: - Updated migration progress tracker (12/23 apps, 52%) - Deleted freshrss directory (app no longer needed) Progress: 12/23 apps (52%) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate wallabag and keybr to ArgoCD GitOpsPaul Buetow
- Added ArgoCD Application manifests for wallabag and keybr - Updated Justfiles to use ArgoCD commands (sync, argocd-status, restart) - Removed Helm commands (install, upgrade, delete) - Tested delete/re-deploy workflow for both apps - All resources sync successfully, zero downtime Apps migrated: 4/23 (17%) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Test GitOps: Scale frontend to 2 replicasPaul Buetow
Testing ArgoCD auto-sync functionality by scaling the tracing-demo frontend deployment from 1 to 2 replicas. This validates the complete GitOps workflow: commit → push → auto-sync → deployment. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07UpdatePaul Buetow
2026-01-07Update AGENT.md with gogios monitoring configuration patternsPaul Buetow
Document how gogios.json.tpl handles server-specific vs service domain checks: - Dedicated bare hostname checks for server FQDNs - Service domain checks with all prefix variants - Why server hostnames must be skipped in @acme_hosts loop - Impact of not skipping: 12 false critical alerts Explains the same skip pattern used across httpd.conf.tpl, relayd.conf.tpl, and gogios.json.tpl for consistent handling of server-specific hostnames. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Fix gogios monitoring to skip server hostname www/standby variantsPaul Buetow
Skip blowfish.buetow.org and fishfinger.buetow.org in the @acme_hosts loop that creates monitoring checks for www and standby prefix variants. These server-specific hostnames: - Don't have DNS records for www/standby prefixes - Already have dedicated bare hostname checks (lines 29-46) - Should only be monitored without prefix variants This prevents 12 false critical alerts for non-existent: - www.blowfish.buetow.org - standby.blowfish.buetow.org - www.fishfinger.buetow.org - standby.fishfinger.buetow.org Follows same pattern as httpd.conf.tpl and relayd.conf.tpl where server hostnames are skipped in shared configuration loops. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-06Refactor AGENT.md to focus on infrastructure knowledgePaul Buetow
Removed troubleshooting narrative and restructured to document the system architecture, configuration patterns, and operational knowledge. Now covers: - Architecture overview and component responsibilities - Configuration array roles (@acme_hosts, @f3s_hosts, @prefixes) - Template processing and variable scoping - Routing configuration logic - TLS certificate management in multi-server deployments - Server block patterns and duplicate prevention - Server-specific vs. shared host configuration - Deployment process and testing procedures - Monitoring system (Gogios) behavior 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-06Add AGENT.md with debugging learnings and architecture insightsPaul Buetow
Documents the investigation process, root cause analysis, and key learnings from debugging the blowfish/fishfinger 404 errors. Includes: - Architecture overview of relayd + httpd routing - Template variable scoping and processing - Common pitfalls with server-specific vs shared configuration - TLS certificate management in multi-server deployments - Debugging methodology and verification approaches 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-06Fix 404 errors for blowfish/fishfinger index.txt URLsPaul Buetow
Added blowfish.buetow.org and fishfinger.buetow.org to @acme_hosts array to ensure proper routing through relayd to localhost instead of falling through to f3s cluster backends. Changes: - Rexfile: Add blowfish.buetow.org and fishfinger.buetow.org to @acme_hosts - httpd.conf.tpl: Skip current server hostname in @acme_hosts loop to avoid duplicate server blocks (already handled by dedicated "Current server's FQDN" block) - relayd.conf.tpl: Skip both server hostnames in TLS keypair loop since each server only has its own certificate (not the other server's cert) This ensures relayd routes these hostnames to localhost:8080 where httpd serves content from /htdocs/buetow.org/self including index.txt health checks. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-06joPaul Buetow
2026-01-06jo promptsPaul Buetow
2026-01-06add gogios.buetow.orgPaul Buetow
2026-01-03Enable WebSocket support in relayd for audiobookshelfPaul Buetow
- Add http websockets directive to relayd.conf.tpl to allow WebSocket upgrade connections - Fix "Socket failed to connect" error in audiobookshelf web interface - Also add immich helm chart configuration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-02add immicPaul Buetow
2026-01-01fixPaul Buetow
2025-12-31Document Admin API and updated out-of-order configurationPaul Buetow
Updated Prometheus documentation to reflect current configuration: - Added web.enable-admin-api flag documentation - Updated outOfOrderTimeWindow from 720h to 744h (31 days) - Added Data Deletion section with cleanup script usage - Documented manual deletion via Admin API endpoints Provides complete guide for data cleanup after benchmark testing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-31UpdatePaul Buetow
2025-12-31Enable Prometheus admin API for data deletionPaul Buetow
Added web.enable-admin-api flag to allow selective deletion of time series data via the /api/v1/admin/tsdb endpoints. This enables cleanup of benchmark data using the delete_series and clean_tombstones APIs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-31Increase Prometheus out-of-order window to 31 daysPaul Buetow
Changes: - outOfOrderTimeWindow: 720h → 744h (30 days → 31 days) Rationale: Provides 1-day buffer for 30-day backfill operations to avoid edge case rejections where the oldest samples exceed the limit due to timing variations between data generation and ingestion. With this configuration: - 30-day benchmarks achieve 99.85% success rate (vs 50% with 720h) - Only 4/2592 batches rejected (first few batches slightly over 30d) - Allows safe backfilling of up to 30 days of historic data 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-31Enable Prometheus historic data ingestion with out-of-order supportPaul Buetow
This commit configures Prometheus to accept historic data via the Remote Write API, enabling backfilling of test metrics for development and troubleshooting purposes. Changes: - Enable Remote Write receiver (--web.enable-remote-write-receiver) - Enable out-of-order ingestion with 30-day window (720h) - Enable exemplar-storage and otlp-write-receiver features - Add Epimetheus dashboard ConfigMap for Grafana provisioning - Remove old prometheus-pusher directory (moved to separate repo) - Document configuration, use cases, and performance considerations Configuration allows backfilling data up to 30 days in the past, supporting tools like Epimetheus for generating synthetic historic metrics. Performance note: This is optimized for ad-hoc troubleshooting, not production use. Out-of-order ingestion increases memory usage, TSDB overhead, and may impact query performance. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Enable Prometheus Remote Write receiver for historic data ingestionPaul Buetow
Updated persistence-values.yaml to enable the Remote Write receiver using the correct flag for Prometheus 3.x: - Changed from enableFeatures (not supported in 3.8.1) - To additionalArgs with web.enable-remote-write-receiver This allows Epimetheus to push historic data with preserved timestamps via the Prometheus Remote Write API endpoint (/api/v1/write). Applied via: just upgrade 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Consolidate all documentation into single comprehensive README.mdPaul Buetow
- Merged content from 10 separate .md files into README.md - Removed: ANSWER.md, AUTO-MODE.md, DASHBOARD.md, HISTORIC.md, LIMITATIONS.md, QUERY_EXAMPLES.md, QUICK-START.md, SETUP-COMPLETE.md, SUMMARY.md, USAGE.md - README.md now includes: * Quick start guide * All operating modes (realtime, historic, backfill, auto) * Data formats (CSV, JSON) * Test metrics documentation * Grafana dashboard setup * Example queries and curl commands * Time range limitations * Troubleshooting guide * Architecture diagram * Best practices 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Rename test metrics with clear prefix and add Grafana dashboardPaul Buetow
Renamed all test metrics with "prometheus_pusher_test_" prefix to clearly indicate they are generated by the prometheus-pusher testing/demo functionality. Metric renaming: - app_requests_total → prometheus_pusher_test_requests_total - app_active_connections → prometheus_pusher_test_active_connections - app_temperature_celsius → prometheus_pusher_test_temperature_celsius - app_request_duration_seconds → prometheus_pusher_test_request_duration_seconds - app_jobs_processed_total → prometheus_pusher_test_jobs_processed_total Grafana Dashboard: - Created comprehensive dashboard with 8 panels - Request rate and total requests visualization - Active connections gauge (0-100 with thresholds) - Temperature gauge (0-50°C with thresholds) - Request duration percentiles (p50, p90, p99) - Average request duration stat - Jobs processed by type (bar gauge) - Jobs status breakdown table - Auto-refresh every 10s, 15-minute default time range Files added: - grafana-dashboard.json: Dashboard definition - deploy-dashboard.sh: Automated deployment script - DASHBOARD.md: Complete dashboard documentation Updated: - internal/metrics/generator.go: Renamed metric names - internal/ingester/remotewrite.go: Updated historic metric names - internal/ingester/remotewrite_test.go: Updated test expectations Tests updated and passing with 63.9% coverage maintained. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Add comprehensive Prometheus query examples documentationPaul Buetow
Document demonstrates actual curl commands and their outputs for all metric types ingested by prometheus-pusher: - Counter metrics (app_requests_total) - Gauge metrics (app_temperature_celsius, app_active_connections) - Histogram metrics (app_request_duration_seconds with buckets, sum, count) - Labeled counter metrics (app_jobs_processed_total with multiple label combinations) Includes: - Complete curl commands - Actual JSON responses from Prometheus API - Explanations of each metric type - Additional query examples (filters, ranges, aggregations) Verifies data ingestion works correctly with real query results. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Add comprehensive unit tests with 63.9% coveragePaul Buetow
Implemented unit tests across all internal packages to achieve 63.9% test coverage, exceeding the 60% target. Test coverage by package: - internal/config: 100.0% (config validation, constants) - internal/metrics: 100.0% (Sample methods, Collectors, Simulate) - internal/parser: 92.3% (CSV/JSON parsing, format detection) - internal/ingester: 44.9% (auto routing, time series conversion) New test files: - internal/config/config_test.go: Config creation and constants - internal/metrics/sample_test.go: Sample type methods (Age, IsRecent) - internal/metrics/generator_test.go: Collectors and simulation - internal/parser/csv_test.go: CSV parsing with various inputs - internal/parser/json_test.go: JSON parsing and validation - internal/parser/parser_test.go: Parser factory and format handling - internal/ingester/auto_test.go: Auto mode routing logic - internal/ingester/remotewrite_test.go: Time series conversion - internal/ingester/pushgateway_test.go: Pushgateway ingester Tests cover: - Happy path and error cases - Context cancellation support - Edge cases (empty input, invalid formats) - Label parsing and timestamp handling - Metric type generation (counter, gauge, histogram) - Table-driven tests for comprehensive coverage All 50+ tests passing ✅ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Refactor prometheus-pusher following Go best practicesPaul Buetow
Major refactoring to improve code organization and follow Go conventions: - Moved main entry point to cmd/prometheus-pusher/main.go - Organized code into internal packages (config, metrics, parser, ingester, version) - Implemented proper dependency injection (no package-level variables) - Added context.Context to all blocking operations - Used value semantics where feasible (Sample, Config, Ingesters) - Proper error wrapping with %w throughout - All functions under 50 lines, focused and single-purpose - Consistent ordering: constants, types, constructors, public, private - Added -version flag to display version from internal/version package Package structure: - cmd/prometheus-pusher: Main entry point with flag parsing and mode routing - internal/config: Configuration types and constants - internal/version: Version constant (0.0.0) - internal/metrics: Sample type and Collectors for metric generation - internal/parser: CSV/JSON parsers with context support - internal/ingester: Pushgateway, RemoteWrite, and Auto ingesters All modes tested and working: realtime, historic, backfill, auto 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Add documentation and tests for ingestion time range limitsPaul Buetow
2025-12-30Add QUICK-START.md for single binary referencePaul Buetow
2025-12-30Consolidate into single prometheus-pusher binaryPaul Buetow
Unified all functionality into one binary instead of multiple variants. ## Changes - Removed: prometheus-pusher-auto, prometheus-pusher-historic - Single binary: prometheus-pusher (supports all modes) - Updated all documentation to reference single binary - Updated run.sh to use unified binary ## Usage One binary, four modes: ```bash # Realtime mode (default) ./prometheus-pusher -mode=realtime -continuous # Historic mode (single datapoint) ./prometheus-pusher -mode=historic -hours-ago=24 # Backfill mode (range of datapoints) ./prometheus-pusher -mode=backfill -start-hours=48 -end-hours=0 -interval=1 # Auto mode (automatic timestamp detection) ./prometheus-pusher -mode=auto -file=data.csv ``` All features accessible from one unified tool! 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Add ANSWER.md documenting support for all time rangesPaul Buetow
2025-12-30Add AUTO mode with automatic timestamp detectionPaul Buetow
This commit adds intelligent auto-detection of data age, automatically choosing the appropriate ingestion method without user intervention. ## New Features 1. **AUTO Mode** (-mode=auto) - Automatically detects timestamp age from input data - Routes realtime data (< 5min) → Pushgateway - Routes historic data (> 5min) → Remote Write API - No manual timestamp calculation needed! 2. **Input Format Support** - CSV format: metric_name,labels,value,timestamp_ms - JSON format: [{metric, labels, value, timestamp_ms}] - Read from file (-file=path) or stdin - Comments supported in CSV (#) 3. **Smart Routing Logic** - 5-minute threshold determines ingestion method - Handles mixed data (current + historic) in single import - Clear logging shows which method is used for each sample 4. **Test Data Generation** - generate-test-data.sh creates samples for all time ranges - Demonstrates: current, 1h, 1d, 1w, 1m old data - Actual timestamps generated dynamically ## Files Added - auto-ingest.go: Core auto-detection logic - AUTO-MODE.md: Complete documentation - generate-test-data.sh: Test data generator - test-data.csv: Example data template - test-all-ages.csv: Generated test data (all ages) ## Example Usage ```bash # Generate test data ./generate-test-data.sh # Auto-import (detects ages automatically) ./prometheus-pusher-auto \ -mode=auto \ -file=test-all-ages.csv \ -pushgateway=http://localhost:9091 \ -prometheus=http://localhost:9090/api/v1/write ``` ## Output Example ``` 📊 Auto-ingest summary: Total samples: 15 Realtime samples (< 5min old): 3 Historic samples (> 5min old): 12 🔄 Ingesting 3 REALTIME samples via Pushgateway... ⏰ Ingesting 12 HISTORIC samples via Remote Write... [1/12] app_requests_total (age: 1.0 hours) [2/12] app_temperature_celsius (age: 1.0 days) ... 🎉 Auto-ingest complete! ``` ## Supported Time Ranges ✅ Current data (< 5min) ✅ 1 hour old data ✅ 1 day old data ✅ 1 week old data ✅ 1 month old data All ages are automatically detected and routed correctly! 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Add historic data ingestion support to prometheus-pusherPaul Buetow
This commit extends prometheus-pusher to support ingesting historic data with custom timestamps via Prometheus Remote Write API. ## Key Changes 1. New Historic Data Module (historic.go) - GenerateHistoricMetrics: Creates metrics for specific past timestamps - PushHistoricData: Sends single datapoint via Remote Write API - BackfillHistoricData: Backfills range of historic data - Uses Protobuf + Snappy encoding per Prometheus spec 2. Enhanced Main Binary (main.go, realtime.go) - Refactored to support multiple modes - Mode 1: realtime - Push to Pushgateway (original behavior) - Mode 2: historic - Push single historic datapoint - Mode 3: backfill - Backfill range of historic data - Command-line flags for configuration 3. Prometheus Configuration (persistence-values.yaml) - Added web.enable-remote-write-receiver flag - Enables Prometheus to accept timestamped samples via Remote Write API - Required for historic data ingestion 4. Documentation (HISTORIC.md) - Complete guide for historic data ingestion - Explains limitations and best practices - Examples for all three modes - Troubleshooting guide ## Technical Details **Problem**: Pushgateway doesn't support custom timestamps - Prometheus always uses "now" when scraping. This prevents backfilling historic data. **Solution**: Use Prometheus Remote Write API which accepts timestamped samples. Requires enabling --web.enable-remote-write-receiver flag. **Data Format**: Protobuf (prompb.WriteRequest) with Snappy compression **Use Cases**: - Backfill missing data (e.g., during outage) - Import historic data from other systems - Testing with specific timestamps - Data migration scenarios ## Usage Examples ```bash # Realtime mode (original behavior) ./prometheus-pusher-historic -mode=realtime -continuous # Push data from 24 hours ago ./prometheus-pusher-historic -mode=historic -hours-ago=24 # Backfill last 48 hours with 1-hour intervals ./prometheus-pusher-historic -mode=backfill -start-hours=48 -end-hours=0 -interval=1 ``` ## Dependencies Added - github.com/prometheus/prometheus (for prompb package) - github.com/golang/snappy (for compression) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Add Prometheus Pushgateway and data ingestion toolPaul Buetow
This commit adds a complete Prometheus data ingestion solution: 1. Pushgateway Helm Chart (f3s/pushgateway/) - Standalone helm chart for Prometheus Pushgateway - Deployed to monitoring namespace - Receives pushed metrics via HTTP POST on port 9091 2. Prometheus Pusher (f3s/prometheus-pusher/) - Standalone Go binary (12MB) for pushing metrics to Pushgateway - Demonstrates all Prometheus metric types: * Counter (app_requests_total) * Gauge (app_active_connections, app_temperature_celsius) * Histogram (app_request_duration_seconds) * Labeled Counter (app_jobs_processed_total) - Pushes metrics every 15 seconds - Includes comprehensive documentation and examples 3. Prometheus Configuration - Updated additional-scrape-configs.yaml to scrape Pushgateway - Uses honor_labels to preserve pushed metric labels Architecture: Go Binary → Pushgateway → Prometheus → Grafana The pusher binary generates realistic example metrics and pushes them to Pushgateway in Prometheus text format. Prometheus then scrapes the Pushgateway and makes the metrics available for querying and alerting. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Fix ArgoCD to preserve password on helm upgradePaul Buetow
Remove fixed password from values.yaml so helm upgrade doesn't reset the admin password when users change it via UI. Changes: - Remove argocdServerAdminPassword from values.yaml - Leave password generation to ArgoCD default behavior - Update Justfile install message to show get-password command Behavior now: - helm install: Generates random password in argocd-initial-admin-secret - helm upgrade: Preserves existing password (does NOT reset) - helm uninstall: Deletes secret along with all resources - User password changes via UI are preserved Verified: - Password hash unchanged after helm upgrade ✅ - Secret deleted on helm uninstall ✅ - Login works before and after upgrade ✅ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Fix ArgoCD secret management - delete with helm uninstallPaul Buetow
Change admin password handling so it's properly managed by Helm and gets deleted when running helm uninstall, while using a fixed password instead of random generation. Changes: - Set fixed argocdServerAdminPassword in values.yaml - Remove configs.secret.createSecret: false (use Helm default: true) - Remove argocd-secret.yaml (Helm creates it now) - Update Justfile to not apply manual secret - Password: "argocd-admin-default" (bcrypt hash in values.yaml) Behavior: - helm install: Creates secret with fixed password - helm upgrade: Updates secret to fixed password (resets any UI changes) - helm uninstall: Deletes secret along with all resources - Secret has Helm annotations (managed by Helm) This is standard Helm behavior - the password in values.yaml is the source of truth. User can change via UI, but helm operations will reset it to the configured value. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Configure ArgoCD to preserve admin password across redeploymentsPaul Buetow
Ensure admin password persists through helm uninstall/install cycles by managing argocd-secret outside of Helm's control. Changes: - Set configs.secret.createSecret: false in values.yaml - Create argocd-secret.yaml with default admin password - Update Justfile to apply secret before helm install - Secret is now managed by kubectl, not Helm - Default password: "argocd-admin-default" (change after first login) Benefits: - Admin password survives helm uninstall/install - Password changes via UI/CLI are preserved - No random password regeneration on redeployments - Secret has no Helm annotations (not managed by Helm) The argocd-secret will persist across redeployments unless explicitly deleted. PVC and admin password are now both persistent. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Configure persistent cache for ArgoCD repo-serverPaul Buetow
Enable PVC mount for ArgoCD repo-server to cache Git repositories and generated manifests, improving performance for subsequent deployments. Changes: - Mount argocd-repo-server-pvc at /home/argocd/repo-cache - Set XDG_CACHE_HOME environment variable to use persistent cache - Avoid conflict with default /tmp mount used by ArgoCD This ensures Git repo clones and Helm charts are cached persistently across pod restarts, reducing network traffic and speeding up syncs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Add ArgoCD deployment to cicd namespacePaul Buetow
Deploy ArgoCD v3.2.3 for GitOps continuous delivery in the k3s cluster. Configuration: - New cicd namespace for CI/CD tooling - Non-HA single instance deployment (following cluster patterns) - Traefik ingress at argocd.f3s.buetow.org - Prometheus ServiceMonitor integration for metrics - 10Gi persistent volume for repo-server cache - Insecure mode with TLS termination at proxy Components deployed: - argocd-server (Web UI and API) - argocd-repo-server (Repository management) - argocd-application-controller (Application sync) - argocd-redis (State cache) - argocd-applicationset-controller (Multi-app management) Also adds argocd.f3s.buetow.org to frontends Rexfile for relayd proxy configuration. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Add automatic fallback for f3s hosts when cluster is downPaul Buetow
Configure OpenBSD relayd and httpd to serve a friendly fallback page when the f3s Kubernetes cluster is unreachable. Changes to relayd.conf.tpl: - Reorder relay forward statements: f3s first, localhost as backup - Remove protocol-level forward rules for f3s hosts to enable relay-level failover - Add explicit localhost routing for non-f3s hosts - Health checks on f3s table trigger automatic failover to localhost Changes to httpd.conf.tpl: - Add request rewrite directive to serve fallback page for ALL paths - Prevents 404 errors for deep links like /login?redirect=/files/ - Ensures consistent fallback experience regardless of requested URL When all f3s nodes fail health checks, traffic automatically routes to localhost:8080 serving static fallback content from /var/www/htdocs/f3s_fallback. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30fallback page displayPaul Buetow
generated by cgit v1.2.3 (git 2.25.1) at 2026-08-08 19:00:36 +0000