| Age | Commit message (Collapse) | Author |
|---|
|
Ensure admin password persists through helm uninstall/install cycles by
managing argocd-secret outside of Helm's control.
Changes:
- Set configs.secret.createSecret: false in values.yaml
- Create argocd-secret.yaml with default admin password
- Update Justfile to apply secret before helm install
- Secret is now managed by kubectl, not Helm
- Default password: "argocd-admin-default" (change after first login)
Benefits:
- Admin password survives helm uninstall/install
- Password changes via UI/CLI are preserved
- No random password regeneration on redeployments
- Secret has no Helm annotations (not managed by Helm)
The argocd-secret will persist across redeployments unless explicitly
deleted. PVC and admin password are now both persistent.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
Enable PVC mount for ArgoCD repo-server to cache Git repositories and
generated manifests, improving performance for subsequent deployments.
Changes:
- Mount argocd-repo-server-pvc at /home/argocd/repo-cache
- Set XDG_CACHE_HOME environment variable to use persistent cache
- Avoid conflict with default /tmp mount used by ArgoCD
This ensures Git repo clones and Helm charts are cached persistently
across pod restarts, reducing network traffic and speeding up syncs.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
Deploy ArgoCD v3.2.3 for GitOps continuous delivery in the k3s cluster.
Configuration:
- New cicd namespace for CI/CD tooling
- Non-HA single instance deployment (following cluster patterns)
- Traefik ingress at argocd.f3s.buetow.org
- Prometheus ServiceMonitor integration for metrics
- 10Gi persistent volume for repo-server cache
- Insecure mode with TLS termination at proxy
Components deployed:
- argocd-server (Web UI and API)
- argocd-repo-server (Repository management)
- argocd-application-controller (Application sync)
- argocd-redis (State cache)
- argocd-applicationset-controller (Multi-app management)
Also adds argocd.f3s.buetow.org to frontends Rexfile for relayd proxy
configuration.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
Remove unnecessary quotes and comments from the Tempo datasource ConfigMap.
This file is now deprecated in favor of the unified grafana-datasources-all.yaml
approach, but keeping it cleaned up for historical reference.
Changes:
- Remove quotes from string values (datasourceUid, spanStartTimeShift, etc.)
- Remove inline comments
- Format tags array properly
- Standardize YAML formatting
Note: This ConfigMap is no longer used. Datasources are now provisioned via
direct ConfigMap mounting using grafana-datasources-all.yaml.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
Problem:
- Only health check traces appeared in Tempo
- API endpoint traces (/api/process) were not visible
- Alloy OTLP receivers were not listening (needed restart)
Root Causes:
1. Health check endpoints were creating massive trace volume from Kubernetes probes
2. Batch processor (100 spans) was filling with health checks before API traces could export
3. Alloy DaemonSet needed restart to activate OTLP receivers after configuration update
Solution:
1. Restarted Alloy to activate OTLP gRPC (4317) and HTTP (4318) receivers
2. Excluded /health endpoint from Flask auto-instrumentation in all three services:
- frontend: FlaskInstrumentor().instrument_app(app, excluded_urls="/health")
- middleware: FlaskInstrumentor().instrument_app(app, excluded_urls="/health")
- backend: FlaskInstrumentor().instrument_app(app, excluded_urls="/health")
Result:
✅ Distributed traces now visible in Tempo with full span chains
✅ Single /api/process request creates 8 spans across 3 services:
- Frontend: GET /api/process, frontend-process, POST (200ms)
- Middleware: POST /api/transform, middleware-transform, GET (180ms)
- Backend: GET /api/data, backend-get-data (100ms)
✅ Complete request flow traced: frontend → middleware → backend
✅ Node graph will now show service dependencies
✅ Traces-to-logs and traces-to-metrics correlation enabled
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
After extensive debugging (documented in problem.md), resolved the issue where Tempo
and Loki datasources would not appear in Grafana despite correct configuration.
Root Cause:
- Sidecar-based provisioning with label discovery was not triggering the provisioner module
- Multi-step indirection (sidecar → watch → write → reload) had silent failures
Solution (following x-rag pattern):
- Disabled sidecar datasource provisioning
- Created unified grafana-datasources-all.yaml with all datasources
- Mount ConfigMap directly to /etc/grafana/provisioning/datasources/
- Grafana now reads datasources on startup via built-in provisioning
Changes:
- NEW: grafana-datasources-all.yaml - Unified datasource configuration (Prometheus, Alertmanager, Loki, Tempo)
- MODIFIED: persistence-values.yaml - Disabled sidecar, added extraVolumes/extraVolumeMounts
- MODIFIED: Justfile - Updated to use unified ConfigMap, removed patch script
- MODIFIED: README.md - Documented new provisioning approach
- NEW: problem.md - Complete debugging journey with 16 attempts documented
- DEPRECATED: loki-datasource.yaml, tempo-datasource.yaml, patch-datasources.sh (kept for history)
Result:
✅ All datasources now successfully provision on Grafana startup
✅ Tempo datasource (uid=tempo) appears in Grafana with traces-to-logs correlation
✅ Loki datasource (uid=loki) appears in Grafana
✅ Simple, maintainable approach without sidecar complexity
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
- Deploy Grafana Tempo in monolithic mode for distributed tracing
- Configure Tempo with OTLP receivers (gRPC:4317, HTTP:4318)
- Set up 10Gi filesystem storage with 7-day retention
- Integrate Tempo datasource in Grafana with traces-to-logs and traces-to-metrics correlation
- Update Grafana Alloy to collect and forward traces
- Add OTLP receiver configuration to alloy-values.yaml
- Configure batch processor for efficient trace forwarding to Tempo
- Patch Alloy service to expose OTLP ports 4317/4318
- Create demo tracing application (frontend, middleware, backend)
- Implement three-tier Python Flask application with OpenTelemetry instrumentation
- Auto-instrument with OpenTelemetry for Flask and requests libraries
- Push Docker images to private registry (registry.lan.buetow.org:30001)
- Deploy via Helm chart with Traefik ingress at tracing-demo.f3s.buetow.org
- Update Grafana configuration in prometheus/persistence-values.yaml
- Add Tempo to additionalDataSources for automatic provisioning
Files added:
- tempo/values.yaml: Tempo Helm chart configuration
- tempo/persistent-volumes.yaml: Storage configuration (10Gi PV/PVC)
- tempo/datasource-configmap.yaml: Grafana datasource with correlations
- tempo/Justfile: Installation automation
- tempo/README.md: Documentation
- tracing-demo/docker/frontend/: Python Flask frontend with OTel
- tracing-demo/docker/middleware/: Python Flask middleware with OTel
- tracing-demo/docker/backend/: Python Flask backend with OTel
- tracing-demo/helm-chart/: Kubernetes deployments, services, ingress
- tracing-demo/docker-image-Justfile: Docker build/push automation
- tracing-demo/Justfile: Helm deployment automation
- tracing-demo/README.md: Documentation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
Implemented complete ZFS monitoring solution including ARC cache statistics,
pool health/capacity metrics, dataset usage, and I/O throughput monitoring.
Changes:
- Add ZFS recording rules (9 calculated metrics for ARC hit rates, memory usage, etc.)
- Add comprehensive Grafana dashboard with 19 panels across 5 rows:
* Pool Overview: capacity, health, size, free space, usage trends
* I/O Throughput: read/write operations and bytes per second
* Dataset Statistics: table showing all datasets with usage details
* ARC Cache Statistics: hit rates, size, memory usage
* ARC Breakdown: data vs metadata, MRU vs MFU with pie charts
- Update Justfile to deploy ZFS recording rules
- Add textfile collector script on FreeBSD servers (f0, f1, f2) for pool/dataset metrics
Metrics collected:
- Pool: size, allocated, free, capacity %, health status
- I/O: read/write operations and throughput (via zpool iostat)
- Dataset: used, available, referenced space per filesystem
- ARC: hit rate, size, memory usage, data/metadata breakdown
Fixes:
- Pool health panel properly displays ONLINE/DEGRADED/FAULTED status
- All stat panels have correct options configuration
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Enable etcd metrics on port 2381
- Add blog post draft documenting the changes
|
|
Reverts hostname relabeling and etcd metrics changes
|
|
- Add relabel_configs to show hostnames for node-exporter targets
- Enable etcd metrics scraping on port 2381
- Update blog post draft
|
|
- Add relabel_configs to additional-scrape-configs.yaml for FreeBSD/OpenBSD hosts
- Add node name relabeling for node-exporter on k3s nodes
- Enable etcd metrics scraping with hostname relabeling
- Add DRAFT blog post documenting the changes
Amp-Thread-ID: https://ampcode.com/threads/T-019b571c-4afc-7789-becf-bc8a3c4e1e1f
Co-authored-by: Amp <amp@ampcode.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Amp-Thread-ID: https://ampcode.com/threads/T-ccf9cd44-5adf-4633-9f3d-d822f733af4d
Co-authored-by: Amp <amp@ampcode.com>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
