summaryrefslogtreecommitdiff
path: root/f3s
AgeCommit message (Collapse)Author
2026-01-09Remove fsGroup to fix nginx worker setgid errorsPaul Buetow
The fsGroup: 1000 was causing nginx workers to fail with setgid errors even after removing 'user nginx;' directive. Since both containers run as root, fsGroup is not needed for repo access.
2026-01-09Remove 'user nginx;' directive to fix nginx worker errorsPaul Buetow
When running as root with fsGroup, nginx workers fail trying to setgid. Remove the 'user nginx;' directive from nginx.conf at startup using sed. This allows nginx to run workers as root without permission errors.
2026-01-09Run spawn-fcgi as root to avoid setgid errorsPaul Buetow
Remove -u nginx -g nginx from spawn-fcgi command to run as root. This avoids nginx worker process setgid permission errors.
2026-01-09Override cgit entrypoint to skip chown/chmodPaul Buetow
The cgit image entrypoint always tries to chown /var/cache/cgit which fails with permission errors. Override the entrypoint to directly: 1. Spawn fcgiwrap as nginx user 2. Start nginx in foreground This skips the problematic chown/chmod and template substitution.
2026-01-09Disable cgit caching to avoid permission issuesPaul Buetow
Instead of fighting permission issues with the cgit cache directory, disable caching entirely by: - Setting cache-size=0 in cgitrc - Removing cgit-cache emptyDir volume and mounts - Simplifying initContainer (only SSH keys setup needed) cgit will work without caching, just slightly slower for large repos.
2026-01-09Run cgit as root - required for nginx and spawn-fcgiPaul Buetow
cgit image needs root to: - Bind sockets with spawn-fcgi - Run nginx master process - Write to /var/run/nginx.pid The initContainer already sets up cache dir with proper permissions.
2026-01-09Set USE_CUSTOM_CONFIG=true to skip cgit template substitutionPaul Buetow
The cgit entrypoint tries to write to /etc/cgitrc which is mounted read-only from our ConfigMap. Set USE_CUSTOM_CONFIG=true to use our custom cgitrc directly without template substitution.
2026-01-09Fix cgit permissions - use UID 101 (nginx user)Paul Buetow
The cgit image runs as nginx user (UID 101), not www-data (UID 33). - Update initContainer to chown cache to 101:1000 - Update cgit securityContext to runAsUser: 101
2026-01-09Fix permissions using fsGroup and initContainer patternPaul Buetow
Follow webdav/filebrowser pattern for proper permission handling: - Add fsGroup: 1000 at pod level for git repo access - Add initContainer to chown emptyDir volumes - Run git-server as root (required for sshd) - Run cgit as user 33 (www-data) - Restore cgit-cache emptyDir volume with proper ownership
2026-01-09Run containers as root and use emptyDir for writeable dirsPaul Buetow
- Mount emptyDir for /etc/ssh to allow SSH host key generation - Mount emptyDir for /var/cache/cgit to allow cache initialization - Run both containers as root with proper capabilities - Copy sshd_config at runtime from /tmp to /etc/ssh - Add imagePullPolicy: Always to force image refresh
2026-01-09Fix SSH host keys and container securityPaul Buetow
- Generate SSH host keys at runtime via entrypoint script - Remove fsGroup security context to fix emptyDir permissions - Allow cgit to initialize cache directory as root
2026-01-09Fix sshd_config and cgit permissionsPaul Buetow
- Remove unsupported UsePAM option from sshd_config - Run cgit as root to allow cache directory initialization - Add CHOWN and DAC_OVERRIDE capabilities for cgit
2026-01-09Fix git-server deploymentPaul Buetow
- Use registry.lan.buetow.org for deployment (internal DNS) - Add emptyDir volume for cgit cache directory - Add README.md with deployment and secret management instructions This fixes image pull issues and cgit permission errors.
2026-01-09Add self-hosted git server with SSH and cgit web UIPaul Buetow
Deploy a self-hosted git repository solution to replace external Codeberg dependency. Components: - SSH git server: Alpine-based container with OpenSSH and git - cgit web UI: Browse repositories at cgit.f3s.buetow.org - Single pod design: git-server + cgit containers sharing storage Infrastructure: - Docker image in git-server/docker-image/ with Justfile build automation - Helm chart in git-server/helm-chart/ for Kubernetes deployment - 5Gi ReadWriteMany PVC for NFS-backed repository storage - ClusterIP service for ArgoCD internal access - NodePort 30022 for external SSH push access - Traefik ingress for cgit web UI ArgoCD Application manifest deployed to cicd namespace. Note: SSH keys must be created as Kubernetes secrets manually, not in git. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-08Enable prune for prometheus ArgoCD app, update READMEPaul Buetow
2026-01-08prunePaul Buetow
2026-01-08Remove invalid radicale scrape job (no metrics endpoint)Paul Buetow
2026-01-08Fix: disable kubeScheduler rules entirelyPaul Buetow
2026-01-08Disable KubeProxyDown and KubeSchedulerDown alerts for k3sPaul Buetow
2026-01-08add agentsPaul Buetow
2026-01-08Add Prometheus NodePort and alert query targets to JustfilePaul Buetow
2026-01-08Add NodePort service for Prometheus on port 30090Paul Buetow
2026-01-08renamePaul Buetow
2026-01-08Add volumeName to PVC for explicit bindingPaul Buetow
2026-01-08Change apache PV/PVC to ReadWriteMany for multi-pod accessPaul Buetow
2026-01-08Disable kube-proxy and kube-scheduler monitoring for k3sPaul Buetow
K3s embeds kube-proxy and kube-scheduler functionality into the main k3s server process, unlike standard Kubernetes where they run as separate components. This change disables monitoring for these components to prevent false-positive critical alerts: - KubeProxyDown - KubeSchedulerDown These alerts were firing because kube-prometheus-stack expects standard Kubernetes architecture with separate kube-proxy and kube-scheduler pods/processes. Cluster info: - Running k3s v1.32.6+k3s1 - 3 control-plane nodes (r0, r1, r2) - Components embedded in k3s binary Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-08Add convenient port-forward targets for Prometheus monitoringPaul Buetow
Added enhanced port-forward targets with helpful UI information: - 'just alerts' - Quick access to Prometheus alerts view - 'just alertmanager' - Quick access to Alertmanager UI - Enhanced output showing all relevant URLs All port-forward commands now display: - Access URLs with direct links to specific views - Clear instructions for stopping (Ctrl+C) Usage: cd prometheus/ just alerts # Opens Prometheus alerts (port 9090) just alertmanager # Opens Alertmanager (port 9093) just port-forward-prometheus [port] just port-forward-grafana [port] After running, access: - Prometheus Alerts: http://localhost:9090/alerts - Alertmanager: http://localhost:9093 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-08Configure Alertmanager routing for ArgoCD application alertsPaul Buetow
Added Alertmanager configuration to: - Route ArgoCD application alerts to dedicated 'argocd-alerts' receiver - Group ArgoCD alerts by alertname, name (app name), and severity - Faster alert grouping for ArgoCD (10s wait vs 30s default) - Repeat ArgoCD alerts every 6 hours - Suppress Watchdog test alerts - Configure inhibit rules to prevent alert spam Alerts are visible in: - Prometheus UI: http://localhost:9090/alerts - Alertmanager UI: http://localhost:9093 - Grafana dashboard: ArgoCD Applications - Health & Sync Status This ensures critical application issues are properly routed and visible in the monitoring UI for immediate action. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-08Update CLAUDE.md instruction formatPaul Buetow
2026-01-08Add Grafana dashboard for ArgoCD applications monitoringPaul Buetow
Created comprehensive Grafana dashboard showing: - Total applications count - Healthy vs unhealthy applications - Out-of-sync status - Detailed table with all applications and their status - Health status timeline graph - Sync operations rate - Active ArgoCD-related alerts Dashboard will auto-load in Grafana via ConfigMap with label grafana_dashboard='1' Access at: https://grafana.f3s.buetow.org → Dashboards → ArgoCD Applications Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-08Add comprehensive ArgoCD application monitoring and alertsPaul Buetow
This implements monitoring for ALL services deployed via ArgoCD by leveraging ArgoCD's native Prometheus metrics instead of scraping individual services. Changes: - Created ArgoCD application alerts for health and sync status monitoring - Alert when applications are unhealthy (Degraded, Missing, Unknown, Suspended) - Alert when applications are out of sync for >10 minutes - Alert when sync operations are failing repeatedly - Alert when applications are stuck in Progressing state - Added recording rules for unhealthy/out-of-sync application counts - Added radicale health monitoring via scrape config - Added radicale to additional-scrape-configs for direct health checks - Monitors radicale web interface availability Benefits: - Single monitoring solution for all 21 ArgoCD-managed applications - Automatic monitoring for new applications added to ArgoCD - Early detection of configuration drift and deployment issues - Centralized alerting with actionable remediation steps Monitored applications include: radicale, registry, alloy, grafana, loki, prometheus, tempo, anki-sync-server, audiobookshelf, filebrowser, immich, keybr, kobo-sync-server, miniflux, opodsync, and more. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Reorganize argocd-apps by namespace for better structurePaul Buetow
- Create subdirectories: monitoring/, services/, infra/, test/ - Move 6 monitoring apps to monitoring/ - Move 13 service apps to services/ - Move 1 infra app to infra/ - Move 1 test app to test/ - Add README.md documenting the structure and usage This organization: - Makes it easier to understand which apps belong to which namespace - Allows applying apps by namespace: kubectl apply -f argocd-apps/monitoring/ - Supports namespace-scoped app-of-apps patterns - Provides better clarity when browsing the repository All 21 applications remain functional and validated with kubectl --dry-run.
2026-01-07Migrate Grafana-Ingress to ArgoCD GitOpsPaul Buetow
- Created ArgoCD Application for grafana-ingress - Simple custom Helm chart exposing Grafana via Traefik - Updated Justfile with ArgoCD commands - Status: Synced and Healthy - Ingress working at https://grafana.f3s.buetow.org 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate Prometheus to ArgoCD GitOpsPaul Buetow
- Successfully migrated kube-prometheus-stack to ArgoCD - Multi-source Application: upstream chart + manifests directory - PostSync hook automatically restarts Grafana to reload datasources - All recording rules applied (FreeBSD, OpenBSD, ZFS) - All dashboards provisioned - Grafana datasources configured (Prometheus, Loki, Tempo, Alertmanager) - Updated Justfile with ArgoCD commands - Status: Synced and Healthy - Grafana restarted successfully by PostSync hook 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Prepare Prometheus for ArgoCD GitOps migrationPaul Buetow
- Created manifests/ directory with all additional resources - Added sync wave annotations for proper ordering - Created PostSync hook for Grafana pod restart - Converted additional-scrape-configs to Kubernetes Secret - Organized: PVs (wave 0), Secrets/ConfigMaps (wave 1), PrometheusRules (wave 3), Dashboards (wave 4), Hook (wave 10) - Created multi-source ArgoCD Application (upstream chart + manifests) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate Loki and Alloy to ArgoCD GitOpsPaul Buetow
- Created two ArgoCD Application manifests (loki and alloy) - Updated Justfile with ArgoCD commands for both apps - Loki: log aggregation (SingleBinary mode, 10Gi storage) - Alloy: log collection DaemonSet + OTLP receiver for traces - Both apps are Synced and Healthy - Alloy forwards logs to Loki and traces to Tempo 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate Tempo to ArgoCD GitOpsPaul Buetow
- Created ArgoCD Application manifest for Tempo - Updated Justfile with ArgoCD commands (sync, argocd-status, restart) - Tested delete/re-deploy workflow - Verified Tempo is Synced and Healthy - OTLP receivers enabled on ports 4317 (gRPC) and 4318 (HTTP) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate remaining 4 apps: example-apache-volume-claim, registry, ↵Paul Buetow
pushgateway, immich Apps migrated in this commit: - example-apache-volume-claim (test namespace, 2 replicas, 1 PVC) - registry (infra namespace, Docker registry, 1 PVC) - pushgateway (monitoring namespace, Prometheus metrics) - immich (multi-component: server, postgres, valkey, ML) Also: - Deleted unused example-apache directory - Updated all Justfiles with ArgoCD commands - All apps synced and healthy Progress: 16/22 active apps (73%) Remaining apps (all in monitoring namespace): - prometheus (kube-prometheus-stack) - loki (umbrella chart) - tempo - grafana-ingress 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate 8 remaining simple service apps to ArgoCDPaul Buetow
Apps migrated in this commit: - anki-sync-server (custom images, 1 PVC) - syncthing (file sync, 2 PVCs) - audiobookshelf (3 PVCs) - radicale (CalDAV/CardDAV) - opodsync (podcast sync, 2-container pod) - kobo-sync-server (eReader sync) - filebrowser (3 PVCs) - webdav (WebDAV server) All apps: - Created ArgoCD Application manifests - Updated Justfiles with ArgoCD commands - All synced successfully and healthy - Zero downtime migrations Also includes: - Updated migration progress tracker (12/23 apps, 52%) - Deleted freshrss directory (app no longer needed) Progress: 12/23 apps (52%) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Migrate wallabag and keybr to ArgoCD GitOpsPaul Buetow
- Added ArgoCD Application manifests for wallabag and keybr - Updated Justfiles to use ArgoCD commands (sync, argocd-status, restart) - Removed Helm commands (install, upgrade, delete) - Tested delete/re-deploy workflow for both apps - All resources sync successfully, zero downtime Apps migrated: 4/23 (17%) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Test GitOps: Scale frontend to 2 replicasPaul Buetow
Testing ArgoCD auto-sync functionality by scaling the tracing-demo frontend deployment from 1 to 2 replicas. This validates the complete GitOps workflow: commit → push → auto-sync → deployment. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-03Enable WebSocket support in relayd for audiobookshelfPaul Buetow
- Add http websockets directive to relayd.conf.tpl to allow WebSocket upgrade connections - Fix "Socket failed to connect" error in audiobookshelf web interface - Also add immich helm chart configuration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-31Document Admin API and updated out-of-order configurationPaul Buetow
Updated Prometheus documentation to reflect current configuration: - Added web.enable-admin-api flag documentation - Updated outOfOrderTimeWindow from 720h to 744h (31 days) - Added Data Deletion section with cleanup script usage - Documented manual deletion via Admin API endpoints Provides complete guide for data cleanup after benchmark testing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-31UpdatePaul Buetow
2025-12-31Enable Prometheus admin API for data deletionPaul Buetow
Added web.enable-admin-api flag to allow selective deletion of time series data via the /api/v1/admin/tsdb endpoints. This enables cleanup of benchmark data using the delete_series and clean_tombstones APIs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-31Increase Prometheus out-of-order window to 31 daysPaul Buetow
Changes: - outOfOrderTimeWindow: 720h → 744h (30 days → 31 days) Rationale: Provides 1-day buffer for 30-day backfill operations to avoid edge case rejections where the oldest samples exceed the limit due to timing variations between data generation and ingestion. With this configuration: - 30-day benchmarks achieve 99.85% success rate (vs 50% with 720h) - Only 4/2592 batches rejected (first few batches slightly over 30d) - Allows safe backfilling of up to 30 days of historic data 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-31Enable Prometheus historic data ingestion with out-of-order supportPaul Buetow
This commit configures Prometheus to accept historic data via the Remote Write API, enabling backfilling of test metrics for development and troubleshooting purposes. Changes: - Enable Remote Write receiver (--web.enable-remote-write-receiver) - Enable out-of-order ingestion with 30-day window (720h) - Enable exemplar-storage and otlp-write-receiver features - Add Epimetheus dashboard ConfigMap for Grafana provisioning - Remove old prometheus-pusher directory (moved to separate repo) - Document configuration, use cases, and performance considerations Configuration allows backfilling data up to 30 days in the past, supporting tools like Epimetheus for generating synthetic historic metrics. Performance note: This is optimized for ad-hoc troubleshooting, not production use. Out-of-order ingestion increases memory usage, TSDB overhead, and may impact query performance. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Enable Prometheus Remote Write receiver for historic data ingestionPaul Buetow
Updated persistence-values.yaml to enable the Remote Write receiver using the correct flag for Prometheus 3.x: - Changed from enableFeatures (not supported in 3.8.1) - To additionalArgs with web.enable-remote-write-receiver This allows Epimetheus to push historic data with preserved timestamps via the Prometheus Remote Write API endpoint (/api/v1/write). Applied via: just upgrade 🤖 Generated with Claude Code Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Consolidate all documentation into single comprehensive README.mdPaul Buetow
- Merged content from 10 separate .md files into README.md - Removed: ANSWER.md, AUTO-MODE.md, DASHBOARD.md, HISTORIC.md, LIMITATIONS.md, QUERY_EXAMPLES.md, QUICK-START.md, SETUP-COMPLETE.md, SUMMARY.md, USAGE.md - README.md now includes: * Quick start guide * All operating modes (realtime, historic, backfill, auto) * Data formats (CSV, JSON) * Test metrics documentation * Grafana dashboard setup * Example queries and curl commands * Time range limitations * Troubleshooting guide * Architecture diagram * Best practices 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-30Rename test metrics with clear prefix and add Grafana dashboardPaul Buetow
Renamed all test metrics with "prometheus_pusher_test_" prefix to clearly indicate they are generated by the prometheus-pusher testing/demo functionality. Metric renaming: - app_requests_total → prometheus_pusher_test_requests_total - app_active_connections → prometheus_pusher_test_active_connections - app_temperature_celsius → prometheus_pusher_test_temperature_celsius - app_request_duration_seconds → prometheus_pusher_test_request_duration_seconds - app_jobs_processed_total → prometheus_pusher_test_jobs_processed_total Grafana Dashboard: - Created comprehensive dashboard with 8 panels - Request rate and total requests visualization - Active connections gauge (0-100 with thresholds) - Temperature gauge (0-50°C with thresholds) - Request duration percentiles (p50, p90, p99) - Average request duration stat - Jobs processed by type (bar gauge) - Jobs status breakdown table - Auto-refresh every 10s, 15-minute default time range Files added: - grafana-dashboard.json: Dashboard definition - deploy-dashboard.sh: Automated deployment script - DASHBOARD.md: Complete dashboard documentation Updated: - internal/metrics/generator.go: Renamed metric names - internal/ingester/remotewrite.go: Updated historic metric names - internal/ingester/remotewrite_test.go: Updated test expectations Tests updated and passing with 63.9% coverage maintained. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
generated by cgit v1.2.3 (git 2.25.1) at 2026-08-12 07:27:57 +0000