summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2026-01-09Migrate all applications from Codeberg to self-hosted gitPaul Buetow
Updated 17 application manifests to use internal git-server: - Monitoring: grafana-ingress, prometheus, pushgateway - Services: anki-sync-server, audiobookshelf, filebrowser, immich, keybr, kobo-sync-server, miniflux, opodsync, radicale, syncthing, tracing-demo, wallabag, webdav - Infra: registry All applications now fetch from: ssh://git@git-server.cicd.svc.cluster.local/repos/repos/conf.git Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Migrate example-apache-volume-claim to self-hosted gitPaul Buetow
Test migration of first application from Codeberg to internal git-server. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Reduce SSH logging from DEBUG3 to INFOPaul Buetow
Debug logging was useful for troubleshooting but not needed in production. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Unlock git user account for SSH authenticationPaul Buetow
SSH requires user accounts to be unlocked (not have ! in shadow). Use 'passwd -u' to unlock the git user account. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Make .ssh directory world-readable for SELinux compatibilityPaul Buetow
SELinux prevents root from accessing 700 directories in some contexts. Use 755 for directory and 644 for authorized_keys to allow access. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Remove readOnly flag from git-ssh-writable mountPaul Buetow
The emptyDir needs to be writable for SSH to access authorized_keys. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Add git-shell to /etc/shells for SSH validityPaul Buetow
SSH requires user shells to be listed in /etc/shells. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Fix nested .ssh directory issuePaul Buetow
Remove extra .ssh directory creation in initContainer. The emptyDir mount point itself is /home/git/.ssh. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Fix authorized_keys permissions via initContainerPaul Buetow
Copy authorized_keys from secret to emptyDir with git user ownership. This allows SSH to read the keys for authentication. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Add SETGID and SETUID capabilities to git-serverPaul Buetow
SSH privilege separation requires setgroups() and setuid() syscalls. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Add SSH connectivity for git-server and configure ArgoCDPaul Buetow
- Enable SYS_CHROOT capability for git-server SSH to work in containers - Configure ArgoCD repo-server to use SSH key for git-server access - Set DEBUG3 logging in sshd for troubleshooting (temp) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Fix fcgiwrap socket permissions for nginx workersPaul Buetow
Nginx workers (running as user nginx) couldn't connect to fcgiwrap.sock created by root. Add chmod 666 to make the socket world-accessible.
2026-01-09Fix cgit port mapping - nginx listens on port 80Paul Buetow
The cgit nginx default.conf listens on port 80, not 8080. Update: - Container port from 8080 to 80 - Service targetPort from 8080 to 80
2026-01-09Add SETGID and SETUID capabilities to cgit containerPaul Buetow
Nginx workers need these capabilities to drop privileges. Add SETGID and SETUID to allow nginx workers to start properly.
2026-01-09Remove fsGroup to fix nginx worker setgid errorsPaul Buetow
The fsGroup: 1000 was causing nginx workers to fail with setgid errors even after removing 'user nginx;' directive. Since both containers run as root, fsGroup is not needed for repo access.
2026-01-09Remove 'user nginx;' directive to fix nginx worker errorsPaul Buetow
When running as root with fsGroup, nginx workers fail trying to setgid. Remove the 'user nginx;' directive from nginx.conf at startup using sed. This allows nginx to run workers as root without permission errors.
2026-01-09Run spawn-fcgi as root to avoid setgid errorsPaul Buetow
Remove -u nginx -g nginx from spawn-fcgi command to run as root. This avoids nginx worker process setgid permission errors.
2026-01-09Override cgit entrypoint to skip chown/chmodPaul Buetow
The cgit image entrypoint always tries to chown /var/cache/cgit which fails with permission errors. Override the entrypoint to directly: 1. Spawn fcgiwrap as nginx user 2. Start nginx in foreground This skips the problematic chown/chmod and template substitution.
2026-01-09Disable cgit caching to avoid permission issuesPaul Buetow
Instead of fighting permission issues with the cgit cache directory, disable caching entirely by: - Setting cache-size=0 in cgitrc - Removing cgit-cache emptyDir volume and mounts - Simplifying initContainer (only SSH keys setup needed) cgit will work without caching, just slightly slower for large repos.
2026-01-09Run cgit as root - required for nginx and spawn-fcgiPaul Buetow
cgit image needs root to: - Bind sockets with spawn-fcgi - Run nginx master process - Write to /var/run/nginx.pid The initContainer already sets up cache dir with proper permissions.
2026-01-09Set USE_CUSTOM_CONFIG=true to skip cgit template substitutionPaul Buetow
The cgit entrypoint tries to write to /etc/cgitrc which is mounted read-only from our ConfigMap. Set USE_CUSTOM_CONFIG=true to use our custom cgitrc directly without template substitution.
2026-01-09Fix cgit permissions - use UID 101 (nginx user)Paul Buetow
The cgit image runs as nginx user (UID 101), not www-data (UID 33). - Update initContainer to chown cache to 101:1000 - Update cgit securityContext to runAsUser: 101
2026-01-09Fix permissions using fsGroup and initContainer patternPaul Buetow
Follow webdav/filebrowser pattern for proper permission handling: - Add fsGroup: 1000 at pod level for git repo access - Add initContainer to chown emptyDir volumes - Run git-server as root (required for sshd) - Run cgit as user 33 (www-data) - Restore cgit-cache emptyDir volume with proper ownership
2026-01-09Run containers as root and use emptyDir for writeable dirsPaul Buetow
- Mount emptyDir for /etc/ssh to allow SSH host key generation - Mount emptyDir for /var/cache/cgit to allow cache initialization - Run both containers as root with proper capabilities - Copy sshd_config at runtime from /tmp to /etc/ssh - Add imagePullPolicy: Always to force image refresh
2026-01-09Fix SSH host keys and container securityPaul Buetow
- Generate SSH host keys at runtime via entrypoint script - Remove fsGroup security context to fix emptyDir permissions - Allow cgit to initialize cache directory as root
2026-01-09Fix sshd_config and cgit permissionsPaul Buetow
- Remove unsupported UsePAM option from sshd_config - Run cgit as root to allow cache directory initialization - Add CHOWN and DAC_OVERRIDE capabilities for cgit
2026-01-09Fix git-server deploymentPaul Buetow
- Use registry.lan.buetow.org for deployment (internal DNS) - Add emptyDir volume for cgit cache directory - Add README.md with deployment and secret management instructions This fixes image pull issues and cgit permission errors.
2026-01-09Add self-hosted git server with SSH and cgit web UIPaul Buetow
Deploy a self-hosted git repository solution to replace external Codeberg dependency. Components: - SSH git server: Alpine-based container with OpenSSH and git - cgit web UI: Browse repositories at cgit.f3s.buetow.org - Single pod design: git-server + cgit containers sharing storage Infrastructure: - Docker image in git-server/docker-image/ with Justfile build automation - Helm chart in git-server/helm-chart/ for Kubernetes deployment - 5Gi ReadWriteMany PVC for NFS-backed repository storage - ClusterIP service for ArgoCD internal access - NodePort 30022 for external SSH push access - Traefik ingress for cgit web UI ArgoCD Application manifest deployed to cicd namespace. Note: SSH keys must be created as Kubernetes secrets manually, not in git. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-09Test: verify SSH push worksPaul Buetow
2026-01-09add cgitPaul Buetow
2026-01-08Enable prune for prometheus ArgoCD app, update READMEPaul Buetow
2026-01-08prunePaul Buetow
2026-01-08Remove invalid radicale scrape job (no metrics endpoint)Paul Buetow
2026-01-08Fix: disable kubeScheduler rules entirelyPaul Buetow
2026-01-08Disable KubeProxyDown and KubeSchedulerDown alerts for k3sPaul Buetow
2026-01-08Add PrometheusHosts to gogios config for f3s cluster alertsPaul Buetow
Amp-Thread-ID: https://ampcode.com/threads/T-019b9eec-b607-7271-9b75-f05255a60742 Co-authored-by: Amp <amp@ampcode.com>
2026-01-08add agentsPaul Buetow
2026-01-08Add Prometheus NodePort and alert query targets to JustfilePaul Buetow
2026-01-08Add NodePort service for Prometheus on port 30090Paul Buetow
2026-01-08renamePaul Buetow
2026-01-08Add volumeName to PVC for explicit bindingPaul Buetow
2026-01-08Change apache PV/PVC to ReadWriteMany for multi-pod accessPaul Buetow
2026-01-08Disable kube-proxy and kube-scheduler monitoring for k3sPaul Buetow
K3s embeds kube-proxy and kube-scheduler functionality into the main k3s server process, unlike standard Kubernetes where they run as separate components. This change disables monitoring for these components to prevent false-positive critical alerts: - KubeProxyDown - KubeSchedulerDown These alerts were firing because kube-prometheus-stack expects standard Kubernetes architecture with separate kube-proxy and kube-scheduler pods/processes. Cluster info: - Running k3s v1.32.6+k3s1 - 3 control-plane nodes (r0, r1, r2) - Components embedded in k3s binary Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-08Add convenient port-forward targets for Prometheus monitoringPaul Buetow
Added enhanced port-forward targets with helpful UI information: - 'just alerts' - Quick access to Prometheus alerts view - 'just alertmanager' - Quick access to Alertmanager UI - Enhanced output showing all relevant URLs All port-forward commands now display: - Access URLs with direct links to specific views - Clear instructions for stopping (Ctrl+C) Usage: cd prometheus/ just alerts # Opens Prometheus alerts (port 9090) just alertmanager # Opens Alertmanager (port 9093) just port-forward-prometheus [port] just port-forward-grafana [port] After running, access: - Prometheus Alerts: http://localhost:9090/alerts - Alertmanager: http://localhost:9093 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-08Configure Alertmanager routing for ArgoCD application alertsPaul Buetow
Added Alertmanager configuration to: - Route ArgoCD application alerts to dedicated 'argocd-alerts' receiver - Group ArgoCD alerts by alertname, name (app name), and severity - Faster alert grouping for ArgoCD (10s wait vs 30s default) - Repeat ArgoCD alerts every 6 hours - Suppress Watchdog test alerts - Configure inhibit rules to prevent alert spam Alerts are visible in: - Prometheus UI: http://localhost:9090/alerts - Alertmanager UI: http://localhost:9093 - Grafana dashboard: ArgoCD Applications - Health & Sync Status This ensures critical application issues are properly routed and visible in the monitoring UI for immediate action. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-08Update CLAUDE.md instruction formatPaul Buetow
2026-01-08Add Grafana dashboard for ArgoCD applications monitoringPaul Buetow
Created comprehensive Grafana dashboard showing: - Total applications count - Healthy vs unhealthy applications - Out-of-sync status - Detailed table with all applications and their status - Health status timeline graph - Sync operations rate - Active ArgoCD-related alerts Dashboard will auto-load in Grafana via ConfigMap with label grafana_dashboard='1' Access at: https://grafana.f3s.buetow.org → Dashboards → ArgoCD Applications Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-08Add comprehensive ArgoCD application monitoring and alertsPaul Buetow
This implements monitoring for ALL services deployed via ArgoCD by leveraging ArgoCD's native Prometheus metrics instead of scraping individual services. Changes: - Created ArgoCD application alerts for health and sync status monitoring - Alert when applications are unhealthy (Degraded, Missing, Unknown, Suspended) - Alert when applications are out of sync for >10 minutes - Alert when sync operations are failing repeatedly - Alert when applications are stuck in Progressing state - Added recording rules for unhealthy/out-of-sync application counts - Added radicale health monitoring via scrape config - Added radicale to additional-scrape-configs for direct health checks - Monitors radicale web interface availability Benefits: - Single monitoring solution for all 21 ArgoCD-managed applications - Automatic monitoring for new applications added to ArgoCD - Early detection of configuration drift and deployment issues - Centralized alerting with actionable remediation steps Monitored applications include: radicale, registry, alloy, grafana, loki, prometheus, tempo, anki-sync-server, audiobookshelf, filebrowser, immich, keybr, kobo-sync-server, miniflux, opodsync, and more. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-01-07Reorganize argocd-apps by namespace for better structurePaul Buetow
- Create subdirectories: monitoring/, services/, infra/, test/ - Move 6 monitoring apps to monitoring/ - Move 13 service apps to services/ - Move 1 infra app to infra/ - Move 1 test app to test/ - Add README.md documenting the structure and usage This organization: - Makes it easier to understand which apps belong to which namespace - Allows applying apps by namespace: kubectl apply -f argocd-apps/monitoring/ - Supports namespace-scoped app-of-apps patterns - Provides better clarity when browsing the repository All 21 applications remain functional and validated with kubectl --dry-run.
2026-01-07Migrate Grafana-Ingress to ArgoCD GitOpsPaul Buetow
- Created ArgoCD Application for grafana-ingress - Simple custom Helm chart exposing Grafana via Traefik - Updated Justfile with ArgoCD commands - Status: Synced and Healthy - Ingress working at https://grafana.f3s.buetow.org 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
generated by cgit v1.2.3 (git 2.25.1) at 2026-08-10 17:45:31 +0000