Update content for html

1 files changed, 530 insertions, 682 deletions

diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html
index 5d58d4c5..2ce86ad7 100644
--- a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html
+++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html
@@ -42,49 +42,32 @@
 <li>⇢ ⇢ <a href='#accessing-argocd'>Accessing ArgoCD</a></li>
 <li>⇢ <a href='#argocd-application-structure'>ArgoCD Application Structure</a></li>
 <li>⇢ <a href='#repository-organization'>Repository Organization</a></li>
-<li>⇢ <a href='#migration-strategy-incremental-one-app-at-a-time'>Migration Strategy: Incremental, One App at a Time</a></li>
 <li>⇢ ⇢ <a href='#migration-phases'>Migration Phases</a></li>
 <li>⇢ <a href='#example-migration-miniflux'>Example Migration: Miniflux</a></li>
 <li>⇢ ⇢ <a href='#before-imperative-helm-deployment'>Before: Imperative Helm deployment</a></li>
 <li>⇢ ⇢ <a href='#after-declarative-gitops-with-argocd'>After: Declarative GitOps with ArgoCD</a></li>
 <li>⇢ ⇢ <a href='#migration-procedure'>Migration procedure</a></li>
-<li>⇢ <a href='#complex-migration-prometheus-with-multi-source'>Complex Migration: Prometheus with Multi-Source</a></li>
-<li>⇢ ⇢ <a href='#sync-waves-and-hooks'>Sync Waves and Hooks</a></li>
-<li>⇢ <a href='#migration-results'>Migration Results</a></li>
-<li>⇢ <a href='#benefits-realized'>Benefits Realized</a></li>
-<li>⇢ ⇢ <a href='#1-single-source-of-truth'>1. Single Source of Truth</a></li>
-<li>⇢ ⇢ <a href='#2-automatic-synchronization'>2. Automatic Synchronization</a></li>
-<li>⇢ ⇢ <a href='#3-drift-detection-and-self-healing'>3. Drift Detection and Self-Healing</a></li>
-<li>⇢ ⇢ <a href='#4-easy-rollbacks'>4. Easy Rollbacks</a></li>
-<li>⇢ ⇢ <a href='#5-disaster-recovery'>5. Disaster Recovery</a></li>
-<li>⇢ ⇢ <a href='#6-documentation-by-default'>6. Documentation by Default</a></li>
-<li>⇢ ⇢ <a href='#7-safe-experimentation'>7. Safe Experimentation</a></li>
-<li>⇢ <a href='#challenges-and-solutions'>Challenges and Solutions</a></li>
-<li>⇢ ⇢ <a href='#challenge-1-helm-release-adoption'>Challenge 1: Helm Release Adoption</a></li>
-<li>⇢ ⇢ <a href='#challenge-2-persistent-volumes-not-tracked-by-helm'>Challenge 2: Persistent Volumes Not Tracked by Helm</a></li>
-<li>⇢ ⇢ <a href='#challenge-3-secrets-management'>Challenge 3: Secrets Management</a></li>
-<li>⇢ ⇢ <a href='#challenge-4-grafana-not-reloading-datasources'>Challenge 4: Grafana Not Reloading Datasources</a></li>
-<li>⇢ ⇢ <a href='#challenge-5-prometheus-with-multiple-sources'>Challenge 5: Prometheus With Multiple Sources</a></li>
-<li>⇢ ⇢ <a href='#challenge-6-sync-ordering-for-prometheus'>Challenge 6: Sync Ordering for Prometheus</a></li>
-<li>⇢ <a href='#justfile-evolution'>Justfile Evolution</a></li>
-<li>⇢ <a href='#lessons-learned'>Lessons Learned</a></li>
-<li>⇢ <a href='#future-improvements'>Future Improvements</a></li>
-<li>⇢ ⇢ <a href='#1-external-secrets-operator'>1. External Secrets Operator</a></li>
-<li>⇢ ⇢ <a href='#2-applicationset-for-similar-apps'>2. ApplicationSet for Similar Apps</a></li>
-<li>⇢ ⇢ <a href='#3-app-of-apps-pattern'>3. App-of-Apps Pattern</a></li>
-<li>⇢ ⇢ <a href='#4-argocd-image-updater'>4. ArgoCD Image Updater</a></li>
-<li>⇢ <a href='#summary'>Summary</a></li>
+<li><a href='#argocd-detects-change-within-3-minutes-and-syncs-automatically'>ArgoCD detects change within 3 minutes and syncs automatically</a></li>
+<li><a href='#argocd-detects-drift-within-3-minutes'>ArgoCD detects drift within 3 minutes</a></li>
+<li><a href='#argocd-automatically-rolls-back-to-the-previous-state'>ArgoCD automatically rolls back to the previous state</a></li>
+<li><a href='#temporarily-point-argocd-at-the-feature-branch'>Temporarily point ArgoCD at the feature branch</a></li>
+<li><a href='#verify-changes-in-argocd-web-ui'>Verify changes in ArgoCD Web UI</a></li>
+<li><a href='#if-good-merge-to-master'>If good: merge to master</a></li>
+<li><a href='#if-bad-revert-the-patch'>If bad: revert the patch</a></li>
+<li><a href='#root-monitoringyaml'>root-monitoring.yaml</a></li>
+<li><a href='#root-app-deploys-all-21-applications-automatically'>Root app deploys all 21 applications automatically</a></li>
+<li><a href='#or-apply-by-namespace'>Or apply by namespace</a></li>
 </ul><br />
 <h2 style='display: inline' id='introduction'>Introduction</h2><br />
 <br />
 <span>In the previous posts, I deployed applications to the k3s cluster using Helm charts and Justfiles—running <span class='inlinecode'>just install</span> or <span class='inlinecode'>just upgrade</span> to imperatively push changes to the cluster. While this approach works, it has several drawbacks:</span><br />
 <br />
 <ul>
-<li>**No single source of truth**: The cluster state depends on which commands were run and when</li>
-<li>**Manual synchronization**: Every change requires manually running commands</li>
-<li>**Drift detection is hard**: No easy way to know if cluster state matches the desired configuration</li>
-<li>**Rollback complexity**: Rolling back changes means re-running old Helm commands</li>
-<li>**No audit trail**: Hard to track who changed what and when</li>
+<li>No single source of truth: The cluster state depends on which commands were run and when</li>
+<li>Manual synchronization: Every change requires manually running commands</li>
+<li>Drift detection is hard: No easy way to know if cluster state matches the desired configuration</li>
+<li>Rollback complexity: Rolling back changes means re-running old Helm commands</li>
+<li>No audit trail: Hard to track who changed what and when</li>
 </ul><br />
 <span>This blog post covers the migration from imperative Helm deployments to declarative GitOps using ArgoCD. After this migration, the Git repository becomes the single source of truth, and ArgoCD automatically ensures the cluster matches what&#39;s defined in Git.</span><br />
 <br />
@@ -95,18 +78,19 @@
 <span>Key principles:</span><br />
 <br />
 <ul>
-<li>**Declarative**: The system&#39;s desired state is described declaratively (YAML manifests, Helm values)</li>
-<li>**Versioned and immutable**: All changes are committed to Git, providing a complete history</li>
-<li>**Pulled automatically**: An agent in the cluster continuously pulls the desired state from Git</li>
-<li>**Continuously reconciled**: The agent ensures the actual state matches the desired state, automatically correcting drift</li>
+<li>Declarative: The system&#39;s desired state is described declaratively (YAML manifests, Helm values)</li>
+<li>Versioned and immutable: All changes are committed to Git, providing a complete history</li>
+<li>Pulled automatically: An agent in the cluster continuously pulls the desired state from Git</li>
+<li>Continuously reconciled: The agent ensures the actual state matches the desired state, automatically correcting drift</li>
 </ul><br />
 <span>For Kubernetes, this means:</span><br />
 <br />
-<span>1. All manifests, Helm charts, and configuration live in a Git repository</span><br />
-<span>2. A tool (ArgoCD in our case) watches the repository</span><br />
-<span>3. When changes are pushed to Git, ArgoCD automatically applies them to the cluster</span><br />
-<span>4. If someone manually changes resources in the cluster, ArgoCD detects the drift and can automatically revert it</span><br />
-<br />
+<ul>
+<li>1. All manifests, Helm charts, and configuration live in a Git repository</li>
+<li>2. A tool (ArgoCD in our case) watches the repository</li>
+<li>3. When changes are pushed to Git, ArgoCD automatically applies them to the cluster</li>
+<li>4. If someone manually changes resources in the cluster, ArgoCD detects the drift and can automatically revert it</li>
+</ul><br />
 <h2 style='display: inline' id='what-is-argocd'>What is ArgoCD?</h2><br />
 <br />
 <span>ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. It&#39;s implemented as a Kubernetes controller that continuously monitors running applications and compares the current, live state against the desired target state defined in Git.</span><br />
@@ -116,32 +100,34 @@
 <span>Key features:</span><br />
 <br />
 <ul>
-<li>**Automated deployment**: Monitors Git repositories and automatically syncs changes to the cluster</li>
-<li>**Application definitions**: Defines applications as CRDs (Custom Resource Definitions)</li>
-<li>**Health assessment**: Understands Kubernetes resources and can determine if an application is healthy</li>
-<li>**Web UI and CLI**: Provides both a web interface and command-line tool for managing applications</li>
-<li>**RBAC**: Role-based access control for team collaboration</li>
-<li>**SSO integration**: Can integrate with existing authentication systems</li>
-<li>**Multi-cluster support**: Can manage applications across multiple Kubernetes clusters</li>
-<li>**Sync waves and hooks**: Control the order of resource deployment and run jobs at specific lifecycle points</li>
+<li>Automated deployment: Monitors Git repositories and automatically syncs changes to the cluster</li>
+<li>Application definitions: Defines applications as CRDs (Custom Resource Definitions)</li>
+<li>Health assessment: Understands Kubernetes resources and can determine if an application is healthy</li>
+<li>Web UI and CLI: Provides both a web interface and command-line tool for managing applications</li>
+<li>RBAC: Role-based access control for team collaboration</li>
+<li>SSO integration: Can integrate with existing authentication systems</li>
+<li>Multi-cluster support: Can manage applications across multiple Kubernetes clusters</li>
+<li>Sync waves and hooks: Control the order of resource deployment and run jobs at specific lifecycle points</li>
 </ul><br />
 <h2 style='display: inline' id='why-argocd-for-f3s'>Why ArgoCD for f3s?</h2><br />
 <br />
 <span>For a home lab cluster, ArgoCD provides several benefits:</span><br />
 <br />
-<span>**Disaster recovery**: If the entire cluster is lost, I can rebuild it by:</span><br />
-<span>1. Bootstrapping a new k3s cluster</span><br />
-<span>2. Installing ArgoCD</span><br />
-<span>3. Pointing ArgoCD at the Git repository</span><br />
-<span>4. All applications automatically deploy to the desired state</span><br />
+<span>Disaster recovery: If the entire cluster is lost, I can rebuild it by:</span><br />
 <br />
-<span>**Experimentation safety**: I can test changes in a separate Git branch without affecting the running cluster. Once validated, merge to master and ArgoCD applies the changes.</span><br />
-<br />
-<span>**Drift detection**: If I manually change something in the cluster (for debugging), ArgoCD shows the difference and can automatically revert it.</span><br />
+<ul>
+<li>1. Bootstrapping a new k3s cluster</li>
+<li>2. Installing ArgoCD</li>
+<li>3. Pointing ArgoCD at the Git repository</li>
+<li>4. All applications automatically deploy to the desired state</li>
+</ul><br />
+<span>Experimentation safety: I can test changes in a separate Git branch without affecting the running cluster. Once validated, merge to master and ArgoCD applies the changes.</span><br />
+<span> </span><br />
+<span>Drift detection: If I manually change something in the cluster (for debugging), ArgoCD shows the difference and can automatically revert it.</span><br />
 <br />
-<span>**Declarative configuration**: The Git repository documents the entire cluster configuration. No need to remember which <span class='inlinecode'>just</span> commands to run or in which order.</span><br />
+<span>Declarative configuration: The Git repository documents the entire cluster configuration. No need to remember which <span class='inlinecode'>just</span> commands to run or in which order.</span><br />
 <br />
-<span>**Automatic sync**: Push to Git, and changes deploy automatically. No need to SSH to a workstation and run Helm commands.</span><br />
+<span>Automatic sync: Push to Git, and changes deploy automatically. No need to SSH to a workstation and run Helm commands.</span><br />
 <br />
 <h2 style='display: inline' id='deploying-argocd'>Deploying ArgoCD</h2><br />
 <br />
@@ -187,7 +173,7 @@ STATUS: deployed
 <br />
 <span>The <span class='inlinecode'>values.yaml</span> file configures several important aspects:</span><br />
 <br />
-<span>**Persistent storage for the repo-server**: ArgoCD clones Git repositories to cache them locally. I configured a persistent volume so the cache survives pod restarts:</span><br />
+<span>Persistent storage for the repo-server: ArgoCD clones Git repositories to cache them locally. I configured a persistent volume so the cache survives pod restarts:</span><br />
 <br />
 <pre>
 repoServer:
@@ -200,7 +186,7 @@ repoServer:
       mountPath: /tmp
 </pre>
 <br />
-<span>**Admin password preservation**: By default, the admin password is auto-generated and stored in a secret. To ensure it persists across Helm upgrades:</span><br />
+<span>Admin password preservation: By default, the admin password is auto-generated and stored in a secret. To ensure it persists across Helm upgrades:</span><br />
 <br />
 <pre>
 configs:
@@ -222,7 +208,7 @@ $ kubectl create secret generic argocd-secret \
 $ echo <font color="#808080">"ArgoCD admin password: $ARGOCD_ADMIN_PASSWORD"</font>
 </pre>
 <br />
-<span>**Server configuration**: Enabled insecure mode since TLS is handled by the OpenBSD edge relays:</span><br />
+<span>Server configuration: Enabled insecure mode since TLS is handled by the OpenBSD edge relays:</span><br />
 <br />
 <pre>
 server:
@@ -261,7 +247,7 @@ metadata:
     traefik.ingress.kubernetes.io/router.entrypoints: web
 spec:
   rules:
-    - host: argocd.f3s.buetow.org
+    - host: argocd.f3s.foo.zone
       http:
         paths:
           - path: /
@@ -275,15 +261,13 @@ spec:
 <br />
 <span>Following the same pattern as other services, the OpenBSD edge relays terminate TLS and forward traffic through WireGuard to the cluster. ArgoCD is now accessible at:</span><br />
 <br />
-<a class='textlink' href='https://argocd.f3s.buetow.org'>ArgoCD Web UI</a><br />
-<br />
 <span>The ArgoCD CLI can also be used for operations:</span><br />
 <br />
 <!-- Generator: GNU source-highlight 3.1.9
 by Lorenzo Bettini
 http://www.lorenzobettini.it
 http://www.gnu.org/software/src-highlite -->
-<pre>$ argocd login argocd.f3s.buetow.org
+<pre>$ argocd login argocd.f3s.foo.zone
 $ argocd app list
 </pre>
 <br />
@@ -292,15 +276,13 @@ $ argocd app list
 <span>ArgoCD uses a CRD called <span class='inlinecode'>Application</span> to define what should be deployed. Each application specifies:</span><br />
 <br />
 <ul>
-<li>**Source**: Where the manifests live (Git repo, Helm chart repository, or both)</li>
-<li>**Destination**: Which cluster and namespace to deploy to</li>
-<li>**Sync policy**: Whether to automatically sync changes</li>
+<li>Source: Where the manifests live (Git repo, Helm chart repository, or both)</li>
+<li>Destination: Which cluster and namespace to deploy to</li>
 </ul><br />
 <span>Here&#39;s a simple example for the miniflux application:</span><br />
 <br />
 <pre>
-apiVersion: argoproj.io/v1alpha1
-kind: Application
+ind: Application
 metadata:
   name: miniflux
   namespace: cicd
@@ -310,9 +292,11 @@ spec:
   project: default
   source:
     repoURL: https://codeberg.org/snonux/conf.git
+    
     targetRevision: master
     path: f3s/miniflux/helm-chart
   destination:
+  
     server: https://kubernetes.default.svc
     namespace: services
   syncPolicy:
@@ -389,57 +373,44 @@ spec:
 <br />
 <span>The application directories (miniflux, prometheus, etc.) remained mostly unchanged—ArgoCD references the same Helm charts. The main additions:</span><br />
 <br />
-<span>1. **argocd-apps/**: Application manifests organized by Kubernetes namespace for better clarity</span><br />
-<span>   - <span class='inlinecode'>monitoring/</span>: 6 observability applications</span><br />
-<span>   - <span class='inlinecode'>services/</span>: 13 user-facing applications</span><br />
-<span>   - <span class='inlinecode'>infra/</span>: 1 infrastructure application (registry)</span><br />
-<span>   - <span class='inlinecode'>test/</span>: 1 test application</span><br />
-<span>2. ***/manifests/**: Additional Kubernetes manifests for complex apps (like Prometheus)</span><br />
-<span>3. **Justfiles updated**: Changed from <span class='inlinecode'>helm install/upgrade</span> to <span class='inlinecode'>argocd app sync</span></span><br />
-<br />
-<span>This organization makes it easy to apply all applications in a specific namespace or manage them independently.</span><br />
-<br />
-<h2 style='display: inline' id='migration-strategy-incremental-one-app-at-a-time'>Migration Strategy: Incremental, One App at a Time</h2><br />
+<span>1. argocd-apps/: Application manifests organized by Kubernetes namespace for better clarity</span><br />
 <br />
-<span>Rather than attempting a "big bang" migration of all 21 applications at once, I migrated them incrementally:</span><br />
-<br />
-<span>1. **Start with a simple app**: Validate the pattern with a low-risk application</span><br />
-<span>2. **Migrate in waves**: Group similar applications and migrate together</span><br />
-<span>3. **Validate thoroughly**: Ensure each app is healthy before moving to the next</span><br />
-<span>4. **Learn and iterate**: Apply lessons from earlier migrations to later ones</span><br />
+<ul>
+<li><span class='inlinecode'>monitoring/</span>: 6 observability applications</li>
+<li><span class='inlinecode'>services/</span>: 13 user-facing applications</li>
+<li><span class='inlinecode'>infra/</span>: 1 infrastructure application (registry)</li>
+<li><span class='inlinecode'>test/</span>: 1 test application</li>
+</ul><br />
+<span>2. */manifests/: Additional Kubernetes manifests for complex apps (like Prometheus)</span><br />
+<span>3. Justfiles updated: Changed from <span class='inlinecode'>helm install/upgrade</span> to <span class='inlinecode'>argocd app sync</span></span><br />
 <br />
-<span>This approach reduced risk and allowed me to refine the migration process.</span><br />
+<span>This organization makes it easy to apply all applications in a specific namespace or manage them independently.</span><br />
 <br />
 <h3 style='display: inline' id='migration-phases'>Migration Phases</h3><br />
 <br />
-<span>**Phase 1: Simple services** (13 apps)</span><br />
+<span>These apps have straightforward Helm charts with no complex dependencies. Pattern established:</span><br />
+<br />
 <ul>
-<li>miniflux, freshrss, wallabag</li>
-<li>anki-sync-server, kobo-sync-server, opodsync</li>
-<li>radicale, syncthing, audiobookshelf</li>
-<li>filebrowser, keybr, webdav</li>
-<li>example-apache, example-apache-volume-claim</li>
+<li>1. Create Application manifest in <span class='inlinecode'>argocd-apps/</span></li>
+<li>2. Apply with <span class='inlinecode'>kubectl apply -f argocd-apps/&lt;app&gt;.yaml</span></li>
+<li>3. Verify sync status: <span class='inlinecode'>argocd app get &lt;app&gt;</span></li>
+<li>4. Update Justfile to use ArgoCD commands</li>
 </ul><br />
-<span>These apps have straightforward Helm charts with no complex dependencies. Pattern established:</span><br />
-<span>1. Create Application manifest in <span class='inlinecode'>argocd-apps/</span></span><br />
-<span>2. Apply with <span class='inlinecode'>kubectl apply -f argocd-apps/&lt;app&gt;.yaml</span></span><br />
-<span>3. Verify sync status: <span class='inlinecode'>argocd app get &lt;app&gt;</span></span><br />
-<span>4. Update Justfile to use ArgoCD commands</span><br />
+<span>Phase 2: Infrastructure apps (3 apps)</span><br />
 <br />
-<span>**Phase 2: Infrastructure apps** (3 apps)</span><br />
 <ul>
 <li>registry (Docker image registry)</li>
 <li>pushgateway (Prometheus metrics ingestion)</li>
 <li>immich (photo management with complex dependencies)</li>
 </ul><br />
-<span>**Phase 3: Monitoring stack** (4 apps)</span><br />
+<span>Phase 3: Monitoring stack (4 apps)</span><br />
 <ul>
 <li>tempo (distributed tracing)</li>
 <li>loki (log aggregation)</li>
 <li>alloy (log collection)</li>
 <li>prometheus (metrics and monitoring)</li>
 </ul><br />
-<span>**Phase 4: Monitoring addons** (1 app)</span><br />
+<span>Phase 4: Monitoring addons (1 app)</span><br />
 <ul>
 <li>grafana-ingress (separate ingress for Grafana)</li>
 </ul><br />
@@ -553,7 +524,7 @@ logs:
 <br />
 <h3 style='display: inline' id='migration-procedure'>Migration procedure</h3><br />
 <br />
-<span>1. **Backup current state**:</span><br />
+<span>1. Backup current state:</span><br />
 <!-- Generator: GNU source-highlight 3.1.9
 by Lorenzo Bettini
 http://www.lorenzobettini.it
@@ -562,7 +533,7 @@ http://www.gnu.org/software/src-highlite -->
 $ kubectl get all,ingress -n services -o yaml &gt; /tmp/miniflux-backup.yaml
 </pre>
 <br />
-<span>2. **Create Application manifest**:</span><br />
+<span>2. Create Application manifest:</span><br />
 <!-- Generator: GNU source-highlight 3.1.9
 by Lorenzo Bettini
 http://www.lorenzobettini.it
@@ -571,7 +542,7 @@ http://www.gnu.org/software/src-highlite -->
 application.argoproj.io/miniflux created
 </pre>
 <br />
-<span>3. **Verify ArgoCD adopted the resources**:</span><br />
+<span>3. Verify ArgoCD adopted the resources:</span><br />
 <!-- Generator: GNU source-highlight 3.1.9
 by Lorenzo Bettini
 http://www.lorenzobettini.it
@@ -581,7 +552,7 @@ Name:               miniflux
 Project:            default
 Server:             https://kubernetes.default.svc
 Namespace:          services
-URL:                https://argocd.f3s.buetow.org/applications/miniflux
+URL:                https://argocd.f3s.foo.zone/applications/miniflux
 Repo:               https://codeberg.org/snonux/conf.git
 Target:             master
 Path:               f3s/miniflux/helm-chart
@@ -591,7 +562,7 @@ Sync Status:        Synced to master (4e3c216)
 Health Status:      Healthy
 </pre>
 <br />
-<span>4. **Monitor for issues**:</span><br />
+<span>4. Monitor for issues:</span><br />
 <!-- Generator: GNU source-highlight 3.1.9
 by Lorenzo Bettini
 http://www.lorenzobettini.it
@@ -599,623 +570,500 @@ http://www.gnu.org/software/src-highlite -->
 <pre>$ kubectl get pods -n services -l app=miniflux -w
 NAME                                READY   STATUS    RESTARTS   AGE
 miniflux-postgres-556444cb8d-xvv2p  <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          54d
-miniflux-server-85d7c64664-stmt<font color="#000000">9</font>    <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          54d
-</pre>
-<br />
-<span>5. **Test the application**:</span><br />
-<!-- Generator: GNU source-highlight 3.1.9
-by Lorenzo Bettini
-http://www.lorenzobettini.it
-http://www.gnu.org/software/src-highlite -->
-<pre>$ curl -I https://flux.f3s.buetow.org
-HTTP/<font color="#000000">2</font> <font color="#000000">200</font>
+``
+
+<font color="#000000">5</font>. Test the application:
 </pre>
-<br />
-<span>6. **Update Justfile** and commit changes</span><br />
-<br />
-<span>Total time: 10 minutes. Zero downtime.</span><br />
-<br />
-<h2 style='display: inline' id='complex-migration-prometheus-with-multi-source'>Complex Migration: Prometheus with Multi-Source</h2><br />
-<br />
-<span>The Prometheus migration was more complex because it combines:</span><br />
-<ul>
-<li>Upstream Helm chart (kube-prometheus-stack)</li>
-<li>Custom manifests (PersistentVolumes, recording rules, dashboards)</li>
-<li>Sync hooks (PostSync job to restart Grafana)</li>
-</ul><br />
-<span>ArgoCD supports "multi-source" Applications that combine multiple sources:</span><br />
-<br />
+<span>$ curl -I https://flux.f3s.foo.zone</span><br />
+<span>HTTP/2 200</span><br />
 <pre>
-apiVersion: argoproj.io/v1alpha1
-kind: Application
-metadata:
-  name: prometheus
-  namespace: cicd
-  finalizers:
-    - resources-finalizer.argocd.argoproj.io
-spec:
-  project: default
-  sources:
-    # Source 1: Upstream Helm chart from prometheus-community
-    - repoURL: https://prometheus-community.github.io/helm-charts
-      chart: kube-prometheus-stack
-      targetRevision: 55.5.0
-      helm:
-        releaseName: prometheus
-        valuesObject:
-          # Full Prometheus configuration embedded here
-          kubeEtcd:
-            enabled: true
-            endpoints:
-              - 192.168.2.120
-              - 192.168.2.121
-              - 192.168.2.122
-          # ... (hundreds of lines of configuration)
-
-    # Source 2: Additional manifests from Git repository
-    - repoURL: https://codeberg.org/snonux/conf.git
-      targetRevision: master
-      path: f3s/prometheus/manifests
+6. Update Justfile and commit changes
 
-  destination:
-    server: https://kubernetes.default.svc
-    namespace: monitoring
+Total time: 10 minutes. Zero downtime.
+
+## Complex Migration: Prometheus with Multi-Source
+
+
+The Prometheus migration was more complex because it combines:
+* Upstream Helm chart (kube-prometheus-stack)
+* Custom manifests (PersistentVolumes, recording rules, dashboards)
+* Sync hooks (PostSync job to restart Grafana)
 
-  syncPolicy:
-    automated:
-      prune: false  # Manual pruning for safety on complex stack
-      selfHeal: true
-    syncOptions:
-      - CreateNamespace=false
-      - ServerSideApply=true
-    retry:
-      limit: 3
-      backoff:
-        duration: 10s
-        factor: 2
-        maxDuration: 3m
 </pre>
-<br />
-<span>The <span class='inlinecode'>prometheus/manifests/</span> directory contains:</span><br />
-<br />
+<span>apiVersion: argoproj.io/v1alpha1</span><br />
+<span>kind: Application</span><br />
+<span>metadata:</span><br />
+<span>  name: prometheus</span><br />
+<span>  namespace: cicd</span><br />
+<span>  finalizers:</span><br />
+<span>    - resources-finalizer.argocd.argoproj.io</span><br />
+<span>spec:</span><br />
+<span>  project: default</span><br />
+<span>  sources:</span><br />
+<span>    # Source 1: Upstream Helm chart from prometheus-community</span><br />
+<span>    - repoURL: https://prometheus-community.github.io/helm-charts</span><br />
+<span>    </span><br />
+<span>      chart: kube-prometheus-stack</span><br />
+<span>      targetRevision: 55.5.0</span><br />
+<span>      helm:</span><br />
+<span>        releaseName: prometheus</span><br />
+<span>        valuesObject:</span><br />
+<span>          # Full Prometheus configuration embedded here</span><br />
+<span>          kubeEtcd:</span><br />
+<span>            enabled: true</span><br />
+<span>            endpoints:</span><br />
+<span>              - 192.168.2.120</span><br />
+<span>              - 192.168.2.121</span><br />
+<span>              - 192.168.2.122</span><br />
+<span>          # ... (hundreds of lines of configuration)</span><br />
+<br />
+<span>    # Source 2: Additional manifests from Git repository</span><br />
+<span>    - repoURL: https://codeberg.org/snonux/conf.git</span><br />
+<span>      targetRevision: master</span><br />
+<span>      path: f3s/prometheus/manifests</span><br />
+<br />
+<span>  destination:</span><br />
+<span>    server: https://kubernetes.default.svc</span><br />
+<span>    namespace: monitoring</span><br />
+<br />
+<span>  syncPolicy:</span><br />
+<span>    automated:</span><br />
+<span>      prune: false  # Manual pruning for safety on complex stack</span><br />
+<span>      selfHeal: true</span><br />
+<span>    syncOptions:</span><br />
+<span>      - CreateNamespace=false</span><br />
+<span>      - ServerSideApply=true</span><br />
+<span>    retry:</span><br />
+<span>      limit: 3</span><br />
+<span>      backoff:</span><br />
+<span>        duration: 10s</span><br />
+<span>        factor: 2</span><br />
+<span>        maxDuration: 3m</span><br />
 <pre>
-f3s/prometheus/manifests/
-├── persistent-volumes.yaml              # Sync wave 0
-├── additional-scrape-configs-secret.yaml # Sync wave 1
-├── grafana-datasources-configmap.yaml   # Sync wave 1
-├── freebsd-recording-rules.yaml         # Sync wave 3
-├── openbsd-recording-rules.yaml         # Sync wave 3
-├── zfs-recording-rules.yaml             # Sync wave 3
-├── epimetheus-dashboard.yaml            # Sync wave 4
-├── zfs-dashboards.yaml                  # Sync wave 4
-├── grafana-restart-hook.yaml            # Sync wave 10 (PostSync)
-└── grafana-restart-rbac.yaml            # Sync wave 0
+The `prometheus/manifests/` directory contains:
+
 </pre>
-<br />
-<h3 style='display: inline' id='sync-waves-and-hooks'>Sync Waves and Hooks</h3><br />
-<br />
-<span>ArgoCD allows controlling the order of resource deployment using sync waves (the <span class='inlinecode'>argocd.argoproj.io/sync-wave</span> annotation):</span><br />
-<br />
-<ul>
-<li>**Wave 0**: Infrastructure (PersistentVolumes, RBAC)</li>
-<li>**Wave 1**: Configuration (Secrets, ConfigMaps)</li>
-<li>**Wave 3**: Recording rules (PrometheusRule CRDs)</li>
-<li>**Wave 4**: Dashboards (ConfigMaps with <span class='inlinecode'>grafana_dashboard: &#39;1&#39;</span> label)</li>
-<li>**Wave 10**: PostSync hooks (Jobs that run after everything else)</li>
-</ul><br />
-<span>The Grafana restart hook ensures Grafana reloads datasources after they&#39;re updated:</span><br />
-<br />
+<span>f3s/prometheus/manifests/</span><br />
+<span>├── persistent-volumes.yaml              # Sync wave 0</span><br />
+<span>├── additional-scrape-configs-secret.yaml # Sync wave 1</span><br />
+<span>├── grafana-datasources-configmap.yaml   # Sync wave 1</span><br />
+<span>├── freebsd-recording-rules.yaml         # Sync wave 3</span><br />
+<span>├── openbsd-recording-rules.yaml         # Sync wave 3</span><br />
+<span>├── zfs-recording-rules.yaml             # Sync wave 3</span><br />
+<span>├── epimetheus-dashboard.yaml            # Sync wave 4</span><br />
+<span>├── zfs-dashboards.yaml                  # Sync wave 4</span><br />
+<span>├── grafana-restart-hook.yaml            # Sync wave 10 (PostSync)</span><br />
+<span>└── grafana-restart-rbac.yaml            # Sync wave 0</span><br />
 <pre>
-apiVersion: batch/v1
-kind: Job
-metadata:
-  name: grafana-restart-hook
-  namespace: monitoring
-  annotations:
-    argocd.argoproj.io/hook: PostSync
-    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
-    argocd.argoproj.io/sync-wave: "10"
-spec:
-  template:
-    spec:
-      serviceAccountName: grafana-restart-sa
-      restartPolicy: OnFailure
-      containers:
-        - name: kubectl
-          image: bitnami/kubectl:latest
-          command:
-            - /bin/sh
-            - -c
-            - |
-              kubectl wait --for=condition=available --timeout=300s deployment/prometheus-grafana -n monitoring || true
-              kubectl delete pod -n monitoring -l app.kubernetes.io/name=grafana --ignore-not-found=true
-  backoffLimit: 2
-</pre>
-<br />
-<span>This replaces the manual step in the old Justfile that required running <span class='inlinecode'>kubectl delete pod</span> after every upgrade.</span><br />
-<br />
-<h2 style='display: inline' id='migration-results'>Migration Results</h2><br />
-<br />
-<span>After migrating all 21 applications to ArgoCD:</span><br />
-<br />
-<!-- Generator: GNU source-highlight 3.1.9
-by Lorenzo Bettini
-http://www.lorenzobettini.it
-http://www.gnu.org/software/src-highlite -->
-<pre>$ argocd app list
-NAME                      CLUSTER                         NAMESPACE    PROJECT  STATUS  HEALTH   SYNCPOLICY
-alloy                     https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto-Prune
-anki-sync-server          https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-audiobookshelf            https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-example-apache            https://kubernetes.default.svc  <b><u><font color="#000000">test</font></u></b>         default  Synced  Healthy  Auto-Prune
-example-apache-volume-... https://kubernetes.default.svc  <b><u><font color="#000000">test</font></u></b>         default  Synced  Healthy  Auto-Prune
-filebrowser               https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-freshrss                  https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-grafana-ingress           https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto-Prune
-immich                    https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-keybr                     https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-kobo-sync-server          https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-loki                      https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto-Prune
-miniflux                  https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-opodsync                  https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-prometheus                https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto
-pushgateway               https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto-Prune
-radicale                  https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-registry                  https://kubernetes.default.svc  infra        default  Synced  Healthy  Auto-Prune
-syncthing                 https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-tempo                     https://kubernetes.default.svc  monitoring   default  Synced  Healthy  Auto-Prune
-wallabag                  https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
-webdav                    https://kubernetes.default.svc  services     default  Synced  Healthy  Auto-Prune
+### Sync Waves and Hooks
+
+ArgoCD allows controlling the order of resource deployment using sync waves (the `argocd.argoproj.io/sync-wave` annotation):
+
+* Wave 0: Infrastructure (PersistentVolumes, RBAC)
+* Wave 1: Configuration (Secrets, ConfigMaps)
+* Wave 3: Recording rules (PrometheusRule CRDs)
+* Wave 4: Dashboards (ConfigMaps with `grafana_dashboard: &#39;1&#39;` label)
+* Wave 10: PostSync hooks (Jobs that run after everything else)
+
+The Grafana restart hook ensures Grafana reloads datasources after they&#39;re updated:
+
 </pre>
-<br />
-<span>All 21 applications: **Synced** and **Healthy**.</span><br />
-<br />
-<span>ArgoCD Web UI:</span><br />
-<br />
-<a href='./f3s-kubernetes-with-freebsd-part-X/argocd-apps-list.png'><img alt='ArgoCD Applications List' title='ArgoCD Applications List' src='./f3s-kubernetes-with-freebsd-part-X/argocd-apps-list.png' /></a><br />
-<br />
-<a href='./f3s-kubernetes-with-freebsd-part-X/argocd-app-tree.png'><img alt='ArgoCD Application Resource Tree' title='ArgoCD Application Resource Tree' src='./f3s-kubernetes-with-freebsd-part-X/argocd-app-tree.png' /></a><br />
-<br />
-<h2 style='display: inline' id='benefits-realized'>Benefits Realized</h2><br />
-<br />
-<h3 style='display: inline' id='1-single-source-of-truth'>1. Single Source of Truth</h3><br />
-<br />
-<span>The Git repository at <span class='inlinecode'>https://codeberg.org/snonux/conf</span> now contains the complete cluster configuration. Anyone can clone it and see exactly what&#39;s deployed:</span><br />
-<br />
-<!-- Generator: GNU source-highlight 3.1.9
-by Lorenzo Bettini
-http://www.lorenzobettini.it
-http://www.gnu.org/software/src-highlite -->
-<pre>$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf/f3s
-$ ls argocd-apps/
-alloy.yaml  anki-sync-server.yaml  audiobookshelf.yaml  ...
+<span>apiVersion: batch/v1</span><br />
+<span>kind: Job</span><br />
+<span>metadata:</span><br />
+<span>  name: grafana-restart-hook</span><br />
+<span>  namespace: monitoring</span><br />
+<span>  annotations:</span><br />
+<span>    argocd.argoproj.io/hook: PostSync</span><br />
+<span>*rgocd.argoproj.io/hook-delete-policy: BeforeHookCreation</span><br />
+<span>*rgocd.argoproj.io/sync-wave: "10"</span><br />
+<span>*</span><br />
+<span>  *plate:</span><br />
+<span>    spec:</span><br />
+<span>      serviceAccountName: grafana-restart-sa</span><br />
+<span>      restartPolicy: OnFailure</span><br />
+<span>      containers:</span><br />
+<span>        - name: kubectl</span><br />
+<span>          image: bitnami/kubectl:latest</span><br />
+<span>          command:</span><br />
+<span>            - /bin/sh</span><br />
+<span>            - -c</span><br />
+<span>            - |</span><br />
+<span>              kubectl wait --for=condition=available --timeout=300s deployment/prometheus-grafana -n monitoring || true</span><br />
+<span>              kubectl delete pod -n monitoring -l app.kubernetes.io/name=grafana --ignore-not-found=true</span><br />
+<span>  backoffLimit: 2</span><br />
+<pre>
+This *he manual