Update content for html

4 files changed, 1328 insertions, 61 deletions

diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html
index 70a8e44f..e17fdf7d 100644
--- a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html
+++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html
@@ -33,6 +33,7 @@
 <ul>
 <li><a href='#f3s-kubernetes-with-freebsd---part-7-k3s-and-first-pod-deployments'>f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a></li>
 <li>⇢ <a href='#introduction'>Introduction</a></li>
+<li>⇢ <a href='#important-note-gitops-migration'>Important Note: GitOps Migration</a></li>
 <li>⇢ <a href='#updating'>Updating</a></li>
 <li>⇢ <a href='#installing-k3s'>Installing k3s</a></li>
 <li>⇢ ⇢ <a href='#generating-k3stoken-and-starting-the-first-k3s-node'>Generating <span class='inlinecode'>K3S_TOKEN</span> and starting the first k3s node</a></li>
@@ -63,6 +64,29 @@
 <br />
 <a class='textlink' href='https://k3s.io'>https://k3s.io</a><br />
 <br />
+<h2 style='display: inline' id='important-note-gitops-migration'>Important Note: GitOps Migration</h2><br />
+<br />
+<span>**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests and Helm charts in the repository have been reorganized for ArgoCD-based continuous deployment.</span><br />
+<br />
+<span>**To view the exact manifests and charts as they existed when this blog post was written** (before the ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf
+$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
+$ cd f3s/
+</pre>
+<br />
+<span>**Current master branch** contains the ArgoCD-managed versions with:</span><br />
+<span>- Application manifests organized under <span class='inlinecode'>argocd-apps/{monitoring,services,infra,test}/</span></span><br />
+<span>- Additional resources under <span class='inlinecode'>*/manifests/</span> directories (e.g., <span class='inlinecode'>prometheus/manifests/</span>)</span><br />
+<span>- Justfiles updated to trigger ArgoCD syncs instead of direct Helm commands</span><br />
+<br />
+<span>The deployment concepts and architecture remain the same—only the deployment method changed from imperative (<span class='inlinecode'>helm install/upgrade</span>) to declarative (GitOps with ArgoCD). For details on the GitOps migration, see Part X of this series.</span><br />
+<br />
 <h2 style='display: inline' id='updating'>Updating</h2><br />
 <br />
 <span>Before proceeding, I bring all systems involved up-to-date. On all three Rocky Linux 9 boxes <span class='inlinecode'>r0</span>, <span class='inlinecode'>r1</span>, and <span class='inlinecode'>r2</span>:</span><br />
@@ -878,21 +902,7 @@ http://www.gnu.org/software/src-highlite -->
 <br />
 <a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s</a><br />
 <br />
-<span>**Note:** After publishing this blog post, the f3s cluster was migrated to ArgoCD GitOps. The Kubernetes manifests and Helm charts in the repository have been reorganized for declarative deployment. To view the exact manifests and charts as they existed when this blog post was written (before ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
-<br />
-<!-- Generator: GNU source-highlight 3.1.9
-by Lorenzo Bettini
-http://www.lorenzobettini.it
-http://www.gnu.org/software/src-highlite -->
-<pre>$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf
-$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
-$ cd f3s/
-</pre>
-<br />
-<span>The current master branch contains the ArgoCD-managed versions with manifests organized under <span class='inlinecode'>argocd-apps/</span> and <span class='inlinecode'>*/manifests/</span> directories.</span><br />
-<br />
-<span>Within that repo, the <span class='inlinecode'>examples/conf/f3s/registry/</span> directory contains the Helm chart, a <span class='inlinecode'>Justfile</span>, and a detailed <span class='inlinecode'>README</span>. Here&#39;s the condensed walkthrough I used to roll out the registry with Helm.</span><br />
+<span>Within that repo, the <span class='inlinecode'>f3s/registry/</span> directory contains the Helm chart, a <span class='inlinecode'>Justfile</span>, and a detailed <span class='inlinecode'>README</span>. Here&#39;s the condensed walkthrough I used to roll out the registry with Helm.</span><br />
 <br />
 <h3 style='display: inline' id='prepare-the-nfs-backed-storage'>Prepare the NFS-backed storage</h3><br />
 <br />
diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
index 00fcadc9..0295f7dc 100644
--- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
+++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html
@@ -33,6 +33,7 @@
 <ul>
 <li><a href='#f3s-kubernetes-with-freebsd---part-8-observability'>f3s: Kubernetes with FreeBSD - Part 8: Observability</a></li>
 <li>⇢ <a href='#introduction'>Introduction</a></li>
+<li>⇢ <a href='#important-note-gitops-migration'>Important Note: GitOps Migration</a></li>
 <li>⇢ <a href='#persistent-storage-recap'>Persistent storage recap</a></li>
 <li>⇢ <a href='#the-monitoring-namespace'>The monitoring namespace</a></li>
 <li>⇢ <a href='#installing-prometheus-and-grafana'>Installing Prometheus and Grafana</a></li>
@@ -74,7 +75,30 @@
 <br />
 <span>All manifests for the f3s stack live in my configuration repository:</span><br />
 <br />
-<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s </a><br />
+<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s</a><br />
+<br />
+<h2 style='display: inline' id='important-note-gitops-migration'>Important Note: GitOps Migration</h2><br />
+<br />
+<span>**Note:** After publishing this blog post, the f3s cluster was migrated from imperative Helm deployments to declarative GitOps using ArgoCD. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for ArgoCD-based continuous deployment.</span><br />
+<br />
+<span>**To view the exact configuration as it existed when this blog post was written** (before the ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ git clone https://codeberg.org/snonux/conf.git
+$ cd conf
+$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
+$ cd f3s/prometheus/
+</pre>
+<br />
+<span>**Current master branch** contains the ArgoCD-managed versions with:</span><br />
+<span>- Application manifests organized under <span class='inlinecode'>argocd-apps/{monitoring,services,infra,test}/</span></span><br />
+<span>- Resources organized under <span class='inlinecode'>prometheus/manifests/</span>, <span class='inlinecode'>loki/</span>, etc.</span><br />
+<span>- Justfiles updated to trigger ArgoCD syncs instead of direct Helm commands</span><br />
+<br />
+<span>The deployment concepts and architecture remain the same—only the deployment method changed from imperative (<span class='inlinecode'>helm install/upgrade</span>) to declarative (GitOps with ArgoCD). For details on the GitOps migration, see Part X of this series. </span><br />
 <br />
 <h2 style='display: inline' id='persistent-storage-recap'>Persistent storage recap</h2><br />
 <br />
@@ -107,20 +131,6 @@ namespace/monitoring created
 <br />
 <h2 style='display: inline' id='installing-prometheus-and-grafana'>Installing Prometheus and Grafana</h2><br />
 <br />
-<span>**Note:** After publishing this blog post, the f3s cluster was migrated to ArgoCD GitOps. The Kubernetes manifests, Helm charts, and Justfiles in the repository have been reorganized for declarative deployment. To view the exact configuration as it existed when this blog post was written (before ArgoCD migration), check out the pre-ArgoCD revision:</span><br />
-<br />
-<!-- Generator: GNU source-highlight 3.1.9
-by Lorenzo Bettini
-http://www.lorenzobettini.it
-http://www.gnu.org/software/src-highlite -->
-<pre>$ git clone https://codeberg.org/snonux/conf.git
-$ cd conf
-$ git checkout 15a86f3  <i><font color="silver"># Last commit before ArgoCD migration</font></i>
-$ cd f3s/prometheus/
-</pre>
-<br />
-<span>The current master branch contains the ArgoCD-managed versions with Application manifests under <span class='inlinecode'>argocd-apps/</span> and resources organized under <span class='inlinecode'>prometheus/manifests/</span>, <span class='inlinecode'>loki/</span>, etc. The Justfiles have been updated to trigger ArgoCD syncs instead of direct Helm commands.</span><br />
-<br />
 <span>Prometheus and Grafana are deployed together using the <span class='inlinecode'>kube-prometheus-stack</span> Helm chart from the Prometheus community. This chart bundles Prometheus, Grafana, Alertmanager, and various exporters (Node Exporter, Kube State Metrics) into a single deployment. Ill explain what each component does in detail later when we look at the running pods.</span><br />
 <br />
 <h3 style='display: inline' id='prerequisites'>Prerequisites</h3><br />
diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html
new file mode 100644
index 00000000..5d58d4c5
--- /dev/null
+++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.html
@@ -0,0 +1,1227 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
+<head>
+<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+<title>f3s: Kubernetes with FreeBSD - Part X: GitOps with ArgoCD</title>
+<link rel="shortcut icon" type="image/gif" href="/favicon.ico" />
+<link rel="stylesheet" href="../style.css" />
+<link rel="stylesheet" href="style-override.css" />
+</head>
+<body>
+<p class="header">
+<a href="https://foo.zone">Home</a> | <a href="https://codeberg.org/snonux/foo.zone/src/branch/content-md/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.md">Markdown</a> | <a href="gemini://foo.zone/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-X.gmi">Gemini</a>
+</p>
+<h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-x-gitops-with-argocd'>f3s: Kubernetes with FreeBSD - Part X: GitOps with ArgoCD</h1><br />
+<br />
+<span class='quote'>DRAFT - Not yet published</span><br />
+<br />
+<span>This is part X of the f3s series for my self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines.</span><br />
+<br />
+<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br />
+<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br />
+<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br />
+<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br />
+<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br />
+<a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br />
+<a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br />
+<a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br />
+<br />
+<a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br />
+<br />
+<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br />
+<br />
+<ul>
+<li><a href='#f3s-kubernetes-with-freebsd---part-x-gitops-with-argocd'>f3s: Kubernetes with FreeBSD - Part X: GitOps with ArgoCD</a></li>
+<li>⇢ <a href='#introduction'>Introduction</a></li>
+<li>⇢ <a href='#what-is-gitops'>What is GitOps?</a></li>
+<li>⇢ <a href='#what-is-argocd'>What is ArgoCD?</a></li>
+<li>⇢ <a href='#why-argocd-for-f3s'>Why ArgoCD for f3s?</a></li>
+<li>⇢ <a href='#deploying-argocd'>Deploying ArgoCD</a></li>
+<li>⇢ ⇢ <a href='#prerequisites'>Prerequisites</a></li>
+<li>⇢ ⇢ <a href='#installing-argocd'>Installing ArgoCD</a></li>
+<li>⇢ ⇢ <a href='#accessing-argocd'>Accessing ArgoCD</a></li>
+<li>⇢ <a href='#argocd-application-structure'>ArgoCD Application Structure</a></li>
+<li>⇢ <a href='#repository-organization'>Repository Organization</a></li>
+<li>⇢ <a href='#migration-strategy-incremental-one-app-at-a-time'>Migration Strategy: Incremental, One App at a Time</a></li>
+<li>⇢ ⇢ <a href='#migration-phases'>Migration Phases</a></li>
+<li>⇢ <a href='#example-migration-miniflux'>Example Migration: Miniflux</a></li>
+<li>⇢ ⇢ <a href='#before-imperative-helm-deployment'>Before: Imperative Helm deployment</a></li>
+<li>⇢ ⇢ <a href='#after-declarative-gitops-with-argocd'>After: Declarative GitOps with ArgoCD</a></li>
+<li>⇢ ⇢ <a href='#migration-procedure'>Migration procedure</a></li>
+<li>⇢ <a href='#complex-migration-prometheus-with-multi-source'>Complex Migration: Prometheus with Multi-Source</a></li>
+<li>⇢ ⇢ <a href='#sync-waves-and-hooks'>Sync Waves and Hooks</a></li>
+<li>⇢ <a href='#migration-results'>Migration Results</a></li>
+<li>⇢ <a href='#benefits-realized'>Benefits Realized</a></li>
+<li>⇢ ⇢ <a href='#1-single-source-of-truth'>1. Single Source of Truth</a></li>
+<li>⇢ ⇢ <a href='#2-automatic-synchronization'>2. Automatic Synchronization</a></li>
+<li>⇢ ⇢ <a href='#3-drift-detection-and-self-healing'>3. Drift Detection and Self-Healing</a></li>
+<li>⇢ ⇢ <a href='#4-easy-rollbacks'>4. Easy Rollbacks</a></li>
+<li>⇢ ⇢ <a href='#5-disaster-recovery'>5. Disaster Recovery</a></li>
+<li>⇢ ⇢ <a href='#6-documentation-by-default'>6. Documentation by Default</a></li>
+<li>⇢ ⇢ <a href='#7-safe-experimentation'>7. Safe Experimentation</a></li>
+<li>⇢ <a href='#challenges-and-solutions'>Challenges and Solutions</a></li>
+<li>⇢ ⇢ <a href='#challenge-1-helm-release-adoption'>Challenge 1: Helm Release Adoption</a></li>
+<li>⇢ ⇢ <a href='#challenge-2-persistent-volumes-not-tracked-by-helm'>Challenge 2: Persistent Volumes Not Tracked by Helm</a></li>
+<li>⇢ ⇢ <a href='#challenge-3-secrets-management'>Challenge 3: Secrets Management</a></li>
+<li>⇢ ⇢ <a href='#challenge-4-grafana-not-reloading-datasources'>Challenge 4: Grafana Not Reloading Datasources</a></li>
+<li>⇢ ⇢ <a href='#challenge-5-prometheus-with-multiple-sources'>Challenge 5: Prometheus With Multiple Sources</a></li>
+<li>⇢ ⇢ <a href='#challenge-6-sync-ordering-for-prometheus'>Challenge 6: Sync Ordering for Prometheus</a></li>
+<li>⇢ <a href='#justfile-evolution'>Justfile Evolution</a></li>
+<li>⇢ <a href='#lessons-learned'>Lessons Learned</a></li>
+<li>⇢ <a href='#future-improvements'>Future Improvements</a></li>
+<li>⇢ ⇢ <a href='#1-external-secrets-operator'>1. External Secrets Operator</a></li>
+<li>⇢ ⇢ <a href='#2-applicationset-for-similar-apps'>2. ApplicationSet for Similar Apps</a></li>
+<li>⇢ ⇢ <a href='#3-app-of-apps-pattern'>3. App-of-Apps Pattern</a></li>
+<li>⇢ ⇢ <a href='#4-argocd-image-updater'>4. ArgoCD Image Updater</a></li>
+<li>⇢ <a href='#summary'>Summary</a></li>
+</ul><br />
+<h2 style='display: inline' id='introduction'>Introduction</h2><br />
+<br />
+<span>In the previous posts, I deployed applications to the k3s cluster using Helm charts and Justfiles—running <span class='inlinecode'>just install</span> or <span class='inlinecode'>just upgrade</span> to imperatively push changes to the cluster. While this approach works, it has several drawbacks:</span><br />
+<br />
+<ul>
+<li>**No single source of truth**: The cluster state depends on which commands were run and when</li>
+<li>**Manual synchronization**: Every change requires manually running commands</li>
+<li>**Drift detection is hard**: No easy way to know if cluster state matches the desired configuration</li>
+<li>**Rollback complexity**: Rolling back changes means re-running old Helm commands</li>
+<li>**No audit trail**: Hard to track who changed what and when</li>
+</ul><br />
+<span>This blog post covers the migration from imperative Helm deployments to declarative GitOps using ArgoCD. After this migration, the Git repository becomes the single source of truth, and ArgoCD automatically ensures the cluster matches what&#39;s defined in Git.</span><br />
+<br />
+<h2 style='display: inline' id='what-is-gitops'>What is GitOps?</h2><br />
+<br />
+<span>GitOps is an operational framework that applies DevOps best practices—like version control, collaboration, and CI/CD—to infrastructure automation. The core idea is simple: the entire desired state of your infrastructure is stored in Git, and automated processes ensure the actual state matches the desired state.</span><br />
+<br />
+<span>Key principles:</span><br />
+<br />
+<ul>
+<li>**Declarative**: The system&#39;s desired state is described declaratively (YAML manifests, Helm values)</li>
+<li>**Versioned and immutable**: All changes are committed to Git, providing a complete history</li>
+<li>**Pulled automatically**: An agent in the cluster continuously pulls the desired state from Git</li>
+<li>**Continuously reconciled**: The agent ensures the actual state matches the desired state, automatically correcting drift</li>
+</ul><br />
+<span>For Kubernetes, this means:</span><br />
+<br />
+<span>1. All manifests, Helm charts, and configuration live in a Git repository</span><br />
+<span>2. A tool (ArgoCD in our case) watches the repository</span><br />
+<span>3. When changes are pushed to Git, ArgoCD automatically applies them to the cluster</span><br />
+<span>4. If someone manually changes resources in the cluster, ArgoCD detects the drift and can automatically revert it</span><br />
+<br />
+<h2 style='display: inline' id='what-is-argocd'>What is ArgoCD?</h2><br />
+<br />
+<span>ArgoCD is a declarative, GitOps continuous delivery tool for Kubernetes. It&#39;s implemented as a Kubernetes controller that continuously monitors running applications and compares the current, live state against the desired target state defined in Git.</span><br />
+<br />
+<a class='textlink' href='https://argo-cd.readthedocs.io'>ArgoCD Documentation</a><br />
+<br />
+<span>Key features:</span><br />
+<br />
+<ul>
+<li>**Automated deployment**: Monitors Git repositories and automatically syncs changes to the cluster</li>
+<li>**Application definitions**: Defines applications as CRDs (Custom Resource Definitions)</li>
+<li>**Health assessment**: Understands Kubernetes resources and can determine if an application is healthy</li>
+<li>**Web UI and CLI**: Provides both a web interface and command-line tool for managing applications</li>
+<li>**RBAC**: Role-based access control for team collaboration</li>
+<li>**SSO integration**: Can integrate with existing authentication systems</li>
+<li>**Multi-cluster support**: Can manage applications across multiple Kubernetes clusters</li>
+<li>**Sync waves and hooks**: Control the order of resource deployment and run jobs at specific lifecycle points</li>
+</ul><br />
+<h2 style='display: inline' id='why-argocd-for-f3s'>Why ArgoCD for f3s?</h2><br />
+<br />
+<span>For a home lab cluster, ArgoCD provides several benefits:</span><br />
+<br />
+<span>**Disaster recovery**: If the entire cluster is lost, I can rebuild it by:</span><br />
+<span>1. Bootstrapping a new k3s cluster</span><br />
+<span>2. Installing ArgoCD</span><br />
+<span>3. Pointing ArgoCD at the Git repository</span><br />
+<span>4. All applications automatically deploy to the desired state</span><br />
+<br />
+<span>**Experimentation safety**: I can test changes in a separate Git branch without affecting the running cluster. Once validated, merge to master and ArgoCD applies the changes.</span><br />
+<br />
+<span>**Drift detection**: If I manually change something in the cluster (for debugging), ArgoCD shows the difference and can automatically revert it.</span><br />
+<br />
+<span>**Declarative configuration**: The Git repository documents the entire cluster configuration. No need to remember which <span class='inlinecode'>just</span> commands to run or in which order.</span><br />
+<br />
+<span>**Automatic sync**: Push to Git, and changes deploy automatically. No need to SSH to a workstation and run Helm commands.</span><br />
+<br />
+<h2 style='display: inline' id='deploying-argocd'>Deploying ArgoCD</h2><br />
+<br />
+<span>ArgoCD itself runs as a set of Kubernetes resources in the cluster. The official installation method uses <span class='inlinecode'>kubectl apply</span>, which is fitting—ArgoCD manages everything else via GitOps, but ArgoCD itself needs a bootstrap.</span><br />
+<br />
+<h3 style='display: inline' id='prerequisites'>Prerequisites</h3><br />
+<br />
+<span>Create the <span class='inlinecode'>cicd</span> namespace where ArgoCD will run:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl create namespace cicd
+namespace/cicd created
+</pre>
+<br />
+<h3 style='display: inline' id='installing-argocd'>Installing ArgoCD</h3><br />
+<br />
+<span>The ArgoCD installation lives in the configuration repository:</span><br />
+<br />
+<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/argocd'>codeberg.org/snonux/conf/f3s/argocd</a><br />
+<br />
+<span>I deployed ArgoCD using Helm instead of the raw manifests. This provides easier upgrades and customization. The installation is managed via a Justfile:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ cd conf/f3s/argocd
+$ just install
+helm repo add argo https://argoproj.github.io/argo-helm
+helm repo update
+helm install argocd argo/argo-cd \
+    --namespace cicd \
+    --version <font color="#000000">7.7</font>.<font color="#000000">12</font> \
+    -f values.yaml
+NAME: argocd
+LAST DEPLOYED: ...
+NAMESPACE: cicd
+STATUS: deployed
+</pre>
+<br />
+<span>The <span class='inlinecode'>values.yaml</span> file configures several important aspects:</span><br />
+<br />
+<span>**Persistent storage for the repo-server**: ArgoCD clones Git repositories to cache them locally. I configured a persistent volume so the cache survives pod restarts:</span><br />
+<br />
+<pre>
+repoServer:
+  volumes:
+    - name: repo-cache
+      persistentVolumeClaim:
+        claimName: argocd-repo-cache-pvc
+  volumeMounts:
+    - name: repo-cache
+      mountPath: /tmp
+</pre>
+<br />
+<span>**Admin password preservation**: By default, the admin password is auto-generated and stored in a secret. To ensure it persists across Helm upgrades:</span><br />
+<br />
+<pre>
+configs:
+  secret:
+    createSecret: false
+</pre>
+<br />
+<span>I manually created the secret before installation:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ ARGOCD_ADMIN_PASSWORD=$(pwgen -s <font color="#000000">32</font> <font color="#000000">1</font>)
+$ BCRYPT_HASH=$(htpasswd -nbBC <font color="#000000">10</font> <font color="#808080">""</font> <font color="#808080">"$ARGOCD_ADMIN_PASSWORD"</font> | tr -d <font color="#808080">':</font>\n<font color="#808080">'</font> | sed <font color="#808080">'s/$2y/$2a/'</font>)
+$ kubectl create secret generic argocd-secret \
+    --from-literal=admin.password=<font color="#808080">"$BCRYPT_HASH"</font> \
+    -n cicd
+$ echo <font color="#808080">"ArgoCD admin password: $ARGOCD_ADMIN_PASSWORD"</font>
+</pre>
+<br />
+<span>**Server configuration**: Enabled insecure mode since TLS is handled by the OpenBSD edge relays:</span><br />
+<br />
+<pre>
+server:
+  insecure: true
+</pre>
+<br />
+<h3 style='display: inline' id='accessing-argocd'>Accessing ArgoCD</h3><br />
+<br />
+<span>After deployment, ArgoCD runs several pods in the <span class='inlinecode'>cicd</span> namespace:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ kubectl get pods -n cicd
+NAME                                                READY   STATUS    RESTARTS   AGE
+argocd-application-controller-<font color="#000000">0</font>                     <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-applicationset-controller-66d6b9b8f4-vhm9k   <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-dex-server-7fb556b7dd-xjr2l                  <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-notifications-controller-6d8dd4c5f5-b8vwl    <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-redis-77b8d6c6d4-mz9hg                       <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-repo-server-5f98f77b97-8xtcq                 <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+argocd-server-6b9c4b4f8d-kxw7p                      <font color="#000000">1</font>/<font color="#000000">1</font>     Running   <font color="#000000">0</font>          45d
+</pre>
+<br />
+<span>I created an ingress to expose the ArgoCD web UI:</span><br />
+<br />
+<pre>
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: argocd-server-ingress
+  namespace: cicd
+  annotations:
+    spec.ingressClassName: traefik
+    traefik.ingress.kubernetes.io/router.entrypoints: web
+spec:
+  rules:
+    - host: argocd.f3s.buetow.org
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: argocd-server
+                port:
+                  number: 80
+</pre>
+<br />
+<span>Following the same pattern as other services, the OpenBSD edge relays terminate TLS and forward traffic through WireGuard to the cluster. ArgoCD is now accessible at:</span><br />
+<br />
+<a class='textlink' href='https://argocd.f3s.buetow.org'>ArgoCD Web UI</a><br />
+<br />
+<span>The ArgoCD CLI can also be used for operations:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>$ argocd login argocd.f3s.buetow.org
+$ argocd app list
+</pre>
+<br />
+<h2 style='display: inline' id='argocd-application-structure'>ArgoCD Application Structure</h2><br />
+<br />
+<span>ArgoCD uses a CRD called <span class='inlinecode'>Application</span> to define what should be deployed. Each application specifies:</span><br />
+<br />
+<ul>
+<li>**Source**: Where the manifests live (Git repo, Helm chart repository, or both)</li>
+<li>**Destination**: Which cluster and namespace to deploy to</li>
+<li>**Sync policy**: Whether to automatically sync changes</li>
+</ul><br />
+<span>Here&#39;s a simple example for the miniflux application:</span><br />
+<br />
+<pre>
+apiVersion: argoproj.io/v1alpha1
+kind: Application
+metadata:
+  name: miniflux
+  namespace: cicd
+  finalizers:
+    - resources-finalizer.argocd.argoproj.io
+spec:
+  project: default
+  source:
+    repoURL: https://codeberg.org/snonux/conf.git
+    targetRevision: master
+    path: f3s/miniflux/helm-chart
+  destination:
+    server: https://kubernetes.default.svc
+    namespace: services
+  syncPolicy:
+    automated:
+      prune: true
+      selfHeal: true
+    syncOptions:
+      - CreateNamespace=false
+    retry:
+      limit: 3
+      backoff:
+        duration: 5s
+        factor: 2
+        maxDuration: 1m
+</pre>
+<br />
+<span>Key fields:</span><br />
+<br />
+<ul>
+<li><span class='inlinecode'>source.path</span>: Points to the Helm chart directory in Git</li>
+<li><span class='inlinecode'>destination.namespace</span>: Where to deploy the application</li>
+<li><span class='inlinecode'>syncPolicy.automated.prune</span>: Delete resources that are removed from Git</li>
+<li><span class='inlinecode'>syncPolicy.automated.selfHeal</span>: Automatically revert manual changes in the cluster</li>
+<li><span class='inlinecode'>finalizers</span>: Ensures ArgoCD deletes all resources when the Application is deleted</li>
+</ul><br />
+<h2 style='display: inline' id='repository-organization'>Repository Organization</h2><br />
+<br />
+<span>I reorganized the configuration repository to support GitOps:</span><br />
+<br />
+<pre>
+/home/paul/git/conf/f3s/
+├── argocd-apps/               # ArgoCD Application manifests (organized by namespace)
+│   ├── README.md              # Documentation of structure
+│   ├── monitoring/            # Observability stack (6 apps)
+│   │   ├── alloy.yaml
+│   │   ├── grafana-ingress.yaml
+│   │   ├── loki.yaml
+│   │   ├── prometheus.yaml
+│   │   ├── pushgateway.yaml
+│   │   └── tempo.yaml
+│   ├── services/              # User-facing applications (13 apps)
+│   │   ├── anki-sync-server.yaml
+│   │   ├── audiobookshelf.yaml
+│   │   ├── filebrowser.yaml
+│   │   ├── immich.yaml
+│   │   ├── keybr.yaml
+│   │   ├── kobo-sync-server.yaml
+│   │   ├── miniflux.yaml
+│   │   ├── opodsync.yaml
+│   │   ├── radicale.yaml
+│   │   ├── syncthing.yaml
+│   │   ├── tracing-demo.yaml
+│   │   ├── wallabag.yaml
+│   │   └── webdav.yaml
+│   ├── infra/                 # Infrastructure services (1 app)
+│   │   └── registry.yaml
+│   └── test/                  # Test/example applications (1 app)
+│       └── example-apache-volume-claim.yaml
+├── miniflux/                  # Application directories (unchanged)
+│   ├── helm-chart/
+│   │   ├── Chart.yaml
+│   │   ├── values.yaml
+│   │   └── templates/
+│   └── Justfile               # Updated for ArgoCD
+├── prometheus/
+│   ├── manifests/             # NEW: Additional manifests
+│   │   ├── persistent-volumes.yaml
+│   │   ├── grafana-restart-hook.yaml
+│   │   ├── freebsd-recording-rules.yaml
+│   │   └── ...
+│   └── Justfile               # Updated for ArgoCD
+└── ...
+</pre>
+<br />
+<span>The application directories (miniflux, prometheus, etc.) remained mostly unchanged—ArgoCD references the same Helm charts. The main additions:</span><br />
+<br />
+<span>1. **argocd-apps/**: Application manifests organized by Kubernetes namespace for better clarity</span><br />
+<span>   - <span class='inlinecode'>monitoring/</span>: 6 observability applications</span><br />
+<span>   - <span class='inlinecode'>services/</span>: 13 user-facing applications</span><br />
+<span>   - <span class='inlinecode'>infra/</span>: 1 infrastructure application (registry)</span><br />
+<span>   - <span class='inlinecode'>test/</span>: 1 test application</span><br />
+<span>2. ***/manifests/**: Additional Kubernetes manifests for complex apps (like Prometheus)</span><br />
+<span>3. **Justfiles updated**: Changed from <span class='inlinecode'>helm install/upgrade</span> to <span class='inlinecode'>argocd app sync</span></span><br />
+<br />
+<span>This organization makes it easy to apply all applications in a specific namespace or manage them independently.</span><br />
+<br />
+<h2 style='display: inline' id='migration-strategy-incremental-one-app-at-a-time'>Migration Strategy: Incremental, One App at a Time</h2><br />
+<br />
+<span>Rather than attempting a "big bang" migration of all 21 applications at once, I migrated them incrementally:</span><br />
+<br />
+<span>1. **Start with a simple app**: Validate the pattern with a low-risk application</span><br />
+<span>2. **Migrate in waves**: Group similar applications and migrate together</span><br />
+<span>3. **Validate thoroughly**: Ensure each app is healthy before moving to the next</span><br />
+<span>4. **Learn and iterate**: Apply lessons from earlier migrations to later ones</span><br />
+<br />
+<span>This approach reduced risk and allowed me to refine the migration process.</span><br />
+<br />
+<h3 style='display: inline' id='migration-phases'>Migration Phases</h3><br />
+<br />
+<span>**Phase 1: Simple services** (13 apps)</span><br />
+<ul>
+<li>miniflux, freshrss, wallabag</li>
+<li>anki-sync-server, kobo-sync-server, opodsync</li>
+<li>radicale, syncthing, audiobookshelf</li>
+<li>filebrowser, keybr, webdav</li>
+<li>example-apache, example-apache-volume-claim</li>
+</ul><br />
+<span>These apps have straightforward Helm charts with no complex dependencies. Pattern established:</span><br />
+<span>1. Create Application manifest in <span class='inlinecode'>argocd-apps/</span></span><br />
+<span>2. Apply with <span class='inlinecode'>kubectl apply -f argocd-apps/&lt;app&gt;.yaml</span></span><br />
+<span>3. Verify sync status: <span class='inlinecode'>argocd app get &lt;app&gt;</span></span><br />
+<span>4. Update Justfile to use ArgoCD commands</span><br />
+<br />
+<span>**Phase 2: Infrastructure apps** (3 apps)</span><br />
+<ul>
+<li>registry (Docker image registry)</li>
+<li>pushgateway (Prometheus metrics ingestion)</li>
+<li>immich (photo management with complex dependencies)</li>
+</ul><br />
+<span>**Phase 3: Monitoring stack** (4 apps)</span><br />
+<ul>
+<li>tempo (distributed tracing)</li>
+<li>loki (log aggregation)</li>
+<li>alloy (log collection)</li>
+<li>prometheus (metrics and monitoring)</li>
+</ul><br />
+<span>**Phase 4: Monitoring addons** (1 app)</span><br />
+<ul>
+<li>grafana-ingress (separate ingress for Grafana)</li>
+</ul><br />
+<h2 style='display: inline' id='example-migration-miniflux'>Example Migration: Miniflux</h2><br />
+<br />
+<span>Let me walk through the migration of miniflux as a concrete example.</span><br />
+<br />
+<h3 style='display: inline' id='before-imperative-helm-deployment'>Before: Imperative Helm deployment</h3><br />
+<br />
+<span>Original Justfile:</span><br />
+<br />
+<!-- Generator: GNU source-highlight 3.1.9
+by Lorenzo Bettini
+http://www.lorenzobettini.it
+http://www.gnu.org/software/src-highlite -->
+<pre>NAMESPACE := <font color="#808080">"services"</font>
+APP_NAME := <font color="#808080">"miniflux"</font>
+
+install:
+    kubectl apply -f helm-chart/persistent-volumes.yaml
+    helm install {{APP_NAME}} ./helm-chart --namespace {{NAMESPACE}}
+
+upgrade:
+    helm upgrade {{APP_NAME}} ./helm-chart --namespace {{NAMESPACE}}
+
+uninstall:
+    helm uninstall {{APP_NAME}} --namespace {{NAMESPACE}}
+    kubectl delete -f helm-chart/persistent-volumes.yaml
+
+status:
+    @kubectl get all -n {{NAMESPACE}} -l app={{APP_NAME}}
+</pre>
+<br />
+<span>Workflow:</span><br />
+<span>1. Make changes to <span class='inlinecode'>helm-chart/</span></span><br />
+<span>2. Run <span class='inlinecode'>just upgrade</span></span><br />
+<span>3. Helm pushes changes to cluster</span><br />
+<br />
+<h3 style='display: inline' id='after-declarative-gitops-with-argocd'>After: Declarative GitOps with ArgoCD</h3><br />
+<br />
+<span>Created <span class='inlinecode'>argocd-apps/services/miniflux.yaml<