diff options
| author | Paul Buetow <paul@buetow.org> | 2026-03-30 22:43:37 +0300 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2026-03-30 22:43:37 +0300 |
| commit | 362916a9e839124716cc0c430677167b56ded5aa (patch) | |
| tree | 1e74858c82bbf24ae096291680e58e7a7987650d | |
| parent | ea0f59bcbd02a340432549ed983468fc450bc42c (diff) | |
Update content for gemtext
17 files changed, 904 insertions, 360 deletions
diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi index fbbbf9a7..257699a4 100644 --- a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi +++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi @@ -397,6 +397,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi index 38d67d98..fafcc686 100644 --- a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi +++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi @@ -676,6 +676,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi index 4dceb52c..9cb57b32 100644 --- a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi +++ b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi @@ -53,6 +53,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi index df9c6e11..776d8ad2 100644 --- a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi +++ b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi @@ -300,6 +300,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD and KISS related posts are: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi index 8e7c7d01..ef1cb5e7 100644 --- a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi +++ b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi @@ -16,6 +16,7 @@ These are all the posts so far: => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -164,6 +165,7 @@ Read the next post of this series: Other *BSD-related posts: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi index 26fa7dcb..1d15eea0 100644 --- a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi +++ b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi @@ -16,6 +16,7 @@ These are all the posts so far: => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -444,6 +445,7 @@ Read the next post of this series: Other *BSD-related posts: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi index b39d0cc8..a32572e1 100644 --- a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi +++ b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi @@ -12,6 +12,7 @@ This is the third blog post about my f3s series for my self-hosting demands in m => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -366,6 +367,7 @@ Read the next post of this series: Other BSD related posts are: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi index dbef3644..ad10c129 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi @@ -12,6 +12,7 @@ This is the fourth blog post about the f3s series for self-hosting demands in a => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -610,6 +611,7 @@ Read the next post of this series: Other *BSD-related posts: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi index 3c8dc1ca..badb977f 100644 --- a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi +++ b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi @@ -18,6 +18,7 @@ These are all the posts so far: => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -1442,6 +1443,7 @@ Read the next post of this series: Other *BSD-related posts: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi index 26254d7c..90d9321c 100644 --- a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi +++ b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi @@ -12,6 +12,7 @@ This is the sixth blog post about the f3s series for self-hosting demands in a h => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this) => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -1932,6 +1933,7 @@ Read the next post of this series: Other *BSD-related posts: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this) diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi index b89f1922..ee6084c9 100644 --- a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi +++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi @@ -12,6 +12,7 @@ This is the seventh blog post about the f3s series for my self-hosting demands i => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this) => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -1328,6 +1329,7 @@ I hope you enjoyed this walkthrough. Read the next post of this series: Other *BSD-related posts: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this) => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi index c0ceefa9..faad3b1c 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi @@ -12,6 +12,7 @@ This is the 8th blog post about the f3s series for my self-hosting demands in a => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability (You are currently reading this) +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -1656,6 +1657,7 @@ All configuration files are available on Codeberg: Other *BSD-related posts: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability (You are currently reading this) => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-9.gmi b/gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi index 3323bb64..1598ea39 100644 --- a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-9.gmi +++ b/gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi @@ -1,6 +1,6 @@ # f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD -> DRAFT - Not yet published +> Published at 2026-04-02T00:00:00+03:00 This is the 9th post in the f3s series about my self-hosting home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines. @@ -12,6 +12,7 @@ This is the 9th post in the f3s series about my self-hosting home lab. f3s? The => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD (You are currently reading this) => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -40,8 +41,6 @@ This is the 9th post in the f3s series about my self-hosting home lab. f3s? The * ⇢ ⇢ ⇢ Secrets * ⇢ ⇢ ⇢ Grafana Not Reloading * ⇢ ⇢ ⇢ Prometheus Multi-Source Ordering -* ⇢ ⇢ Future Ideas -* ⇢ ⇢ Lessons Learned * ⇢ ⇢ Wrapping Up ## Introduction @@ -549,17 +548,12 @@ webdav https://kubernetes.default.svc services default The practical difference is pretty big: -Single source of truth. Clone the repo, look at `argocd-apps/`, and you know exactly what's running. No more `helm list` or guessing. - -Push and forget. Edit a Helm value, commit, push. ArgoCD picks it up within a few minutes. No SSH, no `just upgrade`. - -Self-healing. I've tweaked things manually for debugging, forgotten about it, and ArgoCD quietly reverted it. That's saved me from some confusing "why is this behaving differently?" moments. - -Rollback = git revert. That's it. `git revert HEAD && git push` and ArgoCD syncs back to the previous state. - -Disaster recovery. Bootstrap k3s, install ArgoCD, apply the Application manifests, wait. The cluster rebuilds itself. I haven't had to do this for real yet, but I've tested it and it works. - -Drift detection. The ArgoCD UI shows immediately if something is out of sync. Much better than running `kubectl` commands and comparing output manually. +* Single source of truth--clone the repo, look at `argocd-apps/`, and you know exactly what's running. No more `helm list` or guessing. +* Push and forget--edit a Helm value, commit, push. ArgoCD picks it up within a few minutes. No SSH, no `just upgrade`. +* Self-healing--I've tweaked things manually for debugging, forgotten about it, and ArgoCD quietly reverted it. That's saved me from some confusing "why is this behaving differently?" moments. +* Rollback = git revert--`git revert HEAD && git push` and ArgoCD syncs back to the previous state. +* Disaster recovery--bootstrap k3s, install ArgoCD, apply the Application manifests, wait. The cluster rebuilds itself. I haven't had to do this for real yet, but I've tested it and it works. +* Drift detection--the ArgoCD UI shows immediately if something is out of sync. Much better than running `kubectl` commands and comparing output manually. ## Challenges Along the Way @@ -583,68 +577,9 @@ After updating datasource ConfigMaps, Grafana wouldn't notice until the pod was Without sync waves, Prometheus resources deployed in random order and things broke. PVs need to exist before PVCs, secrets before the operator, recording rules after the CRDs are registered. Adding sync wave annotations to everything in `prometheus/manifests/` fixed all the ordering issues. -## Future Ideas - -External Secrets Operator to manage secrets declaratively without committing them to Git. - -ApplicationSets for apps with nearly identical manifests. One template could replace 10+ individual Application files: - -```yaml -apiVersion: argoproj.io/v1alpha1 -kind: ApplicationSet -metadata: - name: simple-services - namespace: cicd -spec: - generators: - - list: - elements: - - app: miniflux - - app: wallabag - - app: radicale - template: - metadata: - name: '{{app}}' - spec: - source: - repoURL: http://git-server.cicd.svc.cluster.local/conf.git - targetRevision: master - path: 'f3s/{{app}}/helm-chart' - destination: - server: https://kubernetes.default.svc - namespace: services - syncPolicy: - automated: - prune: true - selfHeal: true -``` - -App-of-Apps pattern so even the Application manifests themselves are managed by ArgoCD. One root Application watches `f3s/argocd-apps/` and deploys everything inside. Disaster recovery then becomes a single `kubectl apply`: - -```sh -$ kubectl apply -f root-app.yaml -# ArgoCD deploys all 30 applications automatically -``` - -ArgoCD Image Updater for apps with custom Docker images--automatically update the image tag in Git when a new image is pushed to the registry. - -## Lessons Learned - -Migrate incrementally, one app at a time. It lets you validate the pattern before it affects everything. - -Start with the boring apps. Complex stuff like Prometheus should come last, after you've gotten comfortable with the basics. - -Sync waves matter for anything non-trivial. Random deployment order causes weird failures. - -Multi-source is great for combining upstream charts with custom config. Keeps things cleanly separated. - -PostSync hooks replace the manual steps you'll inevitably forget. Automate them. - -The ArgoCD Web UI is surprisingly useful. Seeing the resource tree and health status at a glance beats running `kubectl` commands. - ## Wrapping Up -The migration took a few weeks, doing one or two apps at a time. The result: 30 applications across 5 namespaces, all managed declaratively through Git. Push a change, it deploys. Break something, `git revert`. Cluster dies, rebuild from the repo. +The migration took a couple of days, doing one or two apps at a time. The result: 30 applications across 5 namespaces, all managed declaratively through Git. Push a change, it deploys. Break something, `git revert`. Cluster dies, rebuild from the repo. All the config lives here: @@ -658,6 +593,7 @@ I can't imagine going back to running Helm commands manually. Other *BSD-related posts: +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD (You are currently reading this) => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-9.gmi.tpl b/gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi.tpl index 87d6b6dd..5b1b878a 100644 --- a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-9.gmi.tpl +++ b/gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi.tpl @@ -1,6 +1,6 @@ # f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD -> DRAFT - Not yet published +> Published at 2026-04-02T00:00:00+03:00 This is the 9th post in the f3s series about my self-hosting home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines. diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index 3b4f21af..ce67a79a 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,12 +1,672 @@ <?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom"> - <updated>2026-03-30T09:16:50+03:00</updated> + <updated>2026-03-30T22:42:23+03:00</updated> <title>foo.zone feed</title> <subtitle>To be in the .zone!</subtitle> <link href="gemini://foo.zone/gemfeed/atom.xml" rel="self" /> <link href="gemini://foo.zone/" /> <id>gemini://foo.zone/</id> <entry> + <title>f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</title> + <link href="gemini://foo.zone/gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi" /> + <id>gemini://foo.zone/gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi</id> + <updated>2026-04-02T00:00:00+03:00</updated> + <author> + <name>Paul Buetow aka snonux</name> + <email>paul@dev.buetow.org</email> + </author> + <summary>This is the 9th post in the f3s series about my self-hosting home lab. f3s? The 'f' stands for FreeBSD, and the '3s' stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines.</summary> + <content type="xhtml"> + <div xmlns="http://www.w3.org/1999/xhtml"> + <h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-9-gitops-with-argocd'>f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</h1><br /> +<br /> +<span class='quote'>Published at 2026-04-02T00:00:00+03:00</span><br /> +<br /> +<span>This is the 9th post in the f3s series about my self-hosting home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines.</span><br /> +<br /> +<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br /> +<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br /> +<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br /> +<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br /> +<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br /> +<a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> +<a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> +<a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD (You are currently reading this)</a><br /> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br /> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-9/argocd-app-tree.png'><img alt='ArgoCD Application Resource Tree' title='ArgoCD Application Resource Tree' src='./f3s-kubernetes-with-freebsd-part-9/argocd-app-tree.png' /></a><br /> +<br /> +<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br /> +<br /> +<ul> +<li><a href='#f3s-kubernetes-with-freebsd---part-9-gitops-with-argocd'>f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a></li> +<li>⇢ <a href='#introduction'>Introduction</a></li> +<li>⇢ <a href='#gitops-in-a-nutshell'>GitOps in a Nutshell</a></li> +<li>⇢ <a href='#argocd'>ArgoCD</a></li> +<li>⇢ <a href='#why-bother-for-a-home-lab'>Why Bother for a Home Lab?</a></li> +<li>⇢ <a href='#deploying-argocd'>Deploying ArgoCD</a></li> +<li>⇢ ⇢ <a href='#accessing-argocd'>Accessing ArgoCD</a></li> +<li>⇢ <a href='#in-cluster-git-server'>In-Cluster Git Server</a></li> +<li>⇢ <a href='#repository-organization'>Repository Organization</a></li> +<li>⇢ <a href='#migrating-an-app-miniflux-as-example'>Migrating an App: Miniflux as Example</a></li> +<li>⇢ ⇢ <a href='#migration-order'>Migration Order</a></li> +<li>⇢ <a href='#complex-migration-prometheus-multi-source'>Complex Migration: Prometheus Multi-Source</a></li> +<li>⇢ ⇢ <a href='#sync-waves'>Sync Waves</a></li> +<li>⇢ <a href='#the-result'>The Result</a></li> +<li>⇢ <a href='#what-changed-day-to-day'>What Changed Day-to-Day</a></li> +<li>⇢ <a href='#challenges-along-the-way'>Challenges Along the Way</a></li> +<li>⇢ ⇢ <a href='#helm-release-adoption'>Helm Release Adoption</a></li> +<li>⇢ ⇢ <a href='#persistentvolumes'>PersistentVolumes</a></li> +<li>⇢ ⇢ <a href='#secrets'>Secrets</a></li> +<li>⇢ ⇢ <a href='#grafana-not-reloading'>Grafana Not Reloading</a></li> +<li>⇢ ⇢ <a href='#prometheus-multi-source-ordering'>Prometheus Multi-Source Ordering</a></li> +<li>⇢ <a href='#wrapping-up'>Wrapping Up</a></li> +</ul><br /> +<h2 style='display: inline' id='introduction'>Introduction</h2><br /> +<br /> +<span>In previous posts, I deployed applications to the k3s cluster using Helm charts and Justfiles--running <span class='inlinecode'>just install</span> or <span class='inlinecode'>just upgrade</span> to push changes to the cluster. That worked, but it had some drawbacks:</span><br /> +<br /> +<ul> +<li>No single source of truth--cluster state depends on which commands were run and when</li> +<li>Every change requires manually running commands</li> +<li>No easy way to tell if the cluster drifted from the desired config</li> +<li>Rolling back means re-running old Helm commands</li> +<li>No audit trail for who changed what</li> +</ul><br /> +<span>This post covers the migration to GitOps with ArgoCD. After this, the Git repo is the single source of truth, and ArgoCD keeps the cluster in sync automatically.</span><br /> +<br /> +<h2 style='display: inline' id='gitops-in-a-nutshell'>GitOps in a Nutshell</h2><br /> +<br /> +<span>The idea behind GitOps is simple: describe your entire desired state in Git, and let an agent in the cluster pull that state and reconcile it continuously. Every change goes through a commit, so you get version history, collaboration, and rollback for free.</span><br /> +<br /> +<span>For Kubernetes specifically:</span><br /> +<br /> +<ul> +<li>All manifests, Helm charts, and config live in a Git repo</li> +<li>ArgoCD watches that repo</li> +<li>Push a change, ArgoCD applies it</li> +<li>If someone manually tweaks something in the cluster, ArgoCD detects the drift and reverts it</li> +</ul><br /> +<h2 style='display: inline' id='argocd'>ArgoCD</h2><br /> +<br /> +<span>ArgoCD is a GitOps continuous delivery tool for Kubernetes. It runs as a controller in the cluster, continuously comparing live state against what's defined in Git.</span><br /> +<br /> +<a class='textlink' href='https://argo-cd.readthedocs.io'>ArgoCD Documentation</a><br /> +<br /> +<span>The features I care about most for f3s:</span><br /> +<br /> +<ul> +<li>Automatic sync--monitors Git and applies changes to the cluster</li> +<li>Application CRDs--each app is a Kubernetes custom resource</li> +<li>Health checks--knows whether an app is healthy or degraded</li> +<li>Web UI--visual overview of all applications and their sync status</li> +<li>Sync waves and hooks--control deployment order and run post-deploy jobs</li> +<li>Multi-source--combine upstream Helm charts with custom manifests</li> +</ul><br /> +<h2 style='display: inline' id='why-bother-for-a-home-lab'>Why Bother for a Home Lab?</h2><br /> +<br /> +<span>Honestly, the biggest reason is disaster recovery. If the cluster dies, I can:</span><br /> +<br /> +<ul> +<li>Bootstrap a fresh k3s cluster</li> +<li>Install ArgoCD</li> +<li>Point it at the Git repo</li> +<li>Everything deploys automatically</li> +</ul><br /> +<span>That's it. No "let me check my shell history to remember how I set this up."</span><br /> +<br /> +<span>It's also a great way to learn. Setting up GitOps for real--even on a small cluster--teaches you things you won't pick up from tutorials alone. Debugging sync issues, figuring out sync waves, dealing with secrets management--all stuff that's directly applicable at work too.</span><br /> +<br /> +<span>Beyond that: push to Git, things deploy. No SSH'ing to a workstation to run Helm commands. And if I manually tweak something while debugging and forget about it, ArgoCD reverts it back to the desired state. That's happened more than once.</span><br /> +<br /> +<h2 style='display: inline' id='deploying-argocd'>Deploying ArgoCD</h2><br /> +<br /> +<span>ArgoCD manages everything else via GitOps, but ArgoCD itself needs a bootstrap. Chicken-and-egg problem.</span><br /> +<br /> +<span>The installation lives in the config repo:</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/argocd'>codeberg.org/snonux/conf/f3s/argocd</a><br /> +<br /> +<span>I deployed it using Helm via a Justfile:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>$ cd conf/f3s/argocd +$ just install +helm repo add argo https://argoproj.github.io/argo-helm +helm repo update +kubectl create namespace cicd +kubectl apply -f persistent-volumes.yaml +helm install argocd argo/argo-cd --namespace cicd -f values.yaml +kubectl apply -f ingress.yaml +</pre> +<br /> +<span>A few things worth noting in the <span class='inlinecode'>values.yaml</span>:</span><br /> +<br /> +<span>Persistent storage for the repo-server so cloned Git repos survive pod restarts:</span><br /> +<br /> +<pre> +repoServer: + volumes: + - name: repo-server-data + persistentVolumeClaim: + claimName: argocd-repo-server-pvc + volumeMounts: + - name: repo-server-data + mountPath: /home/argocd/repo-cache + env: + - name: XDG_CACHE_HOME + value: /home/argocd/repo-cache +</pre> +<br /> +<span>Server runs in insecure mode since TLS is terminated by the OpenBSD edge relays (same pattern as all other f3s services):</span><br /> +<br /> +<pre> +server: + insecure: true +configs: + params: + server.insecure: true +</pre> +<br /> +<span>Dex (SSO) and notifications are disabled--overkill for a single-user home lab:</span><br /> +<br /> +<pre> +dex: + enabled: false +notifications: + enabled: false +</pre> +<br /> +<span>The admin password is auto-generated on first install and stored in <span class='inlinecode'>argocd-initial-admin-secret</span>. It's preserved across Helm upgrades, so no manual secret creation needed:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>$ just get-password +<i><font color="silver"># Reads from argocd-initial-admin-secret</font></i> +</pre> +<br /> +<h3 style='display: inline' id='accessing-argocd'>Accessing ArgoCD</h3><br /> +<br /> +<span>After deployment, ArgoCD runs in the <span class='inlinecode'>cicd</span> namespace:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>$ kubectl get pods -n cicd +NAME READY STATUS RESTARTS AGE +argocd-application-controller-<font color="#000000">0</font> <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 45d +argocd-applicationset-controller-66d6b9b8f4-vhm9k <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 45d +argocd-redis-77b8d6c6d4-mz9hg <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 45d +argocd-repo-server-5f98f77b97-8xtcq <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 45d +argocd-server-6b9c4b4f8d-kxw7p <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 45d +</pre> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-9/argocd-login.png'><img alt='ArgoCD login page' title='ArgoCD login page' src='./f3s-kubernetes-with-freebsd-part-9/argocd-login.png' /></a><br /> +<br /> +<span>The ingress exposes both a WAN and LAN endpoint:</span><br /> +<br /> +<pre> +# WAN access (via OpenBSD relayd) +- host: argocd.f3s.foo.zone +# LAN access (via FreeBSD CARP VIP, with TLS) +- host: argocd.f3s.lan.foo.zone +</pre> +<br /> +<h2 style='display: inline' id='in-cluster-git-server'>In-Cluster Git Server</h2><br /> +<br /> +<span>I didn't want ArgoCD pulling from Codeberg over the internet every time it checks for changes. If Codeberg is down (or my internet is), the cluster can't reconcile. So I set up a Git server inside the cluster itself.</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/conf/src/commit/190473b/f3s/git-server'>codeberg.org/snonux/conf/f3s/git-server (at 190473b)</a><br /> +<br /> +<span>The git-server runs as a single pod in the <span class='inlinecode'>cicd</span> namespace with two containers sharing a PVC:</span><br /> +<br /> +<ul> +<li>An SSH git server (Alpine + OpenSSH + git-shell) for pushing changes from my laptop</li> +<li>A CGit web UI with git-http-backend (nginx + fcgiwrap) for browsing repos and HTTP clones</li> +</ul><br /> +<span>ArgoCD uses the HTTP backend to clone repos. Most Application manifests point at:</span><br /> +<br /> +<pre> +http://git-server.cicd.svc.cluster.local/conf.git +</pre> +<br /> +<span>For pushing, I use SSH via a NodePort (30022). The git user is locked down to git-shell--no actual shell access. SSH keys are managed through a Kubernetes Secret.</span><br /> +<br /> +<span>There's a chicken-and-egg situation here. The git-server's own ArgoCD Application manifest points at Codeberg (not at itself), since ArgoCD needs to bootstrap the git-server before it can use it:</span><br /> +<br /> +<pre> +# argocd-apps/cicd/git-server.yaml +source: + repoURL: https://codeberg.org/snonux/conf.git + targetRevision: master + path: f3s/git-server/helm-chart +</pre> +<br /> +<span>Once the pod is up, all other apps use the in-cluster URL. The dependency chain is: Codeberg -> git-server -> everything else.</span><br /> +<br /> +<span>The repo storage lives on NFS. Initial setup was just cloning the Codeberg repo as a bare repo into the NFS volume, then pointing my laptop's git remote at the NodePort:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>$ git remote add f3s f3s-git:/repos/conf.git +$ git push f3s master +</pre> +<br /> +<span>ArgoCD detects the change within a few minutes and syncs. No internet required. The whole thing is intentionally minimal--no database, no accounts, no webhooks. Just git over SSH for writes and HTTP for reads.</span><br /> +<br /> +<h2 style='display: inline' id='repository-organization'>Repository Organization</h2><br /> +<br /> +<span>I reorganized the config repo to support GitOps. Application manifests are grouped by Kubernetes namespace:</span><br /> +<br /> +<pre> +/home/paul/git/conf/f3s/ +├── argocd-apps/ +│ ├── cicd/ # CI/CD tooling (2 apps) +│ │ ├── argo-rollouts.yaml +│ │ └── git-server.yaml +│ ├── infra/ # Infrastructure (4 apps) +│ │ ├── cert-manager.yaml +│ │ ├── pkgrepo.yaml +│ │ ├── registry.yaml +│ │ └── traefik-config.yaml +│ ├── monitoring/ # Observability stack (6 apps) +│ │ ├── alloy.yaml +│ │ ├── grafana-ingress.yaml +│ │ ├── loki.yaml +│ │ ├── prometheus.yaml +│ │ ├── pushgateway.yaml +│ │ └── tempo.yaml +│ ├── services/ # User-facing applications (18 apps) +│ │ ├── anki-sync-server.yaml +│ │ ├── apache.yaml +│ │ ├── audiobookshelf.yaml +│ │ ├── filebrowser.yaml +│ │ ├── immich.yaml +│ │ ├── ipv6test.yaml +│ │ ├── jellyfin.yaml +│ │ ├── keybr.yaml +│ │ ├── kobo-sync-server.yaml +│ │ ├── miniflux.yaml +│ │ ├── navidrome.yaml +│ │ ├── opodsync.yaml +│ │ ├── pihole.yaml +│ │ ├── radicale.yaml +│ │ ├── syncthing.yaml +│ │ ├── tracing-demo.yaml +│ │ ├── wallabag.yaml +│ │ └── webdav.yaml +│ └── test/ # Test/example applications +├── miniflux/ # Per-app directories (unchanged) +│ ├── helm-chart/ +│ │ ├── Chart.yaml +│ │ ├── values.yaml +│ │ └── templates/ +│ └── Justfile +├── prometheus/ +│ ├── manifests/ # Additional manifests for multi-source +│ └── Justfile +└── ... +</pre> +<br /> +<span>The per-app directories (miniflux, prometheus, etc.) stayed mostly the same--ArgoCD points at the same Helm charts. The main additions are the <span class='inlinecode'>argocd-apps/</span> directory structure and <span class='inlinecode'>manifests/</span> subdirectories for complex apps.</span><br /> +<br /> +<h2 style='display: inline' id='migrating-an-app-miniflux-as-example'>Migrating an App: Miniflux as Example</h2><br /> +<br /> +<span>I migrated all apps incrementally, one at a time. The procedure was the same for each. Here's miniflux as a concrete example.</span><br /> +<br /> +<span>Before ArgoCD, the Justfile looked like this:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>install: + kubectl apply -f helm-chart/persistent-volumes.yaml + helm install miniflux ./helm-chart --namespace services + +upgrade: + helm upgrade miniflux ./helm-chart --namespace services + +uninstall: + helm uninstall miniflux --namespace services +</pre> +<br /> +<span>Workflow: edit chart, run <span class='inlinecode'>just upgrade</span>, hope you didn't forget anything.</span><br /> +<br /> +<span>To migrate, I created an Application manifest telling ArgoCD where the Helm chart lives and how to sync it:</span><br /> +<br /> +<pre> +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: miniflux + namespace: cicd + finalizers: + - resources-finalizer.argocd.argoproj.io +spec: + project: default + source: + repoURL: http://git-server.cicd.svc.cluster.local/conf.git + targetRevision: master + path: f3s/miniflux/helm-chart + destination: + server: https://kubernetes.default.svc + namespace: services + syncPolicy: + automated: + prune: true + selfHeal: true + syncOptions: + - CreateNamespace=false + retry: + limit: 3 + backoff: + duration: 5s + factor: 2 + maxDuration: 1m +</pre> +<br /> +<span>Then applied it:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="silver"># 1. Apply the Application manifest</font></i> +$ kubectl apply -f argocd-apps/services/miniflux.yaml +application.argoproj.io/miniflux created + +<i><font color="silver"># 2. Verify ArgoCD adopted the existing resources</font></i> +$ argocd app get miniflux +Name: miniflux +Sync Status: Synced to master (4e3c216) +Health Status: Healthy + +<i><font color="silver"># 3. Test that the app still works</font></i> +$ curl -I https://flux.f3s.foo.zone +HTTP/<font color="#000000">2</font> <font color="#000000">200</font> +</pre> +<br /> +<span>About 10 minutes, zero downtime. ArgoCD recognised the already-running resources matched the Helm chart in Git and adopted them without re-deploying.</span><br /> +<br /> +<span>After the migration, the Justfile turned into utility commands--no more install/upgrade/uninstall:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>status: + @kubectl get pods -n services -l app=miniflux-server + @kubectl get pods -n services -l app=miniflux-postgres + @kubectl get application miniflux -n cicd \ + -o jsonpath=<font color="#808080">'Sync: {.status.sync.status}, Health: {.status.health.status}'</font> + +sync: + @kubectl annotate application miniflux -n cicd \ + argocd.argoproj.io/refresh=normal --overwrite + +logs: + kubectl logs -n services -l app=miniflux-server --tail=<font color="#000000">100</font> -f + +restart: + kubectl rollout restart -n services deployment/miniflux-server + +port-forward port=<font color="#808080">"8080"</font>: + kubectl port-forward -n services svc/miniflux {{port}}:<font color="#000000">8080</font> + +psql: + kubectl <b><u><font color="#000000">exec</font></u></b> -it -n services deployment/miniflux-postgres -- psql -U miniflux +</pre> +<br /> +<span>New workflow: edit chart, commit, push. ArgoCD picks it up within a few minutes. Run <span class='inlinecode'>just sync</span> if you're impatient.</span><br /> +<br /> +<h3 style='display: inline' id='migration-order'>Migration Order</h3><br /> +<br /> +<span>I started with the simplest services (miniflux, wallabag, radicale, etc.)--apps with straightforward Helm charts and no complex dependencies. This let me validate the pattern before touching anything critical.</span><br /> +<br /> +<span>After that: infrastructure apps (registry, cert-manager, pkgrepo, traefik-config), then the monitoring stack (tempo, loki, alloy, and finally prometheus--the most complex one), and last the CI/CD tools (git-server, argo-rollouts).</span><br /> +<br /> +<h2 style='display: inline' id='complex-migration-prometheus-multi-source'>Complex Migration: Prometheus Multi-Source</h2><br /> +<br /> +<span>Prometheus was the tricky one. It combines an upstream Helm chart with a bunch of custom manifests--recording rules, dashboards, persistent volumes, and a post-sync hook to restart Grafana.</span><br /> +<br /> +<span>ArgoCD's multi-source feature handles this cleanly:</span><br /> +<br /> +<pre> +apiVersion: argoproj.io/v1alpha1 +kind: Application +metadata: + name: prometheus + namespace: cicd +spec: + sources: + # Source 1: Upstream Helm chart + - repoURL: https://prometheus-community.github.io/helm-charts + chart: kube-prometheus-stack + targetRevision: 55.5.0 + helm: + releaseName: prometheus + valuesObject: + kubeEtcd: + enabled: true + endpoints: + - 192.168.2.120 + - 192.168.2.121 + - 192.168.2.122 + # ... hundreds of lines of config + + # Source 2: Custom manifests from Git + - repoURL: http://git-server.cicd.svc.cluster.local/conf.git + targetRevision: master + path: f3s/prometheus/manifests + + syncPolicy: + automated: + prune: false # Manual pruning--too risky for the monitoring stack + selfHeal: true + syncOptions: + - ServerSideApply=true +</pre> +<br /> +<span>The <span class='inlinecode'>prometheus/manifests/</span> directory has 13 files, each with a sync wave annotation to control deployment order:</span><br /> +<br /> +<pre> +f3s/prometheus/manifests/ +├── persistent-volumes.yaml # Wave 0 +├── grafana-restart-rbac.yaml # Wave 0 +├── additional-scrape-configs-secret.yaml # Wave 1 +├── grafana-datasources-configmap.yaml # Wave 1 +├── freebsd-recording-rules.yaml # Wave 3 +├── openbsd-recording-rules.yaml # Wave 3 +├── zfs-recording-rules.yaml # Wave 3 +├── argocd-application-alerts.yaml # Wave 3 +├── epimetheus-dashboard.yaml # Wave 4 +├── zfs-dashboards.yaml # Wave 4 +├── argocd-applications-dashboard.yaml # Wave 4 +├── node-resources-multi-select-dashboard.yaml # Wave 4 +├── prometheus-nodeport.yaml # Wave 4 +└── grafana-restart-hook.yaml # Wave 10 (PostSync) +</pre> +<br /> +<h3 style='display: inline' id='sync-waves'>Sync Waves</h3><br /> +<br /> +<span>Without sync waves, ArgoCD deploys all resources at once in no particular order. That's fine for simple apps, but for something like Prometheus it causes failures--a PersistentVolumeClaim can't bind if the PersistentVolume doesn't exist yet, and a PrometheusRule can't be created if the CRD hasn't been registered.</span><br /> +<br /> +<span>Sync waves fix this. You annotate each resource with a wave number:</span><br /> +<br /> +<pre> +annotations: + argocd.argoproj.io/sync-wave: "3" +</pre> +<br /> +<span>ArgoCD deploys all wave 0 resources first, waits until they're healthy, then moves to wave 1, waits again, and so on. Resources without the annotation default to wave 0.</span><br /> +<br /> +<span>For the Prometheus stack, the waves look like this:</span><br /> +<br /> +<ul> +<li>Wave 0: PersistentVolumes, RBAC--infrastructure that everything else depends on</li> +<li>Wave 1: Secrets, ConfigMaps--config that Prometheus and Grafana need at startup</li> +<li>Wave 3: PrometheusRule CRDs--recording rules for FreeBSD, OpenBSD, ZFS, ArgoCD (the operator from wave 0 needs to be running first)</li> +<li>Wave 4: Dashboard ConfigMaps and nodeport config</li> +<li>Wave 10: PostSync hook--a Job that runs after all waves complete</li> +</ul><br /> +<span>The PostSync hook is worth explaining. ArgoCD supports lifecycle hooks (<span class='inlinecode'>PreSync</span>, <span class='inlinecode'>Sync</span>, <span class='inlinecode'>PostSync</span>) that run Jobs at specific points. The Grafana restart hook is a good example--it restarts Grafana after every sync so it picks up updated datasources and dashboards:</span><br /> +<br /> +<pre> +apiVersion: batch/v1 +kind: Job +metadata: + name: grafana-restart-hook + namespace: monitoring + annotations: + argocd.argoproj.io/hook: PostSync + argocd.argoproj.io/hook-delete-policy: BeforeHookCreation + argocd.argoproj.io/sync-wave: "10" +spec: + template: + spec: + serviceAccountName: grafana-restart-sa + restartPolicy: OnFailure + containers: + - name: kubectl + image: bitnami/kubectl:latest + command: + - /bin/sh + - -c + - | + kubectl wait --for=condition=available --timeout=300s \ + deployment/prometheus-grafana -n monitoring || true + kubectl delete pod -n monitoring \ + -l app.kubernetes.io/name=grafana --ignore-not-found=true + backoffLimit: 2 +</pre> +<br /> +<h2 style='display: inline' id='the-result'>The Result</h2><br /> +<br /> +<span>All 30 applications across 5 namespaces, synced and healthy:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>$ argocd app list +NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY +alloy https://kubernetes.default.svc monitoring default Synced Healthy Auto-Prune +anki-sync-server https://kubernetes.default.svc services default Synced Healthy Auto-Prune +apache https://kubernetes.default.svc services default Synced Healthy Auto-Prune +argo-rollouts https://kubernetes.default.svc cicd default Synced Healthy Auto-Prune +audiobookshelf https://kubernetes.default.svc services default Synced Healthy Auto-Prune +cert-manager https://kubernetes.default.svc infra default Synced Healthy Auto-Prune +filebrowser https://kubernetes.default.svc services default Synced Healthy Auto-Prune +git-server https://kubernetes.default.svc cicd default Synced Healthy Auto-Prune +grafana-ingress https://kubernetes.default.svc monitoring default Synced Healthy Auto-Prune +immich https://kubernetes.default.svc services default Synced Healthy Auto-Prune +ipv6test https://kubernetes.default.svc services default Synced Healthy Auto-Prune +jellyfin https://kubernetes.default.svc services default Synced Healthy Auto-Prune +keybr https://kubernetes.default.svc services default Synced Healthy Auto-Prune +kobo-sync-server https://kubernetes.default.svc services default Synced Healthy Auto-Prune +loki https://kubernetes.default.svc monitoring default Synced Healthy Auto-Prune +miniflux https://kubernetes.default.svc services default Synced Healthy Auto-Prune +navidrome https://kubernetes.default.svc services default Synced Healthy Auto-Prune +opodsync https://kubernetes.default.svc services default Synced Healthy Auto-Prune +pihole https://kubernetes.default.svc services default Synced Healthy Auto-Prune +pkgrepo https://kubernetes.default.svc infra default Synced Healthy Auto-Prune +prometheus https://kubernetes.default.svc monitoring default Synced Healthy Auto +pushgateway https://kubernetes.default.svc monitoring default Synced Healthy Auto-Prune +radicale https://kubernetes.default.svc services default Synced Healthy Auto-Prune +registry https://kubernetes.default.svc infra default Synced Healthy Auto-Prune +syncthing https://kubernetes.default.svc services default Synced Healthy Auto-Prune +tempo https://kubernetes.default.svc monitoring default Synced Healthy Auto-Prune +traefik-config https://kubernetes.default.svc infra default Synced Healthy Auto-Prune +tracing-demo https://kubernetes.default.svc services default Synced Healthy Auto-Prune +wallabag https://kubernetes.default.svc services default Synced Healthy Auto-Prune +webdav https://kubernetes.default.svc services default Synced Healthy Auto-Prune +</pre> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-9/argocd-apps-list.png'><img alt='ArgoCD managing all 30 applications in the f3s cluster' title='ArgoCD managing all 30 applications in the f3s cluster' src='./f3s-kubernetes-with-freebsd-part-9/argocd-apps-list.png' /></a><br /> +<br /> +<h2 style='display: inline' id='what-changed-day-to-day'>What Changed Day-to-Day</h2><br /> +<br /> +<span>The practical difference is pretty big:</span><br /> +<br /> +<ul> +<li>Single source of truth--clone the repo, look at <span class='inlinecode'>argocd-apps/</span>, and you know exactly what's running. No more <span class='inlinecode'>helm list</span> or guessing.</li> +<li>Push and forget--edit a Helm value, commit, push. ArgoCD picks it up within a few minutes. No SSH, no <span class='inlinecode'>just upgrade</span>.</li> +<li>Self-healing--I've tweaked things manually for debugging, forgotten about it, and ArgoCD quietly reverted it. That's saved me from some confusing "why is this behaving differently?" moments.</li> +<li>Rollback = git revert--<span class='inlinecode'>git revert HEAD && git push</span> and ArgoCD syncs back to the previous state.</li> +<li>Disaster recovery--bootstrap k3s, install ArgoCD, apply the Application manifests, wait. The cluster rebuilds itself. I haven't had to do this for real yet, but I've tested it and it works.</li> +<li>Drift detection--the ArgoCD UI shows immediately if something is out of sync. Much better than running <span class='inlinecode'>kubectl</span> commands and comparing output manually.</li> +</ul><br /> +<h2 style='display: inline' id='challenges-along-the-way'>Challenges Along the Way</h2><br /> +<br /> +<h3 style='display: inline' id='helm-release-adoption'>Helm Release Adoption</h3><br /> +<br /> +<span>When ArgoCD tries to manage resources already deployed by Helm, it can get confused. The fix: make sure the Application manifest matches the current Helm values exactly. ArgoCD then recognizes the resources and adopts them without re-deploying.</span><br /> +<br /> +<h3 style='display: inline' id='persistentvolumes'>PersistentVolumes</h3><br /> +<br /> +<span>PVs are cluster-scoped, and many of my Helm charts created them with <span class='inlinecode'>kubectl apply</span> outside of Helm. For simple apps I moved PV definitions into the Helm chart templates. For complex apps like Prometheus, I used the multi-source pattern with PVs in a separate <span class='inlinecode'>manifests/</span> directory at sync wave 0.</span><br /> +<br /> +<h3 style='display: inline' id='secrets'>Secrets</h3><br /> +<br /> +<span>Secrets shouldn't live in Git as plaintext. For now, I create them manually with <span class='inlinecode'>kubectl create secret</span> and reference them from Helm charts. ArgoCD doesn't manage the secrets themselves. This works fine but isn't fully declarative--External Secrets Operator is on the list for the future.</span><br /> +<br /> +<h3 style='display: inline' id='grafana-not-reloading'>Grafana Not Reloading</h3><br /> +<br /> +<span>After updating datasource ConfigMaps, Grafana wouldn't notice until the pod was restarted. The PostSync hook (the Grafana restart Job in sync wave 10) handles this automatically now.</span><br /> +<br /> +<h3 style='display: inline' id='prometheus-multi-source-ordering'>Prometheus Multi-Source Ordering</h3><br /> +<br /> +<span>Without sync waves, Prometheus resources deployed in random order and things broke. PVs need to exist before PVCs, secrets before the operator, recording rules after the CRDs are registered. Adding sync wave annotations to everything in <span class='inlinecode'>prometheus/manifests/</span> fixed all the ordering issues.</span><br /> +<br /> +<h2 style='display: inline' id='wrapping-up'>Wrapping Up</h2><br /> +<br /> +<span>The migration took a couple of days, doing one or two apps at a time. The result: 30 applications across 5 namespaces, all managed declaratively through Git. Push a change, it deploys. Break something, <span class='inlinecode'>git revert</span>. Cluster dies, rebuild from the repo.</span><br /> +<br /> +<span>All the config lives here:</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s'>codeberg.org/snonux/conf/f3s</a><br /> +<br /> +<span>ArgoCD Application manifests organized by namespace:</span><br /> +<br /> +<a class='textlink' href='https://codeberg.org/snonux/conf/src/branch/master/f3s/argocd-apps'>codeberg.org/snonux/conf/f3s/argocd-apps</a><br /> +<br /> +<span>I can't imagine going back to running Helm commands manually.</span><br /> +<br /> +<span>Other *BSD-related posts:</span><br /> +<br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD (You are currently reading this)</a><br /> +<a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> +<a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> +<a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> +<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br /> +<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br /> +<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br /> +<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br /> +<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br /> +<a class='textlink' href='./2024-04-01-KISS-high-availability-with-OpenBSD.html'>2024-04-01 KISS high-availability with OpenBSD</a><br /> +<a class='textlink' href='./2024-01-13-one-reason-why-i-love-openbsd.html'>2024-01-13 One reason why I love OpenBSD</a><br /> +<a class='textlink' href='./2022-10-30-installing-dtail-on-openbsd.html'>2022-10-30 Installing DTail on OpenBSD</a><br /> +<a class='textlink' href='./2022-07-30-lets-encrypt-with-openbsd-and-rex.html'>2022-07-30 Let's Encrypt with OpenBSD and Rex</a><br /> +<a class='textlink' href='./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html'>2016-04-09 Jails and ZFS with Puppet on FreeBSD</a><br /> +<br /> +<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br /> +<br /> +<a class='textlink' href='../'>Back to the main site</a><br /> + </div> + </content> + </entry> + <entry> <title>Distributed Systems Simulator - Part 3: Advanced Examples and Protocol API</title> <link href="gemini://foo.zone/gemfeed/2026-04-02-distributed-systems-simulator-part-3.gmi" /> <id>gemini://foo.zone/gemfeed/2026-04-02-distributed-systems-simulator-part-3.gmi</id> @@ -4539,6 +5199,7 @@ $ curl -s -G "http://localhost:3200/api/search" \ <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability (You are currently reading this)</a><br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <br /> <a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br /> <br /> @@ -4634,10 +5295,10 @@ $ curl -s -G "http://localhost:3200/api/search" \ by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ git clone https</font><font color="#F3E651">:</font><font color="#ff0000">//codeberg</font><font color="#F3E651">.</font><font color="#ff0000">org/snonux/conf</font><font color="#F3E651">.</font><font color="#ff0000">git</font> -<font color="#ff0000">$ cd conf</font> -<font color="#ff0000">$ git checkout 15a86f3 </font><i><font color="#ababab"># Last commit before ArgoCD migration</font></i> -<font color="#ff0000">$ cd f3s/prometheus</font><font color="#F3E651">/</font> +<pre>$ git clone https://codeberg.org/snonux/conf.git +$ cd conf +$ git checkout 15a86f3 <i><font color="silver"># Last commit before ArgoCD migration</font></i> +$ cd f3s/prometheus/ </pre> <br /> <span>**Current master branch** contains the ArgoCD-managed versions with:</span><br /> @@ -4674,8 +5335,8 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ kubectl create namespace monitoring</font> -<font color="#ff0000">namespace/monitoring created</font> +<pre>$ kubectl create namespace monitoring +namespace/monitoring created </pre> <br /> <h2 style='display: inline' id='installing-prometheus-and-grafana'>Installing Prometheus and Grafana</h2><br /> @@ -4690,8 +5351,8 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ helm repo add prometheus-community https</font><font color="#F3E651">:</font><font color="#ff0000">//prometheus-community</font><font color="#F3E651">.</font><font color="#ff0000">github</font><font color="#F3E651">.</font><font color="#ff0000">io/helm-charts</font> -<font color="#ff0000">$ helm repo update</font> +<pre>$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +$ helm repo update </pre> <br /> <span>Create the directories on the NFS server for persistent storage:</span><br /> @@ -4700,8 +5361,8 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># mkdir -p /data/nfs/k3svolumes/prometheus/data</font></i> -<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># mkdir -p /data/nfs/k3svolumes/grafana/data</font></i> +<pre>[root@r0 ~]<i><font color="silver"># mkdir -p /data/nfs/k3svolumes/prometheus/data</font></i> +[root@r0 ~]<i><font color="silver"># mkdir -p /data/nfs/k3svolumes/grafana/data</font></i> </pre> <br /> <h3 style='display: inline' id='deploying-with-the-justfile'>Deploying with the Justfile</h3><br /> @@ -4717,18 +5378,18 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ cd conf/f3s/prometheus</font> -<font color="#ff0000">$ just install</font> -<font color="#ff0000">kubectl apply -f persistent-volumes</font><font color="#F3E651">.</font><font color="#ff0000">yaml</font> -<font color="#ff0000">persistentvolume/prometheus-data-pv created</font> -<font color="#ff0000">persistentvolume/grafana-data-pv created</font> -<font color="#ff0000">persistentvolumeclaim/grafana-data-pvc created</font> -<font color="#ff0000">helm install prometheus prometheus-community/kube-prometheus-stack </font><font color="#F3E651">\</font> -<font color="#ff0000"> --namespace monitoring -f persistence-values</font><font color="#F3E651">.</font><font color="#ff0000">yaml</font> -<font color="#ff0000">NAME</font><font color="#F3E651">:</font><font color="#ff0000"> prometheus</font> -<font color="#ff0000">LAST DEPLOYED</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#F3E651">...</font> -<font color="#ff0000">NAMESPACE</font><font color="#F3E651">:</font><font color="#ff0000"> monitoring</font> -<font color="#ff0000">STATUS</font><font color="#F3E651">:</font><font color="#ff0000"> deployed</font> +<pre>$ cd conf/f3s/prometheus +$ just install +kubectl apply -f persistent-volumes.yaml +persistentvolume/prometheus-data-pv created +persistentvolume/grafana-data-pv created +persistentvolumeclaim/grafana-data-pvc created +helm install prometheus prometheus-community/kube-prometheus-stack \ + --namespace monitoring -f persistence-values.yaml +NAME: prometheus +LAST DEPLOYED: ... +NAMESPACE: monitoring +STATUS: deployed </pre> <br /> <span>The <span class='inlinecode'>persistence-values.yaml</span> configures Prometheus and Grafana to use the NFS-backed persistent volumes I mentioned earlier, ensuring data survives pod restarts. It also enables scraping of etcd and kube-controller-manager metrics:</span><br /> @@ -4767,12 +5428,12 @@ kubeControllerManager: by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># cat >> /etc/rancher/k3s/config.yaml << 'EOF'</font></i> -<font color="#ff0000">kube-controller-manager-arg</font><font color="#F3E651">:</font> -<font color="#ff0000"> - bind-address</font><font color="#F3E651">=</font><font color="#bb00ff">0.0</font><font color="#F3E651">.</font><font color="#bb00ff">0.0</font> -<font color="#ff0000">etcd-expose-metrics</font><font color="#F3E651">:</font><font color="#ff0000"> </font><b><font color="#ffffff">true</font></b> -<font color="#ff0000">EOF</font> -<font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># systemctl restart k3s</font></i> +<pre>[root@r0 ~]<i><font color="silver"># cat >> /etc/rancher/k3s/config.yaml << 'EOF'</font></i> +kube-controller-manager-arg: + - bind-address=<font color="#000000">0.0</font>.<font color="#000000">0.0</font> +etcd-expose-metrics: <b><u><font color="#000000">true</font></u></b> +EOF +[root@r0 ~]<i><font color="silver"># systemctl restart k3s</font></i> </pre> <br /> <span>Repeat for <span class='inlinecode'>r1</span> and <span class='inlinecode'>r2</span>. After restarting all nodes, the controller-manager metrics endpoint will be accessible and etcd metrics are available on port 2381. Prometheus can now scrape both.</span><br /> @@ -4783,8 +5444,8 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader</font></i> -<font color="#ff0000">etcd_server_has_leader </font><font color="#bb00ff">1</font> +<pre>[root@r0 ~]<i><font color="silver"># curl -s http://127.0.0.1:2381/metrics | grep etcd_server_has_leader</font></i> +etcd_server_has_leader <font color="#000000">1</font> </pre> <br /> <span>The full <span class='inlinecode'>persistence-values.yaml</span> and all other Prometheus configuration files are available on Codeberg:</span><br /> @@ -4805,9 +5466,9 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ kubectl get svc -n monitoring prometheus-kube-prometheus-prometheus</font> -<font color="#ff0000">NAME TYPE CLUSTER-IP PORT</font><font color="#F3E651">(</font><font color="#ff0000">S</font><font color="#F3E651">)</font> -<font color="#ff0000">prometheus-kube-prometheus-prometheus ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">152.163</font><font color="#ff0000"> </font><font color="#bb00ff">9090</font><font color="#ff0000">/TCP</font><font color="#F3E651">,</font><font color="#bb00ff">8080</font><font color="#ff0000">/TCP</font> +<pre>$ kubectl get svc -n monitoring prometheus-kube-prometheus-prometheus +NAME TYPE CLUSTER-IP PORT(S) +prometheus-kube-prometheus-prometheus ClusterIP <font color="#000000">10.43</font>.<font color="#000000">152.163</font> <font color="#000000">9090</font>/TCP,<font color="#000000">8080</font>/TCP </pre> <br /> <span>Grafana connects to Prometheus using the internal service URL <span class='inlinecode'>http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090</span>. The default Grafana credentials are <span class='inlinecode'>admin</span>/<span class='inlinecode'>prom-operator</span>, which should be changed immediately after first login.</span><br /> @@ -4832,7 +5493,7 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#F3E651">[</font><font color="#ff0000">root@r0 </font><font color="#F3E651">~]</font><i><font color="#ababab"># mkdir -p /data/nfs/k3svolumes/loki/data</font></i> +<pre>[root@r0 ~]<i><font color="silver"># mkdir -p /data/nfs/k3svolumes/loki/data</font></i> </pre> <br /> <h3 style='display: inline' id='deploying-loki-and-alloy'>Deploying Loki and Alloy</h3><br /> @@ -4847,24 +5508,24 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ cd conf/f3s/loki</font> -<font color="#ff0000">$ just install</font> -<font color="#ff0000">helm repo add grafana https</font><font color="#F3E651">:</font><font color="#ff0000">//grafana</font><font color="#F3E651">.</font><font color="#ff0000">github</font><font color="#F3E651">.</font><font color="#ff0000">io/helm-charts </font><font color="#F3E651">||</font><font color="#ff0000"> </font><b><font color="#ffffff">true</font></b> -<font color="#ff0000">helm repo update</font> -<font color="#ff0000">kubectl apply -f persistent-volumes</font><font color="#F3E651">.</font><font color="#ff0000">yaml</font> -<font color="#ff0000">persistentvolume/loki-data-pv created</font> -<font color="#ff0000">persistentvolumeclaim/loki-data-pvc created</font> -<font color="#ff0000">helm install loki grafana/loki --namespace monitoring -f values</font><font color="#F3E651">.</font><font color="#ff0000">yaml</font> -<font color="#ff0000">NAME</font><font color="#F3E651">:</font><font color="#ff0000"> loki</font> -<font color="#ff0000">LAST DEPLOYED</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#F3E651">...</font> -<font color="#ff0000">NAMESPACE</font><font color="#F3E651">:</font><font color="#ff0000"> monitoring</font> -<font color="#ff0000">STATUS</font><font color="#F3E651">:</font><font color="#ff0000"> deployed</font> -<font color="#F3E651">...</font> -<font color="#ff0000">helm install alloy grafana/alloy --namespace monitoring -f alloy-values</font><font color="#F3E651">.</font><font color="#ff0000">yaml</font> -<font color="#ff0000">NAME</font><font color="#F3E651">:</font><font color="#ff0000"> alloy</font> -<font color="#ff0000">LAST DEPLOYED</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#F3E651">...</font> -<font color="#ff0000">NAMESPACE</font><font color="#F3E651">:</font><font color="#ff0000"> monitoring</font> -<font color="#ff0000">STATUS</font><font color="#F3E651">:</font><font color="#ff0000"> deployed</font> +<pre>$ cd conf/f3s/loki +$ just install +helm repo add grafana https://grafana.github.io/helm-charts || <b><u><font color="#000000">true</font></u></b> +helm repo update +kubectl apply -f persistent-volumes.yaml +persistentvolume/loki-data-pv created +persistentvolumeclaim/loki-data-pvc created +helm install loki grafana/loki --namespace monitoring -f values.yaml +NAME: loki +LAST DEPLOYED: ... +NAMESPACE: monitoring +STATUS: deployed +... +helm install alloy grafana/alloy --namespace monitoring -f alloy-values.yaml +NAME: alloy +LAST DEPLOYED: ... +NAMESPACE: monitoring +STATUS: deployed </pre> <br /> <span>Loki runs in single-binary mode with a single replica (<span class='inlinecode'>loki-0</span>), which is appropriate for a home lab cluster. This means there's only one Loki pod running at any time. If the node hosting Loki fails, Kubernetes will automatically reschedule the pod to another worker node—but there will be a brief downtime (typically under a minute) while this happens. For my home lab use case, this is perfectly acceptable.</span><br /> @@ -4879,44 +5540,44 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">discovery</font><font color="#F3E651">.</font><font color="#ff0000">kubernetes </font><font color="#bb00ff">"pods"</font><font color="#ff0000"> {</font> -<font color="#ff0000"> role </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#bb00ff">"pod"</font> -<font color="#ff0000">}</font> +<pre>discovery.kubernetes <font color="#808080">"pods"</font> { + role = <font color="#808080">"pod"</font> +} -<font color="#ff0000">discovery</font><font color="#F3E651">.</font><font color="#ff0000">relabel </font><font color="#bb00ff">"pods"</font><font color="#ff0000"> {</font> -<font color="#ff0000"> targets </font><font color="#F3E651">=</font><font color="#ff0000"> discovery</font><font color="#F3E651">.</font><font color="#ff0000">kubernetes</font><font color="#F3E651">.</font><font color="#ff0000">pods</font><font color="#F3E651">.</font><font color="#ff0000">targets</font> +discovery.relabel <font color="#808080">"pods"</font> { + targets = discovery.kubernetes.pods.targets -<font color="#ff0000"> rule {</font> -<font color="#ff0000"> source_labels </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#bb00ff">"__meta_kubernetes_namespace"</font><font color="#F3E651">]</font> -<font color="#ff0000"> target_label </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#bb00ff">"namespace"</font> -<font color="#ff0000"> }</font> + rule { + source_labels = [<font color="#808080">"__meta_kubernetes_namespace"</font>] + target_label = <font color="#808080">"namespace"</font> + } -<font color="#ff0000"> rule {</font> -<font color="#ff0000"> source_labels </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#bb00ff">"__meta_kubernetes_pod_name"</font><font color="#F3E651">]</font> -<font color="#ff0000"> target_label </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#bb00ff">"pod"</font> -<font color="#ff0000"> }</font> + rule { + source_labels = [<font color="#808080">"__meta_kubernetes_pod_name"</font>] + target_label = <font color="#808080">"pod"</font> + } -<font color="#ff0000"> rule {</font> -<font color="#ff0000"> source_labels </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#bb00ff">"__meta_kubernetes_pod_container_name"</font><font color="#F3E651">]</font> -<font color="#ff0000"> target_label </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#bb00ff">"container"</font> -<font color="#ff0000"> }</font> + rule { + source_labels = [<font color="#808080">"__meta_kubernetes_pod_container_name"</font>] + target_label = <font color="#808080">"container"</font> + } -<font color="#ff0000"> rule {</font> -<font color="#ff0000"> source_labels </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#bb00ff">"__meta_kubernetes_pod_label_app"</font><font color="#F3E651">]</font> -<font color="#ff0000"> target_label </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#bb00ff">"app"</font> -<font color="#ff0000"> }</font> -<font color="#ff0000">}</font> + rule { + source_labels = [<font color="#808080">"__meta_kubernetes_pod_label_app"</font>] + target_label = <font color="#808080">"app"</font> + } +} -<font color="#ff0000">loki</font><font color="#F3E651">.</font><b><font color="#ffffff">source</font></b><font color="#F3E651">.</font><font color="#ff0000">kubernetes </font><font color="#bb00ff">"pods"</font><font color="#ff0000"> {</font> -<font color="#ff0000"> targets </font><font color="#F3E651">=</font><font color="#ff0000"> discovery</font><font color="#F3E651">.</font><font color="#ff0000">relabel</font><font color="#F3E651">.</font><font color="#ff0000">pods</font><font color="#F3E651">.</font><font color="#ff0000">output</font> -<font color="#ff0000"> forward_to </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#F3E651">[</font><font color="#ff0000">loki</font><font color="#F3E651">.</font><font color="#ff0000">write</font><font color="#F3E651">.</font><font color="#ff0000">default</font><font color="#F3E651">.</font><font color="#ff0000">receiver</font><font color="#F3E651">]</font> -<font color="#ff0000">}</font> +loki.<b><u><font color="#000000">source</font></u></b>.kubernetes <font color="#808080">"pods"</font> { + targets = discovery.relabel.pods.output + forward_to = [loki.write.default.receiver] +} -<font color="#ff0000">loki</font><font color="#F3E651">.</font><font color="#ff0000">write </font><font color="#bb00ff">"default"</font><font color="#ff0000"> {</font> -<font color="#ff0000"> endpoint {</font> -<font color="#ff0000"> url </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#bb00ff">"http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push"</font> -<font color="#ff0000"> }</font> -<font color="#ff0000">}</font> +loki.write <font color="#808080">"default"</font> { + endpoint { + url = <font color="#808080">"http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push"</font> + } +} </pre> <br /> <span>This configuration automatically labels each log line with the namespace, pod name, container name, and app label, making it easy to filter logs in Grafana.</span><br /> @@ -4929,9 +5590,9 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ kubectl get svc -n monitoring loki</font> -<font color="#ff0000">NAME TYPE CLUSTER-IP PORT</font><font color="#F3E651">(</font><font color="#ff0000">S</font><font color="#F3E651">)</font> -<font color="#ff0000">loki ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">64.60</font><font color="#ff0000"> </font><font color="#bb00ff">3100</font><font color="#ff0000">/TCP</font><font color="#F3E651">,</font><font color="#bb00ff">9095</font><font color="#ff0000">/TCP</font> +<pre>$ kubectl get svc -n monitoring loki +NAME TYPE CLUSTER-IP PORT(S) +loki ClusterIP <font color="#000000">10.43</font>.<font color="#000000">64.60</font> <font color="#000000">3100</font>/TCP,<font color="#000000">9095</font>/TCP </pre> <br /> <span>To add Loki as a data source in Grafana:</span><br /> @@ -4955,21 +5616,21 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ kubectl get pods -n monitoring</font> -<font color="#ff0000">NAME READY STATUS RESTARTS AGE</font> -<font color="#ff0000">alertmanager-prometheus-kube-prometheus-alertmanager-</font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#bb00ff">2</font><font color="#F3E651">/</font><font color="#bb00ff">2</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 42d</font> -<font color="#ff0000">alloy-g5fgj </font><font color="#bb00ff">2</font><font color="#F3E651">/</font><font color="#bb00ff">2</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 29m</font> -<font color="#ff0000">alloy-nfw8w </font><font color="#bb00ff">2</font><font color="#F3E651">/</font><font color="#bb00ff">2</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 29m</font> -<font color="#ff0000">alloy-tg9vj </font><font color="#bb00ff">2</font><font color="#F3E651">/</font><font color="#bb00ff">2</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 29m</font> -<font color="#ff0000">loki-</font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#bb00ff">2</font><font color="#F3E651">/</font><font color="#bb00ff">2</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 25m</font> -<font color="#ff0000">prometheus-grafana-868f9dc7cf-lg2vl </font><font color="#bb00ff">3</font><font color="#F3E651">/</font><font color="#bb00ff">3</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 42d</font> -<font color="#ff0000">prometheus-kube-prometheus-operator-8d7bbc48c-p4sf4 </font><font color="#bb00ff">1</font><font color="#F3E651">/</font><font color="#bb00ff">1</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 42d</font> -<font color="#ff0000">prometheus-kube-state-metrics-7c5fb9d798-hh2fx </font><font color="#bb00ff">1</font><font color="#F3E651">/</font><font color="#bb00ff">1</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 42d</font> -<font color="#ff0000">prometheus-prometheus-kube-prometheus-prometheus-</font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#bb00ff">2</font><font color="#F3E651">/</font><font color="#bb00ff">2</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 42d</font> -<font color="#ff0000">prometheus-prometheus-node-exporter-2nsg9 </font><font color="#bb00ff">1</font><font color="#F3E651">/</font><font color="#bb00ff">1</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 42d</font> -<font color="#ff0000">prometheus-prometheus-node-exporter-mqr</font><font color="#bb00ff">25</font><font color="#ff0000"> </font><font color="#bb00ff">1</font><font color="#F3E651">/</font><font color="#bb00ff">1</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 42d</font> -<font color="#ff0000">prometheus-prometheus-node-exporter-wp4ds </font><font color="#bb00ff">1</font><font color="#F3E651">/</font><font color="#bb00ff">1</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 42d</font> -<font color="#ff0000">tempo-</font><font color="#bb00ff">0</font><font color="#ff0000"> </font><font color="#bb00ff">1</font><font color="#F3E651">/</font><font color="#bb00ff">1</font><font color="#ff0000"> Running </font><font color="#bb00ff">0</font><font color="#ff0000"> 1d</font> +<pre>$ kubectl get pods -n monitoring +NAME READY STATUS RESTARTS AGE +alertmanager-prometheus-kube-prometheus-alertmanager-<font color="#000000">0</font> <font color="#000000">2</font>/<font color="#000000">2</font> Running <font color="#000000">0</font> 42d +alloy-g5fgj <font color="#000000">2</font>/<font color="#000000">2</font> Running <font color="#000000">0</font> 29m +alloy-nfw8w <font color="#000000">2</font>/<font color="#000000">2</font> Running <font color="#000000">0</font> 29m +alloy-tg9vj <font color="#000000">2</font>/<font color="#000000">2</font> Running <font color="#000000">0</font> 29m +loki-<font color="#000000">0</font> <font color="#000000">2</font>/<font color="#000000">2</font> Running <font color="#000000">0</font> 25m +prometheus-grafana-868f9dc7cf-lg2vl <font color="#000000">3</font>/<font color="#000000">3</font> Running <font color="#000000">0</font> 42d +prometheus-kube-prometheus-operator-8d7bbc48c-p4sf4 <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 42d +prometheus-kube-state-metrics-7c5fb9d798-hh2fx <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 42d +prometheus-prometheus-kube-prometheus-prometheus-<font color="#000000">0</font> <font color="#000000">2</font>/<font color="#000000">2</font> Running <font color="#000000">0</font> 42d +prometheus-prometheus-node-exporter-2nsg9 <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 42d +prometheus-prometheus-node-exporter-mqr<font color="#000000">25</font> <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 42d +prometheus-prometheus-node-exporter-wp4ds <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 42d +tempo-<font color="#000000">0</font> <font color="#000000">1</font>/<font color="#000000">1</font> Running <font color="#000000">0</font> 1d </pre> <br /> <span>Note: Tempo (<span class='inlinecode'>tempo-0</span>) is deployed later in this post in the "Distributed Tracing with Grafana Tempo" section. It is included in the pod listing here for completeness.</span><br /> @@ -4980,19 +5641,19 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ kubectl get svc -n monitoring</font> -<font color="#ff0000">NAME TYPE CLUSTER-IP PORT</font><font color="#F3E651">(</font><font color="#ff0000">S</font><font color="#F3E651">)</font> -<font color="#ff0000">alertmanager-operated ClusterIP None </font><font color="#bb00ff">9093</font><font color="#ff0000">/TCP</font><font color="#F3E651">,</font><font color="#bb00ff">9094</font><font color="#ff0000">/TCP</font> -<font color="#ff0000">alloy ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">74.14</font><font color="#ff0000"> </font><font color="#bb00ff">12345</font><font color="#ff0000">/TCP</font> -<font color="#ff0000">loki ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">64.60</font><font color="#ff0000"> </font><font color="#bb00ff">3100</font><font color="#ff0000">/TCP</font><font color="#F3E651">,</font><font color="#bb00ff">9095</font><font color="#ff0000">/TCP</font> -<font color="#ff0000">loki-headless ClusterIP None </font><font color="#bb00ff">3100</font><font color="#ff0000">/TCP</font> -<font color="#ff0000">prometheus-grafana ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">46.82</font><font color="#ff0000"> </font><font color="#bb00ff">80</font><font color="#ff0000">/TCP</font> -<font color="#ff0000">prometheus-kube-prometheus-alertmanager ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">208.43</font><font color="#ff0000"> </font><font color="#bb00ff">9093</font><font color="#ff0000">/TCP</font><font color="#F3E651">,</font><font color="#bb00ff">8080</font><font color="#ff0000">/TCP</font> -<font color="#ff0000">prometheus-kube-prometheus-operator ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">246.121</font><font color="#ff0000"> </font><font color="#bb00ff">443</font><font color="#ff0000">/TCP</font> -<font color="#ff0000">prometheus-kube-prometheus-prometheus ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">152.163</font><font color="#ff0000"> </font><font color="#bb00ff">9090</font><font color="#ff0000">/TCP</font><font color="#F3E651">,</font><font color="#bb00ff">8080</font><font color="#ff0000">/TCP</font> -<font color="#ff0000">prometheus-kube-state-metrics ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">64.26</font><font color="#ff0000"> </font><font color="#bb00ff">8080</font><font color="#ff0000">/TCP</font> -<font color="#ff0000">prometheus-prometheus-node-exporter ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">127.242</font><font color="#ff0000"> </font><font color="#bb00ff">9100</font><font color="#ff0000">/TCP</font> -<font color="#ff0000">tempo ClusterIP </font><font color="#bb00ff">10.43</font><font color="#F3E651">.</font><font color="#bb00ff">91.44</font><font color="#ff0000"> </font><font color="#bb00ff">3200</font><font color="#ff0000">/TCP</font><font color="#F3E651">,</font><font color="#bb00ff">4317</font><font color="#ff0000">/TCP</font><font color="#F3E651">,</font><font color="#bb00ff">4318</font><font color="#ff0000">/TCP</font> +<pre>$ kubectl get svc -n monitoring +NAME TYPE CLUSTER-IP PORT(S) +alertmanager-operated ClusterIP None <font color="#000000">9093</font>/TCP,<font color="#000000">9094</font>/TCP +alloy ClusterIP <font color="#000000">10.43</font>.<font color="#000000">74.14</font> <font color="#000000">12345</font>/TCP +loki ClusterIP <font color="#000000">10.43</font>.<font color="#000000">64.60</font> <font color="#000000">3100</font>/TCP,<font color="#000000">9095</font>/TCP +loki-headless ClusterIP None <font color="#000000">3100</font>/TCP +prometheus-grafana ClusterIP <font color="#000000">10.43</font>.<font color="#000000">46.82</font> <font color="#000000">80</font>/TCP +prometheus-kube-prometheus-alertmanager ClusterIP <font color="#000000">10.43</font>.<font color="#000000">208.43</font> <font color="#000000">9093</font>/TCP,<font color="#000000">8080</font>/TCP +prometheus-kube-prometheus-operator ClusterIP <font color="#000000">10.43</font>.<font color="#000000">246.121</font> <font color="#000000">443</font>/TCP +prometheus-kube-prometheus-prometheus ClusterIP <font color="#000000">10.43</font>.<font color="#000000">152.163</font> <font color="#000000">9090</font>/TCP,<font color="#000000">8080</font>/TCP +prometheus-kube-state-metrics ClusterIP <font color="#000000">10.43</font>.<font color="#000000">64.26</font> <font color="#000000">8080</font>/TCP +prometheus-prometheus-node-exporter ClusterIP <font color="#000000">10.43</font>.<font color="#000000">127.242</font> <font color="#000000">9100</font>/TCP +tempo ClusterIP <font color="#000000">10.43</font>.<font color="#000000">91.44</font> <font color="#000000">3200</font>/TCP,<font color="#000000">4317</font>/TCP,<font color="#000000">4318</font>/TCP </pre> <br /> <span>Let me break down what each pod does:</span><br /> @@ -5069,7 +5730,7 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas pkg install -y node_exporter</font> +<pre>paul@f0:~ % doas pkg install -y node_exporter </pre> <br /> <span>Enable the service to start at boot:</span><br /> @@ -5078,8 +5739,8 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas sysrc </font><font color="#ff0000">node_exporter_enable</font><font color="#F3E651">=</font><font color="#ff0000">YES</font> -<font color="#ff0000">node_exporter_enable</font><font color="#F3E651">:</font><font color="#ff0000"> -</font><font color="#F3E651">></font><font color="#ff0000"> YES</font> +<pre>paul@f0:~ % doas sysrc node_exporter_enable=YES +node_exporter_enable: -> YES </pre> <br /> <span>Configure node_exporter to listen on the WireGuard interface. This ensures metrics are only accessible through the secure tunnel, not the public network. Replace the IP with the host's WireGuard address:</span><br /> @@ -5088,8 +5749,8 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas sysrc </font><font color="#ff0000">node_exporter_args</font><font color="#F3E651">=</font><font color="#bb00ff">'--web.listen-address=192.168.2.130:9100'</font> -<font color="#ff0000">node_exporter_args</font><font color="#F3E651">:</font><font color="#ff0000"> -</font><font color="#F3E651">></font><font color="#ff0000"> --web</font><font color="#F3E651">.</font><font color="#ff0000">listen-address</font><font color="#F3E651">=</font><font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">2.130</font><font color="#F3E651">:</font><font color="#bb00ff">9100</font> +<pre>paul@f0:~ % doas sysrc node_exporter_args=<font color="#808080">'--web.listen-address=192.168.2.130:9100'</font> +node_exporter_args: -> --web.listen-address=<font color="#000000">192.168</font>.<font color="#000000">2.130</font>:<font color="#000000">9100</font> </pre> <br /> <span>Start the service:</span><br /> @@ -5098,8 +5759,8 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> doas service node_exporter start</font> -<font color="#ff0000">Starting node_exporter</font><font color="#F3E651">.</font> +<pre>paul@f0:~ % doas service node_exporter start +Starting node_exporter. </pre> <br /> <span>Verify it's running:</span><br /> @@ -5108,10 +5769,10 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">paul@f0</font><font color="#F3E651">:~</font><font color="#ff0000"> </font><font color="#F3E651">%</font><font color="#ff0000"> curl -s http</font><font color="#F3E651">://</font><font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">2.130</font><font color="#F3E651">:</font><font color="#bb00ff">9100</font><font color="#ff0000">/metrics </font><font color="#F3E651">|</font><font color="#ff0000"> head -</font><font color="#bb00ff">3</font> -<i><font color="#ababab"># HELP go_gc_duration_seconds A summary of the wall-time pause...</font></i> -<i><font color="#ababab"># TYPE go_gc_duration_seconds summary</font></i> -<font color="#ff0000">go_gc_duration_seconds{</font><font color="#ff0000">quantile</font><font color="#F3E651">=</font><font color="#bb00ff">"0"</font><font color="#ff0000">} </font><font color="#bb00ff">0</font> +<pre>paul@f0:~ % curl -s http://<font color="#000000">192.168</font>.<font color="#000000">2.130</font>:<font color="#000000">9100</font>/metrics | head -<font color="#000000">3</font> +<i><font color="silver"># HELP go_gc_duration_seconds A summary of the wall-time pause...</font></i> +<i><font color="silver"># TYPE go_gc_duration_seconds summary</font></i> +go_gc_duration_seconds{quantile=<font color="#808080">"0"</font>} <font color="#000000">0</font> </pre> <br /> <span>Repeat for the other FreeBSD hosts (<span class='inlinecode'>f1</span>, <span class='inlinecode'>f2</span>) with their respective WireGuard IPs.</span><br /> @@ -5139,9 +5800,9 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ kubectl create secret generic additional-scrape-configs </font><font color="#F3E651">\</font> -<font color="#ff0000"> --from-file</font><font color="#F3E651">=</font><font color="#ff0000">additional-scrape-configs</font><font color="#F3E651">.</font><font color="#ff0000">yaml </font><font color="#F3E651">\</font> -<font color="#ff0000"> -n monitoring</font> +<pre>$ kubectl create secret generic additional-scrape-configs \ + --from-file=additional-scrape-configs.yaml \ + -n monitoring </pre> <br /> <span>Update <span class='inlinecode'>persistence-values.yaml</span> to reference the secret:</span><br /> @@ -5161,7 +5822,7 @@ prometheus: by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">$ just upgrade</font> +<pre>$ just upgrade </pre> <br /> <span>After a minute or so, the FreeBSD hosts appear in the Prometheus targets and in the Node Exporter dashboards in Grafana.</span><br /> @@ -5529,10 +6190,10 @@ zfs_pool_free_bytes{pool="zdata"} 3.48809678848e+11 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">blowfish</font><font color="#F3E651">:~</font><font color="#ff0000"> $ doas pkg_add node_exporter</font> -<font color="#ff0000">quirks-</font><font color="#bb00ff">7.103</font><font color="#ff0000"> signed on </font><font color="#bb00ff">2025</font><font color="#ff0000">-</font><font color="#bb00ff">10</font><font color="#ff0000">-13T22</font><font color="#F3E651">:</font><font color="#bb00ff">55</font><font color="#F3E651">:</font><font color="#ff0000">16Z</font> -<font color="#ff0000">The following new rcscripts were installed</font><font color="#F3E651">:</font><font color="#ff0000"> /etc/rc</font><font color="#F3E651">.</font><font color="#ff0000">d/node_exporter</font> -<font color="#ff0000">See rcctl</font><font color="#F3E651">(</font><font color="#bb00ff">8</font><font color="#F3E651">)</font><font color="#ff0000"> </font><b><font color="#ffffff">for</font></b><font color="#ff0000"> details</font><font color="#F3E651">.</font> +<pre>blowfish:~ $ doas pkg_add node_exporter +quirks-<font color="#000000">7.103</font> signed on <font color="#000000">2025</font>-<font color="#000000">10</font>-13T22:<font color="#000000">55</font>:16Z +The following new rcscripts were installed: /etc/rc.d/node_exporter +See rcctl(<font color="#000000">8</font>) <b><u><font color="#000000">for</font></u></b> details. </pre> <br /> <span>Enable the service to start at boot:</span><br /> @@ -5541,7 +6202,7 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">blowfish</font><font color="#F3E651">:~</font><font color="#ff0000"> $ doas rcctl </font><b><font color="#ffffff">enable</font></b><font color="#ff0000"> node_exporter</font> +<pre>blowfish:~ $ doas rcctl <b><u><font color="#000000">enable</font></u></b> node_exporter </pre> <br /> <span>Configure node_exporter to listen on the WireGuard interface. This ensures metrics are only accessible through the secure tunnel, not the public network. Replace the IP with the host's WireGuard address:</span><br /> @@ -5550,7 +6211,7 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">blowfish</font><font color="#F3E651">:~</font><font color="#ff0000"> $ doas rcctl </font><b><font color="#ffffff">set</font></b><font color="#ff0000"> node_exporter flags </font><font color="#bb00ff">'--web.listen-address=192.168.2.110:9100'</font> +<pre>blowfish:~ $ doas rcctl <b><u><font color="#000000">set</font></u></b> node_exporter flags <font color="#808080">'--web.listen-address=192.168.2.110:9100'</font> </pre> <br /> <span>Start the service:</span><br /> @@ -5559,8 +6220,8 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">blowfish</font><font color="#F3E651">:~</font><font color="#ff0000"> $ doas rcctl start node_exporter</font> -<font color="#ff0000">node_exporter</font><font color="#F3E651">(</font><font color="#ff0000">ok</font><font color="#F3E651">)</font> +<pre>blowfish:~ $ doas rcctl start node_exporter +node_exporter(ok) </pre> <br /> <span>Verify it's running:</span><br /> @@ -5569,10 +6230,10 @@ http://www.gnu.org/software/src-highlite --> by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ff0000">blowfish</font><font color="#F3E651">:~</font><font color="#ff0000"> $ curl -s http</font><font color="#F3E651">://</font><font color="#bb00ff">192.168</font><font color="#F3E651">.</font><font color="#bb00ff">2.110</font><font color="#F3E651">:</font><font color="#bb00ff">9100</font><font color="#ff0000">/metrics </font><font color="#F3E651">|</font><font color="#ff0000"> head -</font><font color="#bb00ff">3</font> -<i><font color="#ababab"># HELP go_gc_duration_seconds A summary of the wall-time pause...</font></i> -<i><font color="#ababab"># TYPE go_gc_duration_seconds summary</font></i> -<font color="#ff0000">go_gc_duration_seconds{</font><font color="#ff0000">quantile</font><font color="#F3E651">=</font><font color="#bb00ff">"0"</font><font color="#ff0000">} </font><font color="#bb00ff">0</font> +<pre>blowfish:~ $ curl -s http://<font color="#000000">192.168</font>.<font color="#000000">2.110</font>:<font color="#000000">9100</font>/metrics | head -<font color="#000000">3</font> +<i><font color="silver"># HELP go_gc_duration_seconds A summary of the wall-time pause...</font></i> +<i><font color="silver"># TYPE go_gc_duration_seconds summary</font></i> +go_gc_duration_seconds{quantile=<font color="#808080">"0"</font>} <font color="#000000">0</font> </pre> <br /> <span>Repeat for the other OpenBSD host (<span class='inlinecode'>fishfinger</span>) with its respective WireGuard IP (<span class='inlinecode'>192.168.2.111</span>).</span><br /> @@ -5934,35 +6595,35 @@ opentelemetry-instrumentation-requests==0.49b0 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#ababab">from</font><font color="#ff0000"> opentelemetry </font><font color="#ababab">import</font><font color="#ff0000"> trace</font> -<font color="#ababab">from</font><font color="#ff0000"> opentelemetry</font><font color="#F3E651">.</font><font color="#ff0000">sdk</font><font color="#F3E651">.</font><font color="#ff0000">trace </font><font color="#ababab">import</font><font color="#ff0000"> TracerProvider</font> -<font color="#ababab">from</font><font color="#ff0000"> opentelemetry</font><font color="#F3E651">.</font><font color="#ff0000">exporter</font><font color="#F3E651">.</font><font color="#ff0000">otlp</font><font color="#F3E651">.</font><font color="#ff0000">proto</font><font color="#F3E651">.</font><font color="#ff0000">grpc</font><font color="#F3E651">.</font><font color="#ff0000">trace_exporter </font><font color="#ababab">import</font><font color="#ff0000"> OTLPSpanExporter</font> -<font color="#ababab">from</font><font color="#ff0000"> opentelemetry</font><font color="#F3E651">.</font><font color="#ff0000">instrumentation</font><font color="#F3E651">.</font><font color="#ff0000">flask </font><font color="#ababab">import</font><font color="#ff0000"> FlaskInstrumentor</font> -<font color="#ababab">from</font><font color="#ff0000"> opentelemetry</font><font color="#F3E651">.</font><font color="#ff0000">instrumentation</font><font color="#F3E651">.</font><font color="#ff0000">requests </font><font color="#ababab">import</font><font color="#ff0000"> RequestsInstrumentor</font> -<font color="#ababab">from</font><font color="#ff0000"> opentelemetry</font><font color="#F3E651">.</font><font color="#ff0000">sdk</font><font color="#F3E651">.</font><font color="#ff0000">resources </font><font color="#ababab">import</font><font color="#ff0000"> Resource</font> +<pre><b><u><font color="#000000">from</font></u></b> opentelemetry <b><u><font color="#000000">import</font></u></b> trace +<b><u><font color="#000000">from</font></u></b> opentelemetry.sdk.trace <b><u><font color="#000000">import</font></u></b> TracerProvider +<b><u><font color="#000000">from</font></u></b> opentelemetry.exporter.otlp.proto.grpc.trace_exporter <b><u><font color="#000000">import</font></u></b> OTLPSpanExporter +<b><u><font color="#000000">from</font></u></b> opentelemetry.instrumentation.flask <b><u><font color="#000000">import</font></u></b> FlaskInstrumentor +<b><u><font color="#000000">from</font></u></b> opentelemetry.instrumentation.requests <b><u><font color="#000000">import</font></u></b> RequestsInstrumentor +<b><u><font color="#000000">from</font></u></b> opentelemetry.sdk.resources <b><u><font color="#000000">import</font></u></b> Resource -<i><font color="#ababab"># Define service identity</font></i> -<font color="#ff0000">resource </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#7bc710">Resource</font><font color="#F3E651">(</font><font color="#ff0000">attributes</font><font color="#F3E651">={</font> -<font color="#ff0000"> </font><font color="#bb00ff">"service.name"</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">"frontend"</font><font color="#F3E651">,</font> -<font color="#ff0000"> </font><font color="#bb00ff">"service.namespace"</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">"tracing-demo"</font><font color="#F3E651">,</font> -<font color="#ff0000"> </font><font color="#bb00ff">"service.version"</font><font color="#F3E651">:</font><font color="#ff0000"> </font><font color="#bb00ff">"1.0.0"</font> -<font color="#F3E651">})</font> +<i><font color="silver"># Define service identity</font></i> +resource = Resource(attributes={ + <font color="#808080">"service.name"</font>: <font color="#808080">"frontend"</font>, + <font color="#808080">"service.namespace"</font>: <font color="#808080">"tracing-demo"</font>, + <font color="#808080">"service.version"</font>: <font color="#808080">"1.0.0"</font> +}) -<font color="#ff0000">provider </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#7bc710">TracerProvider</font><font color="#F3E651">(</font><font color="#ff0000">resource</font><font color="#F3E651">=</font><font color="#ff0000">resource</font><font color="#F3E651">)</font> +provider = TracerProvider(resource=resource) -<i><font color="#ababab"># Export to Alloy</font></i> -<font color="#ff0000">otlp_exporter </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#7bc710">OTLPSpanExporter</font><font color="#F3E651">(</font> -<font color="#ff0000"> endpoint</font><font color="#F3E651">=</font><font color="#bb00ff">"http://alloy.monitoring.svc.cluster.local:4317"</font><font color="#F3E651">,</font> -<font color="#ff0000"> insecure</font><font color="#F3E651">=</font><font color="#ff0000">True</font> -<font color="#F3E651">)</font> +<i><font color="silver"># Export to Alloy</font></i> +otlp_exporter = OTLPSpanExporter( + endpoint=<font color="#808080">"http://alloy.monitoring.svc.cluster.local:4317"</font>, + insecure=True +) -<font color="#ff0000">processor </font><font color="#F3E651">=</font><font color="#ff0000"> </font><font color="#7bc710">BatchSpanProcessor</font><font color="#F3E651">(</font><font color="#ff0000">otlp_exporter</font><font color="#F3E651">)</font> -<font color="#ff0000">provider</font><font color="#F3E651">.</font><font color="#7bc710">add_span_processor</font><font color="#F3E651">(</font><font color="#ff0000">processor</font><font color="#F3E651">)</font> -<font color="#ff0000">trace</font><font color="#F3E651">.</font><font color="#7bc710">set_tracer_provider</font><font color="#F3E651">(</font><font color="#ff0000">provider</font><font color="#F3E651">)</font> +processor = BatchSpanProcessor(otlp_exporter) +provider.add_span_processor(processor) +trace.set_tracer_provider(provider) -<i><font color="#ababab"># Auto-instrument Flask and requests</font></i> -<font color="#7bc710">FlaskInstrumentor</font><font color="#F3E651">().</font><font color="#7bc710">instrument_app</font><font color="#F3E651">(</font><font color="#ff0000">app</font><font color="#F3E651">)</font> -<font color="#7bc710">RequestsInstrumentor</font><font color="#F3E651">().</font><font color="#7bc710">instrument</font><font color="#F3E651">()</font> +<i><font color="silver"># Auto-instrument Flask and requests</font></i> +FlaskInstrumentor().instrument_app(app) +RequestsInstrumentor().instrument() </pre> <br /> <span>The auto-instrumentation automatically:</span><br /> @@ -6139,29 +6800,29 @@ curl -H "Host: tracing-demo.f3s.buetow.org" http://r0/api/process by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#F3E651">{</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">middleware_response</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">backend_data</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">data</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">id</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#bb00ff">12345</font><font color="#F3E651">,</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">query_time_ms</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#bb00ff">100.0</font><font color="#F3E651">,</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">timestamp</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">2025-12-28T18:35:01.064538</font><font color="#ff0000">"</font><font color="#F3E651">,</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">value</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">Sample data from backend service</font><font color="#ff0000">"</font> -<font color="#ff0000"> </font><font color="#F3E651">},</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">service</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">backend</font><font color="#ff0000">"</font> -<font color="#ff0000"> </font><font color="#F3E651">},</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">middleware_processed</font><font color="#ff0000">"</font><font color="#ff0000">: </font><b><font color="#ffffff">true</font></b><font color="#F3E651">,</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">original_data</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">source</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">GET request</font><font color="#ff0000">"</font> -<font color="#ff0000"> </font><font color="#F3E651">},</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">transformation_time_ms</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#bb00ff">50</font> -<font color="#ff0000"> </font><font color="#F3E651">},</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">request_data</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#F3E651">{</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">source</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">GET request</font><font color="#ff0000">"</font> -<font color="#ff0000"> </font><font color="#F3E651">},</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">service</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">frontend</font><font color="#ff0000">"</font><font color="#F3E651">,</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">status</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">success</font><font color="#ff0000">"</font> -<font color="#F3E651">}</font> +<pre>{ + "middleware_response": { + "backend_data": { + "data": { + "id": <font color="#000000">12345</font>, + "query_time_ms": <font color="#000000">100.0</font>, + "timestamp": "<font color="#808080">2025-12-28T18:35:01.064538</font>", + "value": "<font color="#808080">Sample data from backend service</font>" + }, + "service": "<font color="#808080">backend</font>" + }, + "middleware_processed": <b><u><font color="#000000">true</font></u></b>, + "original_data": { + "source": "<font color="#808080">GET request</font>" + }, + "transformation_time_ms": <font color="#000000">50</font> + }, + "request_data": { + "source": "<font color="#808080">GET request</font>" + }, + "service": "<font color="#808080">frontend</font>", + "status": "<font color="#808080">success</font>" +} </pre> <br /> <span>**2. Find the trace in Tempo via API:**</span><br /> @@ -6180,12 +6841,12 @@ kubectl exec -n monitoring tempo-0 -- wget -qO- \ by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre><font color="#F3E651">{</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">traceID</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">4be1151c0bdcd5625ac7e02b98d95bd5</font><font color="#ff0000">"</font><font color="#F3E651">,</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">rootServiceName</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">frontend</font><font color="#ff0000">"</font><font color="#F3E651">,</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">rootTraceName</font><font color="#ff0000">"</font><font color="#ff0000">:</font><font color="#ff0000"> "</font><font color="#bb00ff">GET /api/process</font><font color="#ff0000">"</font><font color="#F3E651">,</font> -<font color="#ff0000"> </font><font color="#ff0000">"</font><font color="#ff0000">durationMs</font><font color="#ff0000">"</font><font color="#ff0000">: </font><font color="#bb00ff">221</font> -<font color="#F3E651">}</font> +<pre>{ + "traceID": "<font color="#808080">4be1151c0bdcd5625ac7e02b98d95bd5</font>", + "rootServiceName": "<font color="#808080">frontend</font>", + "rootTraceName": "<font color="#808080">GET /api/process</font>", + "durationMs": <font color="#000000">221</font> +} </pre> <br /> <span>**3. Fetch complete trace details:**</span><br /> @@ -6309,6 +6970,7 @@ kubectl exec -n monitoring <tempo-pod> -- df -h /var/tempo <br /> <span>Other *BSD-related posts:</span><br /> <br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability (You are currently reading this)</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> @@ -7233,6 +7895,7 @@ p hash.values_at(:a, :c) <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <br /> <a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br /> <br /> @@ -8700,6 +9363,7 @@ replicaset.apps/miniflux-server-85d7c64664 1 1 1 54d <br /> <span>Other *BSD-related posts:</span><br /> <br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)</a><br /> <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> @@ -10007,6 +10671,7 @@ content = "{CODE}" <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <br /> <a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br /> <br /> @@ -12161,6 +12826,7 @@ http://www.gnu.org/software/src-highlite --> <br /> <span>Other *BSD-related posts:</span><br /> <br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)</a><br /> @@ -13235,6 +13901,7 @@ http://www.gnu.org/software/src-highlite --> <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <br /> <a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br /> <br /> @@ -14780,6 +15447,7 @@ earth$ curl https://ifconfig.me <i><font color="silver"># Should show gateway's <br /> <span>Other *BSD-related posts:</span><br /> <br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> @@ -15373,6 +16041,7 @@ __ejm\___/________dwb`---`______________________ <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <br /> <a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br /> <br /> @@ -16063,6 +16732,7 @@ etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408 <br /> <span>Other *BSD-related posts:</span><br /> <br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> @@ -16788,6 +17458,7 @@ This is perl, v5.<font color="#000000">8.8</font> built <b><u><font color="#0000 <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <br /> <a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br /> <br /> @@ -17183,6 +17854,7 @@ Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd shutdown succeeded <br /> <span>Other BSD related posts are:</span><br /> <br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> @@ -17818,6 +18490,7 @@ Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd shutdown succeeded <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <br /> <a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br /> <br /> @@ -18305,6 +18978,7 @@ Waking up e8:ff:1e:d7:1c:a0... <br /> <span>Other *BSD-related posts:</span><br /> <br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> @@ -18355,6 +19029,7 @@ Waking up e8:ff:1e:d7:1c:a0... <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <br /> <a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br /> <br /> @@ -18506,6 +19181,7 @@ Waking up e8:ff:1e:d7:1c:a0... <br /> <span>Other *BSD-related posts:</span><br /> <br /> +<a class='textlink' href='./2026-04-02-f3s-kubernetes-with-freebsd-part-9.html'>2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD</a><br /> <a class='textlink' href='./2025-12-07-f3s-kubernetes-with-freebsd-part-8.html'>2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability</a><br /> <a class='textlink' href='./2025-10-02-f3s-kubernetes-with-freebsd-part-7.html'>2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments</a><br /> <a class='textlink' href='./2025-07-14-f3s-kubernetes-with-freebsd-part-6.html'>2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage</a><br /> @@ -19254,94 +19930,4 @@ jgs \\`_..---.Y.---.._`// </div> </content> </entry> - <entry> - <title>'The Stoic Challenge' book notes</title> - <link href="gemini://foo.zone/gemfeed/2024-07-07-the-stoic-challenge-book-notes.gmi" /> - <id>gemini://foo.zone/gemfeed/2024-07-07-the-stoic-challenge-book-notes.gmi</id> - <updated>2024-07-07T12:46:55+03:00</updated> - <author> - <name>Paul Buetow aka snonux</name> - <email>paul@dev.buetow.org</email> - </author> - <summary>These are my personal takeaways after reading 'The Stoic Challenge: A Philosopher's Guide to Becoming Tougher, Calmer, and More Resilient' by William B. Irvine. </summary> - <content type="xhtml"> - <div xmlns="http://www.w3.org/1999/xhtml"> - <h1 style='display: inline' id='the-stoic-challenge-book-notes'>"The Stoic Challenge" book notes</h1><br /> -<br /> -<span class='quote'>Published at 2024-07-07T12:46:55+03:00</span><br /> -<br /> -<span>These are my personal takeaways after reading "The Stoic Challenge: A Philosopher's Guide to Becoming Tougher, Calmer, and More Resilient" by William B. Irvine. </span><br /> -<br /> -<pre> - ,.......... .........., - ,..,' '.' ',.., - ,' ,' : ', ', - ,' ,' : ', ', - ,' ,' : ', ', - ,' ,'............., : ,.............', ', -,' '............ '.' ............' ', - '''''''''''''''''';''';'''''''''''''''''' - ''' -</pre> -<br /> -<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br /> -<br /> -<ul> -<li><a href='#the-stoic-challenge-book-notes'>"The Stoic Challenge" book notes</a></li> -<li>⇢ <a href='#god-sets-you-up-for-a-challenge'>God sets you up for a challenge</a></li> -<li>⇢ <a href='#negative-visualization'>Negative visualization</a></li> -<li>⇢ <a href='#oh-nice-trick-you-stoic-god--'>Oh, nice trick, you stoic "god"! ;-)</a></li> -</ul><br /> -<h2 style='display: inline' id='god-sets-you-up-for-a-challenge'>God sets you up for a challenge</h2><br /> -<br /> -<span>Gods set you up for a challenge to see how resilient you are. Is getting angry worth the price? If you stay calm then you can find the optimal workaround for the obstacle. Stay calm even with big setbacks. Practice minimalism of negative emotions.</span><br /> -<br /> -<span>Put a positive spin on everything. What should you do if someone wrong you? Don't get angry, there is no point in that, it just makes you suffer. Do the best what you got now and keep calm and carry on. A resilient person will refuse to play the role of a victim. You can develop the setback response skills. Turn a setback. e.g. a handycap, into a personal triumph.</span><br /> -<br /> -<span>It is not the things done to you or happen to you what matters but how you take the things and react to these things.</span><br /> -<br /> -<span>Don't row against the other boats but against your own lazy bill. It doesn't matter if you are first or last, as long as you defeat your lazy self.</span><br /> -<br /> -<span>Stoics are thankful that they are mortal. As then you can get reminded of how great it is to be alive at all. In dying we are more alive we have ever been as every thing you do could be the last time you do it. Rather than fighting your death you should embrace it if there are no workarounds. Embrace a good death.</span><br /> -<br /> -<h2 style='display: inline' id='negative-visualization'>Negative visualization</h2><br /> -<br /> -<span>It is easy what we have to take for granted.</span><br /> -<br /> -<ul> -<li>Imagine the negative and then think that things are actually much better than they seem to be.</li> -<li>Close your eyes and imagine you are color blind for a minute, then open the eyes again and see all the colours. You will be grateful for being able to see the colours. </li> -<li>Now close your eyes for a minute and imagine you would be blind, so that you will never be able to experience the world again and let it sink in. When you open your eyes again you will feel a lot of gratefulness.</li> -<li>Last time meditation. Lets you appreciate the life as it is now. Life gets vitalised again.</li> -</ul><br /> -<h2 style='display: inline' id='oh-nice-trick-you-stoic-god--'>Oh, nice trick, you stoic "god"! ;-)</h2><br /> -<br /> -<span>Take setbacks as a challenge. Also take it with some humor.</span><br /> -<br /> -<ul> -<li>A setback in a setback, how Genius :-)</li> -<li>A setback in a setback in a setback: the stoic god's work overtime, eh? :-)</li> -</ul><br /> -<span>What would the stoic god's do next? This is just a test strategy by them. Don't be frustrated at all but be astonished of what comes next. Thank the stoic gods of testing you. This is comfort zone extension of the stoics aka toughness Training.</span><br /> -<br /> -<span>E-Mail your comments to <span class='inlinecode'>paul@nospam.buetow.org</span> :-)</span><br /> -<br /> -<span>Other book notes of mine are:</span><br /> -<br /> -<a class='textlink' href='./2025-11-02-the-courage-to-be-disliked-book-notes.html'>2025-11-02 'The Courage To Be Disliked' book notes</a><br /> -<a class='textlink' href='./2025-06-07-a-monks-guide-to-happiness-book-notes.html'>2025-06-07 'A Monk's Guide to Happiness' book notes</a><br /> -<a class='textlink' href='./2025-04-19-when-book-notes.html'>2025-04-19 'When: The Scientific Secrets of Perfect Timing' book notes</a><br /> -<a class='textlink' href='./2024-10-24-staff-engineer-book-notes.html'>2024-10-24 'Staff Engineer' book notes</a><br /> -<a class='textlink' href='./2024-07-07-the-stoic-challenge-book-notes.html'>2024-07-07 'The Stoic Challenge' book notes (You are currently reading this)</a><br /> -<a class='textlink' href='./2024-05-01-slow-productivity-book-notes.html'>2024-05-01 'Slow Productivity' book notes</a><br /> -<a class='textlink' href='./2023-11-11-mind-management-book-notes.html'>2023-11-11 'Mind Management' book notes</a><br /> -<a class='textlink' href='./2023-07-17-career-guide-and-soft-skills-book-notes.html'>2023-07-17 'Software Developers Career Guide and Soft Skills' book notes</a><br /> -<a class='textlink' href='./2023-05-06-the-obstacle-is-the-way-book-notes.html'>2023-05-06 'The Obstacle is the Way' book notes</a><br /> -<a class='textlink' href='./2023-04-01-never-split-the-difference-book-notes.html'>2023-04-01 'Never split the difference' book notes</a><br /> -<a class='textlink' href='./2023-03-16-the-pragmatic-programmer-book-notes.html'>2023-03-16 'The Pragmatic Programmer' book notes</a><br /> -<br /> -<a class='textlink' href='../'>Back to the main site</a><br /> - </div> - </content> - </entry> </feed> diff --git a/gemfeed/index.gmi b/gemfeed/index.gmi index 09aea74c..adc86453 100644 --- a/gemfeed/index.gmi +++ b/gemfeed/index.gmi @@ -2,6 +2,7 @@ ## To be in the .zone! +=> ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 - f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./2026-04-02-distributed-systems-simulator-part-3.gmi 2026-04-02 - Distributed Systems Simulator - Part 3: Advanced Examples and Protocol API => ./2026-04-01-distributed-systems-simulator-part-2.gmi 2026-04-01 - Distributed Systems Simulator - Part 2: Built-in Protocols => ./2026-03-31-distributed-systems-simulator-part-1.gmi 2026-03-31 - Distributed Systems Simulator - Part 1: Introduction and GUI @@ -30,6 +30,7 @@ Everything you read on this site is my personal opinion and experience. You can ### Posts +=> ./gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 - f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD => ./gemfeed/2026-04-02-distributed-systems-simulator-part-3.gmi 2026-04-02 - Distributed Systems Simulator - Part 3: Advanced Examples and Protocol API => ./gemfeed/2026-04-01-distributed-systems-simulator-part-2.gmi 2026-04-01 - Distributed Systems Simulator - Part 2: Built-in Protocols => ./gemfeed/2026-03-31-distributed-systems-simulator-part-1.gmi 2026-03-31 - Distributed Systems Simulator - Part 1: Introduction and GUI |