Update content for gemtext

19 files changed, 1543 insertions, 2207 deletions

diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi
index 257699a4..f271efda 100644
--- a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi
+++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi
@@ -398,6 +398,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-)
 Other *BSD related posts are:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi
index fafcc686..6d240965 100644
--- a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi
+++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi
@@ -677,6 +677,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-)
 Other *BSD related posts are:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi
index 9cb57b32..d692fa6e 100644
--- a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi
+++ b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi
@@ -54,6 +54,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-)
 Other *BSD related posts are:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi
index 776d8ad2..270509ee 100644
--- a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi
+++ b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi
@@ -301,6 +301,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-)
 Other *BSD and KISS related posts are:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi
index ef1cb5e7..a36f2313 100644
--- a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi
+++ b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi
@@ -16,6 +16,7 @@ These are all the posts so far:
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
 
 => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo
@@ -166,6 +167,7 @@ Read the next post of this series:
 Other *BSD-related posts:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi
index 1d15eea0..fac0fda4 100644
--- a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi
+++ b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi
@@ -16,6 +16,7 @@ These are all the posts so far:
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
 
 => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo
@@ -446,6 +447,7 @@ Read the next post of this series:
 Other *BSD-related posts:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi
index a32572e1..19349fa1 100644
--- a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi
+++ b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi
@@ -12,6 +12,7 @@ This is the third blog post about my f3s series for my self-hosting demands in m
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
 
 => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo
@@ -368,6 +369,7 @@ Read the next post of this series:
 Other BSD related posts are:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi
index ad10c129..8a6fd6cc 100644
--- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi
+++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi
@@ -12,6 +12,7 @@ This is the fourth blog post about the f3s series for self-hosting demands in a
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
 
 => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo
@@ -612,6 +613,7 @@ Read the next post of this series:
 Other *BSD-related posts:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi
index badb977f..114f16a4 100644
--- a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi
+++ b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi
@@ -18,6 +18,7 @@ These are all the posts so far:
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
 
 => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo
@@ -1444,6 +1445,7 @@ Read the next post of this series:
 Other *BSD-related posts:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi
index 90d9321c..78d31d76 100644
--- a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi
+++ b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi
@@ -12,6 +12,7 @@ This is the sixth blog post about the f3s series for self-hosting demands in a h
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
 
 => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo
@@ -1934,6 +1935,7 @@ Read the next post of this series:
 Other *BSD-related posts:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)
diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi
index ee6084c9..ca334279 100644
--- a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi
+++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi
@@ -12,6 +12,7 @@ This is the seventh blog post about the f3s series for my self-hosting demands i
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
 
 => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo
@@ -1330,6 +1331,7 @@ I hope you enjoyed this walkthrough. Read the next post of this series:
 Other *BSD-related posts:
 
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi
index faad3b1c..e3da8de6 100644
--- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi
+++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi
@@ -12,6 +12,7 @@ This is the 8th blog post about the f3s series for my self-hosting demands in a
 => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
 => ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
 => ./2025-12-07-f3s-kubernetes-with-freebsd-part-8.gmi 2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability (You are currently reading this)
+=> ./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.gmi 2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
 => ./2026-04-02-f3s-kubernetes-with-freebsd-part-9.gmi 2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
 
 => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo
@@ -55,29 +56,6 @@ This is the 8th blog post about the f3s series for my self-hosting demands in a
 * ⇢ ⇢ ⇢ Installing Node Exporter on OpenBSD
 * ⇢ ⇢ ⇢ Adding OpenBSD hosts to Prometheus
 * ⇢ ⇢ ⇢ OpenBSD memory metrics compatibility
-* ⇢ ⇢ Distributed Tracing with Grafana Tempo
-* ⇢ ⇢ ⇢ Why Distributed Tracing?
-* ⇢ ⇢ ⇢ Deploying Grafana Tempo
-* ⇢ ⇢ ⇢# Configuration Strategy
-* ⇢ ⇢ ⇢# Tempo Deployment Files
-* ⇢ ⇢ ⇢# Installation
-* ⇢ ⇢ ⇢ Configuring Grafana Alloy for Trace Collection
-* ⇢ ⇢ ⇢# OTLP Receiver Configuration
-* ⇢ ⇢ ⇢# Upgrade Alloy
-* ⇢ ⇢ ⇢ Demo Tracing Application
-* ⇢ ⇢ ⇢# Application Architecture
-* ⇢ ⇢ ⇢ Visualizing Traces in Grafana
-* ⇢ ⇢ ⇢# Accessing Traces
-* ⇢ ⇢ ⇢# Service Graph Visualization
-* ⇢ ⇢ ⇢ Correlation Between Observability Signals
-* ⇢ ⇢ ⇢# Traces-to-Logs
-* ⇢ ⇢ ⇢# Traces-to-Metrics
-* ⇢ ⇢ ⇢# Logs-to-Traces
-* ⇢ ⇢ ⇢ Generating Traces for Testing
-* ⇢ ⇢ ⇢ Verifying the Complete Pipeline
-* ⇢ ⇢ ⇢ Practical Example: Viewing a Distributed Trace
-* ⇢ ⇢ ⇢ Storage and Retention
-* ⇢ ⇢ ⇢ Configuration Files
 * ⇢ ⇢ Summary
 
 ## Introduction
@@ -1013,651 +991,28 @@ This file is saved as `openbsd-recording-rules.yaml` and applied alongside the F
 
 After running `just upgrade`, the OpenBSD hosts appear in Prometheus targets and the Node Exporter dashboards.
 
-## Distributed Tracing with Grafana Tempo
-
-After implementing logs (Loki) and metrics (Prometheus), the final pillar of observability is distributed tracing. Grafana Tempo provides distributed tracing capabilities that help understand request flows across microservices.
-
-For a preview of what distributed tracing with Tempo looks like in Grafana, see the X-RAG blog post:
-
-=> ./2025-12-24-x-rag-observability-hackathon.gmi X-RAG Observability Hackathon
-
-### Why Distributed Tracing?
-
-In a microservices architecture, a single user request may traverse multiple services. Distributed tracing:
-
-* Tracks requests across service boundaries
-* Identifies performance bottlenecks
-* Visualizes service dependencies
-* Correlates with logs and metrics
-* Helps debug complex distributed systems
-
-### Deploying Grafana Tempo
-
-Tempo is deployed in monolithic mode, following the same pattern as Loki's SingleBinary deployment.
-
-#### Configuration Strategy
-
-**Deployment Mode:** Monolithic (all components in one process)
-* Simpler operation than microservices mode
-* Suitable for the cluster scale
-* Consistent with Loki deployment pattern
-
-**Storage:** Filesystem backend using hostPath
-* 10Gi storage at /data/nfs/k3svolumes/tempo/data
-* 7-day retention (168h)
-* Local storage is the only option for monolithic mode
-
-**OTLP Receivers:** Standard OpenTelemetry Protocol ports
-* gRPC: 4317
-* HTTP: 4318
-* Bind to 0.0.0.0 to avoid Tempo 2.7+ localhost-only binding issue
-
-#### Tempo Deployment Files
-
-Created in /home/paul/git/conf/f3s/tempo/:
-
-**values.yaml** - Helm chart configuration:
-
-```
-tempo:
-  retention: 168h
-  storage:
-    trace:
-      backend: local
-      local:
-        path: /var/tempo/traces
-      wal:
-        path: /var/tempo/wal
-  receivers:
-    otlp:
-      protocols:
-        grpc:
-          endpoint: 0.0.0.0:4317
-        http:
-          endpoint: 0.0.0.0:4318
-
-persistence:
-  enabled: true
-  size: 10Gi
-  storageClassName: ""
-
-resources:
-  limits:
-    cpu: 1000m
-    memory: 2Gi
-  requests:
-    cpu: 500m
-    memory: 1Gi
-```
-
-**persistent-volumes.yaml** - Storage configuration:
-
-```
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: tempo-data-pv
-spec:
-  capacity:
-    storage: 10Gi
-  accessModes:
-    - ReadWriteOnce
-  persistentVolumeReclaimPolicy: Retain
-  hostPath:
-    path: /data/nfs/k3svolumes/tempo/data
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: tempo-data-pvc
-  namespace: monitoring
-spec:
-  storageClassName: ""
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 10Gi
-```
-
-**Grafana Datasource Provisioning**
-
-All Grafana datasources (Prometheus, Alertmanager, Loki, Tempo) are provisioned via a unified ConfigMap that is directly mounted to the Grafana pod. This approach ensures datasources are loaded on startup without requiring sidecar-based discovery.
-
-In /home/paul/git/conf/f3s/prometheus/grafana-datasources-all.yaml:
-
-```
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: grafana-datasources-all
-  namespace: monitoring
-data:
-  datasources.yaml: |
-    apiVersion: 1
-    datasources:
-      - name: Prometheus
-        type: prometheus
-        uid: prometheus
-        url: http://prometheus-kube-prometheus-prometheus.monitoring:9090/
-        access: proxy
-        isDefault: true
-      - name: Alertmanager
-        type: alertmanager
-        uid: alertmanager
-        url: http://prometheus-kube-prometheus-alertmanager.monitoring:9093/
-      - name: Loki
-        type: loki
-        uid: loki
-        url: http://loki.monitoring.svc.cluster.local:3100
-      - name: Tempo
-        type: tempo
-        uid: tempo
-        url: http://tempo.monitoring.svc.cluster.local:3200
-        jsonData:
-          tracesToLogsV2:
-            datasourceUid: loki
-            spanStartTimeShift: -1h
-            spanEndTimeShift: 1h
-          tracesToMetrics:
-            datasourceUid: prometheus
-          serviceMap:
-            datasourceUid: prometheus
-          nodeGraph:
-            enabled: true
-```
-
-The kube-prometheus-stack Helm values (persistence-values.yaml) are configured to:
-* Disable sidecar-based datasource provisioning
-* Mount grafana-datasources-all ConfigMap directly to /etc/grafana/provisioning/datasources/
-
-This direct mounting approach is simpler and more reliable than sidecar-based discovery.
-
-#### Installation
-
-```
-cd /home/paul/git/conf/f3s/tempo
-just install
-```
-
-Verify Tempo is running:
-
-```
-kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
-kubectl exec -n monitoring <tempo-pod> -- wget -qO- http://localhost:3200/ready
-```
-
-### Configuring Grafana Alloy for Trace Collection
-
-Updated /home/paul/git/conf/f3s/loki/alloy-values.yaml to add OTLP receivers for traces while maintaining existing log collection.
-
-#### OTLP Receiver Configuration
-
-Added to Alloy configuration after the log collection pipeline:
-
-```
-// OTLP receiver for traces via gRPC and HTTP
-otelcol.receiver.otlp "default" {
-  grpc {
-    endpoint = "0.0.0.0:4317"
-  }
-  http {
-    endpoint = "0.0.0.0:4318"
-  }
-  output {
-    traces = [otelcol.processor.batch.default.input]
-  }
-}
-
-// Batch processor for efficient trace forwarding
-otelcol.processor.batch "default" {
-  timeout = "5s"
-  send_batch_size = 100
-  send_batch_max_size = 200
-  output {
-    traces = [otelcol.exporter.otlp.tempo.input]
-  }
-}
-
-// OTLP exporter to send traces to Tempo
-otelcol.exporter.otlp "tempo" {
-  client {
-    endpoint = "tempo.monitoring.svc.cluster.local:4317"
-    tls {
-      insecure = true
-    }
-    compression = "gzip"
-  }
-}
-```
-
-The batch processor reduces network overhead by accumulating spans before forwarding to Tempo.
-
-#### Upgrade Alloy
-
-```
-cd /home/paul/git/conf/f3s/loki
-just upgrade
-```
-
-Verify OTLP receivers are listening:
-
-```
-kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i "otlp.*receiver"
-kubectl exec -n monitoring <alloy-pod> -- netstat -ln | grep -E ':(4317|4318)'
-```
-
-### Demo Tracing Application
-
-Created a three-tier Python application to demonstrate distributed tracing in action.
-
-#### Application Architecture
-
-```
-User → Frontend (Flask:5000) → Middleware (Flask:5001) → Backend (Flask:5002)
-           ↓                          ↓                        ↓
-                    Alloy (OTLP:4317) → Tempo → Grafana
-```
-
-Frontend Service:
-
-* Receives HTTP requests at /api/process
-* Forwards to middleware service
-* Creates parent span for the entire request
-
-Middleware Service:
-
-* Transforms data at /api/transform
-* Calls backend service
-* Creates child span linked to frontend
-
-Backend Service:
-
-* Returns data at /api/data
-* Simulates database query (100ms sleep)
-* Creates leaf span in the trace
-
-OpenTelemetry Instrumentation:
-
-All services use Python OpenTelemetry libraries:
-
-**Dependencies:**
-```
-flask==3.0.0
-requests==2.31.0
-opentelemetry-distro==0.49b0
-opentelemetry-exporter-otlp==1.28.0
-opentelemetry-instrumentation-flask==0.49b0
-opentelemetry-instrumentation-requests==0.49b0
-```
-
-**Auto-instrumentation pattern** (used in all services):
-
-```python
-from opentelemetry import trace
-from opentelemetry.sdk.trace import TracerProvider
-from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
-from opentelemetry.instrumentation.flask import FlaskInstrumentor
-from opentelemetry.instrumentation.requests import RequestsInstrumentor
-from opentelemetry.sdk.resources import Resource
-
-# Define service identity
-resource = Resource(attributes={
-    "service.name": "frontend",
-    "service.namespace": "tracing-demo",
-    "service.version": "1.0.0"
-})
-
-provider = TracerProvider(resource=resource)
-
-# Export to Alloy
-otlp_exporter = OTLPSpanExporter(
-    endpoint="http://alloy.monitoring.svc.cluster.local:4317",
-    insecure=True
-)
-
-processor = BatchSpanProcessor(otlp_exporter)
-provider.add_span_processor(processor)
-trace.set_tracer_provider(provider)
-
-# Auto-instrument Flask and requests
-FlaskInstrumentor().instrument_app(app)
-RequestsInstrumentor().instrument()
-```
-
-The auto-instrumentation automatically:
-* Creates spans for HTTP requests
-* Propagates trace context via W3C Trace Context headers
-* Links parent and child spans across service boundaries
-
-Deployment:
-
-Created Helm chart in /home/paul/git/conf/f3s/tracing-demo/ with three separate deployments, services, and an ingress.
-
-Build and deploy:
-
-```
-cd /home/paul/git/conf/f3s/tracing-demo
-just build
-just import
-just install
-```
-
-Verify deployment:
-
-```
-kubectl get pods -n services | grep tracing-demo
-kubectl get ingress -n services tracing-demo-ingress
-```
-
-Access the application at:
-
-=> http://tracing-demo.f3s.buetow.org
-
-### Visualizing Traces in Grafana
-
-The Tempo datasource is automatically discovered by Grafana through the ConfigMap label.
-
-#### Accessing Traces
-
-Navigate to Grafana → Explore → Select "Tempo" datasource
-
-**Search Interface:**
-* Search by Trace ID
-* Search by service name
-* Search by tags
-
-**TraceQL Queries:**
-
-Find all traces from demo app:
-```
-{ resource.service.namespace = "tracing-demo" }
-```
-
-Find slow requests (>200ms):
-```
-{ duration > 200ms }
-```
-
-Find traces from specific service:
-```
-{ resource.service.name = "frontend" }
-```
-
-Find errors:
-```
-{ status = error }
-```
-
-Complex query - frontend traces calling middleware:
-```
-{ resource.service.namespace = "tracing-demo" } && { span.http.status_code >= 500 }
-```
-
-#### Service Graph Visualization
-
-The service graph shows visual connections between services:
-
-1. Navigate to Explore → Tempo
-2. Enable "Service Graph" view
-3. Shows: Frontend → Middleware → Backend with request rates
-
-The service graph uses Prometheus metrics generated from trace data.
-
-### Correlation Between Observability Signals
-
-Tempo integrates with Loki and Prometheus to provide unified observability.
-
-#### Traces-to-Logs
-
-Click on any span in a trace to see related logs:
-
-1. View trace in Grafana
-2. Click on a span
-3. Select "Logs for this span"
-4. Loki shows logs filtered by:
-   * Time range (span duration ± 1 hour)
-   * Service name
-   * Namespace
-   * Pod
-
-This helps correlate what the service was doing when the span was created.
-
-#### Traces-to-Metrics
-
-View Prometheus metrics for services in the trace:
-
-1. View trace in Grafana
-2. Select "Metrics" tab
-3. Shows metrics like:
-   * Request rate
-   * Error rate
-   * Duration percentiles
-
-#### Logs-to-Traces
-
-From logs, you can jump to related traces:
-
-1. In Loki, logs that contain trace IDs are automatically linked
-2. Click the trace ID to view the full trace
-3. See the complete request flow
-
-### Generating Traces for Testing
-
-Test the demo application:
-
-```
-curl http://tracing-demo.f3s.buetow.org/api/process
-```
-
-Load test (generates 50 traces):
-
-```
-cd /home/paul/git/conf/f3s/tracing-demo
-just load-test
-```
-
-Each request creates a distributed trace spanning all three services.
-
-### Verifying the Complete Pipeline
-
-Check the trace flow end-to-end:
-
-**1. Application generates traces:**
-```
-kubectl logs -n services -l app=tracing-demo-frontend | grep -i trace
-```
-
-**2. Alloy receives traces:**
-```
-kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i otlp
-```
-
-**3. Tempo stores traces:**
-```
-kubectl logs -n monitoring -l app.kubernetes.io/name=tempo | grep -i trace
-```
-
-**4. Grafana displays traces:**
-Navigate to Explore → Tempo → Search for traces
-
-### Practical Example: Viewing a Distributed Trace
-
-Let's generate a trace and examine it in Grafana.
-
-**1. Generate a trace by calling the demo application:**
-
-```
-curl -H "Host: tracing-demo.f3s.buetow.org" http://r0/api/process
-```
-
-**Response (HTTP 200):**
-
-```json
-{
-  "middleware_response": {
-    "backend_data": {
-      "data": {
-        "id": 12345,
-        "query_time_ms": 100.0,
-        "timestamp": "2025-12-28T18:35:01.064538",
-        "value": "Sample data from backend service"
-      },</