Update content for mdcontent-md

16 files changed, 526 insertions, 652 deletions

diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md
index 5375f5ac..06473931 100644
--- a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md
+++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md
@@ -398,6 +398,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-)
 Other *BSD related posts are:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md
index 08106050..27dffeed 100644
--- a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md
+++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md
@@ -677,6 +677,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-)
 Other *BSD related posts are:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md
index 7c74fb30..24790995 100644
--- a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md
+++ b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md
@@ -54,6 +54,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-)
 Other *BSD related posts are:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md
index 2bde7c7f..96cfd902 100644
--- a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md
+++ b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md
@@ -301,6 +301,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-)
 Other *BSD and KISS related posts are:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md
index 91e52953..371a78a8 100644
--- a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md
+++ b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md
@@ -16,6 +16,7 @@ These are all the posts so far:
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
 
 [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png)  
@@ -166,6 +167,7 @@ Read the next post of this series:
 Other *BSD-related posts:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md
index 5f7e646f..15438e03 100644
--- a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md
+++ b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md
@@ -16,6 +16,7 @@ These are all the posts so far:
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
 
 [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png)  
@@ -446,6 +447,7 @@ Read the next post of this series:
 Other *BSD-related posts:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md
index bb4b2aae..a93bdf89 100644
--- a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md
+++ b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md
@@ -12,6 +12,7 @@ This is the third blog post about my f3s series for my self-hosting demands in m
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
 
 [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png)  
@@ -368,6 +369,7 @@ Read the next post of this series:
 Other BSD related posts are:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md
index 8372c042..53a1f58a 100644
--- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md
+++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md
@@ -12,6 +12,7 @@ This is the fourth blog post about the f3s series for self-hosting demands in a
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
 
 [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png)  
@@ -612,6 +613,7 @@ Read the next post of this series:
 Other *BSD-related posts:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md
index 65329eb1..4d8deb2e 100644
--- a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md
+++ b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md
@@ -18,6 +18,7 @@ These are all the posts so far:
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
 
 [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png)  
@@ -1444,6 +1445,7 @@ Read the next post of this series:
 Other *BSD-related posts:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md
index 809c9780..4227f8f6 100644
--- a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md
+++ b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md
@@ -12,6 +12,7 @@ This is the sixth blog post about the f3s series for self-hosting demands in a h
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
 
 [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png)  
@@ -1934,6 +1935,7 @@ Read the next post of this series:
 Other *BSD-related posts:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.md b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.md
index ac108aff..f890ec3f 100644
--- a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.md
+++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.md
@@ -12,6 +12,7 @@ This is the seventh blog post about the f3s series for my self-hosting demands i
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
 
 [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png)  
@@ -1330,6 +1331,7 @@ I hope you enjoyed this walkthrough. Read the next post of this series:
 Other *BSD-related posts:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.md b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.md
index 9b8ed6b8..9d0b88d0 100644
--- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.md
+++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.md
@@ -12,6 +12,7 @@ This is the 8th blog post about the f3s series for my self-hosting demands in a
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability (You are currently reading this)](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
 
 [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png)  
@@ -55,29 +56,6 @@ This is the 8th blog post about the f3s series for my self-hosting demands in a
 * [⇢ ⇢ ⇢ Installing Node Exporter on OpenBSD](#installing-node-exporter-on-openbsd)
 * [⇢ ⇢ ⇢ Adding OpenBSD hosts to Prometheus](#adding-openbsd-hosts-to-prometheus)
 * [⇢ ⇢ ⇢ OpenBSD memory metrics compatibility](#openbsd-memory-metrics-compatibility)
-* [⇢ ⇢ Distributed Tracing with Grafana Tempo](#distributed-tracing-with-grafana-tempo)
-* [⇢ ⇢ ⇢ Why Distributed Tracing?](#why-distributed-tracing)
-* [⇢ ⇢ ⇢ Deploying Grafana Tempo](#deploying-grafana-tempo)
-* [⇢ ⇢ ⇢# Configuration Strategy](#-configuration-strategy)
-* [⇢ ⇢ ⇢# Tempo Deployment Files](#-tempo-deployment-files)
-* [⇢ ⇢ ⇢# Installation](#-installation)
-* [⇢ ⇢ ⇢ Configuring Grafana Alloy for Trace Collection](#configuring-grafana-alloy-for-trace-collection)
-* [⇢ ⇢ ⇢# OTLP Receiver Configuration](#-otlp-receiver-configuration)
-* [⇢ ⇢ ⇢# Upgrade Alloy](#-upgrade-alloy)
-* [⇢ ⇢ ⇢ Demo Tracing Application](#demo-tracing-application)
-* [⇢ ⇢ ⇢# Application Architecture](#-application-architecture)
-* [⇢ ⇢ ⇢ Visualizing Traces in Grafana](#visualizing-traces-in-grafana)
-* [⇢ ⇢ ⇢# Accessing Traces](#-accessing-traces)
-* [⇢ ⇢ ⇢# Service Graph Visualization](#-service-graph-visualization)
-* [⇢ ⇢ ⇢ Correlation Between Observability Signals](#correlation-between-observability-signals)
-* [⇢ ⇢ ⇢# Traces-to-Logs](#-traces-to-logs)
-* [⇢ ⇢ ⇢# Traces-to-Metrics](#-traces-to-metrics)
-* [⇢ ⇢ ⇢# Logs-to-Traces](#-logs-to-traces)
-* [⇢ ⇢ ⇢ Generating Traces for Testing](#generating-traces-for-testing)
-* [⇢ ⇢ ⇢ Verifying the Complete Pipeline](#verifying-the-complete-pipeline)
-* [⇢ ⇢ ⇢ Practical Example: Viewing a Distributed Trace](#practical-example-viewing-a-distributed-trace)
-* [⇢ ⇢ ⇢ Storage and Retention](#storage-and-retention)
-* [⇢ ⇢ ⇢ Configuration Files](#configuration-files)
 * [⇢ ⇢ Summary](#summary)
 
 ## Introduction
@@ -1013,651 +991,28 @@ This file is saved as `openbsd-recording-rules.yaml` and applied alongside the F
 
 After running `just upgrade`, the OpenBSD hosts appear in Prometheus targets and the Node Exporter dashboards.
 
-## Distributed Tracing with Grafana Tempo
-
-After implementing logs (Loki) and metrics (Prometheus), the final pillar of observability is distributed tracing. Grafana Tempo provides distributed tracing capabilities that help understand request flows across microservices.
-
-For a preview of what distributed tracing with Tempo looks like in Grafana, see the X-RAG blog post:
-
-[X-RAG Observability Hackathon](./2025-12-24-x-rag-observability-hackathon.md)  
-
-### Why Distributed Tracing?
-
-In a microservices architecture, a single user request may traverse multiple services. Distributed tracing:
-
-* Tracks requests across service boundaries
-* Identifies performance bottlenecks
-* Visualizes service dependencies
-* Correlates with logs and metrics
-* Helps debug complex distributed systems
-
-### Deploying Grafana Tempo
-
-Tempo is deployed in monolithic mode, following the same pattern as Loki's SingleBinary deployment.
-
-#### Configuration Strategy
-
-**Deployment Mode:** Monolithic (all components in one process)
-* Simpler operation than microservices mode
-* Suitable for the cluster scale
-* Consistent with Loki deployment pattern
-
-**Storage:** Filesystem backend using hostPath
-* 10Gi storage at /data/nfs/k3svolumes/tempo/data
-* 7-day retention (168h)
-* Local storage is the only option for monolithic mode
-
-**OTLP Receivers:** Standard OpenTelemetry Protocol ports
-* gRPC: 4317
-* HTTP: 4318
-* Bind to 0.0.0.0 to avoid Tempo 2.7+ localhost-only binding issue
-
-#### Tempo Deployment Files
-
-Created in /home/paul/git/conf/f3s/tempo/:
-
-**values.yaml** - Helm chart configuration:
-
-```
-tempo:
-  retention: 168h
-  storage:
-    trace:
-      backend: local
-      local:
-        path: /var/tempo/traces
-      wal:
-        path: /var/tempo/wal
-  receivers:
-    otlp:
-      protocols:
-        grpc:
-          endpoint: 0.0.0.0:4317
-        http:
-          endpoint: 0.0.0.0:4318
-
-persistence:
-  enabled: true
-  size: 10Gi
-  storageClassName: ""
-
-resources:
-  limits:
-    cpu: 1000m
-    memory: 2Gi
-  requests:
-    cpu: 500m
-    memory: 1Gi
-```
-
-**persistent-volumes.yaml** - Storage configuration:
-
-```
-apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: tempo-data-pv
-spec:
-  capacity:
-    storage: 10Gi
-  accessModes:
-    - ReadWriteOnce
-  persistentVolumeReclaimPolicy: Retain
-  hostPath:
-    path: /data/nfs/k3svolumes/tempo/data
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: tempo-data-pvc
-  namespace: monitoring
-spec:
-  storageClassName: ""
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 10Gi
-```
-
-**Grafana Datasource Provisioning**
-
-All Grafana datasources (Prometheus, Alertmanager, Loki, Tempo) are provisioned via a unified ConfigMap that is directly mounted to the Grafana pod. This approach ensures datasources are loaded on startup without requiring sidecar-based discovery.
-
-In /home/paul/git/conf/f3s/prometheus/grafana-datasources-all.yaml:
-
-```
-apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: grafana-datasources-all
-  namespace: monitoring
-data:
-  datasources.yaml: |
-    apiVersion: 1
-    datasources:
-      - name: Prometheus
-        type: prometheus
-        uid: prometheus
-        url: http://prometheus-kube-prometheus-prometheus.monitoring:9090/
-        access: proxy
-        isDefault: true
-      - name: Alertmanager
-        type: alertmanager
-        uid: alertmanager
-        url: http://prometheus-kube-prometheus-alertmanager.monitoring:9093/
-      - name: Loki
-        type: loki
-        uid: loki
-        url: http://loki.monitoring.svc.cluster.local:3100
-      - name: Tempo
-        type: tempo
-        uid: tempo
-        url: http://tempo.monitoring.svc.cluster.local:3200
-        jsonData:
-          tracesToLogsV2:
-            datasourceUid: loki
-            spanStartTimeShift: -1h
-            spanEndTimeShift: 1h
-          tracesToMetrics:
-            datasourceUid: prometheus
-          serviceMap:
-            datasourceUid: prometheus
-          nodeGraph:
-            enabled: true
-```
-
-The kube-prometheus-stack Helm values (persistence-values.yaml) are configured to:
-* Disable sidecar-based datasource provisioning
-* Mount grafana-datasources-all ConfigMap directly to /etc/grafana/provisioning/datasources/
-
-This direct mounting approach is simpler and more reliable than sidecar-based discovery.
-
-#### Installation
-
-```
-cd /home/paul/git/conf/f3s/tempo
-just install
-```
-
-Verify Tempo is running:
-
-```
-kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
-kubectl exec -n monitoring <tempo-pod> -- wget -qO- http://localhost:3200/ready
-```
-
-### Configuring Grafana Alloy for Trace Collection
-
-Updated /home/paul/git/conf/f3s/loki/alloy-values.yaml to add OTLP receivers for traces while maintaining existing log collection.
-
-#### OTLP Receiver Configuration
-
-Added to Alloy configuration after the log collection pipeline:
-
-```
-// OTLP receiver for traces via gRPC and HTTP
-otelcol.receiver.otlp "default" {
-  grpc {
-    endpoint = "0.0.0.0:4317"
-  }
-  http {
-    endpoint = "0.0.0.0:4318"
-  }
-  output {
-    traces = [otelcol.processor.batch.default.input]
-  }
-}
-
-// Batch processor for efficient trace forwarding
-otelcol.processor.batch "default" {
-  timeout = "5s"
-  send_batch_size = 100
-  send_batch_max_size = 200
-  output {
-    traces = [otelcol.exporter.otlp.tempo.input]
-  }
-}
-
-// OTLP exporter to send traces to Tempo
-otelcol.exporter.otlp "tempo" {
-  client {
-    endpoint = "tempo.monitoring.svc.cluster.local:4317"
-    tls {
-      insecure = true
-    }
-    compression = "gzip"
-  }
-}
-```
-
-The batch processor reduces network overhead by accumulating spans before forwarding to Tempo.
-
-#### Upgrade Alloy
-
-```
-cd /home/paul/git/conf/f3s/loki
-just upgrade
-```
-
-Verify OTLP receivers are listening:
-
-```
-kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i "otlp.*receiver"
-kubectl exec -n monitoring <alloy-pod> -- netstat -ln | grep -E ':(4317|4318)'
-```
-
-### Demo Tracing Application
-
-Created a three-tier Python application to demonstrate distributed tracing in action.
-
-#### Application Architecture
-
-```
-User → Frontend (Flask:5000) → Middleware (Flask:5001) → Backend (Flask:5002)
-           ↓                          ↓                        ↓
-                    Alloy (OTLP:4317) → Tempo → Grafana
-```
-
-Frontend Service:
-
-* Receives HTTP requests at /api/process
-* Forwards to middleware service
-* Creates parent span for the entire request
-
-Middleware Service:
-
-* Transforms data at /api/transform
-* Calls backend service
-* Creates child span linked to frontend
-
-Backend Service:
-
-* Returns data at /api/data
-* Simulates database query (100ms sleep)
-* Creates leaf span in the trace
-
-OpenTelemetry Instrumentation:
-
-All services use Python OpenTelemetry libraries:
-
-**Dependencies:**
-```
-flask==3.0.0
-requests==2.31.0
-opentelemetry-distro==0.49b0
-opentelemetry-exporter-otlp==1.28.0
-opentelemetry-instrumentation-flask==0.49b0
-opentelemetry-instrumentation-requests==0.49b0
-```
-
-**Auto-instrumentation pattern** (used in all services):
-
-```python
-from opentelemetry import trace
-from opentelemetry.sdk.trace import TracerProvider
-from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
-from opentelemetry.instrumentation.flask import FlaskInstrumentor
-from opentelemetry.instrumentation.requests import RequestsInstrumentor
-from opentelemetry.sdk.resources import Resource
-
-# Define service identity
-resource = Resource(attributes={
-    "service.name": "frontend",
-    "service.namespace": "tracing-demo",
-    "service.version": "1.0.0"
-})
-
-provider = TracerProvider(resource=resource)
-
-# Export to Alloy
-otlp_exporter = OTLPSpanExporter(
-    endpoint="http://alloy.monitoring.svc.cluster.local:4317",
-    insecure=True
-)
-
-processor = BatchSpanProcessor(otlp_exporter)
-provider.add_span_processor(processor)
-trace.set_tracer_provider(provider)
-
-# Auto-instrument Flask and requests
-FlaskInstrumentor().instrument_app(app)
-RequestsInstrumentor().instrument()
-```
-
-The auto-instrumentation automatically:
-* Creates spans for HTTP requests
-* Propagates trace context via W3C Trace Context headers
-* Links parent and child spans across service boundaries
-
-Deployment:
-
-Created Helm chart in /home/paul/git/conf/f3s/tracing-demo/ with three separate deployments, services, and an ingress.
-
-Build and deploy:
-
-```
-cd /home/paul/git/conf/f3s/tracing-demo
-just build
-just import
-just install
-```
-
-Verify deployment:
-
-```
-kubectl get pods -n services | grep tracing-demo
-kubectl get ingress -n services tracing-demo-ingress
-```
-
-Access the application at:
-
-[http://tracing-demo.f3s.buetow.org](http://tracing-demo.f3s.buetow.org)  
-
-### Visualizing Traces in Grafana
-
-The Tempo datasource is automatically discovered by Grafana through the ConfigMap label.
-
-#### Accessing Traces
-
-Navigate to Grafana → Explore → Select "Tempo" datasource
-
-**Search Interface:**
-* Search by Trace ID
-* Search by service name
-* Search by tags
-
-**TraceQL Queries:**
-
-Find all traces from demo app:
-```
-{ resource.service.namespace = "tracing-demo" }
-```
-
-Find slow requests (>200ms):
-```
-{ duration > 200ms }
-```
-
-Find traces from specific service:
-```
-{ resource.service.name = "frontend" }
-```
-
-Find errors:
-```
-{ status = error }
-```
-
-Complex query - frontend traces calling middleware:
-```
-{ resource.service.namespace = "tracing-demo" } && { span.http.status_code >= 500 }
-```
-
-#### Service Graph Visualization
-
-The service graph shows visual connections between services:
-
-1. Navigate to Explore → Tempo
-2. Enable "Service Graph" view
-3. Shows: Frontend → Middleware → Backend with request rates
-
-The service graph uses Prometheus metrics generated from trace data.
-
-### Correlation Between Observability Signals
-
-Tempo integrates with Loki and Prometheus to provide unified observability.
-
-#### Traces-to-Logs
-
-Click on any span in a trace to see related logs:
-
-1. View trace in Grafana
-2. Click on a span
-3. Select "Logs for this span"
-4. Loki shows logs filtered by:
-   * Time range (span duration ± 1 hour)
-   * Service name
-   * Namespace
-   * Pod
-
-This helps correlate what the service was doing when the span was created.
-
-#### Traces-to-Metrics
-
-View Prometheus metrics for services in the trace:
-
-1. View trace in Grafana
-2. Select "Metrics" tab
-3. Shows metrics like:
-   * Request rate
-   * Error rate
-   * Duration percentiles
-
-#### Logs-to-Traces
-
-From logs, you can jump to related traces:
-
-1. In Loki, logs that contain trace IDs are automatically linked
-2. Click the trace ID to view the full trace
-3. See the complete request flow
-
-### Generating Traces for Testing
-
-Test the demo application:
-
-```
-curl http://tracing-demo.f3s.buetow.org/api/process
-```
-
-Load test (generates 50 traces):
-
-```
-cd /home/paul/git/conf/f3s/tracing-demo
-just load-test
-```
-
-Each request creates a distributed trace spanning all three services.
-
-### Verifying the Complete Pipeline
-
-Check the trace flow end-to-end:
-
-**1. Application generates traces:**
-```
-kubectl logs -n services -l app=tracing-demo-frontend | grep -i trace
-```
-
-**2. Alloy receives traces:**
-```
-kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i otlp
-```
-
-**3. Tempo stores traces:**
-```
-kubectl logs -n monitoring -l app.kubernetes.io/name=tempo | grep -i trace
-```
-
-**4. Grafana displays traces:**
-Navigate to Explore → Tempo → Search for traces
-
-### Practical Example: Viewing a Distributed Trace
-
-Let's generate a trace and examine it in Grafana.
-
-**1. Generate a trace by calling the demo application:**
-
-```
-curl -H "Host: tracing-demo.f3s.buetow.org" http://r0/api/process
-```
-
-**Response (HTTP 200):**
-
-```json
-{
-  "middleware_response": {
-    "backend_data": {
-      "data": {
-        "id": 12345,
-        "query_time_ms": 100.0,
-        "timestamp": "2025-12-28T18:35:01.064538",
-        "value": "Sample data from backend service"
-      },
-      "service": "backend"
-    },
-    "middleware_processed": true,
-    "original_data": {
-      "source": "GET request"
-    },
-    "transformation_time_ms": 50
-  },
-  "request_data": {
-    "source": "GET request"
-  },
-  "service": "frontend",
-  "status": "success"
-}
-```
-
-**2. Find the trace in Tempo via API:**
-
-After a few seconds (for batch export), search for recent traces:
-
-```
-kubectl exec -n monitoring tempo-0 -- wget -qO- \
-  'http://localhost:3200/api/search?tags=service.namespace%3Dtracing-demo&limit=5' 2>/dev/null | \
-  python3 -m json.tool
-```
-
-Returns traces including:
-
-```json
-{
-  "traceID": "4be1151c0bdcd5625ac7e02b98d95bd5",
-  "rootServiceName": "frontend",
-  "rootTraceName": "GET /api/process",
-  "durationMs": 221
-}
-```
-
-**3. Fetch complete trace details:**
-
-```
-kubectl exec -n monitoring tempo-0 -- wget -qO- \
-  'http://localhost:3200/api/traces/4be1151c0bdcd5625ac7e02b98d95bd5' 2>/dev/null | \
-  python3 -m json.tool
-```
-
-**Trace structure (8 spans across 3 services):**
-
-```
-Trace ID: 4be1151c0bdcd5625ac7e02b98d95bd5
-Services: 3 (frontend, middleware, backend)
-
-Service: frontend
-  └─ GET /api/process                 221.10ms  (HTTP server span)
-  └─ frontend-process                 216.23ms  (custom business logic span)
-  └─ POST                             209.97ms  (HTTP client span to middleware)
-
-Service: middleware
-  └─ POST /api/transform              186.02ms  (HTTP server span)
-  └─ middleware-transform             180.96ms  (custom business logic span)
-  └─ GET                              127.52ms  (HTTP client span to backend)
-
-Service: backend
-  └─ GET /api/data                    103.93ms  (HTTP server span)
-  └─ backend-get-data                 102.11ms  (custom business logic span with 100ms sleep)
-```
-
-**4. View the trace in Grafana UI:**
-
-Navigate to: Grafana → Explore → Tempo datasource
-
-Search using TraceQL:
-```
-{ resource.service.namespace = "tracing-demo" }
-```
-
-Or directly open the trace by pasting the trace ID in the search box:
-```
-4be1151c0bdcd5625ac7e02b98d95bd5
-```
-
-**5. Trace visualization:**
-
-The trace waterfall view in Grafana shows the complete request flow with timing:
-
-[![Distributed trace visualization in Grafana Tempo showing Frontend → Middleware → Backend spans](./f3s-kubernetes-with-freebsd-part-8/grafana-tempo-trace.png "Distributed trace visualization in Grafana Tempo showing Frontend → Middleware → Backend spans")](./f3s-kubernetes-with-freebsd-part-8/grafana-tempo-trace.png)  
-
-For additional examples of Tempo trace visualization, see also:
-
-[X-RAG Observability Hackathon (more Grafana Tempo screenshots)](https://foo.zone/gemfeed/2025-12-24-x-rag-observability-hackathon.html)  
-
-The trace reveals the distributed request flow:
-
-* Frontend (221ms): Receives GET /api/process, executes business logic, calls middleware
-* Middleware (186ms): Receives POST /api/transform, transforms data, calls backend
-* Backend (104ms): Receives GET /api/data, simulates database query with 100ms sleep
-* Total request time: 221ms end-to-end
-* Span propagation: W3C Trace Context headers automatically link all spans
-
-**6. Service graph visualization:**
-
-The service graph is automatically generated from traces and shows service dependencies. For examples of service graph visualization in Grafana, see the screenshots in the X-RAG Observability Hackathon blog post.
-
-[X-RAG Observability Hackathon (includes service graph screenshots)](./2025-12-24-x-rag-observability-hackathon.md)  
-
-This visualization helps identify:
-
-* Request rates between services
-* Average latency for each hop
-* Error rates (if any)
-* Service dependencies and communication patterns
-
-### Storage and Retention
-
-Monitor Tempo storage usage:
-
-```
-kubectl exec -n monitoring <tempo-pod> -- df -h /var/tempo
-```
-
-With 10Gi storage and 7-day retention, the system handles moderate trace volumes. If storage fills up:
-
-* Reduce retention to 72h (3 days)
-* Implement sampling in Alloy
-* Increase PV size
-
-### Configuration Files
-
-All configuration files are available on Codeberg:
-
-[Tempo configuration](https://codeberg.org/snonux/conf/src/branch/master/f3s/tempo)  
-[Alloy configuration (updated for traces)](https://codeberg.org/snonux/conf/src/branch/master/f3s/loki)  
-[Demo tracing application](https://codeberg.org/snonux/conf/src/branch/master/f3s/tracing-demo)  
-
 ## Summary
 
-With Prometheus, Grafana, Loki, Alloy, and Tempo deployed, I now have complete visibility into the k3s cluster, the FreeBSD storage servers, and the OpenBSD edge relays:
+With Prometheus, Grafana, Loki, and Alloy deployed, I now have visibility into the k3s cluster, the FreeBSD storage servers, and the OpenBSD edge relays:
 
 * Metrics: Prometheus collects and stores time-series data from all components, including etcd and ZFS
 * Logs: Loki aggregates logs from all containers, searchable via Grafana
-* Traces: Tempo provides distributed request tracing with service dependency mapping
-* Visualisation: Grafana provides dashboards and exploration tools with correlation between all three signals
+* Visualisation: Grafana provides dashboards and exploration tools
 * Alerting: Alertmanager can notify on conditions defined in Prometheus rules
 
-This observability stack runs entirely on the home lab infrastructure, with data persisted to the NFS share. It's lightweight enough for a three-node cluster but provides the same capabilities as production-grade setups.
+The next part covers the final pillar of observability: distributed tracing with Grafana Tempo.
+
+[Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 
 All configuration files are available on Codeberg:
 
 [Prometheus, Grafana, and recording rules configuration](https://codeberg.org/snonux/conf/src/branch/master/f3s/prometheus)  
 [Loki and Alloy configuration](https://codeberg.org/snonux/conf/src/branch/master/f3s/loki)  
-[Tempo configuration](https://codeberg.org/snonux/conf/src/branch/master/f3s/tempo)  
-[Demo tracing application](https://codeberg.org/snonux/conf/src/branch/master/f3s/tracing-demo)  
 
 Other *BSD-related posts:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability (You are currently reading this)](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md b/gemfeed/2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md
new file mode 100644
index 00000000..eeb9e84e
--- /dev/null
+++ b/gemfeed/2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md
@@ -0,0 +1,497 @@
+# f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
+
+> Published at 2025-12-14T20:00:00+02:00
+
+This is a follow-up to Part 8 of the f3s series, where I covered Prometheus, Grafana, Loki, and Alloy. Now it's time for the last pillar of observability: distributed tracing with Grafana Tempo.
+
+[Part 8: Observability (Prometheus, Grafana, Loki, Alloy)](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+
+For a preview of what distributed tracing with Tempo looks like in Grafana, check out the X-RAG blog post:
+
+[X-RAG Observability Hackathon](./2025-12-24-x-rag-observability-hackathon.md)  
+
+## Table of Contents
+
+* [⇢ f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](#f3s-kubernetes-with-freebsd---part-8b-distributed-tracing-with-tempo)
+* [⇢ ⇢ Why Distributed Tracing?](#why-distributed-tracing)
+* [⇢ ⇢ Deploying Grafana Tempo](#deploying-grafana-tempo)
+* [⇢ ⇢ ⇢ Tempo Helm Values](#tempo-helm-values)
+* [⇢ ⇢ ⇢ Persistent Volumes](#persistent-volumes)
+* [⇢ ⇢ ⇢ Grafana Datasource Provisioning](#grafana-datasource-provisioning)
+* [⇢ ⇢ ⇢ Installation](#installation)
+* [⇢ ⇢ Configuring Alloy for Trace Collection](#configuring-alloy-for-trace-collection)
+* [⇢ ⇢ Demo Tracing Application](#demo-tracing-application)
+* [⇢ ⇢ ⇢ Architecture](#architecture)
+* [⇢ ⇢ ⇢ OpenTelemetry Instrumentation](#opentelemetry-instrumentation)
+* [⇢ ⇢ ⇢ Deployment](#deployment)
+* [⇢ ⇢ Visualizing Traces in Grafana](#visualizing-traces-in-grafana)
+* [⇢ ⇢ ⇢ Searching for Traces](#searching-for-traces)
+* [⇢ ⇢ ⇢ Service Graph](#service-graph)
+* [⇢ ⇢ Practical Example: End-to-End Trace](#practical-example-end-to-end-trace)
+* [⇢ ⇢ Correlation Between Signals](#correlation-between-signals)
+* [⇢ ⇢ Storage and Retention](#storage-and-retention)
+* [⇢ ⇢ Configuration Files](#configuration-files)
+
+## Why Distributed Tracing?
+
+In a microservices setup, a single user request can hop through multiple services. Tracing gives you:
+
+* Request tracking across service boundaries
+* Performance bottleneck identification
+* Service dependency visualization
+* Correlation with logs and metrics
+
+Without it, you're basically guessing where time gets spent.
+
+## Deploying Grafana Tempo
+
+Tempo runs in monolithic mode — all components in one process, same pattern as Loki's SingleBinary deployment. Keeps things simple for a home lab.
+
+The setup:
+
+* Filesystem backend using hostPath (10Gi at `/data/nfs/k3svolumes/tempo/data`)
+* 7-day retention (168h)
+* OTLP receivers on gRPC (4317) and HTTP (4318)
+* Bind to `0.0.0.0` to avoid Tempo 2.7+ localhost-only binding issue
+
+### Tempo Helm Values
+
+```
+tempo:
+  retention: 168h
+  storage:
+    trace:
+      backend: local
+      local:
+        path: /var/tempo/traces
+      wal:
+        path: /var/tempo/wal
+  receivers:
+    otlp:
+      protocols:
+        grpc:
+          endpoint: 0.0.0.0:4317
+        http:
+          endpoint: 0.0.0.0:4318
+
+persistence:
+  enabled: true
+  size: 10Gi
+  storageClassName: ""
+
+resources:
+  limits:
+    cpu: 1000m
+    memory: 2Gi
+  requests:
+    cpu: 500m
+    memory: 1Gi
+```
+
+### Persistent Volumes
+
+```
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: tempo-data-pv
+spec:
+  capacity:
+    storage: 10Gi
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  hostPath:
+    path: /data/nfs/k3svolumes/tempo/data
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: tempo-data-pvc
+  namespace: monitoring
+spec:
+  storageClassName: ""
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi
+```
+
+### Grafana Datasource Provisioning
+
+All Grafana datasources (Prometheus, Alertmanager, Loki, Tempo) are provisioned via a single ConfigMap mounted directly to the Grafana pod. No sidecar discovery needed.
+
+In `grafana-datasources-all.yaml`:
+
+```
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-datasources-all
+  namespace: monitoring
+data:
+  datasources.yaml: |
+    apiVersion: 1
+    datasources:
+      - name: Prometheus
+        type: prometheus
+        uid: prometheus
+        url: http://prometheus-kube-prometheus-prometheus.monitoring:9090/
+        access: proxy
+        isDefault: true
+      - name: Alertmanager
+        type: alertmanager
+        uid: alertmanager
+        url: http://prometheus-kube-prometheus-alertmanager.monitoring:9093/
+      - name: Loki
+        type: loki
+        uid: loki
+        url: http://loki.monitoring.svc.cluster.local:3100
+      - name: Tempo
+        type: tempo
+        uid: tempo
+        url: http://tempo.monitoring.svc.cluster.local:3200
+        jsonData:
+          tracesToLogsV2:
+            datasourceUid: loki
+            spanStartTimeShift: -1h
+            spanEndTimeShift: 1h
+          tracesToMetrics:
+            datasourceUid: prometheus
+          serviceMap:
+            datasourceUid: prometheus
+          nodeGraph:
+            enabled: true
+```
+
+The Tempo datasource config links traces to Loki logs and Prometheus metrics — so you can jump between signals directly in Grafana.
+
+The kube-prometheus-stack Helm values disable sidecar-based discovery and mount this ConfigMap directly to `/etc/grafana/provisioning/datasources/`.
+
+### Installation
+
+```
+cd /home/paul/git/conf/f3s/tempo
+just install
+```
+
+Verify it's running:
+
+```
+kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
+kubectl exec -n monitoring <tempo-pod> -- wget -qO- http://localhost:3200/ready
+```
+
+## Configuring Alloy for Trace Collection
+
+I updated the Alloy values to add OTLP receivers for traces alongside the existing log collection.
+
+Added to the Alloy config:
+
+```
+// OTLP receiver for traces via gRPC and HTTP
+otelcol.receiver.otlp "default" {
+  grpc {
+    endpoint = "0.0.0.0:4317"
+  }
+  http {
+    endpoint = "0.0.0.0:4318"
+  }
+  output {
+    traces = [otelcol.processor.batch.default.input]
+  }
+}
+
+// Batch processor — accumulates spans before forwarding to Tempo
+otelcol.processor.batch "default" {
+  timeout = "5s"
+  send_batch_size = 100
+  send_batch_max_size = 200
+  output {
+    traces = [otelcol.exporter.otlp.tempo.input]
+  }
+}
+
+// OTLP exporter to Tempo
+otelcol.exporter.otlp "tempo" {
+  client {
+    endpoint = "tempo.monitoring.svc.cluster.local:4317"
+    tls {
+      insecure = true
+    }
+    compression = "gzip"
+  }
+}
+```
+
+Upgrade Alloy:
+
+```
+cd /home/paul/git/conf/f3s/loki
+just upgrade
+```
+
+## Demo Tracing Application
+
+To actually see traces, I built a three-tier Python app. Nothing fancy — just enough to generate real distributed traces.
+
+### Architecture
+
+```
+User -> Frontend (Flask:5000) -> Middleware (Flask:5001) -> Backend (Flask:5002)
+           |                          |                        |
+                    Alloy (OTLP:4317) -> Tempo -> Grafana
+```
+
+* Frontend: receives requests at `/api/process`, forwards to middleware
+* Middleware: transforms data at `/api/transform`, calls backend
+* Backend: returns data at `/api/data`, simulates a 100ms database query
+
+### OpenTelemetry Instrumentation
+
+All three services use Python OpenTelemetry libraries:
+
+Dependencies:
+
+```
+flask==3.0.0
+requests==2.31.0
+opentelemetry-distro==0.49b0
+opentelemetry-exporter-otlp==1.28.0
+opentelemetry-instrumentation-flask==0.49b0
+opentelemetry-instrumentation-requests==0.49b0
+```
+
+Auto-instrumentation pattern (same across all services, just change the service name):
+
+```python
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+from opentelemetry.instrumentation.flask import FlaskInstrumentor
+from opentelemetry.instrumentation.requests import RequestsInstrumentor
+from opentelemetry.sdk.resources import Resource
+
+resource = Resource(attributes={
+    "service.name": "frontend",
+    "service.namespace": "tracing-demo",
+    "service.version": "1.0.0"
+})
+
+provider = TracerProvider(resource=resource)
+
+otlp_exporter = OTLPSpanExporter(
+    endpoint="http://alloy.monitoring.svc.cluster.local:4317",
+    insecure=True
+)
+
+processor = BatchSpanProcessor(otlp_exporter)
+provider.add_span_processor(processor)
+trace.set_tracer_provider(provider)
+
+FlaskInstrumentor().instrument_app(app)
+RequestsInstrumentor().instrument()
+```
+
+The auto-instrumentation creates spans for HTTP requests, propagates trace context via W3C headers, and links parent/child spans across services automatically.
+
+### Deployment
+
+The demo app has a Helm chart in the conf repo. Build, import the container images, and install:
+
+```
+cd /home/paul/git/conf/f3s/tracing-demo
+just build
+just import
+just install
+```
+
+Verify:
+
+```
+kubectl get pods -n services | grep tracing-demo
+kubectl get ingress -n services tracing-demo-ingress
+```
+
+Access at:
+
+[http://tracing-demo.f3s.foo.zone](http://tracing-demo.f3s.foo.zone)  
+
+## Visualizing Traces in Grafana
+
+### Searching for Traces
+
+In Grafana, go to Explore, select the Tempo datasource, and you can search by trace ID, service name, or tags.
+
+Some useful TraceQL queries:
+
+Find all traces from the demo app:
+```
+{ resource.service.namespace = "tracing-demo" }
+```
+
+Find slow requests (>200ms):
+```
+{ duration > 200ms }
+```
+
+Find traces from a specific service:
+```
+{ resource.service.name = "frontend" }
+```
+
+Find errors:
+```
+{ status = error }
+```
+
+Frontend traces with server errors:
+```
+{ resource.service.namespace = "tracing-demo" } && { span.http.status_code >= 500 }
+```
+
+### Service Graph
+
+The service graph view shows visual connections between services — Frontend to Middleware to Backend — with request rates and latencies. It's generated automatically from trace data using Prometheus metrics.
+
+## Practical Example: End-to-End Trace
+
+Here's what it looks like to generate and examine a trace.
+
+Generate a trace:
+
+```
+curl -H "Host: tracing-demo.f3s.foo.zone" http://r0/api/process
+```
+
+Response (HTTP 200):
+
+```json
+{
+  "middleware_response": {
+    "backend_data": {
+      "data": {
+        "id": 12345,
+        "query_time_ms": 100.0,
+        "timestamp": "2025-12-28T18:35:01.064538",
+        "value": "Sample data from backend service"
+      },
+      "service": "backend"
+    },
+    "middleware_processed": true,
+    "original_data": {
+      "source": "GET request"
+    },
+    "transformation_time_ms": 50
+  },
+  "request_data": {
+    "source": "GET request"
+  },
+  "service": "frontend",
+  "status": "success"
+}
+```
+
+After a few seconds (batch export delay), search for traces via Tempo API:
+
+```
+kubectl exec -n monitoring tempo-0 -- wget -qO- \
+  'http://localhost:3200/api/search?tags=service.namespace%3Dtracing-demo&limit=5' 2>/dev/null | \
+  python3 -m json.tool
+```
+
+Returns something like:
+
+```json
+{
+  "traceID": "4be1151c0bdcd5625ac7e02b98d95bd5",
+  "rootServiceName": "frontend",
+  "rootTraceName": "GET /api/process",
+  "durationMs": 221
+}
+```
+
+The full trace has 8 spans across 3 services:
+
+```
+Trace ID: 4be1151c0bdcd5625ac7e02b98d95bd5
+
+Service: frontend
+  GET /api/process                 221.10ms  (HTTP server span)
+  frontend-process                 216.23ms  (business logic)
+  POST                             209.97ms  (HTTP client -> middleware)
+
+Service: middleware
+  POST /api/transform              186.02ms  (HTTP server span)
+  middleware-transform             180.96ms  (business logic)
+  GET                              127.52ms  (HTTP client -> backend)
+
+Service: backend
+  GET /api/data                    103.93ms  (HTTP server span)
+  backend-get-data                 102.11ms  (business logic, 100ms sleep)
+```
+
+In Grafana, paste the trace ID in the Tempo search box or use TraceQL:
+
+```
+{ resource.service.namespace = "tracing-demo" }
+```
+
+The waterfall view shows the complete request flow with timing:
+
+[![Distributed trace in Grafana Tempo: Frontend -> Middleware -> Backend](./f3s-kubernetes-with-freebsd-part-8/grafana-tempo-trace.png "Distributed trace in Grafana Tempo: Frontend -> Middleware -> Backend")](./f3s-kubernetes-with-freebsd-part-8/grafana-tempo-trace.png)  
+
+More Tempo trace screenshots in the X-RAG blog post:
+
+[X-RAG Observability Hackathon](https://foo.zone/gemfeed/2025-12-24-x-rag-observability-hackathon.html)  
+
+## Correlation Between Signals
+
+This is where the observability stack really comes together. Tempo integrates with Loki and Prometheus so you can jump between traces, logs, and metrics.
+
+Traces to logs: click on any span and select "Logs for this span." Loki filters by time range, service name, namespace, and pod. Super useful for figuring out what a service was doing during a specific request.
+
+Traces to metrics: from a trace view, the "Metrics" tab shows Prometheus data like request rate, error rate, and duration percentiles for the services involved.
+
+Logs to traces: in Loki, logs containing trace IDs are automatically linked. Click the trace ID and you jump straight to the full trace in Tempo.
+
+## Storage and Retention
+
+With 10Gi storage and 7-day retention, the system handles moderate trace volumes. Check usage:
+
+```
+kubectl exec -n monitoring <tempo-pod> -- df -h /var/tempo
+```
+
+If storage fills up, you can reduce retention to 72h, add sampling in Alloy, or increase the PV size.
+
+## Configuration Files
+
+All config files are on Codeberg:
+
+[Tempo configuration](https://codeberg.org/snonux/conf/src/branch/master/f3s/tempo)  
+[Alloy configuration (updated for traces)](https://codeberg.org/snonux/conf/src/branch/master/f3s/loki)  
+[Demo tracing application](https://codeberg.org/snonux/conf/src/branch/master/f3s/tracing-demo)  
+
+Other *BSD-related posts:
+
+[2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo (You are currently reading this)](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
+[2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+[2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
+[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
+[2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md)  
+[2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md)  
+[2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md)  
+[2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation](./2024-12-03-f3s-kubernetes-with-freebsd-part-2.md)  
+[2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage](./2024-11-17-f3s-kubernetes-with-freebsd-part-1.md)  
+[2024-04-01 KISS high-availability with OpenBSD](./2024-04-01-KISS-high-availability-with-OpenBSD.md)  
+[2024-01-13 One reason why I love OpenBSD](./2024-01-13-one-reason-why-i-love-openbsd.md)  
+[2022-10-30 Installing DTail on OpenBSD](./2022-10-30-installing-dtail-on-openbsd.md)  
+[2022-07-30 Let's Encrypt with OpenBSD and Rex](./2022-07-30-lets-encrypt-with-openbsd-and-rex.md)  
+[2016-04-09 Jails and ZFS with Puppet on FreeBSD](./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md)  
+
+E-Mail your comments to `paul@nospam.buetow.org`
+
+[Back to the main site](../)  
diff --git a/gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.md b/gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.md
index b4185f0b..8e75220f 100644
--- a/gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.md
+++ b/gemfeed/2026-04-02-f3s-kubernetes-with-freebsd-part-9.md
@@ -12,6 +12,7 @@ This is the 9th post in the f3s series about my self-hosting home lab. f3s? The
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD (You are currently reading this)](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
 
 [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png)  
@@ -594,6 +595,7 @@ I can't imagine going back to running Helm commands manually.
 Other *BSD-related posts:
 
 [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD (You are currently reading this)](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md)  
+[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md)  
 [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md)  
diff --git a/gemfeed/index.md b/gemfeed/index.md
index 1f79d4d3..6cc1af7c 100644
--- a/gemfeed/index.md
+++ b/gemfeed/index.md
@@ -15,6 +15,7 @@
 [2026-01-01 - Posts from July to December 2025](./2026-01-01-posts-from-july-to-december-2025.md)  
 [2026-01-01 - Cloudless Kobo Forma with KOReader](./2026-01-01-cloudless-kobo-forma-with-koreader.md)  
 [2025-12-24 - X-RAG Observability Hackathon](./2025-12-24-x-rag-observability-hackathon.md)  
+[2025-12-14 - f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 - f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-11-02 - 'The Courage To Be Disliked' book notes](./2025-11-02-the-courage-to-be-disliked-book-notes.md)  
 [2025-11-02 - Perl New Features and Foostats](./2025-11-02-perl-new-features-and-foostats.md)  
diff --git a/index.md b/index.md
index 689720a8..54b1ffac 100644
--- a/index.md
+++ b/index.md
@@ -43,6 +43,7 @@ Everything you read on this site is my personal opinion and experience. You can
 [2026-01-01 - Posts from July to December 2025](./gemfeed/2026-01-01-posts-from-july-to-december-2025.md)  
 [2026-01-01 - Cloudless Kobo Forma with KOReader](./gemfeed/2026-01-01-cloudless-kobo-forma-with-koreader.md)  
 [2025-12-24 - X-RAG Observability Hackathon](./gemfeed/2025-12-24-x-rag-observability-hackathon.md)  
+[2025-12-14 - f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./gemfeed/2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md)  
 [2025-12-07 - f3s: Kubernetes with FreeBSD - Part 8: Observability](./gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.md)  
 [2025-11-02 - 'The Courage To Be Disliked' book notes](./gemfeed/2025-11-02-the-courage-to-be-disliked-book-notes.md)  
 [2025-11-02 - Perl New Features and Foostats](./gemfeed/2025-11-02-perl-new-features-and-foostats.md)