diff options
| author | Paul Buetow <paul@buetow.org> | 2026-03-30 22:56:04 +0300 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2026-03-30 22:56:04 +0300 |
| commit | 65acccd143c7648d9165715c75ffe0b6be5bfdbf (patch) | |
| tree | 6921d82aba8ae3fef37f892d707b1f6700d942a1 | |
| parent | b1dca87542f184916ecb1c9f87858db357065c7d (diff) | |
Update content for md
16 files changed, 526 insertions, 652 deletions
diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md index 5375f5ac..06473931 100644 --- a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md +++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md @@ -398,6 +398,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md index 08106050..27dffeed 100644 --- a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md +++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md @@ -677,6 +677,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md index 7c74fb30..24790995 100644 --- a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md +++ b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md @@ -54,6 +54,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md index 2bde7c7f..96cfd902 100644 --- a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md +++ b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md @@ -301,6 +301,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD and KISS related posts are: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md index 91e52953..371a78a8 100644 --- a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md +++ b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md @@ -16,6 +16,7 @@ These are all the posts so far: [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) [](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -166,6 +167,7 @@ Read the next post of this series: Other *BSD-related posts: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md index 5f7e646f..15438e03 100644 --- a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md +++ b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md @@ -16,6 +16,7 @@ These are all the posts so far: [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) [](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -446,6 +447,7 @@ Read the next post of this series: Other *BSD-related posts: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md index bb4b2aae..a93bdf89 100644 --- a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md +++ b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md @@ -12,6 +12,7 @@ This is the third blog post about my f3s series for my self-hosting demands in m [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) [](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -368,6 +369,7 @@ Read the next post of this series: Other BSD related posts are: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md index 8372c042..53a1f58a 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md @@ -12,6 +12,7 @@ This is the fourth blog post about the f3s series for self-hosting demands in a [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) [](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -612,6 +613,7 @@ Read the next post of this series: Other *BSD-related posts: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md index 65329eb1..4d8deb2e 100644 --- a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md +++ b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md @@ -18,6 +18,7 @@ These are all the posts so far: [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) [](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -1444,6 +1445,7 @@ Read the next post of this series: Other *BSD-related posts: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md index 809c9780..4227f8f6 100644 --- a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md +++ b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md @@ -12,6 +12,7 @@ This is the sixth blog post about the f3s series for self-hosting demands in a h [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) [](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -1934,6 +1935,7 @@ Read the next post of this series: Other *BSD-related posts: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.md b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.md index ac108aff..f890ec3f 100644 --- a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.md +++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.md @@ -12,6 +12,7 @@ This is the seventh blog post about the f3s series for my self-hosting demands i [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) [](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -1330,6 +1331,7 @@ I hope you enjoyed this walkthrough. Read the next post of this series: Other *BSD-related posts: [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.md b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.md index 9b8ed6b8..9d0b88d0 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.md +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.md @@ -12,6 +12,7 @@ This is the 8th blog post about the f3s series for my self-hosting demands in a [2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments](./2025-10-02-f3s-kubernetes-with-freebsd-part-7.md) [2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability (You are currently reading this)](./2025-12-07-f3s-kubernetes-with-freebsd-part-8.md) +[2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo](./2025-12-14-f3s-kubernetes-with-freebsd-part-8b.md) [2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD](./2026-04-02-f3s-kubernetes-with-freebsd-part-9.md) [](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -55,29 +56,6 @@ This is the 8th blog post about the f3s series for my self-hosting demands in a * [⇢ ⇢ ⇢ Installing Node Exporter on OpenBSD](#installing-node-exporter-on-openbsd) * [⇢ ⇢ ⇢ Adding OpenBSD hosts to Prometheus](#adding-openbsd-hosts-to-prometheus) * [⇢ ⇢ ⇢ OpenBSD memory metrics compatibility](#openbsd-memory-metrics-compatibility) -* [⇢ ⇢ Distributed Tracing with Grafana Tempo](#distributed-tracing-with-grafana-tempo) -* [⇢ ⇢ ⇢ Why Distributed Tracing?](#why-distributed-tracing) -* [⇢ ⇢ ⇢ Deploying Grafana Tempo](#deploying-grafana-tempo) -* [⇢ ⇢ ⇢# Configuration Strategy](#-configuration-strategy) -* [⇢ ⇢ ⇢# Tempo Deployment Files](#-tempo-deployment-files) -* [⇢ ⇢ ⇢# Installation](#-installation) -* [⇢ ⇢ ⇢ Configuring Grafana Alloy for Trace Collection](#configuring-grafana-alloy-for-trace-collection) -* [⇢ ⇢ ⇢# OTLP Receiver Configuration](#-otlp-receiver-configuration) -* [⇢ ⇢ ⇢# Upgrade Alloy](#-upgrade-alloy) -* [⇢ ⇢ ⇢ Demo Tracing Application](#demo-tracing-application) -* [⇢ ⇢ ⇢# Application Architecture](#-application-architecture) -* [⇢ ⇢ ⇢ Visualizing Traces in Grafana](#visualizing-traces-in-grafana) -* [⇢ ⇢ ⇢# Accessing Traces](#-accessing-traces) -* [⇢ ⇢ ⇢# Service Graph Visualization](#-service-graph-visualization) -* [⇢ ⇢ ⇢ Correlation Between Observability Signals](#correlation-between-observability-signals) -* [⇢ ⇢ ⇢# Traces-to-Logs](#-traces-to-logs) -* [⇢ ⇢ ⇢# Traces-to-Metrics](#-traces-to-metrics) -* [⇢ ⇢ ⇢# Logs-to-Traces](#-logs-to-traces) -* [⇢ ⇢ ⇢ Generating Traces for Testing](#generating-traces-for-testing) -* [⇢ ⇢ ⇢ Verifying the Complete Pipeline](#verifying-the-complete-pipeline) -* [⇢ ⇢ ⇢ Practical Example: Viewing a Distributed Trace](#practical-example-viewing-a-distributed-trace) -* [⇢ ⇢ ⇢ Storage and Retention](#storage-and-retention) -* [⇢ ⇢ ⇢ Configuration Files](#configuration-files) * [⇢ ⇢ Summary](#summary) ## Introduction @@ -1013,651 +991,28 @@ This file is saved as `openbsd-recording-rules.yaml` and applied alongside the F After running `just upgrade`, the OpenBSD hosts appear in Prometheus targets and the Node Exporter dashboards. -## Distributed Tracing with Grafana Tempo - -After implementing logs (Loki) and metrics (Prometheus), the final pillar of observability is distributed tracing. Grafana Tempo provides distributed tracing capabilities that help understand request flows across microservices. - -For a preview of what distributed tracing with Tempo looks like in Grafana, see the X-RAG blog post: - -[X-RAG Observability Hackathon](./2025-12-24-x-rag-observability-hackathon.md) - -### Why Distributed Tracing? - -In a microservices architecture, a single user request may traverse multiple services. Distributed tracing: - -* Tracks requests across service boundaries -* Identifies performance bottlenecks -* Visualizes service dependencies -* Correlates with logs and metrics -* Helps debug complex distributed systems - -### Deploying Grafana Tempo - -Tempo is deployed in monolithic mode, following the same pattern as Loki's SingleBinary deployment. - -#### Configuration Strategy - -**Deployment Mode:** Monolithic (all components in one process) -* Simpler operation than microservices mode -* Suitable for the cluster scale -* Consistent with Loki deployment pattern - -**Storage:** Filesystem backend using hostPath -* 10Gi storage at /data/nfs/k3svolumes/tempo/data -* 7-day retention (168h) -* Local storage is the only option for monolithic mode - -**OTLP Receivers:** Standard OpenTelemetry Protocol ports -* gRPC: 4317 -* HTTP: 4318 -* Bind to 0.0.0.0 to avoid Tempo 2.7+ localhost-only binding issue - -#### Tempo Deployment Files - -Created in /home/paul/git/conf/f3s/tempo/: - -**values.yaml** - Helm chart configuration: - -``` -tempo: - retention: 168h - storage: - trace: - backend: local - local: - path: /var/tempo/traces - wal: - path: /var/tempo/wal - receivers: - otlp: - protocols: - grpc: - endpoint: 0.0.0.0:4317 - http: - endpoint: 0.0.0.0:4318 - -persistence: - enabled: true - size: 10Gi - storageClassName: "" - -resources: - limits: - cpu: 1000m - memory: 2Gi - requests: - cpu: 500m - memory: 1Gi -``` - -**persistent-volumes.yaml** - Storage configuration: - -``` -apiVersion: v1 -kind: PersistentVolume -metadata: - name: tempo-data-pv -spec: - capacity: - storage: 10Gi - accessModes: - - ReadWriteOnce - persistentVolumeReclaimPolicy: Retain - hostPath: - path: /data/nfs/k3svolumes/tempo/data ---- -apiVersion: v1 -kind: PersistentVolumeClaim -metadata: - name: tempo-data-pvc - namespace: monitoring -spec: - storageClassName: "" - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 10Gi -``` - -**Grafana Datasource Provisioning** - -All Grafana datasources (Prometheus, Alertmanager, Loki, Tempo) are provisioned via a unified ConfigMap that is directly mounted to the Grafana pod. This approach ensures datasources are loaded on startup without requiring sidecar-based discovery. - -In /home/paul/git/conf/f3s/prometheus/grafana-datasources-all.yaml: - -``` -apiVersion: v1 -kind: ConfigMap -metadata: - name: grafana-datasources-all - namespace: monitoring -data: - datasources.yaml: | - apiVersion: 1 - datasources: - - name: Prometheus - type: prometheus - uid: prometheus - url: http://prometheus-kube-prometheus-prometheus.monitoring:9090/ - access: proxy - isDefault: true - - name: Alertmanager - type: alertmanager - uid: alertmanager - url: http://prometheus-kube-prometheus-alertmanager.monitoring:9093/ - - name: Loki - type: loki - uid: loki - url: http://loki.monitoring.svc.cluster.local:3100 - - name: Tempo - type: tempo - uid: tempo - url: http://tempo.monitoring.svc.cluster.local:3200 - jsonData: - tracesToLogsV2: - datasourceUid: loki - spanStartTimeShift: -1h - spanEndTimeShift: 1h - tracesToMetrics: - datasourceUid: prometheus - serviceMap: - datasourceUid: prometheus - nodeGraph: - enabled: true -``` - -The kube-prometheus-stack Helm values (persistence-values.yaml) are configured to: -* Disable sidecar-based datasource provisioning -* Mount grafana-datasources-all ConfigMap directly to /etc/grafana/provisioning/datasources/ - -This direct mounting approach is simpler and more reliable than sidecar-based discovery. - -#### Installation - -``` -cd /home/paul/git/conf/f3s/tempo -just install -``` - -Verify Tempo is running: - -``` -kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo -kubectl exec -n monitoring <tempo-pod> -- wget -qO- http://localhost:3200/ready -``` - -### Configuring Grafana Alloy for Trace Collection - -Updated /home/paul/git/conf/f3s/loki/alloy-values.yaml to add OTLP receivers for traces while maintaining existing log collection. - -#### OTLP Receiver Configuration - -Added to Alloy configuration after the log collection pipeline: - -``` -// OTLP receiver for traces via gRPC and HTTP -otelcol.receiver.otlp "default" { - grpc { - endpoint = "0.0.0.0:4317" - } - http { - endpoint = "0.0.0.0:4318" - } - output { - traces = [otelcol.processor.batch.default.input] - } -} - -// Batch processor for efficient trace forwarding -otelcol.processor.batch "default" { - timeout = "5s" - send_batch_size = 100 - send_batch_max_size = 200 - output { - traces = [otelcol.exporter.otlp.tempo.input] - } -} - -// OTLP exporter to send traces to Tempo -otelcol.exporter.otlp "tempo" { - client { - endpoint = "tempo.monitoring.svc.cluster.local:4317" - tls { - insecure = true - } - compression = "gzip" - } -} -``` - -The batch processor reduces network overhead by accumulating spans before forwarding to Tempo. - -#### Upgrade Alloy - -``` -cd /home/paul/git/conf/f3s/loki -just upgrade -``` - -Verify OTLP receivers are listening: - -``` -kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i "otlp.*receiver" -kubectl exec -n monitoring <alloy-pod> -- netstat -ln | grep -E ':(4317|4318)' -``` - -### Demo Tracing Application - -Created a three-tier Python application to demonstrate distributed tracing in action. - -#### Application Architecture - -``` -User → Frontend (Flask:5000) → Middleware (Flask:5001) → Backend (Flask:5002) - ↓ ↓ ↓ - Alloy (OTLP:4317) → Tempo → Grafana -``` - -Frontend Service: - -* Receives HTTP requests at /api/process -* Forwards to middleware service -* Creates parent span for the entire request - -Middleware Service: - -* Transforms data at /api/transform -* Calls backend service -* Creates child span linked to frontend - -Backend Service: - -* Returns data at /api/data -* Simulates database query (100ms sleep) -* Creates leaf span in the trace - -OpenTelemetry Instrumentation: - -All services use Python OpenTelemetry libraries: - -**Dependencies:** -``` -flask==3.0.0 -requests==2.31.0 -opentelemetry-distro==0.49b0 -opentelemetry-exporter-otlp==1.28.0 -opentelemetry-instrumentation-flask==0.49b0 -opentelemetry-instrumentation-requests==0.49b0 -``` - -**Auto-instrumentation pattern** (used in all services): - -```python -from opentelemetry import trace -from opentelemetry.sdk.trace import TracerProvider -from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter -from opentelemetry.instrumentation.flask import FlaskInstrumentor -from opentelemetry.instrumentation.requests import RequestsInstrumentor -from opentelemetry.sdk.resources import Resource - -# Define service identity -resource = Resource(attributes={ - "service.name": "frontend", - "service.namespace": "tracing-demo", - "service.version": "1.0.0" -}) - -provider = TracerProvider(resource=resource) - -# Export to Alloy -otlp_exporter = OTLPSpanExporter( - endpoint="http://alloy.monitoring.svc.cluster.local:4317", - insecure=True -) - -processor = BatchSpanProcessor(otlp_exporter) -provider.add_span_processor(processor) -trace.set_tracer_provider(provider) - -# Auto-instrument Flask and requests -FlaskInstrumentor().instrument_app(app) -RequestsInstrumentor().instrument() -``` - -The auto-instrumentation automatically: -* Creates spans for HTTP requests -* Propagates trace context via W3C Trace Context headers -* Links parent and child spans across service boundaries - -Deployment: - -Created Helm chart in /home/paul/git/conf/f3s/tracing-demo/ with three separate deployments, services, and an ingress. - -Build and deploy: - -``` -cd /home/paul/git/conf/f3s/tracing-demo -just build -just import -just install -``` - -Verify deployment: - -``` -kubectl get pods -n services | grep tracing-demo -kubectl get ingress -n services tracing-demo-ingress -``` - -Access the application at: - -[http://tracing-demo.f3s.buetow.org](http://tracing-demo.f3s.buetow.org) - -### Visualizing Traces in Grafana - -The Tempo datasource is automatically discovered by Grafana through the ConfigMap label. - -#### Accessing Traces - -Navigate to Grafana → Explore → Select "Tempo" datasource - -**Search Interface:** -* Search by Trace ID -* Search by service name -* Search by tags - -**TraceQL Queries:** - -Find all traces from demo app: -``` -{ resource.service.namespace = "tracing-demo" } -``` - -Find slow requests (>200ms): -``` -{ duration > 200ms } -``` - -Find traces from specific service: -``` -{ resource.service.name = "frontend" } -``` - -Find errors: -``` -{ status = error } -``` - -Complex query - frontend traces calling middleware: -``` -{ resource.service.namespace = "tracing-demo" } && { span.http.status_code >= 500 } -``` - -#### Service Graph Visualization - -The service graph shows visual connections between services: - -1. Navigate to Explore → Tempo -2. Enable "Service Graph" view -3. Shows: Frontend → Middleware → Backend with request rates - -The service graph uses Prometheus metrics generated from trace data. - -### Correlation Between Observability Signals - -Tempo integrates with Loki and Prometheus to provide unified observability. - -#### Traces-to-Logs - -Click on any span in a trace to see related logs: - -1. View trace in Grafana -2. Click on a span -3. Select "Logs for this span" -4. Loki shows logs filtered by: - * Time range (span duration ± 1 hour) - * Service name - * Namespace - * Pod - -This helps correlate what the service was doing when the span was created. - -#### Traces-to-Metrics - -View Prometheus metrics for services in the trace: - -1. View trace in Grafana -2. Select "Metrics" tab -3. Shows metrics like: - * Request rate - * Error rate - * Duration percentiles - -#### Logs-to-Traces - -From logs, you can jump to related traces: - -1. In Loki, logs that contain trace IDs are automatically linked -2. Click the trace ID to view the full trace -3. See the complete request flow - -### Generating Traces for Testing - -Test the demo application: - -``` -curl http://tracing-demo.f3s.buetow.org/api/process -``` - -Load test (generates 50 traces): - -``` -cd /home/paul/git/conf/f3s/tracing-demo -just load-test -``` - -Each request creates a distributed trace spanning all three services. - -### Verifying the Complete Pipeline - -Check the trace flow end-to-end: - -**1. Application generates traces:** -``` -kubectl logs -n services -l app=tracing-demo-frontend | grep -i trace -``` - -**2. Alloy receives traces:** -``` -kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i otlp -``` - -**3. Tempo stores traces:** -``` -kubectl logs -n monitoring -l app.kubernetes.io/name=tempo | grep -i trace -``` - -**4. Grafana displays traces:** -Navigate to Explore → Tempo → Search for traces - -### Practical Example: Viewing a Distributed Trace - -Let's generate a trace and examine it in Grafana. - -**1. Generate a trace by calling the demo application:** - -``` -curl -H "Host: tracing-demo.f3s.buetow.org" http://r0/api/process -``` - -**Response (HTTP 200):** - -```json -{ - "middleware_response": { - "backend_data": { - "data": { - "id": 12345, - "query_time_ms": 100.0, - "timestamp": "2025-12-28T18:35:01.064538", - "value": "Sample data from backend service" - }, - "service": "backend" - }, - "middleware_processed": true, - "original_data": { - "source": "GET request" - }, - "transformation_time_ms": 50 - }, - "request_data": { - "source": "GET request" - }, - "service": "frontend", - "status": "success" -} -``` |
