From 4c01fc9e0966bb8a462ff5054229a4f1fb3330ce Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Mon, 30 Mar 2026 22:56:03 +0300 Subject: Update content for html --- ...04-09-jails-and-zfs-on-freebsd-with-puppet.html | 1 + ...22-07-30-lets-encrypt-with-openbsd-and-rex.html | 1 + .../2024-01-13-one-reason-why-i-love-openbsd.html | 1 + ...-04-01-KISS-high-availability-with-OpenBSD.html | 1 + ...4-11-17-f3s-kubernetes-with-freebsd-part-1.html | 2 + ...4-12-03-f3s-kubernetes-with-freebsd-part-2.html | 2 + ...5-02-01-f3s-kubernetes-with-freebsd-part-3.html | 2 + ...5-04-05-f3s-kubernetes-with-freebsd-part-4.html | 2 + ...5-05-11-f3s-kubernetes-with-freebsd-part-5.html | 2 + ...5-07-14-f3s-kubernetes-with-freebsd-part-6.html | 2 + ...5-10-02-f3s-kubernetes-with-freebsd-part-7.html | 2 + ...5-12-07-f3s-kubernetes-with-freebsd-part-8.html | 681 +---- ...-12-14-f3s-kubernetes-with-freebsd-part-8b.html | 532 ++++ ...6-04-02-f3s-kubernetes-with-freebsd-part-9.html | 2 + gemfeed/atom.xml | 2909 +++++++++----------- gemfeed/index.html | 1 + index.html | 1 + 17 files changed, 1827 insertions(+), 2317 deletions(-) create mode 100644 gemfeed/2025-12-14-f3s-kubernetes-with-freebsd-part-8b.html diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html index 05f153a3..6ec2d003 100644 --- a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html +++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.html @@ -414,6 +414,7 @@ Notice: Finished catalog run in 206.09 seconds Other *BSD related posts are:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.html b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.html index 71ccc187..cd045e63 100644 --- a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.html +++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.html @@ -693,6 +693,7 @@ rex commons Other *BSD related posts are:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.html b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.html index f67c7110..a2b7eb99 100644 --- a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.html +++ b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.html @@ -73,6 +73,7 @@ $ doas reboot # Just in case, reboot one more timeOther *BSD related posts are:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html index 3839324e..72dcfec3 100644 --- a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html +++ b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.html @@ -332,6 +332,7 @@ http://www.gnu.org/software/src-highlite --> Other *BSD and KISS related posts are:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.html b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.html index a95ea52c..617ef9a6 100644 --- a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.html +++ b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.html @@ -29,6 +29,7 @@ 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD

f3s logo
@@ -182,6 +183,7 @@ Other *BSD-related posts:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.html b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.html index e478754e..150f9db9 100644 --- a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.html +++ b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.html @@ -29,6 +29,7 @@ 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD

f3s logo
@@ -518,6 +519,7 @@ Waking up e8:ff:1e:d7:1c:a0... Other *BSD-related posts:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.html b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.html index 7e9539f0..620eb55c 100644 --- a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.html +++ b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.html @@ -25,6 +25,7 @@ 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD

f3s logo
@@ -422,6 +423,7 @@ Jan 26 17:36:32 f2 apcupsd[2159]: apcupsd shutdown succeeded Other BSD related posts are:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html index 77bd8bcd..1729d91c 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.html @@ -25,6 +25,7 @@ 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD

f3s logo
@@ -717,6 +718,7 @@ etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 408 Other *BSD-related posts:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.html b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.html index 6b4beb48..20b75e46 100644 --- a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.html +++ b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.html @@ -31,6 +31,7 @@ 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD

f3s logo
@@ -1578,6 +1579,7 @@ earth$ curl https://ifconfig.me # Should show gateway's Other *BSD-related posts:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.html b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.html index 1e60ff46..99e9577e 100644 --- a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.html +++ b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.html @@ -25,6 +25,7 @@ 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD

f3s logo
@@ -2181,6 +2182,7 @@ http://www.gnu.org/software/src-highlite --> Other *BSD-related posts:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)
diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html index 7f13b083..3a8a18cf 100644 --- a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html +++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.html @@ -25,6 +25,7 @@ 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD

f3s logo
@@ -1494,6 +1495,7 @@ replicaset.apps/miniflux-server-85d7c64664 1 1 1 54d Other *BSD-related posts:

2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this)
2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
diff --git a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html index 3954f3dd..c864bb79 100644 --- a/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html +++ b/gemfeed/2025-12-07-f3s-kubernetes-with-freebsd-part-8.html @@ -25,6 +25,7 @@ 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability (You are currently reading this)
+2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD

f3s logo
@@ -69,29 +70,6 @@
  • ⇢ ⇢ Installing Node Exporter on OpenBSD
  • ⇢ ⇢ Adding OpenBSD hosts to Prometheus
  • ⇢ ⇢ OpenBSD memory metrics compatibility
    • -
  • ⇢ Distributed Tracing with Grafana Tempo
    • -
  • ⇢ ⇢ Why Distributed Tracing?
    • -
  • ⇢ ⇢ Deploying Grafana Tempo
    • -
  • ⇢ ⇢# Configuration Strategy
    • -
  • ⇢ ⇢# Tempo Deployment Files
    • -
  • ⇢ ⇢# Installation
    • -
  • ⇢ ⇢ Configuring Grafana Alloy for Trace Collection
    • -
  • ⇢ ⇢# OTLP Receiver Configuration
    • -
  • ⇢ ⇢# Upgrade Alloy
    • -
  • ⇢ ⇢ Demo Tracing Application
    • -
  • ⇢ ⇢# Application Architecture
    • -
  • ⇢ ⇢ Visualizing Traces in Grafana
    • -
  • ⇢ ⇢# Accessing Traces
    • -
  • ⇢ ⇢# Service Graph Visualization
    • -
  • ⇢ ⇢ Correlation Between Observability Signals
    • -
  • ⇢ ⇢# Traces-to-Logs
    • -
  • ⇢ ⇢# Traces-to-Metrics
    • -
  • ⇢ ⇢# Logs-to-Traces
    • -
  • ⇢ ⇢ Generating Traces for Testing
    • -
  • ⇢ ⇢ Verifying the Complete Pipeline
    • -
  • ⇢ ⇢ Practical Example: Viewing a Distributed Trace
    • -
  • ⇢ ⇢ Storage and Retention
    • -
  • ⇢ ⇢ Configuration Files
  • ⇢ Summary

    • Introduction


    @@ -1129,674 +1107,29 @@ spec:
    After running just upgrade, the OpenBSD hosts appear in Prometheus targets and the Node Exporter dashboards.

    -

    Distributed Tracing with Grafana Tempo


    -
    -After implementing logs (Loki) and metrics (Prometheus), the final pillar of observability is distributed tracing. Grafana Tempo provides distributed tracing capabilities that help understand request flows across microservices.
    -
    -For a preview of what distributed tracing with Tempo looks like in Grafana, see the X-RAG blog post:
    -
    -X-RAG Observability Hackathon
    -
    -

    Why Distributed Tracing?


    -
    -In a microservices architecture, a single user request may traverse multiple services. Distributed tracing:
    -
    -
      -
    • Tracks requests across service boundaries
      • -
    • Identifies performance bottlenecks
      • -
    • Visualizes service dependencies
      • -
    • Correlates with logs and metrics
      • -
    • Helps debug complex distributed systems
      • -

    -

    Deploying Grafana Tempo


    -
    -Tempo is deployed in monolithic mode, following the same pattern as Loki's SingleBinary deployment.
    -
    -#### Configuration Strategy
    -
    -**Deployment Mode:** Monolithic (all components in one process)
    -
      -
    • Simpler operation than microservices mode
      • -
    • Suitable for the cluster scale
      • -
    • Consistent with Loki deployment pattern
      • -

    -**Storage:** Filesystem backend using hostPath
    -
      -
    • 10Gi storage at /data/nfs/k3svolumes/tempo/data
      • -
    • 7-day retention (168h)
      • -
    • Local storage is the only option for monolithic mode
      • -

    -**OTLP Receivers:** Standard OpenTelemetry Protocol ports
    -
      -
    • gRPC: 4317
      • -
    • HTTP: 4318
      • -
    • Bind to 0.0.0.0 to avoid Tempo 2.7+ localhost-only binding issue
      • -

    -#### Tempo Deployment Files
    -
    -Created in /home/paul/git/conf/f3s/tempo/:
    -
    -**values.yaml** - Helm chart configuration:
    -
    -
    -tempo:
-  retention: 168h
-  storage:
-    trace:
-      backend: local
-      local:
-        path: /var/tempo/traces
-      wal:
-        path: /var/tempo/wal
-  receivers:
-    otlp:
-      protocols:
-        grpc:
-          endpoint: 0.0.0.0:4317
-        http:
-          endpoint: 0.0.0.0:4318
-
-persistence:
-  enabled: true
-  size: 10Gi
-  storageClassName: ""
-
-resources:
-  limits:
-    cpu: 1000m
-    memory: 2Gi
-  requests:
-    cpu: 500m
-    memory: 1Gi
-
    -
    -**persistent-volumes.yaml** - Storage configuration:
    -
    -
    -apiVersion: v1
-kind: PersistentVolume
-metadata:
-  name: tempo-data-pv
-spec:
-  capacity:
-    storage: 10Gi
-  accessModes:
-    - ReadWriteOnce
-  persistentVolumeReclaimPolicy: Retain
-  hostPath:
-    path: /data/nfs/k3svolumes/tempo/data
----
-apiVersion: v1
-kind: PersistentVolumeClaim
-metadata:
-  name: tempo-data-pvc
-  namespace: monitoring
-spec:
-  storageClassName: ""
-  accessModes:
-    - ReadWriteOnce
-  resources:
-    requests:
-      storage: 10Gi
-
    -
    -**Grafana Datasource Provisioning**
    -
    -All Grafana datasources (Prometheus, Alertmanager, Loki, Tempo) are provisioned via a unified ConfigMap that is directly mounted to the Grafana pod. This approach ensures datasources are loaded on startup without requiring sidecar-based discovery.
    -
    -In /home/paul/git/conf/f3s/prometheus/grafana-datasources-all.yaml:
    -
    -
    -apiVersion: v1
-kind: ConfigMap
-metadata:
-  name: grafana-datasources-all
-  namespace: monitoring
-data:
-  datasources.yaml: |
-    apiVersion: 1
-    datasources:
-      - name: Prometheus
-        type: prometheus
-        uid: prometheus
-        url: http://prometheus-kube-prometheus-prometheus.monitoring:9090/
-        access: proxy
-        isDefault: true
-      - name: Alertmanager
-        type: alertmanager
-        uid: alertmanager
-        url: http://prometheus-kube-prometheus-alertmanager.monitoring:9093/
-      - name: Loki
-        type: loki
-        uid: loki
-        url: http://loki.monitoring.svc.cluster.local:3100
-      - name: Tempo
-        type: tempo
-        uid: tempo
-        url: http://tempo.monitoring.svc.cluster.local:3200
-        jsonData:
-          tracesToLogsV2:
-            datasourceUid: loki
-            spanStartTimeShift: -1h
-            spanEndTimeShift: 1h
-          tracesToMetrics:
-            datasourceUid: prometheus
-          serviceMap:
-            datasourceUid: prometheus
-          nodeGraph:
-            enabled: true
-
    -
    -The kube-prometheus-stack Helm values (persistence-values.yaml) are configured to:
    -
      -
    • Disable sidecar-based datasource provisioning
      • -
    • Mount grafana-datasources-all ConfigMap directly to /etc/grafana/provisioning/datasources/
      • -

    -This direct mounting approach is simpler and more reliable than sidecar-based discovery.
    -
    -#### Installation
    -
    -
    -cd /home/paul/git/conf/f3s/tempo
-just install
-
    -
    -Verify Tempo is running:
    -
    -
    -kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
-kubectl exec -n monitoring <tempo-pod> -- wget -qO- http://localhost:3200/ready
-
    -
    -

    Configuring Grafana Alloy for Trace Collection


    -
    -Updated /home/paul/git/conf/f3s/loki/alloy-values.yaml to add OTLP receivers for traces while maintaining existing log collection.
    -
    -#### OTLP Receiver Configuration
    -
    -Added to Alloy configuration after the log collection pipeline:
    -
    -
    -// OTLP receiver for traces via gRPC and HTTP
-otelcol.receiver.otlp "default" {
-  grpc {
-    endpoint = "0.0.0.0:4317"
-  }
-  http {
-    endpoint = "0.0.0.0:4318"
-  }
-  output {
-    traces = [otelcol.processor.batch.default.input]
-  }
-}
-
-// Batch processor for efficient trace forwarding
-otelcol.processor.batch "default" {
-  timeout = "5s"
-  send_batch_size = 100
-  send_batch_max_size = 200
-  output {
-    traces = [otelcol.exporter.otlp.tempo.input]
-  }
-}
-
-// OTLP exporter to send traces to Tempo
-otelcol.exporter.otlp "tempo" {
-  client {
-    endpoint = "tempo.monitoring.svc.cluster.local:4317"
-    tls {
-      insecure = true
-    }
-    compression = "gzip"
-  }
-}
-
    -
    -The batch processor reduces network overhead by accumulating spans before forwarding to Tempo.
    -
    -#### Upgrade Alloy
    -
    -
    -cd /home/paul/git/conf/f3s/loki
-just upgrade
-
    -
    -Verify OTLP receivers are listening:
    -
    -
    -kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i "otlp.*receiver"
-kubectl exec -n monitoring <alloy-pod> -- netstat -ln | grep -E ':(4317|4318)'
-
    -
    -

    Demo Tracing Application


    -
    -Created a three-tier Python application to demonstrate distributed tracing in action.
    -
    -#### Application Architecture
    -
    -
    -User → Frontend (Flask:5000) → Middleware (Flask:5001) → Backend (Flask:5002)
-           ↓                          ↓                        ↓
-                    Alloy (OTLP:4317) → Tempo → Grafana
-
    -
    -Frontend Service:
    -
    -
      -
    • Receives HTTP requests at /api/process
      • -
    • Forwards to middleware service
      • -
    • Creates parent span for the entire request
      • -

    -Middleware Service:
    -
    -
      -
    • Transforms data at /api/transform
      • -
    • Calls backend service
      • -
    • Creates child span linked to frontend
      • -

    -Backend Service:
    -
    -
      -
    • Returns data at /api/data
      • -
    • Simulates database query (100ms sleep)
      • -
    • Creates leaf span in the trace
      • -

    -OpenTelemetry Instrumentation:
    -
    -All services use Python OpenTelemetry libraries:
    -
    -**Dependencies:**
    -
    -flask==3.0.0
-requests==2.31.0
-opentelemetry-distro==0.49b0
-opentelemetry-exporter-otlp==1.28.0
-opentelemetry-instrumentation-flask==0.49b0
-opentelemetry-instrumentation-requests==0.49b0
-
    -
    -**Auto-instrumentation pattern** (used in all services):
    -
    - -
    from opentelemetry import trace
-from opentelemetry.sdk.trace import TracerProvider
-from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
-from opentelemetry.instrumentation.flask import FlaskInstrumentor
-from opentelemetry.instrumentation.requests import RequestsInstrumentor
-from opentelemetry.sdk.resources import Resource
-
-# Define service identity
-resource = Resource(attributes={
-    "service.name": "frontend",
-    "service.namespace": "tracing-demo",
-    "service.version": "1.0.0"
-})
-
-provider = TracerProvider(resource=resource)
-
-# Export to Alloy
-otlp_exporter = OTLPSpanExporter(
-    endpoint="http://alloy.monitoring.svc.cluster.local:4317",
-    insecure=True
-)
-
-processor = BatchSpanProcessor(otlp_exporter)
-provider.add_span_processor(processor)
-trace.set_tracer_provider(provider)
-
-# Auto-instrument Flask and requests
-FlaskInstrumentor().instrument_app(app)
-RequestsInstrumentor().instrument()
-
    -
    -The auto-instrumentation automatically:
    -
      -
    • Creates spans for HTTP requests
      • -
    • Propagates trace context via W3C Trace Context headers
      • -
    • Links parent and child spans across service boundaries
      • -

    -Deployment:
    -
    -Created Helm chart in /home/paul/git/conf/f3s/tracing-demo/ with three separate deployments, services, and an ingress.
    -
    -Build and deploy:
    -
    -
    -cd /home/paul/git/conf/f3s/tracing-demo
-just build
-just import
-just install
-
    -
    -Verify deployment:
    -
    -
    -kubectl get pods -n services | grep tracing-demo
-kubectl get ingress -n services tracing-demo-ingress
-
    -
    -Access the application at:
    -
    -http://tracing-demo.f3s.buetow.org
    -
    -

    Visualizing Traces in Grafana


    -
    -The Tempo datasource is automatically discovered by Grafana through the ConfigMap label.
    -
    -#### Accessing Traces
    -
    -Navigate to Grafana → Explore → Select "Tempo" datasource
    -
    -**Search Interface:**
    -
      -
    • Search by Trace ID
      • -
    • Search by service name
      • -
    • Search by tags
      • -

    -**TraceQL Queries:**
    -
    -Find all traces from demo app:
    -
    -{ resource.service.namespace = "tracing-demo" }
-
    -
    -Find slow requests (>200ms):
    -
    -{ duration > 200ms }
-
    -
    -Find traces from specific service:
    -
    -{ resource.service.name = "frontend" }
-
    -
    -Find errors:
    -
    -{ status = error }
-
    -
    -Complex query - frontend traces calling middleware:
    -
    -{ resource.service.namespace = "tracing-demo" } && { span.http.status_code >= 500 }
-
    -
    -#### Service Graph Visualization
    -
    -The service graph shows visual connections between services:
    -
    -1. Navigate to Explore → Tempo
    -2. Enable "Service Graph" view
    -3. Shows: Frontend → Middleware → Backend with request rates
    -
    -The service graph uses Prometheus metrics generated from trace data.
    -
    -

    Correlation Between Observability Signals


    -
    -Tempo integrates with Loki and Prometheus to provide unified observability.
    -
    -#### Traces-to-Logs
    -
    -Click on any span in a trace to see related logs:
    -
    -1. View trace in Grafana
    -2. Click on a span
    -3. Select "Logs for this span"
    -4. Loki shows logs filtered by:
    - * Time range (span duration ± 1 hour)
    - * Service name
    - * Namespace
    - * Pod
    -
    -This helps correlate what the service was doing when the span was created.
    -
    -#### Traces-to-Metrics
    -
    -View Prometheus metrics for services in the trace:
    -
    -1. View trace in Grafana
    -2. Select "Metrics" tab
    -3. Shows metrics like:
    - * Request rate
    - * Error rate
    - * Duration percentiles
    -
    -#### Logs-to-Traces
    -
    -From logs, you can jump to related traces:
    -
    -1. In Loki, logs that contain trace IDs are automatically linked
    -2. Click the trace ID to view the full trace
    -3. See the complete request flow
    -
    -

    Generating Traces for Testing


    -
    -Test the demo application:
    -
    -
    -curl http://tracing-demo.f3s.buetow.org/api/process
-
    -
    -Load test (generates 50 traces):
    -
    -
    -cd /home/paul/git/conf/f3s/tracing-demo
-just load-test
-
    -
    -Each request creates a distributed trace spanning all three services.
    -
    -

    Verifying the Complete Pipeline


    -
    -Check the trace flow end-to-end:
    -
    -**1. Application generates traces:**
    -
    -kubectl logs -n services -l app=tracing-demo-frontend | grep -i trace
-
    -
    -**2. Alloy receives traces:**
    -
    -kubectl logs -n monitoring -l app.kubernetes.io/name=alloy | grep -i otlp
-
    -
    -**3. Tempo stores traces:**
    -
    -kubectl logs -n monitoring -l app.kubernetes.io/name=tempo | grep -i trace
-
    -
    -**4. Grafana displays traces:**
    -Navigate to Explore → Tempo → Search for traces
    -
    -

    Practical Example: Viewing a Distributed Trace


    -
    -Let's generate a trace and examine it in Grafana.
    -
    -**1. Generate a trace by calling the demo application:**
    -
    -
    -curl -H "Host: tracing-demo.f3s.buetow.org" http://r0/api/process
-
    -
    -**Response (HTTP 200):**
    -
    - -
    {
-  "middleware_response": {
-    "backend_data": {
-      "data": {
-        "id": 12345,
-        "query_time_ms": 100.0,
-        "timestamp": "2025-12-28T18:35:01.064538",
-        "value": "Sample data from backend service"
-      },
-      "service": "backend"
-    },
-    "middleware_processed": true,
-    "original_data": {
-      "source": "GET request"
-    },
-    "transformation_time_ms": 50
-  },
-  "request_data": {
-    "source": "GET request"
-  },
-  "service": "frontend",
-  "status": "success"
-}
-
    -
    -**2. Find the trace in Tempo via API:**
    -
    -After a few seconds (for batch export), search for recent traces:
    -
    -
    -kubectl exec -n monitoring tempo-0 -- wget -qO- \
-  'http://localhost:3200/api/search?tags=service.namespace%3Dtracing-demo&limit=5' 2>/dev/null | \
-  python3 -m json.tool
-
    -
    -Returns traces including:
    -
    - -
    {
-  "traceID": "4be1151c0bdcd5625ac7e02b98d95bd5",
-  "rootServiceName": "frontend",
-  "rootTraceName": "GET /api/process",
-  "durationMs": 221
-}
-
    -
    -**3. Fetch complete trace details:**
    -
    -
    -kubectl exec -n monitoring tempo-0 -- wget -qO- \
-  'http://localhost:3200/api/traces/4be1151c0bdcd5625ac7e02b98d95bd5' 2>/dev/null | \
-  python3 -m json.tool
-
    -
    -**Trace structure (8 spans across 3 services):**
    -
    -
    -Trace ID: 4be1151c0bdcd5625ac7e02b98d95bd5
-Services: 3 (frontend, middleware, backend)
-
-Service: frontend
-  └─ GET /api/process                 221.10ms  (HTTP server span)
-  └─ frontend-process                 216.23ms  (custom business logic span)
-  └─ POST                             209.97ms  (HTTP client span to middleware)
-
-Service: middleware
-  └─ POST /api/transform              186.02ms  (HTTP server span)
-  └─ middleware-transform             180.96ms  (custom business logic span)
-  └─ GET                              127.52ms  (HTTP client span to backend)
-
-Service: backend
-  └─ GET /api/data                    103.93ms  (HTTP server span)
-  └─ backend-get-data                 102.11ms  (custom business logic span with 100ms sleep)
-
    -
    -**4. View the trace in Grafana UI:**
    -
    -Navigate to: Grafana → Explore → Tempo datasource
    -
    -Search using TraceQL:
    -
    -{ resource.service.namespace = "tracing-demo" }
-
    -
    -Or directly open the trace by pasting the trace ID in the search box:
    -
    -4be1151c0bdcd5625ac7e02b98d95bd5
-
    -
    -**5. Trace visualization:**
    -
    -The trace waterfall view in Grafana shows the complete request flow with timing:
    -
    -Distributed trace visualization in Grafana Tempo showing Frontend → Middleware → Backend spans
    -
    -For additional examples of Tempo trace visualization, see also:
    -
    -X-RAG Observability Hackathon (more Grafana Tempo screenshots)
    -
    -The trace reveals the distributed request flow:
    -
    -
      -
    • Frontend (221ms): Receives GET /api/process, executes business logic, calls middleware
      • -
    • Middleware (186ms): Receives POST /api/transform, transforms data, calls backend
      • -
    • Backend (104ms): Receives GET /api/data, simulates database query with 100ms sleep
      • -
    • Total request time: 221ms end-to-end
      • -
    • Span propagation: W3C Trace Context headers automatically link all spans
      • -

    -**6. Service graph visualization:**
    -
    -The service graph is automatically generated from traces and shows service dependencies. For examples of service graph visualization in Grafana, see the screenshots in the X-RAG Observability Hackathon blog post.
    -
    -X-RAG Observability Hackathon (includes service graph screenshots)
    -
    -This visualization helps identify:
    -
    -
      -
    • Request rates between services
      • -
    • Average latency for each hop
      • -
    • Error rates (if any)
      • -
    • Service dependencies and communication patterns
      • -

    -

    Storage and Retention


    -
    -Monitor Tempo storage usage:
    -
    -
    -kubectl exec -n monitoring <tempo-pod> -- df -h /var/tempo
-
    -
    -With 10Gi storage and 7-day retention, the system handles moderate trace volumes. If storage fills up:
    -
    -
      -
    • Reduce retention to 72h (3 days)
      • -
    • Implement sampling in Alloy
      • -
    • Increase PV size
      • -

    -

    Configuration Files


    -
    -All configuration files are available on Codeberg:
    -
    -Tempo configuration
    -Alloy configuration (updated for traces)
    -Demo tracing application
    -

    Summary



    -With Prometheus, Grafana, Loki, Alloy, and Tempo deployed, I now have complete visibility into the k3s cluster, the FreeBSD storage servers, and the OpenBSD edge relays:
    +With Prometheus, Grafana, Loki, and Alloy deployed, I now have visibility into the k3s cluster, the FreeBSD storage servers, and the OpenBSD edge relays:

    • Metrics: Prometheus collects and stores time-series data from all components, including etcd and ZFS
    • Logs: Loki aggregates logs from all containers, searchable via Grafana
      • -
    • Traces: Tempo provides distributed request tracing with service dependency mapping
      • -
    • Visualisation: Grafana provides dashboards and exploration tools with correlation between all three signals
      • +
    • Visualisation: Grafana provides dashboards and exploration tools
    • Alerting: Alertmanager can notify on conditions defined in Prometheus rules

    -This observability stack runs entirely on the home lab infrastructure, with data persisted to the NFS share. It's lightweight enough for a three-node cluster but provides the same capabilities as production-grade setups.
    +The next part covers the final pillar of observability: distributed tracing with Grafana Tempo.
    +
    +Part 8b: Distributed Tracing with Tempo

    All configuration files are available on Codeberg:

    Prometheus, Grafana, and recording rules configuration
    Loki and Alloy configuration
    -Tempo configuration
    -Demo tracing application

    Other *BSD-related posts:

    2026-04-02 f3s: Kubernetes with FreeBSD - Part 9: GitOps with ArgoCD
    +2025-12-14 f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo
    2025-12-07 f3s: Kubernetes with FreeBSD - Part 8: Observability (You are currently reading this)
    2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments
    2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage
    diff --git a/gemfeed/2025-12-14-f3s-kubernetes-with-freebsd-part-8b.html b/gemfeed/2025-12-14-f3s-kubernetes-with-freebsd-part-8b.html new file mode 100644 index 00000000..e08937b7 --- /dev/null +++ b/gemfeed/2025-12-14-f3s-kubernetes-with-freebsd-part-8b.html @@ -0,0 +1,532 @@ + + + + +f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo + + + + + +

    +Home | Markdown | Gemini +

    +

    f3s: Kubernetes with FreeBSD - Part 8b: Distributed Tracing with Tempo


    +
    +Published at 2025-12-14T20:00:00+02:00
    +
    +This is a follow-up to Part 8 of the f3s series, where I covered Prometheus, Grafana, Loki, and Alloy. Now it's time for the last pillar of observability: distributed tracing with Grafana Tempo.
    +
    +Part 8: Observability (Prometheus, Grafana, Loki, Alloy)
    +
    +For a preview of what distributed tracing with Tempo looks like in Grafana, check out the X-RAG blog post:
    +
    +X-RAG Observability Hackathon
    +
    +

    Table of Contents


    +
    +
    +

    Why Distributed Tracing?


    +
    +In a microservices setup, a single user request can hop through multiple services. Tracing gives you:
    +
    +
      +
    • Request tracking across service boundaries
      • +
    • Performance bottleneck identification
      • +
    • Service dependency visualization
      • +
    • Correlation with logs and metrics
      • +

    +Without it, you're basically guessing where time gets spent.
    +
    +

    Deploying Grafana Tempo


    +
    +Tempo runs in monolithic mode — all components in one process, same pattern as Loki's SingleBinary deployment. Keeps things simple for a home lab.
    +
    +The setup:
    +
    +
      +
    • Filesystem backend using hostPath (10Gi at /data/nfs/k3svolumes/tempo/data)
      • +
    • 7-day retention (168h)
      • +
    • OTLP receivers on gRPC (4317) and HTTP (4318)
      • +
    • Bind to 0.0.0.0 to avoid Tempo 2.7+ localhost-only binding issue
      • +

    +

    Tempo Helm Values


    +
    +
    +tempo:
+  retention: 168h
+  storage:
+    trace:
+      backend: local
+      local:
+        path: /var/tempo/traces
+      wal:
+        path: /var/tempo/wal
+  receivers:
+    otlp:
+      protocols:
+        grpc:
+          endpoint: 0.0.0.0:4317
+        http:
+          endpoint: 0.0.0.0:4318
+
+persistence:
+  enabled: true
+  size: 10Gi
+  storageClassName: ""
+
+resources:
+  limits:
+    cpu: 1000m
+    memory: 2Gi
+  requests:
+    cpu: 500m
+    memory: 1Gi
+
    +
    +

    Persistent Volumes


    +
    +
    +apiVersion: v1
+kind: PersistentVolume
+metadata:
+  name: tempo-data-pv
+spec:
+  capacity:
+    storage: 10Gi
+  accessModes:
+    - ReadWriteOnce
+  persistentVolumeReclaimPolicy: Retain
+  hostPath:
+    path: /data/nfs/k3svolumes/tempo/data
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: tempo-data-pvc
+  namespace: monitoring
+spec:
+  storageClassName: ""
+  accessModes:
+    - ReadWriteOnce
+  resources:
+    requests:
+      storage: 10Gi
+
    +
    +

    Grafana Datasource Provisioning


    +
    +All Grafana datasources (Prometheus, Alertmanager, Loki, Tempo) are provisioned via a single ConfigMap mounted directly to the Grafana pod. No sidecar discovery needed.
    +
    +In grafana-datasources-all.yaml:
    +
    +
    +apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-datasources-all
+  namespace: monitoring
+data:
+  datasources.yaml: |
+    apiVersion: 1
+    datasources:
+      - name: Prometheus
+        type: prometheus
+        uid: prometheus
+        url: http://prometheus-kube-prometheus-prometheus.monitoring:9090/
+        access: proxy
+        isDefault: true
+      - name: Alertmanager
+        type: alertmanager
+        uid: alertmanager
+        url: http://prometheus-kube-prometheus-alertmanager.monitoring:9093/
+      - name: Loki
+        type: loki
+        uid: loki
+        url: http://loki.monitoring.svc.cluster.local:3100
+      - name: Tempo
+        type: tempo
+        uid: tempo
+        url: http://tempo.monitoring.svc.cluster.local:3200
+        jsonData:
+          tracesToLogsV2:
+            datasourceUid: loki
+            spanStartTimeShift: -1h
+            spanEndTimeShift: 1h
+          tracesToMetrics:
+            datasourceUid: prometheus
+          serviceMap:
+            datasourceUid: prometheus
+          nodeGraph:
+            enabled: true
+
    +
    +The Tempo datasource config links traces to Loki logs and Prometheus metrics — so you can jump between signals directly in Grafana.
    +
    +The kube-prometheus-stack Helm values disable sidecar-based discovery and mount this ConfigMap directly to /etc/grafana/provisioning/datasources/.
    +
    +

    Installation


    +
    +
    +cd /home/paul/git/conf/f3s/tempo
+just install
+
    +
    +Verify it's running:
    +
    +
    +kubectl get pods -n monitoring -l app.kubernetes.io/name=tempo
+kubectl exec -n monitoring <tempo-pod> -- wget -qO- http://localhost:3200/ready
+
    +
    +

    Configuring Alloy for Trace Collection


    +
    +I updated the Alloy values to add OTLP receivers for traces alongside the existing log collection.
    +
    +Added to the Alloy config:
    +
    +
    +// OTLP receiver for traces via gRPC and HTTP
+otelcol.receiver.otlp "default" {
+  grpc {
+    endpoint = "0.0.0.0:4317"
+  }
+  http {
+    endpoint = "0.0.0.0:4318"
+  }
+  output {
+    traces = [otelcol.processor.batch.default.input]
+  }
+}
+
+// Batch processor — accumulates spans before forwarding to Tempo
+otelcol.processor.batch "default" {
+  timeout = "5s"
+  send_batch_size = 100
+  send_batch_max_size = 200
+  output {
+    traces = [otelcol.exporter.otlp.tempo.input]
+  }
+}
+
+// OTLP exporter to Tempo
+otelcol.exporter.otlp "tempo" {
+  client {
+    endpoint = "tempo.monitoring.svc.cluster.local:4317"
+    tls {
+      insecure = true
+    }
+    compression = "gzip"
+  }
+}
+
    +
    +Upgrade Alloy:
    +
    +
    +cd /home/paul/git/conf/f3s/loki
+just upgrade
+
    +
    +

    Demo Tracing Application


    +
    +To actually see traces, I built a three-tier Python app. Nothing fancy — just enough to generate real distributed traces.
    +
    +

    Architecture


    +
    +
    +User -> Frontend (Flask:5000) -> Middleware (Flask:5001) -> Backend (Flask:5002)
+           |                          |                        |
+                    Alloy (OTLP:4317) -> Tempo -> Grafana
+
    +
    +
      +
    • Frontend: receives requests at /api/process, forwards to middleware
      • +
    • Middleware: transforms data at /api/transform, calls backend
      • +
    • Backend: returns data at /api/data, simulates a 100ms database query
      • +

    +

    OpenTelemetry Instrumentation


    +
    +All three services use Python OpenTelemetry libraries:
    +
    +Dependencies:
    +
    +
    +flask==3.0.0
+requests==2.31.0
+opentelemetry-distro==0.49b0
+opentelemetry-exporter-otlp==1.28.0
+opentelemetry-instrumentation-flask==0.49b0
+opentelemetry-instrumentation-requests==0.49b0
+
    +
    +Auto-instrumentation pattern (same across all services, just change the service name):
    +
    + +
    from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+from opentelemetry.instrumentation.flask import FlaskInstrumentor
+from opentelemetry.instrumentation.requests import RequestsInstrumentor
+from opentelemetry.sdk.resources import Resource
+
+resource = Resource(attributes={
+    "service.name": "frontend",
+    "service.namespace": "tracing-demo",
+    "service.version": "1.0.0"
+})
+
+provider = TracerProvider(resource=resource)
+
+otlp_exporter = OTLPSpanExporter(
+    endpoint="http://alloy.monitoring.svc.cluster.local:4317",
+    insecure=True
+)
+
+processor = BatchSpanProcessor(otlp_exporter)
+provider.add_span_processor(processor)
+trace.set_tracer_provider(provider)
+
+FlaskInstrumentor().instrument_app(app)
+RequestsInstrumentor().instrument()
+
    +
    +The auto-instrumentation creates spans for HTTP requests, propagates trace context via W3C headers, and links parent/child span