Add Grafana Tempo distributed tracing with demo application

- Deploy Grafana Tempo in monolithic mode for distributed tracing
- Configure Tempo with OTLP receivers (gRPC:4317, HTTP:4318)
- Set up 10Gi filesystem storage with 7-day retention
- Integrate Tempo datasource in Grafana with traces-to-logs and traces-to-metrics correlation

- Update Grafana Alloy to collect and forward traces
- Add OTLP receiver configuration to alloy-values.yaml
- Configure batch processor for efficient trace forwarding to Tempo
- Patch Alloy service to expose OTLP ports 4317/4318

- Create demo tracing application (frontend, middleware, backend)
- Implement three-tier Python Flask application with OpenTelemetry instrumentation
- Auto-instrument with OpenTelemetry for Flask and requests libraries
- Push Docker images to private registry (registry.lan.buetow.org:30001)
- Deploy via Helm chart with Traefik ingress at tracing-demo.f3s.buetow.org

- Update Grafana configuration in prometheus/persistence-values.yaml
- Add Tempo to additionalDataSources for automatic provisioning

Files added:
- tempo/values.yaml: Tempo Helm chart configuration
- tempo/persistent-volumes.yaml: Storage configuration (10Gi PV/PVC)
- tempo/datasource-configmap.yaml: Grafana datasource with correlations
- tempo/Justfile: Installation automation
- tempo/README.md: Documentation

- tracing-demo/docker/frontend/: Python Flask frontend with OTel
- tracing-demo/docker/middleware/: Python Flask middleware with OTel
- tracing-demo/docker/backend/: Python Flask backend with OTel
- tracing-demo/helm-chart/: Kubernetes deployments, services, ingress
- tracing-demo/docker-image-Justfile: Docker build/push automation
- tracing-demo/Justfile: Helm deployment automation
- tracing-demo/README.md: Documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

1 files changed, 67 insertions, 0 deletions

diff --git a/f3s/loki/alloy-values.yaml b/f3s/loki/alloy-values.yaml
index 09da220..f53fd12 100644
--- a/f3s/loki/alloy-values.yaml
+++ b/f3s/loki/alloy-values.yaml
@@ -1,4 +1,17 @@
 alloy:
+  service:
+    ports:
+      otlp-grpc:
+        enabled: true
+        port: 4317
+        targetPort: 4317
+        protocol: TCP
+      otlp-http:
+        enabled: true
+        port: 4318
+        targetPort: 4318
+        protocol: TCP
+
   configMap:
     content: |
       discovery.kubernetes "pods" {
@@ -39,3 +52,57 @@ alloy:
           url = "http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push"
         }
       }
+
+      // ========================================
+      // TRACES COLLECTION
+      // ========================================
+
+      // OTLP receiver for traces via gRPC and HTTP
+      // Accepts traces from applications instrumented with OpenTelemetry
+      otelcol.receiver.otlp "default" {
+        // Accept OTLP over gRPC on port 4317 (standard OTLP port)
+        grpc {
+          endpoint = "0.0.0.0:4317"
+        }
+
+        // Accept OTLP over HTTP on port 4318 (standard OTLP HTTP port)
+        http {
+          endpoint = "0.0.0.0:4318"
+        }
+
+        output {
+          traces = [otelcol.processor.batch.default.input]
+        }
+      }
+
+      // Batch processor for efficient trace forwarding to Tempo
+      // Reduces network calls by batching spans before sending
+      otelcol.processor.batch "default" {
+        // Send batch every 5 seconds
+        timeout = "5s"
+
+        // Or when 100 spans have accumulated
+        send_batch_size = 100
+
+        // Maximum batch size as safety limit
+        send_batch_max_size = 200
+
+        output {
+          traces = [otelcol.exporter.otlp.tempo.input]
+        }
+      }
+
+      // OTLP exporter to send traces to Tempo
+      otelcol.exporter.otlp "tempo" {
+        client {
+          endpoint = "tempo.monitoring.svc.cluster.local:4317"
+
+          // Tempo doesn't use TLS for internal cluster communication
+          tls {
+            insecure = true
+          }
+
+          // Enable compression for efficiency
+          compression = "gzip"
+        }
+      }