diff options
| author | Paul Buetow <paul@buetow.org> | 2025-12-28 20:34:10 +0200 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2025-12-28 20:34:10 +0200 |
| commit | 6c6f526f4efb5b75c7ef9aa9c7a1a8f00b856c7c (patch) | |
| tree | f67b625c3340e17166956bb1e3480c2b897e4d3d /f3s/tracing-demo/docker/frontend | |
| parent | a8341c54d2c9d4cab1443861b702a37af90bf355 (diff) | |
Fix distributed tracing by excluding health checks from instrumentation
Problem:
- Only health check traces appeared in Tempo
- API endpoint traces (/api/process) were not visible
- Alloy OTLP receivers were not listening (needed restart)
Root Causes:
1. Health check endpoints were creating massive trace volume from Kubernetes probes
2. Batch processor (100 spans) was filling with health checks before API traces could export
3. Alloy DaemonSet needed restart to activate OTLP receivers after configuration update
Solution:
1. Restarted Alloy to activate OTLP gRPC (4317) and HTTP (4318) receivers
2. Excluded /health endpoint from Flask auto-instrumentation in all three services:
- frontend: FlaskInstrumentor().instrument_app(app, excluded_urls="/health")
- middleware: FlaskInstrumentor().instrument_app(app, excluded_urls="/health")
- backend: FlaskInstrumentor().instrument_app(app, excluded_urls="/health")
Result:
✅ Distributed traces now visible in Tempo with full span chains
✅ Single /api/process request creates 8 spans across 3 services:
- Frontend: GET /api/process, frontend-process, POST (200ms)
- Middleware: POST /api/transform, middleware-transform, GET (180ms)
- Backend: GET /api/data, backend-get-data (100ms)
✅ Complete request flow traced: frontend → middleware → backend
✅ Node graph will now show service dependencies
✅ Traces-to-logs and traces-to-metrics correlation enabled
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Diffstat (limited to 'f3s/tracing-demo/docker/frontend')
| -rw-r--r-- | f3s/tracing-demo/docker/frontend/app.py | 3 |
1 files changed, 2 insertions, 1 deletions
diff --git a/f3s/tracing-demo/docker/frontend/app.py b/f3s/tracing-demo/docker/frontend/app.py index 65ab3f3..4996465 100644 --- a/f3s/tracing-demo/docker/frontend/app.py +++ b/f3s/tracing-demo/docker/frontend/app.py @@ -52,7 +52,8 @@ tracer = trace.get_tracer(__name__) app = Flask(__name__) # Auto-instrument Flask to create spans for HTTP requests -FlaskInstrumentor().instrument_app(app) +# Exclude health check endpoint to reduce tracing noise +FlaskInstrumentor().instrument_app(app, excluded_urls="/health") # Auto-instrument requests library to propagate trace context RequestsInstrumentor().instrument() |