diff options
| author | Paul Buetow <paul@buetow.org> | 2025-12-28 20:34:10 +0200 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2025-12-28 20:34:10 +0200 |
| commit | 6c6f526f4efb5b75c7ef9aa9c7a1a8f00b856c7c (patch) | |
| tree | f67b625c3340e17166956bb1e3480c2b897e4d3d /frontends | |
| parent | a8341c54d2c9d4cab1443861b702a37af90bf355 (diff) | |
Fix distributed tracing by excluding health checks from instrumentation
Problem:
- Only health check traces appeared in Tempo
- API endpoint traces (/api/process) were not visible
- Alloy OTLP receivers were not listening (needed restart)
Root Causes:
1. Health check endpoints were creating massive trace volume from Kubernetes probes
2. Batch processor (100 spans) was filling with health checks before API traces could export
3. Alloy DaemonSet needed restart to activate OTLP receivers after configuration update
Solution:
1. Restarted Alloy to activate OTLP gRPC (4317) and HTTP (4318) receivers
2. Excluded /health endpoint from Flask auto-instrumentation in all three services:
- frontend: FlaskInstrumentor().instrument_app(app, excluded_urls="/health")
- middleware: FlaskInstrumentor().instrument_app(app, excluded_urls="/health")
- backend: FlaskInstrumentor().instrument_app(app, excluded_urls="/health")
Result:
✅ Distributed traces now visible in Tempo with full span chains
✅ Single /api/process request creates 8 spans across 3 services:
- Frontend: GET /api/process, frontend-process, POST (200ms)
- Middleware: POST /api/transform, middleware-transform, GET (180ms)
- Backend: GET /api/data, backend-get-data (100ms)
✅ Complete request flow traced: frontend → middleware → backend
✅ Node graph will now show service dependencies
✅ Traces-to-logs and traces-to-metrics correlation enabled
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Diffstat (limited to 'frontends')
0 files changed, 0 insertions, 0 deletions
