From 3004a7100e325c006971cc2e8d0f157338c0ce5c Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Wed, 22 Jul 2026 23:52:52 +0300 Subject: =?UTF-8?q?docs:=20DTail=20fork=20=E2=80=94=20documentation,=20age?= =?UTF-8?q?nt=20guide,=20example=20configs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Squashed development of the documentation and example configuration: - AGENTS.md / CLAUDE.md: repository guide describing build/test/benchmark/PGO workflows and the single default read/output path (formerly "turbo"). - doc/ and docs/: query-language reference, log formats, auth-key fast reconnect, journal source reads, performance analyses (dated point-in-time records kept under historical-note disclaimers), and the turbo-vs-normal benchmark report with its result CSVs. - README.md updates; examples/ config + JSON schema aligned with the current Output* server tuning fields (the removed TurboBoost* keys dropped). Co-Authored-By: Claude Opus 4.8 --- doc/performance_optimization_summary.md | 69 +++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 doc/performance_optimization_summary.md (limited to 'doc/performance_optimization_summary.md') diff --git a/doc/performance_optimization_summary.md b/doc/performance_optimization_summary.md new file mode 100644 index 0000000..e54d9a9 --- /dev/null +++ b/doc/performance_optimization_summary.md @@ -0,0 +1,69 @@ +# DTail Performance Optimization Summary + +> **Historical note:** This is a point-in-time record of a specific optimization +> effort (trace-logging and buffering fixes) with the measurements taken at that +> time; the numbers and file paths (e.g. `turbo_writer.go`, since renamed to +> `line_writer.go`) are preserved as-is. The "turbo mode" it compares against a +> "non-turbo mode" is now the single, default read/output path — there is no mode +> split any more, and the `DTAIL_TURBOBOOST_DISABLE` toggle has been removed. + +## Changes Made + +### 1. Optimized Trace Logging (`/internal/io/dlog/dlog.go`) + +**Problem**: The `Trace()` and `Devel()` functions were calling `runtime.Caller(1)` for every invocation, even when trace logging was disabled. This was causing ~497ns overhead per call. + +**Solution**: Added early level checks before the expensive `runtime.Caller()` operation: + +```go +func (d *DLog) Trace(args ...interface{}) string { + // Early check to avoid expensive runtime.Caller when trace is disabled + if d.maxLevel < Trace { + return "" + } + // ... rest of function +} +``` + +### 2. Improved Buffering in Turbo Mode (`/internal/server/handlers/turbo_writer.go`) + +**Problem**: Turbo mode was forcing immediate flush after every line in serverless mode, defeating the purpose of buffering. + +**Solution**: Removed the immediate flush condition for serverless mode, allowing proper buffering: + +```go +// Changed from: +if w.writeBuf.Len() >= w.bufSize || w.serverless { + return w.flushBuffer() +} + +// To: +if w.writeBuf.Len() >= w.bufSize { + return w.flushBuffer() +} +``` + +## Performance Results + +### Before Optimization +- Turbo mode was **3-5x slower** than non-turbo mode +- DCat (10MB): 678ms (turbo) vs 210ms (non-turbo) +- DGrep (10MB): 570ms (turbo) vs 96ms (non-turbo) + +### After Optimization +- Turbo mode is now **2.87x faster** than non-turbo mode +- DCat (1M lines): 0.66s (turbo) vs 1.89s (non-turbo) - **65% improvement** +- DCat with colors: 1.69s (turbo) vs 2.24s (non-turbo) - **24% improvement** + +## Verification +- ✅ All unit tests pass +- ✅ All integration tests pass +- ✅ No functionality regression +- ✅ Backward compatible + +## Files Modified +1. `/internal/io/dlog/dlog.go` - Lines 196-216 +2. `/internal/server/handlers/turbo_writer.go` - Lines 104-108 + +## Key Takeaway +The trace logging overhead was the primary bottleneck, causing DTail to spend more time logging than processing data. By adding simple level checks before expensive operations, we achieved a ~3x performance improvement in turbo mode. \ No newline at end of file -- cgit v1.2.3