diff options
Diffstat (limited to 'docs/turbo-vs-normal-benchmark-20260718.md')
| -rw-r--r-- | docs/turbo-vs-normal-benchmark-20260718.md | 126 |
1 files changed, 126 insertions, 0 deletions
diff --git a/docs/turbo-vs-normal-benchmark-20260718.md b/docs/turbo-vs-normal-benchmark-20260718.md new file mode 100644 index 0000000..3d86fa9 --- /dev/null +++ b/docs/turbo-vs-normal-benchmark-20260718.md @@ -0,0 +1,126 @@ +# Turbo vs Normal (Non-Turbo) Benchmark Report — 2026-07-18 + +**Question:** After the recent server/turbo-path work (in particular task `ws0`, +which gave `TurboAggregate` an input-exhausted `FinishInput` signal), is turbo +mode still faster than the normal (channel-based) path — and does the +server-mode `dmap` deadlock the previous report flagged still exist? + +**Answer: Yes — turbo remains faster in every scenario where it applies, and the +result is reproducible across two independent full runs. The server-mode `dmap` +turbo deadlock documented in the 2026-07-10 report is FIXED: turbo `dmap` in +server mode now completes and is ~1.6×–2.6× faster than non-turbo. Server-mode +`dcat`, previously transport-bound and neutral, is now ~6–8× faster on the turbo +path.** + +## Methodology + +- Binaries rebuilt from clean at HEAD `1a079d9` (`make clean && make build`). +- Turbo is default-on; non-turbo runs set `DTAIL_TURBOBOOST_DISABLE=yes` on the + process that runs the server handlers (the client itself in serverless mode, + `dserver` in server mode). +- Serverless scenarios: client binaries run directly against local files. +- Server scenarios: two `dserver` instances on localhost (one turbo, one + non-turbo, `--cfg none`, real SSH auth via a dedicated RSA key + `CacheDir` + authorized_keys) — *not* `DTAIL_INTEGRATION_TEST_RUN_MODE`, because that mode + force-disables turbo (`internal/config/initializer.go`) and would compare + non-turbo against non-turbo. +- Turbo/non-turbo iterations alternate A/B to spread thermal/load drift (dcat, + dgrep, dmap scenarios); the dtail follow scenario runs all turbo iterations + first, then all non-turbo. 1 warmup + 5–7 measured iterations per + scenario/mode; medians reported. +- Wall-clock timing, client stdout to `/dev/null` (except dtail follow). +- Datasets: `benchmarks/testdata/medium.log` (100 MB, 1.22 M lines), + `large.log` (1 GB), and a generated 100 MB `MAPREDUCE:STATS` file (800 k + lines, 50 hostnames). dgrep patterns: `"user999 "` (low hit rate, ~0.07 %) and + `"ERROR"` (high hit rate, ~19 %). +- dtail follow: client follows an empty file over SSH; after the stream is + established (probe line round-trip), a 10 MB / 100 k-line burst plus end + marker is appended; time from append to marker arrival at the client is + measured, and delivered-line completeness is counted. +- The benchmark was run twice end to end (run 1 and run 2) to check + reproducibility. Raw per-iteration data is committed alongside this report: + `turbo-vs-normal-benchmark-20260718-run1.csv` and `...-run2.csv`. Harness: + `benchmarks/turbo_vs_normal_bench.sh`. +- Machine: 11th Gen Intel i7-1185G7 @ 3.00 GHz (8 threads), Fedora Linux 44, + Go 1.26.4. Caveat: laptop; absolute numbers are noisy (thermal/background + load), so the **turbo/non-turbo ratios are the meaningful signal**, not the + absolute seconds. This is visible in the server-mode rows, where run 2's + non-turbo times are lower than run 1's (lighter machine load), which + compresses those ratios without any change to the turbo path. +- Non-turbo serverless runs go through an `env` wrapper that turbo runs do not + (~1 ms extra process spawn). Negligible at these magnitudes, and it biases + against turbo, not for it. + +## Results (median wall time, seconds) + +### Serverless mode + +| Scenario | Turbo (r1 / r2) | Non-Turbo (r1 / r2) | Speedup (r1 / r2) | Verdict | +|------------------------------|-----------------|---------------------|-------------------|---------| +| dcat 100 MB | 0.094 / 0.094 | 1.310 / 1.322 | 13.9× / 14.1× | turbo much faster | +| dcat 1 GB | 0.991 / 1.034 | 17.776 / 17.912 | 17.9× / 17.3× | turbo much faster | +| dgrep 100 MB, low hit rate | 0.106 / 0.111 | 2.104 / 2.289 | 19.8× / 20.6× | turbo much faster | +| dgrep 100 MB, high hit rate | 0.098 / 0.104 | 2.184 / 2.342 | 22.3× / 22.5× | turbo much faster | +| dmap agg 100 MB (control) | 3.072 / 2.999 | 3.279 / 3.068 | 1.07× / 1.02× | expected: turbo N/A serverless | + +Serverless ratios are extremely stable between the two runs (dcat ~14–18×, +dgrep ~20–22×). Serverless `dmap` shows parity because turbo is a server-side +optimization; client-side aggregation does not use it (as documented). + +### Server mode (localhost SSH) + +| Scenario | Turbo (r1 / r2) | Non-Turbo (r1 / r2) | Speedup (r1 / r2) | Verdict | +|------------------------------|-----------------|---------------------|-------------------|---------| +| dcat 100 MB | 1.962 / 1.253 | 15.263 / 8.274 | 7.8× / 6.6× | turbo much faster | +| dgrep 100 MB, low hit rate | 0.783 / 0.725 | 5.524 / 2.176 | 7.1× / 3.0× | turbo faster | +| dgrep 100 MB, high hit rate | 0.962 / 0.763 | 6.199 / 2.893 | 6.4× / 3.8× | turbo faster | +| dmap count, 100 MB stats | 2.442 / 1.491 | 6.113 / 3.841 | 2.5× / 2.6× | **turbo faster — now completes (was deadlock)** | +| dmap group-by-agg, 100 MB | 2.729 / 1.692 | 5.318 / 2.667 | 1.9× / 1.6× | **turbo faster — now completes (was deadlock)** | +| dtail follow 10 MB burst | 0.240 / 0.115, **100 % delivered** | timeout, ~39–47 % delivered* | n/a | turbo strictly better | + +\* Non-turbo tail drops lines when the consumer lags (documented behavior, +`internal/io/line/line.go`). Across both runs the non-turbo follow delivered +only ~39 000–47 000 of 100 000 lines and generally never received the end +marker, timing out at >90 s. Turbo delivered all 100 000 lines in ~0.1–0.24 s in +every iteration. + +## Per-tool verdicts + +- **dcat**: turbo a big win serverless (~14–18×). In server mode it is now + ~6–8× faster (previously transport-bound and neutral) — the turbo network + writer path (64 KB buffering, `bytesWritten` accounting) pays off over SSH. +- **dgrep**: turbo a big win everywhere (serverless ~20–22×; server ~3–7×, + larger under heavier load). +- **dtail**: turbo is strictly better — complete delivery in ~0.1 s vs lossy + (~40 %) delivery that times out on the normal path. +- **dmap (serverless)**: turbo not applicable; parity confirmed. +- **dmap (server mode)**: **now works.** Turbo aggregation completes and is + ~1.6×–2.6× faster than non-turbo. See the "what changed" note below. + +## What changed since the 2026-07-10 report + +The previous report (`benchmarks/turbo_vs_normal_report_20260710.md`) found +turbo dramatically faster everywhere it applied **but** uncovered a functional +deadlock: server-mode `dmap` with turbo enabled (the default) never terminated +the client session, because nothing signaled "input exhausted" to +`TurboAggregate` — the map command stayed active, so the shutdown coordinator's +`activeCommands == 0` finalization never ran. + +Task `ws0` fixed this by giving `TurboAggregate` a `FinishInput()` that performs +the final flush + serialize + `done.Shutdown()` once input is exhausted, +analogous to the regular aggregate's channel-close path. This benchmark confirms +the fix end to end: server-mode turbo `dmap` (both `count group by` and the +multi-aggregation query) completes cleanly in every iteration of both runs and +is faster than the non-turbo path. No hangs, no `timeout`/`rc124` rows. + +## Reproducing + +```bash +make clean && make build +bash benchmarks/turbo_vs_normal_bench.sh /tmp/dtail_turbo_bench +# medians: +awk -F, 'NR>1 && $4!="" {k=$1","$2; v[k]=v[k]" "$4} + END{for(x in v){n=split(v[x],a," ");asort(a); + m=(n%2)?a[(n+1)/2]:(a[n/2]+a[n/2+1])/2;printf "%s,%.3f\n",x,m}}' \ + /tmp/dtail_turbo_bench/results.csv | sort +``` |