docs: DTail fork — documentation, agent guide, example configsHEADmaster

Squashed development of the documentation and example configuration:

- AGENTS.md / CLAUDE.md: repository guide describing build/test/benchmark/PGO
  workflows and the single default read/output path (formerly "turbo").
- doc/ and docs/: query-language reference, log formats, auth-key fast reconnect,
  journal source reads, performance analyses (dated point-in-time records kept
  under historical-note disclaimers), and the turbo-vs-normal benchmark report
  with its result CSVs.
- README.md updates; examples/ config + JSON schema aligned with the current
  Output* server tuning fields (the removed TurboBoost* keys dropped).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

1 files changed, 126 insertions, 0 deletions

diff --git a/docs/turbo-vs-normal-benchmark-20260718.md b/docs/turbo-vs-normal-benchmark-20260718.md
new file mode 100644
index 0000000..3d86fa9
--- /dev/null
+++ b/docs/turbo-vs-normal-benchmark-20260718.md
@@ -0,0 +1,126 @@
+# Turbo vs Normal (Non-Turbo) Benchmark Report — 2026-07-18
+
+**Question:** After the recent server/turbo-path work (in particular task `ws0`,
+which gave `TurboAggregate` an input-exhausted `FinishInput` signal), is turbo
+mode still faster than the normal (channel-based) path — and does the
+server-mode `dmap` deadlock the previous report flagged still exist?
+
+**Answer: Yes — turbo remains faster in every scenario where it applies, and the
+result is reproducible across two independent full runs. The server-mode `dmap`
+turbo deadlock documented in the 2026-07-10 report is FIXED: turbo `dmap` in
+server mode now completes and is ~1.6×–2.6× faster than non-turbo. Server-mode
+`dcat`, previously transport-bound and neutral, is now ~6–8× faster on the turbo
+path.**
+
+## Methodology
+
+- Binaries rebuilt from clean at HEAD `1a079d9` (`make clean && make build`).
+- Turbo is default-on; non-turbo runs set `DTAIL_TURBOBOOST_DISABLE=yes` on the
+  process that runs the server handlers (the client itself in serverless mode,
+  `dserver` in server mode).
+- Serverless scenarios: client binaries run directly against local files.
+- Server scenarios: two `dserver` instances on localhost (one turbo, one
+  non-turbo, `--cfg none`, real SSH auth via a dedicated RSA key + `CacheDir`
+  authorized_keys) — *not* `DTAIL_INTEGRATION_TEST_RUN_MODE`, because that mode
+  force-disables turbo (`internal/config/initializer.go`) and would compare
+  non-turbo against non-turbo.
+- Turbo/non-turbo iterations alternate A/B to spread thermal/load drift (dcat,
+  dgrep, dmap scenarios); the dtail follow scenario runs all turbo iterations
+  first, then all non-turbo. 1 warmup + 5–7 measured iterations per
+  scenario/mode; medians reported.
+- Wall-clock timing, client stdout to `/dev/null` (except dtail follow).
+- Datasets: `benchmarks/testdata/medium.log` (100 MB, 1.22 M lines),
+  `large.log` (1 GB), and a generated 100 MB `MAPREDUCE:STATS` file (800 k
+  lines, 50 hostnames). dgrep patterns: `"user999 "` (low hit rate, ~0.07 %) and
+  `"ERROR"` (high hit rate, ~19 %).
+- dtail follow: client follows an empty file over SSH; after the stream is
+  established (probe line round-trip), a 10 MB / 100 k-line burst plus end
+  marker is appended; time from append to marker arrival at the client is
+  measured, and delivered-line completeness is counted.
+- The benchmark was run twice end to end (run 1 and run 2) to check
+  reproducibility. Raw per-iteration data is committed alongside this report:
+  `turbo-vs-normal-benchmark-20260718-run1.csv` and `...-run2.csv`. Harness:
+  `benchmarks/turbo_vs_normal_bench.sh`.
+- Machine: 11th Gen Intel i7-1185G7 @ 3.00 GHz (8 threads), Fedora Linux 44,
+  Go 1.26.4. Caveat: laptop; absolute numbers are noisy (thermal/background
+  load), so the **turbo/non-turbo ratios are the meaningful signal**, not the
+  absolute seconds. This is visible in the server-mode rows, where run 2's
+  non-turbo times are lower than run 1's (lighter machine load), which
+  compresses those ratios without any change to the turbo path.
+- Non-turbo serverless runs go through an `env` wrapper that turbo runs do not
+  (~1 ms extra process spawn). Negligible at these magnitudes, and it biases
+  against turbo, not for it.
+
+## Results (median wall time, seconds)
+
+### Serverless mode
+
+| Scenario                     | Turbo (r1 / r2) | Non-Turbo (r1 / r2) | Speedup (r1 / r2) | Verdict |
+|------------------------------|-----------------|---------------------|-------------------|---------|
+| dcat 100 MB                  | 0.094 / 0.094   | 1.310 / 1.322       | 13.9× / 14.1×     | turbo much faster |
+| dcat 1 GB                    | 0.991 / 1.034   | 17.776 / 17.912     | 17.9× / 17.3×     | turbo much faster |
+| dgrep 100 MB, low hit rate   | 0.106 / 0.111   | 2.104 / 2.289       | 19.8× / 20.6×     | turbo much faster |
+| dgrep 100 MB, high hit rate  | 0.098 / 0.104   | 2.184 / 2.342       | 22.3× / 22.5×     | turbo much faster |
+| dmap agg 100 MB (control)    | 3.072 / 2.999   | 3.279 / 3.068       | 1.07× / 1.02×     | expected: turbo N/A serverless |
+
+Serverless ratios are extremely stable between the two runs (dcat ~14–18×,
+dgrep ~20–22×). Serverless `dmap` shows parity because turbo is a server-side
+optimization; client-side aggregation does not use it (as documented).
+
+### Server mode (localhost SSH)
+
+| Scenario                     | Turbo (r1 / r2) | Non-Turbo (r1 / r2) | Speedup (r1 / r2) | Verdict |
+|------------------------------|-----------------|---------------------|-------------------|---------|
+| dcat 100 MB                  | 1.962 / 1.253   | 15.263 / 8.274      | 7.8× / 6.6×       | turbo much faster |
+| dgrep 100 MB, low hit rate   | 0.783 / 0.725   | 5.524 / 2.176       | 7.1× / 3.0×       | turbo faster |
+| dgrep 100 MB, high hit rate  | 0.962 / 0.763   | 6.199 / 2.893       | 6.4× / 3.8×       | turbo faster |
+| dmap count, 100 MB stats     | 2.442 / 1.491   | 6.113 / 3.841       | 2.5× / 2.6×       | **turbo faster — now completes (was deadlock)** |
+| dmap group-by-agg, 100 MB    | 2.729 / 1.692   | 5.318 / 2.667       | 1.9× / 1.6×       | **turbo faster — now completes (was deadlock)** |
+| dtail follow 10 MB burst     | 0.240 / 0.115, **100 % delivered** | timeout, ~39–47 % delivered* | n/a | turbo strictly better |
+
+\* Non-turbo tail drops lines when the consumer lags (documented behavior,
+`internal/io/line/line.go`). Across both runs the non-turbo follow delivered
+only ~39 000–47 000 of 100 000 lines and generally never received the end
+marker, timing out at >90 s. Turbo delivered all 100 000 lines in ~0.1–0.24 s in
+every iteration.
+
+## Per-tool verdicts
+
+- **dcat**: turbo a big win serverless (~14–18×). In server mode it is now
+  ~6–8× faster (previously transport-bound and neutral) — the turbo network
+  writer path (64 KB buffering, `bytesWritten` accounting) pays off over SSH.
+- **dgrep**: turbo a big win everywhere (serverless ~20–22×; server ~3–7×,
+  larger under heavier load).
+- **dtail**: turbo is strictly better — complete delivery in ~0.1 s vs lossy
+  (~40 %) delivery that times out on the normal path.
+- **dmap (serverless)**: turbo not applicable; parity confirmed.
+- **dmap (server mode)**: **now works.** Turbo aggregation completes and is
+  ~1.6×–2.6× faster than non-turbo. See the "what changed" note below.
+
+## What changed since the 2026-07-10 report
+
+The previous report (`benchmarks/turbo_vs_normal_report_20260710.md`) found
+turbo dramatically faster everywhere it applied **but** uncovered a functional
+deadlock: server-mode `dmap` with turbo enabled (the default) never terminated
+the client session, because nothing signaled "input exhausted" to
+`TurboAggregate` — the map command stayed active, so the shutdown coordinator's
+`activeCommands == 0` finalization never ran.
+
+Task `ws0` fixed this by giving `TurboAggregate` a `FinishInput()` that performs
+the final flush + serialize + `done.Shutdown()` once input is exhausted,
+analogous to the regular aggregate's channel-close path. This benchmark confirms
+the fix end to end: server-mode turbo `dmap` (both `count group by` and the
+multi-aggregation query) completes cleanly in every iteration of both runs and
+is faster than the non-turbo path. No hangs, no `timeout`/`rc124` rows.
+
+## Reproducing
+
+```bash
+make clean && make build
+bash benchmarks/turbo_vs_normal_bench.sh /tmp/dtail_turbo_bench
+# medians:
+awk -F, 'NR>1 && $4!="" {k=$1","$2; v[k]=v[k]" "$4}
+  END{for(x in v){n=split(v[x],a," ");asort(a);
+  m=(n%2)?a[(n+1)/2]:(a[n/2]+a[n/2+1])/2;printf "%s,%.3f\n",x,m}}' \
+  /tmp/dtail_turbo_bench/results.csv | sort
+```