docs: DTail fork — documentation, agent guide, example configsHEADmaster

Squashed development of the documentation and example configuration:

- AGENTS.md / CLAUDE.md: repository guide describing build/test/benchmark/PGO
  workflows and the single default read/output path (formerly "turbo").
- doc/ and docs/: query-language reference, log formats, auth-key fast reconnect,
  journal source reads, performance analyses (dated point-in-time records kept
  under historical-note disclaimers), and the turbo-vs-normal benchmark report
  with its result CSVs.
- README.md updates; examples/ config + JSON schema aligned with the current
  Output* server tuning fields (the removed TurboBoost* keys dropped).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

13 files changed, 1595 insertions, 46 deletions

diff --git a/doc/asciinema/README.md b/doc/asciinema/README.md
deleted file mode 100644
index 1eb1a2e..0000000
--- a/doc/asciinema/README.md
+++ /dev/null
@@ -1,41 +0,0 @@
-asciinema
-=========
-
-The animated Gifs you find in the DTail docs were created using:
-
-* [asciinema](https://asciinema.org) 
-* [asciicast2gif](https://github.com/asciinema/asciicast2gif)
-
-On Fedora Linux 35.
-
-## Installing prerequisites
-
-On Fedora Linux 35 install `asciinema`:
-
-```shell
-% sudo dnf install -y asciinema
-```
-
-and `asciicast2gif` (for simplicity, the Docker image was used):
-
-```shell
-% docker pull asciinema/asciicast2gif
-```
-
-This of course assumes that Docker is up and running on your machine (out of scope for this documentation).
-
-## Record a shell session
-
-This is as simple as running
-
-```shell
-% asciinema rec recording.json
-```
-
-This will launch a sub-shell to be recorded. Once done, exit the sub-shell with `exit`.
-
-## Convert the recording to a gif
-
-```shell
-% docker run --rm -v $PWD:/data asciinema/asciicast2gif -t tango -s 2 recording.json recording.gif
-```
diff --git a/doc/auth-key-fast-reconnect.md b/doc/auth-key-fast-reconnect.md
new file mode 100644
index 0000000..cb9884a
--- /dev/null
+++ b/doc/auth-key-fast-reconnect.md
@@ -0,0 +1,200 @@
+# Auth-Key Fast-Reconnect for DTail
+
+## Problem
+
+When using a YubiKey for SSH authentication, each DTail connection requires a
+physical touch of the YubiKey during the SSH handshake. This is slow and becomes
+painful when connecting to many servers concurrently — the YubiKey serialises
+all signing requests, turning parallel connections into sequential ones.
+
+## Solution
+
+Allow the DTail client to register a local SSH public key with the DTail server
+over an already-authenticated SSH session. The server caches this key
+**in-memory only** (never written to disk). On subsequent connections the client
+offers that local key first — a pure in-memory RSA verify with no YubiKey
+interaction — and falls back to the original auth method if the server does not
+recognise the key.
+
+## Design Principles
+
+1. **Transparent fallback** — Go's `golang.org/x/crypto/ssh` tries each
+   `AuthMethod` in order; if the fast key is rejected the client silently falls
+   back to the SSH agent / YubiKey. No user interaction required.
+2. **Server keys are ephemeral** — the in-memory store is lost on server
+   restart. No file I/O, no persistence.
+3. **Trust chain preserved** — an auth-key can only be registered over a session
+   that was already authenticated via the normal (YubiKey) path.
+4. **Minimal protocol addition** — a single `AUTHKEY <base64-pubkey>` command
+   sent over the existing SSH session text protocol.
+
+## Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────┐
+│ DTail Client                                            │
+│                                                         │
+│  Auth methods (tried in order):                         │
+│    1. Local private key (~/.ssh/id_rsa)  ← FAST         │
+│    2. SSH Agent / YubiKey               ← SLOW fallback │
+│                                                         │
+│  After slow-path auth:                                  │
+│    → sends AUTHKEY <~/.ssh/id_rsa.pub> to server        │
+└────────────────────────┬────────────────────────────────┘
+                         │ SSH
+┌────────────────────────▼────────────────────────────────┐
+│ DTail Server (dserver)                                  │
+│                                                         │
+│  PublicKeyCallback:                                     │
+│    1. Check in-memory authkeystore  ← FAST              │
+│    2. Check authorized_keys file    ← existing path     │
+│                                                         │
+│  AUTHKEY command handler:                               │
+│    → authkeystore.Add(user, pubkey)                     │
+│    → responds AUTHKEY OK / AUTHKEY ERR                  │
+│                                                         │
+│  authkeystore (in-memory only):                         │
+│    map[username] → []PublicKey (with TTL, max per user) │
+└─────────────────────────────────────────────────────────┘
+```
+
+## Sequence of Events
+
+### First Connection (slow path — YubiKey)
+
+1. Client checks for local private key at `~/.ssh/id_rsa` (or `--auth-key-path`).
+2. Client builds auth methods list: `[localKey, sshAgent]`.
+3. SSH handshake begins; server's `PublicKeyCallback` is called with local key.
+4. Server checks in-memory authkeystore → not found.
+5. Server checks `authorized_keys` file → not found (this key isn't in there).
+6. Server rejects the key.
+7. Go SSH client automatically tries next auth method: SSH agent (YubiKey).
+8. YubiKey signs the challenge; server finds the YubiKey pubkey in
+   `authorized_keys` → auth succeeds.
+9. Session is established; client sends DTail commands as usual.
+10. Client reads `~/.ssh/id_rsa.pub` and sends `AUTHKEY <base64-pubkey>`.
+11. Server's handler parses the command, calls `authkeystore.Add(user, pubkey)`.
+12. Server responds `AUTHKEY OK`.
+
+### Subsequent Connections (fast path — no YubiKey)
+
+1. Client builds auth methods list: `[localKey, sshAgent]`.
+2. SSH handshake begins; server's `PublicKeyCallback` is called with local key.
+3. Server checks in-memory authkeystore → **found** → auth succeeds immediately.
+4. No YubiKey touch needed. Session is established instantly.
+
+### Fallback (server restarted, key expired)
+
+1. Client offers local key → server's authkeystore is empty → rejected.
+2. Client falls back to SSH agent → YubiKey auth succeeds.
+3. Client re-registers local pubkey via `AUTHKEY` command.
+
+## Components
+
+### 1. Server: In-Memory Auth-Key Store
+
+**New file:** `internal/ssh/server/authkeystore.go`
+
+- Thread-safe store using `sync.RWMutex`.
+- Data structure: `map[string][]authKeyEntry` where key is username.
+- Each `authKeyEntry` holds `gossh.PublicKey` + `time.Time` (registered at).
+- Methods: `Add(user, pubkey)`, `Has(user, pubkey) bool`, `Remove(user, pubkey)`.
+- Per-user max key limit (default 5, configurable via `AuthKeyMaxPerUser`).
+- TTL-based expiry (default 24h, configurable via `AuthKeyTTLSeconds`).
+- Lazy expiry: check TTL on `Has()` calls; optionally a background reaper.
+- Package-level singleton or passed via dependency injection.
+
+### 2. Server: Extend PublicKeyCallback
+
+**Modified file:** `internal/ssh/server/publickeycallback.go`
+
+- Before the existing `authorizedKeysFile` lookup, check `authkeystore.Has(user, offeredPubKey)`.
+- If found → return success immediately (fast path).
+- If not found → fall through to existing file-based logic (no behaviour change).
+
+### 3. Server: AUTHKEY Command Handler
+
+**Modified file:** `internal/server/handlers/serverhandler.go` (or relevant handler)
+
+- Parse incoming line for `AUTHKEY <base64-pubkey>` prefix.
+- Decode the base64 public key using `gossh.ParsePublicKey()`.
+- Call `authkeystore.Add(user, pubkey)`.
+- Write `AUTHKEY OK\n` or `AUTHKEY ERR <reason>\n` back to the client.
+- Guard: only accept if `AuthKeyEnabled` is true in server config.
+
+### 4. Client: Auth Method Ordering (Multi-Method Support)
+
+**Modified file:** `internal/ssh/client/authmethods.go`
+
+- Change `initKnownHostsAuthMethods` to **collect multiple auth methods**
+  instead of returning after the first successful one.
+- Order: local private key first (from `--auth-key-path`, default `~/.ssh/id_rsa`),
+  then SSH agent, then other default keys.
+- This ensures Go's SSH client tries the fast key before the YubiKey.
+
+### 5. Client: Auth-Key Registration After Slow-Path Connection
+
+**Modified file:** `internal/clients/connectors/serverconnection.go` (or handler layer)
+
+- After session is established and DTail commands are sent, determine whether
+  the connection used the fast path or slow path.
+- If slow path (YubiKey was used): read the public key file
+  (`--auth-key-path` + `.pub`), send `AUTHKEY <base64-pubkey>` command.
+- Parse `AUTHKEY OK` / `AUTHKEY ERR` response.
+- A simple heuristic: if the auth-key-path private key exists and we have a
+  corresponding `.pub` file, always send the registration — sending it again is
+  idempotent and cheap.
+
+### 6. Configuration
+
+**Modified files:** `internal/config/server.go`, `internal/config/client.go`, `internal/config/args.go`
+
+Server config (`dtail.json`):
+- `AuthKeyEnabled` (bool, default `true`)
+- `AuthKeyTTLSeconds` (int, default `86400` = 24h)
+- `AuthKeyMaxPerUser` (int, default `5`)
+
+Client config / CLI flags:
+- `--auth-key-path` (string, default `~/.ssh/id_rsa`) — path to the local
+  private key to try first and whose `.pub` counterpart is registered
+- `--no-auth-key` (bool, default `false`) — disable auth-key feature entirely
+
+### 7. Integration Tests
+
+**Modified/new files in:** `integrationtests/`
+
+- Test that auth-key registration works end-to-end.
+- Test that fast-path auth succeeds after registration.
+- Test fallback when server has no cached key (simulating restart).
+- Test TTL expiry and max-keys-per-user limits.
+- Test `--no-auth-key` disables the feature.
+
+### 8. Documentation
+
+- Update `README.md` with auth-key feature description.
+- Update `AGENTS.md` / `CLAUDE.md` with new config options and architecture notes.
+
+## Security Considerations
+
+- **No server-side disk persistence** — keys exist only in memory, lost on restart.
+- **Trust chain** — auth-keys can only be registered over an already-authenticated
+  session. An attacker cannot register a key without first proving identity.
+- **TTL expiry** — keys auto-expire (default 24h), limiting exposure window.
+- **Per-user limits** — max 5 keys per user prevents memory exhaustion.
+- **Same security model as `~/.ssh/id_rsa`** — the local key is protected by
+  filesystem permissions (0600). If an attacker has access to `~/.ssh/id_rsa`,
+  they already have SSH access anyway.
+- **No new attack surface** — the `AUTHKEY` command is only processed inside an
+  authenticated session. The `PublicKeyCallback` fast-path is equivalent to
+  having the key in `authorized_keys`.
+
+## Implementation Order
+
+1. Auth-key store (server, standalone, unit-testable)
+2. Extend `PublicKeyCallback` (server, minimal change)
+3. `AUTHKEY` command handler (server handler)
+4. Client auth method ordering (multi-method collection)
+5. Client auth-key registration (send pubkey after slow-path)
+6. Configuration and CLI flags
+7. Integration tests
+8. Documentation
diff --git a/doc/index.md b/doc/index.md
index 565253b..c45569d 100644
--- a/doc/index.md
+++ b/doc/index.md
@@ -10,5 +10,6 @@ DTail Documentation
 ## Advanced topics
 
 * The [DTail Query Language](./querylanguage.md) is the starting point to dig deeper into DTail's own SQL-like mapreduce language for extraction/aggregation stats from log files.
+* The [Interactive Query Reload](../README.md#interactive-query-reload) section in the main README documents `:reload`, `:show`, `:help`, `:quit`, capability fallback on mixed-version servers, and session reuse semantics.
 * [Log Formats](./logformats.md) explains how to create your own custom log format for use with mapreduce queries.
 * Check out the [Testing Guide](./testing.md) for unit and integration testing.
diff --git a/doc/installation.md b/doc/installation.md
index 1f54050..c80e011 100644
--- a/doc/installation.md
+++ b/doc/installation.md
@@ -25,7 +25,17 @@ Set the `DTAIL_USE_ACL` environment variable before invoking the make command.
 % export DTAIL_USE_ACL=yes
 ```
 
-Alternatively, you could add `-tags linuxacl` to the Go compiler. 
+Alternatively, you could add `-tags linuxacl` to the Go compiler.
+
+## Build without zstd (optional)
+
+For targets where CGO-based zstd is unavailable (for example cross-compiling `dserver` for another architecture), build with the `nozstd` tag. Compressed `.zst` log files will not be supported in that binary.
+
+```console
+% export DTAIL_NO_ZSTD=yes
+```
+
+This sets `-tags nozstd` via the Makefile. Plain `go build` users can pass `-tags nozstd` directly.
 
 # Install it
 
@@ -61,15 +71,28 @@ uid=1001(dserver) 1001=670(dserver) groups=1001(dserver)
     sudo tee /etc/dserver/dtail.json
 ```
 
+### SSH listen address (``SSHBindAddress``)
+
+The example config sets ``Server.SSHBindAddress`` to ``0.0.0.0``, so dserver listens on **every** local IPv4 address, including your LAN (e.g. ``192.168.1.x`` on eth0) and any other interface (loopback, WireGuard, etc.). Clients reach it as ``<that-host-LAN-IP>:2222``; you do **not** need to change this for normal LAN access.
+
+To listen **only** on a specific address—for example only the home LAN and not on a VPN—set ``SSHBindAddress`` in ``/etc/dserver/dtail.json`` to **that machine’s** address (each host needs its own value), e.g. ``192.168.1.125`` on ``pi0``, ``192.168.1.126`` on ``pi1``. Alternatively, override from the command line (after ``-cfg``): ``dserver -cfg /etc/dserver/dtail.json -bindAddress 192.168.1.125``. Then reload or restart dserver.
+
 5. It is recommended to configure DTail server as a service to ``systemd``. An example unit file for ``systemd`` can be found [here](../examples/dserver.service.example).
 
 ```console
 % curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver.service.example |
     sudo tee /etc/systemd/system/dserver.service
 % sudo systemctl daemon-reload
-% sudo systemctl enable dserver
 ```
 
+The unit is intended to stay **disabled** until you opt in. Start DTail server manually when needed:
+
+```console
+% sudo systemctl start dserver
+```
+
+To start it automatically at boot, run once: `sudo systemctl enable dserver`.
+
 # Start it
 
 To start the DTail server via ``systemd`` run:
@@ -93,6 +116,20 @@ To start the DTail server via ``systemd`` run:
     Dec 06 13:21:24 serv-001.lan.example.org dserver[12296]: SERVER|serv-001|INFO|Binding server|1.2.3.4:2222
 ```
 
+### Firewall (firewalld on RHEL, Rocky Linux, Fedora, …)
+
+The DTail server listens on TCP port ``2222`` (see ``SSHPort`` in ``dtail.json``). **ICMP (ping) may work while TCP to 2222 is blocked**, because host firewalls often allow ping but not arbitrary ports.
+
+If ``firewalld`` is active, allow the DTail port permanently and reload:
+
+```console
+% sudo firewall-cmd --permanent --add-port=2222/tcp
+% sudo firewall-cmd --reload
+% sudo firewall-cmd --list-ports
+```
+
+Clients may report ``dial tcp …: connect: no route to host`` when the firewall rejects the connection with an ICMP unreachable—opening ``2222/tcp`` fixes that. For other firewalls (nftables, ufw, …), add an equivalent allow rule for ``2222/tcp``. A small helper script is [firewalld-dserver-port.sh.example](../examples/firewalld-dserver-port.sh.example).
+
 # Register SSH public keys in DTail server
 
 The DTail server now runs as a ``systemd`` service under system user ``dserver``. However, the system user ``dserver`` has no permissions to read the SSH public keys from ``/home/USER/.ssh/authorized_keys``. Therefore, no user would be able to establish an SSH session to DTail server. As an alternative path DTail server also checks for public SSH key files in ``/var/run/dserver/cache/USER.authorized_keys``.
@@ -113,6 +150,24 @@ It is recommended to execute [update_key_cache.sh](../examples/update_key_cache.
 % sudo systemctl start dserver-update-keycache.timer
 ```
 
+# Prune old dserver log files
+
+Log files live under ``/var/run/dserver/log`` (see ``LogDir`` in ``dtail.json``). To remove ``*.log`` files **older than seven days**, install [prune_dserver_logs.sh](../examples/prune_dserver_logs.sh.example) and a systemd timer (runs daily with a randomized delay):
+
+```console
+% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/prune_dserver_logs.sh.example |
+    sudo tee /var/run/dserver/prune_dserver_logs.sh
+% sudo chmod 755 /var/run/dserver/prune_dserver_logs.sh
+% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver-prune-logs.service.example |
+    sudo tee /etc/systemd/system/dserver-prune-logs.service
+% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver-prune-logs.timer.example |
+    sudo tee /etc/systemd/system/dserver-prune-logs.timer
+% sudo systemctl daemon-reload
+% sudo systemctl enable --now dserver-prune-logs.timer
+```
+
+The script uses ``find /var/run/dserver/log -type f -name '*.log' -mtime +7 -delete``.
+
 # Run DTail client
 
 Now you should be able to use DTail client like outlined in the [Quick Starting Guide](quickstart.md). Also, have a look at the [Examples](examples.md).
diff --git a/doc/logformats.md b/doc/logformats.md
index dbf2051..29e2559 100644
--- a/doc/logformats.md
+++ b/doc/logformats.md
@@ -42,7 +42,7 @@ func newGenericKVParser(hostname, timeZoneName string, timeZoneOffset int) (*gen
 	return &genericKVParser{defaultParser: *defaultParser}, nil
 }
 
-func (p *genericKVParser) MakeFields(maprLine string) (map[string]string, error) {
+func (p *genericKVParser) MakeFields(maprLine, _ string) (map[string]string, error) {
 	splitted := strings.Split(maprLine, protocol.FieldDelimiter)
 	fields := make(map[string]string, len(splitted))
 
@@ -70,6 +70,7 @@ func (p *genericKVParser) MakeFields(maprLine string) (map[string]string, error)
 ... whereas:
 
 * `maprLine` is the whole raw log line to be parsed by the log format.
+* `sourceID` is the stable identifier of the log file / stream the line came from. Stateful parsers (e.g. CSV with a header row per file) should key their per-file state by this value; stateless parsers may ignore it.
 * `protocol.FieldDelimiter` is the field delimiter used by the log format, here: `|`.
 * All field names starting with `$` are variables. They store some custom values.
 * All other fields are bareword-fields and are extracted from the log lines directly, e.g. `field1=value1|field2=value2|...`
@@ -139,7 +140,7 @@ func newFooParser(hostname, timeZoneName string, timeZoneOffset int) (*fooParser
 	return &fooParser{defaultParser: *defaultParser}, nil
 }
 
-func (p *fooParser) MakeFields(maprLine string) (map[string]string, error) {
+func (p *fooParser) MakeFields(maprLine, sourceID string) (map[string]string, error) {
 	fields := make(map[string]string, 3)
 
 	..
diff --git a/doc/performance_optimization_summary.md b/doc/performance_optimization_summary.md
new file mode 100644
index 0000000..e54d9a9
--- /dev/null
+++ b/doc/performance_optimization_summary.md
@@ -0,0 +1,69 @@
+# DTail Performance Optimization Summary
+
+> **Historical note:** This is a point-in-time record of a specific optimization
+> effort (trace-logging and buffering fixes) with the measurements taken at that
+> time; the numbers and file paths (e.g. `turbo_writer.go`, since renamed to
+> `line_writer.go`) are preserved as-is. The "turbo mode" it compares against a
+> "non-turbo mode" is now the single, default read/output path — there is no mode
+> split any more, and the `DTAIL_TURBOBOOST_DISABLE` toggle has been removed.
+
+## Changes Made
+
+### 1. Optimized Trace Logging (`/internal/io/dlog/dlog.go`)
+
+**Problem**: The `Trace()` and `Devel()` functions were calling `runtime.Caller(1)` for every invocation, even when trace logging was disabled. This was causing ~497ns overhead per call.
+
+**Solution**: Added early level checks before the expensive `runtime.Caller()` operation:
+
+```go
+func (d *DLog) Trace(args ...interface{}) string {
+    // Early check to avoid expensive runtime.Caller when trace is disabled
+    if d.maxLevel < Trace {
+        return ""
+    }
+    // ... rest of function
+}
+```
+
+### 2. Improved Buffering in Turbo Mode (`/internal/server/handlers/turbo_writer.go`)
+
+**Problem**: Turbo mode was forcing immediate flush after every line in serverless mode, defeating the purpose of buffering.
+
+**Solution**: Removed the immediate flush condition for serverless mode, allowing proper buffering:
+
+```go
+// Changed from:
+if w.writeBuf.Len() >= w.bufSize || w.serverless {
+    return w.flushBuffer()
+}
+
+// To:
+if w.writeBuf.Len() >= w.bufSize {
+    return w.flushBuffer()
+}
+```
+
+## Performance Results
+
+### Before Optimization
+- Turbo mode was **3-5x slower** than non-turbo mode
+- DCat (10MB): 678ms (turbo) vs 210ms (non-turbo)
+- DGrep (10MB): 570ms (turbo) vs 96ms (non-turbo)
+
+### After Optimization
+- Turbo mode is now **2.87x faster** than non-turbo mode
+- DCat (1M lines): 0.66s (turbo) vs 1.89s (non-turbo) - **65% improvement**
+- DCat with colors: 1.69s (turbo) vs 2.24s (non-turbo) - **24% improvement**
+
+## Verification
+- ✅ All unit tests pass
+- ✅ All integration tests pass
+- ✅ No functionality regression
+- ✅ Backward compatible
+
+## Files Modified
+1. `/internal/io/dlog/dlog.go` - Lines 196-216
+2. `/internal/server/handlers/turbo_writer.go` - Lines 104-108
+
+## Key Takeaway
+The trace logging overhead was the primary bottleneck, causing DTail to spend more time logging than processing data. By adding simple level checks before expensive operations, we achieved a ~3x performance improvement in turbo mode.
\ No newline at end of file
diff --git a/doc/pgo_commands_detail.md b/doc/pgo_commands_detail.md
new file mode 100644
index 0000000..e874ba8
--- /dev/null
+++ b/doc/pgo_commands_detail.md
@@ -0,0 +1,216 @@
+# PGO Command Execution Details
+
+This document shows the exact commands executed during Profile-Guided Optimization (PGO) generation for DTail tools.
+
+## Overview
+
+When running `make pgo-generate` or `dtail-tools pgo`, the following commands are executed to generate performance profiles for each tool.
+
+## Commands Executed
+
+### 1. Building Baseline Binaries
+
+```bash
+go build -o pgo-build/dtail-baseline ./cmd/dtail
+go build -o pgo-build/dcat-baseline ./cmd/dcat
+go build -o pgo-build/dgrep-baseline ./cmd/dgrep
+go build -o pgo-build/dmap-baseline ./cmd/dmap
+go build -o pgo-build/dserver-baseline ./cmd/dserver
+```
+
+### 2. Profile Generation Commands
+
+#### DTail Profile Generation
+```bash
+# Background log writer process
+bash -c "for i in {1..200}; do 
+  level=$((i % 4))
+  case $level in 
+    0) lvl=INFO;; 
+    1) lvl=WARN;; 
+    2) lvl=ERROR;; 
+    3) lvl=DEBUG;; 
+  esac
+  echo \"[2025-07-04 15:00:00] $lvl - Test log line number $i with some additional text to process\" >> growing.log
+  sleep 0.015
+done"
+
+# DTail command
+pgo-build/dtail-baseline \
+  -cfg none \
+  -plain \
+  -profile \
+  -profiledir pgo-profiles/iter_dtail_TIMESTAMP \
+  -regex "ERROR|WARN" \
+  -shutdownAfter 3 \
+  pgo-profiles/growing.log
+```
+
+#### DCat Profile Generation
+```bash
+pgo-build/dcat-baseline \
+  -cfg none \
+  -plain \
+  -profile \
+  -profiledir pgo-profiles/iter_dcat_TIMESTAMP \
+  pgo-profiles/test.log
+```
+
+#### DGrep Profile Generation
+```bash
+pgo-build/dgrep-baseline \
+  -cfg none \
+  -plain \
+  -profile \
+  -profiledir pgo-profiles/iter_dgrep_TIMESTAMP \
+  -regex "ERROR|WARN" \
+  pgo-profiles/test.log
+```
+
+#### DMap Profile Generation
+```bash
+pgo-build/dmap-baseline \
+  -cfg none \
+  -plain \
+  -profile \
+  -profiledir pgo-profiles/iter_dmap_TIMESTAMP \
+  -files pgo-profiles/test.csv \
+  -query "select status, count(*) group by status"
+```
+
+#### DServer Profile Generation
+```bash
+# Start dserver with pprof endpoint
+pgo-build/dserver-baseline \
+  -cfg none \
+  -pprof localhost:16060 \
+  -port 12222
+
+# Client commands to generate server load (run concurrently):
+pgo-build/dcat-baseline \
+  -cfg none \
+  -server localhost:12222 \
+  pgo-profiles/test.log
+
+pgo-build/dgrep-baseline \
+  -cfg none \
+  -server localhost:12222 \
+  -regex "ERROR|WARN" \
+  pgo-profiles/test.log
+
+pgo-build/dgrep-baseline \
+  -cfg none \
+  -server localhost:12222 \
+  -regex "INFO.*action" \
+  pgo-profiles/test.log
+
+pgo-build/dmap-baseline \
+  -cfg none \
+  -server localhost:12222 \
+  -files pgo-profiles/test.csv \
+  -query "select status, count(*) group by status"
+
+pgo-build/dmap-baseline \
+  -cfg none \
+  -server localhost:12222 \
+  -files pgo-profiles/test.csv \
+  -query "select department, avg(salary) group by department"
+
+# Capture CPU profile via HTTP
+curl http://localhost:16060/debug/pprof/profile?seconds=5 > dserver.pprof
+```
+
+### 3. Profile Merging
+
+When multiple iterations are run, profiles are merged:
+
+```bash
+# Merge multiple profile iterations
+go tool pprof -proto \
+  pgo-profiles/dcat.pprof.0.pprof \
+  pgo-profiles/dcat.pprof.1.pprof \
+  > pgo-profiles/dcat.pprof
+```
+
+### 4. Building with PGO
+
+```bash
+# Build optimized binaries using profiles
+go build -pgo=pgo-profiles/dcat.pprof -o pgo-build/dcat ./cmd/dcat
+go build -pgo=pgo-profiles/dgrep.pprof -o pgo-build/dgrep ./cmd/dgrep
+go build -pgo=pgo-profiles/dmap.pprof -o pgo-build/dmap ./cmd/dmap
+go build -pgo=pgo-profiles/dtail.pprof -o pgo-build/dtail ./cmd/dtail
+go build -pgo=pgo-profiles/dserver.pprof -o pgo-build/dserver ./cmd/dserver
+```
+
+### 5. Performance Comparison Commands
+
+Quick benchmarks are run to compare baseline vs optimized:
+
+```bash
+# Baseline benchmark
+pgo-build/dcat-baseline -cfg none -plain /tmp/pgo_bench.log
+
+# Optimized benchmark
+pgo-build/dcat -cfg none -plain /tmp/pgo_bench.log
+
+# Similar commands for dgrep and dmap
+pgo-build/dgrep-baseline -cfg none -plain -regex ERROR /tmp/pgo_bench.log
+pgo-build/dmap-baseline -cfg none -plain -files /tmp/pgo_bench.csv -query "select count(*)"
+```
+
+## Test Data Generation
+
+The PGO framework generates realistic test data:
+
+### Log File (test.log)
+- Contains timestamps, log levels (INFO, WARN, ERROR, DEBUG)
+- Includes user actions, durations, and status information
+- Default size: 1,000,000 lines (configurable with -datasize)
+
+### CSV File (test.csv)
+- Contains employee data with departments, salaries, status
+- Used for MapReduce queries
+- Default size: 100,000 rows (1/10 of log file size)
+
+### Growing Log File (growing.log)
+- Used specifically for dtail testing
+- Simulates real-time log generation
+- Writes ~200 lines over 3 seconds with mixed log levels
+
+## Customization Options
+
+### Adjust Test Data Size
+```bash
+dtail-tools pgo -datasize 5000000  # 5 million lines
+```
+
+### Run More Iterations
+```bash
+dtail-tools pgo -iterations 5  # Run 5 iterations per command
+```
+
+### Profile Specific Commands Only
+```bash
+dtail-tools pgo dcat dgrep  # Only optimize dcat and dgrep
+```
+
+### Verbose Output
+```bash
+dtail-tools pgo -v  # Show all command execution details
+```
+
+### Profile Generation Only
+```bash
+dtail-tools pgo -profileonly  # Skip building optimized binaries
+```
+
+## Notes
+
+1. **Empty Profiles**: Some commands (like dtail) may generate empty profiles if they are I/O-bound. This is handled gracefully.
+
+2. **DServer Profiling**: Uses HTTP pprof endpoint instead of command-line profiling to capture server-side performance data.
+
+3. **Concurrent Execution**: Multiple client commands are run concurrently against dserver to generate realistic load patterns.
+
+4. **Profile Quality**: The effectiveness of PGO depends on how well the test workload represents real-world usage patterns.
\ No newline at end of file
diff --git a/doc/pgo_implementation.md b/doc/pgo_implementation.md
new file mode 100644
index 0000000..b404668
--- /dev/null
+++ b/doc/pgo_implementation.md
@@ -0,0 +1,173 @@
+# Profile-Guided Optimization (PGO) Implementation for DTail
+
+## Overview
+
+This document describes the Profile-Guided Optimization (PGO) implementation for DTail tools. PGO is a compiler optimization technique that uses runtime profiling data to guide optimization decisions, resulting in better performance for real-world usage patterns.
+
+## Implementation Details
+
+### Architecture
+
+The PGO implementation is integrated into the dtail-tools command as a subcommand:
+
+```bash
+dtail-tools pgo [options] [commands...]
+```
+
+### Core Components
+
+1. **PGO Module** (`internal/tools/pgo/pgo.go`)
+   - Handles the complete PGO workflow
+   - Manages profile generation, merging, and PGO builds
+   - Provides performance comparison
+
+2. **Profiling Integration**
+   - All dtail commands now support the `-profile` flag
+   - dserver uses HTTP pprof endpoint for profiling
+   - Profiles are generated during realistic workloads
+
+3. **Makefile Integration**
+   - `make pgo` - Complete PGO workflow
+   - `make pgo-quick` - Quick PGO with smaller datasets
+   - `make pgo-generate` - Generate profiles only
+   - `make build-pgo` - Build with existing profiles
+   - `make install-pgo` - Install PGO-optimized binaries
+
+### Workflow
+
+1. **Build Baseline Binaries**: Standard Go builds without PGO
+2. **Generate Profiles**: Run workloads to collect CPU profiles
+3. **Merge Profiles**: Combine multiple profile iterations
+4. **Build with PGO**: Use profiles to guide optimization
+5. **Compare Performance**: Measure improvement
+
+### Profile Generation Details
+
+Each command has specific workloads designed to exercise common code paths:
+
+- **dcat**: Reading large log files
+- **dgrep**: Pattern matching with various regex patterns
+- **dmap**: MapReduce queries on CSV data
+- **dtail**: Following growing log files with filtering
+- **dserver**: Handling concurrent client connections
+
+### Special Handling
+
+1. **Empty Profiles**: I/O-bound operations may generate empty profiles. The implementation handles this gracefully by creating empty profile files that allow the workflow to continue.
+
+2. **dserver Profiling**: Uses HTTP pprof endpoint instead of command-line flags, allowing profile capture during server operation.
+
+3. **dtail Workload**: Simulates a growing log file with various log levels to exercise the tail functionality.
+
+## Performance Results
+
+Based on testing with PGO optimization:
+
+### Individual Command Improvements
+- **dcat**: 3.75-5.40% improvement
+- **dgrep**: Up to 19% improvement (varies by pattern hit rate)
+- **dmap**: Up to 39% improvement for specific queries
+
+### Overall Performance Progression
+From the original pre-optimization baseline to the current PGO-optimized build
+(the channel-less read/output path, formerly called "turbo", is now the default
+and only path):
+- **dcat**: 14-21x faster overall
+- **dgrep**: 9-15x faster overall
+- **dmap**: 9-29% faster overall
+
+## Usage Examples
+
+### Generate PGO-Optimized Binaries
+```bash
+# Full PGO workflow
+make pgo
+
+# Quick PGO with smaller datasets
+make pgo-quick
+
+# Generate profiles only
+make pgo-generate
+
+# Build with existing profiles
+make build-pgo
+```
+
+### Using dtail-tools Directly
+```bash
+# Optimize all commands
+dtail-tools pgo
+
+# Optimize specific commands
+dtail-tools pgo dcat dgrep
+
+# Verbose mode with custom iterations
+dtail-tools pgo -v -iterations 5
+
+# Generate profiles only
+dtail-tools pgo -profileonly
+```
+
+### Custom PGO Options
+```bash
+# Custom data size
+dtail-tools pgo -datasize 5000000
+
+# Custom profile directory
+dtail-tools pgo -profiledir my-profiles
+
+# Custom output directory
+dtail-tools pgo -outdir my-pgo-build
+```
+
+## Technical Considerations
+
+1. **Profile Quality**: The quality of PGO optimization depends on how representative the profiling workload is of real-world usage.
+
+2. **Binary Size**: PGO-optimized binaries may be slightly larger due to function cloning and inlining decisions.
+
+3. **Build Time**: Building with PGO takes longer than standard builds due to profile processing.
+
+4. **Go Version**: PGO requires Go 1.20 or later.
+
+## Integration with CI/CD
+
+To integrate PGO into your build pipeline:
+
+1. Generate profiles periodically with production-like workloads
+2. Store profiles in version control or artifact repository
+3. Use `make build-pgo` in your build process
+4. Monitor performance metrics to validate improvements
+
+## Profile Files
+
+Profile files are stored in the `pgo-profiles/` directory:
+- `dcat.pprof` - DCat CPU profile
+- `dgrep.pprof` - DGrep CPU profile
+- `dmap.pprof` - DMap CPU profile
+- `dtail.pprof` - DTail CPU profile (may be empty for I/O-bound operations)
+- `dserver.pprof` - DServer CPU profile
+
+## Troubleshooting
+
+### Empty Profiles
+Some commands may generate empty profiles if they are I/O-bound. This is normal and the PGO workflow handles it gracefully.
+
+### Profile Merge Failures
+If profile merging fails, check that:
+- All profile files are valid
+- Go tools are properly installed
+- Sufficient disk space is available
+
+### Performance Not Improving
+If PGO doesn't show improvement:
+- Ensure profiles represent real workloads
+- Check that the profile has sufficient samples
+- Verify the correct profile is being used during build
+
+## Future Enhancements
+
+1. **Automated Profile Collection**: Collect profiles from production deployments
+2. **Profile Versioning**: Track profile versions with code changes
+3. **Multi-Architecture Support**: Generate architecture-specific profiles
+4. **Continuous Profiling**: Regular profile updates based on usage patterns
\ No newline at end of file
diff --git a/doc/profiling.md b/doc/profiling.md
new file mode 100644
index 0000000..7925fb3
--- /dev/null
+++ b/doc/profiling.md
@@ -0,0 +1,376 @@
+# DTail Profiling Framewo