diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/asciinema/README.md | 41 | ||||
| -rw-r--r-- | doc/auth-key-fast-reconnect.md | 200 | ||||
| -rw-r--r-- | doc/index.md | 1 | ||||
| -rw-r--r-- | doc/installation.md | 59 | ||||
| -rw-r--r-- | doc/logformats.md | 5 | ||||
| -rw-r--r-- | doc/performance_optimization_summary.md | 69 | ||||
| -rw-r--r-- | doc/pgo_commands_detail.md | 216 | ||||
| -rw-r--r-- | doc/pgo_implementation.md | 173 | ||||
| -rw-r--r-- | doc/profiling.md | 376 | ||||
| -rw-r--r-- | doc/querylanguage.md | 40 | ||||
| -rw-r--r-- | doc/refactoring_guide.md | 240 | ||||
| -rw-r--r-- | doc/turbo_performance_analysis.md | 104 | ||||
| -rw-r--r-- | doc/turboboost_optimization.md | 117 |
13 files changed, 1595 insertions, 46 deletions
diff --git a/doc/asciinema/README.md b/doc/asciinema/README.md deleted file mode 100644 index 1eb1a2e..0000000 --- a/doc/asciinema/README.md +++ /dev/null @@ -1,41 +0,0 @@ -asciinema -========= - -The animated Gifs you find in the DTail docs were created using: - -* [asciinema](https://asciinema.org) -* [asciicast2gif](https://github.com/asciinema/asciicast2gif) - -On Fedora Linux 35. - -## Installing prerequisites - -On Fedora Linux 35 install `asciinema`: - -```shell -% sudo dnf install -y asciinema -``` - -and `asciicast2gif` (for simplicity, the Docker image was used): - -```shell -% docker pull asciinema/asciicast2gif -``` - -This of course assumes that Docker is up and running on your machine (out of scope for this documentation). - -## Record a shell session - -This is as simple as running - -```shell -% asciinema rec recording.json -``` - -This will launch a sub-shell to be recorded. Once done, exit the sub-shell with `exit`. - -## Convert the recording to a gif - -```shell -% docker run --rm -v $PWD:/data asciinema/asciicast2gif -t tango -s 2 recording.json recording.gif -``` diff --git a/doc/auth-key-fast-reconnect.md b/doc/auth-key-fast-reconnect.md new file mode 100644 index 0000000..cb9884a --- /dev/null +++ b/doc/auth-key-fast-reconnect.md @@ -0,0 +1,200 @@ +# Auth-Key Fast-Reconnect for DTail + +## Problem + +When using a YubiKey for SSH authentication, each DTail connection requires a +physical touch of the YubiKey during the SSH handshake. This is slow and becomes +painful when connecting to many servers concurrently — the YubiKey serialises +all signing requests, turning parallel connections into sequential ones. + +## Solution + +Allow the DTail client to register a local SSH public key with the DTail server +over an already-authenticated SSH session. The server caches this key +**in-memory only** (never written to disk). On subsequent connections the client +offers that local key first — a pure in-memory RSA verify with no YubiKey +interaction — and falls back to the original auth method if the server does not +recognise the key. + +## Design Principles + +1. **Transparent fallback** — Go's `golang.org/x/crypto/ssh` tries each + `AuthMethod` in order; if the fast key is rejected the client silently falls + back to the SSH agent / YubiKey. No user interaction required. +2. **Server keys are ephemeral** — the in-memory store is lost on server + restart. No file I/O, no persistence. +3. **Trust chain preserved** — an auth-key can only be registered over a session + that was already authenticated via the normal (YubiKey) path. +4. **Minimal protocol addition** — a single `AUTHKEY <base64-pubkey>` command + sent over the existing SSH session text protocol. + +## Architecture Overview + +``` +┌─────────────────────────────────────────────────────────┐ +│ DTail Client │ +│ │ +│ Auth methods (tried in order): │ +│ 1. Local private key (~/.ssh/id_rsa) ← FAST │ +│ 2. SSH Agent / YubiKey ← SLOW fallback │ +│ │ +│ After slow-path auth: │ +│ → sends AUTHKEY <~/.ssh/id_rsa.pub> to server │ +└────────────────────────┬────────────────────────────────┘ + │ SSH +┌────────────────────────▼────────────────────────────────┐ +│ DTail Server (dserver) │ +│ │ +│ PublicKeyCallback: │ +│ 1. Check in-memory authkeystore ← FAST │ +│ 2. Check authorized_keys file ← existing path │ +│ │ +│ AUTHKEY command handler: │ +│ → authkeystore.Add(user, pubkey) │ +│ → responds AUTHKEY OK / AUTHKEY ERR │ +│ │ +│ authkeystore (in-memory only): │ +│ map[username] → []PublicKey (with TTL, max per user) │ +└─────────────────────────────────────────────────────────┘ +``` + +## Sequence of Events + +### First Connection (slow path — YubiKey) + +1. Client checks for local private key at `~/.ssh/id_rsa` (or `--auth-key-path`). +2. Client builds auth methods list: `[localKey, sshAgent]`. +3. SSH handshake begins; server's `PublicKeyCallback` is called with local key. +4. Server checks in-memory authkeystore → not found. +5. Server checks `authorized_keys` file → not found (this key isn't in there). +6. Server rejects the key. +7. Go SSH client automatically tries next auth method: SSH agent (YubiKey). +8. YubiKey signs the challenge; server finds the YubiKey pubkey in + `authorized_keys` → auth succeeds. +9. Session is established; client sends DTail commands as usual. +10. Client reads `~/.ssh/id_rsa.pub` and sends `AUTHKEY <base64-pubkey>`. +11. Server's handler parses the command, calls `authkeystore.Add(user, pubkey)`. +12. Server responds `AUTHKEY OK`. + +### Subsequent Connections (fast path — no YubiKey) + +1. Client builds auth methods list: `[localKey, sshAgent]`. +2. SSH handshake begins; server's `PublicKeyCallback` is called with local key. +3. Server checks in-memory authkeystore → **found** → auth succeeds immediately. +4. No YubiKey touch needed. Session is established instantly. + +### Fallback (server restarted, key expired) + +1. Client offers local key → server's authkeystore is empty → rejected. +2. Client falls back to SSH agent → YubiKey auth succeeds. +3. Client re-registers local pubkey via `AUTHKEY` command. + +## Components + +### 1. Server: In-Memory Auth-Key Store + +**New file:** `internal/ssh/server/authkeystore.go` + +- Thread-safe store using `sync.RWMutex`. +- Data structure: `map[string][]authKeyEntry` where key is username. +- Each `authKeyEntry` holds `gossh.PublicKey` + `time.Time` (registered at). +- Methods: `Add(user, pubkey)`, `Has(user, pubkey) bool`, `Remove(user, pubkey)`. +- Per-user max key limit (default 5, configurable via `AuthKeyMaxPerUser`). +- TTL-based expiry (default 24h, configurable via `AuthKeyTTLSeconds`). +- Lazy expiry: check TTL on `Has()` calls; optionally a background reaper. +- Package-level singleton or passed via dependency injection. + +### 2. Server: Extend PublicKeyCallback + +**Modified file:** `internal/ssh/server/publickeycallback.go` + +- Before the existing `authorizedKeysFile` lookup, check `authkeystore.Has(user, offeredPubKey)`. +- If found → return success immediately (fast path). +- If not found → fall through to existing file-based logic (no behaviour change). + +### 3. Server: AUTHKEY Command Handler + +**Modified file:** `internal/server/handlers/serverhandler.go` (or relevant handler) + +- Parse incoming line for `AUTHKEY <base64-pubkey>` prefix. +- Decode the base64 public key using `gossh.ParsePublicKey()`. +- Call `authkeystore.Add(user, pubkey)`. +- Write `AUTHKEY OK\n` or `AUTHKEY ERR <reason>\n` back to the client. +- Guard: only accept if `AuthKeyEnabled` is true in server config. + +### 4. Client: Auth Method Ordering (Multi-Method Support) + +**Modified file:** `internal/ssh/client/authmethods.go` + +- Change `initKnownHostsAuthMethods` to **collect multiple auth methods** + instead of returning after the first successful one. +- Order: local private key first (from `--auth-key-path`, default `~/.ssh/id_rsa`), + then SSH agent, then other default keys. +- This ensures Go's SSH client tries the fast key before the YubiKey. + +### 5. Client: Auth-Key Registration After Slow-Path Connection + +**Modified file:** `internal/clients/connectors/serverconnection.go` (or handler layer) + +- After session is established and DTail commands are sent, determine whether + the connection used the fast path or slow path. +- If slow path (YubiKey was used): read the public key file + (`--auth-key-path` + `.pub`), send `AUTHKEY <base64-pubkey>` command. +- Parse `AUTHKEY OK` / `AUTHKEY ERR` response. +- A simple heuristic: if the auth-key-path private key exists and we have a + corresponding `.pub` file, always send the registration — sending it again is + idempotent and cheap. + +### 6. Configuration + +**Modified files:** `internal/config/server.go`, `internal/config/client.go`, `internal/config/args.go` + +Server config (`dtail.json`): +- `AuthKeyEnabled` (bool, default `true`) +- `AuthKeyTTLSeconds` (int, default `86400` = 24h) +- `AuthKeyMaxPerUser` (int, default `5`) + +Client config / CLI flags: +- `--auth-key-path` (string, default `~/.ssh/id_rsa`) — path to the local + private key to try first and whose `.pub` counterpart is registered +- `--no-auth-key` (bool, default `false`) — disable auth-key feature entirely + +### 7. Integration Tests + +**Modified/new files in:** `integrationtests/` + +- Test that auth-key registration works end-to-end. +- Test that fast-path auth succeeds after registration. +- Test fallback when server has no cached key (simulating restart). +- Test TTL expiry and max-keys-per-user limits. +- Test `--no-auth-key` disables the feature. + +### 8. Documentation + +- Update `README.md` with auth-key feature description. +- Update `AGENTS.md` / `CLAUDE.md` with new config options and architecture notes. + +## Security Considerations + +- **No server-side disk persistence** — keys exist only in memory, lost on restart. +- **Trust chain** — auth-keys can only be registered over an already-authenticated + session. An attacker cannot register a key without first proving identity. +- **TTL expiry** — keys auto-expire (default 24h), limiting exposure window. +- **Per-user limits** — max 5 keys per user prevents memory exhaustion. +- **Same security model as `~/.ssh/id_rsa`** — the local key is protected by + filesystem permissions (0600). If an attacker has access to `~/.ssh/id_rsa`, + they already have SSH access anyway. +- **No new attack surface** — the `AUTHKEY` command is only processed inside an + authenticated session. The `PublicKeyCallback` fast-path is equivalent to + having the key in `authorized_keys`. + +## Implementation Order + +1. Auth-key store (server, standalone, unit-testable) +2. Extend `PublicKeyCallback` (server, minimal change) +3. `AUTHKEY` command handler (server handler) +4. Client auth method ordering (multi-method collection) +5. Client auth-key registration (send pubkey after slow-path) +6. Configuration and CLI flags +7. Integration tests +8. Documentation diff --git a/doc/index.md b/doc/index.md index 565253b..c45569d 100644 --- a/doc/index.md +++ b/doc/index.md @@ -10,5 +10,6 @@ DTail Documentation ## Advanced topics * The [DTail Query Language](./querylanguage.md) is the starting point to dig deeper into DTail's own SQL-like mapreduce language for extraction/aggregation stats from log files. +* The [Interactive Query Reload](../README.md#interactive-query-reload) section in the main README documents `:reload`, `:show`, `:help`, `:quit`, capability fallback on mixed-version servers, and session reuse semantics. * [Log Formats](./logformats.md) explains how to create your own custom log format for use with mapreduce queries. * Check out the [Testing Guide](./testing.md) for unit and integration testing. diff --git a/doc/installation.md b/doc/installation.md index 1f54050..c80e011 100644 --- a/doc/installation.md +++ b/doc/installation.md @@ -25,7 +25,17 @@ Set the `DTAIL_USE_ACL` environment variable before invoking the make command. % export DTAIL_USE_ACL=yes ``` -Alternatively, you could add `-tags linuxacl` to the Go compiler. +Alternatively, you could add `-tags linuxacl` to the Go compiler. + +## Build without zstd (optional) + +For targets where CGO-based zstd is unavailable (for example cross-compiling `dserver` for another architecture), build with the `nozstd` tag. Compressed `.zst` log files will not be supported in that binary. + +```console +% export DTAIL_NO_ZSTD=yes +``` + +This sets `-tags nozstd` via the Makefile. Plain `go build` users can pass `-tags nozstd` directly. # Install it @@ -61,15 +71,28 @@ uid=1001(dserver) 1001=670(dserver) groups=1001(dserver) sudo tee /etc/dserver/dtail.json ``` +### SSH listen address (``SSHBindAddress``) + +The example config sets ``Server.SSHBindAddress`` to ``0.0.0.0``, so dserver listens on **every** local IPv4 address, including your LAN (e.g. ``192.168.1.x`` on eth0) and any other interface (loopback, WireGuard, etc.). Clients reach it as ``<that-host-LAN-IP>:2222``; you do **not** need to change this for normal LAN access. + +To listen **only** on a specific address—for example only the home LAN and not on a VPN—set ``SSHBindAddress`` in ``/etc/dserver/dtail.json`` to **that machine’s** address (each host needs its own value), e.g. ``192.168.1.125`` on ``pi0``, ``192.168.1.126`` on ``pi1``. Alternatively, override from the command line (after ``-cfg``): ``dserver -cfg /etc/dserver/dtail.json -bindAddress 192.168.1.125``. Then reload or restart dserver. + 5. It is recommended to configure DTail server as a service to ``systemd``. An example unit file for ``systemd`` can be found [here](../examples/dserver.service.example). ```console % curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver.service.example | sudo tee /etc/systemd/system/dserver.service % sudo systemctl daemon-reload -% sudo systemctl enable dserver ``` +The unit is intended to stay **disabled** until you opt in. Start DTail server manually when needed: + +```console +% sudo systemctl start dserver +``` + +To start it automatically at boot, run once: `sudo systemctl enable dserver`. + # Start it To start the DTail server via ``systemd`` run: @@ -93,6 +116,20 @@ To start the DTail server via ``systemd`` run: Dec 06 13:21:24 serv-001.lan.example.org dserver[12296]: SERVER|serv-001|INFO|Binding server|1.2.3.4:2222 ``` +### Firewall (firewalld on RHEL, Rocky Linux, Fedora, …) + +The DTail server listens on TCP port ``2222`` (see ``SSHPort`` in ``dtail.json``). **ICMP (ping) may work while TCP to 2222 is blocked**, because host firewalls often allow ping but not arbitrary ports. + +If ``firewalld`` is active, allow the DTail port permanently and reload: + +```console +% sudo firewall-cmd --permanent --add-port=2222/tcp +% sudo firewall-cmd --reload +% sudo firewall-cmd --list-ports +``` + +Clients may report ``dial tcp …: connect: no route to host`` when the firewall rejects the connection with an ICMP unreachable—opening ``2222/tcp`` fixes that. For other firewalls (nftables, ufw, …), add an equivalent allow rule for ``2222/tcp``. A small helper script is [firewalld-dserver-port.sh.example](../examples/firewalld-dserver-port.sh.example). + # Register SSH public keys in DTail server The DTail server now runs as a ``systemd`` service under system user ``dserver``. However, the system user ``dserver`` has no permissions to read the SSH public keys from ``/home/USER/.ssh/authorized_keys``. Therefore, no user would be able to establish an SSH session to DTail server. As an alternative path DTail server also checks for public SSH key files in ``/var/run/dserver/cache/USER.authorized_keys``. @@ -113,6 +150,24 @@ It is recommended to execute [update_key_cache.sh](../examples/update_key_cache. % sudo systemctl start dserver-update-keycache.timer ``` +# Prune old dserver log files + +Log files live under ``/var/run/dserver/log`` (see ``LogDir`` in ``dtail.json``). To remove ``*.log`` files **older than seven days**, install [prune_dserver_logs.sh](../examples/prune_dserver_logs.sh.example) and a systemd timer (runs daily with a randomized delay): + +```console +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/prune_dserver_logs.sh.example | + sudo tee /var/run/dserver/prune_dserver_logs.sh +% sudo chmod 755 /var/run/dserver/prune_dserver_logs.sh +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver-prune-logs.service.example | + sudo tee /etc/systemd/system/dserver-prune-logs.service +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver-prune-logs.timer.example | + sudo tee /etc/systemd/system/dserver-prune-logs.timer +% sudo systemctl daemon-reload +% sudo systemctl enable --now dserver-prune-logs.timer +``` + +The script uses ``find /var/run/dserver/log -type f -name '*.log' -mtime +7 -delete``. + # Run DTail client Now you should be able to use DTail client like outlined in the [Quick Starting Guide](quickstart.md). Also, have a look at the [Examples](examples.md). diff --git a/doc/logformats.md b/doc/logformats.md index dbf2051..29e2559 100644 --- a/doc/logformats.md +++ b/doc/logformats.md @@ -42,7 +42,7 @@ func newGenericKVParser(hostname, timeZoneName string, timeZoneOffset int) (*gen return &genericKVParser{defaultParser: *defaultParser}, nil } -func (p *genericKVParser) MakeFields(maprLine string) (map[string]string, error) { +func (p *genericKVParser) MakeFields(maprLine, _ string) (map[string]string, error) { splitted := strings.Split(maprLine, protocol.FieldDelimiter) fields := make(map[string]string, len(splitted)) @@ -70,6 +70,7 @@ func (p *genericKVParser) MakeFields(maprLine string) (map[string]string, error) ... whereas: * `maprLine` is the whole raw log line to be parsed by the log format. +* `sourceID` is the stable identifier of the log file / stream the line came from. Stateful parsers (e.g. CSV with a header row per file) should key their per-file state by this value; stateless parsers may ignore it. * `protocol.FieldDelimiter` is the field delimiter used by the log format, here: `|`. * All field names starting with `$` are variables. They store some custom values. * All other fields are bareword-fields and are extracted from the log lines directly, e.g. `field1=value1|field2=value2|...` @@ -139,7 +140,7 @@ func newFooParser(hostname, timeZoneName string, timeZoneOffset int) (*fooParser return &fooParser{defaultParser: *defaultParser}, nil } -func (p *fooParser) MakeFields(maprLine string) (map[string]string, error) { +func (p *fooParser) MakeFields(maprLine, sourceID string) (map[string]string, error) { fields := make(map[string]string, 3) .. diff --git a/doc/performance_optimization_summary.md b/doc/performance_optimization_summary.md new file mode 100644 index 0000000..e54d9a9 --- /dev/null +++ b/doc/performance_optimization_summary.md @@ -0,0 +1,69 @@ +# DTail Performance Optimization Summary + +> **Historical note:** This is a point-in-time record of a specific optimization +> effort (trace-logging and buffering fixes) with the measurements taken at that +> time; the numbers and file paths (e.g. `turbo_writer.go`, since renamed to +> `line_writer.go`) are preserved as-is. The "turbo mode" it compares against a +> "non-turbo mode" is now the single, default read/output path — there is no mode +> split any more, and the `DTAIL_TURBOBOOST_DISABLE` toggle has been removed. + +## Changes Made + +### 1. Optimized Trace Logging (`/internal/io/dlog/dlog.go`) + +**Problem**: The `Trace()` and `Devel()` functions were calling `runtime.Caller(1)` for every invocation, even when trace logging was disabled. This was causing ~497ns overhead per call. + +**Solution**: Added early level checks before the expensive `runtime.Caller()` operation: + +```go +func (d *DLog) Trace(args ...interface{}) string { + // Early check to avoid expensive runtime.Caller when trace is disabled + if d.maxLevel < Trace { + return "" + } + // ... rest of function +} +``` + +### 2. Improved Buffering in Turbo Mode (`/internal/server/handlers/turbo_writer.go`) + +**Problem**: Turbo mode was forcing immediate flush after every line in serverless mode, defeating the purpose of buffering. + +**Solution**: Removed the immediate flush condition for serverless mode, allowing proper buffering: + +```go +// Changed from: +if w.writeBuf.Len() >= w.bufSize || w.serverless { + return w.flushBuffer() +} + +// To: +if w.writeBuf.Len() >= w.bufSize { + return w.flushBuffer() +} +``` + +## Performance Results + +### Before Optimization +- Turbo mode was **3-5x slower** than non-turbo mode +- DCat (10MB): 678ms (turbo) vs 210ms (non-turbo) +- DGrep (10MB): 570ms (turbo) vs 96ms (non-turbo) + +### After Optimization +- Turbo mode is now **2.87x faster** than non-turbo mode +- DCat (1M lines): 0.66s (turbo) vs 1.89s (non-turbo) - **65% improvement** +- DCat with colors: 1.69s (turbo) vs 2.24s (non-turbo) - **24% improvement** + +## Verification +- ✅ All unit tests pass +- ✅ All integration tests pass +- ✅ No functionality regression +- ✅ Backward compatible + +## Files Modified +1. `/internal/io/dlog/dlog.go` - Lines 196-216 +2. `/internal/server/handlers/turbo_writer.go` - Lines 104-108 + +## Key Takeaway +The trace logging overhead was the primary bottleneck, causing DTail to spend more time logging than processing data. By adding simple level checks before expensive operations, we achieved a ~3x performance improvement in turbo mode.
\ No newline at end of file diff --git a/doc/pgo_commands_detail.md b/doc/pgo_commands_detail.md new file mode 100644 index 0000000..e874ba8 --- /dev/null +++ b/doc/pgo_commands_detail.md @@ -0,0 +1,216 @@ +# PGO Command Execution Details + +This document shows the exact commands executed during Profile-Guided Optimization (PGO) generation for DTail tools. + +## Overview + +When running `make pgo-generate` or `dtail-tools pgo`, the following commands are executed to generate performance profiles for each tool. + +## Commands Executed + +### 1. Building Baseline Binaries + +```bash +go build -o pgo-build/dtail-baseline ./cmd/dtail +go build -o pgo-build/dcat-baseline ./cmd/dcat +go build -o pgo-build/dgrep-baseline ./cmd/dgrep +go build -o pgo-build/dmap-baseline ./cmd/dmap +go build -o pgo-build/dserver-baseline ./cmd/dserver +``` + +### 2. Profile Generation Commands + +#### DTail Profile Generation +```bash +# Background log writer process +bash -c "for i in {1..200}; do + level=$((i % 4)) + case $level in + 0) lvl=INFO;; + 1) lvl=WARN;; + 2) lvl=ERROR;; + 3) lvl=DEBUG;; + esac + echo \"[2025-07-04 15:00:00] $lvl - Test log line number $i with some additional text to process\" >> growing.log + sleep 0.015 +done" + +# DTail command +pgo-build/dtail-baseline \ + -cfg none \ + -plain \ + -profile \ + -profiledir pgo-profiles/iter_dtail_TIMESTAMP \ + -regex "ERROR|WARN" \ + -shutdownAfter 3 \ + pgo-profiles/growing.log +``` + +#### DCat Profile Generation +```bash +pgo-build/dcat-baseline \ + -cfg none \ + -plain \ + -profile \ + -profiledir pgo-profiles/iter_dcat_TIMESTAMP \ + pgo-profiles/test.log +``` + +#### DGrep Profile Generation +```bash +pgo-build/dgrep-baseline \ + -cfg none \ + -plain \ + -profile \ + -profiledir pgo-profiles/iter_dgrep_TIMESTAMP \ + -regex "ERROR|WARN" \ + pgo-profiles/test.log +``` + +#### DMap Profile Generation +```bash +pgo-build/dmap-baseline \ + -cfg none \ + -plain \ + -profile \ + -profiledir pgo-profiles/iter_dmap_TIMESTAMP \ + -files pgo-profiles/test.csv \ + -query "select status, count(*) group by status" +``` + +#### DServer Profile Generation +```bash +# Start dserver with pprof endpoint +pgo-build/dserver-baseline \ + -cfg none \ + -pprof localhost:16060 \ + -port 12222 + +# Client commands to generate server load (run concurrently): +pgo-build/dcat-baseline \ + -cfg none \ + -server localhost:12222 \ + pgo-profiles/test.log + +pgo-build/dgrep-baseline \ + -cfg none \ + -server localhost:12222 \ + -regex "ERROR|WARN" \ + pgo-profiles/test.log + +pgo-build/dgrep-baseline \ + -cfg none \ + -server localhost:12222 \ + -regex "INFO.*action" \ + pgo-profiles/test.log + +pgo-build/dmap-baseline \ + -cfg none \ + -server localhost:12222 \ + -files pgo-profiles/test.csv \ + -query "select status, count(*) group by status" + +pgo-build/dmap-baseline \ + -cfg none \ + -server localhost:12222 \ + -files pgo-profiles/test.csv \ + -query "select department, avg(salary) group by department" + +# Capture CPU profile via HTTP +curl http://localhost:16060/debug/pprof/profile?seconds=5 > dserver.pprof +``` + +### 3. Profile Merging + +When multiple iterations are run, profiles are merged: + +```bash +# Merge multiple profile iterations +go tool pprof -proto \ + pgo-profiles/dcat.pprof.0.pprof \ + pgo-profiles/dcat.pprof.1.pprof \ + > pgo-profiles/dcat.pprof +``` + +### 4. Building with PGO + +```bash +# Build optimized binaries using profiles +go build -pgo=pgo-profiles/dcat.pprof -o pgo-build/dcat ./cmd/dcat +go build -pgo=pgo-profiles/dgrep.pprof -o pgo-build/dgrep ./cmd/dgrep +go build -pgo=pgo-profiles/dmap.pprof -o pgo-build/dmap ./cmd/dmap +go build -pgo=pgo-profiles/dtail.pprof -o pgo-build/dtail ./cmd/dtail +go build -pgo=pgo-profiles/dserver.pprof -o pgo-build/dserver ./cmd/dserver +``` + +### 5. Performance Comparison Commands + +Quick benchmarks are run to compare baseline vs optimized: + +```bash +# Baseline benchmark +pgo-build/dcat-baseline -cfg none -plain /tmp/pgo_bench.log + +# Optimized benchmark +pgo-build/dcat -cfg none -plain /tmp/pgo_bench.log + +# Similar commands for dgrep and dmap +pgo-build/dgrep-baseline -cfg none -plain -regex ERROR /tmp/pgo_bench.log +pgo-build/dmap-baseline -cfg none -plain -files /tmp/pgo_bench.csv -query "select count(*)" +``` + +## Test Data Generation + +The PGO framework generates realistic test data: + +### Log File (test.log) +- Contains timestamps, log levels (INFO, WARN, ERROR, DEBUG) +- Includes user actions, durations, and status information +- Default size: 1,000,000 lines (configurable with -datasize) + +### CSV File (test.csv) +- Contains employee data with departments, salaries, status +- Used for MapReduce queries +- Default size: 100,000 rows (1/10 of log file size) + +### Growing Log File (growing.log) +- Used specifically for dtail testing +- Simulates real-time log generation +- Writes ~200 lines over 3 seconds with mixed log levels + +## Customization Options + +### Adjust Test Data Size +```bash +dtail-tools pgo -datasize 5000000 # 5 million lines +``` + +### Run More Iterations +```bash +dtail-tools pgo -iterations 5 # Run 5 iterations per command +``` + +### Profile Specific Commands Only +```bash +dtail-tools pgo dcat dgrep # Only optimize dcat and dgrep +``` + +### Verbose Output +```bash +dtail-tools pgo -v # Show all command execution details +``` + +### Profile Generation Only +```bash +dtail-tools pgo -profileonly # Skip building optimized binaries +``` + +## Notes + +1. **Empty Profiles**: Some commands (like dtail) may generate empty profiles if they are I/O-bound. This is handled gracefully. + +2. **DServer Profiling**: Uses HTTP pprof endpoint instead of command-line profiling to capture server-side performance data. + +3. **Concurrent Execution**: Multiple client commands are run concurrently against dserver to generate realistic load patterns. + +4. **Profile Quality**: The effectiveness of PGO depends on how well the test workload represents real-world usage patterns.
\ No newline at end of file diff --git a/doc/pgo_implementation.md b/doc/pgo_implementation.md new file mode 100644 index 0000000..b404668 --- /dev/null +++ b/doc/pgo_implementation.md @@ -0,0 +1,173 @@ +# Profile-Guided Optimization (PGO) Implementation for DTail + +## Overview + +This document describes the Profile-Guided Optimization (PGO) implementation for DTail tools. PGO is a compiler optimization technique that uses runtime profiling data to guide optimization decisions, resulting in better performance for real-world usage patterns. + +## Implementation Details + +### Architecture + +The PGO implementation is integrated into the dtail-tools command as a subcommand: + +```bash +dtail-tools pgo [options] [commands...] +``` + +### Core Components + +1. **PGO Module** (`internal/tools/pgo/pgo.go`) + - Handles the complete PGO workflow + - Manages profile generation, merging, and PGO builds + - Provides performance comparison + +2. **Profiling Integration** + - All dtail commands now support the `-profile` flag + - dserver uses HTTP pprof endpoint for profiling + - Profiles are generated during realistic workloads + +3. **Makefile Integration** + - `make pgo` - Complete PGO workflow + - `make pgo-quick` - Quick PGO with smaller datasets + - `make pgo-generate` - Generate profiles only + - `make build-pgo` - Build with existing profiles + - `make install-pgo` - Install PGO-optimized binaries + +### Workflow + +1. **Build Baseline Binaries**: Standard Go builds without PGO +2. **Generate Profiles**: Run workloads to collect CPU profiles +3. **Merge Profiles**: Combine multiple profile iterations +4. **Build with PGO**: Use profiles to guide optimization +5. **Compare Performance**: Measure improvement + +### Profile Generation Details + +Each command has specific workloads designed to exercise common code paths: + +- **dcat**: Reading large log files +- **dgrep**: Pattern matching with various regex patterns +- **dmap**: MapReduce queries on CSV data +- **dtail**: Following growing log files with filtering +- **dserver**: Handling concurrent client connections + +### Special Handling + +1. **Empty Profiles**: I/O-bound operations may generate empty profiles. The implementation handles this gracefully by creating empty profile files that allow the workflow to continue. + +2. **dserver Profiling**: Uses HTTP pprof endpoint instead of command-line flags, allowing profile capture during server operation. + +3. **dtail Workload**: Simulates a growing log file with various log levels to exercise the tail functionality. + +## Performance Results + +Based on testing with PGO optimization: + +### Individual Command Improvements +- **dcat**: 3.75-5.40% improvement +- **dgrep**: Up to 19% improvement (varies by pattern hit rate) +- **dmap**: Up to 39% improvement for specific queries + +### Overall Performance Progression +From the original pre-optimization baseline to the current PGO-optimized build +(the channel-less read/output path, formerly called "turbo", is now the default +and only path): +- **dcat**: 14-21x faster overall +- **dgrep**: 9-15x faster overall +- **dmap**: 9-29% faster overall + +## Usage Examples + +### Generate PGO-Optimized Binaries +```bash +# Full PGO workflow +make pgo + +# Quick PGO with smaller datasets +make pgo-quick + +# Generate profiles only +make pgo-generate + +# Build with existing profiles +make build-pgo +``` + +### Using dtail-tools Directly +```bash +# Optimize all commands +dtail-tools pgo + +# Optimize specific commands +dtail-tools pgo dcat dgrep + +# Verbose mode with custom iterations +dtail-tools pgo -v -iterations 5 + +# Generate profiles only +dtail-tools pgo -profileonly +``` + +### Custom PGO Options +```bash +# Custom data size +dtail-tools pgo -datasize 5000000 + +# Custom profile directory +dtail-tools pgo -profiledir my-profiles + +# Custom output directory +dtail-tools pgo -outdir my-pgo-build +``` + +## Technical Considerations + +1. **Profile Quality**: The quality of PGO optimization depends on how representative the profiling workload is of real-world usage. + +2. **Binary Size**: PGO-optimized binaries may be slightly larger due to function cloning and inlining decisions. + +3. **Build Time**: Building with PGO takes longer than standard builds due to profile processing. + +4. **Go Version**: PGO requires Go 1.20 or later. + +## Integration with CI/CD + +To integrate PGO into your build pipeline: + +1. Generate profiles periodically with production-like workloads +2. Store profiles in version control or artifact repository +3. Use `make build-pgo` in your build process +4. Monitor performance metrics to validate improvements + +## Profile Files + +Profile files are stored in the `pgo-profiles/` directory: +- `dcat.pprof` - DCat CPU profile +- `dgrep.pprof` - DGrep CPU profile +- `dmap.pprof` - DMap CPU profile +- `dtail.pprof` - DTail CPU profile (may be empty for I/O-bound operations) +- `dserver.pprof` - DServer CPU profile + +## Troubleshooting + +### Empty Profiles +Some commands may generate empty profiles if they are I/O-bound. This is normal and the PGO workflow handles it gracefully. + +### Profile Merge Failures +If profile merging fails, check that: +- All profile files are valid +- Go tools are properly installed +- Sufficient disk space is available + +### Performance Not Improving +If PGO doesn't show improvement: +- Ensure profiles represent real workloads +- Check that the profile has sufficient samples +- Verify the correct profile is being used during build + +## Future Enhancements + +1. **Automated Profile Collection**: Collect profiles from production deployments +2. **Profile Versioning**: Track profile versions with code changes +3. **Multi-Architecture Support**: Generate architecture-specific profiles +4. **Continuous Profiling**: Regular profile updates based on usage patterns
\ No newline at end of file diff --git a/doc/profiling.md b/doc/profiling.md new file mode 100644 index 0000000..7925fb3 --- /dev/null +++ b/doc/profiling.md @@ -0,0 +1,376 @@ +# DTail Profiling Framework + +This document describes the profiling framework for dtail commands (dcat, dgrep, dmap) to analyze CPU usage and memory allocations. + +## Overview + +The profiling framework provides: +- CPU profiling to identify performance bottlenecks +- Memory profiling to track allocations and detect leaks +- Integration with existing benchmarks +- Analysis tools for profile interpretation + +## Quick Start + +### 1. Build the Tools + +```bash +make build # Builds all tools including dprofile +``` + +### 2. Run Commands with Profiling + +Each command now supports profiling flags: + +```bash +# Profile dcat +./dcat -profile -profiledir profiles -plain -cfg none /path/to/file.log + +# Profile dgrep with specific profiling types +./dgrep -cpuprofile -memprofile -profiledir profiles -regex "error" /path/to/file.log + +# Profile dmap +./dmap -profile -query "select count(*) from data.csv" +``` + +### 3. Analyze Profiles + +Use dtail-tools for quick analysis: + +```bash +# List all profiles +./dtail-tools profile -mode list + +# Analyze a specific profile +./dtail-tools profile -mode analyze profiles/dcat_cpu_20240101_120000.prof + +# Open web browser with flame graph +./dtail-tools profile -mode analyze profiles/dcat_cpu_*.prof -web + +# You can also use go tool pprof directly: +go tool pprof profiles/dcat_cpu_20240101_120000.prof +``` + +## Profiling Options + +### Command-line Flags + +All dtail commands support these profiling flags: + +- `-cpuprofile`: Enable CPU profiling only +- `-memprofile`: Enable memory profiling only +- `-profile`: Enable both CPU and memory profiling +- `-profiledir <dir>`: Directory to store profiles (default: "profiles") + +### Profile Types + +1. **CPU Profile** (`*_cpu_*.prof`) + - Samples CPU usage during execution + - Id |