From 3004a7100e325c006971cc2e8d0f157338c0ce5c Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Wed, 22 Jul 2026 23:52:52 +0300 Subject: =?UTF-8?q?docs:=20DTail=20fork=20=E2=80=94=20documentation,=20age?= =?UTF-8?q?nt=20guide,=20example=20configs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Squashed development of the documentation and example configuration: - AGENTS.md / CLAUDE.md: repository guide describing build/test/benchmark/PGO workflows and the single default read/output path (formerly "turbo"). - doc/ and docs/: query-language reference, log formats, auth-key fast reconnect, journal source reads, performance analyses (dated point-in-time records kept under historical-note disclaimers), and the turbo-vs-normal benchmark report with its result CSVs. - README.md updates; examples/ config + JSON schema aligned with the current Output* server tuning fields (the removed TurboBoost* keys dropped). Co-Authored-By: Claude Opus 4.8 --- AGENTS.md | 426 +++++++++++++++++++++++ CLAUDE.md | 1 + README.md | 98 +++++- doc/asciinema/README.md | 41 --- doc/auth-key-fast-reconnect.md | 200 +++++++++++ doc/index.md | 1 + doc/installation.md | 59 +++- doc/logformats.md | 5 +- doc/performance_optimization_summary.md | 69 ++++ doc/pgo_commands_detail.md | 216 ++++++++++++ doc/pgo_implementation.md | 173 +++++++++ doc/profiling.md | 376 ++++++++++++++++++++ doc/querylanguage.md | 40 ++- doc/refactoring_guide.md | 240 +++++++++++++ doc/turbo_performance_analysis.md | 104 ++++++ doc/turboboost_optimization.md | 117 +++++++ docs/SERVERLESS_LARGE_FILES_ISSUE.md | 61 ++++ docs/turbo-vs-normal-benchmark-20260718-run1.csv | 127 +++++++ docs/turbo-vs-normal-benchmark-20260718-run2.csv | 127 +++++++ docs/turbo-vs-normal-benchmark-20260718.md | 126 +++++++ examples/dserver-prune-logs.service.example | 8 + examples/dserver-prune-logs.timer.example | 9 + examples/dserver.service.example | 2 + examples/firewalld-dserver-port.sh.example | 21 ++ examples/prune_dserver_logs.sh.example | 10 + 25 files changed, 2609 insertions(+), 48 deletions(-) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md delete mode 100644 doc/asciinema/README.md create mode 100644 doc/auth-key-fast-reconnect.md create mode 100644 doc/performance_optimization_summary.md create mode 100644 doc/pgo_commands_detail.md create mode 100644 doc/pgo_implementation.md create mode 100644 doc/profiling.md create mode 100644 doc/refactoring_guide.md create mode 100644 doc/turbo_performance_analysis.md create mode 100644 doc/turboboost_optimization.md create mode 100644 docs/SERVERLESS_LARGE_FILES_ISSUE.md create mode 100644 docs/turbo-vs-normal-benchmark-20260718-run1.csv create mode 100644 docs/turbo-vs-normal-benchmark-20260718-run2.csv create mode 100644 docs/turbo-vs-normal-benchmark-20260718.md create mode 100644 examples/dserver-prune-logs.service.example create mode 100644 examples/dserver-prune-logs.timer.example create mode 100644 examples/firewalld-dserver-port.sh.example create mode 100644 examples/prune_dserver_logs.sh.example diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..90ac627 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,426 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +DTail (Distributed Tail) is a DevOps tool written in Go for distributed log operations across multiple servers. It provides secure, concurrent access to logs on many machines using SSH protocol, supporting tail, cat, grep, MapReduce operations, and auth-key fast reconnect optimization for repeated SSH connections. + +## Build Commands + +```bash +# Build all binaries +make build + +# Build individual components +make dtail # Client for tailing log files +make dserver # Server component (required on target machines) +make dcat # Client for displaying files +make dgrep # Client for searching files +make dmap # Client for MapReduce queries +make dtailhealth # Health check client + +# Clean build artifacts +make clean + +# Enable ACL support (requires libacl-devel) +DTAIL_USE_ACL=yes make build + +# Build without zstd (CGO-free cross-compiles; .zst logs unsupported) +DTAIL_NO_ZSTD=yes make build + +# Enable proprietary features +DTAIL_USE_PROPRIETARY=yes make build + +# Build PGO-optimized binaries (requires existing profiles) +make build-pgo + +# Generate PGO profiles and build optimized binaries +make pgo +``` + +## Testing & Development + +```bash +# Run all tests (unit tests only) +make test + +# Run all tests including integration tests +# IMPORTANT: Always rebuild binaries before running integration tests +make clean && make build +DTAIL_INTEGRATION_TEST_RUN_MODE=yes make test + +# Quick integration test workflow (recommended) +make build && DTAIL_INTEGRATION_TEST_RUN_MODE=yes make test + +# Run linting +make lint + +# Run go vet +make vet + +# Run integration tests individually (requires binaries built first) +cd integrationtests && go test +``` + +## Benchmarking + +```bash +# Run all benchmarks +make benchmark + +# Quick benchmarks (subset of tests) +make benchmark-quick + +# Full benchmarks with longer runs +make benchmark-full + +# Create a baseline for comparison +make benchmark-baseline + +# Compare current performance against a baseline +make benchmark-compare BASELINE=benchmarks/baselines/baseline_TIMESTAMP.txt +``` + +## Profile-Guided Optimization (PGO) + +```bash +# Full PGO workflow: generate profiles and build optimized binaries +make pgo + +# Quick PGO with smaller datasets (faster) +make pgo-quick + +# PGO for specific commands only +make pgo-commands COMMANDS='dcat dgrep' + +# Generate PGO profiles only (without building) +make pgo-generate + +# Build PGO-optimized binaries using existing profiles +make build-pgo + +# Install PGO-optimized binaries to system +make install-pgo + +# Clean PGO artifacts +make pgo-clean + +# Show PGO help +make pgo-help +``` + +### PGO Notes + +- PGO provides additional performance improvements on top of DTail's default optimized read/output path +- Measured improvements are workload-dependent and modest: on a 100 MB + serverless run, DGrep ~7-8%, DCat ~3%, DMap within measurement noise. PGO + output is byte-identical to non-PGO output (verified with `cmp`). +- Profiles are saved in `pgo-profiles/` directory +- `pgo-profiles/` and `pgo-build/` are **gitignored**: profiles are regenerated + locally with `make pgo` / `make pgo-generate`, not committed. They are + workload-specific; regenerate them after significant hot-path changes. +- Optimized binaries are built in `pgo-build/` directory +- Use `make build-pgo` to rebuild optimized binaries without regenerating profiles +- The tooling verifies every captured profile has non-zero CPU samples and + fails loudly otherwise (see `internal/tools/pgo`), so an idle-server or + I/O-bound capture can no longer silently produce an empty/zero-sample profile. +- **dtail is intentionally excluded from PGO.** Its follow client does not + return from `client.Start` under `-shutdownAfter`, SIGINT or SIGTERM (the + pre-existing "auto shutdown does not work" bug noted in `cmd/dtail/main.go`), + so it never flushes a CPU profile. Attempting to profile it produced the + 0-byte `dtail.pprof`. The `dtail-tools pgo` default command set therefore + covers dcat, dgrep, dmap and dserver only, matching the existing + `internal/tools/profile` harness which also omits dtail. Re-enable dtail here + once its follow shutdown is fixed. + +## Profiling + +```bash +# Profile all commands (dcat, dgrep, dmap) +make profile-all + +# Profile individual commands +make profile-dcat # Profile dcat with test data +make profile-dgrep # Profile dgrep with test data +make profile-dmap # Profile dmap MapReduce queries + +# Quick profiling with smaller datasets +make profile-quick + +# Full automated profiling (includes larger files) +make profile-auto + +# Clean all profile data +make profile-clean + +# Analyze a specific profile interactively +make profile-analyze PROFILE=profiles/dcat_cpu_*.prof + +# Generate flame graph visualization +make profile-flamegraph PROFILE=profiles/dcat_cpu_*.prof + +# Custom profiling options +PROFILE_SIZE=10000000 make profile-all # Profile with 10M lines +PROFILE_DIR=myprofiles make profile-dcat # Custom profile directory + +# Show all profiling options +make profile-help +``` + +### Profiling Notes + +- Profiles are saved in the `profiles/` directory by default +- Each command generates CPU, memory, and allocation profiles +- Use `go tool pprof` for detailed analysis of profile files + +## Test Execution Details + +- Integration tests require binaries to be built before execution +- **IMPORTANT:** Always recompile binaries after code changes before running integration tests: + ```bash + make clean && make build + DTAIL_INTEGRATION_TEST_RUN_MODE=yes make test + ``` +- Integration tests are run by setting DTAIL_INTEGRATION_TEST_RUN_MODE to yes, and by running 'make test' +- Integration tests verify: DCat, DGrep, DMap (MapReduce), DServer, DTail, DTailHealth, and auth-key fast reconnect functionality +- All tests run with race detection enabled (`--race` flag) + +## Known Limitations + +### Interactive Query Reload +Interactive query control is opt-in on the client with `--interactive-query`. +The controlling TTY accepts `:reload `, `:show`, `:help`, and `:quit`. + +**Compatibility and session semantics:** +- Initial interactive bootstrap prefers `SESSION START` when the server + advertises capability `query-update-v1` +- If that capability is absent, startup falls back to the legacy command stream + automatically so mixed-version client/server combinations still run +- Live `:reload` updates require every active server connection to advertise + `query-update-v1`; unsupported servers cause the reload to be rejected while + the current workload keeps running +- Successful reloads reuse the existing SSH session and advance a generation + boundary so stale output from older workloads is dropped + +### Auth-Key Fast Reconnect +Auth-key fast reconnect is enabled by default. The client can register a public key with `dserver` over an already-authenticated session, and subsequent connections can use this in-memory key before falling back to normal SSH auth. + +**Technical Details:** +- Client sends `AUTHKEY ` command during session setup +- Server stores keys in memory, per user, with TTL and max-keys limits +- SSH `PublicKeyCallback` checks in-memory auth-key store before `authorized_keys` +- If fast-path auth misses (restart/expiry/mismatch), normal SSH auth is used automatically + +**Config and Flags:** +- Client flag: `--auth-key-path` (default `~/.ssh/id_rsa`) +- Client flag: `--no-auth-key` (disable feature) +- Client env: `DTAIL_AUTH_KEY_PATH` (primary env alias for auth key path; takes precedence over `DTAIL_SSH_PRIVATE_KEYFILE_PATH`) +- Client env: `DTAIL_SSH_PRIVATE_KEYFILE_PATH` (legacy alias; used only when `DTAIL_AUTH_KEY_PATH` is not set) +- Env var precedence (highest to lowest): CLI flag → `DTAIL_AUTH_KEY_PATH` → `DTAIL_SSH_PRIVATE_KEYFILE_PATH` +- Server config: `AuthKeyEnabled` (default `true`) +- Server config: `AuthKeyTTLSeconds` (default `86400`) +- Server config: `AuthKeyMaxPerUser` (default `5`) + +### Journal Source Reads +Journal targets use `journal:unit.service` syntax. Permission rules should match the full target, for example `Server.Permissions.Users[user]: ["readfiles:^journal:.*\\.service$"]`. + +**Technical Details:** +- Server capability `journal-v1` is required for any journal-backed read target; clients reject journal sources when the server does not advertise it. +- The capability is advertised only on Linux when `journalctl` is available on `PATH`. +- Journal reads use `journalctl` via `os/exec` only; there is no cgo or `libsystemd` path, and the feature is Linux-only at runtime. +- Non-follow reads the current journal snapshot once. Follow mode adds `-f -n 0` and keeps restarting `journalctl` until canceled. + +### Client Log Contents: Diagnostics vs Payload +The default client logger is `fout` (stdout + a daily file at +`/YYYYMMDD.log`, `LogDir` defaults to `~/log`). By default that file +records DIAGNOSTICS ONLY — the small connection/audit lines (INFO/WARN/ERROR). +The full retrieved PAYLOAD (the bulk `dcat`/`dgrep`/`dtail` output) is NOT written +to the file by default, so a large read no longer silently grows the daily log by +the full payload size. Payload always still goes to STDOUT/terminal, unchanged. + +**Restoring the legacy full-payload tee (opt-in):** +- Client flag: `--log-payload` (all client commands: dcat, dgrep, dtail, dmap) +- Client config: `Client.LogPayload` (default `false`) +- When enabled, retrieved payload is teed into the daily log file as before. + +**Seam and scope:** +- The split happens in the `fout` logger: diagnostics arrive via `Log`/`LogWithColors` + (always written to both stdout and file); payload arrives via `Raw`/`RawWithColors` + (always to stdout, to the file only when `LogPayload` is set). +- Only the default `fout` logger is affected. `--logger stdout` (no file sink) and + `--logger none` are unaffected. `--logger file` is a pure file sink with no stdout + tee; it deliberately still writes payload to the file, because otherwise its output + file would be empty — payload teeing there is not gated by `--log-payload`. +- Note: the serverless direct-output path can bypass the logger and write payload + directly to stdout; that path never wrote payload to the file and is unaffected. + The disk-fill footgun lives on the `fout` file path (the server-mode receive + path), which is what this setting gates. + +### Output Path and MapReduce Operations +DTail uses a single, channel-less read/output path for both direct output +operations (cat, grep, tail) and MapReduce operations in server mode. This was +formerly called "turbo boost" mode and offered as an opt-out optimization; it is +now the one and only mode. There is no on/off toggle: the old +`DTAIL_TURBOBOOST_DISABLE` environment variable and the `Server.TurboBoostDisable` +config field have been removed. `DTAIL_TURBOBOOST_DISABLE` is now inert (a no-op), +and a leftover `TurboBoostDisable` key in an old config file is silently ignored +(config decoding does not reject unknown keys). + +**Technical Details:** +- For cat/grep/tail: the read path writes directly to the output/connection + without channel hand-offs. +- For MapReduce in server mode: lines are processed directly without channels. +- For MapReduce in serverless/client mode: the server-side direct processing does + not apply — client-side aggregation runs on the client. + +**Server-Side MapReduce (dserver):** +- Lines are processed directly without channel overhead +- Batch processing reduces lock contention +- Memory pooling reduces garbage collection pressure +- Same output format and accuracy regardless of workload + +**Tuning knobs (server config):** +The output path timing and buffer knobs live in the `Output*` / `Shutdown*` config +namespace (formerly `Turbo*Ms` / `TurboChannelBufferSize` / +`ShutdownTurboSerializeWaitMs`). See `internal/config/server.go` for the current +fields: `OutputTransmissionDelayMs`, `OutputEOFWaitBaseMs`, +`OutputEOFWaitPerFileMs`, `OutputEOFWaitMaxMs`, `OutputChannelBufferSize`, +`OutputFlushTimeoutMs`, `OutputFlushPollIntervalMs`, `OutputReadRetryIntervalMs`, +`OutputEOFAckTimeoutMs`, and `ShutdownOutputSerializeWaitMs`. All are optional +(`omitempty`) internal tuning knobs with sensible defaults; the old `Turbo*` keys +are gone, so a config that still sets them reverts silently to the defaults. + +**Best Practices for High-Concurrency MapReduce:** +1. Increase MaxConcurrentCats in the server configuration to match workload +2. Use server mode for large-scale MapReduce operations +3. Monitor logs for performance metrics + +Note: three operator-facing diagnostic log lines still contain the word "turbo" +verbatim ("Using turbo mode for reading", "Using turbo aggregate processor for +MapReduce", "Creating turbo aggregate for MapReduce"). These are deliberately kept +as stable log strings and do not imply a separate mode. + +## Benchmarking & Profiling + +```bash +# Run benchmarks +make benchmark + +# Run performance profiling +make profile + +# Generate profiling reports +make profile-report + +# Run specific benchmark suites +make benchmark-network +make benchmark-mapreduce +make benchmark-ssh +``` + +## Profile-Guided Optimization (PGO) + +```bash +# Run PGO for all commands +make pgo + +# Quick PGO with smaller datasets +make pgo-quick + +# PGO for specific commands +make pgo-commands COMMANDS='dcat dgrep' + +# Clean PGO artifacts +make pgo-clean + +# Show PGO help +make pgo-help + +# Direct usage with dtail-tools +dtail-tools pgo # Optimize all commands +dtail-tools pgo dcat dgrep # Optimize specific commands +dtail-tools pgo -v -iterations 5 # Verbose with 5 iterations + +# After PGO, optimized binaries are in pgo-build/ +``` + +### PGO Notes + +- PGO uses profile data from real workloads to optimize binary performance +- The process involves: building baseline → generating profiles → building with PGO +- Typical improvements range from 5-20% depending on the workload +- Optimized binaries are placed in the `pgo-build/` directory + +## Architecture & Code Organization + +### Binary Entry Points +- `/cmd/dtail/` - Remote log tailing client +- `/cmd/dserver/` - Server daemon +- `/cmd/dcat/` - Remote file reading client +- `/cmd/dgrep/` - Remote file searching client +- `/cmd/dmap/` - MapReduce query client +- `/cmd/dtailhealth/` - Health check client + +### Core Implementation +- `/internal/clients/` - Client implementations for each tool +- `/internal/server/` - Server daemon logic +- `/internal/mapr/` - MapReduce engine and query parsing +- `/internal/ssh/` - SSH client/server components +- `/internal/config/` - Configuration management +- `/internal/io/` - File operations, logging, compression handling + +### Key Architectural Patterns + +1. **Client-Server Communication**: All clients communicate with dserver instances via SSH protocol on port 2222 (configurable) + +2. **MapReduce Query Engine**: Located in `/internal/mapr/`, implements SQL-like query language for distributed log aggregation + +3. **Configuration System**: JSON-based configuration in `/internal/config/`, supports both client and server settings + +4. **SSH Integration**: Custom SSH server implementation in `/internal/ssh/server/` and client in `/internal/ssh/client/` + +5. **Compression Support**: Automatic handling of gzip and zstd compressed files in `/internal/io/` + +6. **Auth-Key Fast Reconnect**: Client registers a public key via `AUTHKEY`; server validates against in-memory auth-key cache before falling back to `authorized_keys` + +## Important Implementation Details + +- **Main Server Loop**: `/internal/server/server.go` - Core server processing logic +- **Client Base**: `/internal/clients/baseClient.go` - Common client functionality +- **MapReduce Parser**: `/internal/mapr/parse/` - SQL-like query language parser +- **Log Format Parsers**: `/internal/mapr/logformat/` - Extensible log parsing system +- **SSH Authorization Callback**: `/internal/ssh/server/publickeycallback.go` - auth-key fast-path + `authorized_keys` fallback +- **Auth-Key Cache**: `/internal/ssh/server/authkeystore.go` - in-memory per-user key cache (TTL/max-keys) +- **AUTHKEY Handler**: `/internal/server/handlers/serverhandler.go` - session command handling for auth-key registration + +## Configuration Files + +- Server config: `/etc/dserver/dtail.json` or `./dtail.json` +- Example configs: `/examples/` +- Docker configs: `/docker/` + +### Auth-Key Related Options + +- Client: `--auth-key-path`, `--no-auth-key` +- Client config: `Client.AuthKeyPath`, `Client.AuthKeyDisable` +- Client env: `DTAIL_AUTH_KEY_PATH` (takes precedence over `DTAIL_SSH_PRIVATE_KEYFILE_PATH`) +- Client env: `DTAIL_SSH_PRIVATE_KEYFILE_PATH` (legacy; used only when `DTAIL_AUTH_KEY_PATH` is unset) +- Server config: `Server.AuthKeyEnabled`, `Server.AuthKeyTTLSeconds`, `Server.AuthKeyMaxPerUser` + +## Common Development Tasks + +When modifying client behavior: +1. Check `/internal/clients/` for the specific client implementation +2. Common functionality is in `baseClient.go` +3. Client-specific logic is in respective files (e.g., `tail.go`, `cat.go`) + +When modifying server behavior: +1. Core server logic is in `/internal/server/server.go` +2. User authentication in `/internal/server/user/` +3. Handler implementations in `/internal/server/handlers/` + +When working with MapReduce: +1. Query parsing in `/internal/mapr/parse/` +2. Aggregation logic in `/internal/mapr/reducer/` +3. Log format parsing in `/internal/mapr/logformat/` diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..43c994c --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md diff --git a/README.md b/README.md index 8a0ac57..dbc7d62 100644 --- a/README.md +++ b/README.md @@ -13,13 +13,108 @@ The DTail binary operates in either client or server mode. The DTail server must ![DTail](doc/dtail.gif "Example") -If you like what you see [look here for more examples](doc/examples.md)! You can also read through the [DTail Mimecast Engineering Blog Post](https://medium.com/mimecast-engineering/dtail-the-distributed-log-tail-program-79b8087904bb). There is also a GitHub Page at [dtail.dev](https://dtail.dev). +If you like what you see [look here for more examples](doc/examples.md)! You can also read through the [DTail Mimecast Engineering Blog Post](https://medium.com/mimecast-engineering/dtail-the-distributed-log-tail-program-79b8087904bb). Installation and Usage ====================== * Check out the [DTail Documentation](doc/index.md) +Interactive Query Reload +======================== + +`dtail`, `dgrep`, `dcat`, and `dmap` accept `--interactive-query` to keep the +current run open and listen for control commands on the controlling TTY. + +Available control commands: + +* `:reload ` apply a new workload by reusing the current session when the + active servers support it +* `:show` print the current interactive state, including capability counts +* `:help` print the interactive command help text +* `:quit` stop the interactive session + +Reload flags are mode-specific: + +* `dtail` and `dgrep`: `--grep`/`--regex`, `--before`, `--after`, `--max`, + `--invert`, plus shared flags such as `--files`, `--plain`, `--quiet`, and + `--timeout` +* `dmap`, and query-driven `dtail`: `--query` plus the shared flags above +* `dcat`: shared flags such as `--files`, `--plain`, `--quiet`, and `--timeout` + +Examples: + +```bash +dtail --servers app01 --files /var/log/app.log --grep ERROR --interactive-query +# then type: +:reload --grep WARN + +dgrep --servers app01 --files /var/log/app.log --grep ERROR --interactive-query +# then type: +:reload --grep WARN --before 2 --after 3 + +dmap --servers app01 --files /var/log/app.log \ + --query 'from STATS select count($line) group by hostname' \ + --interactive-query +# then type: +:reload --query "from STATS select count($line),avg(latency) group by hostname" +``` + +Compatibility and session reuse: + +* On startup, an interactive client first tries `SESSION START` when the remote + side advertises the `query-update-v1` capability +* If a server is older or does not advertise that capability, startup falls + back to the legacy command stream automatically, so mixed-version + client/server combinations still run the original workload normally +* Live `:reload` updates require every active server to advertise + `query-update-v1`; otherwise the reload is rejected and the current workload + keeps running unchanged +* On capable servers, DTail reuses the existing SSH session and sends + `SESSION UPDATE` messages instead of reconnecting +* Every successful reload advances a generation boundary; late output from the + previous workload is dropped so stale matches do not leak into the new result + stream + +Auth-Key Fast Reconnect +======================= + +DTail supports an optional SSH auth optimization for repeated reconnects. +After a normal authenticated SSH session is established, the client can +register a local public key with `dserver` using an `AUTHKEY` command. The +server stores this key in memory only and checks it before `authorized_keys` +on subsequent connections. + +This reduces repeated hardware-token signing (for example YubiKey-backed SSH +agent keys) while keeping transparent fallback to normal SSH authentication. + +Client options: + +* `--auth-key-path` path to the private key to offer first and register + (default: `~/.ssh/id_rsa`) +* `--no-auth-key` disable auth-key registration/fast-path and use normal SSH + behavior only + +Server configuration (`dtail.json`): + +```json +{ + "Server": { + "AuthKeyEnabled": true, + "AuthKeyTTLSeconds": 86400, + "AuthKeyMaxPerUser": 5 + } +} +``` + +Security notes: + +* Registered keys are stored in memory only (no disk persistence) +* Registration is accepted only over an already-authenticated session +* TTL expiry and per-user key limits bound key lifetime and memory growth +* If fast-path auth is unavailable (restart/expiry/mismatch), DTail falls back + to normal SSH auth automatically + More ==== @@ -34,4 +129,3 @@ Credits * Thank you [Mimecast](https://www.mimecast.com) for supporting this Open-Source project. * Thank you to **Vlad-Marian Marian** for creating the DTail (dog) logo. * The Gopher was generated at https://gopherize.me -* The animated Gifs were created using `asciinema` with `asciicast2gif`. Check out [how this was done](./doc/asciinema/README.md) for more information. diff --git a/doc/asciinema/README.md b/doc/asciinema/README.md deleted file mode 100644 index 1eb1a2e..0000000 --- a/doc/asciinema/README.md +++ /dev/null @@ -1,41 +0,0 @@ -asciinema -========= - -The animated Gifs you find in the DTail docs were created using: - -* [asciinema](https://asciinema.org) -* [asciicast2gif](https://github.com/asciinema/asciicast2gif) - -On Fedora Linux 35. - -## Installing prerequisites - -On Fedora Linux 35 install `asciinema`: - -```shell -% sudo dnf install -y asciinema -``` - -and `asciicast2gif` (for simplicity, the Docker image was used): - -```shell -% docker pull asciinema/asciicast2gif -``` - -This of course assumes that Docker is up and running on your machine (out of scope for this documentation). - -## Record a shell session - -This is as simple as running - -```shell -% asciinema rec recording.json -``` - -This will launch a sub-shell to be recorded. Once done, exit the sub-shell with `exit`. - -## Convert the recording to a gif - -```shell -% docker run --rm -v $PWD:/data asciinema/asciicast2gif -t tango -s 2 recording.json recording.gif -``` diff --git a/doc/auth-key-fast-reconnect.md b/doc/auth-key-fast-reconnect.md new file mode 100644 index 0000000..cb9884a --- /dev/null +++ b/doc/auth-key-fast-reconnect.md @@ -0,0 +1,200 @@ +# Auth-Key Fast-Reconnect for DTail + +## Problem + +When using a YubiKey for SSH authentication, each DTail connection requires a +physical touch of the YubiKey during the SSH handshake. This is slow and becomes +painful when connecting to many servers concurrently — the YubiKey serialises +all signing requests, turning parallel connections into sequential ones. + +## Solution + +Allow the DTail client to register a local SSH public key with the DTail server +over an already-authenticated SSH session. The server caches this key +**in-memory only** (never written to disk). On subsequent connections the client +offers that local key first — a pure in-memory RSA verify with no YubiKey +interaction — and falls back to the original auth method if the server does not +recognise the key. + +## Design Principles + +1. **Transparent fallback** — Go's `golang.org/x/crypto/ssh` tries each + `AuthMethod` in order; if the fast key is rejected the client silently falls + back to the SSH agent / YubiKey. No user interaction required. +2. **Server keys are ephemeral** — the in-memory store is lost on server + restart. No file I/O, no persistence. +3. **Trust chain preserved** — an auth-key can only be registered over a session + that was already authenticated via the normal (YubiKey) path. +4. **Minimal protocol addition** — a single `AUTHKEY ` command + sent over the existing SSH session text protocol. + +## Architecture Overview + +``` +┌─────────────────────────────────────────────────────────┐ +│ DTail Client │ +│ │ +│ Auth methods (tried in order): │ +│ 1. Local private key (~/.ssh/id_rsa) ← FAST │ +│ 2. SSH Agent / YubiKey ← SLOW fallback │ +│ │ +│ After slow-path auth: │ +│ → sends AUTHKEY <~/.ssh/id_rsa.pub> to server │ +└────────────────────────┬────────────────────────────────┘ + │ SSH +┌────────────────────────▼────────────────────────────────┐ +│ DTail Server (dserver) │ +│ │ +│ PublicKeyCallback: │ +│ 1. Check in-memory authkeystore ← FAST │ +│ 2. Check authorized_keys file ← existing path │ +│ │ +│ AUTHKEY command handler: │ +│ → authkeystore.Add(user, pubkey) │ +│ → responds AUTHKEY OK / AUTHKEY ERR │ +│ │ +│ authkeystore (in-memory only): │ +│ map[username] → []PublicKey (with TTL, max per user) │ +└─────────────────────────────────────────────────────────┘ +``` + +## Sequence of Events + +### First Connection (slow path — YubiKey) + +1. Client checks for local private key at `~/.ssh/id_rsa` (or `--auth-key-path`). +2. Client builds auth methods list: `[localKey, sshAgent]`. +3. SSH handshake begins; server's `PublicKeyCallback` is called with local key. +4. Server checks in-memory authkeystore → not found. +5. Server checks `authorized_keys` file → not found (this key isn't in there). +6. Server rejects the key. +7. Go SSH client automatically tries next auth method: SSH agent (YubiKey). +8. YubiKey signs the challenge; server finds the YubiKey pubkey in + `authorized_keys` → auth succeeds. +9. Session is established; client sends DTail commands as usual. +10. Client reads `~/.ssh/id_rsa.pub` and sends `AUTHKEY `. +11. Server's handler parses the command, calls `authkeystore.Add(user, pubkey)`. +12. Server responds `AUTHKEY OK`. + +### Subsequent Connections (fast path — no YubiKey) + +1. Client builds auth methods list: `[localKey, sshAgent]`. +2. SSH handshake begins; server's `PublicKeyCallback` is called with local key. +3. Server checks in-memory authkeystore → **found** → auth succeeds immediately. +4. No YubiKey touch needed. Session is established instantly. + +### Fallback (server restarted, key expired) + +1. Client offers local key → server's authkeystore is empty → rejected. +2. Client falls back to SSH agent → YubiKey auth succeeds. +3. Client re-registers local pubkey via `AUTHKEY` command. + +## Components + +### 1. Server: In-Memory Auth-Key Store + +**New file:** `internal/ssh/server/authkeystore.go` + +- Thread-safe store using `sync.RWMutex`. +- Data structure: `map[string][]authKeyEntry` where key is username. +- Each `authKeyEntry` holds `gossh.PublicKey` + `time.Time` (registered at). +- Methods: `Add(user, pubkey)`, `Has(user, pubkey) bool`, `Remove(user, pubkey)`. +- Per-user max key limit (default 5, configurable via `AuthKeyMaxPerUser`). +- TTL-based expiry (default 24h, configurable via `AuthKeyTTLSeconds`). +- Lazy expiry: check TTL on `Has()` calls; optionally a background reaper. +- Package-level singleton or passed via dependency injection. + +### 2. Server: Extend PublicKeyCallback + +**Modified file:** `internal/ssh/server/publickeycallback.go` + +- Before the existing `authorizedKeysFile` lookup, check `authkeystore.Has(user, offeredPubKey)`. +- If found → return success immediately (fast path). +- If not found → fall through to existing file-based logic (no behaviour change). + +### 3. Server: AUTHKEY Command Handler + +**Modified file:** `internal/server/handlers/serverhandler.go` (or relevant handler) + +- Parse incoming line for `AUTHKEY ` prefix. +- Decode the base64 public key using `gossh.ParsePublicKey()`. +- Call `authkeystore.Add(user, pubkey)`. +- Write `AUTHKEY OK\n` or `AUTHKEY ERR \n` back to the client. +- Guard: only accept if `AuthKeyEnabled` is true in server config. + +### 4. Client: Auth Method Ordering (Multi-Method Support) + +**Modified file:** `internal/ssh/client/authmethods.go` + +- Change `initKnownHostsAuthMethods` to **collect multiple auth methods** + instead of returning after the first successful one. +- Order: local private key first (from `--auth-key-path`, default `~/.ssh/id_rsa`), + then SSH agent, then other default keys. +- This ensures Go's SSH client tries the fast key before the YubiKey. + +### 5. Client: Auth-Key Registration After Slow-Path Connection + +**Modified file:** `internal/clients/connectors/serverconnection.go` (or handler layer) + +- After session is established and DTail commands are sent, determine whether + the connection used the fast path or slow path. +- If slow path (YubiKey was used): read the public key file + (`--auth-key-path` + `.pub`), send `AUTHKEY ` command. +- Parse `AUTHKEY OK` / `AUTHKEY ERR` response. +- A simple heuristic: if the auth-key-path private key exists and we have a + corresponding `.pub` file, always send the registration — sending it again is + idempotent and cheap. + +### 6. Configuration + +**Modified files:** `internal/config/server.go`, `internal/config/client.go`, `internal/config/args.go` + +Server config (`dtail.json`): +- `AuthKeyEnabled` (bool, default `true`) +- `AuthKeyTTLSeconds` (int, default `86400` = 24h) +- `AuthKeyMaxPerUser` (int, default `5`) + +Client config / CLI flags: +- `--auth-key-path` (string, default `~/.ssh/id_rsa`) — path to the local + private key to try first and whose `.pub` counterpart is registered +- `--no-auth-key` (bool, default `false`) — disable auth-key feature entirely + +### 7. Integration Tests + +**Modified/new files in:** `integrationtests/` + +- Test that auth-key registration works end-to-end. +- Test that fast-path auth succeeds after registration. +- Test fallback when server has no cached key (simulating restart). +- Test TTL expiry and max-keys-per-user limits. +- Test `--no-auth-key` disables the feature. + +### 8. Documentation + +- Update `README.md` with auth-key feature description. +- Update `AGENTS.md` / `CLAUDE.md` with new config options and architecture notes. + +## Security Considerations + +- **No server-side disk persistence** — keys exist only in memory, lost on restart. +- **Trust chain** — auth-keys can only be registered over an already-authenticated + session. An attacker cannot register a key without first proving identity. +- **TTL expiry** — keys auto-expire (default 24h), limiting exposure window. +- **Per-user limits** — max 5 keys per user prevents memory exhaustion. +- **Same security model as `~/.ssh/id_rsa`** — the local key is protected by + filesystem permissions (0600). If an attacker has access to `~/.ssh/id_rsa`, + they already have SSH access anyway. +- **No new attack surface** — the `AUTHKEY` command is only processed inside an + authenticated session. The `PublicKeyCallback` fast-path is equivalent to + having the key in `authorized_keys`. + +## Implementation Order + +1. Auth-key store (server, standalone, unit-testable) +2. Extend `PublicKeyCallback` (server, minimal change) +3. `AUTHKEY` command handler (server handler) +4. Client auth method ordering (multi-method collection) +5. Client auth-key registration (send pubkey after slow-path) +6. Configuration and CLI flags +7. Integration tests +8. Documentation diff --git a/doc/index.md b/doc/index.md index 565253b..c45569d 100644 --- a/doc/index.md +++ b/doc/index.md @@ -10,5 +10,6 @@ DTail Documentation ## Advanced topics * The [DTail Query Language](./querylanguage.md) is the starting point to dig deeper into DTail's own SQL-like mapreduce language for extraction/aggregation stats from log files. +* The [Interactive Query Reload](../README.md#interactive-query-reload) section in the main README documents `:reload`, `:show`, `:help`, `:quit`, capability fallback on mixed-version servers, and session reuse semantics. * [Log Formats](./logformats.md) explains how to create your own custom log format for use with mapreduce queries. * Check out the [Testing Guide](./testing.md) for unit and integration testing. diff --git a/doc/installation.md b/doc/installation.md index 1f54050..c80e011 100644 --- a/doc/installation.md +++ b/doc/installation.md @@ -25,7 +25,17 @@ Set the `DTAIL_USE_ACL` environment variable before invoking the make command. % export DTAIL_USE_ACL=yes ``` -Alternatively, you could add `-tags linuxacl` to the Go compiler. +Alternatively, you could add `-tags linuxacl` to the Go compiler. + +## Build without zstd (optional) + +For targets where CGO-based zstd is unavailable (for example cross-compiling `dserver` for another architecture), build with the `nozstd` tag. Compressed `.zst` log files will not be supported in that binary. + +```console +% export DTAIL_NO_ZSTD=yes +``` + +This sets `-tags nozstd` via the Makefile. Plain `go build` users can pass `-tags nozstd` directly. # Install it @@ -61,15 +71,28 @@ uid=1001(dserver) 1001=670(dserver) groups=1001(dserver) sudo tee /etc/dserver/dtail.json ``` +### SSH listen address (``SSHBindAddress``) + +The example config sets ``Server.SSHBindAddress`` to ``0.0.0.0``, so dserver listens on **every** local IPv4 address, including your LAN (e.g. ``192.168.1.x`` on eth0) and any other interface (loopback, WireGuard, etc.). Clients reach it as ``:2222``; you do **not** need to change this for normal LAN access. + +To listen **only** on a specific address—for example only the home LAN and not on a VPN—set ``SSHBindAddress`` in ``/etc/dserver/dtail.json`` to **that machine’s** address (each host needs its own value), e.g. ``192.168.1.125`` on ``pi0``, ``192.168.1.126`` on ``pi1``. Alternatively, override from the command line (after ``-cfg``): ``dserver -cfg /etc/dserver/dtail.json -bindAddress 192.168.1.125``. Then reload or restart dserver. + 5. It is recommended to configure DTail server as a service to ``systemd``. An example unit file for ``systemd`` can be found [here](../examples/dserver.service.example). ```console % curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver.service.example | sudo tee /etc/systemd/system/dserver.service % sudo systemctl daemon-reload -% sudo systemctl enable dserver ``` +The unit is intended to stay **disabled** until you opt in. Start DTail server manually when needed: + +```console +% sudo systemctl start dserver +``` + +To start it automatically at boot, run once: `sudo systemctl enable dserver`. + # Start it To start the DTail server via ``systemd`` run: @@ -93,6 +116,20 @@ To start the DTail server via ``systemd`` run: Dec 06 13:21:24 serv-001.lan.example.org dserver[12296]: SERVER|serv-001|INFO|Binding server|1.2.3.4:2222 ``` +### Firewall (firewalld on RHEL, Rocky Linux, Fedora, …) + +The DTail server listens on TCP port ``2222`` (see ``SSHPort`` in ``dtail.json``). **ICMP (ping) may work while TCP to 2222 is blocked**, because host firewalls often allow ping but not arbitrary ports. + +If ``firewalld`` is active, allow the DTail port permanently and reload: + +```console +% sudo firewall-cmd --permanent --add-port=2222/tcp +% sudo firewall-cmd --reload +% sudo firewall-cmd --list-ports +``` + +Clients may report ``dial tcp …: connect: no route to host`` when the firewall rejects the connection with an ICMP unreachable—opening ``2222/tcp`` fixes that. For other firewalls (nftables, ufw, …), add an equivalent allow rule for ``2222/tcp``. A small helper script is [firewalld-dserver-port.sh.example](../examples/firewalld-dserver-port.sh.example). + # Register SSH public keys in DTail server The DTail server now runs as a ``systemd`` service under system user ``dserver``. However, the system user ``dserver`` has no permissions to read the SSH public keys from ``/home/USER/.ssh/authorized_keys``. Therefore, no user would be able to establish an SSH session to DTail server. As an alternative path DTail server also checks for public SSH key files in ``/var/run/dserver/cache/USER.authorized_keys``. @@ -113,6 +150,24 @@ It is recommended to execute [update_key_cache.sh](../examples/update_key_cache. % sudo systemctl start dserver-update-keycache.timer ``` +# Prune old dserver log files + +Log files live under ``/var/run/dserver/log`` (see ``LogDir`` in ``dtail.json``). To remove ``*.log`` files **older than seven days**, install [prune_dserver_logs.sh](../examples/prune_dserver_logs.sh.example) and a systemd timer (runs daily with a randomized delay): + +```console +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/prune_dserver_logs.sh.example | + sudo tee /var/run/dserver/prune_dserver_logs.sh +% sudo chmod 755 /var/run/dserver/prune_dserver_logs.sh +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver-prune-logs.service.example | + sudo tee /etc/systemd/system/dserver-prune-logs.service +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver-prune-logs.timer.example | + sudo tee /etc/systemd/system/dserver-prune-logs.timer +% sudo systemctl daemon-reload +% sudo systemctl enable --now dserver-prune-logs.timer +``` + +The script uses ``find /var/run/dserver/log -type f -name '*.log' -mtime +7 -delete``. + # Run DTail client Now you should be able to use DTail client like outlined in the [Quick Starting Guide](quickstart.md). Also, have a look at the [Examples](examples.md). diff --git a/doc/logformats.md b/doc/logformats.md index dbf2051..29e2559 100644 --- a/doc/logformats.md +++ b/doc/logformats.md @@ -42,7 +42,7 @@ func newGenericKVParser(hostname, timeZoneName string, timeZoneOffset int) (*gen return &genericKVParser{defaultParser: *defaultParser}, nil } -func (p *genericKVParser) MakeFields(maprLine string) (map[string]string, error) { +func (p *genericKVParser) MakeFields(maprLine, _ string) (map[string]string, error) { splitted := strings.Split(maprLine, protocol.FieldDelimiter) fields := make(map[string]string, len(splitted)) @@ -70,6 +70,7 @@ func (p *genericKVParser) MakeFields(maprLine string) (map[string]string, error) ... whereas: * `maprLine` is the whole raw log line to be parsed by the log format. +* `sourceID` is the stable identifier of the log file / stream the line came from. Stateful parsers (e.g. CSV with a header row per file) should key their per-file state by this value; stateless parsers may ignore it. * `protocol.FieldDelimiter` is the field delimiter used by the log format, here: `|`. * All field names starting with `$` are variables. They store some custom values. * All other fields are bareword-fields and are extracted from the log lines directly, e.g. `field1=value1|field2=value2|...` @@ -139,7 +140,7 @@ func newFooParser(hostname, timeZoneName string, timeZoneOffset int) (*fooParser return &fooParser{defaultParser: *defaultParser}, nil } -func (p *fooParser) MakeFields(maprLine string) (map[string]string, error) { +func (p *fooParser) MakeFields(maprLine, sourceID string) (map[string]string, error) { fields := make(map[string]string, 3) .. diff --git a/doc/performance_optimization_summary.md b/doc/performance_optimization_summary.md new file mode 100644 index 0000000..e54d9a9 --- /dev/null +++ b/doc/performance_optimization_summary.md @@ -0,0 +1,69 @@ +# DTail Performance Optimization Summary + +> **Historical note:** This is a point-in-time record of a specific optimization +> effort (trace-logging and buffering fixes) with the measurements taken at that +> time; the numbers and file paths (e.g. `turbo_writer.go`, since renamed to +> `line_writer.go`) are preserved as-is. The "turbo mode" it compares against a +> "non-turbo mode" is now the single, default read/output path — there is no mode +> split any more, and the `DTAIL_TURBOBOOST_DISABLE` toggle has been removed. + +## Changes Made + +### 1. Optimized Trace Logging (`/internal/io/dlog/dlog.go`) + +**Problem**: The `Trace()` and `Devel()` functions were calling `runtime.Caller(1)` for every invocation, even when trace logging was disabled. This was causing ~497ns overhead per call. + +**Solution**: Added early level checks before the expensive `runtime.Caller()` operation: + +```go +func (d *DLog) Trace(args ...interface{}) string { + // Early check to avoid expensive runtime.Caller when trace is disabled + if d.maxLevel < Trace { + return "" + } + // ... rest of function +} +``` + +### 2. Improved Buffering in Turbo Mode (`/internal/server/handlers/turbo_writer.go`) + +**Problem**: Turbo mode was forcing immediate flush after every line in serverless mode, defeating the purpose of buffering. + +**Solution**: Removed the immediate flush condition for serverless mode, allowing proper buffering: + +```go +// Changed from: +if w.writeBuf.Len() >= w.bufSize || w.serverless { + return w.flushBuffer() +} + +// To: +if w.writeBuf.Len() >= w.bufSize { + return w.flushBuffer() +} +``` + +## Performance Results + +### Before Optimization +- Turbo mode was **3-5x slower** than non-turbo mode +- DCat (10MB): 678ms (turbo) vs 210ms (non-turbo) +- DGrep (10MB): 570ms (turbo) vs 96ms (non-turbo) + +### After Optimization +- Turbo mode is now **2.87x faster** than non-turbo mode +- DCat (1M lines): 0.66s (turbo) vs 1.89s (non-turbo) - **65% improvement** +- DCat with colors: 1.69s (turbo) vs 2.24s (non-turbo) - **24% improvement** + +## Verification +- ✅ All unit tests pass +- ✅ All integration tests pass +- ✅ No functionality regression +- ✅ Backward compatible + +## Files Modified +1. `/internal/io/dlog/dlog.go` - Lines 196-216 +2. `/internal/server/handlers/turbo_writer.go` - Lines 104-108 + +## Key Takeaway +The trace logging overhead was the primary bottleneck, causing DTail to spend more time logging than processing data. By adding simple level checks before expensive operations, we achieved a ~3x performance improvement in turbo mode. \ No newline at end of file diff --git a/doc/pgo_commands_detail.md b/doc/pgo_commands_detail.md new file mode 100644 index 0000000..e874ba8 --- /dev/null +++ b/doc/pgo_commands_detail.md @@ -0,0 +1,216 @@ +# PGO Command Execution Details + +This document shows the exact commands executed during Profile-Guided Optimization (PGO) generation for DTail tools. + +## Overview + +When running `make pgo-generate` or `dtail-tools pgo`, the following commands are executed to generate performance profiles for each tool. + +## Commands Executed + +### 1. Building Baseline Binaries + +```bash +go build -o pgo-build/dtail-baseline ./cmd/dtail +go build -o pgo-build/dcat-baseline ./cmd/dcat +go build -o pgo-build/dgrep-baseline ./cmd/dgrep +go build -o pgo-build/dmap-baseline ./cmd/dmap +go build -o pgo-build/dserver-baseline ./cmd/dserver +``` + +### 2. Profile Generation Commands + +#### DTail Profile Generation +```bash +# Background log writer process +bash -c "for i in {1..200}; do + level=$((i % 4)) + case $level in + 0) lvl=INFO;; + 1) lvl=WARN;; + 2) lvl=ERROR;; + 3) lvl=DEBUG;; + esac + echo \"[2025-07-04 15:00:00] $lvl - Test log line number $i with some additional text to process\" >> growing.log + sleep 0.015 +done" + +# DTail command +pgo-build/dtail-baseline \ + -cfg none \ + -plain \ + -profile \ + -profiledir pgo-profiles/iter_dtail_TIMESTAMP \ + -regex "ERROR|WARN" \ + -shutdownAfter 3 \ + pgo-profiles/growing.log +``` + +#### DCat Profile Generation +```bash +pgo-build/dcat-baseline \ + -cfg none \ + -plain \ + -profile \ + -profiledir pgo-profiles/iter_dcat_TIMESTAMP \ + pgo-profiles/test.log +``` + +#### DGrep Profile Generation +```bash +pgo-build/dgrep-baseline \ + -cfg none \ + -plain \ + -profile \ + -profiledir pgo-profiles/iter_dgrep_TIMESTAMP \ + -regex "ERROR|WARN" \ + pgo-profiles/test.log +``` + +#### DMap Profile Generation +```bash +pgo-build/dmap-baseline \ + -cfg none \ + -plain \ + -profile \ + -profiledir pgo-profiles/iter_dmap_TIMESTAMP \ + -files pgo-profiles/test.csv \ + -query "select status, count(*) group by status" +``` + +#### DServer Profile Generation +```bash +# Start dserver with pprof endpoint +pgo-build/dserver-baseline \ + -cfg none \ + -pprof localhost:16060 \ + -port 12222 + +# Client commands to generate server load (run concurrently): +pgo-build/dcat-baseline \ + -cfg none \ + -server localhost:12222 \ + pgo-profiles/test.log + +pgo-build/dgrep-baseline \ + -cfg none \ + -server localhost:12222 \ + -regex "ERROR|WARN" \ + pgo-profiles/test.log + +pgo-build/dgrep-baseline \ + -cfg none \ + -server localhost:12222 \ + -regex "INFO.*action" \ + pgo-profiles/test.log + +pgo-build/dmap-baseline \ + -cfg none \ + -server localhost:12222 \ + -files pgo-profiles/test.csv \ + -query "select status, count(*) group by status" + +pgo-build/dmap-baseline \ + -cfg none \ + -server localhost:12222 \ + -files pgo-profiles/test.csv \ + -query "select department, avg(salary) group by department" + +# Capture CPU profile via HTTP +curl http://localhost:16060/debug/pprof/profile?seconds=5 > dserver.pprof +``` + +### 3. Profile Merging + +When multiple iterations are run, profiles are merged: + +```bash +# Merge multiple profile iterations +go tool pprof -proto \ + pgo-profiles/dcat.pprof.0.pprof \ + pgo-profiles/dcat.pprof.1.pprof \ + > pgo-profiles/dcat.pprof +``` + +### 4. Building with PGO + +```bash +# Build optimized binaries using profiles +go build -pgo=pgo-profiles/dcat.pprof -o pgo-build/dcat ./cmd/dcat +go build -pgo=pgo-profiles/dgrep.pprof -o pgo-build/dgrep ./cmd/dgrep +go build -pgo=pgo-profiles/dmap.pprof -o pgo-build/dmap ./cmd/dmap +go build -pgo=pgo-profiles/dtail.pprof -o pgo-build/dtail ./cmd/dtail +go build -pgo=pgo-profiles/dserver.pprof -o pgo-build/dserver ./cmd/dserver +``` + +### 5. Performance Comparison Commands + +Quick benchmarks are run to compare baseline vs optimized: + +```bash +# Baseline benchmark +pgo-build/dcat-baseline -cfg none -plain /tmp/pgo_bench.log + +# Optimized benchmark +pgo-build/dcat -cfg none -plain /tmp/pgo_bench.log + +# Similar commands for dgrep and dmap +pgo-build/dgrep-baseline -cfg none -plain -regex ERROR /tmp/pgo_bench.log +pgo-build/dmap-baseline -cfg none -plain -files /tmp/pgo_bench.csv -query "select count(*)" +``` + +## Test Data Generation + +The PGO framework generates realistic test data: + +### Log File (test.log) +- Contains timestamps, log levels (INFO, WARN, ERROR, DEBUG) +- Includes user actions, durations, and status information +- Default size: 1,000,000 lines (configurable with -datasize) + +### CSV File (test.csv) +- Contains employee data with departments, salaries, status +- Used for MapReduce queries +- Default size: 100,000 rows (1/10 of log file size) + +### Growing Log File (growing.log) +- Used specifically for dtail testing +- Simulates real-time log generation +- Writes ~200 lines over 3 seconds with mixed log levels + +## Customization Options + +### Adjust Test Data Size +```bash +dtail-tools pgo -datasize 5000000 # 5 million lines +``` + +### Run More Iterations +```bash +dtail-tools pgo -iterations 5 # Run 5 iterations per command +``` + +### Profile Specific Commands Only +```bash +dtail-tools pgo dcat dgrep # Only optimize dcat and dgrep +``` + +### Verbose Output +```bash +dtail-tools pgo -v # Show all command execution details +``` + +### Profile Generation Only +```bash +dtail-tools pgo -profileonly # Skip building optimized binaries +``` + +## Notes + +1. **Empty Profiles**: Some commands (like dtail) may generate empty profiles if they are I/O-bound. This is handled gracefully. + +2. **DServer Profiling**: Uses HTTP pprof endpoint instead of command-line profiling to capture server-side performance data. + +3. **Concurrent Execution**: Multiple client commands are run concurrently against dserver to generate realistic load patterns. + +4. **Profile Quality**: The effectiveness of PGO depends on how well the test workload represents real-world usage patterns. \ No newline at end of file diff --git a/doc/pgo_implementation.md b/doc/pgo_implementation.md new file mode 100644 index 0000000..b404668 --- /dev/null +++ b/doc/pgo_implementation.md @@ -0,0 +1,173 @@ +# Profile-Guided Optimization (PGO) Implementation for DTail + +## Overview + +This document describes the Profile-Guided Optimization (PGO) implementation for DTail tools. PGO is a compiler optimization technique that uses runtime profiling data to guide optimization decisions, resulting in better performance for real-world usage patterns. + +## Implementation Details + +### Architecture + +The PGO implementation is integrated into the dtail-tools command as a subcommand: + +```bash +dtail-tools pgo [options] [commands...] +``` + +### Core Components + +1. **PGO Module** (`internal/tools/pgo/pgo.go`) + - Handles the complete PGO workflow + - Manages profile generation, merging, and PGO builds + - Provides performance comparison + +2. **Profiling Integration** + - All dtail commands now support the `-profile` flag + - dserver uses HTTP pprof endpoint for profiling + - Profiles are generated during realistic workloads + +3. **Makefile Integration** + - `make pgo` - Complete PGO workflow + - `make pgo-quick` - Quick PGO with smaller datasets + - `make pgo-generate` - Generate profiles only + - `make build-pgo` - Build with existing profiles + - `make install-pgo` - Install PGO-optimized binaries + +### Workflow + +1. **Build Baseline Binaries**: Standard Go builds without PGO +2. **Generate Profiles**: Run workloads to collect CPU profiles +3. **Merge Profiles**: Combine multiple profile iterations +4. **Build with PGO**: Use profiles to guide optimization +5. **Compare Performance**: Measure improvement + +### Profile Generation Details + +Each command has specific workloads designed to exercise common code paths: + +- **dcat**: Reading large log files +- **dgrep**: Pattern matching with various regex patterns +- **dmap**: MapReduce queries on CSV data +- **dtail**: Following growing log files with filtering +- **dserver**: Handling concurrent client connections + +### Special Handling + +1. **Empty Profiles**: I/O-bound operations may generate empty profiles. The implementation handles this gracefully by creating empty profile files that allow the workflow to continue. + +2. **dserver Profiling**: Uses HTTP pprof endpoint instead of command-line flags, allowing profile capture during server operation. + +3. **dtail Workload**: Simulates a growing log file with various log levels to exercise the tail functionality. + +## Performance Results + +Based on testing with PGO optimization: + +### Individual Command Improvements +- **dcat**: 3.75-5.40% improvement +- **dgrep**: Up to 19% improvement (varies by pattern hit rate) +- **dmap**: Up to 39% improvement for specific queries + +### Overall Performance Progression +From the original pre-optimization baseline to the current PGO-optimized build +(the channel-less read/output path, formerly called "turbo", is now the default +and only path): +- **dcat**: 14-21x faster overall +- **dgrep**: 9-15x faster overall +- **dmap**: 9-29% faster overall + +## Usage Examples + +### Generate PGO-Optimized Binaries +```bash +# Full PGO workflow +make pgo + +# Quick PGO with smaller datasets +make pgo-quick + +# Generate profiles only +make pgo-generate + +# Build with existing profiles +make build-pgo +``` + +### Using dtail-tools Directly +```bash +# Optimize all commands +dtail-tools pgo + +# Optimize specific commands +dtail-tools pgo dcat dgrep + +# Verbose mode with custom iterations +dtail-tools pgo -v -iterations 5 + +# Generate profiles only +dtail-tools pgo -profileonly +``` + +### Custom PGO Options +```bash +# Custom data size +dtail-tools pgo -datasize 5000000 + +# Custom profile directory +dtail-tools pgo -profiledir my-profiles + +# Custom output directory +dtail-tools pgo -outdir my-pgo-build +``` + +## Technical Considerations + +1. **Profile Quality**: The quality of PGO optimization depends on how representative the profiling workload is of real-world usage. + +2. **Binary Size**: PGO-optimized binaries may be slightly larger due to function cloning and inlining decisions. + +3. **Build Time**: Building with PGO takes longer than standard builds due to profile processing. + +4. **Go Version**: PGO requires Go 1.20 or later. + +## Integration with CI/CD + +To integrate PGO into your build pipeline: + +1. Generate profiles periodically with production-like workloads +2. Store profiles in version control or artifact repository +3. Use `make build-pgo` in your build process +4. Monitor performance metrics to validate improvements + +## Profile Files + +Profile files are stored in the `pgo-profiles/` directory: +- `dcat.pprof` - DCat CPU profile +- `dgrep.pprof` - DGrep CPU profile +- `dmap.pprof` - DMap CPU profile +- `dtail.pprof` - DTail CPU profile (may be empty for I/O-bound operations) +- `dserver.pprof` - DServer CPU profile + +## Troubleshooting + +### Empty Profiles +Some commands may generate empty profiles if they are I/O-bound. This is normal and the PGO workflow handles it gracefully. + +### Profile Merge Failures +If profile merging fails, check that: +- All profile files are valid +- Go tools are properly installed +- Sufficient disk space is available + +### Performance Not Improving +If PGO doesn't show improvement: +- Ensure profiles represent real workloads +- Check that the profile has sufficient samples +- Verify the correct profile is being used during build + +## Future Enhancements + +1. **Automated Profile Collection**: Collect profiles from production deployments +2. **Profile Versioning**: Track profile versions with code changes +3. **Multi-Architecture Support**: Generate architecture-specific profiles +4. **Continuous Profiling**: Regular profile updates based on usage patterns \ No newline at end of file diff --git a/doc/profiling.md b/doc/profiling.md new file mode 100644 index 0000000..7925fb3 --- /dev/null +++ b/doc/profiling.md @@ -0,0 +1,376 @@ +# DTail Profiling Framework + +This document describes the profiling framework for dtail commands (dcat, dgrep, dmap) to analyze CPU usage and memory allocations. + +## Overview + +The profiling framework provides: +- CPU profiling to identify performance bottlenecks +- Memory profiling to track allocations and detect leaks +- Integration with existing benchmarks +- Analysis tools for profile interpretation + +## Quick Start + +### 1. Build the Tools + +```bash +make build # Builds all tools including dprofile +``` + +### 2. Run Commands with Profiling + +Each command now supports profiling flags: + +```bash +# Profile dcat +./dcat -profile -profiledir profiles -plain -cfg none /path/to/file.log + +# Profile dgrep with specific profiling types +./dgrep -cpuprofile -memprofile -profiledir profiles -regex "error" /path/to/file.log + +# Profile dmap +./dmap -profile -query "select count(*) from data.csv" +``` + +### 3. Analyze Profiles + +Use dtail-tools for quick analysis: + +```bash +# List all profiles +./dtail-tools profile -mode list + +# Analyze a specific profile +./dtail-tools profile -mode analyze profiles/dcat_cpu_20240101_120000.prof + +# Open web browser with flame graph +./dtail-tools profile -mode analyze profiles/dcat_cpu_*.prof -web + +# You can also use go tool pprof directly: +go tool pprof profiles/dcat_cpu_20240101_120000.prof +``` + +## Profiling Options + +### Command-line Flags + +All dtail commands support these profiling flags: + +- `-cpuprofile`: Enable CPU profiling only +- `-memprofile`: Enable memory profiling only +- `-profile`: Enable both CPU and memory profiling +- `-profiledir `: Directory to store profiles (default: "profiles") + +### Profile Types + +1. **CPU Profile** (`*_cpu_*.prof`) + - Samples CPU usage during execution + - Identifies hot functions and code paths + - Useful for optimizing computational bottlenecks + +2. **Memory Profile** (`*_mem_*.prof`) + - Captures heap allocations at end of execution + - Shows memory usage by function + - Helps identify memory leaks + +3. **Allocation Profile** (`*_alloc_*.prof`) + - Tracks all allocations during execution + - More detailed than memory profile + - Useful for reducing allocation pressure + +## Using with Benchmarks + +### Automated Profiling + +Run profiling using dtail-tools: + +```bash +# Quick profiling with small datasets +./dtail-tools profile -mode quick + +# Full profiling suite +./dtail-tools profile -mode full + +# Profile dmap specifically (with MapReduce format) +./dtail-tools profile -mode dmap +``` + +This tool: +- Generates test data of various sizes +- Profiles dcat, dgrep, and dmap with different workloads +- Stores profiles in the `profiles` directory +- Provides immediate analysis of results + +### Using Make Targets + +```bash +# Quick profiling with immediate results +make profile-quick + +# Full profiling suite +make profile-all + +# Profile dmap specifically +make profile-dmap + +# List available profiles +make profile-list + +# Analyze a specific profile +make profile-analyze PROFILE=profiles/dcat_cpu_*.prof + +# Open web interface for profile +make profile-web PROFILE=profiles/dcat_cpu_*.prof +``` + +### Benchmark Integration + +Run profiling-enabled benchmarks: + +```bash +cd benchmarks +go test -bench="WithProfiling" -benchtime=1x +``` + +### Custom Profile Runner + +Use the profile runner in your benchmarks: + +```go +import "github.com/mimecast/dtail/benchmarks" + +func BenchmarkMyFeature(b *testing.B) { + benchmarks.ProfileBenchmark(b, "MyFeature", "dcat", + "--plain", "--cfg", "none", "testfile.log") +} +``` + +## Profile Analysis + +### Using go tool pprof + +For interactive analysis: + +```bash +# Interactive mode +go tool pprof profiles/dcat_cpu_*.prof + +# Common pprof commands: +# top - Show top functions +# list func - Show source code for function +# web - Generate SVG graph +# peek func - Show callers/callees of function +``` + +Generate visualizations: + +```bash +# Flame graph (requires graphviz) +go tool pprof -http=:8080 profiles/dcat_cpu_*.prof + +# Generate SVG +go tool pprof -svg profiles/dgrep_mem_*.prof > profile.svg + +# Generate text report +go tool pprof -text profiles/dmap_alloc_*.prof > report.txt +``` + +### Using dtail-tools profile + +The dtail-tools profile command provides quick summaries: + +```bash +# List all profiles +./dtail-tools profile -mode list + +# Analyze specific profile +./dtail-tools profile -mode analyze profiles/dcat_cpu_20240101_120000.prof + +# Get help +./dtail-tools profile -h +``` + +## Optimization Workflow + +1. **Baseline Performance** + ```bash + # Run benchmarks without profilin