From 3004a7100e325c006971cc2e8d0f157338c0ce5c Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Wed, 22 Jul 2026 23:52:52 +0300 Subject: =?UTF-8?q?docs:=20DTail=20fork=20=E2=80=94=20documentation,=20age?= =?UTF-8?q?nt=20guide,=20example=20configs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Squashed development of the documentation and example configuration: - AGENTS.md / CLAUDE.md: repository guide describing build/test/benchmark/PGO workflows and the single default read/output path (formerly "turbo"). - doc/ and docs/: query-language reference, log formats, auth-key fast reconnect, journal source reads, performance analyses (dated point-in-time records kept under historical-note disclaimers), and the turbo-vs-normal benchmark report with its result CSVs. - README.md updates; examples/ config + JSON schema aligned with the current Output* server tuning fields (the removed TurboBoost* keys dropped). Co-Authored-By: Claude Opus 4.8 --- doc/querylanguage.md | 40 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) (limited to 'doc/querylanguage.md') diff --git a/doc/querylanguage.md b/doc/querylanguage.md index c3e567e..a405f35 100644 --- a/doc/querylanguage.md +++ b/doc/querylanguage.md @@ -52,7 +52,7 @@ STRINGOPERATOR := eq|ne|contains|ncontains|lacks|hasprefix|nhasprefix|hassuffix| ORDERFIELD := FIELD|AGGREGATION(FIELD) SET := $VARIABLE = FLOAT|STRING|FIELD|FUNCTION(FIELD) LOGFORMAT := default|generic|generickv|... -AGGREGATION := count|sum|min|max|avg|last|len +AGGREGATION := count|sum|min|max|avg|last|len|percentage|percentile FUNCTION := md5sum|maskdigits ``` @@ -61,3 +61,41 @@ FUNCTION := md5sum|maskdigits * `rorder` stands for reverse order. * `lacks` is an alias for `ncontains` (not contains). * Available fields (variables and barewords) vary from the log format used. Check out the [log format](./logformats.md) documentation for more information. +* `percentage(field)` returns the selected group's share of the total for that field across all groups. For non-negative inputs, the result is between 0 and 100; with mixed positive and negative values, it can fall outside that range. +* `percentile(field)` returns the percentile rank of the selected group's value among all grouped values for that field, also expressed as a value between 0 and 100. Equal values share the same rank. + +## Selecting the log format and dynamic fields + +Two things commonly trip people up when a `$field`-style reference "does not +resolve" while positional/built-in fields work. Both are by design: + +1. **Dynamic `key=value` fields are barewords, not `$`-variables.** A log line + like `...|service=web|bytes=100` exposes `service` and `bytes` as *barewords*. + Query them as `select service,sum(bytes)` — **not** `$service`/`$bytes`. The + `$` prefix is reserved for values DTail sets itself (e.g. `$time`, `$hostname`, + `$line`). A `$name` that is not one of those built-ins silently resolves to + the empty string, which is exactly what "did not resolve" looks like: + everything collapses into a single empty group. To catch this early, the + client prints a plan-time warning to stderr for every `$`-variable the + selected parser cannot populate, e.g. + `warning: $service is not a known variable for log format "default"; did you + mean bareword service?`. It is only a warning (resolution behaviour is + unchanged), and it is never emitted for barewords, for built-ins like + `$empty`, or for variables defined via a `set` clause. + +2. **The `from TABLE` clause selects the rich parser.** Although `from TABLE` is + written as optional in the grammar above, omitting it (and not passing an + explicit `logformat`) downgrades the query to the `generic` log format, which + exposes only the common variables (`$line`, `$hostname`, ...) and **no** + dynamic `key=value` fields and **no** default-format `$`-variables such as + `$time`. To query DTail's own default-format logs (lines containing + `MAPREDUCE:STATS`), use `from STATS`, for example: + + ```shell + % dmap --files /var/log/dserver/dserver.log \ + --query 'from STATS select $hostname,max($goroutines),lifetimeConnections group by $hostname' + ``` + + Alternatively, name the parser explicitly with the `logformat` keyword (e.g. + `logformat generickv`), which works regardless of the `from` clause. See the + [log formats](./logformats.md) documentation for details. -- cgit v1.2.3