From a7cfc60cd607f7c1b33abf1e93ae476ada171826 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Tue, 7 Dec 2021 09:23:00 +0000 Subject: initial logformats and querylanguage documentation --- doc/logformats.md | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 doc/logformats.md (limited to 'doc/logformats.md') diff --git a/doc/logformats.md b/doc/logformats.md new file mode 100644 index 0000000..a9f9865 --- /dev/null +++ b/doc/logformats.md @@ -0,0 +1,4 @@ +Log Formats +=========== + +TODO: document log formats. -- cgit v1.2.3 From bfdd5fb6f0cfac9acce62d70010355c801f00998 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 9 Dec 2021 10:22:25 +0000 Subject: Documenting log formats --- doc/logformats.md | 70 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 69 insertions(+), 1 deletion(-) (limited to 'doc/logformats.md') diff --git a/doc/logformats.md b/doc/logformats.md index a9f9865..06fff76 100644 --- a/doc/logformats.md +++ b/doc/logformats.md @@ -1,4 +1,72 @@ Log Formats =========== -TODO: document log formats. +You may have looked at the [DTail Query Language](./querylanguage.md) and wondered how to make DTail understand your own log formats. Otherwise, DTail won't be able to extract information from your logs (e.g. extract fields and variables from your log lines to be used in the query language). + +You could either make your application follow the DTail default log format, or you would need to implement a custom log format in Go. + +## Current log formats + +The following log formats are currently available out of the box: + +* `default` - The default DTail log format. +* `generic` - A generic log format with a very simple set of fields. +* `generickv` - A simple log format expecting all log lines in form of `field1=value1|field2=value2|...`. + +For details, have a look at the implementations at `./internal/mapr/logformat/`. + +### Selecting a log format + +By default, DTail will use the `default` log format. You can override the log format with the `logformat` keyword: + +```shell +% dmap --files /var/log/example.log --query 'from EXAMPLE select ....queryhere.... logformat generickv' +``` + +Alternatively, you can override the default log format via `MapreduceLogFormat` in the Server section of `dtail.json`. + +## Log format fields + +TODO: Difference between field and variables. + +## Log format variables + +This is the list of pre-defined variables. Please note that these vary depending on the log format used. + +### Common variables: + +The common variables may exist in all log formats. + +* `$empty` - The empty string `""` +* `$hostname` - The server FQDN +* `$line` - The current log line +* `$server` - Alias for `$hostname` +* `$timeoffset` - Offset of $timezone +* `$timezone` - The current time zone +* `*` - Special placeholder + +### Default log format variables: + +These variables may only exist when your logs are in the DTail default log format: + +*Date and time:* + +* `$hour` - The current hour in format HH +* `$minute` - The current minute in format MM +* `$second` - The current second in format SS. +* `$time` - The current time in format YYYYMMDD-HHMMSS + +*Log level/severity:* + +* `$loglevel` - Alias for `$severity` +* `$severity` - The log severity + +*System and Go runtime:* + +* `$caller` - DTail server caller of the logger +* `$cgocalls` - Num of DTail server CGo calls +* `$cpus` - Num of DTail server CPUs used +* `$goroutines` - Num of DTail server Goroutines used +* `$loadavg` - 1 min. average load average +* `$pid` - DTail server process ID +* `$uptime` - DTail server uptime -- cgit v1.2.3 From ff18a1aaa338de54a4e0a7f88cea66cbe8a94879 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Wed, 15 Dec 2021 16:06:48 +0000 Subject: documenting how to implement a custom log format --- doc/logformats.md | 100 +++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 77 insertions(+), 23 deletions(-) (limited to 'doc/logformats.md') diff --git a/doc/logformats.md b/doc/logformats.md index 06fff76..dd49c7c 100644 --- a/doc/logformats.md +++ b/doc/logformats.md @@ -1,19 +1,17 @@ Log Formats =========== -You may have looked at the [DTail Query Language](./querylanguage.md) and wondered how to make DTail understand your own log formats. Otherwise, DTail won't be able to extract information from your logs (e.g. extract fields and variables from your log lines to be used in the query language). +You may have looked at the [DTail Query Language](./querylanguage.md) and wondered how to make DTail understand your own log format(s). If DTail doesn't know your log format, it won't be able to extract much useful information from your logs. This information then can be used as fields (e.g. variables) by the Query Language. -You could either make your application follow the DTail default log format, or you would need to implement a custom log format in Go. +You could either make your application follow the DTail default log format, or you would need to implement a custom log format. Have a look at `./integrationtests/mapr_testdata.log` for an example a log file in the DTail default format. -## Current log formats +## Available log formats The following log formats are currently available out of the box: -* `default` - The default DTail log format. -* `generic` - A generic log format with a very simple set of fields. -* `generickv` - A simple log format expecting all log lines in form of `field1=value1|field2=value2|...`. - -For details, have a look at the implementations at `./internal/mapr/logformat/`. +* `default` - The default DTail log format +* `generic` - A generic log format with a very simple set of fields +* `generickv` - A simple log format expecting all log lines in form of `field1=value1|field2=value2|...` ### Selecting a log format @@ -23,43 +21,75 @@ By default, DTail will use the `default` log format. You can override the log fo % dmap --files /var/log/example.log --query 'from EXAMPLE select ....queryhere.... logformat generickv' ``` -Alternatively, you can override the default log format via `MapreduceLogFormat` in the Server section of `dtail.json`. +Alternatively, you can override the default log format with `MapreduceLogFormat` in the Server section of `dtail.json`. -## Log format fields +## Under the hood: generickv -TODO: Difference between field and variables. +As an example, let's have a look at the `generickv` log format's implementation. It's located at `internal/mapr/logformat/generickv.go`: -## Log format variables +```shell +// MakeFieldsGENERIGKV is the generic key-value logfile parser. +func (p *Parser) MakeFieldsGENERIGKV(maprLine string) (map[string]string, error) { + splitted := strings.Split(maprLine, protocol.FieldDelimiter) + fields := make(map[string]string, len(splitted)) + + fields["*"] = "*" + fields["$line"] = maprLine + fields["$empty"] = "" + fields["$hostname"] = p.hostname + fields["$server"] = p.hostname + fields["$timezone"] = p.timeZoneName + fields["$timeoffset"] = p.timeZoneOffset + + for _, kv := range splitted[0:] { + keyAndValue := strings.SplitN(kv, "=", 2) + if len(keyAndValue) != 2 { + // dlog.Common.Debug("Unable to parse key-value token, ignoring it", kv) + continue + } + fields[strings.ToLower(keyAndValue[0])] = keyAndValue[1] + } + + return fields, nil +} +``` + +... whereas: -This is the list of pre-defined variables. Please note that these vary depending on the log format used. +* `maprLine` is the whole raw log line to be parsed by the log format. +* `protocol.FieldDelimiter` is the field delimiter used by the log format, here: `|`. +* All field names starting with `$` are variables. They store some custom values. +* All other fields are bareword-fields and are extracted from the log lines directly, e.g. `field1=value1|field2=value2|...` + +## Log format variables ### Common variables: -The common variables may exist in all log formats. +The common variables may exist in all log formats: * `$empty` - The empty string `""` * `$hostname` - The server FQDN -* `$line` - The current log line +* `$line` - The whole log line * `$server` - Alias for `$hostname` * `$timeoffset` - Offset of $timezone * `$timezone` - The current time zone -* `*` - Special placeholder +* `*` - Special placeholder. E.g. sometimes used by the query language to group by everything. ### Default log format variables: -These variables may only exist when your logs are in the DTail default log format: +These variables may only exist in the DTail default log format (see `internal/mapr/logformat/default.go` more details): *Date and time:* -* `$hour` - The current hour in format HH -* `$minute` - The current minute in format MM -* `$second` - The current second in format SS. -* `$time` - The current time in format YYYYMMDD-HHMMSS +* `$hour` - The hour in format HH +* `$minute` - The minute in format MM +* `$second` - The second in format SS. +* `$time` - The time in format YYYYMMDD-HHMMSS *Log level/severity:* * `$loglevel` - Alias for `$severity` -* `$severity` - The log severity +* `$severity` - The log severity, one of `FATAL`, `ERROR`, `WARN`, `INFO`, `VERBOSE`, `DEBUG`, `DEVEL`, `TRACE` *System and Go runtime:* @@ -67,6 +97,30 @@ These variables may only exist when your logs are in the DTail default log forma * `$cgocalls` - Num of DTail server CGo calls * `$cpus` - Num of DTail server CPUs used * `$goroutines` - Num of DTail server Goroutines used -* `$loadavg` - 1 min. average load average +* `$loadavg` - 1 min. load average * `$pid` - DTail server process ID * `$uptime` - DTail server uptime + +## Implementing your own log format + +All what needs to be done is to place your own implementation into the `logformat` source directory. As a template, you can copy an existing format ... + +```shell +% cp internal/mapr/logformat/generic.go internal/mapr/logformat/yourcustomformat.go +``` + +... and replace `GENERIGKV` with your format's name in capital letters (the method name string is used by DTail to reflect the log format parser method, so it is important to name it correctly): + +```shell +// MakeFieldsCUSTOMLOGFORMAT is your own custom log format. +func (p *Parser) MakeFieldsCUSTOMLOGFORMAT(maprLine string) (map[string]string, error) { + // .. Your own format implementation goes here + // .. you can parse maprLine and store values into the fields map. +.. +. +. + return fields, nil +} +``` + +Once done, recompile DTail. DTail now understands `... logformat customlogformat` (see "Seleting a log format" above). -- cgit v1.2.3