documenting how to implement a custom log format

1 files changed, 77 insertions, 23 deletions

diff --git a/doc/logformats.md b/doc/logformats.md
index 06fff76..dd49c7c 100644
--- a/doc/logformats.md
+++ b/doc/logformats.md
@@ -1,19 +1,17 @@
 Log Formats
 ===========
 
-You may have looked at the [DTail Query Language](./querylanguage.md) and wondered how to make DTail understand your own log formats. Otherwise, DTail won't be able to extract information from your logs (e.g. extract fields and variables from your log lines to be used in the query language).
+You may have looked at the [DTail Query Language](./querylanguage.md) and wondered how to make DTail understand your own log format(s). If DTail doesn't know your log format, it won't be able to extract much useful information from your logs. This information then can be used as fields (e.g. variables) by the Query Language.
 
-You could either make your application follow the DTail default log format, or you would need to implement a custom log format in Go.
+You could either make your application follow the DTail default log format, or you would need to implement a custom log format. Have a look at `./integrationtests/mapr_testdata.log` for an example a log file in the DTail default format.
 
-## Current log formats
+## Available log formats
 
 The following log formats are currently available out of the box:
 
-* `default` - The default DTail log format.
-* `generic` - A generic log format with a very simple set of fields.
-* `generickv` - A simple log format expecting all log lines in form of `field1=value1|field2=value2|...`.
-
-For details, have a look at the implementations at `./internal/mapr/logformat/`.
+* `default` - The default DTail log format
+* `generic` - A generic log format with a very simple set of fields
+* `generickv` - A simple log format expecting all log lines in form of `field1=value1|field2=value2|...`
 
 ### Selecting a log format
 
@@ -23,43 +21,75 @@ By default, DTail will use the `default` log format. You can override the log fo
 % dmap --files /var/log/example.log --query 'from EXAMPLE select ....queryhere.... logformat generickv'
 ```
 
-Alternatively, you can override the default log format via `MapreduceLogFormat` in the Server section of `dtail.json`.
+Alternatively, you can override the default log format with `MapreduceLogFormat` in the Server section of `dtail.json`.
 
-## Log format fields
+## Under the hood: generickv
 
-TODO: Difference between field and variables.
+As an example, let's have a look at the `generickv` log format's implementation. It's located at `internal/mapr/logformat/generickv.go`:
 
-## Log format variables
+```shell
+// MakeFieldsGENERIGKV is the generic key-value logfile parser.
+func (p *Parser) MakeFieldsGENERIGKV(maprLine string) (map[string]string, error) {
+	splitted := strings.Split(maprLine, protocol.FieldDelimiter)
+	fields := make(map[string]string, len(splitted))
+
+	fields["*"] = "*"
+	fields["$line"] = maprLine
+	fields["$empty"] = ""
+	fields["$hostname"] = p.hostname
+	fields["$server"] = p.hostname
+	fields["$timezone"] = p.timeZoneName
+	fields["$timeoffset"] = p.timeZoneOffset
+
+	for _, kv := range splitted[0:] {
+		keyAndValue := strings.SplitN(kv, "=", 2)
+		if len(keyAndValue) != 2 {
+			// dlog.Common.Debug("Unable to parse key-value token, ignoring it", kv)
+			continue
+		}
+		fields[strings.ToLower(keyAndValue[0])] = keyAndValue[1]
+	}
+
+	return fields, nil
+}
+```
+
+... whereas:
 
-This is the list of pre-defined variables. Please note that these vary depending on the log format used. 
+* `maprLine` is the whole raw log line to be parsed by the log format.
+* `protocol.FieldDelimiter` is the field delimiter used by the log format, here: `|`.
+* All field names starting with `$` are variables. They store some custom values.
+* All other fields are bareword-fields and are extracted from the log lines directly, e.g. `field1=value1|field2=value2|...`
+
+## Log format variables
 
 ### Common variables:
 
-The common variables may exist in all log formats.
+The common variables may exist in all log formats:
 
 * `$empty` - The empty string `""`
 * `$hostname` - The server FQDN
-* `$line` - The current log line
+* `$line` - The whole log line
 * `$server` - Alias for `$hostname`
 * `$timeoffset` -  Offset of $timezone
 * `$timezone` -  The current time zone
-* `*` - Special placeholder 
+* `*` - Special placeholder. E.g. sometimes used by the query language to group by everything.
 
 ### Default log format variables:
 
-These variables may only exist when your logs are in the DTail default log format:
+These variables may only exist in the DTail default log format (see `internal/mapr/logformat/default.go` more details):
 
 *Date and time:*
 
-* `$hour` - The current hour in format HH
-* `$minute` - The current minute in format MM
-* `$second` - The current second in format SS.
-* `$time` - The current time in format YYYYMMDD-HHMMSS
+* `$hour` - The hour in format HH
+* `$minute` - The minute in format MM
+* `$second` - The second in format SS.
+* `$time` - The time in format YYYYMMDD-HHMMSS
 
 *Log level/severity:*
 
 * `$loglevel` - Alias for `$severity`
-* `$severity` - The log severity
+* `$severity` - The log severity, one of `FATAL`, `ERROR`, `WARN`, `INFO`, `VERBOSE`, `DEBUG`, `DEVEL`, `TRACE`
 
 *System and Go runtime:*
 
@@ -67,6 +97,30 @@ These variables may only exist when your logs are in the DTail default log forma
 * `$cgocalls` - Num of DTail server CGo calls
 * `$cpus` - Num of DTail server CPUs used
 * `$goroutines` - Num of DTail server Goroutines used
-* `$loadavg` - 1 min. average load average
+* `$loadavg` - 1 min. load average
 * `$pid` - DTail server process ID
 * `$uptime` - DTail server uptime
+
+## Implementing your own log format
+
+All what needs to be done is to place your own implementation into the `logformat` source directory. As a template, you can copy an existing format ...
+
+```shell
+% cp internal/mapr/logformat/generic.go internal/mapr/logformat/yourcustomformat.go
+```
+
+... and replace `GENERIGKV` with your format's name in capital letters (the method name string is used by DTail to reflect the log format parser method, so it is important to name it correctly):
+
+```shell
+// MakeFieldsCUSTOMLOGFORMAT is your own custom log format.
+func (p *Parser) MakeFieldsCUSTOMLOGFORMAT(maprLine string) (map[string]string, error) {
+	// .. Your own format implementation goes here
+	// .. you can parse maprLine and store values into the fields map. 
+..
+.
+.
+	return fields, nil
+}
+```
+
+Once done, recompile DTail. DTail now understands `... logformat customlogformat` (see "Seleting a log format" above).