From d3e98fb73a5966736f9c5a9479bad27eff2cccd3 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Wed, 21 Jun 2023 11:26:31 +0300 Subject: rename samples to examples --- doc/installation.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) (limited to 'doc') diff --git a/doc/installation.md b/doc/installation.md index 0f6143b..a2c271a 100644 --- a/doc/installation.md +++ b/doc/installation.md @@ -53,18 +53,18 @@ uid=1001(dserver) 1001=670(dserver) groups=1001(dserver) % sudo chown -R dserver:dserver /var/run/dserver ``` -4. Install the ``dtail.json`` config to ``/etc/dserver/dtail.json``. An example can be found [here](../samples/dtail.json.sample). +4. Install the ``dtail.json`` config to ``/etc/dserver/dtail.json``. An example can be found [here](../examples/dtail.json.example). ```console % sudo mkdir /etc/dserver -% curl https://raw.githubusercontent.com/mimecast/dtail/master/samples/dtail.json.sample | +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dtail.json.example | sudo tee /etc/dserver/dtail.json ``` -5. It is recommended to configure DTail server as a service to ``systemd``. An example unit file for ``systemd`` can be found [here](../samples/dserver.service.sample). +5. It is recommended to configure DTail server as a service to ``systemd``. An example unit file for ``systemd`` can be found [here](../examples/dserver.service.example). ```console -% curl https://raw.githubusercontent.com/mimecast/dtail/master/samples/dserver.service.sample | +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver.service.example | sudo tee /etc/systemd/system/dserver.service % sudo systemctl daemon-reload % sudo systemctl enable dserver @@ -97,15 +97,15 @@ To start the DTail server via ``systemd`` run: The DTail server now runs as a ``systemd`` service under system user ``dserver``. However, the system user ``dserver`` has no permissions to read the SSH public keys from ``/home/USER/.ssh/authorized_keys``. Therefore, no user would be able to establish an SSH session to DTail server. As an alternative path DTail server also checks for public SSH key files in ``/var/run/dserver/cache/USER.authorized_keys``. -It is recommended to execute [update_key_cache.sh](../samples/update_key_cache.sh.sample) periodically to update the key cache. In case you manage your public SSH keys via Puppet you could subscribe the script to corresponding module. Or alternatively just configure a cron job or a systemd timer to run every once in a while, e.g. every 30 minutes: +It is recommended to execute [update_key_cache.sh](../examples/update_key_cache.sh.example) periodically to update the key cache. In case you manage your public SSH keys via Puppet you could subscribe the script to corresponding module. Or alternatively just configure a cron job or a systemd timer to run every once in a while, e.g. every 30 minutes: ```console -% curl https://raw.githubusercontent.com/mimecast/dtail/master/samples/update_key_cache.sh.sample | +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/update_key_cache.sh.example | sudo tee /var/run/dserver/update_key_cache.sh % sudo chmod 755 /var/run/dserver/update_key_cache.sh -% curl https://raw.githubusercontent.com/mimecast/dtail/master/samples/dserver-update-keycache.service.sample | +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver-update-keycache.service.example | sudo tee /etc/systemd/system/dserver-update-keycache.service -% curl https://raw.githubusercontent.com/mimecast/dtail/master/samples/dserver-update-keycache.timer.sample | +% curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dserver-update-keycache.timer.example | sudo tee /etc/systemd/system/dserver-update-keycache.timer % sudo systemctl daemon-reload % sudo systemctl start dserver-update-keycache.service @@ -119,7 +119,7 @@ Now you should be able to use DTail client like outlined in the [Quick Starting # Monitor it -To verify that DTail server is up and running and functioning as expected, you should configure the Nagios check [check_dserver.sh](../samples/check_dserver.sh.sample) in your monitoring system. The check has to be executed locally on the server (e.g. via NRPE). How to configure the monitoring system in detail is out of scope of this guide. +To verify that DTail server is up and running and functioning as expected, you should configure the Nagios check [check_dserver.sh](../examples/check_dserver.sh.example) in your monitoring system. The check has to be executed locally on the server (e.g. via NRPE). How to configure the monitoring system in detail is out of scope of this guide. ```console % ./check_dserver.sh -- cgit v1.2.3 From cc17406ae125f40c4100afbedcd8c40ba319cfa8 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Mon, 18 Jul 2022 10:20:59 +0100 Subject: Include example for "outfile append". --- doc/examples.md | 8 ++++++++ 1 file changed, 8 insertions(+) (limited to 'doc') diff --git a/doc/examples.md b/doc/examples.md index 56744f5..26ce002 100644 --- a/doc/examples.md +++ b/doc/examples.md @@ -65,6 +65,14 @@ Here is another example: ![dtail-map](dtail-map2.gif "Tail mapreduce example 2") +You can also continuously append the results to a CSV file by adding `outfile append filename.csv` to the query: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select ... outfile append result.csv' +``` + ## How to use `dcat` The following example demonstrates how to cat files (display the full content of the files) of multiple servers at once. -- cgit v1.2.3 From bf39452c1f9a06d9f4e6eb3a06a23068a2451ca5 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Tue, 5 Sep 2023 14:13:48 +0300 Subject: Update creating your own logformat docs, to reflect the recent changes. --- doc/logformats.md | 61 +++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 43 insertions(+), 18 deletions(-) (limited to 'doc') diff --git a/doc/logformats.md b/doc/logformats.md index c3f0c63..9d4e55d 100644 --- a/doc/logformats.md +++ b/doc/logformats.md @@ -21,15 +21,26 @@ By default, DTail will use the `default` log format. You can override the log fo % dmap --files /var/log/example.log --query 'from EXAMPLE select ....queryhere.... logformat generickv' ``` -Alternatively, you can override the default log format with `MapreduceLogFormat` in the Server section of `dtail.json`. +You can override the default log format with `MapreduceLogFormat` in the Server section of `dtail.json`. ## Under the hood: generickv As an example, let's have a look at the `generickv` log format's implementation. It's located at `internal/mapr/logformat/generickv.go`: -```shell -// MakeFieldsGENERIGKV is the generic key-value logfile parser. -func (p *Parser) MakeFieldsGENERIGKV(maprLine string) (map[string]string, error) { +```go +type genericKVParser struct { + defaultParser +} + +func newGenericKVParser(hostname, timeZoneName string, timeZoneOffset int) (*genericKVParser, error) { + defaultParser, err := newDefaultParser(hostname, timeZoneName, timeZoneOffset) + if err != nil { + return &genericKVParser{}, err + } + return &genericKVParser{defaultParser: *defaultParser}, nil +} + +func (p *genericKVParser) MakeFields(maprLine string) (map[string]string, error) { splitted := strings.Split(maprLine, protocol.FieldDelimiter) fields := make(map[string]string, len(splitted)) @@ -44,7 +55,7 @@ func (p *Parser) MakeFieldsGENERIGKV(maprLine string) (map[string]string, error) for _, kv := range splitted[0:] { keyAndValue := strings.SplitN(kv, "=", 2) if len(keyAndValue) != 2 { - // dlog.Common.Debug("Unable to parse key-value token, ignoring it", kv) + //dlog.Common.Debug("Unable to parse key-value token, ignoring it", kv) continue } fields[keyAndValue[0]] = keyAndValue[1] @@ -101,26 +112,40 @@ These variables may only exist in the DTail default log format (see `internal/ma * `$pid` - DTail server process ID * `$uptime` - DTail server uptime -## Implementing your own log format +## Implementing your own log format `Foo` -All what needs to be done is to place your own implementation into the `logformat` source directory. As a template, you can copy an existing format ... +What needs to be done is to place your own implementation into the `logformat` source directory. As a template, you can copy an existing format ... ```shell -% cp internal/mapr/logformat/generic.go internal/mapr/logformat/yourcustomformat.go +% cp internal/mapr/logformat/generic.go internal/mapr/logformat/foo.go ``` -... and replace `GENERIGKV` with your format's name in capital letters (the method name string is used by DTail to reflect the log format parser method, so it is important to name it correctly): +... and replace `generic` ` with your format's name `foo`: + +```go +package logformat + +type fooParser struct { + defaultParser +} + +func newFooParser(hostname, timeZoneName string, timeZoneOffset int) (*fooParser, error) { + defaultParser, err := newDefaultParser(hostname, timeZoneName, timeZoneOffset) + if err != nil { + return &fooParser{}, err + } + return &fooParser{defaultParser: *defaultParser}, nil +} + +func (p *fooParser) MakeFields(maprLine string) (map[string]string, error) { + fields := make(map[string]string, 3) + + .. + + .. -```shell -// MakeFieldsCUSTOMLOGFORMAT is your own custom log format. -func (p *Parser) MakeFieldsCUSTOMLOGFORMAT(maprLine string) (map[string]string, error) { - // .. Your own format implementation goes here - // .. you can parse maprLine and store values into the fields map. -.. -. -. return fields, nil } ``` -Once done, recompile DTail. DTail now understands `... logformat customlogformat` (see "Seleting a log format" above). +Once done, recompile DTail. DTail now understands `... logformat foo` (see "Seleting a log format" above). -- cgit v1.2.3 From d427574824a1a4b6a5a0e15a3696f3df1c698424 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Tue, 5 Sep 2023 14:39:21 +0300 Subject: Add `custom1` and `custom2` log formats. --- doc/logformats.md | 2 ++ 1 file changed, 2 insertions(+) (limited to 'doc') diff --git a/doc/logformats.md b/doc/logformats.md index 9d4e55d..839b050 100644 --- a/doc/logformats.md +++ b/doc/logformats.md @@ -148,4 +148,6 @@ func (p *fooParser) MakeFields(maprLine string) (map[string]string, error) { } ``` +Next, `NewParser(...)` in `internal/mapr/logformat/parser.go` needs to be extended, so that the new log format is part of the switch statement. If you don't want to edit `parser.go` then you could instead use `custom1` or `custom2` log formats, there are ready templates available in the `logformat` package. + Once done, recompile DTail. DTail now understands `... logformat foo` (see "Seleting a log format" above). -- cgit v1.2.3 From 05ef7d56f945242fecb97cf03a3a9abab47013ee Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Tue, 5 Sep 2023 17:41:02 +0300 Subject: add CSV aggr example to docs --- doc/examples.md | 19 +++++++++++++++++++ doc/logformats.md | 4 +++- 2 files changed, 22 insertions(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/examples.md b/doc/examples.md index 26ce002..4937cc5 100644 --- a/doc/examples.md +++ b/doc/examples.md @@ -151,6 +151,25 @@ You can also use a file input pipe as follows: dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' ``` +### Aggregating CSV files + +In essence, this works exactly like aggregating logs. All files operated on must be valid CSV files and the first line of the CSV must be the header. E.g.: + +```shell +% cat example.csv +name,lastname,age,profession +Michael,Jordan,40,Basketball player +Michael,Jackson,100,Singer +Albert,Einstein,200,Physician +% dmap --query 'select lastname,name where age > 40 logformat csv outfile result.csv' example.csv +% cat result.csv +lastname,name +Jackson,Michael +Einstein,Albert +``` + +DMap can also be used to query and aggregate CSV files from remote servers. + ### Other serverless commands The serverless mode works transparently with all other DTail commands. Here are some examples: diff --git a/doc/logformats.md b/doc/logformats.md index 839b050..dbf2051 100644 --- a/doc/logformats.md +++ b/doc/logformats.md @@ -10,8 +10,10 @@ You could either make your application follow the DTail default log format, or y The following log formats are currently available out of the box: * `default` - The default DTail log format -* `generic` - A generic log format with a very simple set of fields +* `generic` - A generic log format with a simple set of fields * `generickv` - A simple log format expecting all log lines in form of `field1=value1|field2=value2|...` +* `csv` - A simple CSV format expecting all files a comma separated CSV file. The first line of the file must be the CSV header. +* `custom1` and `custom2` - Customizable log formats. ### Selecting a log format -- cgit v1.2.3