diff options
Diffstat (limited to 'doc/examples.md')
| -rw-r--r-- | doc/examples.md | 159 |
1 files changed, 133 insertions, 26 deletions
diff --git a/doc/examples.md b/doc/examples.md index 91ab7f2..56744f5 100644 --- a/doc/examples.md +++ b/doc/examples.md @@ -1,67 +1,174 @@ Examples ======== -This page demonstrates the primary usage of DTail. Please also see ``dtail --help`` for more available options. +This page demonstrates the primary usage of DTail. Please also see `--help` for more available options. -# How to use ``dtail`` +## Table of contents -## Tailing logs +* How to use `dtail` to follow logs +* How to use `dtail` to aggregate logs +* How to use `dcat` +* How to use `dgrep` +* How to use `dmap` +* How to use the DTail serverless mode -The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all records containing the string ``STAT``. Any other Go compatible regular expression can be used instead of ``STAT``. +## How to use `dtail` + +### Following logs + +The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all records containing the string `INFO`. Any other Go compatible regular expression can also be used instead of `INFO`. ```shell -% dtail --servers serverlist.txt --files "/var/log/service/*.log" --regex STAT +% dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log" ``` +Hint: you can also provide a comma separated server list, e.g.: `servers server1.example.org,server2.example.org:PORT,...` +  -## Aggregating logs +Hint: You can also use the shorthand version (omitting the `--files`) + +```shell +% dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log" +``` + +### Aggregating logs + +To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the result to standard output. -To run ad-hoc MapReduce aggregations on newly written log lines, you also must add a query. The following example follows all remote log lines and prints out every 5 seconds the top 10 servers with the most average free memory. To run a MapReduce query across log lines written in the past, please use the ``dmap`` command instead. +Hint: To run a map-reduce query across log lines written in the past, please use the `dmap` command instead. ```shell -% dtail --servers serverlist.txt \ - --query 'select avg(memfree), $hostname from MCVMSTATS group by $hostname order by avg(memfree) limit 10 interval 5' \ - --files '/var/log/service/*.log' +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select sum($goroutines),sum($cgocalls),last($time),max(lifetimeConnections)' ``` -For MapReduce queries to work, you have to ensure that DTail supports your log format. You can either use the ones already defined in ``internal/mapr/log format`` or add an extension to support a custom log format. +Beware: For map-reduce queries to work, you have to ensure that DTail supports your log format. Check out the [query language](./querylanguage.md) and [log formats](./logformats.md) for more information.  -# How to use ``dcat`` +Hint: You can also use the shorthand version: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + 'from STATS select sum($goroutines),sum($cgocalls),last($time),max(lifetimeConnections)' +``` +Here is another example: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' +``` + + -The following example demonstrates how to cat files (display the full content of the files) of multiple servers at once. The servers are provided as a comma-separated list this time. +## How to use `dcat` + +The following example demonstrates how to cat files (display the full content of the files) of multiple servers at once. + +As you can see in this example, a DTail client also creates a local log file of all received data in `~/log`. You can also use the `noColor` and `-plain` flags (this all also work with other DTail commands than `dcat`). ```shell -% dcat --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org \ - --files /etc/hostname +% dcat --servers serverlist.txt --files /etc/hostname ```  -# How to use ``dgrep`` +Hint: You can also use the shorthand version: + +```shell +% dcat --servers serverlist.txt /etc/hostname +``` + +## How to use `dgrep` -The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after the swap partition in ``/etc/fstab``. We do that only on the first 20 servers from ``serverlist.txt``. ``dgrep`` is also very useful for searching log files of the past. +The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in `/etc/passwd` This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the `-before`, `-after` and `-max` flags (see animation). ```shell -% dgrep --servers <(head -n 20 serverlist.txt) \ - --files /etc/fstab \ - --regex swap +% dgrep --servers server1.example.org:2223 \ + --files /etc/passwd \ + --regex nologin ``` +Generally, `dgrep` is also a very useful way to search historic application logs for certain content. +  -# How to use ``dmap`` +Hint: `-regex` is an alias for `-grep`. + +## How to use `dmap` -To run a MapReduce aggregation over logs written in the past, the ``dmap`` command can be used. For example, the following command aggregates all MapReduce fields of all the records and calculates the average memory free grouped by day of the month, hour, minute and the server hostname. ``dmap`` will print interim results every few seconds. The final product, however, will be written to file ``mapreduce.csv``. +To run a map-reduce aggregation over logs written in the past, the `dmap` command can be used. The following example aggregates all map-reduce fields `dmap` will print interim results every few seconds. You can also write the result to an CSV file by adding `outfile result.csv` to the query. ```shell -% dmap --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org,serv-021.lan.example.org,serv-022.lan.example.org,serv-023.lan.example.org \ - --query 'select avg(memfree), $day, $hour, $minute, $hostname from MCVMSTATS group by $day, $hour, $minute, $hostname order by avg(memfree) limit 10 outfile mapreduce.csv' \ - --files "/var/log/service/*.log" +% dmap --servers serverlist.txt \ + --files '/var/log/dserver/*.log' + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' ``` -Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in ``internal/mapr/log format`` or add an extension to support a custom log format. +Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in `internal/mapr/logformat` or add an extension to support a custom log format. The example here works out of the box though, as DTail understands its own log format already.  + +## How to use the DTail serverless mode + +Until now, all examples so far required to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files. + +The serverless mode does not require any `dserver` up and running and therefore there is no networking/SSH involved. + +All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode. + +### Serverless map-reduce query + +The following `dmap` example is the same as the previously shown one, but the difference is that it operates on a local log file directly: + +```shell +% dmap --files /var/log/dserver/dserver.log + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +As a shorthand version the following command can be used: + +```shell +% dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' /var/log/dsever/dserver.log +``` + +You can also use a file input pipe as follows: + +```shell +% cat /var/log/dserver/dserver.log | \ + dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +### Other serverless commands + +The serverless mode works transparently with all other DTail commands. Here are some examples: + +```shell +dtail /var/log/dserver/dserver.log +``` + +```shell +dtail --logLevel trace /var/log/dserver/dserver.log +``` + +```shell +dcat /etc/passwd +``` + +```shell +dcat --plain /etc/passwd > /etc/test +# Should show no differences. +diff /etc/test /etc/passwd +``` + +```shell +dgrep --regex ERROR --files /var/log/dserver/dsever.log +``` + +```shell +dgrep --before 10 --after 10 --max 10 --grep ERROR /var/log/dserver/dsever.log +``` |