diff options
| author | Paul Buetow <paul@buetow.org> | 2021-11-07 14:43:21 +0200 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2021-11-07 14:43:21 +0200 |
| commit | 84166ab05f72ed23ada3802fc0baf0cec5eb1233 (patch) | |
| tree | 414927eb2f3f4b336b5c730a1f3b03a8f39b96d5 /doc | |
| parent | d9baf981c567e61c57da49cc01f5e1ef2997769f (diff) | |
add serverless examples
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/examples.md | 102 |
1 files changed, 86 insertions, 16 deletions
diff --git a/doc/examples.md b/doc/examples.md index 5dd898d..803b1af 100644 --- a/doc/examples.md +++ b/doc/examples.md @@ -1,31 +1,40 @@ Examples ======== -This page demonstrates the primary usage of DTail. Please also see ``dtail --help`` for more available options. +This page demonstrates the primary usage of DTail. Please also see `dtail --help` for more available options. -# How to use ``dtail`` +## Table of contents -## Tailing logs +* How to use `dtail` to follow logs +* How to use `dtail` to aggregate logs +* How to use `dcat` +* How to use `dgrep` +* How to use `dmap` +* How to use the DTail serverless mode -The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all records containing the string ``INFO``. Any other Go compatible regular expression can be used instead of ``INFO``. +## How to use `dtail` + +### Following logs + +The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all records containing the string `INFO` Any other Go compatible regular expression can be used instead of `INFO` ```shell % dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log" ``` -Hint: you can also provide a comma separated server list, e.g.: `--servers server1.example.org,server2.example.org:PORT,...`. +Hint: you can also provide a comma separated server list, e.g.: `servers server1.example.org,server2.example.org:PORT,...`  -Hint: You can also use the shorthand version (omitting the `--files`): +Hint: You can also use the shorthand version (omitting the `files` ```shell % dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log" ``` -## Aggregating logs +### Aggregating logs -To run ad-hoc MapReduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the top 10 servers with the most average free memory. To run a MapReduce query across log lines written in the past, please use the ``dmap`` command instead. +To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the top 10 servers with the most average free memory. To run a map-reduce query across log lines written in the past, please use the `dmap` command instead. ```shell % dtail --servers serverlist.txt \ @@ -33,7 +42,7 @@ To run ad-hoc MapReduce aggregations on newly written log lines you must add a q --query 'from STATS select sum($goroutines),sum($cgocalls),last($time),max(lifetimeConnections)' ``` -For MapReduce queries to work, you have to ensure that DTail supports your log format. You can either use the ones already defined in ``internal/mapr/log format`` or add an extension to support a custom log format. +For map-reduce queries to work, you have to ensure that DTail supports your log format. You can either use the ones already defined in `internal/mapr/log format` or add an extension to support a custom log format.  @@ -54,11 +63,11 @@ Here is yet another example:  -# How to use ``dcat`` +## How to use `dcat` The following example demonstrates how to cat files (display the full content of the files) of multiple servers at once. -As you can see in this example, a DTail client also creates a local log file of all received data in `~/log`. You can also use the `-noColor` and `-plain` flags (they also work with other commands than `dcat`). +As you can see in this example, a DTail client also creates a local log file of all received data in `log` You can also use the `noColor` and `-plain` flags (they also work with other commands than `dcat`). ```shell % dcat --servers serverlist.txt --files /etc/hostname @@ -72,9 +81,9 @@ Hint: You can also use the shorthand version: % dcat --servers serverlist.txt /etc/hostname ``` -# How to use ``dgrep`` +## How to use `dgrep` -The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in ``/etc/passwd``. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the `-before`, `-after` and `-max` flags. +The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in `etc/passwd` This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the `-before`, `-after` and `-max` flags. ```shell % dgrep --servers server1.example.org:2223 \ @@ -82,13 +91,15 @@ The following example demonstrates how to grep files (display only the lines whi --regex nologin ``` +Generally, this is also a very useful way to search historic application logs. +  Hint: `-regex` is an alias for `-grep`. -# How to use ``dmap`` +## How to use `dmap` -To run a MapReduce aggregation over logs written in the past, the ``dmap`` command has be used. Fhe following example aggregates all MapReduce fields ``dmap`` will print interim results every few seconds. You can also write the result to an CSV file by adding `outfile result.csv` to the query. +To run a map-reduce aggregation over logs written in the past, the `dmap` command has be used. The following example aggregates all map-reduce fields `dmap` will print interim results every few seconds. You can also write the result to an CSV file by adding `outfile result.csv` to the query. ```shell % dmap --servers serverlist.txt \ @@ -96,6 +107,65 @@ To run a MapReduce aggregation over logs written in the past, the ``dmap`` comma --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' ``` -Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in ``internal/mapr/logformat`` or add an extension to support a custom log format. Te example here works out of the box though, as DTail understands its own log format already. +Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in `internal/mapr/logformat` or add an extension to support a custom log format. Te example here works out of the box though, as DTail understands its own log format already.  + +## How to use the DTail serverless mode + +Until now, all examples so far assumed to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files. + +All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode. + +### Serverless map-reduce query + +The following `dmap` example is the same as the previously shown one, but the difference is that it operates on a local log file directly: + +```shell +% dmap --files /var/log/dserver/dserver.log + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +As a shorthand version the following command can be used: + +```shell +% dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' /var/log/dsever/dserver.log +``` + +You can also use a file input pipe as follows: + +```shell +% cat /var/log/dserver/dserver.log | \ + dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +### Other serverless commands + +It works transparently with all other DTail commands. Here are some examples: + +```shell +dtail /var/log/dserver/dserver.log +``` + +```shell +dtail --logLevel trace /var/log/dserver/dserver.log +``` + +```shell +dcat /etc/passwd +``` + +```shell +dcat --plain /etc/passwd > /etc/test +# Should show no differences. +diff /etc/test /etc/passwd +``` + +```shell +dgrep --regex ERROR --files /var/log/dserver/dsever.log +``` + +```shell +dgrep --before 10 --after 10 --max 10 --grep ERROR \ + /var/log/dserver/dsever.log +``` |
