summaryrefslogtreecommitdiff
path: root/doc/examples.md
diff options
context:
space:
mode:
authorPaul Buetow <paul@buetow.org>2021-11-07 14:43:21 +0200
committerPaul Buetow <paul@buetow.org>2021-11-07 14:43:21 +0200
commit84166ab05f72ed23ada3802fc0baf0cec5eb1233 (patch)
tree414927eb2f3f4b336b5c730a1f3b03a8f39b96d5 /doc/examples.md
parentd9baf981c567e61c57da49cc01f5e1ef2997769f (diff)
add serverless examples
Diffstat (limited to 'doc/examples.md')
-rw-r--r--doc/examples.md102
1 files changed, 86 insertions, 16 deletions
diff --git a/doc/examples.md b/doc/examples.md
index 5dd898d..803b1af 100644
--- a/doc/examples.md
+++ b/doc/examples.md
@@ -1,31 +1,40 @@
Examples
========
-This page demonstrates the primary usage of DTail. Please also see ``dtail --help`` for more available options.
+This page demonstrates the primary usage of DTail. Please also see `dtail --help` for more available options.
-# How to use ``dtail``
+## Table of contents
-## Tailing logs
+* How to use `dtail` to follow logs
+* How to use `dtail` to aggregate logs
+* How to use `dcat`
+* How to use `dgrep`
+* How to use `dmap`
+* How to use the DTail serverless mode
-The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all records containing the string ``INFO``. Any other Go compatible regular expression can be used instead of ``INFO``.
+## How to use `dtail`
+
+### Following logs
+
+The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all records containing the string `INFO` Any other Go compatible regular expression can be used instead of `INFO`
```shell
% dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log"
```
-Hint: you can also provide a comma separated server list, e.g.: `--servers server1.example.org,server2.example.org:PORT,...`.
+Hint: you can also provide a comma separated server list, e.g.: `servers server1.example.org,server2.example.org:PORT,...`
![dtail](dtail.gif "Tail example")
-Hint: You can also use the shorthand version (omitting the `--files`):
+Hint: You can also use the shorthand version (omitting the `files`
```shell
% dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log"
```
-## Aggregating logs
+### Aggregating logs
-To run ad-hoc MapReduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the top 10 servers with the most average free memory. To run a MapReduce query across log lines written in the past, please use the ``dmap`` command instead.
+To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the top 10 servers with the most average free memory. To run a map-reduce query across log lines written in the past, please use the `dmap` command instead.
```shell
% dtail --servers serverlist.txt \
@@ -33,7 +42,7 @@ To run ad-hoc MapReduce aggregations on newly written log lines you must add a q
--query 'from STATS select sum($goroutines),sum($cgocalls),last($time),max(lifetimeConnections)'
```
-For MapReduce queries to work, you have to ensure that DTail supports your log format. You can either use the ones already defined in ``internal/mapr/log format`` or add an extension to support a custom log format.
+For map-reduce queries to work, you have to ensure that DTail supports your log format. You can either use the ones already defined in `internal/mapr/log format` or add an extension to support a custom log format.
![dtail-map](dtail-map.gif "Tail mapreduce example")
@@ -54,11 +63,11 @@ Here is yet another example:
![dtail-map](dtail-map2.gif "Tail mapreduce example 2")
-# How to use ``dcat``
+## How to use `dcat`
The following example demonstrates how to cat files (display the full content of the files) of multiple servers at once.
-As you can see in this example, a DTail client also creates a local log file of all received data in `~/log`. You can also use the `-noColor` and `-plain` flags (they also work with other commands than `dcat`).
+As you can see in this example, a DTail client also creates a local log file of all received data in `log` You can also use the `noColor` and `-plain` flags (they also work with other commands than `dcat`).
```shell
% dcat --servers serverlist.txt --files /etc/hostname
@@ -72,9 +81,9 @@ Hint: You can also use the shorthand version:
% dcat --servers serverlist.txt /etc/hostname
```
-# How to use ``dgrep``
+## How to use `dgrep`
-The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in ``/etc/passwd``. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the `-before`, `-after` and `-max` flags.
+The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in `etc/passwd` This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the `-before`, `-after` and `-max` flags.
```shell
% dgrep --servers server1.example.org:2223 \
@@ -82,13 +91,15 @@ The following example demonstrates how to grep files (display only the lines whi
--regex nologin
```
+Generally, this is also a very useful way to search historic application logs.
+
![dgrep](dgrep.gif "Grep example")
Hint: `-regex` is an alias for `-grep`.
-# How to use ``dmap``
+## How to use `dmap`
-To run a MapReduce aggregation over logs written in the past, the ``dmap`` command has be used. Fhe following example aggregates all MapReduce fields ``dmap`` will print interim results every few seconds. You can also write the result to an CSV file by adding `outfile result.csv` to the query.
+To run a map-reduce aggregation over logs written in the past, the `dmap` command has be used. The following example aggregates all map-reduce fields `dmap` will print interim results every few seconds. You can also write the result to an CSV file by adding `outfile result.csv` to the query.
```shell
% dmap --servers serverlist.txt \
@@ -96,6 +107,65 @@ To run a MapReduce aggregation over logs written in the past, the ``dmap`` comma
--query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)'
```
-Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in ``internal/mapr/logformat`` or add an extension to support a custom log format. Te example here works out of the box though, as DTail understands its own log format already.
+Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in `internal/mapr/logformat` or add an extension to support a custom log format. Te example here works out of the box though, as DTail understands its own log format already.
![dmap](dmap.gif "DMap example")
+
+## How to use the DTail serverless mode
+
+Until now, all examples so far assumed to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files.
+
+All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode.
+
+### Serverless map-reduce query
+
+The following `dmap` example is the same as the previously shown one, but the difference is that it operates on a local log file directly:
+
+```shell
+% dmap --files /var/log/dserver/dserver.log
+ --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)'
+```
+
+As a shorthand version the following command can be used:
+
+```shell
+% dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)' /var/log/dsever/dserver.log
+```
+
+You can also use a file input pipe as follows:
+
+```shell
+% cat /var/log/dserver/dserver.log | \
+ dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,lifetimeConnections group by $hostname order by max($cgocalls)'
+```
+
+### Other serverless commands
+
+It works transparently with all other DTail commands. Here are some examples:
+
+```shell
+dtail /var/log/dserver/dserver.log
+```
+
+```shell
+dtail --logLevel trace /var/log/dserver/dserver.log
+```
+
+```shell
+dcat /etc/passwd
+```
+
+```shell
+dcat --plain /etc/passwd > /etc/test
+# Should show no differences.
+diff /etc/test /etc/passwd
+```
+
+```shell
+dgrep --regex ERROR --files /var/log/dserver/dsever.log
+```
+
+```shell
+dgrep --before 10 --after 10 --max 10 --grep ERROR \
+ /var/log/dserver/dsever.log
+```