From 17507d832ccb1c5215fe6a50b30c94e7b9a08b8c Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Mon, 25 Sep 2023 15:14:36 +0300 Subject: Update content for gemtext --- ...4-22-dtail-the-distributed-log-tail-program.gmi | 1 + gemfeed/2022-03-06-the-release-of-dtail-4.0.0.gmi | 1 + gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi | 1 + gemfeed/2023-09-25-dtail-usage-examples.gmi | 244 ++ gemfeed/2023-09-25-dtail-usage-examples.gmi.tpl | 241 ++ gemfeed/DRAFT-dtail-usage-examples.gmi | 230 ++ gemfeed/DRAFT-dtail-usage-examples.gmi.tpl | 214 -- gemfeed/W | 228 ++ gemfeed/atom.xml | 814 +++--- gemfeed/atom.xml.tmp | 2825 -------------------- gemfeed/dtail-usage-examples/dcat.gif | Bin 0 -> 602213 bytes gemfeed/dtail-usage-examples/dgrep.gif | Bin 0 -> 1309227 bytes gemfeed/dtail-usage-examples/dmap.gif | Bin 0 -> 1154423 bytes gemfeed/dtail-usage-examples/dtail-map.gif | Bin 0 -> 298895 bytes gemfeed/dtail-usage-examples/dtail-map2.gif | Bin 0 -> 271416 bytes gemfeed/dtail-usage-examples/dtail.gif | Bin 0 -> 2290260 bytes gemfeed/dtail-usage-examples/testing.gif | Bin 0 -> 2637253 bytes gemfeed/index.gmi | 1 + index.gmi | 3 +- uptime-stats.gmi | 94 +- 20 files changed, 1361 insertions(+), 3536 deletions(-) create mode 100644 gemfeed/2023-09-25-dtail-usage-examples.gmi create mode 100644 gemfeed/2023-09-25-dtail-usage-examples.gmi.tpl create mode 100644 gemfeed/DRAFT-dtail-usage-examples.gmi delete mode 100644 gemfeed/DRAFT-dtail-usage-examples.gmi.tpl create mode 100644 gemfeed/W delete mode 100644 gemfeed/atom.xml.tmp create mode 100644 gemfeed/dtail-usage-examples/dcat.gif create mode 100644 gemfeed/dtail-usage-examples/dgrep.gif create mode 100644 gemfeed/dtail-usage-examples/dmap.gif create mode 100644 gemfeed/dtail-usage-examples/dtail-map.gif create mode 100644 gemfeed/dtail-usage-examples/dtail-map2.gif create mode 100644 gemfeed/dtail-usage-examples/dtail.gif create mode 100644 gemfeed/dtail-usage-examples/testing.gif diff --git a/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi b/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi index f42c625a..c2cd31f4 100644 --- a/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi +++ b/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi @@ -108,6 +108,7 @@ Other related posts are: => ./2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 DTail - The distributed log tail program (You are currently reading this) => ./2022-03-06-the-release-of-dtail-4.0.0.gmi 2022-03-06 The release of DTail 4.0.0 => ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD +=> ./2023-09-25-dtail-usage-examples.gmi 2023-09-25 DTail usage examples E-Mail your comments to `foo@paul.cyou` :-) diff --git a/gemfeed/2022-03-06-the-release-of-dtail-4.0.0.gmi b/gemfeed/2022-03-06-the-release-of-dtail-4.0.0.gmi index 637740a4..63c4714f 100644 --- a/gemfeed/2022-03-06-the-release-of-dtail-4.0.0.gmi +++ b/gemfeed/2022-03-06-the-release-of-dtail-4.0.0.gmi @@ -291,6 +291,7 @@ Other related posts are: => ./2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 DTail - The distributed log tail program => ./2022-03-06-the-release-of-dtail-4.0.0.gmi 2022-03-06 The release of DTail 4.0.0 (You are currently reading this) => ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD +=> ./2023-09-25-dtail-usage-examples.gmi 2023-09-25 DTail usage examples Thanks! diff --git a/gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi b/gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi index 79d3c0ce..90273d3d 100644 --- a/gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi +++ b/gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi @@ -344,6 +344,7 @@ Other related posts are: => ./2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 DTail - The distributed log tail program => ./2022-03-06-the-release-of-dtail-4.0.0.gmi 2022-03-06 The release of DTail 4.0.0 => ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD (You are currently reading this) +=> ./2023-09-25-dtail-usage-examples.gmi 2023-09-25 DTail usage examples E-Mail your comments to `foo@paul.cyou` :-) diff --git a/gemfeed/2023-09-25-dtail-usage-examples.gmi b/gemfeed/2023-09-25-dtail-usage-examples.gmi new file mode 100644 index 00000000..62b1921e --- /dev/null +++ b/gemfeed/2023-09-25-dtail-usage-examples.gmi @@ -0,0 +1,244 @@ +# DTail usage examples + +> Published at 2023-09-25T14:57:42+03:00 + +Hey there. As I am pretty busy this month personally (I am now on Paternity Leave) and as I still want to post once monthly, the blog post of this month will only be some DTail usage examples. They're from the DTail documentation, but not all readers of my blog may be aware of those! + +DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once which I programmed in Go. + +=> https://dtail.dev + +``` + ,_---~~~~~----._ + _,,_,*^____ _____``*g*\"*, + ____ _____ _ _ / __/ /' ^. / \ ^@q f + | _ \_ _|_ _(_) | @f | ((@| |@)) l 0 _/ + | | | || |/ _` | | | \`/ \~____ / __ \_____/ \ + | |_| || | (_| | | | | _l__l_ I + |____/ |_|\__,_|_|_| } [______] I + ] | | | | + ] ~ ~ | + | Let's tail those logs! | + | | +``` + +DTail consists out of a server and several client binaries. In this post, I am showcasing their use! + +* Use `dtail` to follow logs +* Use `dtail` to aggregate logs while they are followed +* Use `dcat` to display logs and other text files already written +* Use `dgrep` to grep (search) logs and other text files already written +* Use `dmap` to aggregate logs and other text files already written +* `dserver` is the DTail server, where all the clients can connect to + +## Following logs + +The following example demonstrates how to follow logs of several servers at once. The server list is provided as a flat text file. The example filters all records containing the string `INFO`. Any other Go compatible regular expression can also be used instead of `INFO`. + +```shell +% dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log" +``` + +Hint: you can also provide a comma separated server list, e.g.: `servers server1.example.org,server2.example.org:PORT,...` + +=> ./dtail-usage-examples/dtail.gif Tail example + +> Hint: You can also use the shorthand version (omitting the `--files`) + +```shell +% dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log" +``` + +## Aggregating logs + +To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the result to standard output. + +> Hint: To run a map-reduce query across log lines written in the past, please use the `dmap` command instead. + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select sum($goroutines),sum($cgocalls), + last($time),max(lifetimeConnections)' +``` + +Beware: For map-reduce queries to work, you have to ensure that DTail supports your log format. Check out the documentaiton of the DTail query language and the DTail log formats on the DTail homepage for more information. + +=> ./dtail-usage-examples/dtail-map.gif Tail map-reduce example + +> Hint: You can also use the shorthand version: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + 'from STATS select sum($goroutines),sum($cgocalls), + last($time),max(lifetimeConnections)' +``` + +Here is another example: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +=> ./dtail-usage-examples/dtail-map2.gif Tail map-reduce example 2 + +You can also continuously append the results to a CSV file by adding `outfile append filename.csv` to the query: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select ... outfile append result.csv' +``` + +## How to use `dcat` + +The following example demonstrates how to cat files (display the full content of the files) on several servers at once. + +As you can see in this example, a DTail client also creates a local log file of all received data in `~/log`. You can also use the `noColor` and `-plain` flags (this all also work with other DTail commands than `dcat`). + +```shell +% dcat --servers serverlist.txt --files /etc/hostname +``` + +=> ./dtail-usage-examples/dcat.gif Cat example + +> Hint: You can also use the shorthand version: + +```shell +% dcat --servers serverlist.txt /etc/hostname +``` + +## How to use `dgrep` + +The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in `/etc/passwd`. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the `-before`, `-after` and `-max` flags (see animation). + +```shell +% dgrep --servers server1.example.org:2223 \ + --files /etc/passwd \ + --regex nologin +``` + +Generally, `dgrep` is also a very useful way to search historic application logs for certain content. + +=> ./dtail-usage-examples/dgrep.gif Grep example + +> Hint: `-regex` is an alias for `-grep`. + +## How to use `dmap` + +To run a map-reduce aggregation over logs written in the past, the `dmap` command can be used. The following example aggregates all map-reduce fields `dmap` will print interim results every few seconds. You can also write the result to an CSV file by adding `outfile result.csv` to the query. + +```shell +% dmap --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in `internal/mapr/logformat` or add an extension to support a custom log format. The example here works out of the box though, as DTail understands its own log format already. + +=> ./dtail-usage-examples/dmap.gif DMap example + +## How to use the DTail serverless mode + +Until now, all examples so far required to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files. + +The serverless mode does not require any `dserver` up and running and therefore there is no networking/SSH involved. + +All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode. + +### Serverless map-reduce query + +The following `dmap` example is the same as the previously shown one, but the difference is that it operates on a local log file directly: + +```shell +% dmap --files /var/log/dserver/dserver.log + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +As a shorthand version the following command can be used: + +```shell +% dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' \ + /var/log/dsever/dserver.log +``` + +You can also use a file input pipe as follows: + +```shell +% cat /var/log/dserver/dserver.log | \ + dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +### Aggregating CSV files + +In essence, this works exactly like aggregating logs. All files operated on must be valid CSV files and the first line of the CSV must be the header. E.g.: + +```shell +% cat example.csv +name,lastname,age,profession +Michael,Jordan,40,Basketball player +Michael,Jackson,100,Singer +Albert,Einstein,200,Physician +% dmap --query 'select lastname,name where age > 40 logformat csv outfile result.csv' example.csv +% cat result.csv +lastname,name +Jackson,Michael +Einstein,Albert +``` + +DMap can also be used to query and aggregate CSV files from remote servers. + +### Other serverless commands + +The serverless mode works transparently with all other DTail commands. Here are some examples: + +```shell +% dtail /var/log/dserver/dserver.log +``` + +```shell +% dtail --logLevel trace /var/log/dserver/dserver.log +``` + +```shell +% dcat /etc/passwd +``` + +```shell +% dcat --plain /etc/passwd > /etc/test +# Should show no differences. +diff /etc/test /etc/passwd +``` + +```shell +% dgrep --regex ERROR --files /var/log/dserver/dsever.log +``` + +```shell +% dgrep --before 10 --after 10 --max 10 --grep ERROR /var/log/dserver/dsever.log +``` + +Use `--help` for more available options. Or go to the DTail page for more information! Hope you find DTail useful! + +Other related posts are: + +=> ./2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 DTail - The distributed log tail program +=> ./2022-03-06-the-release-of-dtail-4.0.0.gmi 2022-03-06 The release of DTail 4.0.0 +=> ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD +=> ./2023-09-25-dtail-usage-examples.gmi 2023-09-25 DTail usage examples (You are currently reading this) + +I hope you find the tools presented in this post useful! + +Paul + +E-Mail your comments to `foo@paul.cyou` :-) + +=> ../ Back to the main site diff --git a/gemfeed/2023-09-25-dtail-usage-examples.gmi.tpl b/gemfeed/2023-09-25-dtail-usage-examples.gmi.tpl new file mode 100644 index 00000000..1fa89cdc --- /dev/null +++ b/gemfeed/2023-09-25-dtail-usage-examples.gmi.tpl @@ -0,0 +1,241 @@ +# DTail usage examples + +> Published at 2023-09-25T14:57:42+03:00 + +Hey there. As I am pretty busy this month personally (I am now on Paternity Leave) and as I still want to post once monthly, the blog post of this month will only be some DTail usage examples. They're from the DTail documentation, but not all readers of my blog may be aware of those! + +DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once which I programmed in Go. + +=> https://dtail.dev + +``` + ,_---~~~~~----._ + _,,_,*^____ _____``*g*\"*, + ____ _____ _ _ / __/ /' ^. / \ ^@q f + | _ \_ _|_ _(_) | @f | ((@| |@)) l 0 _/ + | | | || |/ _` | | | \`/ \~____ / __ \_____/ \ + | |_| || | (_| | | | | _l__l_ I + |____/ |_|\__,_|_|_| } [______] I + ] | | | | + ] ~ ~ | + | Let's tail those logs! | + | | +``` + +DTail consists out of a server and several client binaries. In this post, I am showcasing their use! + +* Use `dtail` to follow logs +* Use `dtail` to aggregate logs while they are followed +* Use `dcat` to display logs and other text files already written +* Use `dgrep` to grep (search) logs and other text files already written +* Use `dmap` to aggregate logs and other text files already written +* `dserver` is the DTail server, where all the clients can connect to + +## Following logs + +The following example demonstrates how to follow logs of several servers at once. The server list is provided as a flat text file. The example filters all records containing the string `INFO`. Any other Go compatible regular expression can also be used instead of `INFO`. + +```shell +% dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log" +``` + +Hint: you can also provide a comma separated server list, e.g.: `servers server1.example.org,server2.example.org:PORT,...` + +=> ./dtail-usage-examples/dtail.gif Tail example + +> Hint: You can also use the shorthand version (omitting the `--files`) + +```shell +% dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log" +``` + +## Aggregating logs + +To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the result to standard output. + +> Hint: To run a map-reduce query across log lines written in the past, please use the `dmap` command instead. + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select sum($goroutines),sum($cgocalls), + last($time),max(lifetimeConnections)' +``` + +Beware: For map-reduce queries to work, you have to ensure that DTail supports your log format. Check out the documentaiton of the DTail query language and the DTail log formats on the DTail homepage for more information. + +=> ./dtail-usage-examples/dtail-map.gif Tail map-reduce example + +> Hint: You can also use the shorthand version: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + 'from STATS select sum($goroutines),sum($cgocalls), + last($time),max(lifetimeConnections)' +``` + +Here is another example: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +=> ./dtail-usage-examples/dtail-map2.gif Tail map-reduce example 2 + +You can also continuously append the results to a CSV file by adding `outfile append filename.csv` to the query: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select ... outfile append result.csv' +``` + +## How to use `dcat` + +The following example demonstrates how to cat files (display the full content of the files) on several servers at once. + +As you can see in this example, a DTail client also creates a local log file of all received data in `~/log`. You can also use the `noColor` and `-plain` flags (this all also work with other DTail commands than `dcat`). + +```shell +% dcat --servers serverlist.txt --files /etc/hostname +``` + +=> ./dtail-usage-examples/dcat.gif Cat example + +> Hint: You can also use the shorthand version: + +```shell +% dcat --servers serverlist.txt /etc/hostname +``` + +## How to use `dgrep` + +The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in `/etc/passwd`. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the `-before`, `-after` and `-max` flags (see animation). + +```shell +% dgrep --servers server1.example.org:2223 \ + --files /etc/passwd \ + --regex nologin +``` + +Generally, `dgrep` is also a very useful way to search historic application logs for certain content. + +=> ./dtail-usage-examples/dgrep.gif Grep example + +> Hint: `-regex` is an alias for `-grep`. + +## How to use `dmap` + +To run a map-reduce aggregation over logs written in the past, the `dmap` command can be used. The following example aggregates all map-reduce fields `dmap` will print interim results every few seconds. You can also write the result to an CSV file by adding `outfile result.csv` to the query. + +```shell +% dmap --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in `internal/mapr/logformat` or add an extension to support a custom log format. The example here works out of the box though, as DTail understands its own log format already. + +=> ./dtail-usage-examples/dmap.gif DMap example + +## How to use the DTail serverless mode + +Until now, all examples so far required to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files. + +The serverless mode does not require any `dserver` up and running and therefore there is no networking/SSH involved. + +All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode. + +### Serverless map-reduce query + +The following `dmap` example is the same as the previously shown one, but the difference is that it operates on a local log file directly: + +```shell +% dmap --files /var/log/dserver/dserver.log + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +As a shorthand version the following command can be used: + +```shell +% dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' \ + /var/log/dsever/dserver.log +``` + +You can also use a file input pipe as follows: + +```shell +% cat /var/log/dserver/dserver.log | \ + dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +### Aggregating CSV files + +In essence, this works exactly like aggregating logs. All files operated on must be valid CSV files and the first line of the CSV must be the header. E.g.: + +```shell +% cat example.csv +name,lastname,age,profession +Michael,Jordan,40,Basketball player +Michael,Jackson,100,Singer +Albert,Einstein,200,Physician +% dmap --query 'select lastname,name where age > 40 logformat csv outfile result.csv' example.csv +% cat result.csv +lastname,name +Jackson,Michael +Einstein,Albert +``` + +DMap can also be used to query and aggregate CSV files from remote servers. + +### Other serverless commands + +The serverless mode works transparently with all other DTail commands. Here are some examples: + +```shell +% dtail /var/log/dserver/dserver.log +``` + +```shell +% dtail --logLevel trace /var/log/dserver/dserver.log +``` + +```shell +% dcat /etc/passwd +``` + +```shell +% dcat --plain /etc/passwd > /etc/test +# Should show no differences. +diff /etc/test /etc/passwd +``` + +```shell +% dgrep --regex ERROR --files /var/log/dserver/dsever.log +``` + +```shell +% dgrep --before 10 --after 10 --max 10 --grep ERROR /var/log/dserver/dsever.log +``` + +Use `--help` for more available options. Or go to the DTail page for more information! Hope you find DTail useful! + +Other related posts are: + +<< template::inline::index dtail + +I hope you find the tools presented in this post useful! + +Paul + +E-Mail your comments to `foo@paul.cyou` :-) + +=> ../ Back to the main site diff --git a/gemfeed/DRAFT-dtail-usage-examples.gmi b/gemfeed/DRAFT-dtail-usage-examples.gmi new file mode 100644 index 00000000..77b42aa7 --- /dev/null +++ b/gemfeed/DRAFT-dtail-usage-examples.gmi @@ -0,0 +1,230 @@ +# DTail usage examples + +DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once which I programmed in Go. + +=> https://dtail.dev + +``` + ,_---~~~~~----._ + _,,_,*^____ _____``*g*\"*, + ____ _____ _ _ / __/ /' ^. / \ ^@q f + | _ \_ _|_ _(_) | @f | ((@| |@)) l 0 _/ + | | | || |/ _` | | | \`/ \~____ / __ \_____/ \ + | |_| || | (_| | | | | _l__l_ I + |____/ |_|\__,_|_|_| } [______] I + ] | | | | + ] ~ ~ | + | Let's tail those logs! | + | | +``` + +DTail consists out of a server and several client binaries. In this post, I am showcasing their use! + +## Following logs + +The following example demonstrates how to follow logs of several servers at once. The server list is provided as a flat text file. The example filters all records containing the string `INFO`. Any other Go compatible regular expression can also be used instead of `INFO`. + +```shell +% dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log" +``` + +Hint: you can also provide a comma separated server list, e.g.: `servers server1.example.org,server2.example.org:PORT,...` + +=> ./dtail-usage-examples/dtail.gif Tail example + +> Hint: You can also use the shorthand version (omitting the `--files`) + +```shell +% dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log" +``` + +## Aggregating logs + +To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the result to standard output. + +> Hint: To run a map-reduce query across log lines written in the past, please use the `dmap` command instead. + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select sum($goroutines),sum($cgocalls), + last($time),max(lifetimeConnections)' +``` + +Beware: For map-reduce queries to work, you have to ensure that DTail supports your log format. Check out the documentaiton of the DTail query language and the DTail log formats on the DTail homepage for more information. + +=> ./dtail-usage-examples/dtail-map.gif Tail map-reduce example + +> Hint: You can also use the shorthand version: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + 'from STATS select sum($goroutines),sum($cgocalls), + last($time),max(lifetimeConnections)' +``` + +Here is another example: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +=> ./dtail-usage-examples/dtail-map2.gif Tail map-reduce example 2 + +You can also continuously append the results to a CSV file by adding `outfile append filename.csv` to the query: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select ... outfile append result.csv' +``` + +## How to use `dcat` + +The following example demonstrates how to cat files (display the full content of the files) on several servers at once. + +As you can see in this example, a DTail client also creates a local log file of all received data in `~/log`. You can also use the `noColor` and `-plain` flags (this all also work with other DTail commands than `dcat`). + +```shell +% dcat --servers serverlist.txt --files /etc/hostname +``` + +=> ./dtail-usage-examples/dcat.gif Cat example + +> Hint: You can also use the shorthand version: + +```shell +% dcat --servers serverlist.txt /etc/hostname +``` + +## How to use `dgrep` + +The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in `/etc/passwd`. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the `-before`, `-after` and `-max` flags (see animation). + +```shell +% dgrep --servers server1.example.org:2223 \ + --files /etc/passwd \ + --regex nologin +``` + +Generally, `dgrep` is also a very useful way to search historic application logs for certain content. + +=> ./dtail-usage-examples/dgrep.gif Grep example + +Hint: `-regex` is an alias for `-grep`. + +## How to use `dmap` + +To run a map-reduce aggregation over logs written in the past, the `dmap` command can be used. The following example aggregates all map-reduce fields `dmap` will print interim results every few seconds. You can also write the result to an CSV file by adding `outfile result.csv` to the query. + +```shell +% dmap --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in `internal/mapr/logformat` or add an extension to support a custom log format. The example here works out of the box though, as DTail understands its own log format already. + +=> ./dtail-usage-examples/dmap.gif DMap example + +## How to use the DTail serverless mode + +Until now, all examples so far required to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files. + +The serverless mode does not require any `dserver` up and running and therefore there is no networking/SSH involved. + +All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode. + +### Serverless map-reduce query + +The following `dmap` example is the same as the previously shown one, but the difference is that it operates on a local log file directly: + +```shell +% dmap --files /var/log/dserver/dserver.log + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +As a shorthand version the following command can be used: + +```shell +% dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, +lifetimeConnections group by $hostname order by max($cgocalls)' /var/log/dsever/dserver.log +``` + +You can also use a file input pipe as follows: + +```shell +% cat /var/log/dserver/dserver.log | \ + dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +### Aggregating CSV files + +In essence, this works exactly like aggregating logs. All files operated on must be valid CSV files and the first line of the CSV must be the header. E.g.: + +```shell +% cat example.csv +name,lastname,age,profession +Michael,Jordan,40,Basketball player +Michael,Jackson,100,Singer +Albert,Einstein,200,Physician +% dmap --query 'select lastname,name where age > 40 logformat csv outfile result.csv' example.csv +% cat result.csv +lastname,name +Jackson,Michael +Einstein,Albert +``` + +DMap can also be used to query and aggregate CSV files from remote servers. + +### Other serverless commands + +The serverless mode works transparently with all other DTail commands. Here are some examples: + +```shell +% dtail /var/log/dserver/dserver.log +``` + +```shell +% dtail --logLevel trace /var/log/dserver/dserver.log +``` + +```shell +% dcat /etc/passwd +``` + +```shell +% dcat --plain /etc/passwd > /etc/test +# Should show no differences. +diff /etc/test /etc/passwd +``` + +```shell +% dgrep --regex ERROR --files /var/log/dserver/dsever.log +``` + +```shell +% dgrep --before 10 --after 10 --max 10 --grep ERROR /var/log/dserver/dsever.log + +Use `--help` for more available options. Or go to the DTail page for more information! Hope you find DTail useful! + +Other related posts are: + +=> ./2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 DTail - The distributed log tail program +=> ./2022-03-06-the-release-of-dtail-4.0.0.gmi 2022-03-06 The release of DTail 4.0.0 +=> ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD + +Thanks! + +Paul + +E-Mail your comments to `foo@paul.cyou` :-) + +=> ../ Back to the main site diff --git a/gemfeed/DRAFT-dtail-usage-examples.gmi.tpl b/gemfeed/DRAFT-dtail-usage-examples.gmi.tpl deleted file mode 100644 index 29538df4..00000000 --- a/gemfeed/DRAFT-dtail-usage-examples.gmi.tpl +++ /dev/null @@ -1,214 +0,0 @@ -# DTail usage examples - -DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once which I programmed in Go. - -=> https://dtail.dev - -DTail consists out of a server and several client binaries. In this post, I am showcasing their use! - -## Following logs - -The following example demonstrates how to follow logs of several servers at once. The server list is provided as a flat text file. The example filters all records containing the string `INFO`. Any other Go compatible regular expression can also be used instead of `INFO`. - -```shell -% dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log" -``` - -Hint: you can also provide a comma separated server list, e.g.: `servers server1.example.org,server2.example.org:PORT,...` - -=> ./dtail-usage-examples/dtail.gif Tail example - -> Hint: You can also use the shorthand version (omitting the `--files`) - -```shell -% dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log" -``` - -## Aggregating logs - -To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the result to standard output. - -> Hint: To run a map-reduce query across log lines written in the past, please use the `dmap` command instead. - -```shell -% dtail --servers serverlist.txt \ - --files '/var/log/dserver/*.log' \ - --query 'from STATS select sum($goroutines),sum($cgocalls), - last($time),max(lifetimeConnections)' -``` - -> Beware: For map-reduce queries to work, you have to ensure that DTail supports your log format. Check out the documentaiton of the DTail query language and the DTail log formats on the DTail homepage for more information. - -=> ./dtail-usage-examples/dtail-map.gif Tail map-reduce example - -> Hint: You can also use the shorthand version: - -```shell -% dtail --servers serverlist.txt \ - --files '/var/log/dserver/*.log' \ - 'from STATS select sum($goroutines),sum($cgocalls), - last($time),max(lifetimeConnections)' -``` - -Here is another example: - -```shell -% dtail --servers serverlist.txt \ - --files '/var/log/dserver/*.log' \ - --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, - lifetimeConnections group by $hostname order by max($cgocalls)' -``` - -=> ./dtail-usage-examples/dtail-map2.gif Tail map-reduce example 2 - -You can also continuously append the results to a CSV file by adding `outfile append filename.csv` to the query: - -```shell -% dtail --servers serverlist.txt \ - --files '/var/log/dserver/*.log' \ - --query 'from STATS select ... outfile append result.csv' -``` - -## How to use `dcat` - -The following example demonstrates how to cat files (display the full content of the files) on several servers at once. - -As you can see in this example, a DTail client also creates a local log file of all received data in `~/log`. You can also use the `noColor` and `-plain` flags (this all also work with other DTail commands than `dcat`). - -```shell -% dcat --servers serverlist.txt --files /etc/hostname -``` - -=> ./dtail-usage-examples/dcat.gif Cat example - -> Hint: You can also use the shorthand version: - -```shell -% dcat --servers serverlist.txt /etc/hostname -``` - -## How to use `dgrep` - -The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in `/etc/passwd`. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the `-before`, `-after` and `-max` flags (see animation). - -```shell -% dgrep --servers server1.example.org:2223 \ - --files /etc/passwd \ - --regex nologin -``` - -Generally, `dgrep` is also a very useful way to search historic application logs for certain content. - -=> ./dtail-usage-examples/dgrep.gif Grep example - -Hint: `-regex` is an alias for `-grep`. - -## How to use `dmap` - -To run a map-reduce aggregation over logs written in the past, the `dmap` command can be used. The following example aggregates all map-reduce fields `dmap` will print interim results every few seconds. You can also write the result to an CSV file by adding `outfile result.csv` to the query. - -```shell -% dmap --servers serverlist.txt \ - --files '/var/log/dserver/*.log' \ - --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, - lifetimeConnections group by $hostname order by max($cgocalls)' -``` - -Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in `internal/mapr/logformat` or add an extension to support a custom log format. The example here works out of the box though, as DTail understands its own log format already. - -=> ./dtail-usage-examples/dmap.gif DMap example - -## How to use the DTail serverless mode - -Until now, all examples so far required to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files. - -The serverless mode does not require any `dserver` up and running and therefore there is no networking/SSH involved. - -All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode. - -### Serverless map-reduce query - -The following `dmap` example is the same as the previously shown one, but the difference is that it operates on a local log file directly: - -```shell -% dmap --files /var/log/dserver/dserver.log - --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, - lifetimeConnections group by $hostname order by max($cgocalls)' -``` - -As a shorthand version the following command can be used: - -```shell -% dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, -lifetimeConnections group by $hostname order by max($cgocalls)' /var/log/dsever/dserver.log -``` - -You can also use a file input pipe as follows: - -```shell -% cat /var/log/dserver/dserver.log | \ - dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, - lifetimeConnections group by $hostname order by max($cgocalls)' -``` - -### Aggregating CSV files - -In essence, this works exactly like aggregating logs. All files operated on must be valid CSV files and the first line of the CSV must be the header. E.g.: - -```shell -% cat example.csv -name,lastname,age,profession -Michael,Jordan,40,Basketball player -Michael,Jackson,100,Singer -Albert,Einstein,200,Physician -% dmap --query 'select lastname,name where age > 40 logformat csv outfile result.csv' example.csv -% cat result.csv -lastname,name -Jackson,Michael -Einstein,Albert -``` - -DMap can also be used to query and aggregate CSV files from remote servers. - -### Other serverless commands - -The serverless mode works transparently with all other DTail commands. Here are some examples: - -```shell -% dtail /var/log/dserver/dserver.log -``` - -```shell -% dtail --logLevel trace /var/log/dserver/dserver.log -``` - -```shell -% dcat /etc/passwd -``` - -```shell -% dcat --plain /etc/passwd > /etc/test -# Should show no differences. -diff /etc/test /etc/passwd -``` - -```shell -% dgrep --regex ERROR --files /var/log/dserver/dsever.log -``` - -```shell -% dgrep --before 10 --after 10 --max 10 --grep ERROR /var/log/dserver/dsever.log - -Use `--help` for more available options. Or go to the DTail page for more information! Hope you find DTail useful! - -Other related posts are: - -<< template::inline::index dtail - -Thanks! - -Paul - -E-Mail your comments to `foo@paul.cyou` :-) - -=> ../ Back to the main site diff --git a/gemfeed/W b/gemfeed/W new file mode 100644 index 00000000..7b753922 --- /dev/null +++ b/gemfeed/W @@ -0,0 +1,228 @@ +# DTail usage examples + +DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once which I programmed in Go. + +=> https://dtail.dev + +``` + ,_---~~~~~----._ + _,,_,*^____ _____``*g*\"*, + ____ _____ _ _ / __/ /' ^. / \ ^@q f + | _ \_ _|_ _(_) | @f | ((@| |@)) l 0 _/ + | | | || |/ _` | | | \`/ \~____ / __ \_____/ \ + | |_| || | (_| | | | | _l__l_ I + |____/ |_|\__,_|_|_| } [______] I + ] | | | | + ] ~ ~ | + | Let's tail those logs! | + | | +``` + +DTail consists out of a server and several client binaries. In this post, I am showcasing their use! + +## Following logs + +The following example demonstrates how to follow logs of several servers at once. The server list is provided as a flat text file. The example filters all records containing the string `INFO`. Any other Go compatible regular expression can also be used instead of `INFO`. + +```shell +% dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log" +``` + +Hint: you can also provide a comma separated server list, e.g.: `servers server1.example.org,server2.example.org:PORT,...` + +=> ./dtail-usage-examples/dtail.gif Tail example + +> Hint: You can also use the shorthand version (omitting the `--files`) + +```shell +% dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log" +``` + +## Aggregating logs + +To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the result to standard output. + +> Hint: To run a map-reduce query across log lines written in the past, please use the `dmap` command instead. + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select sum($goroutines),sum($cgocalls), + last($time),max(lifetimeConnections)' +``` + +Beware: For map-reduce queries to work, you have to ensure that DTail supports your log format. Check out the documentaiton of the DTail query language and the DTail log formats on the DTail homepage for more information. + +=> ./dtail-usage-examples/dtail-map.gif Tail map-reduce example + +> Hint: You can also use the shorthand version: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + 'from STATS select sum($goroutines),sum($cgocalls), + last($time),max(lifetimeConnections)' +``` + +Here is another example: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +=> ./dtail-usage-examples/dtail-map2.gif Tail map-reduce example 2 + +You can also continuously append the results to a CSV file by adding `outfile append filename.csv` to the query: + +```shell +% dtail --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select ... outfile append result.csv' +``` + +## How to use `dcat` + +The following example demonstrates how to cat files (display the full content of the files) on several servers at once. + +As you can see in this example, a DTail client also creates a local log file of all received data in `~/log`. You can also use the `noColor` and `-plain` flags (this all also work with other DTail commands than `dcat`). + +```shell +% dcat --servers serverlist.txt --files /etc/hostname +``` + +=> ./dtail-usage-examples/dcat.gif Cat example + +> Hint: You can also use the shorthand version: + +```shell +% dcat --servers serverlist.txt /etc/hostname +``` + +## How to use `dgrep` + +The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in `/etc/passwd`. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the `-before`, `-after` and `-max` flags (see animation). + +```shell +% dgrep --servers server1.example.org:2223 \ + --files /etc/passwd \ + --regex nologin +``` + +Generally, `dgrep` is also a very useful way to search historic application logs for certain content. + +=> ./dtail-usage-examples/dgrep.gif Grep example + +Hint: `-regex` is an alias for `-grep`. + +## How to use `dmap` + +To run a map-reduce aggregation over logs written in the past, the `dmap` command can be used. The following example aggregates all map-reduce fields `dmap` will print interim results every few seconds. You can also write the result to an CSV file by adding `outfile result.csv` to the query. + +```shell +% dmap --servers serverlist.txt \ + --files '/var/log/dserver/*.log' \ + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in `internal/mapr/logformat` or add an extension to support a custom log format. The example here works out of the box though, as DTail understands its own log format already. + +=> ./dtail-usage-examples/dmap.gif DMap example + +## How to use the DTail serverless mode + +Until now, all examples so far required to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files. + +The serverless mode does not require any `dserver` up and running and therefore there is no networking/SSH involved. + +All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode. + +### Serverless map-reduce query + +The following `dmap` example is the same as the previously shown one, but the difference is that it operates on a local log file directly: + +```shell +% dmap --files /var/log/dserver/dserver.log + --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +As a shorthand version the following command can be used: + +```shell +% dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, +lifetimeConnections group by $hostname order by max($cgocalls)' /var/log/dsever/dserver.log +``` + +You can also use a file input pipe as follows: + +```shell +% cat /var/log/dserver/dserver.log | \ + dmap 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg, + lifetimeConnections group by $hostname order by max($cgocalls)' +``` + +### Aggregating CSV files + +In essence, this works exactly like aggregating logs. All files operated on must be valid CSV files and the first line of the CSV must be the header. E.g.: + +```shell +% cat example.csv +name,lastname,age,profession +Michael,Jordan,40,Basketball player +Michael,Jackson,100,Singer +Albert,Einstein,200,Physician +% dmap --query 'select lastname,name where age > 40 logformat csv outfile result.csv' example.csv +% cat result.csv +lastname,name +Jackson,Michael +Einstein,Albert +``` + +DMap can also be used to query and aggregate CSV files from remote servers. + +### Other serverless commands + +The serverless mode works transparently with all other DTail commands. Here are some examples: + +```shell +% dtail /var/log/dserver/dserver.log +``` + +```shell +% dtail --logLevel trace /var/log/dserver/dserver.log +``` + +```shell +% dcat /etc/passwd +``` + +```shell +% dcat --plain /etc/passwd > /etc/test +# Should show no differences. +diff /etc/test /etc/passwd +``` + +```shell +% dgrep --regex ERROR --files /var/log/dserver/dsever.log +``` + +```shell +% dgrep --before 10 --after 10 --max 10 --grep ERROR /var/log/dserver/dsever.log + +Use `--help` for more available options. Or go to the DTail page for more information! Hope you find DTail useful! + +Other related posts are: + +<< template::inline::index dtail + +Thanks! + +Paul + +E-Mail your comments to `foo@paul.cyou` :-) + +=> ../ Back to the main site diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index f41efc63..4ac849b6 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,11 +1,331 @@ - 2023-08-22T00:06:26+03:00 + 2023-09-25T15:14:22+03:00 foo.zone feed To be in the .zone! gemini://foo.zone/ + + DTail usage examples + + gemini://foo.zone/gemfeed/2023-09-25-dtail-usage-examples.gmi + 2023-09-25T14:57:42+03:00 + + Paul Buetow aka snonux + paul@dev.buetow.org + + Hey there. As I am pretty busy this month personally (I am now on Paternity Leave) and as I still want to post once monthly, the blog post of this month will only be some DTail usage examples. They're from the DTail documentation, but not all readers of my blog may be aware of those! + +
+

DTail usage examples


+
+Published at 2023-09-25T14:57:42+03:00
+
+Hey there. As I am pretty busy this month personally (I am now on Paternity Leave) and as I still want to post once monthly, the blog post of this month will only be some DTail usage examples. They're from the DTail documentation, but not all readers of my blog may be aware of those!
+
+DTail is a distributed DevOps tool for tailing, grepping, catting logs and other text files on many remote machines at once which I programmed in Go.
+
+https://dtail.dev
+
+
+                              ,_---~~~~~----._
+                        _,,_,*^____      _____``*g*\"*,
+  ____ _____     _ _   / __/ /'     ^.  /      \ ^@q   f
+ |  _ \_   _|_ _(_) |   @f |      ((@|  |@))    l  0 _/
+ | | | || |/ _` | | |  \`/   \~____ / __ \_____/    \
+ | |_| || | (_| | | |   |           _l__l_           I
+ |____/ |_|\__,_|_|_|   }          [______]           I
+                        ]            | | |            |
+                        ]             ~ ~             |
+                        |   Let's tail those logs!   |
+                         |                           |
+
+
+DTail consists out of a server and several client binaries. In this post, I am showcasing their use!
+
+
    +
  • Use dtail to follow logs
    • +
  • Use dtail to aggregate logs while they are followed
    • +
  • Use dcat to display logs and other text files already written
    • +
  • Use dgrep to grep (search) logs and other text files already written
    • +
  • Use dmap to aggregate logs and other text files already written
    • +
  • dserver is the DTail server, where all the clients can connect to
    • +

+

Following logs


+
+The following example demonstrates how to follow logs of several servers at once. The server list is provided as a flat text file. The example filters all records containing the string INFO. Any other Go compatible regular expression can also be used instead of INFO.
+
+ +
% dtail --servers serverlist.txt --grep INFO --files "/var/log/dserver/*.log"
+
+
+Hint: you can also provide a comma separated server list, e.g.: servers server1.example.org,server2.example.org:PORT,...
+
+Tail example
+
+Hint: You can also use the shorthand version (omitting the --files)
+
+ +
% dtail --servers serverlist.txt --grep INFO "/var/log/dserver/*.log"
+
+
+

Aggregating logs


+
+To run ad-hoc map-reduce aggregations on newly written log lines you must add a query. The following example follows all remote log lines and prints out every few seconds the result to standard output.
+
+Hint: To run a map-reduce query across log lines written in the past, please use the dmap command instead.
+
+ +
% dtail --servers serverlist.txt \
+    --files '/var/log/dserver/*.log' \
+    --query 'from STATS select sum($goroutines),sum($cgocalls),
+             last($time),max(lifetimeConnections)'
+
+
+Beware: For map-reduce queries to work, you have to ensure that DTail supports your log format. Check out the documentaiton of the DTail query language and the DTail log formats on the DTail homepage for more information.
+
+Tail map-reduce example
+
+Hint: You can also use the shorthand version:
+
+ +
% dtail --servers serverlist.txt \
+    --files '/var/log/dserver/*.log' \
+    'from STATS select sum($goroutines),sum($cgocalls),
+     last($time),max(lifetimeConnections)'
+
+
+Here is another example:
+
+ +
% dtail --servers serverlist.txt \
+    --files '/var/log/dserver/*.log' \
+    --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
+             lifetimeConnections group by $hostname order by max($cgocalls)'
+
+
+Tail map-reduce example 2
+
+You can also continuously append the results to a CSV file by adding outfile append filename.csv to the query:
+
+ +
% dtail --servers serverlist.txt \
+    --files '/var/log/dserver/*.log' \
+    --query 'from STATS select ... outfile append result.csv'
+
+
+

How to use dcat


+
+The following example demonstrates how to cat files (display the full content of the files) on several servers at once.
+
+As you can see in this example, a DTail client also creates a local log file of all received data in ~/log. You can also use the noColor and -plain flags (this all also work with other DTail commands than dcat).
+
+ +
% dcat --servers serverlist.txt --files /etc/hostname
+
+
+Cat example
+
+Hint: You can also use the shorthand version:
+
+ +
% dcat --servers serverlist.txt /etc/hostname
+
+
+

How to use dgrep


+
+The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after some entries in /etc/passwd. This time, we don't provide the server list via an file but rather via a comma separated list directly on the command line. We also explore the -before, -after and -max flags (see animation).
+
+ +
% dgrep --servers server1.example.org:2223 \
+    --files /etc/passwd \
+    --regex nologin
+
+
+Generally, dgrep is also a very useful way to search historic application logs for certain content.
+
+Grep example
+
+Hint: -regex is an alias for -grep.
+
+

How to use dmap


+
+To run a map-reduce aggregation over logs written in the past, the dmap command can be used. The following example aggregates all map-reduce fields dmap will print interim results every few seconds. You can also write the result to an CSV file by adding outfile result.csv to the query.
+
+ +
% dmap --servers serverlist.txt \
+    --files '/var/log/dserver/*.log' \
+    --query 'from STATS select $hostname,max($goroutines),max($cgocalls),$loadavg,
+             lifetimeConnections group by $hostname order by max($cgocalls)'
+
+
+Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in internal/mapr/logformat or add an extension to support a custom log format. The example here works out of the box though, as DTail understands its own log format already.
+
+DMap example
+
+

How to use the DTail serverless mode


+
+Until now, all examples so far required to have remote server(s) to connect to. That makes sense, as after all DTail is a *distributed* tool. However, there are circumstances where you don't really need to connect to a server remotely. For example, you already have a login shell open to the server an all what you want is to run some queries directly on local log files.
+
+The serverless mode does not require any dserver up and running and therefore there is no networking/SSH involved.
+
+All commands shown so far also work in a serverless mode. All what needs to be done is to omit a server list. The DTail client then starts in serverless mode.
+
+

Serverless map-reduce query


+
+The following dmap example is the same as the previously shown one, but the difference is that it operates on a local log file directly:
+
+