diff options
| -rw-r--r-- | doc/examples.md | 37 | ||||
| -rw-r--r-- | doc/installation.md | 4 | ||||
| -rw-r--r-- | doc/quickstart.md | 20 |
3 files changed, 51 insertions, 10 deletions
diff --git a/doc/examples.md b/doc/examples.md index 91ab7f2..6c23120 100644 --- a/doc/examples.md +++ b/doc/examples.md @@ -15,6 +15,12 @@ The following example demonstrates how to follow logs of multiple servers at onc  +You can also use the shorthand version: + +```shell +% dtail --servers serverlist.txt --regex STAT "/var/log/service/*.log" +``` + ## Aggregating logs To run ad-hoc MapReduce aggregations on newly written log lines, you also must add a query. The following example follows all remote log lines and prints out every 5 seconds the top 10 servers with the most average free memory. To run a MapReduce query across log lines written in the past, please use the ``dmap`` command instead. @@ -29,6 +35,14 @@ For MapReduce queries to work, you have to ensure that DTail supports your log f  +You can also use the shorthand version: + +```shell +% dtail --servers serverlist.txt \ + 'select avg(memfree), $hostname from MCVMSTATS group by $hostname order by avg(memfree) limit 10 interval 5' \ + '/var/log/service/*.log' +``` + # How to use ``dcat`` The following example demonstrates how to cat files (display the full content of the files) of multiple servers at once. The servers are provided as a comma-separated list this time. @@ -40,6 +54,13 @@ The following example demonstrates how to cat files (display the full content of  +You can also use the shorthand version: + +```shell +% dcat --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org \ + /etc/hostname +``` + # How to use ``dgrep`` The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after the swap partition in ``/etc/fstab``. We do that only on the first 20 servers from ``serverlist.txt``. ``dgrep`` is also very useful for searching log files of the past. @@ -52,6 +73,14 @@ The following example demonstrates how to grep files (display only the lines whi  +You can also use the shorthand version: + +TODO: Auto detect that swap is a regex. +```shell +% dgrep --servers <(head -n 20 serverlist.txt) \ + /etc/fstab swap +``` + # How to use ``dmap`` To run a MapReduce aggregation over logs written in the past, the ``dmap`` command can be used. For example, the following command aggregates all MapReduce fields of all the records and calculates the average memory free grouped by day of the month, hour, minute and the server hostname. ``dmap`` will print interim results every few seconds. The final product, however, will be written to file ``mapreduce.csv``. @@ -65,3 +94,11 @@ To run a MapReduce aggregation over logs written in the past, the ``dmap`` comma Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in ``internal/mapr/log format`` or add an extension to support a custom log format.  + +You can also use the shorthand version: + +```shell +% dmap --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org,serv-021.lan.example.org,serv-022.lan.example.org,serv-023.lan.example.org \ + 'select avg(memfree), $day, $hour, $minute, $hostname from MCVMSTATS group by $day, $hour, $minute, $hostname order by avg(memfree) limit 10 outfile mapreduce.csv' \ + "/var/log/service/*.log" +``` diff --git a/doc/installation.md b/doc/installation.md index 8f3892c..0f6143b 100644 --- a/doc/installation.md +++ b/doc/installation.md @@ -19,10 +19,10 @@ This is optional, but it gives you better security. On Linux, you have the optio ### 2. Enable ACL via a Go build flag -Set the `USE_ACL` environment variable before invoking the make command. +Set the `DTAIL_USE_ACL` environment variable before invoking the make command. ```console -% export USE_ACL=yes +% export DTAIL_USE_ACL=yes ``` Alternatively, you could add `-tags linuxacl` to the Go compiler. diff --git a/doc/quickstart.md b/doc/quickstart.md index 21274ff..ea943d3 100644 --- a/doc/quickstart.md +++ b/doc/quickstart.md @@ -10,7 +10,7 @@ This guide assumes that you know how to generate and configure a public/private To compile and install all DTail binaries directly from GitHub run: ```console -% for cmd in dcat dgrep dmap dtail dserver; do +% for cmd in dcat dgrep dmap dtail dserver dtailhealth; do go get github.com/mimecast/dtail/cmd/$cmd; done ``` @@ -21,6 +21,7 @@ It produces the following executables in ``$GOPATH/bin``: * ``dgrep``: Client for searching whole files remotely using a regex (distributed grep) * ``dmap``: Client for executing distributed MapReduce queries (may consume a lot of RAM and CPU) * ``dtail``: Client for tailing/following log files remotely (distributed tail) +* ``dtailhealth``: Client for dserver health checks * ``dserver``: The DTail server # Start DTail server @@ -28,12 +29,15 @@ It produces the following executables in ``$GOPATH/bin``: Copy the ``dserver`` binary to the remote server machines of your choice (e.g. ``serv-001.lan.example.org`` and ``serv-002.lan.example.org``) and start it on each of the servers as follows: ```console -% ./dserver -SERVER|serv-001|INFO|Launching server|server|DTail 1.0.0 -SERVER|serv-001|INFO|Creating server|DTail 1.0.0 -SERVER|serv-001|INFO|Generating private server RSA host key -SERVER|serv-001|INFO|Starting server -SERVER|serv-001|INFO|Binding server|0.0.0.0:2222 +❯ ./dserver --logger Stdout --logLevel debug --bindAddress $(hostname) --port 2222 +DTail 4.0.0 Protocol 4 Have a lot of fun! +INFO|20211027-102513|Creating server|DTail 4.0.0-RC2 Protocol 4 Have a lot of fun! +INFO|20211027-102513|Reading private server RSA host key from file|./ssh_host_key +INFO|20211027-102513|Starting server +INFO|20211027-102513|Binding server|X.Y.Z.W:2222 +INFO|20211027-102513|Starting continuous job runner after 10s +DEBUG|20211027-102513|Starting listener loop +INFO|20211027-102513|Starting scheduled job runner after 10s ``` ``dserver`` is now listening on TCP port 2222 and waiting for incoming connections. All SSH keys listed in ``~/.ssh/authorized_keys`` are now respected by the DTail server for authorization. @@ -79,7 +83,7 @@ Now it is time to connect to the DTail servers through the DTail client: ```console % dtail --servers serv-001.lan.example.org,server-002.lan.example.org --files "/var/log/service/*.log" -CLIENT|workstation01|INFO|Launching client|tail|DTail 1.0.0 +CLIENT|workstation01|INFO|Launching client|tail|DTail 4.0.0 CLIENT|workstation01|INFO|Initiating base client CLIENT|workstation01|INFO|Added SSH Agent to list of auth methods CLIENT|workstation01|INFO|Deduped server list|1|1 |