diff options
| -rw-r--r-- | TODO.md | 21 | ||||
| -rw-r--r-- | doc/dtail-gopher.png | bin | 0 -> 227162 bytes | |||
| -rw-r--r-- | doc/examples.md | 16 | ||||
| -rw-r--r-- | doc/installation.md | 14 | ||||
| -rw-r--r-- | doc/quickstart.md | 18 | ||||
| -rw-r--r-- | doc/title.png | bin | 0 -> 84933 bytes | |||
| -rw-r--r-- | doc/title.xcf | bin | 0 -> 124226 bytes | |||
| -rw-r--r-- | integrationtests/fileutils.go | 1 |
8 files changed, 37 insertions, 33 deletions
@@ -3,15 +3,18 @@ TODO This is a loose list of what to do. Maybe for the next releae or maybe for a later one. -[ ] Client 4.x should print a warning when trying to connect to a 3.x server. -[ ] Client 3.x should print a warning when trying to connect to a 4.x server. -[ ] Update docs for color configuration -[ ] Update animated gifs -[ ] Document that can use additional args as file lists -[ ] Document spartan mode -[ ] Document serverless mode -[ ] Go through the git history and document more stuff +[ ] Manually add back all changes from mimecast master to this branch. +[ ] Client 4.x should print an error and exit when trying to connect to a 3.x server. +[ ] Client 3.x should print an error and exit when trying to connect to a 4.x server. +[ ] Create a GitHub Wiki + [ ] Migrate existing documentation + update animated Gifs + [ ] Document that can use additional args as file lists + [ ] Document spartan mode + [ ] Document serverless mode + [ ] Document color configuratio + [ ] Go through the git history and document more stuff [ ] Manual test/adjust dtail colors [ ] More integration test colors (via dcat?) +[ ] Integration test grep context' [ ] Integration test for dtail in serverless mode -[ ] Integration test for dtail normal mode +[x] Integration test for dtail normal mode diff --git a/doc/dtail-gopher.png b/doc/dtail-gopher.png Binary files differnew file mode 100644 index 0000000..5d9727e --- /dev/null +++ b/doc/dtail-gopher.png diff --git a/doc/examples.md b/doc/examples.md index 964660a..91ab7f2 100644 --- a/doc/examples.md +++ b/doc/examples.md @@ -1,13 +1,13 @@ Examples ======== -This page demonstrate the basic usage of DTail. Please also see ``dtail --help`` for more available options. +This page demonstrates the primary usage of DTail. Please also see ``dtail --help`` for more available options. # How to use ``dtail`` ## Tailing logs -The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all logs containing the string ``STAT``. Any other Go compatible regular expression can be used instead of ``STAT``. +The following example demonstrates how to follow logs of multiple servers at once. The server list is provided as a flat text file. The example filters all records containing the string ``STAT``. Any other Go compatible regular expression can be used instead of ``STAT``. ```shell % dtail --servers serverlist.txt --files "/var/log/service/*.log" --regex STAT @@ -17,7 +17,7 @@ The following example demonstrates how to follow logs of multiple servers at onc ## Aggregating logs -To run ad-hoc mapreduce aggregations on newly written log lines you also must add a query. This example follows all remote log lines and prints out every 5 seconds the top 10 servers with most average free memory according to the logs. To run a mapreduce query across log lines written in the past please use the ``dmap`` command instead. +To run ad-hoc MapReduce aggregations on newly written log lines, you also must add a query. The following example follows all remote log lines and prints out every 5 seconds the top 10 servers with the most average free memory. To run a MapReduce query across log lines written in the past, please use the ``dmap`` command instead. ```shell % dtail --servers serverlist.txt \ @@ -25,13 +25,13 @@ To run ad-hoc mapreduce aggregations on newly written log lines you also must ad --files '/var/log/service/*.log' ``` -In order for mapreduce queries to work you have to make sure that your log format is supported by DTail. You can either use the ones which are already defined in ``internal/mapr/logformat`` or add an extension to support a custom log format. +For MapReduce queries to work, you have to ensure that DTail supports your log format. You can either use the ones already defined in ``internal/mapr/log format`` or add an extension to support a custom log format.  # How to use ``dcat`` -The following example demonstrates how to cat files (display the whole content of the files) of multiple servers at once. The servers are provided as a comma separated list this time. +The following example demonstrates how to cat files (display the full content of the files) of multiple servers at once. The servers are provided as a comma-separated list this time. ```shell % dcat --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org \ @@ -42,7 +42,7 @@ The following example demonstrates how to cat files (display the whole content o # How to use ``dgrep`` -The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example we look after the swap partition in ``/etc/fstab``. We do that only on the first 20 servers from ``serverlist.txt``. ``dgrep`` is also very useful for searching log files of the past. +The following example demonstrates how to grep files (display only the lines which match a given regular expression) of multiple servers at once. In this example, we look after the swap partition in ``/etc/fstab``. We do that only on the first 20 servers from ``serverlist.txt``. ``dgrep`` is also very useful for searching log files of the past. ```shell % dgrep --servers <(head -n 20 serverlist.txt) \ @@ -54,7 +54,7 @@ The following example demonstrates how to grep files (display only the lines whi # How to use ``dmap`` -To run a mapreduce aggregation over logs written in the past the ``dmap`` command can be used. For example the following command aggregates all mapreduce fields of all the logs and calculates the average memory free grouped by day of the month, hour, minute and the server hostname. ``dmap`` will print interim results every few seconds. The final result however will be written to file ``mapreduce.csv``. +To run a MapReduce aggregation over logs written in the past, the ``dmap`` command can be used. For example, the following command aggregates all MapReduce fields of all the records and calculates the average memory free grouped by day of the month, hour, minute and the server hostname. ``dmap`` will print interim results every few seconds. The final product, however, will be written to file ``mapreduce.csv``. ```shell % dmap --servers serv-011.lan.example.org,serv-012.lan.example.org,serv-013.lan.example.org,serv-021.lan.example.org,serv-022.lan.example.org,serv-023.lan.example.org \ @@ -62,6 +62,6 @@ To run a mapreduce aggregation over logs written in the past the ``dmap`` comman --files "/var/log/service/*.log" ``` -Remember: In order for that to work you have to make sure that your log format is supported by DTail. You can either use the ones which are already defined in ``internal/mapr/logformat`` or add an extension to support a custom log format. +Remember: For that to work, you have to make sure that DTail supports your log format. You can either use the ones already defined in ``internal/mapr/log format`` or add an extension to support a custom log format.  diff --git a/doc/installation.md b/doc/installation.md index 6bf17b0..8f3892c 100644 --- a/doc/installation.md +++ b/doc/installation.md @@ -5,13 +5,13 @@ The following installation guide has been tested successfully on CentOS 7. You m # Compile it -Please check the [Quick Starting Guide](quickstart.md) for instructions how to compile DTail. It is recommended to automate the build process via your build pipeline (e.g. produce a deployable RPM via Jenkins). You don't have to use ``go get...`` to compile and install the binaries. You can also clone the repository and use ``make`` instead. +Please check the [Quick Starting Guide](quickstart.md) for instructions on compiling DTail. It is recommended to automate the build process via your build pipeline (e.g. produce a deployable (.rpm, .deb, ...) via Jenkins). You don't have to use ``go get...`` to compile and install the binaries. You can also clone the repository and use ``make`` instead. ## Linux ACL support -This is optional but it gives you better security. On Linux you have the option to compile `dserver` with File System Access Control List support. For that you need: +This is optional, but it gives you better security. On Linux, you have the option to compile `dserver` with File System Access Control List support. For that, you need: -### 1. Install the `libacl` development library. On RHEL, CentOS and Fedora it would be +### 1. Install the `libacl` development library. On RHEL, CentOS and Fedora, it would be ```console % sudo dnf install libacl-devel -y @@ -25,7 +25,7 @@ Set the `USE_ACL` environment variable before invoking the make command. % export USE_ACL=yes ``` -Alternatively you could just add `-tags linuxacl` to the Go compiler. +Alternatively, you could add `-tags linuxacl` to the Go compiler. # Install it @@ -95,7 +95,7 @@ To start the DTail server via ``systemd`` run: # Register SSH public keys in DTail server -The DTail server now runs as a ``systemd`` service under system user ``dserver``. The system user ``dserver`` however has no permissions to read the SSH public keys from ``/home/USER/.ssh/authorized_keys``. Therefore, no user would be able to establish a SSH session to DTail server. As an alternative path DTail server also checks for public SSH key files in ``/var/run/dserver/cache/USER.authorized_keys``. +The DTail server now runs as a ``systemd`` service under system user ``dserver``. However, the system user ``dserver`` has no permissions to read the SSH public keys from ``/home/USER/.ssh/authorized_keys``. Therefore, no user would be able to establish an SSH session to DTail server. As an alternative path DTail server also checks for public SSH key files in ``/var/run/dserver/cache/USER.authorized_keys``. It is recommended to execute [update_key_cache.sh](../samples/update_key_cache.sh.sample) periodically to update the key cache. In case you manage your public SSH keys via Puppet you could subscribe the script to corresponding module. Or alternatively just configure a cron job or a systemd timer to run every once in a while, e.g. every 30 minutes: @@ -115,11 +115,11 @@ It is recommended to execute [update_key_cache.sh](../samples/update_key_cache.s # Run DTail client -Now you should be able to use DTail client like outlined in the [Quick Starting Guide](quickstart.md). Also have a look at the [Examples](examples.md). +Now you should be able to use DTail client like outlined in the [Quick Starting Guide](quickstart.md). Also, have a look at the [Examples](examples.md). # Monitor it -To verify that DTail server is up and running and functioning as expected you should configure the Nagios check [check_dserver.sh](../samples/check_dserver.sh.sample) in your monitoring system. The check has to be executed locally on the server (e.g. via NRPE). How to configure the monitoring system in detail is out of scope of this guide. +To verify that DTail server is up and running and functioning as expected, you should configure the Nagios check [check_dserver.sh](../samples/check_dserver.sh.sample) in your monitoring system. The check has to be executed locally on the server (e.g. via NRPE). How to configure the monitoring system in detail is out of scope of this guide. ```console % ./check_dserver.sh diff --git a/doc/quickstart.md b/doc/quickstart.md index f1ac000..21274ff 100644 --- a/doc/quickstart.md +++ b/doc/quickstart.md @@ -1,9 +1,9 @@ Quick Starting Guide ==================== -This is the quick starting guide. For a more sustainable setup, involving how to create a background service via ``systemd``, recommendations about automation via Jenkins and/or Puppet and health monitoring via Nagios please also follow the [Installation Guide](installation.md). +This is the quick starting guide. For a more sustainable setup involving creating a background service via ``systemd``, recommendations about automation via Jenkins and Puppet and health monitoring via Nagios, please follow the [Installation Guide](installation.md). -This guide assumes that you know how to generate and configure a public/private SSH key pair for secure authorization and shell access. For more information please have a look at the OpenSSH documentation of your distribution. +This guide assumes that you know how to generate and configure a public/private SSH key pair for secure authorization and shell access. For more information, please have a look at the OpenSSH documentation of your distribution. # Install it @@ -18,8 +18,8 @@ To compile and install all DTail binaries directly from GitHub run: It produces the following executables in ``$GOPATH/bin``: * ``dcat``: Client for displaying whole files remotely (distributed cat) -* ``dgrep``: Client for searching whole files files remotely using a regex (distributed grep) -* ``dmap``: Client for executing distributed mapreduce queries (may will consume a lot of RAM and CPU) +* ``dgrep``: Client for searching whole files remotely using a regex (distributed grep) +* ``dmap``: Client for executing distributed MapReduce queries (may consume a lot of RAM and CPU) * ``dtail``: Client for tailing/following log files remotely (distributed tail) * ``dserver``: The DTail server @@ -42,13 +42,13 @@ SERVER|serv-001|INFO|Binding server|0.0.0.0:2222 ## Setup SSH -Make sure that your public SSH key is listed in ``~/.ssh/authorized_keys`` on all server machines involved. The private SSH key counterpart should preferably stay on your Laptop or workstation in ``~/.ssh/id_rsa`` or ``~/.ssh/id_dsa``. +Ensure that your public SSH key is listed in ``~/.ssh/authorized_keys`` on all server machines involved. The private SSH key counterpart should preferably stay on your Laptop or workstation in ``~/.ssh/id_rsa`` or ``~/.ssh/id_dsa``. -DTail relies on SSH for secure authentication and communication. You can either use a SSH Agent or a private SSH key file directly. +DTail relies on SSH for secure authentication and communication. You can either use an SSH Agent or a private SSH key file directly. ### SSH Agent -The clients (all client binaries such as ``dtail``, ``dgrep`` and so on...) communicate with an auth backend via the SSH auth socket. The SSH auth socket is configured via the environment variable ``SSH_AUTH_SOCK`` which usually points to ``~/.ssh/ssh_auth_socket`` or similar (depending on your configuration it may also point to other auth backends such as GPG Agent, in which case ``SSH_AUTH_SOCK`` would point to ``~/.gnupg/S.gpg-agent.ssh`` or similar). +The clients (all client binaries such as ``dtail``, ``dgrep`` and so on...) communicate with an auth backend via the SSH auth socket. The SSH auth socket is configured via the environment variable ``SSH_AUTH_SOCK`` which usually points to ``~/.ssh/ssh_auth_socket`` or similar (depending on your configuration, it may also point to other auth backends such as GPG Agent, in which case ``SSH_AUTH_SOCK`` would point to ``~/.gnupg/S.gpg-agent.ssh`` or similar). Usually you would use the SSH Auth Agent. For this the private SSH key has to be registered at the SSH Agent: @@ -58,7 +58,7 @@ Enter passphrase for ~/.ssh/id_rsa: ********** Identity added: ~/.ssh/id_rsa (~/.ssh/id_rsa) ``` -To test whether SSH is setup correctly you should be able to SSH into the servers with the OpenSSH client and your private SSH key through the SSH Agent without entering the private keys passphrase. The following assumes to have an OpenSSH server running on ``serv-001.lan.example.org`` and an OpenSSH client installed on your laptop or workstation. Please notice that DTail does not require to have an OpenSSH infrastructure set up but DTail uses by default the same public/private key file paths as OpenSSH. OpenSSH can be of a great help to verify that the SSH keys are configured correctly: +To test whether SSH is set up correctly, you should be able to SSH into the servers with the OpenSSH client and your private SSH key through the SSH Agent without entering the private key's passphrase. The following assumes to have an OpenSSH server running on ``serv-001.lan.example.org`` and an OpenSSH client installed on your laptop or workstation. Please notice that DTail does not require to have an OpenSSH infrastructure set up, but DTail uses by default the same public/private key file paths as OpenSSH. OpenSSH can be of great help to verify that the SSH keys are configured correctly: ```console workstation01 ~ % ssh serv-001.lan.example.org @@ -71,7 +71,7 @@ Please consult the OpenSSH documentation of your distribution if the test above ### SSH Private Key file -As an alternative to using a SSH Agent a SSH private key file can be used directly. Just add the argument ``--key ~/.ssh/id_rsa`` (pointing to your private key) to the DTail client. This currently does not work with password protected keys. Use the SSH Agent method instead in case your key comes with a password (recommended). +As an alternative to using an SSH Agent, an SSH private key file can be used directly. Just add the argument ``--key ~/.ssh/id_rsa`` (pointing to your private key) to the DTail client. This currently does not work with password-protected keys. Use the SSH Agent method instead, in case your key comes with a password (recommended). ## Run DTail client diff --git a/doc/title.png b/doc/title.png Binary files differnew file mode 100644 index 0000000..4e343c4 --- /dev/null +++ b/doc/title.png diff --git a/doc/title.xcf b/doc/title.xcf Binary files differnew file mode 100644 index 0000000..257d36c --- /dev/null +++ b/doc/title.xcf diff --git a/integrationtests/fileutils.go b/integrationtests/fileutils.go index d771607..8ab66a0 100644 --- a/integrationtests/fileutils.go +++ b/integrationtests/fileutils.go @@ -47,6 +47,7 @@ func compareFilesContents(t *testing.T, fileA, fileB string) error { return nil } + // Read files into maps. a, err := mapFile(fileA) if err != nil { return err |
