From 8931f6ae200ff54cbd2e2ee52e29463429f40a32 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Fri, 24 Mar 2023 12:14:14 +0200 Subject: Update content for gemtext --- ...4-22-dtail-the-distributed-log-tail-program.gmi | 6 + ...-dtail-the-distributed-log-tail-program.gmi.tpl | 112 +++++ gemfeed/2021-04-24-welcome-to-the-geminispace.gmi | 6 + .../2021-04-24-welcome-to-the-geminispace.gmi.tpl | 87 ++++ ...2021-05-16-personal-bash-coding-style-guide.gmi | 7 + ...-05-16-personal-bash-coding-style-guide.gmi.tpl | 389 ++++++++++++++++ ...05-gemtexter-one-bash-script-to-rule-it-all.gmi | 13 +- ...emtexter-one-bash-script-to-rule-it-all.gmi.tpl | 173 ++++++++ gemfeed/2021-11-29-bash-golf-part-1.gmi | 11 +- gemfeed/2021-11-29-bash-golf-part-1.gmi.tpl | 471 ++++++++++++++++++++ gemfeed/2022-01-01-bash-golf-part-2.gmi | 11 +- gemfeed/2022-01-01-bash-golf-part-2.gmi.tpl | 488 +++++++++++++++++++++ gemfeed/2022-03-06-the-release-of-dtail-4.0.0.gmi | 12 +- .../2022-03-06-the-release-of-dtail-4.0.0.gmi.tpl | 299 +++++++++++++ ...22-08-27-gemtexter-1.1.0-lets-gemtext-again.gmi | 11 +- ...8-27-gemtexter-1.1.0-lets-gemtext-again.gmi.tpl | 92 ++++ gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi | 6 + .../2022-10-30-installing-dtail-on-openbsd.gmi.tpl | 348 +++++++++++++++ gemfeed/atom.xml | 69 ++- index.gmi | 2 +- uptime-stats.gmi | 2 +- 21 files changed, 2581 insertions(+), 34 deletions(-) create mode 100644 gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi.tpl create mode 100644 gemfeed/2021-04-24-welcome-to-the-geminispace.gmi.tpl create mode 100644 gemfeed/2021-05-16-personal-bash-coding-style-guide.gmi.tpl create mode 100644 gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi.tpl create mode 100644 gemfeed/2021-11-29-bash-golf-part-1.gmi.tpl create mode 100644 gemfeed/2022-01-01-bash-golf-part-2.gmi.tpl create mode 100644 gemfeed/2022-03-06-the-release-of-dtail-4.0.0.gmi.tpl create mode 100644 gemfeed/2022-08-27-gemtexter-1.1.0-lets-gemtext-again.gmi.tpl create mode 100644 gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi.tpl diff --git a/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi b/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi index 7fad7f1b..250e7ffe 100644 --- a/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi +++ b/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi @@ -103,6 +103,12 @@ Mimecast highly encourages you to have a look at DTail and submit an issue for a => https://dtail.dev +More related posts are: + +=> ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD +=> ./2022-03-06-the-release-of-dtail-4.0.0.gmi 2022-03-06 The release of DTail 4.0.0 +=> ./2021-04-22-dtail-the-distributed-log-tail-program.gmi 2021-04-22 DTail - The distributed log tail program (You are currently reading this) + E-Mail your comments to hi@paul.cyou :-) => ../ Back to the main site diff --git a/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi.tpl b/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi.tpl new file mode 100644 index 00000000..861ac636 --- /dev/null +++ b/gemfeed/2021-04-22-dtail-the-distributed-log-tail-program.gmi.tpl @@ -0,0 +1,112 @@ +# DTail - The distributed log tail program + +> Published at 2021-04-22T19:28:41+01:00; Updated at 2021-04-26 + +=> ./2021-04-22-dtail-the-distributed-log-tail-program/title.png DTail logo image + +This article first appeared at the Mimecast Engineering Blog but I made it available here in my personal internet site too. + +=> https://medium.com/mimecast-engineering/dtail-the-distributed-log-tail-program-79b8087904bb Original Mimecast Engineering Blog post at Medium + +Running a large cloud-based service requires monitoring the state of huge numbers of machines, a task for which many standard UNIX tools were not really designed. In this post, I will describe a simple program, DTail, that Mimecast has built and released as Open-Source, which enables us to monitor log files of many servers at once without the costly overhead of a full-blown log management system. + +At Mimecast, we run over 10 thousand server boxes. Most of them host multiple microservices and each of them produces log files. Even with the use of time series databases and monitoring systems, raw application logs are still an important source of information when it comes to analysing, debugging, and troubleshooting services. + +Every engineer familiar with UNIX or a UNIX-like platform (e.g., Linux) is well aware of tail, a command-line program for displaying a text file content on the terminal which is also especially useful for following application or system log files with tail -f logfile. + +Think of DTail as a distributed version of the tail program which is very useful when you have a distributed application running on many servers. DTail is an Open-Source, cross-platform, fairly easy to use, support and maintain log file analysis & statistics gathering tool designed for Engineers and Systems Administrators. It is programmed in Google Go. + +## A Mimecast Pet Project + +DTail got its inspiration from public domain tools available already in this area but it is a blue sky from-scratch development which was first presented at Mimecast’s annual internal Pet Project competition (awarded with a Bronze prize). It has gained popularity since and is one of the most widely deployed DevOps tools at Mimecast (reaching nearly 10k server installations) and many engineers use it on a regular basis. The Open-Source version of DTail is available at: + +=> https://dtail.dev + +Try it out — We would love any feedback. But first, read on… + +## Differentiating from log management systems + +Why not just use a full-blown log management system? There are various Open-Source and commercial log management solutions available on the market you could choose from (e.g. the ELK stack). Most of them store the logs in a centralized location and are fairly complex to set up and operate. Possibly they are also pretty expensive to operate if you have to buy dedicated hardware (or pay fees to your cloud provider) and have to hire support staff for it. + +DTail does not aim to replace any of the log management tools already available but is rather an additional tool crafted especially for ad-hoc debugging and troubleshooting purposes. DTail is cheap to operate as it does not require any dedicated hardware for log storage as it operates directly on the source of the logs. It means that there is a DTail server installed on all server boxes producing logs. This decentralized comes with the direct advantages that there is no introduced delay because the logs are not shipped to a central log storage device. The reduced complexity also makes it more robust against outages. You won’t be able to troubleshoot your distributed application very well if the log management infrastructure isn’t working either. + +=> ./2021-04-22-dtail-the-distributed-log-tail-program/dtail.gif DTail sample session animated gif + +As a downside, you won’t be able to access any logs with DTail when the server is down. Furthermore, a server can store logs only up to a certain capacity as disks will fill up. For the purpose of ad-hoc debugging, these are not typically issues. Usually, it’s the application you want to debug and not the server. And disk space is rarely an issue for bare metal and VM-based systems these days, with sufficient space for several weeks’ worth of log storage being available. DTail also supports reading compressed logs. The currently supported compression algorithms are gzip and zstd. + +## Combining simplicity, security and efficiency + +DTail also has a client component that connects to multiple servers concurrently for log files (or any other text files). + +The DTail client interacts with a DTail server on port TCP/2222 via SSH protocol and does not interact in any way with the system’s SSH server (e.g., OpenSSH Server) which might be running at port TCP/22 already. As a matter of fact, you don’t need a regular SSH server running for DTail at all. There is no support for interactive login shells at TCP/2222 either, as by design that port can only be used for text data streaming. The SSH protocol is used for the public/private key infrastructure and transport encryption only and DTail implements its own protocol on top of SSH for the features provided. There is no need to set up or buy any additional TLS certificates. The port 2222 can be easily reconfigured if you preferred to use a different one. + +The DTail server, which is a single static binary, will not fork an external process. This means that all features are implemented in native Go code (exception: Linux ACL support is implemented in C, but it must be enabled explicitly on compile time) and therefore helping to make it robust, secure, efficient, and easy to deploy. A single client, running on a standard Laptop, can connect to thousands of servers concurrently while still maintaining a small resource footprint. + +Recent log files are very likely still in the file system caches on the servers. Therefore, there tends to be a minimal I/O overhead involved. + +## The DTail family of commands + +Following the UNIX philosophy, DTail includes multiple command-line commands each of them for a different purpose: + +* dserver: The DTail server, the only binary required to be installed on the servers involved. +* dtail: The distributed log tail client for following log files. +* dcat: The distributed cat client for concatenating and displaying text files. +* dgrep: The distributed grep client for searching text files for a regular expression pattern. +* dmap: The distributed map-reduce client for aggregating stats from log files. + +=> ./2021-04-22-dtail-the-distributed-log-tail-program/dgrep.gif DGrep sample session animated gif + +## Usage example + +The use of these commands is almost self-explanatory for a person already used to the standard command line in Unix systems. One of the main goals is to make DTail easy to use. A tool that is too complicated to use under high-pressure scenarios (e.g., during an incident) can be quite detrimental. + +The basic idea is to start one of the clients from the command line and provide a list of servers to connect to with –servers. You also must provide a path of remote (log) files via –files. If you want to process multiple files per server, you could either provide a comma-separated list of file paths or make use of file system globbing (or a combination of both). + +The following example would connect to all DTail servers listed in the serverlist.txt, follow all files with the ending .log and filter for lines containing the string error. You can specify any Go compatible regular expression. In this example we add the case-insensitive flag to the regex: + +``` +dtail –servers serverlist.txt –files ‘/var/log/*.log’ –regex ‘(?i:error)’ +``` + +You usually want to specify a regular expression as a client argument. This will mean that responses are pre-filtered for all matching lines on the server-side and thus sending back only the relevant lines to the client. If your logs are growing very rapidly and the regex is not specific enough there might be the chance that your client is not fast enough to keep up processing all of the responses. This could be due to a network bottleneck or just as simple as a slow terminal emulator displaying the log lines on the client-side. + +A green 100 in the client output before each log line received from the server always indicates that there were no such problems and 100% of all log lines could be displayed on your terminal (have a look at the animated Gifs in this post). If the percentage falls below 100 it means that some of the channels used by the servers to send data to the client are congested and lines were dropped. In this case, the color will change from green to red. The user then could decide to run the same query but with a more specific regex. + +You could also provide a comma-separated list of servers as opposed to a text file. There are many more options you could use. The ones listed here are just the very basic ones. There are more instructions and usage examples on the GitHub page. Also, you can study even more of the available options via the –help switch (some real treasures might be hidden there). + +## Fitting it in + +DTail integrates nicely into the user management of existing infrastructure. It follows normal system permissions and does not open new “holes” on the server which helps to keep security departments happy. The user would not have more or less file read permissions than he would have via a regular SSH login shell. There is a full SSH key, traditional UNIX permissions, and Linux ACL support. There is also a very low resource footprint involved. On average for tailing and searching log files less than 100MB RAM and less than a quarter of a CPU core per participating server are required. Complex map-reduce queries on big data sets will require more resources accordingly. + +## Advanced features + +The features listed here are out of the scope of this blog post but are worthwhile to mention: + +* Distributed map-reduce queries on stats provided in log files with dmap. dmap comes with its own SQL-like aggregation query language. +* Stats streaming with continuous map-reduce queries. The difference to normal queries is that the stats are aggregated over a specified interval only on the newly written log lines. Thus, giving a de-facto live stat view for each interval. +* Server-side scheduled queries on log files. The queries are configured in the DTail server configuration file and scheduled at certain time intervals. Results are written to CSV files. This is useful for generating daily stats from the log files without the need for an interactive client. +* Server-side stats streaming with continuous map-reduce queries. This for example can be used to periodically generate stats from the logs at a configured interval, e.g., log error counts by the minute. These then can be sent to a time-series database (e.g., Graphite) and then plotted in a Grafana dashboard. +* Support for custom extensions. E.g., for different server discovery methods (so you don’t have to rely on plain server lists) and log file formats (so that map-reduce queries can parse more stats from the logs). + +## For the future + +There are various features we want to see in the future. + +* A spartan mode, not printing out any extra information but the raw remote log files would be a nice feature to have. This will make it easier to post-process the data produced by the DTail client with common UNIX tools. (To some degree this is possible already, just disable the ANSI terminal color output of the client with -noColors and pipe the output to another program). +* Tempting would be implementing the dgoawk command, a distributed version of the AWK programming language purely implemented in Go, for advanced text data stream processing capabilities. There are 3rd party libraries available implementing AWK in pure Go which could be used. +* A more complex change would be the support of federated queries. You can connect to thousands of servers from a single client running on a laptop. But does it scale to 100k of servers? Some of the servers could be used as middleware for connecting to even more servers. +* Another aspect is to extend the documentation. Especially the advanced features such as map-reduce query language and how to configure the server-side queries currently do require more documentation. For now, you can read the code, sample config files or just ask the author for that! But this will be certainly addressed in the future. + +## Open Source + +Mimecast highly encourages you to have a look at DTail and submit an issue for any features you would like to see. Have you found a bug? Maybe you just have a question or comment? If you want to go a step further: We would also love to see pull requests for any features or improvements. Either way, if in doubt just contact us via the DTail GitHub page. + +=> https://dtail.dev + +More related posts are: + +<< template::inline::index dtail + +E-Mail your comments to hi@paul.cyou :-) + +=> ../ Back to the main site diff --git a/gemfeed/2021-04-24-welcome-to-the-geminispace.gmi b/gemfeed/2021-04-24-welcome-to-the-geminispace.gmi index 8b1864a6..8cf3ea2e 100644 --- a/gemfeed/2021-04-24-welcome-to-the-geminispace.gmi +++ b/gemfeed/2021-04-24-welcome-to-the-geminispace.gmi @@ -78,6 +78,12 @@ Check out one of the following links for more information about Gemini. For exam => gemini://gemini.circumlunar.space => https://gemini.circumlunar.space +More related posts are: + +=> ./2022-08-27-gemtexter-1.1.0-lets-gemtext-again.gmi 2022-08-27 Gemtexter 1.1.0 - Let's Gemtext again +=> ./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi 2021-06-05 Gemtexter - One Bash script to rule it all +=> ./2021-04-24-welcome-to-the-geminispace.gmi 2021-04-24 Welcome to the Geminispace (You are currently reading this) + E-Mail your comments to hi@paul.cyou :-) => ../ Back to the main site diff --git a/gemfeed/2021-04-24-welcome-to-the-geminispace.gmi.tpl b/gemfeed/2021-04-24-welcome-to-the-geminispace.gmi.tpl new file mode 100644 index 00000000..385ccf4e --- /dev/null +++ b/gemfeed/2021-04-24-welcome-to-the-geminispace.gmi.tpl @@ -0,0 +1,87 @@ +# Welcome to the Geminispace + +> Published at 2021-04-24T19:28:41+01:00; Updated at 2021-06-18 + +ASCII Art by Andy Hood! + +Have you reached this article already via Gemini? It requires a Gemini client; web browsers such as Firefox, Chrome, Safari, etc., don't support the Gemini protocol. The Gemini address of this site (or the address of this capsule as people say in Geminispace) is: + +=> gemini://foo.zone + +However, if you still use HTTP, you are just surfing the fallback HTML version of this capsule. In that case, I suggest reading on what this is all about :-). + +``` + + /\ + / \ + | | + |NASA| + | | + | | + | | + ' ` + |Gemini| + | | + |______| + '-`'-` . + / . \'\ . .' + ''( .'\.' ' .;' +'.;.;' ;'.;' ..;;' AsH + +``` + +## Motivation + +### My urge to revamp my personal website + +For some time, I had to urge to revamp my personal website. Not to update the technology and its design but to update all the content (+ keep it current) and start a small tech blog again. So unconsciously, I began to search for an excellent platform to do all of that in a KISS (keep it simple & stupid) way. + +### My still great Laptop running hot + +Earlier this year (2021), I noticed that my almost seven-year-old but still great Laptop started to become hot and slowed down while surfing the web. Also, the Laptop's fan became quite noisy. This was all due to the additional bloat such as JavaScript, excessive use of CSS, tracking cookies+pixels, ads, and so on there was on the website. + +All I wanted was to read an interesting article, but after a big advertising pop-up banner appeared and made everything worse, I gave up and closed the browser tab. + +## Discovering the Gemini internet protocol + +Around the same time, I discovered a relatively new, more lightweight protocol named Gemini, which does not support all these CPU-intensive features like HTML, JavaScript, and CSS. Also, tracking and ads are unsupported by the Gemini protocol. + +The "downside" is that due to the limited capabilities of the Gemini protocol, all sites look very old and spartan. But that is not a downside; that is, in fact, a design choice people made. It is up to the client software how your capsule looks. For example, you could use a graphical client, such as Lagrange, with nice font renderings and colours to improve the appearance. Or you could use a very minimalistic command line black-and-white Gemini client. It's your (the user's) choice. + +=> ./2021-04-24-welcome-to-the-geminispace/amfora-screenshot.png Screenshot Amfora Gemini terminal client surfing this site +=> ./2021-04-24-welcome-to-the-geminispace/lagrange-screenshot.png Screenshot graphical Lagrange Gemini client surfing this site + +Why is there a need for a new protocol? As the modern web is a superset of Gemini, can't we use simple HTML 1.0 instead? That's a good and valid question. It is not a technical problem but a human problem. We tend to abuse the features once they are available. You can ensure that things stay efficient and straightforward as long as you are using the Gemini protocol. On the other hand, you can't force every website on the modern web to only create plain and straightforward-looking HTML pages. + +## My own Gemini capsule + +As it is effortless to set up and maintain your own Gemini capsule (Gemini server + content composed via the Gemtext markup language), I decided to create my own. What I like about Gemini is that I can use my favourite text editor and get typing. I don't need to worry about the style and design of the presence, and I also don't have to test anything in ten different web browsers. I can only focus on the content! As a matter of fact, I am using the Vim editor + its spellchecker + auto word completion functionality to write this. + +This site was generated with Gemtexter. You can read more about it here: + +=> ./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi Gemtexter - One Bash script to rule it all + +## Gemini advantages summarised + +* Supports an alternative to the modern bloated web +* Easy to operate and easy to write content +* No need to worry about various web browser compatibilities +* It's the client's responsibility how the content is designed+presented +* Lightweight (although not as lightweight as the Gopher protocol) +* Supports privacy (no cookies, no request header fingerprinting, TLS encryption) +* Fun to play with (it's a bit geeky, yes, but a lot of fun!) + +## Dive into deep Gemini space + +Check out one of the following links for more information about Gemini. For example, you will find a FAQ that explains why the protocol is named Gemini. Many Gemini capsules are dual-hosted via Gemini and HTTP(S) so that people new to Gemini can sneak peek at the content with a regular web browser. Some people go as far as tri-hosting all their content via HTTP(S), Gemini and Gopher. + +=> gemini://gemini.circumlunar.space +=> https://gemini.circumlunar.space + +More related posts are: + +<< template::inline::index gemtext gemini + +E-Mail your comments to hi@paul.cyou :-) + +=> ../ Back to the main site diff --git a/gemfeed/2021-05-16-personal-bash-coding-style-guide.gmi b/gemfeed/2021-05-16-personal-bash-coding-style-guide.gmi index 2db178a3..15fdcdeb 100644 --- a/gemfeed/2021-05-16-personal-bash-coding-style-guide.gmi +++ b/gemfeed/2021-05-16-personal-bash-coding-style-guide.gmi @@ -380,6 +380,13 @@ I also highly recommend having a read through the "Advanced Bash-Scripting Guide => https://tldp.org/LDP/abs/html/ Advanced Bash-Scripting Guide +More related posts are: + +=> ./2022-01-01-bash-golf-part-2.gmi 2022-01-01 Bash Golf Part 2 +=> ./2021-11-29-bash-golf-part-1.gmi 2021-11-29 Bash Golf Part 1 +=> ./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi 2021-06-05 Gemtexter - One Bash script to rule it all +=> ./2021-05-16-personal-bash-coding-style-guide.gmi 2021-05-16 Personal Bash coding style guide (You are currently reading this) + E-Mail your comments to hi@paul.cyou :-) => ../ Back to the main site diff --git a/gemfeed/2021-05-16-personal-bash-coding-style-guide.gmi.tpl b/gemfeed/2021-05-16-personal-bash-coding-style-guide.gmi.tpl new file mode 100644 index 00000000..060313eb --- /dev/null +++ b/gemfeed/2021-05-16-personal-bash-coding-style-guide.gmi.tpl @@ -0,0 +1,389 @@ +# Personal Bash coding style guide + +> Published at 2021-05-16T14:51:57+01:00 + +``` + .---------------------------. + /,--..---..---..---..---..--. `. + //___||___||___||___||___||___\_| + [j__ ######################## [_| + \============================| + .==| |"""||"""||"""||"""| |"""|| +/======"---""---""---""---"=| =|| +|____ []* ____ | ==|| +// \\ // \\ |===|| hjw +"\__/"---------------"\__/"-+---+' +``` + +Lately, I have been polishing and writing a lot of Bash code. Not that I never wrote a lot of Bash, but now as I also looked through the Google Shell Style Guide, I thought it is time also to write my thoughts on that. I agree with that guide in most, but not in all points. + +=> https://google.github.io/styleguide/shellguide.html Google Shell Style Guide + +## My modifications + +These are my modifications to the Google Guide. + +### Shebang + +Google recommends using always... + +``` +#!/bin/bash +``` + +... as the shebang line, but that does not work on all Unix and Unix-like operating systems (e.g., the *BSDs don't have Bash installed to /bin/bash). Better is: + +``` +#!/usr/bin/env bash +``` + +### Two space soft-tabs indentation + +I know there have been many tab- and soft-tab wars on this planet. Google recommends using two space soft-tabs for Bash scripts. + +I don't care if I use two or four space indentations. I agree, however, that we should not use tabs. I tend to use four-space soft-tabs as that's how I currently configured Vim for any programming language. What matters most, though, is consistency within the same script/project. + +Google also recommends limiting the line length to 80 characters. For some people, that seems to be an old habit from the '80s, where all computer terminals couldn't display longer lines. But I think that the 80 character mark is still a good practice, at least for shell scripts. For example, I am often writing code on a Microsoft Go Tablet PC (running Linux, of course), and it comes in convenient if the lines are not too long due to the relatively small display on the device. + +I hit the 80 character line length quicker with the four spaces than with two spaces, but that makes me refactor the Bash code more aggressively, which is a good thing. + +### Breaking long pipes + +Google recommends breaking up long pipes like this: + +``` +# All fits on one line +command1 | command2 + +# Long commands +command1 \ + | command2 \ + | command3 \ + | command4 +``` + +I think there is a better way like the following, which is less noisy. The pipe | already indicates the Bash that another command is expected, thus making the explicit line breaks with \ obsolete: + +``` +# Long commands +command1 | + command2 | + command3 | + command4 +``` + +### Quoting your variables + +Google recommends always quote your variables. Generally, it would be best if you did that only for variables where you are unsure about the content/values of the variables (e.g., content is from an external input source and may contain whitespace or other special characters). In my opinion, the code will become quite noisy when you always quote your variables like this: + +``` +greet () { + local -r greeting="${1}" + local -r name="${2}" + echo "${greeting} ${name}!" +} +``` + +In this particular example, I agree that you should quote them as you don't know the input (are there, for example, whitespace characters?). But if you are sure that you are only using simple bare words, then I think that the code looks much cleaner when you do this instead: + +``` +say_hello_to_paul () { + local -r greeting=Hello + local -r name=Paul + echo "$greeting $name!" +} +``` + +You see, I also omitted the curly braces { } around the variables. I only use the curly braces around variables when it makes the code either easier/clearer to read or if it is necessary to use them: + +``` +declare FOO=bar +# Curly braces around FOO are necessary +echo "foo${FOO}baz" +``` + +A few more words on always quoting the variables: For the sake of consistency (and for making ShellCheck happy), I am not against quoting everything I encounter. I also think that the larger the Bash script becomes, the more critical it becomes always to quote variables. That's because it will be more likely that you might not remember that some of the functions don't work on values with spaces in them, for example. It's just that I won't quote everything in every small script I write. + +### Prefer built-in commands over external commands + +Google recommends using the built-in commands over available external commands where possible: + +``` +# Prefer this: +addition=$(( X + Y )) +substitution="${string/#foo/bar}" + +# Instead of this: +addition="$(expr "${X}" + "${Y}")" +substitution="$(echo "${string}" | sed -e 's/^foo/bar/')" +``` + +I can't entirely agree here. The external commands (especially sed) are much more sophisticated and powerful than the built-in Bash versions. Sed can do much more than the Bash can ever do by itself when it comes to text manipulation (the name "sed" stands for streaming editor, after all). + +I prefer to do light text processing with the Bash built-ins and more complicated text processing with external programs such as sed, grep, awk, cut, and tr. However, there is also medium-light text processing where I would want to use external programs. That is so because I remember using them better than the Bash built-ins. The Bash can get relatively obscure here (even Perl will be more readable then - Side note: I love Perl). + +Also, you would like to use an external command for floating-point calculation (e.g., bc) instead of using the Bash built-ins (worth noticing that ZSH supports built-in floating-points). + +I even didn't get started with what you can do with awk (especially GNU Awk), a fully-fledged programming language. Tiny Awk snippets tend to be used quite often in Shell scripts without honouring the real power of Awk. But if you did everything in Perl or Awk or another scripting language, then it wouldn't be a Bash script anymore, wouldn't it? ;-) + +## My additions + +### Use of 'yes' and 'no' + +Bash does not support a boolean type. I tend just to use the strings 'yes' and 'no' here. I used 0 for false and 1 for true for some time, but I think that the yes/no strings are easier to read. Yes, the Bash script would need to perform string comparisons on every check, but if performance is crucial to you, you wouldn't want to use a Bash script anyway, correct? + +``` +declare -r SUGAR_FREE=yes +declare -r I_NEED_THE_BUZZ=no + +buy_soda () { + local -r sugar_free=$1 + + if [[ $sugar_free == yes ]]; then + echo 'Diet Dr. Pepper' + else + echo 'Pepsi Coke' + fi +} + +buy_soda $I_NEED_THE_BUZZ +``` + +### Non-evil alternative to variable assignments via eval + +Google is in the opinion that eval should be avoided. I think so too. They list these examples in their guide: + +``` +# What does this set? +# Did it succeed? In part or whole? +eval $(set_my_variables) + +# What happens if one of the returned values has a space in it? +variable="$(eval some_function)" + +``` + +However, if I want to read variables from another file, I don't have to use eval here. I only have to source the file: + +``` +% cat vars.source.sh +declare foo=bar +declare bar=baz +declare bay=foo + +% bash -c 'source vars.source.sh; echo $foo $bar $baz' +bar baz foo +``` + +And suppose I want to assign variables dynamically. In that case, I could just run an external script and source its output (This is how you could do metaprogramming in Bash without the use of eval - write code which produces code for immediate execution): + +``` +% cat vars.sh +#!/usr/bin/env bash +cat <&2 + grep ... | sed .... +} + +process_lines () { + echo 'Start processing line by line!' >&2 + while read -r line; do + ... do something and produce a result... + echo "$result" + done +} + +# Do some post-processing of the data +postprocess_lines () { + echo 'Start removing duplicates!' >&2 + sort -u +} + +genreate_report () { + echo 'My boss wants to have a report!' >&2 + tee outfile.txt + wc -l outfile.txt +} + +main () { + filter_lines | + process_lines | + postprocess_lines | + generate_report +} + +main +``` + +The stdout is always passed as a pipe to the next following stage. The stderr is used for info logging. + +### Assign-then-shift + +I often refactor existing Bash code. That leads me to add and removing function arguments quite often. It's pretty repetitive work changing the $1, $2.... function argument numbers every time you change the order or add/remove possible arguments. + +The solution is to use of the "assign-then-shift"-method, which goes like this: "local -r var1=$1; shift; local -r var2=$1; shift". The idea is that you only use "$1" to assign function arguments to named (better readable) local function variables. You will never have to bother about "$2" or above. That is very useful when you constantly refactor your code and remove or add function arguments. It's something that I picked up from a colleague (a pure Bash wizard) some time ago: + +``` +some_function () { + local -r param_foo="$1"; shift + local -r param_baz="$1"; shift + local -r param_bay="$1"; shift + ... +} +``` + +Want to add a param_baz? Just do this: + +``` +some_function () { + local -r param_foo="$1"; shift + local -r param_bar="$1"; shift + local -r param_baz="$1"; shift + local -r param_bay="$1"; shift + ... +} +``` + +Want to remove param_foo? Nothing easier than that: + +``` +some_function () { + local -r param_bar="$1"; shift + local -r param_baz="$1"; shift + local -r param_bay="$1"; shift + ... +} +``` + +As you can see, I didn't need to change any other assignments within the function. Of course, you would also need to change the function argument lists at every occasion where the function is invoked - you would do that within the same refactoring session. + +### Paranoid mode + +I call this the paranoid mode. The Bash will stop executing when a command exits with a status not equal to 0: + +``` +set -e +grep -q foo <<< bar +echo Jo +``` + +Here 'Jo' will never be printed out as the grep didn't find any match. It's unrealistic for most scripts to run in paranoid mode purely, so there must be a way to add exceptions. Critical Bash scripts of mine tend to look like this: + +``` +#!/usr/bin/env bash + +set -e + +some_function () { + .. some critical code + ... + + set +e + # Grep might fail, but that's OK now + grep .... + local -i ec=$? + set -e + + .. critical code continues ... + if [[ $ec -ne 0 ]]; then + ... + fi + ... +} +``` + +## Learned + +There are also a couple of things I've learned from Google's guide. + +### Unintended lexicographical comparison. + +The following looks like a valid Bash code: + +``` +if [[ "${my_var}" > 3 ]]; then + # True for 4, false for 22. + do_something +fi +``` + +... but it is probably an unintended lexicographical comparison. A correct way would be: + +``` +if (( my_var > 3 )); then + do_something +fi +``` + +or + +``` +if [[ "${my_var}" -gt 3 ]]; then + do_something +fi +``` + +### PIPESTATUS + +I have never used the PIPESTATUS variable before. I knew that it's there, but I never bothered to understand how it works until now thoroughly. + +The PIPESTATUS variable in Bash allows checking of the return code from all parts of a pipe. If it's only necessary to check the success or failure of the whole pipe, then the following is acceptable: + +``` +tar -cf - ./* | ( cd "${dir}" && tar -xf - ) +if (( PIPESTATUS[0] != 0 || PIPESTATUS[1] != 0 )); then + echo "Unable to tar files to ${dir}" >&2 +fi +``` + +However, as PIPESTATUS will be overwritten as soon as you do any other command, if you need to act differently on errors based on where it happened in the pipe, you'll need to assign PIPESTATUS to another variable immediately after running the command (don't forget that [ is a command and will wipe out PIPESTATUS). + +``` +tar -cf - ./* | ( cd "${DIR}" && tar -xf - ) +return_codes=( "${PIPESTATUS[@]}" ) +if (( return_codes[0] != 0 )); then + do_something +fi +if (( return_codes[1] != 0 )); then + do_something_else +fi +``` + +## Use common sense and BE CONSISTENT. + +The following two paragraphs are thoroughly quoted from the Google guidelines. But they hit the hammer on the head: + +> If you are editing code, take a few minutes to look at the code around you and determine its style. If they use spaces around their if clauses, you should, too. If their comments have little boxes of stars around them, make your comments have little boxes of stars around them too. + +> The point of having style guidelines is to have a common vocabulary of coding so people can concentrate on what you are saying rather than on how you are saying it. We present global style rules here, so people know the vocabulary. But local style is also important. If the code you add to a file looks drastically different from the existing code around it, the discontinuity throws readers out of their rhythm when they go to read it. Try to avoid this. + + +## Advanced Bash learning pro tip + +I also highly recommend having a read through the "Advanced Bash-Scripting Guide" (not from Google). I use it as the universal Bash reference and learn something new every time I look at it. + +=> https://tldp.org/LDP/abs/html/ Advanced Bash-Scripting Guide + +More related posts are: + +<< template::inline::index bash + +E-Mail your comments to hi@paul.cyou :-) + +=> ../ Back to the main site diff --git a/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi b/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi index f98f651d..05027f79 100644 --- a/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi +++ b/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi @@ -43,9 +43,7 @@ `+a:f:......jrei''' ``` -You might have read my previous blog post about entering the Geminispace, where I pointed out the benefits of having and maintaining an internet presence there. This whole site (the blog and all other pages) is composed in the Gemtext markup language. - -=> ./2021-04-24-welcome-to-the-geminispace.gmi Welcome to the Geminispace +You might have read my previous blog posts about entering the Geminispace, where I pointed out the benefits of having and maintaining an internet presence there. This whole site (the blog and all other pages) is composed in the Gemtext markup language. This comes with the benefit that I can write content in my favourite text editor (Vim). @@ -166,6 +164,15 @@ It was quite a lot of fun writing Gemtexter. It's a relatively small project, bu I finally revamped my personal internet site and started to blog again. I wanted the result to be exactly how it is now: A slightly retro-inspired internet site built for fun with unconventional tools. +More related posts are: + +=> ./2022-08-27-gemtexter-1.1.0-lets-gemtext-again.gmi 2022-08-27 Gemtexter 1.1.0 - Let's Gemtext again +=> ./2022-01-01-bash-golf-part-2.gmi 2022-01-01 Bash Golf Part 2 +=> ./2021-11-29-bash-golf-part-1.gmi 2021-11-29 Bash Golf Part 1 +=> ./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi 2021-06-05 Gemtexter - One Bash script to rule it all (You are currently reading this) +=> ./2021-05-16-personal-bash-coding-style-guide.gmi 2021-05-16 Personal Bash coding style guide +=> ./2021-04-24-welcome-to-the-geminispace.gmi 2021-04-24 Welcome to the Geminispace + E-Mail your comments to hi@paul.cyou :-) => ../ Back to the main site diff --git a/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi.tpl b/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi.tpl new file mode 100644 index 00000000..19426d54 --- /dev/null +++ b/gemfeed/2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi.tpl @@ -0,0 +1,173 @@ +# Gemtexter - One Bash script to rule it all + +> Published at 2021-06-05T19:03:32+01:00 + +``` + o .,<>., o + |\/\/\/\/| + '========' + (_ SSSSSSs + )a'`SSSSSs + /_ SSSSSS + .=## SSSSS + .#### SSSSs + ###::::SSSSS + .;:::""""SSS + .:;:' . . \\ + .::/ ' .'| + .::( . | + :::) \ + /\( / + /) ( | + .' \ . ./ / + _-' |\ . | + _..--.. . /"---\ | ` | . | + -=====================,' _ \=(*#(7.#####() | `/_.. , ( + _.-''``';'-''-) ,. \ ' '+/// | .'/ \ ``-.) \ + ,' _.- (( `-' `._\ `` \_/_.' ) /`-._ ) | + ,'\ ,' _.'.`:-. \.-' / <_L )" | + _/ `._,' ,')`; `-'`' | L / / + / `. ,' ,|_/ / \ ( <_-' \ + \ / `./ ' / /,' \ /|` `. | + )\ /`._ ,'`._.-\ |) \' + / `.' )-'.-,' )__) |\ `| + : /`. `.._(--.`':`':/ \ ) \ \ + |::::\ ,'/::;-)) / ( )`. | + ||::::: . .::': :`-( |/ . | + ||::::| . :| |==[]=: . - \ + |||:::| : || : | | /\ ` | + ___ ___ '|;:::| | |' \=[]=| / \ \ +| /_ ||``|||::::: | ; | | | \_.'\_ `-. +: \_``[]--[]|::::'\_;' )-'..`._ .-'\``:: ` . \ + \___.>`''-.||:.__,' SSt |_______`> <_____:::. . . \ _/ + `+a:f:......jrei''' +``` + +You might have read my previous blog posts about entering the Geminispace, where I pointed out the benefits of having and maintaining an internet presence there. This whole site (the blog and all other pages) is composed in the Gemtext markup language. + +This comes with the benefit that I can write content in my favourite text editor (Vim). + +## Motivation + +Another benefit of using Gemini is that the Gemtext markup language is easy to parse. As my site is dual-hosted (Gemini+HTTP), I could, in theory, just write a shell script to deal with the conversion from Gemtext to HTML; there is no need for a full-featured programming language here. I have done a lot of Bash in the past, but I am also often revisiting old tools and techniques for refreshing and keeping the knowledge up to date here. + +=> ./2021-06-05-gemtexter-one-bash-script-to-rule-it-all/blog-engine.jpg Motivational comic strip + +I have exactly done that - I wrote a Bash script, named Gemtexter, for that: + +=> https://codeberg.org/snonux/gemtexter + +In short, Gemtexter is a static site generator and blogging engine that uses Gemtext as its input format. + +## Output formats + +Gemtexter takes the Gemntext Markup files as the input and generates the following outputs from it (you find examples for each of these output formats on the Gemtexter GitHub page): + +* HTML files for my website +* Markdown files for a GitHub page +* A Gemtext Atom feed for my blog posts +* A Gemfeed for my blog posts (a particular feed format commonly used in Geminispace. The Gemfeed can be used as an alternative to the Atom feed). +* An HTML Atom feed of my blog posts + +I could have done all of that with a more robust language than Bash (such as Perl, Ruby, Go...), but I didn't. The purpose of this exercise was to challenge what I can do with a "simple" Bash script and learn new things. + +## Taking it as far as I should, but no farther + +The Bash is suitable very well for small scripts and ad-hoc automation on the command line. But it is for sure not a robust programming language. Writing this blog post, Gemtexter is nearing 1000 lines of code, which is actually a pretty large Bash script. + +### Modularization + +I modularized the code so that each core functionality has its own file in ./lib. All the modules are included from the main Gemtexter script. For example, there is one module for HTML generation, one for Markdown generation, and so on. + +``` +paul in uranus in gemtexter on 🌱 main +❯ wc -l gemtexter lib/* + 117 gemtexter + 59 lib/assert.source.sh + 128 lib/atomfeed.source.sh + 64 lib/gemfeed.source.sh + 161 lib/generate.source.sh + 50 lib/git.source.sh + 162 lib/html.source.sh + 30 lib/log.source.sh + 63 lib/md.source.sh + 834 total +``` + +This way, the script could grow far beyond 1000 lines of code and still be maintainable. With more features, execution speed may slowly become a problem, though. I already notice that Gemtexter doesn't produce results instantly but requires few seconds of runtime already. That's not a problem yet, though. + +### Bash best practises and ShellCheck + +While working on Gemtexter, I also had a look at the Google Shell Style Guide and wrote a blog post on that: + +=> ./2021-05-16-personal-bash-coding-style-guide.gmi Personal bash coding style guide + +I followed all these best practices, and in my opinion, the result is a pretty maintainable Bash script (given that you are fluent with all the sed and grep commands I used). + +ShellCheck, a shell script analysis tool written in Haskell, is run on Gemtexter ensuring that all code is acceptable. I am pretty impressed with what ShellCheck found. + +It, for example, detected "some_command | while read var; do ...; done" loops and hinted that these create a new subprocess for the while part. The result is that all variable modifications taking place in the while-subprocess won't reflect the primary Bash process. ShellSheck then recommended rewriting the loop so that no subprocess is spawned as "while read -r var; do ...; done < <(some_command)". ShellCheck also pointed out to add a "-r" to "read"; otherwise, there could be an issue with backspaces in the loop data. + +Furthermore, ShellCheck recommended many more improvements. Declaration of unused variables and missing variable and string quotations were the most common ones. ShellSheck immensely helped to improve the robustness of the script. + +=> https://shellcheck.net + +### Unit testing + +There is a basic unit test module in ./lib/assert.source.sh, which is used for unit testing. I found this to be very beneficial for cross-platform development. For example, I noticed that some unit tests failed on macOS while everything still worked fine on my Fedora Linux laptop. + +After digging a bit, I noticed that I had to install the GNU versions of the sed and grep commands on macOS and a newer version of the Bash to make all unit tests pass and Gemtexter work. + +It has been proven quite helpful to have unit tests in place for the HTML part already when working on the Markdown generator part. To test the Markdown part, I copied the HTML unit tests and changed the expected outcome in the assertions. This way, I could implement the Markdown generator in a test-driven way (writing the test first and afterwards the implementation). + +### HTML unit test example + +``` +gemtext='=> http://example.org Description of the link' +assert::equals "$(generate::make_link html "$gemtext")" \ + 'Description of the link
' + +``` + +### Markdown unit test example + +``` +gemtext='=> http://example.org Description of the link' +assert::equals "$(generate::make_link md "$gemtext")" \ + '[Description of the link](http://example.org) ' +``` + +## Handcrafted HTML styles + +I had a look at some ready off the shelf CSS styles, but they all seemed too bloated. There is a whole industry selling CSS styles on the interweb. I preferred an effortless and minimalist style for the HTML site. So I handcrafted the Cascading Style Sheets manually with love and included them in the HTML header template. + +For now, I have to re-generate all HTML files whenever the CSS changes. That should not be an issue now, but I might move the CSS into a separate file one day. + +It's worth mentioning that all generated HTML files and Atom feeds pass the W3C validation tests. + +## Configurability + +In case someone else than me wants to use Gemtexter for his own site, it is pretty much configurable. It is possible to specify your own configuration file and your own HTML templates. Have a look at the GitHub page for examples. + +## Future features + +I could think of the following features added to a future version of Gemtexter: + +* Templating of Gemtext files so that the .gmi files are generated from .gmi.tpl files. The template engine could do such things as an automatic table of contents and sitemap generation. It could also include the output of inlined shell code, e.g. a fortune quote. +* Add support for more output formats, such as Groff, PDF, plain text, Gopher, etc. +* External CSS file for HTML. +* Improve speed by introducing parallelism and/or concurrency and/or better caching. + +## Conclusion + +It was quite a lot of fun writing Gemtexter. It's a relatively small project, but given that I worked on that in my spare time once in a while, it kept me busy for several weeks. + +I finally revamped my personal internet site and started to blog again. I wanted the result to be exactly how it is now: A slightly retro-inspired internet site built for fun with unconventional tools. + +More related posts are: + +<< template::inline::index gemtext gemini bash + +E-Mail your comments to hi@paul.cyou :-) + +=> ../ Back to the main site diff --git a/gemfeed/2021-11-29-bash-golf-part-1.gmi b/gemfeed/2021-11-29-bash-golf-part-1.gmi index 3b1155c6..120999ba 100644 --- a/gemfeed/2021-11-29-bash-golf-part-1.gmi +++ b/gemfeed/2021-11-29-bash-golf-part-1.gmi @@ -16,8 +16,8 @@ jgs^^^^^^^`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This is the first blog post about my Bash Golf series. This series is about random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content. -=> ./2021-11-29-bash-golf-part-1.gmi Bash Golf Part 1 (you are reading this atm.) -=> ./2022-01-01-bash-golf-part-2.gmi Bash Golf Part 2 +=> ./2022-01-01-bash-golf-part-2.gmi 2022-01-01 Bash Golf Part 2 +=> ./2021-11-29-bash-golf-part-1.gmi 2021-11-29 Bash Golf Part 1 (You are currently reading this) ## TCP/IP networking @@ -463,6 +463,13 @@ In the Bash you will have to fall back to an external command like "bc" (the arb See you later for the next post of this series. +More related posts are: + +=> ./2022-01-01-bash-golf-part-2.gmi 2022-01-01 Bash Golf Part 2 +=> ./2021-11-29-bash-golf-part-1.gmi 2021-11-29 Bash Golf Part 1 (You are currently reading this) +=> ./2021-06-05-gemtexter-one-bash-script-to-rule-it-all.gmi 2021-06-05 Gemtexter - One Bash script to rule it all +=> ./2021-05-16-personal-bash-coding-style-guide.gmi 2021-05-16 Personal Bash coding style guide + E-Mail your comments to hi@paul.cyou :-) => ../ Back to the main site diff --git a/gemfeed/2021-11-29-bash-golf-part-1.gmi.tpl b/gemfeed/2021-11-29-bash-golf-part-1.gmi.tpl new file mode 100644 index 00000000..55ab158d --- /dev/null +++ b/gemfeed/2021-11-29-bash-golf-part-1.gmi.tpl @@ -0,0 +1,471 @@ +# Bash Golf Part 1 + +> Published at 2021-11-29T14:06:14+00:00; Updated at 2022-01-05 + +``` + + '\ . . |>18>> + \ . ' . | + O>> . 'o | + \ . | + /\ . | + / / .' | +jgs^^^^^^^`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Art by Joan Stark +``` + +This is the first blog post about my Bash Golf series. This series is about random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content. + +<< template::inline::index bash-golf + +## TCP/IP networking + +You probably know the Netcat tool, which is a swiss army knife for TCP/IP networking on the command line. But did you know that the Bash natively supports TCP/IP networking? + +Have a look here how that works: + +``` +❯ cat < /dev/tcp/time.nist.gov/13 + +59536 21-11-18 08:09:16 00 0 0 153.6 UTC(NIST) * +``` + +The Bash treats /dev/tcp/HOST/PORT in a special way so that it is actually establishing a TCP connection to HOST:PORT. The example above redirects the TCP output of the time-server to cat and cat is printing it on standard output (stdout). + +A more sophisticated example is firing up an HTTP request. Let's create a new read-write (rw) file descriptor (fd) 5, redirect the HTTP request string to it, and then read the response back: + +``` +❯ exec 5<>/dev/tcp/google.de/80 +❯ echo -e "GET / HTTP/1.1\nhost: google.de\n\n" >&5 +❯ cat <&5 | head +HTTP/1.1 301 Moved Permanently +Location: http://www.google.de/ +Content-Type: text/html; charset=UTF-8 +Date: Thu, 18 Nov 2021 08:27:18 GMT +Expires: Sat, 18 Dec 2021 08:27:18 GMT +Cache-Control: public, max-age=2592000 +Server: gws +Content-Length: 218 +X-XSS-Protection: 0 +X-Frame-Options: SAMEORIGIN +``` + +You would assume that this also works with the ZSH, but it doesn't. This is one of the few things which don't work with the ZSH but in the Bash. There might be plugins you could use for ZSH to do something similar, though. + +## Process substitution + +The idea here is, that you can read the output (stdout) of a command from a file descriptor: + +``` +❯ uptime # Without process substitution + 10:58:03 up 4 days, 22:08, 1 user, load average: 0.16, 0.34, 0.41 + +❯ cat <(uptime) # With process substitution + 10:58:16 up 4 days, 22:08, 1 user, load average: 0.14, 0.33, 0.41 + +❯ stat <(uptime) + File: /dev/fd/63 -> pipe:[468130] + Size: 64 Blocks: 0 IO Block: 1024 symbolic link +Device: 16h/22d Inode: 468137 Links: 1 +Access: (0500/lr-x------) Uid: ( 1001/ paul) Gid: ( 1001/ paul) +Context: unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 +Access: 2021-11-20 10:59:31.482411961 +0000 +Modify: 2021-11-20 10:59:31.482411961 +0000 +Change: 2021-11-20 10:59:31.482411961 +0000 + Birth: - +``` + +This example doesn't make any sense practically speaking, but it clearly demonstrates how process substitution works. The standard output pipe of "uptime" is redirected to an anonymous file descriptor. That fd then is opened by the "cat" command as a regular file. + +A useful use case is displaying the differences of two sorted files: + +``` +❯ echo a > /tmp/file-a.txt +❯ echo b >> /tmp/file-a.txt +❯ echo c >> /tmp/file-a.txt +❯ echo b > /tmp/file-b.txt +❯ echo a >> /tmp/file-b.txt +❯ echo c >> /tmp/file-b.txt +❯ echo X >> /tmp/file-b.txt +❯ diff -u <(sort /tmp/file-a.txt) <(sort /tmp/file-b.txt) +--- /dev/fd/63 2021-11-20 11:05:03.667713554 +0000 ++++ /dev/fd/62 2021-11-20 11:05:03.667713554 +0000 +@@ -1,3 +1,4 @@ + a + b + c ++X +❯ echo X >> /tmp/file-a.txt # Now, both files have the same content again. +❯ diff -u <(sort /tmp/file-a.txt) <(sort /tmp/file-b.txt) +❯ +``` + +Another example is displaying the differences of two directories: + +``` +❯ diff -u <(ls ./dir1/ | sort) <(ls ./dir2/ | sort) +``` + +More (Bash golfing) examples: + +``` +❯ wc -l <(ls /tmp/) /etc/passwd <(env) + 24 /dev/fd/63 + 49 /etc/passwd + 24 /dev/fd/62 + 97 total +❯ + +❯ while read foo; do +> echo $foo +> done < <(echo foo bar baz) +foo bar baz +❯ +``` + +So far, we only used process substitution for stdout redirection. But it also works for stdin. The following two commands result into the same outcome, but the second one is writing the tar data stream to an anonymous file descriptor which is substituted by the "bzip2" command reading the data stream from stdin and compressing it to its own stdout, which then gets redirected to a file: + +``` +❯ tar cjf file.tar.bz2 foo +❯ tar cjf >(bzip2 -c > file.tar.bz2) foo +``` + +Just think a while and see whether you understand fully what is happening here. + +## Grouping + +Command grouping can be quite useful for combining the output of multiple commands: + +``` +❯ { ls /tmp; cat /etc/passwd; env; } | wc -l +97 +❯ ( ls /tmp; cat /etc/passwd; env; ) | wc -l +97 +``` + +But wait, what is the difference between curly braces and normal braces? I assumed that the normal braces create a subprocess whereas the curly ones don't, but I was wrong: + +``` +❯ echo $$ +62676 +❯ { echo $$; } +62676 +❯ ( echo $$; ) +62676 +``` + +One difference is, that the curly braces require you to end the last statement with a semicolon, whereas with the normal braces you can omit the last semicolon: + +``` +❯ ( env; ls ) | wc -l +27 +❯ { env; ls } | wc -l +> +> ^C +``` + +In case you know more (subtle) differences, please write me an E-Mail and let me know. + +> Update: A reader sent me an E-Mail and pointed me to the Bash manual page, which explains the difference between () and {} (I should have checked that by myself): + +``` +(list) list is executed in a subshell environment (see COMMAND EXECUTION ENVIRONMENT + below). Variable assignments and builtin commands that affect the shell's + environment do not remain in effect after the command completes. The return + status is the exit status of list. + +{ list; } + list is simply executed in the current shell environment. list must be ter‐ + minated with a newline or semicolon. This is known as a group command. The + return status is the exit status of list. Note that unlike the metacharac‐ + ters ( and ), { and } are reserved words and must occur where a reserved word + is permitted to be recognized. Since they do not cause a word break, they + must be separated from list by whitespace or another shell metacharacter. +``` + +So I was right that () is executed in a subprocess. But why does $$ not show a different PID? Also here (as pointed out by the reader) is the answer in the manual page: + +``` +$ Expands to the process ID of the shell. In a () subshell, it expands to the + process ID of the current shell, not the subshell. +``` + +If we want print the subprocess PID, we can use the BASHPID variable: + +``` +❯ echo $BASHPID; { echo $BASHPID; }; ( echo $BASHPID; ) +1028465 +1028465 +1028739 +``` + +## Expansions + +Let's start with simple examples: + +``` +❯ echo {0..5} +0 1 2 3 4 5 +❯ for i in {0..5}; do echo $i; done +0 +1 +2 +3 +4 +5 +``` + +You can also add leading 0 or expand to any number range: + +``` +❯ echo {00..05} +00 01 02 03 04 05 +❯ echo {000..005} +000 001 002 003 004 005 +❯ echo {201..205} +201 202 203 204 205 +``` + +It also works with letters: + +``` +❯ echo {a..e} +a b c d e +``` + +Now it gets interesting. The following takes a list of words and expands it so that all words are quoted: + +``` +❯ echo \"{These,words,are,quoted}\" +"These" "words" "are" "quoted" +``` + +Let's also expand to the cross product of two given lists: + +``` +❯ echo {one,two}\:{A,B,C} +one:A one:B one:C two:A two:B two:C +❯ echo \"{one,two}\:{A,B,C}\" +"one:A" "one:B" "one:C" "two:A" "two:B" "two:C" +``` + +Just because we can: + +``` +❯ echo Linux-{one,two,three}\:{A,B,C}-FreeBSD +Linux-one:A-FreeBSD Linux-one:B-FreeBSD Linux-one:C-FreeBSD Linux-two:A-FreeBSD Linux-two:B-FreeBSD Linux-two:C-FreeBSD Linux-three:A-FreeBSD Linux-three:B-FreeBSD Linux-three:C-FreeBSD +``` + +## - aka stdin and stdout placeholder + +Some commands and Bash builtins use "-" as a placeholder for stdin and stdout: + +``` +❯ echo Hello world +Hello world +❯ echo Hello world | cat - +Hello world +❯ cat - <