From 97decc24069b655e0a1d32e3e6217ab0d8653351 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Fri, 27 Sep 2024 23:28:33 +0300 Subject: Update content for gemtext --- about/index.gmi | 2 +- about/resources.gmi | 152 ++-- ...-quickly-logging-ideas-programmed-in-golang.gmi | 1 - gemfeed/atom.xml | 810 +++++++++++++-------- gemfeed/index.gmi | 1 - index.gmi | 3 +- notes/search-inside-yourself.gmi | 20 +- uptime-stats.gmi | 2 +- 8 files changed, 620 insertions(+), 371 deletions(-) diff --git a/about/index.gmi b/about/index.gmi index de2b76e4..788b135b 100644 --- a/about/index.gmi +++ b/about/index.gmi @@ -2,7 +2,7 @@ * Name: Paul Buetow * Profession: Computerist - Solving problems with computers that we wouldn't have without them -* Current job title: Site Reliability Engineer +* Current job title: Principal Site Reliability Engineer * E-Mail: `paul@nospam.buetow.org` => ./paul.jpg Paul Buetow diff --git a/about/resources.gmi b/about/resources.gmi index e89cd7bb..cf3ee99b 100644 --- a/about/resources.gmi +++ b/about/resources.gmi @@ -33,93 +33,93 @@ You won't find any links on this site because, over time, the links will break. In random order: -* Modern Perl; Chromatic ; Onyx Neon Press -* Effective awk programming; Arnold Robbins; O'Reilly -* DNS and BIND; Cricket Liu; O'Reilly +* 97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly * Raku Recipes; J.J. Merelo; Apress -* Perl New Features; Joshua McAdams, brian d foy; Perl School -* Raku Fundamentals; Moritz Lenz; Apress -* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications -* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly -* Effective Java; Joshua Bloch; Addison-Wesley Professional -* Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner -* Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly +* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly +* Concurrency in Go; Katherine Cox-Buday; O'Reilly +* Leanring eBPF; Liz Rice; O'Reilly * The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional -* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible -* Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press -* Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers -* Funktionale Programmierung; Peter Pepper; Springer -* C++ Programming Language; Bjarne Stroustrup; -* Systemprogrammierung in Go; Frank Müller; dpunkt -* The Docker Book; James Turnbull; Kindle -* Site Reliability Engineering; How Google runs production systems; O'Reilly -* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible +* Polished Ruby Programming; Jeremy Evans; Packt Publishing +* Modern Perl; Chromatic ; Onyx Neon Press * Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press -* Java ist auch eine Insel; Christian Ullenboom; -* Concurrency in Go; Katherine Cox-Buday; O'Reilly +* Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press +* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible +* Ultimate Go Notebook; Bill Kennedy * Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications +* Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner +* Data Science at the Command Line; Jeroen Janssens; O'Reilly +* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications * Pro Puppet; James Turnbull, Jeffrey McCune; Apress +* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible +* Site Reliability Engineering; How Google runs production systems; O'Reilly +* The Docker Book; James Turnbull; Kindle +* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson +* Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly +* Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly +* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly * The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress * The Pragmatic Programmer; David Thomas; Addison-Wesley -* Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly +* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly +* Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers +* DNS and BIND; Cricket Liu; O'Reilly +* Java ist auch eine Insel; Christian Ullenboom; * Higher Order Perl; Mark Dominus; Morgan Kaufmann -* Leanring eBPF; Liz Rice; O'Reilly -* Ultimate Go Notebook; Bill Kennedy -* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson -* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly -* Polished Ruby Programming; Jeremy Evans; Packt Publishing -* Data Science at the Command Line; Jeroen Janssens; O'Reilly -* 97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly -* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers +* Raku Fundamentals; Moritz Lenz; Apress +* Effective Java; Joshua Bloch; Addison-Wesley Professional * Developing Games in Java; David Brackeen and others...; New Riders -* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly +* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers +* Perl New Features; Joshua McAdams, brian d foy; Perl School +* Effective awk programming; Arnold Robbins; O'Reilly +* Systemprogrammierung in Go; Frank Müller; dpunkt +* C++ Programming Language; Bjarne Stroustrup; +* Funktionale Programmierung; Peter Pepper; Springer ## Technical references I didn't read them from the beginning to the end, but I am using them to look up things. The books are in random order: -* Implementing Service Level Objectives; Alex Hidalgo; O'Reilly -* Relayd and Httpd Mastery; Michael W Lucas -* Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley -* The Linux Programming Interface; Michael Kerrisk; No Starch Press -* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley * Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly +* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley +* Relayd and Httpd Mastery; Michael W Lucas +* Implementing Service Level Objectives; Alex Hidalgo; O'Reilly * Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly +* The Linux Programming Interface; Michael Kerrisk; No Starch Press +* Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley ## Self-development and soft-skills books In random order: -* Deep Work; Cal Newport; Piatkus +* The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook +* Slow Productivity; Cal Newport; Penguin Random House * Stop starting, start finishing; Arne Roock; Lean-Kanban University -* The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select -* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly +* The Off Switch; Mark Cropley; Virgin Books +* 101 Essays that change the way you think; Brianna Wiest; Audible +* Psycho-Cybernetics; Maxwell Maltz; Perigee Books * The Bullet Journal Method; Ryder Carroll; Fourth Estate -* So Good They Can't Ignore You; Cal Newport; Business Plus -* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business -* Staff Engineer: Leadership beyond the management track; Will Larson; Audible -* Slow Productivity; Cal Newport; Penguin Random House +* The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd +* Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press * Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne +* The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books +* Influence without Authority; A. Cohen, D. Bradford; Wiley +* So Good They Can't Ignore You; Cal Newport; Business Plus * Eat That Frog!; Brian Tracy; Hodder Paperbacks -* The Off Switch; Mark Cropley; Virgin Books -* The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd * Atomic Habits; James Clear; Random House Business -* Digital Minimalism; Cal Newport; Portofolio Penguin -* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion -* The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook +* Soft Skills; John Sommez; Manning Publications * The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK * The Power of Now; Eckhard Tolle; Yellow Kite +* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business +* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion +* The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select +* Ultralearning; Anna Laurent; Self-published via Amazon * The Good Enough Job; Simone Stolzoff; Ebury Edge -* The Joy of Missing Out; Christina Crook; New Society Publishers -* Psycho-Cybernetics; Maxwell Maltz; Perigee Books +* Digital Minimalism; Cal Newport; Portofolio Penguin +* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly * Ultralearning; Scott Young; Thorsons -* Soft Skills; John Sommez; Manning Publications -* 101 Essays that change the way you think; Brianna Wiest; Audible * Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing -* Influence without Authority; A. Cohen, D. Bradford; Wiley -* The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books -* Ultralearning; Anna Laurent; Self-published via Amazon -* Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press +* The Joy of Missing Out; Christina Crook; New Society Publishers +* Staff Engineer: Leadership beyond the management track; Will Larson; Audible +* Deep Work; Cal Newport; Piatkus => https://foo.zone/notes/index.html Here are notes of mine for some of the books (HTTP) => gemini://foo.zone/notes/index.gmi Here are notes of mine for some of the books (Gemini) @@ -128,22 +128,22 @@ In random order: Some of these were in-person with exams; others were online learning lectures only. In random order: -* The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online -* Developing IaC with Terraform (with Live Lessons); O'Reilly Online -* AWS Immersion Day; Amazon; 1-day interactive online training -* Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training -* Protocol buffers; O'Reilly Online * Apache Tomcat Best Practises; 3-day on-site training +* F5 Loadbalancers Training; 2-day on-site training; F5, Inc. * The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online +* MySQL Deep Dive Workshop; 2-day on-site training +* AWS Immersion Day; Amazon; 1-day interactive online training +* Developing IaC with Terraform (with Live Lessons); O'Reilly Online +* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need) +* Functional programming lecture; Remote University of Hagen * Scripting Vim; Damian Conway; O'Reilly Online +* Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon * Structure and Interpretation of Computer Programs; Harold Abelson and more...; -* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need) -* MySQL Deep Dive Workshop; 2-day on-site training +* Protocol buffers; O'Reilly Online +* The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online * Ultimate Go Programming; Bill Kennedy; O'Reilly Online -* F5 Loadbalancers Training; 2-day on-site training; F5, Inc. -* Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon * Algorithms Video Lectures; Robert Sedgewick; O'Reilly Online -* Functional programming lecture; Remote University of Hagen +* Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training ## Technical guides @@ -156,30 +156,30 @@ These are not whole books, but guides (smaller or larger) which I found very use In random order: +* Ship it (Changelog) +* Maintainable * Backend Banter -* Modern Mentor * Cup o' Go [Golang] -* Ship it (Changelog) -* Hidden Brain * Deep Questions with Cal Newport +* Hidden Brain +* Go Time (Changelog) +* Modern Mentor * Java Pub House * Dev Interrupted -* Maintainable -* Go Time (Changelog) ## Newsletters I like This is a mix of tech and non-tech newsletters I am subscribed to. In random order: -* byteSizeGo -* VK Newsletter +* Golang Weekly * Register Spill -* Ruby Weekly -* The Imperfectionist +* VK Newsletter * Andreas Brandhorst Newsletter (Sci-Fi author) +* Ruby Weekly * The Valuable Dev -* Golang Weekly * Applied Go Weekly Newsletter +* The Imperfectionist +* byteSizeGo # Formal education diff --git a/gemfeed/2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.gmi b/gemfeed/2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.gmi index 1d62ffa7..e767ecb3 100644 --- a/gemfeed/2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.gmi +++ b/gemfeed/2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.gmi @@ -54,7 +54,6 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other Go related posts are: -=> ./2023-04-09-algorithms-and-data-structures-in-golang-part-1.gmi 2023-04-09 Algorithms and Data Structures in Go - Part 1 => ./2024-03-03-a-fine-fyne-android-app-for-quickly-logging-ideas-programmed-in-golang.gmi 2024-03-03 A fine Fyne Android app for quickly logging ideas programmed in Go (You are currently reading this) => ../ Back to the main site diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index 3ee013e2..3debfd20 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,6 +1,6 @@ - 2024-09-17T05:21:15+03:00 + 2024-09-27T23:27:37+03:00 foo.zone feed To be in the .zone! @@ -2275,7 +2275,6 @@ http://www.gnu.org/software/src-highlite -->
Other Go related posts are:

-2023-04-09 Algorithms and Data Structures in Go - Part 1
2024-03-03 A fine Fyne Android app for quickly logging ideas programmed in Go (You are currently reading this)

Back to the main site
@@ -5117,289 +5116,6 @@ no1 in 455 days, 18:52:44 | at Sun Jul 21 07:37:51 2024 2022-06-15 Sweating the small stuff - Tiny projects of mine
2023-05-01 Unveiling guprecords.raku: Global Uptime Records with Raku (You are currently reading this)

-Back to the main site
- - - - - Algorithms and Data Structures in Go - Part 1 - - gemini://foo.zone/gemfeed/2023-04-09-algorithms-and-data-structures-in-golang-part-1.gmi - 2023-04-09T22:31:42+03:00 - - Paul Buetow aka snonux - paul@dev.buetow.org - - This is the first blog post about my Algorithms and Data Structures in Go series. I am not a Software Developer in my day job. In my current role, programming and scripting skills are desirable but not mandatory. I have been learning about Data Structures and Algorithms many years ago at University. I thought it would be fun to revisit/refresh my knowledge here and implement many of the algorithms in Go. - -
-

Algorithms and Data Structures in Go - Part 1


-
-Published at 2023-04-09T22:31:42+03:00
-
-This is the first blog post about my Algorithms and Data Structures in Go series. I am not a Software Developer in my day job. In my current role, programming and scripting skills are desirable but not mandatory. I have been learning about Data Structures and Algorithms many years ago at University. I thought it would be fun to revisit/refresh my knowledge here and implement many of the algorithms in Go.
-
-2023-04-09 Algorithms and Data Structures in Go - Part 1 (You are currently reading this)
-
-This post is about setting up some basic data structures and methods for this blog series. I promise, everything will be easy to follow in this post. It will become more interesting later in this series.
-
-
-         ,_---~~~~~----._         
-  _,,_,*^____      _____``*g*\"*, 
- / __/ /'     ^.  /      \ ^@q   f 
-[  @f | @))    |  | @))   l  0 _/  
- \`/   \~____ / __ \_____/    \   
-  |           _l__l_           I   
-  }          [______]           I  
-  ]            | | |            |  
-  ]             ~ ~             |  
-  |                            |   
-   |                           |   
-
-
-

Table of Contents


-
-
-

Type constraints


-
-First, the package ds (data structures) defines the types.go. All examples will either operate on the Integer or Number type:
-
- -
package ds
-
-import (
-	"golang.org/x/exp/constraints"
-)
-
-type Integer interface {
-	constraints.Integer
-}
-
-type Number interface {
-	constraints.Integer | constraints.Float
-}
-
-
-
-

ArrayList


-
-Next comes the arraylist.go, which defines the underlying data structure all the algorithms of this series will use. ArrayList is just a type alias of a Go array (or slice) with custom methods on it:
-
- -
package ds
-
-import (
-	"fmt"
-	"math/rand"
-	"strings"
-)
-
-type ArrayList[V Number] []V
-
-func NewArrayList[V Number](l int) ArrayList[V] {
-	return make(ArrayList[V], l)
-}
-
-
-As you can see, the code uses Go generics, which I refactored recently. Besides the default constructor (which only returns an empty ArrayList with a given capacity), there are also a bunch of special constructors. NewRandomArrayList is returning an ArrayList with random numbers, NewAscendingArrayList and NewDescendingArrayList are returning ArrayLists in either ascending or descending order. They all will be used later on for testing and benchmarking the algorithms.
-
- -
func NewRandomArrayList[V Number](l, max int) ArrayList[V] {
-	a := make(ArrayList[V], l)
-	for i := 0; i < l; i++ {
-		if max > 0 {
-			a[i] = V(rand.Intn(max))
-			continue
-		}
-		a[i] = V(rand.Int())
-	}
-	return a
-}
-
-func NewAscendingArrayList[V Number](l int) ArrayList[V] {
-	a := make(ArrayList[V], l)
-	for i := 0; i < l; i++ {
-		a[i] = V(i)
-	}
-	return a
-}
-
-func NewDescendingArrayList[V Number](l int) ArrayList[V] {
-	a := make(ArrayList[V], l)
-	j := l - 1
-	for i := 0; i < l; i++ {
-		a[i] = V(j)
-		j--
-	}
-	return a
-}
-
-
-

Helper methods


-
-The FirstN method only returns the first N elements of the ArrayList. This is useful for printing out only parts of the data structure:
-
- -
func (a ArrayList[V]) FirstN(n int) string {
-	var sb strings.Builder
-	j := n
-
-	l := len(a)
-	if j > l {
-		j = l
-	}
-
-	for i := 0; i < j; i++ {
-		fmt.Fprintf(&sb, "%v ", a[i])
-	}
-
-	if j < l {
-		fmt.Fprintf(&sb, "... ")
-	}
-
-	return sb.String()
-}
-
-
-The Sorted method checks whether the ArrayList is sorted. This will be used by the unit tests later on:
-
- -
func (a ArrayList[V]) Sorted() bool {
-	for i := len(a) - 1; i > 0; i-- {
-		if a[i] < a[i-1] {
-			return false
-		}
-	}
-	return true
-}
-
-
-And the last utility method used is Swap, which allows swapping the values of two indices in the ArrayList:
-
- -
func (a ArrayList[V]) Swap(i, j int) {
-	aux := a[i]
-	a[i] = a[j]
-	a[j] = aux
-}
-
-
-
-

Sleep sort


-
-Let's implement our first algorithm, sleep sort. Sleep sort is a non-traditional and unconventional sorting algorithm based on the idea of waiting a certain amount of time corresponding to the value of each element in the input ArrayList. It's more of a fun, creative concept rather than an efficient or practical sorting technique. This is not a sorting algorithm you would use in any production code. As you can imagine, it is quite an inefficient sorting algorithm (it's only listed here as a warm-up exercise). This sorting method may also return false results depending on how the Goroutines are scheduled by the Go runtime.
-
-
- -
package sort
-
-import (
-	"codeberg.org/snonux/algorithms/ds"
-	"sync"
-	"time"
-)
-
-func Sleep[V ds.Integer](a ds.ArrayList[V]) ds.ArrayList[V] {
-	sorted := ds.NewArrayList[V](len(a))
-
-	numCh := make(chan V)
-	var wg sync.WaitGroup
-	wg.Add(len(a))
-
-	go func() {
-		wg.Wait()
-		close(numCh)
-	}()
-
-	for _, num := range a {
-		go func(num V) {
-			defer wg.Done()
-			time.Sleep(time.Duration(num) * time.Second)
-			numCh <- num
-		}(num)
-	}
-
-	for num := range numCh {
-		sorted = append(sorted, num)
-	}
-
-	return sorted
-}
-
-
-This Go code implements the sleep sort algorithm using generics and goroutines. The main function Sleep[V ds.Integer](a ds.ArrayList[V]) ds.ArrayList[V] takes a generic ArrayList as input and returns a sorted ArrayList. The code creates a separate goroutine for each element in the input array, sleeps for a duration proportional to the element's value, and then sends the element to a channel. Another goroutine waits for all the sleeping goroutines to finish and then closes the channel. The sorted result ArrayList is constructed by appending the elements received from the channel in the order they arrive. The sync.WaitGroup is used to synchronize goroutines and ensure that all of them have completed before closing the channel.
-
-

Testing


-
-For testing, we only allow values up to 10, as otherwise, it would take too long to finish:
-
- -
package sort
-
-import (
-	"fmt"
-	"testing"
-
-	"codeberg.org/snonux/algorithms/ds"
-)
-
-func TestSleepSort(t *testing.T) {
-	a := ds.NewRandomArrayList[int](10, 10)
-	a = Sleep(a)
-	if !a.Sorted() {
-		t.Errorf("Array not sorted: %v", a)
-	}
-}
-
-
-As you can see, it takes 9s here for the algorithm to finish (which is the highest value in the ArrayList):
-
- -
❯ go test ./sort -v -run SleepSort
-=== RUN   TestSleepSort
---- PASS: TestSleepSort (9.00s)
-PASS
-ok      codeberg.org/snonux/algorithms/sort     9.002s
-
-
-I won't write any benchmark for sleep sort; that will be done for the algorithms to come in this series :-).
-
-E-Mail your comments to paul@nospam.buetow.org :-)
-
Back to the main site
 @@ -9175,6 +8891,530 @@ GNU/kFreeBSD rhea.buetow.org 8.0-RELEASE-p5 FreeBSD 8.0-RELEASE-p5 #2: Sat Nov 2
E-Mail your comments to paul@nospam.buetow.org :-)

+Back to the main site
+ + + + + Bash Golf Part 2 + + gemini://foo.zone/gemfeed/2022-01-01-bash-golf-part-2.gmi + 2022-01-01T23:36:15+00:00 + + Paul Buetow aka snonux + paul@dev.buetow.org + + This is the second blog post about my Bash Golf series. This series is random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content. + +
+

Bash Golf Part 2


+
+Published at 2022-01-01T23:36:15+00:00; Updated at 2022-01-05
+
+This is the second blog post about my Bash Golf series. This series is random Bash tips, tricks and weirdnesses I came across. It's a collection of smaller articles I wrote in an older (in German language) blog, which I translated and refreshed with some new content.
+
+2021-11-29 Bash Golf Part 1
+2022-01-01 Bash Golf Part 2 (You are currently reading this)
+2023-12-10 Bash Golf Part 3
+
+
+    '\       '\                   .  .                |>18>>
+      \        \              .         ' .           |
+     O>>      O>>         .                 'o        |
+      \       .\. ..   .                              |
+      /\    .  /\    . .                              |
+     / /   .  / /  .'    .                            |
+jgs^^^^^^^`^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+                        Art by Joan Stark, mod. by Paul Buetow
+
+
+

Table of Contents


+
+
+

Redirection


+
+Let's have a closer look at Bash redirection. As you might already know that there are 3 standard file descriptors:
+
+
    +
  • 0 aka stdin (standard input)
    • +
  • 1 aka stdout (standard output)
    • +
  • 2 aka stderr (standard error output)
    • +

+These are most certainly the ones you are using on regular basis. "/proc/self/fd" lists all file descriptors which are open by the current process (in this case: the current Bash shell itself):
+
+
+❯ ls -l /proc/self/fd/
+total 0
+lrwx------. 1 paul paul 64 Nov 23 09:46 0 -> /dev/pts/9
+lrwx------. 1 paul paul 64 Nov 23 09:46 1 -> /dev/pts/9
+lrwx------. 1 paul paul 64 Nov 23 09:46 2 -> /dev/pts/9
+lr-x------. 1 paul paul 64 Nov 23 09:46 3 -> /proc/162912/fd
+
+
+The following examples demonstrate two different ways to accomplish the same thing. The difference is that the first command is directly printing out "Foo" to stdout and the second command is explicitly redirecting stdout to its own stdout file descriptor:
+
+
+❯ echo Foo
+Foo
+❯ echo Foo > /proc/self/fd/0
+Foo
+
+
+Update: A reader pointed out, that the redirection should actually go to /proc/self/fd/1 and not 0. But apparently, either way works for this particular example. Do you know why?
+
+Other useful redirections are:
+
+
    +
  • Redirect stderr to stdin: "echo foo 2>&1"
    • +
  • Redirect stdin to stderr: "echo foo >&2"
    • +

+It is, however, not possible to redirect multiple times within the same command. E.g. the following won't work. You would expect stdin to be redirected to stderr and then stderr to be redirected to /dev/null. But as the example shows, Foo is still printed out:
+
+
+❯ echo Foo 1>&2 2>/dev/null
+Foo
+
+
+Update: A reader sent me an email and pointed out that the order of the redirections is important.
+
+As you can see, the following will not print out anything:
+
+
+❯ echo Foo 2>/dev/null 1>&2
+❯
+
+
+A good description (also pointed out by the reader) can be found here:
+
+Order of redirection
+
+Ok, back to the original blog post. You can also use grouping here (neither of these commands will print out anything to stdout):
+
+
+❯ { echo Foo 1>&2; } 2>/dev/null
+❯ ( echo Foo 1>&2; ) 2>/dev/null
+❯ { { { echo Foo 1>&2; } 2>&1; } 1>&2; } 2>/dev/null
+❯ ( ( ( echo Foo 1>&2; ) 2>&1; ) 1>&2; ) 2>/dev/null
+❯
+
+
+A handy way to list all open file descriptors is to use the "lsof" command (that's not a Bash built-in), whereas $$ is the process id (pid) of the current shell process:
+
+
+❯ lsof -a -p $$ -d0,1,2
+COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
+bash    62676 paul    0u   CHR  136,9      0t0   12 /dev/pts/9
+bash    62676 paul    1u   CHR  136,9      0t0   12 /dev/pts/9
+bash    62676 paul    2u   CHR  136,9      0t0   12 /dev/pts/9
+
+
+Let's create our own descriptor "3" for redirection to a file named "foo":
+
+
+❯ touch foo
+❯ exec 3>foo # This opens fd 3 and binds it to file foo.
+❯ ls -l /proc/self/fd/3
+l-wx------. 1 paul paul 64 Nov 23 10:10 \
+    /proc/self/fd/3 -> /home/paul/foo
+❯ cat foo
+❯ echo Bratwurst >&3
+❯ cat foo
+Bratwurst
+❯ exec 3>&- # This closes fd 3.
+❯ echo Steak >&3
+-bash: 3: Bad file descriptor
+
+
+You can also override the default file descriptors, as the following example script demonstrates:
+
+
+❯ cat grandmaster.sh
+#!/usr/bin/env bash
+
+# Write a file data-file containing two lines
+echo Learn You a Haskell > data-file
+echo for Great Good >> data-file
+
+# Link fd with fd 6 (saves default stdin)
+exec 6<&0
+
+# Overwrite stdin with data-file
+exec < data-file
+
+# Read the first two lines from it
+declare LINE1 LINE2
+read LINE1
+read LINE2
+
+# Print them
+echo First line: $LINE1
+echo Second line: $LINE2
+
+# Restore default stdin and delete fd 6
+exec 0<&6 6<&-
+
+
+Let's execute it:
+
+
+❯ chmod 750 ./grandmaster.sh
+❯ ./grandmaster.sh
+First line: Learn You a Haskell
+Second line: for Great Good
+
+
+

HERE


+
+I have mentioned HERE-documents and HERE-strings already in this post. Let's do some more examples. The following "cat" receives a multi line string from stdin. In this case, the input multi line string is a HERE-document. As you can see, it also interpolates variables (in this case the output of "date" running in a subshell).
+
+
+❯ cat <<END
+> Hello World
+> It’s $(date)
+> END
+Hello World
+It's Fri 26 Nov 08:46:52 GMT 2021
+
+
+You can also write it the following way, but that's less readable (it's good for an obfuscation contest):
+
+
+❯ <<END cat
+> Hello Universe
+> It’s $(date)
+> END
+Hello Universe
+It's Fri 26 Nov 08:47:32 GMT 2021
+
+
+Besides of an HERE-document, there is also a so-called HERE-string. Besides of...
+
+
+❯ declare VAR=foo
+❯ if echo "$VAR" | grep -q foo; then
+> echo '$VAR ontains foo'
+> fi
+$VAR ontains foo
+
+
+...you can use a HERE-string like that:
+
+
+❯ if grep -q foo <<< "$VAR"; then
+> echo '$VAR contains foo'
+> fi
+$VAR contains foo
+
+
+Or even shorter, you can do:
+
+
+❯ grep -q foo <<< "$VAR" && echo '$VAR contains foo'
+$VAR contains foo
+
+
+You can also use a Bash regex to accomplish the same thing, but the points of the examples so far were to demonstrate HERE-{documents,strings} and not Bash regular expressions:
+
+
+❯ if [[ "$VAR" =~ foo ]]; then echo yay; fi
+yay
+
+
+You can also use it with "read":
+
+
+❯ read a <<< ja
+❯ echo $a
+ja
+❯ read b <<< 'NEIN!!!'
+❯ echo $b
+NEIN!!!
+❯ dumdidumstring='Learn you a Golang for Great Good'
+❯ read -a words <<< "$dumdidumstring"
+❯ echo ${words[0]}
+Learn
+❯ echo ${words[3]}
+Golang
+
+
+The following is good for an obfuscation contest too:
+
+
+❯ echo 'I like Perl too' > perllove.txt
+❯ cat - perllove.txt <<< "$dumdidumstring"
+Learn you a Golang for Great Good
+I like Perl too
+
+
+

RANDOM


+
+Random is a special built-in variable containing a different pseudo random number each time it's used.
+
+
+❯ echo $RANDOM
+11811
+❯ echo $RANDOM
+14997
+❯ echo $RANDOM
+9104
+
+
+That's very useful if you want to randomly delay the execution of your scripts when you run it on many servers concurrently, just to spread the server load (which might be caused by the script run) better.
+
+Let's say you want to introduce a random delay of 1 minute. You can accomplish it with:
+
+
+❯ cat ./calc_answer_to_ultimate_question_in_life.sh
+#!/usr/bin/env bash
+
+declare -i MAX_DELAY=60
+
+random_delay () {
+    local -i sleep_for=$((RANDOM % MAX_DELAY))
+    echo "Delaying script execution for $sleep_for seconds..."
+    sleep $sleep_for
+    echo 'Continuing script execution...'
+}
+
+main () {
+    random_delay
+    # From here, do the real work. Calculating the answer to
+    # the ultimate question can take billions of years....
+    : ....
+}
+
+main
+
+❯
+❯ ./calc_answer_to_ultimate_question_in_life.sh
+Delaying script execution for 42 seconds...
+Continuing script execution...
+
+
+

set -x and set -e and pipefile


+
+In my opinion, -x and -e and pipefile are the most useful Bash options. Let's have a look at them one after another.
+
+

-x


+
+-x prints commands and their arguments as they are executed. This helps to develop and debug your Bash code:
+
+
+❯ set -x
+❯ square () { local -i num=$1; echo $((num*num)); }
+❯ num=11; echo "Square of $num is $(square $num)"
++ num=11
+++ square 11
+++ local -i num=11
+++ echo 121
++ echo 'Square of 11 is 121'
+Square of 11 is 121
+
+
+You can also set it when calling an external script without modifying the script itself:
+
+
+❯ bash -x ./half_broken_script_to_be_debugged.sh
+
+
+Let's do that on one of the example scripts we covered earlier:
+
+
+❯ bash -x ./grandmaster.sh
++ bash -x ./grandmaster.sh
++ echo Learn You a Haskell
++ echo for Great Good
++ exec
++ exec
++ declare LINE1 LINE2
++ read LINE1
++ read LINE2
++ echo First line: Learn You a Haskell
+First line: Learn You a Haskell
++ echo Second line: for Great Good
+Second line: for Great Good
++ exec
+❯
+
+
+

-e


+
+This is a very important option you want to use when you are paranoid. This means, you should always "set -e" in your scripts when you need to make absolutely sure that your script runs successfully (with that I mean that no command should exit with an unexpected status code).
+
+Ok, let's dig deeper:
+
+
+❯ help set | grep -- -e
+      -e  Exit immediately if a command exits with a non-zero status.
+
+
+As you can see in the following example, the Bash terminates after the execution of "grep" as "foo" is not matching "bar". Therefore, grep exits with 1 (unsuccessfully) and the shell aborts. And therefore, "bar" will not be printed out anymore:
+
+
+❯ bash -c 'set -e; echo hello; grep -q bar <<< foo; echo bar'
+hello
+❯ echo $?
+1
+
+
+Whereas the outcome changes when the regex matches:
+
+
+❯ bash -c 'set -e; echo hello; grep -q bar <<< barman; echo bar'
+hello
+bar
+❯ echo $?
+0
+
+
+So does it mean that grep will always make the shell terminate whenever its exit code isn't 0? This will render "set -e" quite unusable. Frankly, there are other commands where an exit status other than 0 should not terminate the whole script abruptly. Usually, what you want is to branch your code based on the outcome (exit code) of a command:
+
+
+❯ bash -c 'set -e
+>    grep -q bar <<< foo
+>    if [ $? -eq 0 ]; then
+>        echo "matching"
+>    else
+>        echo "not matching"
+>    fi'
+❯ echo $?
+1
+
+
+...but the example above won't reach any of the branches and won't print out anything, as the script terminates right after grep.
+
+The proper solution is to use grep as an expression in a conditional (e.g. in an if-else statement):
+
+
+❯ bash -c 'set -e
+>    if grep -q bar <<< foo; then
+>        echo "matching"
+>    else
+>        echo "not matching"
+>    fi'
+not matching
+❯ echo $?
+0
+❯ bash -c 'set -e
+>    if grep -q bar <<< barman; then
+>        echo "matching"
+>    else
+>        echo "not matching"
+>    fi'
+matching
+❯ echo $?
+0
+
+
+You can also temporally undo "set -e" if there is no other way:
+
+
+❯ cat ./e.sh
+#!/usr/bin/env bash
+
+set -e
+
+foo () {
+    local arg="$1"; shift
+
+    if [ -z "$arg" ]; then
+        arg='You!'
+    fi
+    echo "Hello $arg"
+}
+
+bar () {
+    # Temporally disable e
+    set +e
+    local arg="$1"; shift
+    # Enabl