foo.zone feed

Ultra(re)learning Java - My takeaways

2022-12-24T23:18:40+02:00

Ultra(re)learning Java - My takeaways

Published by Paul at 2022-12-24

+
+

As a regular participant in the annual Pet Project competition at work, I always try to find a project where I can learn something new. In this post, I would like to share my takeaways after revisiting Java (Spoiler: I am still not the biggest Java fan, but Java has its uses). You can read about my motivations in my "Creative universe" post:

+Creative universe
+

I have been programming in Java back in the days as a university student, and even my Diploma Thesis I implemented in Java (it would require some overhaul so that it is fully compatible with a recent version of Java, though - It still compiles and runs, but with a lot of warnings, though!):

+VS-Sim: Distributed systems simulator
+

However, after that, I became a Linux Sysadmin and mainly continued programming in Perl, Puppet, bash, and a little Python. For personal use, I also programmed a bit in Haskell and C. After my Sysadmin role, I moved to London and became a Site Reliability Engineer (SRE), where I mainly programmed in Ruby, bash, Puppet and Golang and a little bit of C.

At my workplace, as an SRE, I don't do Java a lot. I have been reading Java code to understand the software better so I can apply and suggest workarounds or fixes to existing issues and bugs. However, most of our stack is in Java, and our Software Engineers use Java as their primary programming language.

Stuck at Java 1.4

Over time, I had been missing out on many new features that were added to the language since Java 1.4, so I decided to implement my next Pet Project in Java and learn every further aspect of the language as my main goal. Of course, I still liked the idea of winning a Pet Project Prize, but my main objective was to level up my Java skills.

Ultra(re)lerning & upskilling to Java 18

Effective Java

This book was recommended by my brother and also by at least another colleague at work to be one of the best, if not the best, book about Java programming. I read the whole book from the beginning to the end and immersed myself in it. I fully agree; this is a great book. Every Java developer or Java software engineer should read it!

+
+

I recommend reading the 90-part effective Java Series on dev.to. It's a perfect companion to the book as it explains all the chapters again but from a slightly different perspective and helps you to really understand the content.

+Kyle Carter's 90-part Effective Java Series
+

Java Pub House

During my lunch breaks, I usually have a walk around the block or in a nearby park. I used that time to listen to the Java Pub House podcast. I listened to *every* episode and learned tons of new stuff. I can highly recommend this podcast. Especially GraalVM, a high-performance JDK distribution written for Java and other JVM languages, captured my attention. GraalVM can compile Java code into native binaries, improving performance and easing the distribution of Java programs. Because of the latter, I should release a VS-Sim GraalVM edition one day through a Linux AppImage ;-).

+https://www.javapubhouse.com
+https://www.graalvm.org
+

Java Concurrency course

I also watched a course on O'Reilly Safari Books online about Java Concurrency. That gave an excellent refresher on how the Java thread pools work and what were the concurrency primitives available in the standard library.

Read a lot of Java code

First, the source code is often the best documentation (if programmed nicely), and second, it helps to get the hang of the language and standard practices. I started to read more and more Java code at work. I did that whenever I had to understand how something, in particular, worked (e.g. while troubleshooting and debugging an issue).

Observed Java code reviews

Another great way to get the hang of Java again was to sneak into the code reviews of the Software Engineer colleagues. They are the expert on the matter and are a great source to copy knowledge. It's OK to stay passive and only follow the reviews. Sometimes, it's OK to step up and take ownership of the review. The developers will also always be happy to answer any naive questions which come up.

Took ownership of a roadmap-Java project

Besides my Pet Project, I also took ownership of a regular roadmap Java project at work, making an internal Java service capable of running in Kubernetes. This was a bunch of minor changes and adding a bunch of classes and unit tests dealing with the statelessness and a persistent job queue in Redis. The job also involved reading and understanding a lot of already existing Java code. It wasn't part of my job description, but it was fun, and I learned a lot. The service runs smoothly in production now. Of course, all of my code got reviewed by my Software Engineering colleagues.

The good

From the new language features and syntaxes, there are many personal takeaways, and I can't possibly list them all, but here are some of my personal highlights:

Static factory methods and public constructors both have their uses, and it pays to understand their relative merits. Often static factories are preferable (cleaner and easier to read), so avoid the reflex to provide public constructors without first considering static factories.
Java streams were utterly new to me. I love how they can help to produce more compact code. But it's challenging to set the line of when enough is enough. Overusing streams can have the opposite effect: Code becomes more complex and challenging to understand. And it is so easy to parallelize the computation of streams by "just" marking the stream as .parallel() (more on that later in this post).
Overall, object-oriented languages tend to include more and more functional paradigms. The functional interfaces, which Java provides now, are fantastic. Their full powers shine in combination with the use of streams. An entire book can be written about Java functional interfaces, so I leave it to you to do any further digging.
Local type inference help to reduce even more boilerplate code. E.g. instead of Hash> foo = new Hash>(); it's possible to just write var foo = new Hash>();
Class inheritance isn't the preferred way anymore to structure reusable code. Now, it's composition over inheritance. E.g. use dependency injection (inject one object to another object through its constructor) or prefer interfaces (which now also support default implementations of methods) over class inheritance. This makes sense to me as I do that already when I program in Ruby.
I learned the try-with-resources pattern. Very useful in ensuring closing resources again correctly. No need anymore for complicated and nested finally-blocks, which used to be almost impossible to get right previously in case of an error condition (e.g. I/O error somewhere deeply nested in an input or output stream).
Optimize only when required. It's considered to be cleaner to prefer immutable variables (declaring them as final). I knew that already, but for Java, it always seemed to be a waste of resources (creating entirely new objects whenever states change), but apparently, it's okay. Java also does many internal tricks for performance optimization here, e.g. interning strings.
I learned about the concept of static member classes and the difference between non-static member classes (also sometimes known as inner classes). Non-static member classes have full access to all members of their outer class (think of closure). In contrast, static member classes act like completely separate classes without such access but provide the benefit of a nested name that can help group functionality in the code.
I learned about the existence of thread-local variables. These are only available to the current thread and aren't shared with other threads.
I learned about the concept of Java modules, which help to structure larger code bases better. The traditional Java packages are different.
I learned to love the new Optional type. I already knew the concept from Haskell, where Maybe would be the corresponding type. Optional helps to avoid null-pointers but comes with some (minimal) performance penalty. So, in the end, you end up with both Optional types and null-pointers in your code (depending on the requirements). But I like to prefer Optional over null-pointer when "no result" is a valid return value from a method.
The enum type is way more powerful than I thought. Initially, I felt an enum could only be used to define a list of constants and then to compare an instance to another instance of the same. An enum is still there to define a list of constants, but it's also almost like a class (you can implement constructors, and methods, inherit from other enums). There are quite a lot of possible use cases.
A small but almost the most helpful thing I learned is always to use the @Override annotation when overriding a method from a parent class. If done, Java helps to detect any typos or type errors when overriding methods. That's useful and spares a lot of time debugging where a method was mistakenly overloaded but not overridden.
Lambdas are much cleaner, shorter and easier to read than anonymous classes. Many Java libraries require passing instances of (anonymous) classes (e.g. in Swing) to other objects. Lambdas are so lovely because they are primarily compatible with the passing of anonymous classes, so they are a 1:1 replacement in many instances. Lambdas also play very nicely together with the Java functional interfaces, as each Lambda got a type, and the type can be an already existing functional interface (or, if you got a particular case, you could define your custom functional interface for your own set of Lambdas, of course).
I love the concept of Java records. You can think of a record as an immutable object holding some data (as members). They are ideal for pipe and stream processing. They are much easier to define (with much less boilerplate) and come with write protection out of the box.

The bad and the ugly

There are also many ugly corners in Java. Many are doomed to stay there forever due to historical decisions and ensuring backward compatibility with older versions of the Java language and the Java standard library.

Finalizers and cleaners seem obsolete, fragile and still, you can use them.
In many cases, extreme caution needs to be taken to minimize the accessibility of class members. You might think that Java provides the best "out-of-the-box" solution for proper encapsulation, but the language has many loopholes.
In the early days, Java didn't support generics yet. So what you would use is to cast everything to Object. Java now fully supports generics (for a while already), but you can still cast everything to Object and back to whatever type you want. That can lead to nasty runtime errors. Also, there's a particular case to convert between an Array of Object to an Array of String or from an Array of String to a List of String. Java can't convert between these types automatically, and extreme caution needs to be taken when enforcing so (e.g. through explicit type casts). In many of these cases, Java would print out warnings that need to be manually suppressed via annotations. Programming that way, converting data between old and new best practices, is clunky.
If you don't know what you do, Java streams can be all wrong. Side effects in functions used in streams can be nasty to debug. Also, don't just blindly add a .parallel() to your stream. You need to understand what the stream does and how it exactly works; otherwise, parallelizing a stream can impact the performance drastically (in a negative way). There need to be language constructs preventing you from doing the wrong things. That's so much easier to do it right in a purely functional programming language like Haskell.
Java is a pretty old language (already), so there are many obstacles to consider. There are too many exceptions and different outcomes of how Java code can behave. In most cases, when you write an API, every method you program needs to be documented so the user won't encounter any surprises using your code. Writing and reading a lot of documentation seems to be quite the overhead when the method name is already descriptive.
Java serialization is broken. It works, and the language still supports it, but you better not use Java's native way of object serialization and deserialization. Unbelievable how much can get wrong here, especially regarding security (injecting arbitrary code).
Being a bit spoiled by Golang's Goroutines, I was shocked about the limitations of the Java threads. They are resource hungry, and you can't just spin up millions of them as you would with Goroutines. I knew this limitation of threads already (as it's not a problem of the language but of how threads work in the OS), but still, I was pretty shocked when I got reminded of them again. Of course, there's a workaround: Use asynchronous sockets so that you don't waste a whole thread on a single I/O operation (in my case, waiting for a network response). Golang's runtime does that automatically for you: An OS thread will be re-used for other tasks until the network socket unblocks. Every modern programming language should support lightweight threads or Coroutines like Go's Goroutines.

Conclusion

While (re)learning Java, I felt like a student again and was quite enthusiastic about it initially. I invested around half a year, immersing myself intensively in Java (again). The last time I did that was many years ago as a university student. I even won a Silver Prize at work, implementing a project this year (2022 as of writing this). I feel confident now with understanding, debugging and patching Java code at work, which boosted my debugging and troubleshooting skills.

I don't hate Java, but I don't love programming in it, either. I will, I guess, always see Java as the necessary "evil" to get stuff done (reading code to understand how the service works, adding a tiny feature to make my life easier, adding a quick bug fix to overcome an obstacle...). But still, I lost the Java groove on the way.

Although Java has significantly improved since 1.4, its code still tends to be more boilerplate. Not mainly because due to lines of code (Golang code tends to be quite repetitive, primarily when no generics are used), but due to the levels of abstractions it uses. Class hierarchies can be ten classes or deeper, and it is challenging to understand what the code is doing. Good test coverage and much documentation can mitigate the problem partially. Big enterprises use Java, and so also look at the languages. There are too many libraries and too many abstractions that are bundled with too many legacy abstractions and interfaces and too many exceptions in the library APIs. There's even an external library named Lombok, which aims to reduce Java boilerplate code. Why is there a need for an external library? It should be all part of Java itself.

+https://projectlombok.org/
+

Java needs a clean cut. The clean cut shall be incompatible with previous versions of Java and only promote modern best practices without all the legacy burden carried around. The same can be said for other languages, e.g. Perl, but in Perl, they already attack the problem with the use of flags which change the behaviour of the language to more modern standards. Or do it like Python, where they had a hard (incompatible) cut from version 2 to version 3. It will be painful, for sure. But that would be the only way I would enjoy using that language as one of my primary languages to code new stuff regularly. Currently, my Java will stay limited to very few projects and the more minor things already mentioned in this post.

Am I a Java expert now? No, by far not. But I will look at features I don't know once in a while when encountering them accidentally.

E-Mail your comments to paul at buetow dot org! :-)

I tried Doom Emacs, but I switched back to (Neo)Vim

2022-11-24T11:17:15+02:00

I tried (Doom) Emacs, but I switched back to (Neo)Vim

Published by Paul at 2022-11-24 11:17:15 EET, last updated at 2022-11-26

+             _/  \    _(\(o
+             /     \  /  _  ^^^o
+            /   !   \/  ! '!!!v'
+           !  !  \ _' ( \____
+           ! . \ _!\   \===^\)
+Art by      \ \_!  / __!
+ Gunnar Z.   \!   /    \    <--- Emacs is a giant dragon
+       (\_      _/   _\ )
+        \ ^^--^^ __-^ /(__ 
+         ^^----^^    "^--v'
+

As a long-lasting user of Vim (and NeoVim), I always wondered what GNU Emacs is really about, so I decided to try it. I didn't try vanilla GNU Emacs, but Doom Emacs. I chose Doom Emacs as it is a neat distribution of Emacs with Evil mode enabled by default. Evil mode allows Vi(m) key bindings (so to speak, it's emulating Vim within Emacs), and I am pretty sure I won't be ready to give up all the muscle memory I have built over more than a decade.

+GNU Emacs
+Doom Emacs
+

I used Doom Emacs for around two months. Still, ultimately I decided to switch back to NeoVim as my primary editor and IDE and Vim (usually pre-installed on Linux-based systems) and Nvi (usually pre-installed on *BSD systems) as my "always available editor" for quick edits. (It is worth mentioning that I don't have a high opinion on whether Vim or NeoVim is the better editor, I prefer NeoVim as it comes with better defaults out of the box, but there is no real blocker to use Vim instead).

+Vim
+NeoVim
+

So why did I switch back to the Vi-family?

Emacs is a giant dragon

Emacs feels like a giant dragon as it is much more than an editor or an integrated development environment. Emacs is a whole platform on its own. There's an E-Mail client, an IRC client, or even games you can run within Emacs. And you can also change Emacs within Emacs using its own Lisp dialect, Emacs Lisp (Emacs is programmed in Emacs Lisp). Therefore, Emacs is also its own programming language. You can change every aspect of Emacs within Emacs itself. People jokingly state Emacs is an operating system and that you should directly use it as the init 1 process (if you don't know what the init 1 process is: Under UNIX and similar operating systems, it's the very first userland processed launched. That's usually systemd on Linux-based systems, launchd on macOS, or any other init script or init system used by the OS)!

In many aspects, Emacs is like shooting at everything with a bazooka! However, I prefer it simple. I only wanted Emacs to be a good editor (which it is, too), but there's too much other stuff in Emacs that I don't need to care about! Vim and NeoVim do one thing excellent: Being great text editors and, when loaded with plugins, decent IDEs, too.

Magit love

I almost fell in love with Magit, an integrated Git client for Emacs. But I think the best way to interact with Git is to use the git command line directly. I don't worry about typing out all the commands, as the most commonly used commands are in my shell history. Other useful Git programs I use frequently are bit and tig. Also, get a mechanical keyboard that makes hammering whole commands into the terminal even more enjoyable.

+Magit
+Tig
+

Magit is pretty neat for basic Git operations, but I found myself searching the internet for the correct sub-commands to do the things I wanted to do in Git. Mainly, the way how branches are managed is confusing. Often, I fell back to the command line to fix up the mess I produced with Magit (e.g. accidentally pushing to the wrong remote branch, so I found myself fixing things manually on the terminal with the git command with forced pushes....). Magit is hotkey driven, and common commands are quickly explorable through built-in hotkey menus. Still, I found it challenging to navigate to more advanced Git sub-commands that way which was much easier accomplished by using the git command directly.

Graphical UI

If there is one thing I envy about Emacs is that it's a graphical program, whereas the Vi-family of editors are purely terminal-based. I see the benefits of being a graphical program as this enables the use of multiple fonts simultaneously to embed pictures and graphs (that would be neat as a Markdown preview, for example). There's also GVim (Vim with GTK UI), but that's more of an afterthought.

There are now graphical front-end clients for NeoVim, but I still need to dig into them. Let me know your experience if you have one. Luckily, I don't rely on something graphical in my text editor, but it would improve how the editor looks and feels. UTF8 can already do a lot in the terminal, and terminal emulators also allow you to use TrueType fonts. Still, you will always be limited to one TTF font for the whole terminal, and it isn't possible to have, for example, a different font for headings, paragraphs, etc... you get the idea. TTF+UTF8 can't beat authentic graphics.

Scripting it

It is possible to customize every aspect of Emacs through Emacs Lisp. I have done some Elk Scheme programming in the past (a dialect of Lisp), but that was a long time ago, and I am not willing to dive here again to customize my environment. I would instead take the pragmatic approach and script what I need in VimScript (a terrible language, but it gets the job done!). I watched Damian Conway's VimScript course on O'Reilly Safari Books Online, which I greatly recommend. Yes, VimScript feels clunky, funky and weird and is far less elegant than Lisp, but it gets its job done - in most cases! (That reminds me that the Vim team has announced a new major version of VimScript with improvements and language changes made - I haven't gotten to it yet - but I assume that VimScript will always stay VimScript).

+Emacs Lisp
+Elk Scheme
+VimScript
+Scripting Vim by Damian Conway
+

NeoVim is also programmable with Lua, which seems to be a step up and Vim comes with a Perl plugin API (which was removed from NeoVim, but that is a different story - why would someone remove the most potent mature text manipulation programming language from one of the most powerful text editors?).

+NeoVim Lua API
+

One example is my workflow of how I compose my blog articles (e.g. this one you are currently reading): I am writing everything in NeoVim, but I also want to have every paragraph checked against Grammarly (as English is not my first language). So I write a whole paragraph, then I select the entire paragraph via visual selection with SHIFT+v, and then I press ,y to yank the paragraph to the systems clipboard, then I paste the paragraph to Grammarly's browser window with CTRL+v, let Grammarly suggest the improvements, and then I copy the result back with CTRL+c to the system clipboard and in NeoVim I type ,i to insert the result back overriding the old paragraph (which is still selected in visual mode) with the new content. That all sounds a bit complicated, but it's surprisingly natural and efficient.

To come back to the example, for the clipboard integration, I use this small VimScript snippet, and I didn't have to dig into any Lisp or Perl for this:

+" Clipboard
+vnoremap ,y !pbcopyugv
+vnoremap ,i !pbpaste
+nmap ,i !wpbpaste
+

That's only a very few lines and does precisely what I want. It's quick and dirty but get's the job done! If VimScript becomes too cumbersome, I can use Lua for NeoVim scripting.

The famous Emacs Org mode

Org-mode is an Emacs mode for keeping notes, authoring documents, computational notebooks, literate programming, maintaining to-do lists, planning projects, and more — in a fast and effective plain-text system. There's even a dedicated website for it:

+https://orgmode.org/
+

In short, Org-mode is an "interactive markup language" that helps you organize everything mentioned above. I rarely touched the surface during my two-month experiment with Emacs, and I am impressed by it, so I see the benefits of having that. But it's not for me.

I use "Dead Tree Mode" to organize my work and notes. Dead tree? Yeah, I use an actual pen and a real paper journal (Leuchtturm or a Moleskine and a set of coloured 0.5 Muji Pens are excellent choices). That's far more immersive and flexible than a computer program can ever be. Yes, some automation and interaction with the computer (like calendar scheduling etc.) are missing. Still, an actual paper journal forces you to stay simple and focus on the actual work rather than tinkering with your computer program. (But I could not resist, and I wrote a VimScript which parses a table of contents page in Markdown format of my scanned paper journals, and NeoVim allows me to select a topic so that the corresponding PDF scan on the right journal page gets opened in an external PDF viewer (the PDF viewer is zathura, it uses Vi-keybindings, of course) :-). (See the appendix of this blog post for that script).

+Zathura
+

On the road, I also write some of my notes in Markdown format to NextCloud Notes, which is editable from my phone and via NeoVim on my computers. Markdown is much less powerful than Org-mode, but I prefer it the simple way. There's a neat terminal application, ranger, which I use to browse my NextCloud Notes when they are synced to a local folder on my machine. ranger is a file manager inspired by Vim and therefore makes use of Vim keybindings and it feels just natural to me.

+Ranger - A Vim inspired file manager
+

Did I mention that I also use my zsh (my default shell) and my tmux (terminal multiplexer) in Vi-mode?

+Z shell
+tmux terminal multiplexer
+

Seeking simplicity

I am not ready to dive deep into the whole world of Emacs. I prefer small and simple tools as opposed to complex tools. Emacs comes with many features out of the box, whereas in Vim/NeoVim, you would need to install many plugins to replicate some of the behaviour. Yes, I need to invest time managing all the Vim/NeoVim plugins I use, but I feel more in control compared to Doom Emacs, where a framework around vanilla Emacs manages all the plugins. I could use vanilla Emacs and manage all my plugins the vanilla way, but for me, it's not worth the effort to learn and dive into that as all that I want to do I can already do with Vim/NeoVim.

I am not saying that Vim/NeoVim are simple programs, but they are much simpler than Emacs with much smaller footprints; furthermore, they appear to be more straightforward as I am used to them. I only need Vim/NeoVim to be an editor, an IDE (through some plugins), and nothing more.

Conclusion

I understand the Emacs users now. Emacs is an incredibly powerful platform for almost everything, not just text editing. With Emacs, you can do nearly everything (Writing, editing, programming, calendar scheduling and note taking, Jira integration, playing games, listening to music, reading/writing emails, browsing the web, using as a calculator, generating HTML pages, configuring interactive menus, jumping around between every feature and every file within one single session, chat on IRC, surf the Gopherspace, ... the options are endless....). If you want to have one piece of software which rules it all and you are happy to invest a large part of your time in your platform: Pick Emacs, and over time Emacs will become "your" Emacs, customized to your own needs and change the way it works, which makes the Emacs users stick even more to it.

Vim/NeoVim also comes with a very high degree of customization options, but to a lesser extreme than Emacs (but still, a much higher degree than most other editors out there). If you want the best text editor in the world, which can also be tweaked to be a decent IDE, you are only looking for: Pick Vim or NeoVim! You would also need to invest a lot of time in learning, tweaking and customizing Vim/NeoVim, but that's a little more straightforward, and the result is much more lightweight once you get used to the "Vi way of doing things" you never would want to change back. I haven't tried the Emacs vanilla keystrokes, but they are terrible (that's probably one of the reasons why Doom Emacs uses Vim keybindings by default).

Update: One reader recommended to have a look at NvChad. NvChad is a NeoVim config written in Lua aiming to provide a base configuration with very beautiful UI and blazing fast startuptime (around 0.02 secs ~ 0.07 secs). They tweak UI plugins such as telescope, nvim-tree, bufferline etc well to provide an aesthetic UI experience. That sounds interesting!

+https://github.com/NvChad/NvChad
+

E-Mail your comments to paul at buetow dot org! :-)

Appendix

This is the VimScript I mentioned earlier, which parses a table of contents index of my scanned paper journals and opens the corresponding PDF at the right page in zathura:

+function! ReadJournalPageNumber()
+    let page = expand("")
+    if page !~# '^\d\+$'
+        for str in split(getline("."), "[ ,]")
+            if str =~# '^\d\+$'
+                let page = str
+                break
+            end
+        endfor
+    endif
+    return page
+endfunction
+
+function! ReadJournalMeta()
+    normal! mj
+
+    1/MetaFilePath:/
+    normal! 3w
+    let s:metaFilePath = expand("")
+    echom s:metaFilePath
+
+    1/MetaOffset:/
+    normal! 3w
+    let s:metaOffset = expand("")
+    echom s:metaOffset
+
+    1/MetaPageAtOffset:/
+    normal! 3w
+    let s:metaPageAtOffset = expand("")
+    echom s:metaPageAtOffset
+
+    1/MetaPagesPerScan:/
+    normal! 3w
+    let s:metaPagesPerScan = expand("")
+    echom s:metaPagesPerScan
+
+    normal! `j
+endfunction
+
+function! GetPdfPage(page)
+    return s:metaOffset + (a:page - s:metaPageAtOffset) / s:metaPagesPerScan
+endfunction
+
+function! OpenJournalPage()
+    let page = ReadJournalPageNumber()
+    if page !~# '^\d\+$'
+        echoerr "Could not identify Journal page number"
+    end
+    call ReadJournalMeta()
+    let pdfPage = GetPdfPage(page)
+    echon "Location is " . s:metaFilePath . ":" . pdfPage
+    call system("zathura --mode fullscreen -P " . pdfPage . " " . s:metaFilePath)
+    " call system("evince -p " . pdfPage . " " . s:metaFilePath)
+endfunction
+
+nmap ,j :call OpenJournalPage()
+

Installing DTail on OpenBSD

2022-10-30T11:03:19+02:00

Installing DTail on OpenBSD

Published by Paul at 2022-10-30 11:03:19 EET

+       ,_---~~~~~----._
+ _,,_,*^____      _____``*g*\"*,
+/ __/ /'     ^.  /      \ ^@q   f
+ @f   |       |  |       |  0 _/
+\`/   \~__((@/ __ \__((@/    \
+ |           _l__l_           I    <--- The Go Gopher
+ }          [______]           I
+ ]            | | |            |
+ ]             ~ ~             |
+ |                            |
+  |                           |
+  |                           |       A       ;
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~,--,-/ \---,-/|~~,~~~~~~~~~~~~~~~~~~~~~~~~~~~
+                           _|\,'. /|      /|   `/|-.
+                       \`.'    /|      ,            `;.
+                      ,'\   A     A         A   A _ /| `.;
+                    ,/  _              A       _  / _   /|  ;
+                   /\  / \   ,  ,           A  /    /     `/|
+                  /_| | _ \         ,     ,             ,/  \
+                 // | |/ `.\  ,-      ,       ,   ,/ ,/      \/
+                 / @| |@  / /'   \  \      ,              >  /|    ,--.
+                |\_/   \_/ /      |  |           ,  ,/        \  ./' __:..
+                |  __ __  |       |  | .--.  ,         >  >   |-'   /     `
+              ,/| /  '  \ |       |  |     \      ,           |    /
+             /  |<--.__,->|       |  | .    `.        >  >    /   (
+            /_,' \\  ^  /  \     /  /   `.    >--            /^\   |
+                  \\___/    \   /  /      \__'     \   \   \/   \  |
+                   `.   |/          ,  ,                  /`\    \  )
+                     \  '  |/    ,       V    \          /        `-\
+ OpenBSD Puffy --->   `|/  '  V      V           \    \.'            \_
+                       '`-.       V       V        \./'\
+                           `|/-.      \ /   \ /,---`\         kat
+                            /   `._____V_____V'
+                                       '     '
+

This will be a quick blog post, as I am busy with my personal life now. I have relocated to a different country and am still busy arranging things. So bear with me :-)

In this post, I want to give a quick overview (or how-to) about installing DTail on OpenBSD, as the official documentation only covers Red Hat and Fedora Linux! And this blog post will also be used as my reference!

+https://dtail.dev
+

I am using Rexify for my OpenBSD automation. Check out the following article covering my Rex setup in a little bit more detail:

+Let's Encrypt with OpenBSD and Rex
+

I will also mention some relevant Rexfile snippets in this post!

Compile it

First of all, DTail needs to be downloaded and compiled. For that, git, go, and gmake are required:

+$ doas pkg_add git go gmake
+

I am happy that the Go Programming Language is readily available in the OpenBSD packaging system. Once the dependencies got installed, clone DTail and compile it:

+$ mkdir git
+$ cd git
+$ git clone https://github.com/mimecast/dtail
+$ cd dtail
+$ gmake 
+

You can verify the version by running the following command:

+$ ./dtail --version
+ DTail  4.1.0  Protocol 4.1  Have a lot of fun!
+$ file dtail
+ dtail: ELF 64-bit LSB executable, x86-64, version 1
+

Now, there isn't any need anymore to keep git, go and gmake, so they can be deinstalled now:

+$ doas pkg_delete git go gmake
+

One day I shall create an official OpenBSD port for DTail.

Install it

Installing the binaries is now just a matter of copying them to /usr/local/bin as follows:

+$ for bin in dserver dcat dgrep dmap dtail dtailhealth; do
+  doas cp -p $bin /usr/local/bin/$bin
+  doas chown root:wheel /usr/local/bin/$bin
+done
+

Also, we will be creating the _dserver service user:

+$ doas adduser -class nologin -group _dserver -batch _dserver
+$ doas usermod -d /var/run/dserver/ _dserver
+

The OpenBSD init script is created from scratch (not part of the official DTail project). Run the following to install the bespoke script:

+$ cat <<'END' | doas tee /etc/rc.d/dserver
+#!/bin/ksh
+
+daemon="/usr/local/bin/dserver"
+daemon_flags="-cfg /etc/dserver/dtail.json"
+daemon_user="_dserver"
+
+. /etc/rc.d/rc.subr
+
+rc_reload=NO
+
+rc_pre() {
+    install -d -o _dserver /var/log/dserver
+    install -d -o _dserver /var/run/dserver/cache
+}
+
+rc_cmd $1 &
+END
+$ doas chmod 755 /etc/rc.d/dserver
+

Rexification

This is the task for setting it up via Rex. Note the . . . ., that's a placeholder which we will fill up more and more during this blog post:

+desc 'Setup DTail';
+task 'dtail', group => 'frontends',
+   sub {
+      my $restart = FALSE;
+
+      file '/etc/rc.d/dserver':
+        content => template('./etc/rc.d/dserver.tpl'),
+        owner => 'root',
+        group => 'wheel',
+        mode => '755',
+        on_change => sub { $restart = TRUE };
+
+        .
+        .
+        .
+        .
+
+      service 'dserver' => 'restart' if $restart;
+      service 'dserver', ensure => 'started';
+   };
+

Configure it

Now, DTail is fully installed but still needs to be configured. Grab the default config file from GitHub ...

+$ doas mkdir /etc/dserver
+$ curl https://raw.githubusercontent.com/mimecast/dtail/master/samples/dtail.json.sample |
+    doas tee /etc/dserver/dtail.json
+

... and then edit it and adjust LogDir in the Common section to /var/log/dserver. The result will look like this:

+  "Common": {
+    "LogDir": "/var/log/dserver",
+    "Logger": "Fout",
+    "LogRotation": "Daily",
+    "CacheDir": "cache",
+    "SSHPort": 2222,
+    "LogLevel": "Info"
+  }
+

Rexification

That's as simple as adding the following to the Rex task:

+file '/etc/dserver',
+  ensure => 'directory';
+
+file '/etc/dserver/dtail.json',
+  content => template('./etc/dserver/dtail.json.tpl'),
+  owner => 'root',
+  group => 'wheel',
+  mode => '755',
+  on_change => sub { $restart = TRUE };
+

Update the key cache for it

DTail relies on SSH for secure authentication and communication. However, the system user _dserver has no permission to read the SSH public keys from the user's home directories, so the DTail server also checks for available public keys in an alternative path /var/run/dserver/cache.

The following script, populating the DTail server key cache, can be run periodically via CRON:

+$ cat <<'END' | doas tee /usr/local/bin/dserver-update-key-cache.sh
+#!/bin/ksh
+
+CACHEDIR=/var/run/dserver/cache
+DSERVER_USER=_dserver
+DSERVER_GROUP=_dserver
+
+echo 'Updating SSH key cache'
+
+ls /home/ | while read remoteuser; do
+    keysfile=/home/$remoteuser/.ssh/authorized_keys
+
+    if [ -f $keysfile ]; then
+        cachefile=$CACHEDIR/$remoteuser.authorized_keys
+        echo "Caching $keysfile -> $cachefile"
+
+        cp $keysfile $cachefile
+        chown $DSERVER_USER:$DSERVER_GROUP $cachefile
+        chmod 600 $cachefile
+    fi
+done
+
+# Cleanup obsolete public SSH keys
+find $CACHEDIR -name \*.authorized_keys -type f |
+while read cachefile; do
+    remoteuser=$(basename $cachefile | cut -d. -f1)
+    keysfile=/home/$remoteuser/.ssh/authorized_keys
+
+    if [ ! -f $keysfile ]; then
+        echo 'Deleting obsolete cache file $cachefile'
+        rm $cachefile
+    fi
+done
+
+echo 'All set...'
+END
+$ doas chmod 500 /usr/local/bin/dserver-update-key-cache.sh
+

Note that the script above is a slight variation of the official DTail script. The official DTail one is a bash script, but on OpenBSD, there's ksh. I run it once daily by adding it to the daily.local:

+$ echo /usr/local/bin/dserver-update-key-cache.sh | doas tee -a /etc/daily.local
+/usr/local/bin/dserver-update-key-cache.sh
+

Rexification

That's done by adding ...

+file '/usr/local/bin/dserver-update-key-cache.sh',
+  content => template('./scripts/dserver-update-key-cache.sh.tpl'),
+  owner => 'root',
+  group => 'wheel',
+  mode => '500';
+
+append_if_no_such_line '/etc/daily.local', '/usr/local/bin/dserver-update-key-cache.sh';
+

... to the Rex task!

Start it

Now, it's time to enable and start the DTail server:

+$ sudo rcctl enable dserver
+$ sudo rcctl start dserver
+$ tail -f /var/log/dserver/*.log
+INFO|1022-090634|Starting scheduled job runner after 2s
+INFO|1022-090634|Starting continuous job runner after 2s
+INFO|1022-090644|24204|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
+INFO|1022-090654|24204|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
+INFO|1022-090719|Starting server|DTail 4.1.0 Protocol 4.1 Have a lot of fun!
+INFO|1022-090719|Generating private server RSA host key
+INFO|1022-090719|Starting server
+INFO|1022-090719|Binding server|0.0.0.0:2222
+INFO|1022-090719|Starting scheduled job runner after 2s
+INFO|1022-090719|Starting continuous job runner after 2s
+INFO|1022-090729|86050|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
+INFO|1022-090739|86050|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnect
+.
+.
+.
+Ctr+C
+

As we don't want to wait until tomorrow, let's populate the key cache manually:

+$ doas /usr/local/bin/dserver-update-key-cache.sh
+Updating SSH key cache
+Caching /home/_dserver/.ssh/authorized_keys -> /var/cache/dserver/_dserver.authorized_keys
+Caching /home/admin/.ssh/authorized_keys -> /var/cache/dserver/admin.authorized_keys
+Caching /home/failunderd/.ssh/authorized_keys -> /var/cache/dserver/failunderd.authorized_keys
+Caching /home/git/.ssh/authorized_keys -> /var/cache/dserver/git.authorized_keys
+Caching /home/paul/.ssh/authorized_keys -> /var/cache/dserver/paul.authorized_keys
+Caching /home/rex/.ssh/authorized_keys -> /var/cache/dserver/rex.authorized_keys
+All set...
+

Use it

The DTail server is now ready to serve connections. You can use any DTail commands, such as dtail, dgrep, dmap, dcat, dtailhealth, to do so. Checkout out all the usage examples on the official DTail page.

I have installed DTail server this way on my personal OpenBSD frontends blowfish, and fishfinger, and the following command connects as user rex to both machines and greps the file /etc/fstab for the string local:

+❯ ./dgrep -user rex -servers blowfish.buetow.org,fishfinger.buetow.org --regex local /etc/fstab
+CLIENT|earth|WARN|Encountered unknown host|{blowfish.buetow.org:2222 0xc0000a00f0 0xc0000a61e0 [blowfish.buetow.org]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9ZnF/LAk14SgqCzk38yENVTNfqibcluMTuKx1u53cKSp2xwHWzy0Ni5smFPpJDIQQljQEJl14ZdXvhhjp1kKHxJ79ubqRtIXBlC0PhlnP8Kd+mVLLHYpH9VO4rnaSfHE1kBjWkI7U6lLc6ks4flgAgGTS5Bb7pLAjwdWg794GWcnRh6kSUEQd3SftANqQLgCunDcP2Vc4KR9R78zBmEzXH/OPzl/ANgNA6wWO2OoKKy2VrjwVAab6FW15h3Lr6rYIw3KztpG+UMmEj5ReexIjXi/jUptdnUFWspvAmzIl6kwzzF8ExVyT9D75JRuHvmxXKKjyJRxqb8UnSh2JD4JN [23.88.35.144]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9ZnF/LAk14SgqCzk38yENVTNfqibcluMTuKx1u53cKSp2xwHWzy0Ni5smFPpJDIQQljQEJl14ZdXvhhjp1kKHxJ79ubqRtIXBlC0PhlnP8Kd+mVLLHYpH9VO4rnaSfHE1kBjWkI7U6lLc6ks4flgAgGTS5Bb7pLAjwdWg794GWcnRh6kSUEQd3SftANqQLgCunDcP2Vc4KR9R78zBmEzXH/OPzl/ANgNA6wWO2OoKKy2VrjwVAab6FW15h3Lr6rYIw3KztpG+UMmEj5ReexIjXi/jUptdnUFWspvAmzIl6kwzzF8ExVyT9D75JRuHvmxXKKjyJRxqb8UnSh2JD4JN 0xc0000a2180}
+CLIENT|earth|WARN|Encountered unknown host|{fishfinger.buetow.org:2222 0xc0000a0150 0xc000460110 [fishfinger.buetow.org]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNiikdL7+tWSN0rCaw1tOd9aQgeUFgb830V9ejkyJ5h93PKLCWZSMMCtiabc1aUeUZR//rZjcPHFLuLq/YC+Y3naYtGd6j8qVrcfG8jy3gCbs4tV9SZ9qd5E24mtYqYdGlee6JN6kEWhJxFkEwPfNlG+YAr3KC8lvEAE2JdWvaZavqsqMvHZtAX3b25WCBf2HGkyLZ+d9cnimRUOt+/+353BQFCEct/2mhMVlkr4I23CY6Tsufx0vtxx25nbFdZias6wmhxaE9p3LiWXygPWGU5iZ4RSQSImQz4zyOc9rnJeP1rwGk0OWDJhdKNXuf0kIPdzMfwxv2otgY32/DJj6L [46.23.94.99]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNiikdL7+tWSN0rCaw1tOd9aQgeUFgb830V9ejkyJ5h93PKLCWZSMMCtiabc1aUeUZR//rZjcPHFLuLq/YC+Y3naYtGd6j8qVrcfG8jy3gCbs4tV9SZ9qd5E24mtYqYdGlee6JN6kEWhJxFkEwPfNlG+YAr3KC8lvEAE2JdWvaZavqsqMvHZtAX3b25WCBf2HGkyLZ+d9cnimRUOt+/+353BQFCEct/2mhMVlkr4I23CY6Tsufx0vtxx25nbFdZias6wmhxaE9p3LiWXygPWGU5iZ4RSQSImQz4zyOc9rnJeP1rwGk0OWDJhdKNXuf0kIPdzMfwxv2otgY32/DJj6L 0xc0000a2240}
+Encountered 2 unknown hosts: 'blowfish.buetow.org:2222,fishfinger.buetow.org:2222'
+Do you want to trust these hosts?? (y=yes,a=all,n=no,d=details): a
+CLIENT|earth|INFO|STATS:STATS|cgocalls=11|cpu=8|connected=2|servers=2|connected%=100|new=2|throttle=0|goroutines=19
+CLIENT|earth|INFO|Added hosts to known hosts file|/home/paul/.ssh/known_hosts
+REMOTE|blowfish|100|7|fstab|31bfd9d9a6788844.h /usr/local ffs rw,wxallowed,nodev 1 2
+REMOTE|fishfinger|100|7|fstab|093f510ec5c0f512.h /usr/local ffs rw,wxallowed,nodev 1 2
+

Running it the second time, and given that you trusted the keys the first time, it won't prompt you for the host keys anymore:

+❯ ./dgrep -user rex -servers blowfish.buetow.org,fishfinger.buetow.org --regex local /etc/fstab
+REMOTE|blowfish|100|7|fstab|31bfd9d9a6788844.h /usr/local ffs rw,wxallowed,nodev 1 2
+REMOTE|fishfinger|100|7|fstab|093f510ec5c0f512.h /usr/local ffs rw,wxallowed,nodev 1 2
+

Conclusions

It's a bit of manual work, but it's ok on this small scale! I shall invest time in creating an official OpenBSD port, though. That would render most of the manual steps obsolete, as outlined in this post!

Check out the following for more information:

+https://dtail.dev
+https://github.com/mimecast/dtail
+https://www.rexify.org
+

E-Mail your comments to paul at buetow dot org! :-)

After a bad night's sleep

2022-09-30T09:53:23+03:00

After a bad night's sleep

Published by Paul at 2022-09-30 09:53:23 EEST, last updated at 2022-10-12

+               z
+                z
+                 Z
+       .--.  Z Z
+      / _(c\   .-.     __
+     | / /  '-;   \'-'`  `\______
+     \_\/'/ __/ )  /  )   |      \--,
+     | \`""`__-/ .'--/   /--------\  \
+      \\`  ///-\/   /   /---;-.    '-'
+jgs                (________\  \
+                             '-'
+

Everyone has it once in a while: A bad night's sleep. Here I attempt to list valuable tips on how to deal with it.

Don't take the day off.

Don't take a day off after not sleeping enough the previous night. That would be wasting the holiday allowance. It wouldn't be possible to enjoy my free time anyway, so why not just work? There's still a way for an IT Engineer to be productive (sometimes even more) with half or less of the concentration power available!

Start work early

Probably I am already awake early and am unable to fall asleep again. My strategy here is to "attack" the day: Start work early and finish early. The early bird will also encounter fewer distractions from colleagues.

Sweat the small stuff

There's never a shortage of small items to hook off my list. Most of these items don't require my full concentration power, and I will be happy to get them off my list so that the next day, after a good night's sleep, I can immerse myself again in focused, deep work with all concentration powers at hand.

Examples of "small work items" are:

Tidying up the workspace.
Installing pending computer software updates.
Going through the work backlog: Create new tickets, close obsolete ones, and roughly pre-plan upcoming work.
Finishing off the easy tickets from the current sprint.
Going through any tedious paperwork.
Catch up with the journal and mark off all trivial action items.

Enter the flow state

I find it easy to enter the "flow state" after a bad night's sleep. All I need to do is to put on some ambient music (preferably instrumental chill house) and start to work on a not-too-difficult ticket.

Usually, the "flow state" is associated with deep-focused work, but deep-focused work isn't easily possible under sleep deprivation. It's still possible to be in the flow by working on more manageable tasks and leaving the difficult ones for the next day.

Reschedule meetings

I find engaging in discussions and demanding meetings challenging after a lousy night's sleep. I still attend the sessions I am invited to as "only" a participant, but I prefer to reschedule all meetings I am the primary driver of.

This, unfortunately, also includes interviews. Interviews require full concentration power. So for interviews, I would find a colleague to step in for me or ask to reschedule the interview altogether. Everything else wouldn't make it justice and would waste everyone's time!

Invent

The mind works differently under sleep deprivation: It's easier to invent new stuff as it's easier to have a look at things from different perspectives. Until an hour ago, I didn't know yet what I would be blogging about for this month, and then I just started writing this, and it took me only half an hour to write the first draft of this blog post!

Fast

I don't eat breakfast, and I don't eat lunch on these days. I only have dinner. Not eating means my mind doesn't get foggy, and I keep up the work momentum. This is called intermittent fasting, which not only generally helps to keep the weight under control and boosts the concentration power. Furthermore, intermittent fasting is healthy. You should include it in your routine, even after a good night's sleep.

Stretch

I won't have enough energy for strenuous physical exercise on those days, but a 30 to a 60-minute stretching session can make the day. Stretching will even hurt less under sleep deprivation! The stretching could also be substituted with a light Yoga session.

Walk

Walking is healthy, and the time can be used to listen to interesting podcasts. The available concentration power might not be enough for more sophisticated audio literature. I will have enough energy for one or two daily walks (~10k steps for the day in total). Sometimes, I listen to music during walks. I also try to catch the bright sunlight.

Red Bull

I don't think that Red Bull is a healthy drink. But once in a while, a can in the early afternoon brings wonders, and productivity will skyrocket. Other than Red Bull, drink a lot of water throughout the day. Don't forget to drink the sugar-free version; otherwise, your intermittent fast will be broken.

Power nap

I don't know how to "enforce" a nap, but sometimes I manage to power nap, and it helps wonders. A 30-minute nap sometimes brings me back to normal. If you don't tend to fast as you are too hungry, it helps to try to nap approximately 30 minutes after eating something.

Don't take anything personally.

It's much more challenging to keep the mind "under control" in this state. Every annoyance can potentially upset, which could reflect on the work colleagues. It is wise to attempt to go with a positive attitude into the day, always smile and be polite to the family and colleagues at work. Don't let anything drop out to the people next; they don't deserve it as they didn't do anything wrong! Also, remember, it can't be controlled at all. It's time to let go of the annoyances for the day.

Meditate

To keep the good vibe, it helps to meditate for 10 minutes. Meditation must nothing be fancy. It can be just lying on the sofa and observing your thoughts as they come and go. Don't judge your thoughts, as that could put you in a negative mood. It's not necessary to sit in an uncomfortable Yoga pose, and it is not required to chant "Ohhmmmmm".

Write things down

Sometimes something requiring more concentration power demands time. This is where it helps to write a note in a journal and return to it another day. This doesn't mean slacking off but managing the rarely available concentration power for the day. I might repeat myself: Today, sweat all the small stuff. Tomorrow, do the deep-focused work on that crucial project again.

It's easier to forget things on those days, so everything should be written down so that it can be worked off later. Things written down will not be overlooked!

Social media

I wouldn't say I like checking social media, as it can consume a lot of time and can become addictive. But once in a while, I want to catch up with my "networks". After a bad night's sleep, it's the perfect time to check your social media. Once done, you don't have to do it anymore for the next couple of days!

E-Mail your comments to paul at buetow dot org! :-)