From 07920d17ce75e84100842fb6b367bd2e5233635a Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Fri, 4 Apr 2025 23:22:09 +0300 Subject: Update content for gemtext --- gemfeed/atom.xml | 1010 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 623 insertions(+), 387 deletions(-) (limited to 'gemfeed/atom.xml') diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index 0faf0fdc..0a678298 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,11 +1,613 @@ - 2025-03-05T19:58:06+02:00 + 2025-04-04T23:21:02+03:00 foo.zone feed To be in the .zone! gemini://foo.zone/ + + f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs + + gemini://foo.zone/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi + 2025-04-04T23:21:01+03:00 + + Paul Buetow aka snonux + paul@dev.buetow.org + + This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The 'f' stands for FreeBSD, and the '3s' stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines. + +
+

f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs


+
+This is the fourth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.
+
+2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
+2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
+2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
+2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs (You are currently reading this)
+
+f3s logo
+
+

Table of Contents


+
+
+

Introduction


+
+In this blog post, we are going to install the Bhyve hypervisor.
+
+The FreeBSD Bhyve hypervisor is a lightweight, modern hypervisor that enables virtualization on FreeBSD systems. Bhyve's strengths include its minimal overhead, which allows it to achieve near-native performance for virtual machines. It's efficient and lightweight, leveraging the capabilities of the FreeBSD operating system for performance and network management.
+
+Bhyve supports running various guest operating systems, including FreeBSD, Linux, and Windows, on hardware platforms that support hardware virtualization extensions (such as Intel VT-x or AMD-V). In our case, we are going to virtualize Rocky Linux, which will later in this series be used to run k3s.
+
+

Check for POPCNT CPU support


+
+POPCNT is a CPU instruction that counts the number of set bits (ones) in a binary number. CPU virtualization and Bhyve support for the POPCNT instruction are important because guest operating systems utilize this instruction to perform various tasks more efficiently. If the host CPU supports POPCNT, Bhyve can pass this capability to virtual machines for better performance. Without POPCNT support, some applications might not run or perform sub-optimally in virtualized environments.
+
+To check for POPCNT support, run:
+
+ +
paul@f0:~ % dmesg | grep 'Features2=.*POPCNT'
+  Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,
+	FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,
+	OSXSAVE,AVX,F16C,RDRAND>
+
+
+So it's there! All good.
+
+

Basic Bhyve setup


+
+For managing the Bhyve VMs, we are using vm-bhyve, a tool not part of the FreeBSD operating system but available as a ready-to-use package. It eases VM management and reduces a lot of overhead. We also install the required package to make Bhyve work with the UEFI firmware.
+
+https://github.com/churchers/vm-bhyve
+
+The following commands are executed on all three hosts f0, f1, and f2, where re0 is the name of the Ethernet interface (which may need to be adjusted if your hardware is different):
+
+ +
paul@f0:~ % doas pkg install vm-bhyve bhyve-firmware
+paul@f0:~ % doas sysrc vm_enable=YES
+vm_enable:  -> YES
+paul@f0:~ % doas sysrc vm_dir=zfs:zroot/bhyve
+vm_dir:  -> zfs:zroot/bhyve
+paul@f0:~ % doas zfs create zroot/bhyve
+paul@f0:~ % doas vm init
+paul@f0:~ % doas vm switch create public
+paul@f0:~ % doas vm switch add public re0
+
+
+Bhyve stores all it's data in the /bhyve of the zroot ZFS pool:
+
+ +
paul@f0:~ % zfs list | grep bhyve
+zroot/bhyve                                   1.74M   453G  1.74M  /zroot/bhyve
+
+
+For convenience, we also create this symlink:
+
+ +
paul@f0:~ % doas ln -s /zroot/bhyve/ /bhyve
+
+
+
+Now, Bhyve is ready to rumble, but no VMs are there yet:
+
+ +
paul@f0:~ % doas vm list
+NAME  DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO  STATE
+
+
+

Rocky Linux VMs


+
+As guest VMs I decided to use Rocky Linux.
+
+Using Rocky Linux 9 as a VM-based OS is beneficial primarily because of its long-term support and stable release cycle. This ensures a reliable environment that receives security updates and bug fixes for an extended period, reducing the need for frequent upgrades. Rocky Linux is community-driven and aims to be fully compatible with enterprise Linux, making it a solid choice for consistency and performance in various deployment scenarios.
+
+https://rockylinux.org/
+
+

ISO download


+
+We're going to install the Rocky Linux from the latest minimal iso:
+
+ +
paul@f0:~ % doas vm iso \
+ https://download.rockylinux.org/pub/rocky/9/isos/x86_64/Rocky-9.5-x86_64-minimal.iso
+/zroot/bhyve/.iso/Rocky-9.5-x86_64-minimal.iso        1808 MB 4780 kBps 06m28s
+paul@f0:/bhyve % doas vm create rocky
+
+

VM configuration


+
+The default Bhyve VM configuration looks like this now:
+
+ +
paul@f0:/bhyve/rocky % cat rocky.conf
+loader="bhyveload"
+cpu=1
+memory=256M
+network0_type="virtio-net"
+network0_switch="public"
+disk0_type="virtio-blk"
+disk0_name="disk0.img"
+uuid="1c4655ac-c828-11ef-a920-e8ff1ed71ca0"
+network0_mac="58:9c:fc:0d:13:3f"
+
+
+The uuid and the network0_mac differ for each of the three VMs.
+
+But to make Rocky Linux boot it (plus some other adjustments, e.g. as we intend to run the majority of the workload in the k3s cluster running on those Linux VMs, we give them beefy specs like 4 CPU cores and 14GB RAM). So we run doas vm configure rocky and modified it to:
+
+
+guest="linux"
+loader="uefi"
+uefi_vars="yes"
+cpu=4
+memory=14G
+network0_type="virtio-net"
+network0_switch="public"
+disk0_type="virtio-blk"
+disk0_name="disk0.img"
+graphics="yes"
+graphics_vga=io
+uuid="1c45400b-c828-11ef-8871-e8ff1ed71cac"
+network0_mac="58:9c:fc:0d:13:3f"
+
+
+

VM installation


+
+To start the installer from the downloaded ISO, we run:
+
+ +
paul@f0:~ % doas vm install rocky Rocky-9.5-x86_64-minimal.iso
+Starting rocky
+  * found guest in /zroot/bhyve/rocky
+  * booting...
+
+paul@f0:/bhyve/rocky % doas vm list
+NAME   DATASTORE  LOADER  CPU  MEMORY  VNC           AUTO  STATE
+rocky  default    uefi    4    14G     0.0.0.0:5900  No    Locked (f0.lan.buetow.org)
+
+paul@f0:/bhyve/rocky % doas sockstat -4 | grep 5900
+root     bhyve       6079 8   tcp4   *:5900                *:*
+
+
+Port 5900 now also opens for VNC connections, so I connected it with a VNC client and ran through the installation dialogues. This could be done unattended or more automated, but there are only three VMs to install, and the automation doesn't seem worth it as we do it only once a year or less often.
+
+

Increase of the disk image


+
+By default, the VM disk image is only 20G, which is a bit small for our purposes, so I stopped the VMs again, ran truncate on the image file to enlarge them to 100G, and re-started the installation:
+
+ +
paul@f0:/bhyve/rocky % doas vm stop rocky
+paul@f0:/bhyve/rocky % doas truncate -s 100G disk0.img
+paul@f0:/bhyve/rocky % doas vm install rocky Rocky-9.5-x86_64-minimal.iso
+
+
+

Connect to VNC


+
+For the installation, I opened the VNC client on my Fedora laptop (GNOME comes with a simple VNC client) and manually ran through the base installation for each of the VMs. Again, I am sure this could have been automated a bit more, but there were just three VMs, and it wasn't worth the effort. The three VNC addresses of the VMs were vnc://f0:5900, vnc://f1:5900, and vnc://f0:5900.
+
+
+
+
+
+I primarily selected the default settings (auto partitioning on the 100GB drive and a root user password). After the installation, the VMs were rebooted.
+
+
+
+
+
+

After install


+
+We perform the following steps for all 3 VMs. In the following, the examples are all executed on f0 (the VM r0 running on f0):
+
+

VM auto-start after host reboot


+
+To automatically start the VM on the servers, we add the following to the rc.conf on the FreeBSD hosts:
+
+ +
paul@f0:/bhyve/rocky % cat <<END | doas tee -a /etc/rc.conf
+vm_list="rocky"
+vm_delay="5"
+
+
+The vm_delay isn't really required. It is used to wait 5 seconds before starting each VM, but there is currently only one VM per host. Maybe later, when there are more, this will be useful. After adding, there's now a Yes indicator in the AUTO column.
+
+ +
paul@f0:~ % doas vm list
+NAME   DATASTORE  LOADER  CPU  MEMORY  VNC           AUTO     STATE
+rocky  default    uefi    4    14G     0.0.0.0:5900  Yes [1]  Running (2063)
+
+
+

Static IP configuration


+
+After that, we change the network configuration of the VMs to be static (from DHCP) here. As per the previous post of this series, the 3 FreeBSD hosts were already in my /etc/hosts file:
+
+
+192.168.1.130 f0 f0.lan f0.lan.buetow.org
+192.168.1.131 f1 f1.lan f1.lan.buetow.org
+192.168.1.132 f2 f2.lan f2.lan.buetow.org
+
+
+For the Rocky VMs, we add those to the FreeBSD host systems as well:
+
+ +
paul@f0:/bhyve/rocky % cat <<END | doas tee -a /etc/hosts
+192.168.1.120 r0 r0.lan r0.lan.buetow.org
+192.168.1.121 r1 r1.lan r1.lan.buetow.org
+192.168.1.122 r2 r2.lan r2.lan.buetow.org
+END
+
+
+And we configure the IPs accordingly on the VMs themselves by opening a root shell via RDP to the VMs and entering the following commands on each of the VMs:
+
+ +
[root@r0 ~] % dnmcli connection modify enp0s5 ipv4.address 192.168.1.120/24
+[root@r0 ~] % dnmcli connection modify enp0s5 ipv4.gateway 192.168.1.1
+[root@r0 ~] % dnmcli connection modify enp0s5 ipv4.DNS 192.168.1.1
+[root@r0 ~] % dnmcli connection modify enp0s5 ipv4.method manual
+[root@r0 ~] % dnmcli connection down enp0s5
+[root@r0 ~] % dnmcli connection up enp0s5
+[root@r0 ~] % hostnamectl set-hostname r0.lan.buetow.org
+[root@r0 ~] % cat <<END >>/etc/hosts
+192.168.1.120 r0 r0.lan r0.lan.buetow.org
+192.168.1.121 r1 r1.lan r1.lan.buetow.org
+192.168.1.122 r2 r2.lan r2.lan.buetow.org
+END
+
+
+Whereas:
+
+
    +
  • 192.168.1.120 is the IP of the VM itself (here: r0.lan.buetow.org)
    • +
  • 192.168.1.1 is the address of my home router, which also does DNS.
    • +

+

Permitting root login


+
+As these VMs aren't directly reachable via SSH from the internet, we enable root login by adding a line with PermitRootLogin yes to /etc/sshd/sshd_config.
+
+Once done, we reboot the VM by running reboot inside the VM to test whether everything was configured and persisted correctly.
+
+After reboot, I copied my public key from my Laptop to the 3 VMs:
+
+ +
% for i in 0 1 2; do ssh-copy-id root@r$i.lan.buetow.org; done
+
+
+Then, I edited the /etc/ssh/sshd_config file again on all 3 VMs and configured PasswordAuthentication no to only allow SSH key authentication from now on.
+
+

Install latest updates


+
+ +
[root@r0 ~] % dnf update
+[root@r0 ~] % reboot
+
+
+

Stress testing CPU


+
+The aim is to prove that bhyve VMs are CPU efficient. As I could not find an off-the-shelf benchmarking tool available in the same version for FreeBSD as well as for Rocky Linux 9, I wrote my own silly CPU benchmarking tool in Go:
+
+ +
package main
+
+import "testing"
+
+func BenchmarkCPUSilly1(b *testing.B) {
+	for i := 0; i < b.N; i++ {
+		_ = i * i
+	}
+}
+
+func BenchmarkCPUSilly2(b *testing.B) {
+	var sillyResult float64
+	for i := 0; i < b.N; i++ {
+		sillyResult += float64(i)
+		sillyResult *= float64(i)
+		divisor := float64(i) + 1
+		if divisor > 0 {
+			sillyResult /= divisor
+		}
+	}
+	_ = sillyResult // to avoid compiler optimization
+}
+
+
+You can find the repository here:
+
+https://codeberg.org/snonux/sillybench
+
+

Silly FreeBSD host benchmark


+
+To install it on FreeBSD, we run:
+
+ +
paul@f0:~ % doas pkg install git go
+paul@f0:~ % mkdir ~/git && cd ~/git && \
+  git clone https://codeberg.org/snonux/sillybench && \
+  cd sillybench
+
+
+And to run it:
+
+ +
paul@f0:~/git/sillybench % go version
+go version go1.24.1 freebsd/amd64
+
+paul@f0:~/git/sillybench % go test -bench=.
+goos: freebsd
+goarch: amd64
+pkg: codeberg.org/snonux/sillybench
+cpu: Intel(R) N100
+BenchmarkCPUSilly1-4    1000000000               0.4022 ns/op
+BenchmarkCPUSilly2-4    1000000000               0.4027 ns/op
+PASS
+ok      codeberg.org/snonux/sillybench 0.891s
+
+
+

Silly Rocky Linux VM @ Bhyve benchmark


+
+OK, let's compare this with the Rocky Linux VM running on Bhyve:
+
+ +
[root@r0 ~]# dnf install golang git
+[root@r0 ~]# mkdir ~/git && cd ~/git && \
+  git clone https://codeberg.org/snonux/sillybench && \
+  cd sillybench
+
+
+And to run it:
+
+ +
[root@r0 sillybench]# go version
+go version go1.22.9 (Red Hat 1.22.9-2.el9_5) linux/amd64
+[root@r0 sillybench]# go test -bench=.
+goos: linux
+goarch: amd64
+pkg: codeberg.org/snonux/sillybench
+cpu: Intel(R) N100
+BenchmarkCPUSilly1-4    1000000000               0.4347 ns/op
+BenchmarkCPUSilly2-4    1000000000               0.4345 ns/op
+
+The Linux benchmark is slightly slower than the FreeBSD one. The Go version is also a bit older. I tried the same with the up-to-date version of Go (1.24.x) with similar results. There could be a slight Bhyve overhead, or FreeBSD is just slightly more efficient in this benchmark. Overall, this shows that Bhyve performs excellently.
+
+

Silly FreeBSD VM @ Bhyve benchmark


+
+But as I am curious and don't want to compare apples with bananas, I decided to install a FreeBSD Bhyve VM to run the same silly benchmark in it. I am not going through the details of how to install a FreeBSD Bhyve VM here; you can easily look it up in the documentation.
+
+But here are the results running the same silly benchmark in a FreeBSD Bhyve VM with the same FreeBSD and Go versions as the host system (I have the VM 4 vCPUs and 14GB of RAM; the benchmark won't use as many CPUs anyway):
+
+ +
root@freebsd:~/git/sillybench # go test -bench=.
+goos: freebsd
+goarch: amd64
+pkg: codeberg.org/snonux/sillybench
+cpu: Intel(R) N100
+BenchmarkCPUSilly1      1000000000               0.4273 ns/op
+BenchmarkCPUSilly2      1000000000               0.4286 ns/op
+PASS
+ok      codeberg.org/snonux/sillybench  0.949s
+
+
+It's a bit better than Linux! I am sure that this is not really a scientific benchmark, so take the results with a grain of salt!
+
+

Benchmarking with ubench


+
+Let's run another, more sophisticated benchmark using ubench, the Unix Benchmark Utility available for FreeBSD. It was installed by simply running doas pkg install ubench. It can benchmark CPU and memory performance. Here, we limit it to one CPU for the first run with -s, and then let it run at full speed in the second run.
+
+

FreeBSD host ubench benchmark


+
+Single CPU:
+
+ +
paul@f0:~ % doas ubench -s 1
+Unix Benchmark Utility v.0.3
+Copyright (C) July, 1999 PhysTech, Inc.
+Author: Sergei Viznyuk <sv@phystech.com>
+http://www.phystech.com/download/ubench.html
+FreeBSD 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64
+Ubench Single CPU:   671010 (0.40s)
+Ubench Single MEM:  1705237 (0.48s)
+-----------------------------------
+Ubench Single AVG:  1188123
+
+
+
+All CPUs (with all Bhyve VMs stopped):
+
+ +
paul@f0:~ % doas ubench
+Unix Benchmark Utility v.0.3
+Copyright (C) July, 1999 PhysTech, Inc.
+Author: Sergei Viznyuk <sv@phystech.com>
+http://www.phystech.com/download/ubench.html
+FreeBSD 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64
+Ubench CPU:  2660220
+Ubench MEM:  3095182
+--------------------
+Ubench AVG:  2877701
+
+
+

FreeBSD VM @ Bhyve ubench benchmark


+
+Single CPU:
+
+ +
root@freebsd:~ # ubench -s 1
+Unix Benchmark Utility v.0.3
+Copyright (C) July, 1999 PhysTech, Inc.
+Author: Sergei Viznyuk <sv@phystech.com>
+http://www.phystech.com/download/ubench.html
+FreeBSD 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64
+Ubench Single CPU:   672792 (0.40s)
+Ubench Single MEM:   852757 (0.48s)
+-----------------------------------
+Ubench Single AVG:   762774
+
+
+All CPUs:
+
+ +
root@freebsd:~ # ubench
+Unix Benchmark Utility v.0.3
+Copyright (C) July, 1999 PhysTech, Inc.
+Author: Sergei Viznyuk <sv@phystech.com>
+http://www.phystech.com/download/ubench.html
+FreeBSD 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64
+Ubench CPU:  2652857
+swap_pager: out of swap space
+swp_pager_getswapspace(27): failed
+swap_pager: out of swap space
+swp_pager_getswapspace(18): failed
+Apr  4 23:02:43 freebsd kernel: pid 862 (ubench), jid 0, uid 0, was killed: failed to reclaim memory
+swp_pager_getswapspace(6): failed
+Apr  4 23:02:46 freebsd kernel: pid 863 (ubench), jid 0, uid 0, was killed: failed to reclaim memory
+Apr  4 23:02:47 freebsd kernel: pid 864 (ubench), jid 0, uid 0, was killed: failed to reclaim memory
+Apr  4 23:02:48 freebsd kernel: pid 865 (ubench), jid 0, uid 0, was killed: failed to reclaim memory
+Apr  4 23:02:49 freebsd kernel: pid 861 (ubench), jid 0, uid 0, was killed: failed to reclaim memory
+Apr  4 23:02:51 freebsd kernel: pid 839 (ubench), jid 0, uid 0, was killed: failed to reclaim memory
+
+
+The multi-CPU benchmark in the Bhyve VM ran with almost identical results to the FreeBSD host system. However, the memory benchmark failed with out-of-swap space errors. I am unsure why, as the VM has 14GB RAM, but I am not investigating further.
+
+Also, during the benchmark, I noticed the bhyve process on the host was constantly using 399% of the CPU (all 4 CPUs).
+
+
+  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
+ 7449 root         14  20    0    14G    78M kqread   2   2:12 399.81% bhyve
+
+
+Overall, Bhyve has a small overhead, but the CPU performance difference is negligible. The FreeBSD host is slightly faster than the FreeBSD VM running on Bhyve, but the difference is small enough for our use cases. The memory benchmark seems slightly off, but I don't know whether to trust it. Do you have an idea?
+
+

Rocky Linux VM @ Bhyve ubench benchmark


+
+Unfortunately, I wasn't able to find ubench in any of the Rocky Linux repositories. So, I skipped this test.
+
+

Conclusion


+
+Having Linux VMs running inside FreeBSD's Bhyve is a solid move for future F3s hosting in my home lab. Bhyve provides a reliable way to manage VMs without much hassle. With Linux VMs, I can tap into all the cool stuff (e.g., Kubernetes) in the Linux world while keeping the steady reliability of FreeBSD.
+
+Future uses (out of scope for this blog series) would be additional VMs for different workloads. For example, how about a Windows or NetBSD VM to tinker with?
+
+This flexibility is great for keeping options open and managing different workloads without overcomplicating things. Overall, it's a nice setup for getting the most out of my hardware and keeping things running smoothly.
+
+See you in the next blog post of this series. Maybe we will be installing highly available storage with HAST or we start setting up k3s on the Rocky Linux VMs.
+
+Other *BSD-related posts:
+
+2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs (You are currently reading this)
+2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
+2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
+2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
+2024-04-01 KISS high-availability with OpenBSD
+2024-01-13 One reason why I love OpenBSD
+2022-10-30 Installing DTail on OpenBSD
+2022-07-30 Let's Encrypt with OpenBSD and Rex
+2016-04-09 Jails and ZFS with Puppet on FreeBSD
+
+E-Mail your comments to paul@nospam.buetow.org
+
+Back to the main site
+
+ + Sharing on Social Media with Gos v1.0.0 @@ -476,7 +1078,7 @@ http://www.gnu.org/software/src-highlite -->

13. Go functions can have methods



-Functions on struct types? Well, know. Functions on types like int and string? It's also known of, but a bit lesser. Functions on function types? That sounds a bit funky, but it's possible, too! For demonstration, have a look at this snippet:
+Functions on struct types? Well known. Functions on types like int and string? It's also known of, but a bit lesser. Functions on function types? That sounds a bit funky, but it's possible, too! For demonstration, have a look at this snippet:


14. ß and ss are treated the same



-Know German? In German, the letter "sarp s" is written as ß. ß is treated the same as ss on macOS.
+Know German? In German, the letter "sharp s" is written as ß. ß is treated the same as ss on macOS.

On a case-insensitive file system like macOS, not only are uppercase and lowercase letters treated the same, but non-Latin characters like the German "ß" are also considered equivalent to their Latin counterparts (in this case, "ss").

@@ -709,6 +1311,7 @@ This is perl, v5.8.8 built 8.8 built
+
+See you in the next post of this series!

Other BSD related posts are:

+2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts (You are currently reading this)
2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
@@ -1797,6 +2408,7 @@ http://www.gnu.org/software/src-highlite -->
These are all the posts so far:

+2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation (You are currently reading this)
2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
@@ -2130,6 +2742,7 @@ dev.cpu.0.freq: 2922
Other *BSD-related posts:

+2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation (You are currently reading this)
2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
@@ -2167,6 +2780,7 @@ dev.cpu.0.freq: 2922
These are all the posts so far:

+2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage (You are currently reading this)
@@ -2193,7 +2807,7 @@ dev.cpu.0.freq: 2922
  • ⇢ Monitoring: Keeping an eye on everything
  • ⇢ ⇢ Prometheus and Grafana
  • ⇢ ⇢ Gogios: My custom alerting system
    • -
  • ⇢ What's after this all?
    • +
  • ⇢ Conclusion

    • Why this setup?



    @@ -2303,7 +2917,7 @@ dev.cpu.0.freq: 2922
    Ironically, I implemented Gogios to avoid using more complex alerting systems like Prometheus, but here we go—it integrates well now.

    -

    What's after this all?


    +

    Conclusion



    This setup may be just the beginning. Some ideas I'm thinking about for the future:

    @@ -2321,6 +2935,7 @@ dev.cpu.0.freq: 2922
    Other *BSD-related posts:

    +2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
    2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
    2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
    2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage (You are currently reading this)
    @@ -4809,6 +5424,7 @@ http://www.gnu.org/software/src-highlite -->
    Other *BSD and KISS related posts are:

    +2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
    2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
    2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
    2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
    @@ -5170,6 +5786,7 @@ $ doas reboot # Just in case, reboot one more time Other *BSD related posts are:

    +2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs
    2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts
    2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation
    2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage
    @@ -8771,387 +9388,6 @@ nmap ,i !wpbpaste<CR>
    E-Mail your comments to paul@nospam.buetow.org :-)

    -Back to the main site
    - - -     - - Installing DTail on OpenBSD - - gemini://foo.zone/gemfeed/2022-10-30-installing-dtail-on-openbsd.gmi - 2022-10-30T11:03:19+02:00 - - Paul Buetow aka snonux - paul@dev.buetow.org - - This will be a quick blog post, as I am busy with my personal life now. I have relocated to a different country and am still busy arranging things. So bear with me :-) - -
    -

    Installing DTail on OpenBSD


    -
    -Published at 2022-10-30T11:03:19+02:00
    -
    -This will be a quick blog post, as I am busy with my personal life now. I have relocated to a different country and am still busy arranging things. So bear with me :-)
    -
    - In this post, I want to give a quick overview (or how-to) about installing DTail on OpenBSD, as the official documentation only covers Red Hat and Fedora Linux! And this blog post will also be used as my reference!
    -
    -https://dtail.dev
    -
    -I am using Rexify for my OpenBSD automation. Check out the following article covering my Rex setup in a little bit more detail:
    -
    -Let's Encrypt with OpenBSD and Rex
    -
    -I will also mention some relevant Rexfile snippets in this post!
    -
    -
    -       ,_---~~~~~----._
- _,,_,*^____      _____``*g*\"*,
-/ __/ /'     ^.  /      \ ^@q   f
- @f   |       |  |       |  0 _/
-\`/   \~__((@/ __ \__((@/    \
- |           _l__l_           I    <--- The Go Gopher
- }          [______]           I
- ]            | | |            |
- ]             ~ ~             |
- |                            |
-  |                           |
-  |                           |       A       ;
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~|~~~,--,-/ \---,-/|~~,~~~~~~~~~~~~~~~~~~~~~~~~~~~
-                           _|\,'. /|      /|   `/|-.
-                       \`.'    /|      ,            `;.
-                      ,'\   A     A         A   A _ /| `.;
-                    ,/  _              A       _  / _   /|  ;
-                   /\  / \   ,  ,           A  /    /     `/|
-                  /_| | _ \         ,     ,             ,/  \
-                 // | |/ `.\  ,-      ,       ,   ,/ ,/      \/
-                 / @| |@  / /'   \  \      ,              >  /|    ,--.
-                |\_/   \_/ /      |  |           ,  ,/        \  ./' __:..
-                |  __ __  |       |  | .--.  ,         >  >   |-'   /     `
-              ,/| /  '  \ |       |  |     \      ,           |    /
-             /  |<--.__,->|       |  | .    `.        >  >    /   (
-            /_,' \\  ^  /  \     /  /   `.    >--            /^\   |
-                  \\___/    \   /  /      \__'     \   \   \/   \  |
-                   `.   |/          ,  ,                  /`\    \  )
-                     \  '  |/    ,       V    \          /        `-\
- OpenBSD Puffy --->   `|/  '  V      V           \    \.'            \_
-                       '`-.       V       V        \./'\
-                           `|/-.      \ /   \ /,---`\         kat
-                            /   `._____V_____V'
-                                       '     '
-
    -
    -

    Table of Contents


    -
    -
    -

    Compile it


    -
    -First of all, DTail needs to be downloaded and compiled. For that, git, go, and gmake are required:
    -
    -
    -$ doas pkg_add git go gmake
-
    -
    -I am happy that the Go Programming Language is readily available in the OpenBSD packaging system. Once the dependencies got installed, clone DTail and compile it:
    -
    -
    -$ mkdir git
-$ cd git
-$ git clone https://github.com/mimecast/dtail
-$ cd dtail
-$ gmake 
-
    -
    -You can verify the version by running the following command:
    -
    -
    -$ ./dtail --version
- DTail  4.1.0  Protocol 4.1  Have a lot of fun!
-$ file dtail
- dtail: ELF 64-bit LSB executable, x86-64, version 1
-
    -
    -Now, there isn't any need anymore to keep git, go and gmake, so they can be deinstalled now:
    -
    -
    -$ doas pkg_delete git go gmake
-
    -
    -One day I shall create an official OpenBSD port for DTail.
    -
    -

    Install it


    -
    -Installing the binaries is now just a matter of copying them to /usr/local/bin as follows:
    -
    -
    -$ for bin in dserver dcat dgrep dmap dtail dtailhealth; do
-  doas cp -p $bin /usr/local/bin/$bin
-  doas chown root:wheel /usr/local/bin/$bin
-done
-
    -
    -Also, we will be creating the _dserver service user:
    -
    -
    -$ doas adduser -class nologin -group _dserver -batch _dserver
-$ doas usermod -d /var/run/dserver/ _dserver
-
    -
    -The OpenBSD init script is created from scratch (not part of the official DTail project). Run the following to install the bespoke script:
    -
    -
    -$ cat <<'END' | doas tee /etc/rc.d/dserver
-#!/bin/ksh
-
-daemon="/usr/local/bin/dserver"
-daemon_flags="-cfg /etc/dserver/dtail.json"
-daemon_user="_dserver"
-
-. /etc/rc.d/rc.subr
-
-rc_reload=NO
-
-rc_pre() {
-    install -d -o _dserver /var/log/dserver
-    install -d -o _dserver /var/run/dserver/cache
-}
-
-rc_cmd $1 &
-END
-$ doas chmod 755 /etc/rc.d/dserver
-
    -
    -

    Rexification


    -
    -This is the task for setting it up via Rex. Note the . . . ., that's a placeholder which we will fill up more and more during this blog post:
    -
    -
    -desc 'Setup DTail';
-task 'dtail', group => 'frontends',
-   sub {
-      my $restart = FALSE;
-
-      file '/etc/rc.d/dserver':
-        content => template('./etc/rc.d/dserver.tpl'),
-        owner => 'root',
-        group => 'wheel',
-        mode => '755',
-        on_change => sub { $restart = TRUE };
-
-        .
-        .
-        .
-        .
-
-      service 'dserver' => 'restart' if $restart;
-      service 'dserver', ensure => 'started';
-   };
-
    -
    -

    Configure it


    -
    -Now, DTail is fully installed but still needs to be configured. Grab the default config file from GitHub ...
    -
    -
    -$ doas mkdir /etc/dserver
-$ curl https://raw.githubusercontent.com/mimecast/dtail/master/examples/dtail.json.examples |
-    doas tee /etc/dserver/dtail.json
-
    -
    -... and then edit it and adjust LogDir in the Common section to /var/log/dserver. The result will look like this:
    -
    -
    -  "Common": {
-    "LogDir": "/var/log/dserver",
-    "Logger": "Fout",
-    "LogRotation": "Daily",
-    "CacheDir": "cache",
-    "SSHPort": 2222,
-    "LogLevel": "Info"
-  }
-
    -
    -

    Rexification


    -
    -That's as simple as adding the following to the Rex task:
    -
    -
    -file '/etc/dserver',
-  ensure => 'directory';
-
-file '/etc/dserver/dtail.json',
-  content => template('./etc/dserver/dtail.json.tpl'),
-  owner => 'root',
-  group => 'wheel',
-  mode => '755',
-  on_change => sub { $restart = TRUE };
-
    -
    -

    Update the key cache for it


    -
    -DTail relies on SSH for secure authentication and communication. However, the system user _dserver has no permission to read the SSH public keys from the user's home directories, so the DTail server also checks for available public keys in an alternative path /var/run/dserver/cache.
    -
    -The following script, populating the DTail server key cache, can be run periodically via CRON:
    -
    -
    -$ cat <<'END' | doas tee /usr/local/bin/dserver-update-key-cache.sh
-#!/bin/ksh
-
-CACHEDIR=/var/run/dserver/cache
-DSERVER_USER=_dserver
-DSERVER_GROUP=_dserver
-
-echo 'Updating SSH key cache'
-
-ls /home/ | while read remoteuser; do
-    keysfile=/home/$remoteuser/.ssh/authorized_keys
-
-    if [ -f $keysfile ]; then
-        cachefile=$CACHEDIR/$remoteuser.authorized_keys
-        echo "Caching $keysfile -> $cachefile"
-
-        cp $keysfile $cachefile
-        chown $DSERVER_USER:$DSERVER_GROUP $cachefile
-        chmod 600 $cachefile
-    fi
-done
-
-# Cleanup obsolete public SSH keys
-find $CACHEDIR -name \*.authorized_keys -type f |
-while read cachefile; do
-    remoteuser=$(basename $cachefile | cut -d. -f1)
-    keysfile=/home/$remoteuser/.ssh/authorized_keys
-
-    if [ ! -f $keysfile ]; then
-        echo 'Deleting obsolete cache file $cachefile'
-        rm $cachefile
-    fi
-done
-
-echo 'All set...'
-END
-$ doas chmod 500 /usr/local/bin/dserver-update-key-cache.sh
-
    -
    -Note that the script above is a slight variation of the official DTail script. The official DTail one is a bash script, but on OpenBSD, there's ksh. I run it once daily by adding it to the daily.local:
    -
    -
    -$ echo /usr/local/bin/dserver-update-key-cache.sh | doas tee -a /etc/daily.local
-/usr/local/bin/dserver-update-key-cache.sh
-
    -
    -

    Rexification


    -
    -That's done by adding ...
    -
    -
    -file '/usr/local/bin/dserver-update-key-cache.sh',
-  content => template('./scripts/dserver-update-key-cache.sh.tpl'),
-  owner => 'root',
-  group => 'wheel',
-  mode => '500';
-
-append_if_no_such_line '/etc/daily.local', '/usr/local/bin/dserver-update-key-cache.sh';
-
    -
    -... to the Rex task!
    -
    -

    Start it


    -
    -Now, it's time to enable and start the DTail server:
    -
    -
    -$ sudo rcctl enable dserver
-$ sudo rcctl start dserver
-$ tail -f /var/log/dserver/*.log
-INFO|1022-090634|Starting scheduled job runner after 2s
-INFO|1022-090634|Starting continuous job runner after 2s
-INFO|1022-090644|24204|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
-INFO|1022-090654|24204|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
-INFO|1022-090719|Starting server|DTail 4.1.0 Protocol 4.1 Have a lot of fun!
-INFO|1022-090719|Generating private server RSA host key
-INFO|1022-090719|Starting server
-INFO|1022-090719|Binding server|0.0.0.0:2222
-INFO|1022-090719|Starting scheduled job runner after 2s
-INFO|1022-090719|Starting continuous job runner after 2s
-INFO|1022-090729|86050|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnections=0
-INFO|1022-090739|86050|stats.go:53|2|11|7|||MAPREDUCE:STATS|currentConnections=0|lifetimeConnect
-.
-.
-.
-Ctr+C
-
    -
    -As we don't want to wait until tomorrow, let's populate the key cache manually:
    -
    -
    -$ doas /usr/local/bin/dserver-update-key-cache.sh
-Updating SSH key cache
-Caching /home/_dserver/.ssh/authorized_keys -> /var/cache/dserver/_dserver.authorized_keys
-Caching /home/admin/.ssh/authorized_keys -> /var/cache/dserver/admin.authorized_keys
-Caching /home/failunderd/.ssh/authorized_keys -> /var/cache/dserver/failunderd.authorized_keys
-Caching /home/git/.ssh/authorized_keys -> /var/cache/dserver/git.authorized_keys
-Caching /home/paul/.ssh/authorized_keys -> /var/cache/dserver/paul.authorized_keys
-Caching /home/rex/.ssh/authorized_keys -> /var/cache/dserver/rex.authorized_keys
-All set...
-
    -
    -

    Use it


    -
    -The DTail server is now ready to serve connections. You can use any DTail commands, such as dtail, dgrep, dmap, dcat, dtailhealth, to do so. Checkout out all the usage examples on the official DTail page.
    -
    -I have installed DTail server this way on my personal OpenBSD frontends blowfish, and fishfinger, and the following command connects as user rex to both machines and greps the file /etc/fstab for the string local:
    -
    -
    -❯ ./dgrep -user rex -servers blowfish.buetow.org,fishfinger.buetow.org --regex local /etc/fstab
-CLIENT|earth|WARN|Encountered unknown host|{blowfish.buetow.org:2222 0xc0000a00f0 0xc0000a61e0 [blowfish.buetow.org]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9ZnF/LAk14SgqCzk38yENVTNfqibcluMTuKx1u53cKSp2xwHWzy0Ni5smFPpJDIQQljQEJl14ZdXvhhjp1kKHxJ79ubqRtIXBlC0PhlnP8Kd+mVLLHYpH9VO4rnaSfHE1kBjWkI7U6lLc6ks4flgAgGTS5Bb7pLAjwdWg794GWcnRh6kSUEQd3SftANqQLgCunDcP2Vc4KR9R78zBmEzXH/OPzl/ANgNA6wWO2OoKKy2VrjwVAab6FW15h3Lr6rYIw3KztpG+UMmEj5ReexIjXi/jUptdnUFWspvAmzIl6kwzzF8ExVyT9D75JRuHvmxXKKjyJRxqb8UnSh2JD4JN [23.88.35.144]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9ZnF/LAk14SgqCzk38yENVTNfqibcluMTuKx1u53cKSp2xwHWzy0Ni5smFPpJDIQQljQEJl14ZdXvhhjp1kKHxJ79ubqRtIXBlC0PhlnP8Kd+mVLLHYpH9VO4rnaSfHE1kBjWkI7U6lLc6ks4flgAgGTS5Bb7pLAjwdWg794GWcnRh6kSUEQd3SftANqQLgCunDcP2Vc4KR9R78zBmEzXH/OPzl/ANgNA6wWO2OoKKy2VrjwVAab6FW15h3Lr6rYIw3KztpG+UMmEj5ReexIjXi/jUptdnUFWspvAmzIl6kwzzF8ExVyT9D75JRuHvmxXKKjyJRxqb8UnSh2JD4JN 0xc0000a2180}
-CLIENT|earth|WARN|Encountered unknown host|{fishfinger.buetow.org:2222 0xc0000a0150 0xc000460110 [fishfinger.buetow.org]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNiikdL7+tWSN0rCaw1tOd9aQgeUFgb830V9ejkyJ5h93PKLCWZSMMCtiabc1aUeUZR//rZjcPHFLuLq/YC+Y3naYtGd6j8qVrcfG8jy3gCbs4tV9SZ9qd5E24mtYqYdGlee6JN6kEWhJxFkEwPfNlG+YAr3KC8lvEAE2JdWvaZavqsqMvHZtAX3b25WCBf2HGkyLZ+d9cnimRUOt+/+353BQFCEct/2mhMVlkr4I23CY6Tsufx0vtxx25nbFdZias6wmhxaE9p3LiWXygPWGU5iZ4RSQSImQz4zyOc9rnJeP1rwGk0OWDJhdKNXuf0kIPdzMfwxv2otgY32/DJj6L [46.23.94.99]:2222 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNiikdL7+tWSN0rCaw1tOd9aQgeUFgb830V9ejkyJ5h93PKLCWZSMMCtiabc1aUeUZR//rZjcPHFLuLq/YC+Y3naYtGd6j8qVrcfG8jy3gCbs4tV9SZ9qd5E24mtYqYdGlee6JN6kEWhJxFkEwPfNlG+YAr3KC8lvEAE2JdWvaZavqsqMvHZtAX3b25WCBf2HGkyLZ+d9cnimRUOt+/+353BQFCEct/2mhMVlkr4I23CY6Tsufx0vtxx25nbFdZias6wmhxaE9p3LiWXygPWGU5iZ4RSQSImQz4zyOc9rnJeP1rwGk0OWDJhdKNXuf0kIPdzMfwxv2otgY32/DJj6L 0xc0000a2240}
-Encountered 2 unknown hosts: 'blowfish.buetow.org:2222,fishfinger.buetow.org:2222'
-Do you want to trust these hosts?? (y=yes,a=all,n=no,d=details): a
-CLIENT|earth|INFO|STATS:STATS|cgocalls=11|cpu=8|connected=2|servers=2|connected%=100|new=2|throttle=0|goroutines=19
-CLIENT|earth|INFO|Added hosts to known hosts file|/home/paul/.ssh/known_hosts
-REMOTE|blowfish|100|7|fstab|31bfd9d9a6788844.h /usr/local ffs rw,wxallowed,nodev 1 2
-REMOTE|fishfinger|100|7|fstab|093f510ec5c0f512.h /usr/local ffs rw,wxallowed,nodev 1 2
-
    -
    -Running it the second time, and given that you trusted the keys the first time, it won't prompt you for the host keys anymore:
    -
    -
    -❯ ./dgrep -user rex -servers blowfish.buetow.org,fishfinger.buetow.org --regex local /etc/fstab
-REMOTE|blowfish|100|7|fstab|31bfd9d9a6788844.h /usr/local ffs rw,wxallowed,nodev 1 2
-REMOTE|fishfinger|100|7|fstab|093f510ec5c0f512.h /usr/local ffs rw,wxallowed,nodev 1 2
-
    -
    -

    Conclusions


    -
    -It's a bit of manual work, but it's ok on this small scale! I shall invest time in creating an official OpenBSD port, though. That would render most of the manual steps obsolete, as outlined in this post!
    -
    -Check out the following for more information:
    -
    -https://dtail.dev
    -https://github.com/mimecast/dtail
    -https://www.rexify.org
    -
    -E-Mail your comments to paul@nospam.buetow.org :-)
    -
    -Other related posts are:
    -
    -2023-09-25 DTail usage examples
    -2022-10-30 Installing DTail on OpenBSD (You are currently reading this)
    -2022-03-06 The release of DTail 4.0.0
    -2021-04-22 DTail - The distributed log tail program
    -
    Back to the main site
     -- cgit v1.2.3