f3s: Kubernetes with FreeBSD - Setting the stage - Part 1

+
+This is the first blog post about my f3s series for my self-hosting demands in my home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I will use on FreeBSD-based physical machines.
+
+I will post a new entry every month or so (there are too many other side projects for more frequent updates—I bet you can understand).
+
+2024-11-17 f3s: Kubernetes with FreeBSD - Setting the stage - Part 1 (You are currently reading this)
+
+

+
+Let's begin...
+
+

+
+

f3s: Kubernetes with FreeBSD - Setting the stage - Part 1
⇢ Why this setup?
⇢ The infrastructure
⇢ ⇢ Physical FreeBSD nodes and Linux VMs
⇢ ⇢ Kubernetes with k3s
⇢ ⇢ HA volumes for k3s with HAST/ZFS and NFS
⇢ ⇢ OpenBSD/relayd to the rescue for external connectivity
⇢ Data integrity
⇢ ⇢ Periodic backups
⇢ ⇢ Power protection
⇢ Monitoring: Keeping an Eye on Everything
⇢ ⇢ Prometheus and Grafana
⇢ ⇢ Gogios: My Custom Alerting System
⇢ What's after this all?

Why this setup?

+
+Look at my previous setup, which was great to learn Terraform and AWS, but the setup is too expensive. Costs are under control there, but only because I am shutting down all containers after use (so they are offline ninety per cent of the time and still cost around 20 bucks monthly). With the new setup, I could run all containers 24/7 at home, which would still be cheaper for electricity consumption.
+
+From babylon5.buetow.org to .cloud
+
+Migrating off all my containers from AWS ECS means I need a reliable and scalable environment to host my workloads. I wanted something:
+
+

To self-host all my open-source apps (Docker containers).
Fully under my control (goodbye cloud vendor lock-in).
Secure and redundant.
Cost-efficient (after the initial hardware investment).
Something I can poke around with and also pick up new skills.

The infrastructure

+
+This is still in progress, and I need to own the hardware. But in this first part of the blog series, I will outline what I intend to do.
+
+

+
+

Physical FreeBSD nodes and Linux VMs

+
+The setup starts with three physical FreeBSD nodes. On these, I'm running Rocky Linux virtual machines with bhyve. Why Linux VMs in FreeBSD and not Linux directly? I want to leverage the great ZFS integration in FreeBSD (among other features), and I have been using FreeBSD for a while in my home lab. And with bhyve, there is a very performant hypervisor available which makes the Linux VMs de-facto run at native speed (another use case of mine would be maybe running a Windows bhyve VM on one of the nodes - but out of scope for this blog series).
+
+https://www.freebsd.org/
+https://wiki.freebsd.org/bhyve
+
+I selected Rocky Linux because it comes with long-term support (I don't want to upgrade the VMs every 6 months). Rocky Linux 9 will reach its end of life in 2032, which is plenty of time! Of course, there will be minor upgrades, but nothing will significantly break my setup.
+
+https://rockylinux.org/
+https://wiki.rockylinux.org/rocky/version/
+
+Furthermore, I am already using "RHEL-family" related distros at work and Fedora on my main personal laptop. Rocky Linux belongs to the same type of Linux distribution family, so I already feel at home here. I also used Rocky 9 before I switched to AWS ECS. Now, I am switching back in one sense or another ;-)
+
+

Kubernetes with k3s

+
+These Linux VMs form a three-node k3s Kubernetes cluster, where my containers will reside moving forward. The 3-node k3s cluster will be highly available (in etcd mode), and all apps will probably be deployed with Helm. Prometheus will also be running in k3s, collecting time-series metrics and handling monitoring. Additionally, a private Docker registry will be deployed into the k3s cluster, where I will store some of my self-created Docker images. k3s is the perfect distribution of Kubernetes for homelabbers due to its simplicity and the inclusion of the most useful features out of the box!
+
+https://k3s.io/
+
+

HA volumes for k3s with HAST/ZFS and NFS

+
+Persistent storage for the k3s cluster will be handled by highly available (HA) NFS shares backed by ZFS on the FreeBSD hosts.
+
+On two of the three physical FreeBSD nodes, I will add a second SSD drive to each and dedicate it to a pool ZFS pool. With HAST (FreeBSD's solution for highly available storage), this pool will be replicated at the byte level to a standby node.
+
+A virtual IP (VIP) will point to the master node. When the master node goes down, the VIP will failover to the standby node, where the ZFS pool will be mounted. An NFS server will listen to both nodes. k3s will use the VIP to access the NFS shares.
+
+https://wiki.freebsd.org/HighlyAvailableStorage
+
+

OpenBSD/relayd to the rescue for external connectivity

+
+All apps should be reachable through the internet (e.g., from my phone or computer when travelling). For external connectivity and TLS management, I've got two OpenBSD VMs (one hosted by OpenBSD Amsterdam and another hosted by Hetzner) handling public-facing services like DNS, relaying traffic, and automating Let's Encrypt certificates.
+
+All of this (every Linux VM to every OpenBSD box) will be connected via WireGuard tunnels, keeping everything private and secure. There will be 6 WireGuard tunnels (3 k3s nodes times two OpenBSD VMs).
+
+https://en.wikipedia.org/wiki/WireGuard
+
+So, when I want to access a service running in k3s, I will hit an external DNS endpoint (with the authoritative DNS servers being the OpenBSD boxes). The DNS will resolve to the master OpenBSD VM (see my KISS highly-available with OpenBSD blog post), and from there, the relayd process (with a Let's Encrypt certificate—see my Let's Encrypt with OpenBSD and Rex blog post) will accept the TCP connection and forward it through the WireGuard tunnel to a reachable node port of one of the k3s nodes, thus serving the traffic.
+
+KISS high-availability with OpenBSD
+Le's Encrypt with OpenBSD and Rex
+
+The OpenBSD setup described here already exists and is ready to use. The only thing that does not yet exist is the configuration of relayd to forward requests to k3s through the WireGuard tunnel(s).
+
+

Data integrity

+
+

Periodic backups

+
+Let's face it, backups are non-negotiable.
+
+On the HAST master node, incremental and encrypted ZFS snapshots are created daily and automatically backed up to AWS S3 Glacier Deep Archive via CRON. I have a bunch of scripts already available, which I currently use for a similar purpose on my FreeBSD Home NAS server (an old ThinkPad T440 with an external USB drive enclosure, which I will eventually retire when the HAST setup is ready). I will copy them and slightly modify them to fit the purpose.
+
+https://www.freshports.org/sysutils/zfstools
+
+The backup scripts also perform some zpool scrubbing now and then. A scrub once in a while keeps the trouble away.
+
+

Power protection

+
+Power outages are regularly in my area, so a UPS keeps the infrastructure running during short outages and protects the hardware. I'm still trying to decide which hardware to get, and I still need one, as my previous NAS is simply an older laptop that already has a battery for power outages. However, there are plenty of options to choose from. My main criterion is that the UPS should be silent, as the whole setup will be installed in an upper shelf unit in my daughter's room. ;-)
+
+

Monitoring: Keeping an Eye on Everything

+
+Robust monitoring is vital to any infrastructure, especially one as distributed as mine. I've thought about a setup that ensures I'll always be aware of what's happening in my environment.
+
+

Prometheus and Grafana

+
+Inside the k3s cluster, Prometheus will be deployed to handle metrics collection. It will be configured to scrape data from my Kubernetes workloads, nodes, and any services I monitor. Prometheus also integrates with Alertmanager to generate alerts based on predefined thresholds or conditions.
+
+https://prometheus.io
+
+For visualization, Grafana will be deployed alongside Prometheus. Grafana lets me build dynamic, customizable dashboards that provide a real-time view of everything from resource utilization to application performance. Whether it's keeping track of CPU load, memory usage, or the health of Kubernetes pods, Grafana has it covered. This will also make troubleshooting easier, as I can quickly pinpoint where issues are arising.
+
+https://grafana.com
+
+

Gogios: My Custom Alerting System

+
+Alerts generated by Prometheus are forwarded to Alertmanager, which I will configure to work with Gogios, a lightweight monitoring and alerting system I wrote myself. Gogios runs on one of my OpenBSD VMs. At regular intervals, Gogios scrapes the alerts generated in the k3s cluster and notifies me via Email.
+
+KISS server monitoring with Gogios
+
+Ironically, I implemented Gogios to avoid using more complex alerting systems like Prometheus, but here we go—it integrates well now.
+
+

What's after this all?

+
+This setup is just the beginning. Some ideas I'm thinking about for the future:
+
+

Adding more FreeBSD nodes (in different physical locations, maybe at my wider family's places?) for better redundancy. (HA storage then might be trickier)
Deploying more Docker apps (data-intensive ones, like a picture gallery, my entire audiobook catalogue, or even a music server) to k3s.

+For now, though, I'm focused on completing the migration from AWS ECS and getting all my Docker containers running smoothly in k3s.
+
+What's your take on self-hosting? Are you planning to move away from managed cloud services? Stay tuned for the second part of this series, where I will likely write about the hardware and the OS setups.
+
+Other *BSD-related posts:
+
+2016-04-09 Jails and ZFS with Puppet on FreeBSD
+2022-07-30 Let's Encrypt with OpenBSD and Rex
+2022-10-30 Installing DTail on OpenBSD
+2024-01-13 One reason why I love OpenBSD
+2024-04-01 KISS high-availability with OpenBSD
+2024-11-17 f3s: Kubernetes with FreeBSD - Setting the stage - Part 1 (You are currently reading this)
+
+E-Mail your comments to paul@nospam.buetow.org :-)
+
+Back to the main site
+

Computer operating systems I use(d)

-
-Published at 2022-02-04T09:58:22+00:00; Updated at 2022-02-18
-
-This is a list of Operating Systems I currently use. This list is in no particular order and also will be updated over time. The very first operating system I used was MS-DOS (mainly for games) and the very first Unix like operating system I used was SuSE Linux 5.3. My first smartphone OS was Symbian on a clunky Sony Ericsson device.
-
-

-              /(        )`
-              \ \___   / |
-              /- _  `-/  '
-             (/\/ \ \   /\
-             / /   | `    \
-             O O   ) /    |
-             `-^--'`<     '
-            (_.)  _  )   /
-             `.___/`    /
-               `-----' /
-  <----.     __ / __   \
-  <----|====O)))==) \) /====
-  <----'    `--' `.__,' \
-               |        |
-                \       /
-           ______( (_  / \______
-  (FL)   ,'  ,-----'   |        \
-         `--{__________)        \/   "Berkeley Unix Daemon"
-

-
-

Computer operating systems I use(d)
⇢ Fedora Linux
⇢ EndeavourOS
⇢ FreeBSD
⇢ CentOS 7
⇢ OpenBSD
⇢ macOS (proprietary)
⇢ LineageOS (mobile)
⇢ Samsung's Stock Android (mobile proprietary)
⇢ iOS (mobile proprietary)
⇢ Other OSes
⇢ ⇢ InfinyTime (smartwatch)
⇢ ⇢ motionEyeOS
⇢ ⇢ Kobo OS (proprietary)
⇢ ⇢ Android TV (proprietary)
⇢ Other OSes..
⇢ ⇢ NetBSD
⇢ ⇢ Other OSes in use...
⇢ ⇢ Other OSes not used any more...
⇢ ⇢ Other OSes I only had a glance at...
⇢ ⇢ Other OSes which seem interesting...

Fedora Linux

-
-Fedora Linux is the operating system I use on my primary (personal) laptop. It's a ThinkPad X1 Carbon Gen. 9. Lenovo which comes along with official Lenovo Linux support. I already noticed hardware firmware updates being installed directly through Fedora from Lenovo. Fedora is a real powerhouse, cutting-edge and reasonably stable at the same time. It's baked by Red Hat.
-
-I also use Fedora on my Microsoft Surface Go 2 convertible tablet. Fedora works quite OK (and much better than Windows) on this device. It's also the perfect travel companion.
-
-I use the GNOME Desktop on my Fedora boxes. I have memorized and customized a bunch of keyboard shortcuts. But the fact that I mostly work in the terminal (with tmux) makes the Desktop environment I use only secondary.
-
-

EndeavourOS

-
-I installed EndeavourOS on my (older) ThinkPad X240 to try out an Arch based Linux distribution. I also could have installed plain Arch, but I don't see the point when there is EndeavourOS. EndeavourOS is as close as you can get to the plain Arch experience but with an easy installer. I am not saying that it's difficult to install plain Arch but it's, unless you are new to Linux and want to learn about the installation procedure, just waste of time in my humble opinion. Give Linux From Scratch a shot instead if you really want to learn about Linux.
-
-https://www.linuxfromscratch.org/
-
-On EndeavourOS, I use the Xfce desktop environment which feels very snappy and fast on the X240 (which I purchased back in 2014). Usually, I have my X240 standing right next to my work laptop and use it for playing music (mainly online radio streams), for personal note taking and occasional emailing and instant messaging.
-
-As this is a rolling Linux distribution there are a lot of software updates coming through every day. Sometimes, it only takes a minute until the next version of a package is available. Honestly, I find that a bit annoying to constantly catch up with all the updates. As for now I will live with it and/or automate it a bit more. It'll be OK if it breaks occasionally, as this is not my primary laptop anyway.
-
-Arch Linux and EndeavourOS are community distributions. This means, that there is no big corporation in the backyard lurking around. They won't give you the firmware updates for cutting edge hardware out of the box, though, but they are still a very good choice for hobbyist and also for older hardware where future firmware updates are less likely to happen.
-
-I am very happy with the package availability through the official repository and AUR.
-
-https://endeavouros.com/
-
-

FreeBSD

-
-I have run FreeBSD in many occasions. Right after SuSE Linux, FreeBSD (around 4.x) was the second open source system I used in my life on regular basis. I didn't even go to university yet then I started using it :-). Also, a former employer of mine even allowed me to install FreeBSD on my main workstation (which I actually did and used it for a couple of years).
-
-I remember it used to be a pain bootstrapping Java for FreeBSD due to the lack of pre-compiled binary packages. You had first to enable the Linux compatibility layer, then install Linux Java, and then compile FreeBSD Java with the bootstrapped Linux Java (yes, Java is mainly programmed in C++, but for some reason compiling Java for FreeBSD also required an installation of Java). Nowadays, there are ready OpenJDK binary packages you could install. So things have improved a lot since.
-
-FreeBSD always had a place somewhere in my life:
-
-

On a Desktop PC (personal and work)
On a Laptop
On a webserver, FTP server, DNS server, mail server
On a server offering FreeBSD jails to customers for rent
As an experiment running Debian GNU/kFreeBSD inside of jails

-Debian GNU/kFreeBSD is now dead (same is my experiment)...
-
-https://www.debian.org/ports/kfreebsd-gnu/
-
-...but I still have saved and old uname output :-):
-
-

-[root@saturn /usr/jail/serv14/etc] # jexec 21 bash
-root@rhea:/ # uname -a
-GNU/kFreeBSD rhea.buetow.org 8.0-RELEASE-p5 FreeBSD 8.0-RELEASE-p5 #2: Sat Nov 27 13:10:09 CET
-  2010 root@saturn.buetow.org:/usr/obj/usr/srcs/freebsd.src8/src/sys/SERV10 x86 64 amd64 Intel(R)
-  Core(TM) i7 CPU 920 @ 2.67GHz GNU/kFreeBSD
-

-
-Currently, I use FreeBSD on my personal NAS server. The server is a regular PC with a bunch of hard drives and a ZFS RAIDZ (with 4x2TB drives) + a couple of external backup drives.
-
-https://www.FreeBSD.org
-
-

CentOS 7

-
-While CentOS 8 is already out of support, I still use CentOS 7 (which will receive security updates until 2024). CentOS 7 runs in a cloud VM and is the home to my personal NextCloud and Wallabag installations. You probably know already NextCloud. About Wallabag: It is a great free and open source alternative to Pocket (for reading articles from the web offline later). Yes, you can pay for a Wallabag subscription, but you can also host it for free on your own server.
-
-NextCloud
-Wallabag
-
-The reason I use Linux and not *BSD at the moment for these services is Docker. With Docker, it's so easy-peasy to get these up and running. I will have to switch to another OS before CentOS 7 runs out of support, though. It might be CentOS Stream, Rocky Linux, or, more likely, I will use FreeBSD. On FreeBSD there isn't Docker, but what can be done is to create a self-contained Jail for each of the web-apps.
-
-I have been using FreeBSD Jails for LAMP stacks before I started using CentOS. The reason why I switched to CentOS (it was still CentOS 6 at that time) in the first place was, that I wanted to try out something new.
-
-https://www.centos.org
-
-

OpenBSD

-
-I use two small OpenBSD "cloud" boxes for my "public facing internet front-ends". The services I run here are:
-
-

HTTP server (serving this site via https://foo.zone)
Gemini server (serving this site via https://foo.zone)
MTA server (for receiving E-Mails to my hosts)
Authorative DNS server (for all of my "domains")
Some personal/private git repositories (accessible only via SSH)

-OpenBSD is a complete operating system. I love it due to it's "simplicity" and "correctness" and the good documentation (I love the manual pages in particular). OpenBSD is also known for its innovations in security. I must admin, though, that most Unix like operating system would be secure enough for my personal needs and that I don't really need to use OpenBSD here. But nevertheless, I think it's the ideal operating system for what I am using it for.
-
-The only softwares which were not part of the base system and I had to install additionally were the Gemini server (vger) and Git, which both were available as pre-compiled OpenBSD binary packages. So, besides of these two packages, it is indeed a pretty complete operating system for my use case.
-
-https://www.openbsd.org
-
-

macOS (proprietary)

-
-I have to use a MacBook Pro with macOS for work. What else can I say but that this would have never been my personal choice. At least macOS is a UNIX under the hood and comes with a decent terminal and there are plenty of terminal apps available via Brew. Some of the inner workings of macOS were actually forked from the FreeBSD project.
-
-developer.apple.com: BSD in macOS/Darwin
-
-I find the macOS UI rather confusing.
-
-

LineageOS (mobile)

-
-At some point I got fed up with big tech, like Google and Samsung (or Apple, but personally I don't use Apple), spying on me. So I purchased a Google phone (a midrange Pixel phone) and installed LineageOS, a free and open source distribution of Android, on it. I don't have anything from Google installed on it (not even the play store, I install my apps from F-Droid). It's my daily driver since mid 2021 now.
-
-So far the experience is not great but good. The main culprits are not having Google Maps, Google Gboard and the camera app. The latter lacks some features on LineageOS (e.g. No wide angle lens support). Also, I can't use my banking apps anymore. Sometimes apps crash for no apparent reason(s) but I get around it so far. I shouldn't spend so much time on my smartphone anyway! And the whole point of switching to LineageOS was to get away of big tech and therefore I should not complain :-). What I do like is that 95% the things I used to do on a proprietary mobile phone also can be done with LineageOS.
-
-Read also "The Midle Way" section of this blog post regarding smartphones.
-
-There's also the excellent Termux app in the F-Droid store, which transforms the phone into a small Linux handheld device. I am able to run all of my Linux/Unix terminal apps with it.
-
-https://lineageos.org/
-https://termux.com/
-
-

Samsung's Stock Android (mobile proprietary)

-
-Unfortunatley, I still have to keep my proprietary Android phone around. Sometimes, I really need to use some proprietary apps which are only available form the Google play store and also require the Google services installed on the phone. I don't carry this phone around all the time and I only use it intentionally for very specific use cases. I think this is the best compromise I can make.
-
-

iOS (mobile proprietary)

-
-I have to use an iPhone for work. I like the hardware but I hate the OS (you can also call it spyOS), but it's the necessarries evil, unfortunately. Apple is even worse than Google here (despite claiming for themselves to produce the most secure phone(s)). I don't have it with me all the time or switched off when I don't need it. I also find iOS quite unintuitive to use.
-
-Being on-call for work means to to be reachable 24/7. This implies that the phone is carried around all the time (in an switched-on state). 1984 is now.
-https://en.wikipedia.org/wiki/Nineteen_Eighty-Four
-
-

Other OSes

-
-

InfinyTime (smartwatch)

-
-I use it on my PineTime smartwatch. Other than checking the time and my step count, I really don't do anything else fancy with it (yet).
-
-https://www.pine64.org/pinetime/
-https://infinitime.io/
-
-

motionEyeOS

-
-I usually install an army of RaspberryPi 3's in my house before I travel for a prolonged amount of time. All Pi's are equipped with an camera and have motionEyeOS (Linux based video surveillance system) installed. There's a neat Android app in the F-Droid store which let's me keep an eye on everything. I make the Pi's accessible from the internet via reverse SSH tunnels through one of my frontend servers.
-
-https://github.com/ccrisan/motioneyeos
-
-

Kobo OS (proprietary)

-
-I use a Kobo Forma as my e-reader device. I have started to switch off the Wifi and to only sideload DRM free ePubs on it. Even offline, it's a fully capable reader device. I wouldn't like the Kobo to call home to Rakuten. I would love to replace it one day with an open source e-reader alternative like the PineNote. There are also some interesting attempts installing postmarketOS Linux on Kobo devices. The latter boots already, but is far from being usable as a normal e-reader.
-
-The PineNote
-Kobo Clara HD becomes an e-ink Linux tablet
-
-But as a fall-back, someone could still use the good old dead tree format!
-
-

Android TV (proprietary)

-
-An Android TV box is used for watching movies and series on Netflix and Amazon Prime video (yes, I am human too and rely once in a while on big tech streaming services). The Android TV box is currently in the process of being replaced by OSMC, though. Most services seem to work fine with OSMC, but didn't get around tinkering with Netflix and Amazon there yet.
-
-https://osmc.tv/
-
-

Other OSes..

-
-This section is just for the sake of having a complete list of all OSes I used for some significant amount of time. I might not use all of them any more...
-
-

NetBSD

-
-I have been using NetBSD on an old Sun Sparcstation 10 as a student. I also have run NetBSD on a very old ThinkPad with 96MB!!! of RAM (even with X/evilWM). I also installed (but never really used) NetBSD on an HP Jornada 680. But that's all more than 10 years ago. I haven't looked at NetBSD for long time. I want to revive it on an "old" ThinkPad T450 of mine which I currently don't use.
-
-https://netbsd.org
-
-

Technical references

Technical guides

Podcasts

Podcasts I liked

Newsletters I like

Formal education

f3s: Kubernetes with FreeBSD - Setting the stage - Part 1

Table of Contents

Why this setup?

The infrastructure

Physical FreeBSD nodes and Linux VMs

Kubernetes with k3s

HA volumes for k3s with HAST/ZFS and NFS

OpenBSD/relayd to the rescue for external connectivity

Data integrity

Periodic backups

Power protection

Monitoring: Keeping an Eye on Everything

Prometheus and Grafana

Gogios: My Custom Alerting System

What's after this all?

f3s: Kubernetes with FreeBSD - Setting the stage - Part 1

Table of Contents

Why this setup?

The infrastructure

Physical FreeBSD nodes and Linux VMs

Kubernetes with k3s

HA volumes for k3s with HAST/ZFS and NFS

OpenBSD/relayd to the rescue for external connectivity

Data integrity

Periodic backups

Power protection

Monitoring: Keeping an Eye on Everything

Prometheus and Grafana

Gogios: My Custom Alerting System

What's after this all?

Computer operating systems I use(d)

Table of Contents

Fedora Linux

EndeavourOS

FreeBSD

CentOS 7

OpenBSD

macOS (proprietary)

LineageOS (mobile)

Samsung's Stock Android (mobile proprietary)

iOS (mobile proprietary)

Other OSes

InfinyTime (smartwatch)

motionEyeOS

Kobo OS (proprietary)

Android TV (proprietary)

Other OSes..

NetBSD

Other OSes in use...

Other OSes not used any more...

Other OSes I only had a glance at...

Other OSes which seem interesting...

To be in the .zone!

foo.zone

Posts

My machine uptime stats