From 157c9b2080a3f41eea0eeba11f6ef307f40e9b9e Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Thu, 2 Oct 2025 11:31:40 +0300 Subject: Update content for gemtext --- about/resources.gmi | 198 +-- gemfeed/.gitignore | 1 - ...-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi | 1 + ...022-07-30-lets-encrypt-with-openbsd-and-rex.gmi | 1 + .../2024-01-13-one-reason-why-i-love-openbsd.gmi | 1 + ...4-04-01-KISS-high-availability-with-OpenBSD.gmi | 1 + ...24-11-17-f3s-kubernetes-with-freebsd-part-1.gmi | 2 + ...24-12-03-f3s-kubernetes-with-freebsd-part-2.gmi | 5 +- ...2-03-f3s-kubernetes-with-freebsd-part-2.gmi.tpl | 2 +- ...25-02-01-f3s-kubernetes-with-freebsd-part-3.gmi | 2 + ...25-04-05-f3s-kubernetes-with-freebsd-part-4.gmi | 2 + ...25-05-11-f3s-kubernetes-with-freebsd-part-5.gmi | 2 + ...25-07-14-f3s-kubernetes-with-freebsd-part-6.gmi | 10 +- ...7-14-f3s-kubernetes-with-freebsd-part-6.gmi.tpl | 4 +- ...25-10-02-f3s-kubernetes-with-freebsd-part-7.gmi | 960 +++++++++++++ ...0-02-f3s-kubernetes-with-freebsd-part-7.gmi.tpl | 919 ++++++++++++ gemfeed/DRAFT-kubernetes-with-freebsd-part-7.gmi | 941 ------------- .../DRAFT-kubernetes-with-freebsd-part-7.gmi.tpl | 913 ------------ gemfeed/atom.xml | 1472 +++++++++++++++----- gemfeed/index.gmi | 3 +- gemfeed/stunnel-nfs-quick-reference.txt | 78 -- index.gmi | 5 +- uptime-stats.gmi | 2 +- 23 files changed, 3115 insertions(+), 2410 deletions(-) delete mode 100644 gemfeed/.gitignore create mode 100644 gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi create mode 100644 gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi.tpl delete mode 100644 gemfeed/DRAFT-kubernetes-with-freebsd-part-7.gmi delete mode 100644 gemfeed/DRAFT-kubernetes-with-freebsd-part-7.gmi.tpl delete mode 100644 gemfeed/stunnel-nfs-quick-reference.txt diff --git a/about/resources.gmi b/about/resources.gmi index df7bd80b..3a83d551 100644 --- a/about/resources.gmi +++ b/about/resources.gmi @@ -35,107 +35,107 @@ You won't find any links on this site because, over time, the links will break. In random order: -* The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton -* C++ Programming Language; Bjarne Stroustrup; -* DNS and BIND; Cricket Liu; O'Reilly -* Site Reliability Engineering; How Google runs production systems; O'Reilly -* The Kubernetes Book; Nigel Poulton; Unabridged Audiobook -* Higher Order Perl; Mark Dominus; Morgan Kaufmann -* Funktionale Programmierung; Peter Pepper; Springer -* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson -* Developing Games in Java; David Brackeen and others...; New Riders -* Raku Fundamentals; Moritz Lenz; Apress -* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible +* Modern Perl; Chromatic ; Onyx Neon Press * Terraform Cookbook; Mikael Krief; Packt Publishing -* Effective Java; Joshua Bloch; Addison-Wesley Professional -* Perl New Features; Joshua McAdams, brian d foy; Perl School -* The Docker Book; James Turnbull; Kindle +* Java ist auch eine Insel; Christian Ullenboom; * The Pragmatic Programmer; David Thomas; Addison-Wesley -* Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook +* Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly +* Perl New Features; Joshua McAdams, brian d foy; Perl School * Raku Recipes; J.J. Merelo; Apress -* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications -* The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional +* The Kubernetes Book; Nigel Poulton; Unabridged Audiobook +* The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton +* Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner +* Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press +* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible +* Chaos Engineering - System Resiliency in Practice; Casey Rosenthal and Nora Jones; eBook +* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly +* The Docker Book; James Turnbull; Kindle +* Ultimate Go Notebook; Bill Kennedy +* Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly * 97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly -* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible -* Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly +* Effective awk programming; Arnold Robbins; O'Reilly +* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications * Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers +* Pro Puppet; James Turnbull, Jeffrey McCune; Apress +* Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly * The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress -* Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications -* Data Science at the Command Line; Jeroen Janssens; O'Reilly -* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly -* Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly -* Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly -* Effective awk programming; Arnold Robbins; O'Reilly +* C++ Programming Language; Bjarne Stroustrup; * Concurrency in Go; Katherine Cox-Buday; O'Reilly -* Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner -* Modern Perl; Chromatic ; Onyx Neon Press -* Java ist auch eine Insel; Christian Ullenboom; -* Leanring eBPF; Liz Rice; O'Reilly -* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers +* Raku Fundamentals; Moritz Lenz; Apress +* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly * Polished Ruby Programming; Jeremy Evans; Packt Publishing -* Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press -* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly -* Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press -* Systemprogrammierung in Go; Frank Müller; dpunkt * Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt -* Pro Puppet; James Turnbull, Jeffrey McCune; Apress +* Effective Java; Joshua Bloch; Addison-Wesley Professional +* Leanring eBPF; Liz Rice; O'Reilly +* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson +* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible +* Funktionale Programmierung; Peter Pepper; Springer +* Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press +* Higher Order Perl; Mark Dominus; Morgan Kaufmann * Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf -* Ultimate Go Notebook; Bill Kennedy -* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly +* Developing Games in Java; David Brackeen and others...; New Riders +* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly +* Site Reliability Engineering; How Google runs production systems; O'Reilly +* Systemprogrammierung in Go; Frank Müller; dpunkt +* The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional +* Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications +* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers +* Data Science at the Command Line; Jeroen Janssens; O'Reilly +* DNS and BIND; Cricket Liu; O'Reilly ## Technical references I didn't read them from the beginning to the end, but I am using them to look up things. The books are in random order: -* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley +* Implementing Service Level Objectives; Alex Hidalgo; O'Reilly +* Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly +* Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley * The Linux Programming Interface; Michael Kerrisk; No Starch Press * Go: Design Patterns for Real-World Projects; Mat Ryer; Packt -* Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly * Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly +* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley * Relayd and Httpd Mastery; Michael W Lucas -* Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley -* Implementing Service Level Objectives; Alex Hidalgo; O'Reilly ## Self-development and soft-skills books In random order: -* Ultralearning; Scott Young; Thorsons -* Getting Things Done; David Allen -* 97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook -* Solve for Happy; Mo Gawdat (RE-READ 1ST TIME) +* The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd * The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books -* Stop starting, start finishing; Arne Roock; Lean-Kanban University -* Atomic Habits; James Clear; Random House Business -* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business -* The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK * So Good They Can't Ignore You; Cal Newport; Business Plus -* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion -* Influence without Authority; A. Cohen, D. Bradford; Wiley -* The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook -* Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing -* The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd -* Eat That Frog!; Brian Tracy; Hodder Paperbacks +* The Good Enough Job; Simone Stolzoff; Ebury Edge * 101 Essays that change the way you think; Brianna Wiest; Audiobook -* Slow Productivity; Cal Newport; Penguin Random House -* Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook -* Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook -* The Power of Now; Eckhard Tolle; Yellow Kite -* Soft Skills; John Sommez; Manning Publications -* The Joy of Missing Out; Christina Crook; New Society Publishers -* Eat That Frog; Brian Tracy +* Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne * Deep Work; Cal Newport; Piatkus +* Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook * The Bullet Journal Method; Ryder Carroll; Fourth Estate -* The Good Enough Job; Simone Stolzoff; Ebury Edge -* Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press -* Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne -* The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME) +* Eat That Frog; Brian Tracy +* Influence without Authority; A. Cohen, D. Bradford; Wiley +* Solve for Happy; Mo Gawdat (RE-READ 1ST TIME) +* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business +* Meditation for Mortals, Oliver Burkeman, Audiobook +* The Power of Now; Eckhard Tolle; Yellow Kite +* 97 Things Every Engineering Manager Should Know; Camille Fournier; Audiobook +* The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook * The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select -* Digital Minimalism; Cal Newport; Portofolio Penguin +* Getting Things Done; David Allen * Ultralearning; Anna Laurent; Self-published via Amazon -* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly +* Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press +* The Joy of Missing Out; Christina Crook; New Society Publishers * Psycho-Cybernetics; Maxwell Maltz; Perigee Books -* Meditation for Mortals, Oliver Burkeman, Audiobook +* Soft Skills; John Sommez; Manning Publications +* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly +* Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook +* Eat That Frog!; Brian Tracy; Hodder Paperbacks +* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion +* The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME) +* Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing +* Ultralearning; Scott Young; Thorsons +* Slow Productivity; Cal Newport; Penguin Random House +* Atomic Habits; James Clear; Random House Business +* Digital Minimalism; Cal Newport; Portofolio Penguin +* Stop starting, start finishing; Arne Roock; Lean-Kanban University +* The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK => ../notes/index.gmi Here are notes of mine for some of the books @@ -143,30 +143,30 @@ In random order: Some of these were in-person with exams; others were online learning lectures only. In random order: -* Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon +* AWS Immersion Day; Amazon; 1-day interactive online training +* The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online +* Apache Tomcat Best Practises; 3-day on-site training +* Ultimate Go Programming; Bill Kennedy; O'Reilly Online * F5 Loadbalancers Training; 2-day on-site training; F5, Inc. * Functional programming lecture; Remote University of Hagen -* The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online * Developing IaC with Terraform (with Live Lessons); O'Reilly Online +* Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training +* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need) * MySQL Deep Dive Workshop; 2-day on-site training * The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online -* Structure and Interpretation of Computer Programs; Harold Abelson and more...; -* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need) -* AWS Immersion Day; Amazon; 1-day interactive online training -* Ultimate Go Programming; Bill Kennedy; O'Reilly Online -* Apache Tomcat Best Practises; 3-day on-site training +* Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon * Protocol buffers; O'Reilly Online -* Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training -* Algorithms Video Lectures; Robert Sedgewick; O'Reilly Online +* Structure and Interpretation of Computer Programs; Harold Abelson and more...; * Scripting Vim; Damian Conway; O'Reilly Online +* Algorithms Video Lectures; Robert Sedgewick; O'Reilly Online ## Technical guides These are not whole books, but guides (smaller or larger) which I found very useful. in random order: +* Raku Guide at https://raku.guide * Advanced Bash-Scripting Guide * How CPUs work at https://cpu.land -* Raku Guide at https://raku.guide ## Podcasts @@ -174,21 +174,21 @@ These are not whole books, but guides (smaller or larger) which I found very use In random order: -* The ProdCast (Google SRE Podcast) +* Dev Interrupted * Backend Banter -* The Changelog Podcast(s) +* BSD Now [BSD] * Maintainable -* Hidden Brain -* Deep Questions with Cal Newport +* The ProdCast (Google SRE Podcast) +* Fallthrough [Golang] * Wednesday Wisdom -* Modern Mentor -* BSD Now [BSD] +* The Changelog Podcast(s) * Fork Around And Find Out -* Fallthrough [Golang] -* Dev Interrupted -* The Pragmatic Engineer Podcast -* Pratical AI * Cup o' Go [Golang] +* Deep Questions with Cal Newport +* Hidden Brain +* Pratical AI +* The Pragmatic Engineer Podcast +* Modern Mentor ### Podcasts I liked @@ -196,36 +196,36 @@ I liked them but am not listening to them anymore. The podcasts have either "fin * Go Time (predecessor of fallthrough) * CRE: Chaosradio Express [german] -* Java Pub House * Ship It (predecessor of Fork Around And Find Out) -* Modern Mentor * FLOSS weekly +* Java Pub House +* Modern Mentor ## Newsletters I like This is a mix of tech and non-tech newsletters I am subscribed to. In random order: +* Changelog News +* The Valuable Dev * Andreas Brandhorst Newsletter (Sci-Fi author) * The Imperfectionist -* Register Spill -* Applied Go Weekly Newsletter -* VK Newsletter * Golang Weekly * Monospace Mentor -* The Valuable Dev -* Changelog News -* The Pragmatic Engineer * Ruby Weekly +* Applied Go Weekly Newsletter +* The Pragmatic Engineer +* VK Newsletter * byteSizeGo +* Register Spill ## Magazines I like(d) This is a mix of tech I like(d). I may not be a current subscriber, but now and then, I buy an issue. In random order: -* LWN (online only) * Linux User -* freeX (not published anymore) +* LWN (online only) * Linux Magazine +* freeX (not published anymore) # Formal education diff --git a/gemfeed/.gitignore b/gemfeed/.gitignore deleted file mode 100644 index 1e107f52..00000000 --- a/gemfeed/.gitignore +++ /dev/null @@ -1 +0,0 @@ -examples diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi index 0d67f9ba..41e5feaa 100644 --- a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi +++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi @@ -397,6 +397,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi index 1efa22e1..88481536 100644 --- a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi +++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi @@ -676,6 +676,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs diff --git a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi index 7b3759b7..b1057a1a 100644 --- a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi +++ b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi @@ -53,6 +53,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs diff --git a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi index c79cb573..7eb69d32 100644 --- a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi +++ b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.gmi @@ -300,6 +300,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD and KISS related posts are: +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs diff --git a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi index 0d01ac9d..9b9a0ebc 100644 --- a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi +++ b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi @@ -14,6 +14,7 @@ These are all the posts so far: => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -162,6 +163,7 @@ Read the next post of this series: Other *BSD-related posts: +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs diff --git a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi index 70036e8a..827d4ff8 100644 --- a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi +++ b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi @@ -1,4 +1,4 @@ - f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation +# f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation > Published at 2024-12-02T23:48:21+02:00 @@ -14,6 +14,7 @@ These are all the posts so far: => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -23,6 +24,7 @@ Let's continue... ## Table of Contents +* ⇢ f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation * ⇢ Deciding on the hardware * ⇢ ⇢ Not ARM but Intel N100 * ⇢ ⇢ Beelink unboxing @@ -302,6 +304,7 @@ Read the next post of this series: Other *BSD-related posts: +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs diff --git a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi.tpl b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi.tpl index 03dff1d0..cc9b2903 100644 --- a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi.tpl +++ b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi.tpl @@ -1,4 +1,4 @@ - f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation +# f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation > Published at 2024-12-02T23:48:21+02:00 diff --git a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi index 65c1637b..14b9b35e 100644 --- a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi +++ b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi @@ -10,6 +10,7 @@ This is the third blog post about my f3s series for my self-hosting demands in m => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -364,6 +365,7 @@ Read the next post of this series: Other BSD related posts are: +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi index e5c3a5ae..9f532d6a 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi @@ -10,6 +10,7 @@ This is the fourth blog post about the f3s series for self-hosting demands in a => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs (You are currently reading this) => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -510,6 +511,7 @@ Read the next post of this series: Other *BSD-related posts: +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs (You are currently reading this) diff --git a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi index f22ddc15..ae88f848 100644 --- a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi +++ b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi @@ -14,6 +14,7 @@ These are all the posts so far: => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network (You are currently reading this) => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -931,6 +932,7 @@ Read the next post of this series: Other *BSD-related posts: +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network (You are currently reading this) => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs diff --git a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi index ba315359..77117a8d 100644 --- a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi +++ b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi @@ -10,6 +10,7 @@ This is the sixth blog post about the f3s series for self-hosting demands in a h => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this) +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo @@ -719,7 +720,7 @@ Whereas: Next, update `/etc/hosts` on all nodes (`f0`, `f1`, `f2`, `r0`, `r1`, `r2`) to resolve the VIP hostname: ``` -192.168.1.138 f3s-storage-ha f3s-storage-ha.lan f3s-storage-ha.lan.buetow.org +192.168.2.138 f3s-storage-ha f3s-storage-ha.wg0 f3s-storage-ha.wg0.wan.buetow.org ``` This allows clients to connect to `f3s-storage-ha` regardless of which physical server is currently the MASTER. @@ -1400,7 +1401,7 @@ To mount NFS through the stunnel encrypted tunnel, we run: clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1) # For persistent mount, add to /etc/fstab: -127.0.0.1:/data/nfs/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev 0 0 +127.0.0.1:/k3svolumes /data/nfs/k3svolumes nfs4 port=2323,_netdev,soft,timeo=10,retrans=2,intr 0 0 ``` Note: The mount uses localhost (`127.0.0.1`) because stunnel is listening locally and forwarding the encrypted traffic to the remote server. @@ -1650,10 +1651,13 @@ MooseFS is a fault-tolerant, distributed file system that could provide proper h Both technologies could run on top of our encrypted ZFS volumes, combining ZFS's data integrity and encryption features with distributed storage capabilities. This would be particularly interesting for workloads that need either S3-compatible APIs (MinIO) or transparent distributed POSIX storage (MooseFS). What about Ceph and GlusterFS? Unfortunately, there doesn't seem to be great native FreeBSD support for them. However, other alternatives also appear suitable for my use case. -I'm looking forward to the next post in this series, where we will set up k3s (Kubernetes) on the Linux VMs. +Read the next post of this series: + +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments Other *BSD-related posts: +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments => ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this) => ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network => ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs diff --git a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi.tpl b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi.tpl index d62100cd..d0843866 100644 --- a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi.tpl +++ b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi.tpl @@ -1602,7 +1602,9 @@ MooseFS is a fault-tolerant, distributed file system that could provide proper h Both technologies could run on top of our encrypted ZFS volumes, combining ZFS's data integrity and encryption features with distributed storage capabilities. This would be particularly interesting for workloads that need either S3-compatible APIs (MinIO) or transparent distributed POSIX storage (MooseFS). What about Ceph and GlusterFS? Unfortunately, there doesn't seem to be great native FreeBSD support for them. However, other alternatives also appear suitable for my use case. -I'm looking forward to the next post in this series, where we will set up k3s (Kubernetes) on the Linux VMs. +Read the next post of this series: + +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments Other *BSD-related posts: diff --git a/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi new file mode 100644 index 00000000..c9f8c2b5 --- /dev/null +++ b/gemfeed/2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi @@ -0,0 +1,960 @@ +# f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments + +> Published at 2025-10-02T11:27:19+03:00 + +This is the seventh blog post about the f3s series for my self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution I use on FreeBSD-based physical machines. + +=> ./2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi 2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage +=> ./2024-12-03-f3s-kubernetes-with-freebsd-part-2.gmi 2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation +=> ./2025-02-01-f3s-kubernetes-with-freebsd-part-3.gmi 2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts +=> ./2025-04-05-f3s-kubernetes-with-freebsd-part-4.gmi 2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs +=> ./2025-05-11-f3s-kubernetes-with-freebsd-part-5.gmi 2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network +=> ./2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi 2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage +=> ./2025-10-02-f3s-kubernetes-with-freebsd-part-7.gmi 2025-10-02 f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments (You are currently reading this) + +=> ./f3s-kubernetes-with-freebsd-part-1/f3slogo.png f3s logo + +## Table of Contents + +* ⇢ f3s: Kubernetes with FreeBSD - Part 7: k3s and first pod deployments +* ⇢ ⇢ Introduction +* ⇢ ⇢ Updating +* ⇢ ⇢ Installing k3s +* ⇢ ⇢ ⇢ Generating `K3S_TOKEN` and starting the first k3s node +* ⇢ ⇢ ⇢ Adding the remaining nodes to the cluster +* ⇢ ⇢ Test deployments +* ⇢ ⇢ ⇢ Test deployment to Kubernetes +* ⇢ ⇢ ⇢ Test deployment with persistent volume claim +* ⇢ ⇢ ⇢ Scaling Traefik for faster failover +* ⇢ ⇢ Make it accessible from the public internet +* ⇢ ⇢ ⇢ OpenBSD relayd configuration +* ⇢ ⇢ Deploying the private Docker image registry +* ⇢ ⇢ ⇢ Prepare the NFS-backed storage +* ⇢ ⇢ ⇢ Install (or upgrade) the chart +* ⇢ ⇢ ⇢ Allow nodes and workstations to trust the registry +* ⇢ ⇢ ⇢ Pushing and pulling images +* ⇢ ⇢ Example: Anki Sync Server from the private registry +* ⇢ ⇢ ⇢ Build and push the image +* ⇢ ⇢ ⇢ Create the Anki secret and storage on the cluster +* ⇢ ⇢ ⇢ Deploy the chart +* ⇢ ⇢ NFSv4 UID mapping for Postgres-backed (and other) apps +* ⇢ ⇢ ⇢ Helm charts currently in service + +## Introduction + +In this blog post, I am finally going to install k3s (the Kubernetes distribution I use) to the whole setup and deploy the first workloads (helm charts, and a private registry) to it. + +=> https://k3s.io + +## Updating + +Before proceeding, I bring all systems involved up-to-date. On all three Rocky Linux 9 boxes `r0`, `r1`, and `r2`: + +```sh +dnf update -y +reboot +``` + +On the FreeBSD hosts, I upgraded from FreeBSD 14.2 to 14.3-RELEASE, running this on all three hosts `f0`, `f1` and `f2`: + +```sh +paul@f0:~ % doas freebsd-update fetch +paul@f0:~ % doas freebsd-update install +paul@f0:~ % doas reboot +. +. +. +paul@f0:~ % doas freebsd-update -r 14.3-RELEASE upgrade +paul@f0:~ % doas freebsd-update install +paul@f0:~ % doas freebsd-update install +paul@f0:~ % doas reboot +. +. +. +paul@f0:~ % doas freebsd-update install +paul@f0:~ % doas pkg update +paul@f0:~ % doas pkg upgrade +paul@f0:~ % doas reboot +. +. +. +paul@f0:~ % uname -a +FreeBSD f0.lan.buetow.org 14.3-RELEASE FreeBSD 14.3-RELEASE + releng/14.3-n271432-8c9ce319fef7 GENERIC amd64 +``` + +## Installing k3s + +### Generating `K3S_TOKEN` and starting the first k3s node + +I generated the k3s token on my Fedora laptop with `pwgen -n 32` and selected one of the results. Then, on all three `r` hosts, I ran the following (replace SECRET_TOKEN with the actual secret): + +```sh +[root@r0 ~]# echo -n SECRET_TOKEN > ~/.k3s_token +``` + +The following steps are also documented on the k3s website: + +=> https://docs.k3s.io/datastore/ha-embedded + +To bootstrap k3s on the first node, I ran this on `r0`: + +```sh +[root@r0 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \ + sh -s - server --cluster-init --tls-san=r0.wg0.wan.buetow.org +[INFO] Finding release for channel stable +[INFO] Using v1.32.6+k3s1 as release +. +. +. +[INFO] systemd: Starting k3s +``` + +### Adding the remaining nodes to the cluster + +Then I ran on the other two nodes `r1` and `r2`: + +```sh +[root@r1 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \ + sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \ + --tls-san=r1.wg0.wan.buetow.org + +[root@r2 ~]# curl -sfL https://get.k3s.io | K3S_TOKEN=$(cat ~/.k3s_token) \ + sh -s - server --server https://r0.wg0.wan.buetow.org:6443 \ + --tls-san=r2.wg0.wan.buetow.org +. +. +. + +``` + +Once done, I had a three-node Kubernetes cluster control plane: + +```sh +[root@r0 ~]# kubectl get nodes +NAME STATUS ROLES AGE VERSION +r0.lan.buetow.org Ready control-plane,etcd,master 4m44s v1.32.6+k3s1 +r1.lan.buetow.org Ready control-plane,etcd,master 3m13s v1.32.6+k3s1 +r2.lan.buetow.org Ready control-plane,etcd,master 30s v1.32.6+k3s1 + +[root@r0 ~]# kubectl get pods --all-namespaces +NAMESPACE NAME READY STATUS RESTARTS AGE +kube-system coredns-5688667fd4-fs2jj 1/1 Running 0 5m27s +kube-system helm-install-traefik-crd-f9hgd 0/1 Completed 0 5m27s +kube-system helm-install-traefik-zqqqk 0/1 Completed 2 5m27s +kube-system local-path-provisioner-774c6665dc-jqlnc 1/1 Running 0 5m27s +kube-system metrics-server-6f4c6675d5-5xpmp 1/1 Running 0 5m27s +kube-system svclb-traefik-411cec5b-cdp2l 2/2 Running 0 78s +kube-system svclb-traefik-411cec5b-f625r 2/2 Running 0 4m58s +kube-system svclb-traefik-411cec5b-twrd7 2/2 Running 0 4m2s +kube-system traefik-c98fdf6fb-lt6fx 1/1 Running 0 4m58s +``` + +In order to connect with `kubectl` from my Fedora laptop, I had to copy `/etc/rancher/k3s/k3s.yaml` from `r0` to `~/.kube/config` and then replace the value of the server field with `r0.lan.buetow.org`. kubectl can now manage the cluster. Note that this step has to be repeated when I want to connect to another node of the cluster (e.g. when `r0` is down). + +## Test deployments + +### Test deployment to Kubernetes + +Let's create a test namespace: + +```sh +> ~ kubectl create namespace test +namespace/test created + +> ~ kubectl get namespaces +NAME STATUS AGE +default Active 6h11m +kube-node-lease Active 6h11m +kube-public Active 6h11m +kube-system Active 6h11m +test Active 5s + +> ~ kubectl config set-context --current --namespace=test +Context "default" modified. +``` + +And let's also create an Apache test pod: + +```sh +> ~ cat < apache-deployment.yaml +# Apache HTTP Server Deployment +apiVersion: apps/v1 +kind: Deployment +metadata: + name: apache-deployment +spec: + replicas: 1 + selector: + matchLabels: + app: apache + template: + metadata: + labels: + app: apache + spec: + containers: + - name: apache + image: httpd:latest + ports: + # Container port where Apache listens + - containerPort: 80 +END + +> ~ kubectl apply -f apache-deployment.yaml +deployment.apps/apache-deployment created + +> ~ kubectl get all +NAME READY STATUS RESTARTS AGE +pod/apache-deployment-5fd955856f-4pjmf 1/1 Running 0 7s + +NAME READY UP-TO-DATE AVAILABLE AGE +deployment.apps/apache-deployment 1/1 1 1 7s + +NAME DESIRED CURRENT READY AGE +replicaset.apps/apache-deployment-5fd955856f 1 1 1 7s +``` + +Let's also create a service: + +```sh +> ~ cat < apache-service.yaml +apiVersion: v1 +kind: Service +metadata: + labels: + app: apache + name: apache-service +spec: + ports: + - name: web + port: 80 + protocol: TCP + # Expose port 80 on the service + targetPort: 80 + selector: + # Link this service to pods with the label app=apache + app: apache +END + +> ~ kubectl apply -f apache-service.yaml +service/apache-service created + +> ~ kubectl get service +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +apache-service ClusterIP 10.43.249.165 80/TCP 4s +``` + +Now let's create an ingress: + +> Note: I've modified the hosts listed in this example after I published this blog post to ensure that there aren't any bots scraping it. + +```sh +> ~ cat < apache-ingress.yaml + +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: apache-ingress + namespace: test + annotations: + spec.ingressClassName: traefik + traefik.ingress.kubernetes.io/router.entrypoints: web +spec: + rules: + - host: f3s.foo.zone + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: apache-service + port: + number: 80 + - host: standby.f3s.foo.zone + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: apache-service + port: + number: 80 + - host: www.f3s.foo.zone + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: apache-service + port: + number: 80 +END + +> ~ kubectl apply -f apache-ingress.yaml +ingress.networking.k8s.io/apache-ingress created + +> ~ kubectl describe ingress +Name: apache-ingress +Labels: +Namespace: test +Address: 192.168.1.120,192.168.1.121,192.168.1.122 +Ingress Class: traefik +Default backend: +Rules: + Host Path Backends + ---- ---- -------- + f3s.foo.zone + / apache-service:80 (10.42.1.11:80) + standby.f3s.foo.zone + / apache-service:80 (10.42.1.11:80) + www.f3s.foo.zone + / apache-service:80 (10.42.1.11:80) +Annotations: spec.ingressClassName: traefik + traefik.ingress.kubernetes.io/router.entrypoints: web +Events: +``` + +Notes: + +* In the ingress, I use plain HTTP (web) for the Traefik rule, as all the "production" traffic will be routed through a WireGuard tunnel anyway, as I will show later. + +So I tested the Apache web server through the ingress rule: + +```sh +> ~ curl -H "Host: www.f3s.foo.zone" http://r0.lan.buetow.org:80 +

It works!

+``` + +### Test deployment with persistent volume claim + +Next, I modified the Apache example to serve the `htdocs` directory from the NFS share I created in the previous blog post. I used the following manifests. Most of them are the same as before, except for the persistent volume claim and the volume mount in the Apache deployment. + +```sh +> ~ cat < apache-deployment.yaml +# Apache HTTP Server Deployment +apiVersion: apps/v1 +kind: Deployment +metadata: + name: apache-deployment + namespace: test +spec: + replicas: 2 + selector: + matchLabels: + app: apache + template: + metadata: + labels: + app: apache + spec: + containers: + - name: apache + image: httpd:latest + ports: + # Container port where Apache listens + - containerPort: 80 + readinessProbe: + httpGet: + path: / + port: 80 + initialDelaySeconds: 5 + periodSeconds: 10 + livenessProbe: + httpGet: + path: / + port: 80 + initialDelaySeconds: 15 + periodSeconds: 10 + volumeMounts: + - name: apache-htdocs + mountPath: /usr/local/apache2/htdocs/ + volumes: + - name: apache-htdocs + persistentVolumeClaim: + claimName: example-apache-pvc +END + +> ~ cat < apache-ingress.yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: apache-ingress + namespace: test + annotations: + spec.ingressClassName: traefik + traefik.ingress.kubernetes.io/router.entrypoints: web +spec: + rules: + - host: f3s.foo.zone + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: apache-service + port: + number: 80 + - host: standby.f3s.foo.zone + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: apache-service + port: + number: 80 + - host: www.f3s.foo.zone + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: apache-service + port: + number: 80 +END + +> ~ cat < apache-persistent-volume.yaml +apiVersion: v1 +kind: PersistentVolume +metadata: + name: example-apache-pv +spec: + capacity: + storage: 1Gi + volumeMode: Filesystem + accessModes: + - ReadWriteOnce + persistentVolumeReclaimPolicy: Retain + hostPath: + path: /data/nfs/k3svolumes/example-apache-volume-claim + type: Directory +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: example-apache-pvc + namespace: test +spec: + storageClassName: "" + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi +END + +> ~ cat < apache-service.yaml +apiVersion: v1 +kind: Service +metadata: + labels: + app: apache + name: apache-service + namespace: test +spec: + ports: + - name: web + port: 80 + protocol: TCP + # Expose port 80 on the service + targetPort: 80 + selector: + # Link this service to pods with the label app=apache + app: apache +END +``` + +I applied the manifests: + +```sh +> ~ kubectl apply -f apache-persistent-volume.yaml +> ~ kubectl apply -f apache-service.yaml +> ~ kubectl apply -f apache-deployment.yaml +> ~ kubectl apply -f apache-ingress.yaml +``` + +Looking at the deployment, I could see it failed because the directory didn't exist yet on the NFS share (note that I also increased the replica count to 2 so if one node goes down there's already a replica running on another node for faster failover): + +```sh +> ~ kubectl get pods +NAME READY STATUS RESTARTS AGE +apache-deployment-5b96bd6b6b-fv2jx 0/1 ContainerCreating 0 9m15s +apache-deployment-5b96bd6b6b-ax2ji 0/1 ContainerCreating 0 9m15s + +> ~ kubectl describe pod apache-deployment-5b96bd6b6b-fv2jx | tail -n 5 +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal Scheduled 9m34s default-scheduler Successfully + assigned test/apache-deployment-5b96bd6b6b-fv2jx to r2.lan.buetow.org + Warning FailedMount 80s (x12 over 9m34s) kubelet MountVolume.SetUp + failed for volume "example-apache-pv" : hostPath type check failed: + /data/nfs/k3svolumes/example-apache is not a directory +``` + +That's intentional—I needed to create the directory on the NFS share first, so I did that (e.g. on `r0`): + +```sh +[root@r0 ~]# mkdir /data/nfs/k3svolumes/example-apache-volume-claim/ + +[root@r0 ~]# cat < /data/nfs/k3svolumes/example-apache-volume-claim/index.html + + + + Hello, it works + + +

Hello, it works!

+

This site is served via a PVC!

+ + +END +``` + +The `index.html` file gives us some actual content to serve. After deleting the pod, it recreates itself and the volume mounts correctly: + +```sh +> ~ kubectl delete pod apache-deployment-5b96bd6b6b-fv2jx + +> ~ curl -H "Host: www.f3s.foo.zone" http://r0.lan.buetow.org:80 + + + + Hello, it works + + +

Hello, it works!

+

This site is served via a PVC!

+ + +``` + +### Scaling Traefik for faster failover + +Traefik (used for ingress on k3s) ships with a single replica by default, but for faster failover I bumped it to two replicas so each worker node runs one pod. That way, if a node disappears, the service stays up while Kubernetes schedules a replacement. Here's the command I used: + +```sh +> ~ kubectl -n kube-system scale deployment traefik --replicas=2 +``` + +And the result: + +```sh +> ~ kubectl -n kube-system get pods -l app.kubernetes.io/name=traefik +kube-system traefik-c98fdf6fb-97kqk 1/1 Running 19 (53d ago) 64d +kube-system traefik-c98fdf6fb-9npg2 1/1 Running 11 (53d ago) 61d +``` + +## Make it accessible from the public internet + +Next, I made this accessible through the public internet via the `www.f3s.foo.zone` hosts. As a reminder from part 1 of this series, I reviewed the section titled "OpenBSD/relayd to the rescue for external connectivity": + +=> ./2024-11-17-f3s-kubernetes-with-freebsd-part-1.gmi f3s: Kubernetes with FreeBSD - Part 1: Setting the stage + +> All apps should be reachable through the internet (e.g., from my phone or computer when travelling). For external connectivity and TLS management, I've got two OpenBSD VMs (one hosted by OpenBSD Amsterdam and another hosted by Hetzner) handling public-facing services like DNS, relaying traffic, and automating Let's Encrypt certificates. + +> All of this (every Linux VM to every OpenBSD box) will be connected via WireGuard tunnels, keeping everything private and secure. There will be 6 WireGuard tunnels (3 k3s nodes times two OpenBSD VMs). + +> So, when I want to access a service running in k3s, I will hit an external DNS endpoint (with the authoritative DNS servers being the OpenBSD boxes). The DNS will resolve to the master OpenBSD VM (see my KISS highly-available with OpenBSD blog post), and from there, the relayd process (with a Let's Encrypt certificate—see my Let's Encrypt with OpenBSD and Rex blog post) will accept the TCP connection and forward it through the WireGuard tunnel to a reachable node port of one of the k3s nodes, thus serving the traffic. + +```sh +> ~ curl https://f3s.foo.zone +

It works!

+ +> ~ curl https://www.f3s.foo.zone +

It works!

+ +> ~ curl https://standby.f3s.foo.zone +

It works!

+``` + +This is how it works in `relayd.conf` on OpenBSD: + +### OpenBSD relayd configuration + +The OpenBSD edge relays keep the Kubernetes-facing addresses for the f3s ingress endpoints in a shared backend table so TLS traffic for every `f3s` hostname lands on the same pool of k3s nodes (pointing to the WireGuard IP addresses of those nodes - remember, they are running locally in my LAN, wheras the OpenBSD edge relays operate in the public internet): + +``` +table { + 192.168.2.120 + 192.168.2.121 + 192.168.2.122 +} +``` + +Inside the `http protocol "https"` block each public hostname gets its Let's Encrypt certificate and is matched to that backend table. Besides the primary trio, every service-specific hostname (`anki`, `bag`, `flux`, `audiobookshelf`, `gpodder`, `radicale`, `vault`, `syncthing`, `uprecords`) and their `www` / `standby` aliases reuse the same pool so new apps can go live just by publishing an ingress rule, whereas they will all map to a service running in k3s: + +``` +http protocol "https" { + tls keypair f3s.foo.zone + tls keypair www.f3s.foo.zone + tls keypair standby.f3s.foo.zone + tls keypair anki.f3s.foo.zone + tls keypair www.anki.f3s.foo.zone + tls keypair standby.anki.f3s.foo.zone + tls keypair bag.f3s.foo.zone + tls keypair www.bag.f3s.foo.zone + tls keypair standby.bag.f3s.foo.zone + tls keypair flux.f3s.foo.zone + tls keypair www.flux.f3s.foo.zone + tls keypair standby.flux.f3s.foo.zone + tls keypair audiobookshelf.f3s.foo.zone + tls keypair www.audiobookshelf.f3s.foo.zone + tls keypair standby.audiobookshelf.f3s.foo.zone + tls keypair gpodder.f3s.foo.zone + tls keypair www.gpodder.f3s.foo.zone + tls keypair standby.gpodder.f3s.foo.zone + tls keypair radicale.f3s.foo.zone + tls keypair www.radicale.f3s.foo.zone + tls keypair standby.radicale.f3s.foo.zone + tls keypair vault.f3s.foo.zone + tls keypair www.vault.f3s.foo.zone + tls keypair standby.vault.f3s.foo.zone + tls keypair syncthing.f3s.foo.zone + tls keypair www.syncthing.f3s.foo.zone + tls keypair standby.syncthing.f3s.foo.zone + tls keypair uprecords.f3s.foo.zone + tls keypair www.uprecords.f3s.foo.zone + tls keypair standby.uprecords.f3s.foo.zone + + match request quick header "Host" value "f3s.foo.zone" forward to + match request quick header "Host" value "www.f3s.foo.zone" forward to + match request quick header "Host" value "standby.f3s.foo.zone" forward to + match request quick header "Host" value "anki.f3s.foo.zone" forward to + match request quick header "Host" value "www.anki.f3s.foo.zone" forward to + match request quick header "Host" value "standby.anki.f3s.foo.zone" forward to + match request quick header "Host" value "bag.f3s.foo.zone" forward to + match request quick header "Host" value "www.bag.f3s.foo.zone" forward to + match request quick header "Host" value "standby.bag.f3s.foo.zone" forward to + match request quick header "Host" value "flux.f3s.foo.zone" forward to + match request quick header "Host" value "www.flux.f3s.foo.zone" forward to + match request quick header "Host" value "standby.flux.f3s.foo.zone" forward to + match request quick header "Host" value "audiobookshelf.f3s.foo.zone" forward to + match request quick header "Host" value "www.audiobookshelf.f3s.foo.zone" forward to + match request quick header "Host" value "standby.audiobookshelf.f3s.foo.zone" forward to + match request quick header "Host" value "gpodder.f3s.foo.zone" forward to + match request quick header "Host" value "www.gpodder.f3s.foo.zone" forward to + match request quick header "Host" value "standby.gpodder.f3s.foo.zone" forward to + match request quick header "Host" value "radicale.f3s.foo.zone" forward to + match request quick header "Host" value "www.radicale.f3s.foo.zone" forward to + match request quick header "Host" value "standby.radicale.f3s.foo.zone" forward to + match request quick header "Host" value "vault.f3s.foo.zone" forward to + match request quick header "Host" value "www.vault.f3s.foo.zone" forward to + match request quick header "Host" value "standby.vault.f3s.foo.zone" forward to + match request quick header "Host" value "syncthing.f3s.foo.zone" forward to + match request quick header "Host" value "www.syncthing.f3s.foo.zone" forward to + match request quick header "Host" value "standby.syncthing.f3s.foo.zone" forward to + match request quick header "Host" value "uprecords.f3s.foo.zone" forward to + match request quick header "Host" value "www.uprecords.f3s.foo.zone" forward to + match request quick header "Host" value "standby.uprecords.f3s.foo.zone" forward to +} +``` + +Both IPv4 and IPv6 listeners reuse the same protocol definition, making the relay transparent for dual-stack clients while still health checking every k3s backend before forwarding traffic over WireGuard: + +``` +relay "https4" { + listen on 46.23.94.99 port 443 tls + protocol "https" + forward to port 80 check tcp +} + +relay "https6" { + listen on 2a03:6000:6f67:624::99 port 443 tls + protocol "https" + forward to port 80 check tcp +} +``` + +In practice, that means relayd terminates TLS with the correct certificate, keeps the three WireGuard-connected backends in rotation, and ships each request to whichever bhyve VM answers first. + +## Deploying the private Docker image registry + +As not all Docker images I want to deploy are available on public Docker registries and as I also build some of them by myself, there is the need of a private registry. + +All manifests for the f3s stack live in my configuration repository: + +=> https://codeberg.org/snonux/conf/src/branch/master/f3s codeberg.org/snonux/conf/f3s + +Within that repo, the `examples/conf/f3s/registry/` directory contains the Helm chart, a `Justfile`, and a detailed `README`. Here's the condensed walkthrough I used to roll out the registry with Helm. + +### Prepare the NFS-backed storage + +Create the directory that will hold the registry blobs on the NFS share (I ran this on `r0`, but any node that exports `/data/nfs/k3svolumes` works): + +```sh +[root@r0 ~]# mkdir -p /data/nfs/k3svolumes/registry +``` + +### Install (or upgrade) the chart + +Clone the repo (or pull the latest changes) on a workstation that has `helm` configured for the cluster, then deploy the chart. The Justfile wraps the commands, but the raw Helm invocation looks like this: + +```sh +$ git clone https://codeberg.org/snonux/conf/f3s.git +$ cd conf/f3s/examples/conf/f3s/registry +$ helm upgrade --install registry ./helm-chart --namespace infra --create-namespace +``` + +Helm creates the `infra` namespace if it does not exist, provisions a `PersistentVolume`/`PersistentVolumeClaim` pair that points at `/data/nfs/k3svolumes/registry`, and spins up a single registry pod exposed via the `docker-registry-service` NodePort (`30001`). Verify everything is up before continuing: + +```sh +$ kubectl get pods --namespace infra +NAME READY STATUS RESTARTS AGE +docker-registry-6bc9bb46bb-6grkr 1/1 Running 6 (53d ago) 54d + +$ kubectl get svc docker-registry-service -n infra +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +docker-registry-service NodePort 10.43.141.56 5000:30001/TCP 54d +``` + +### Allow nodes and workstations to trust the registry + +The registry listens on plain HTTP, so both Docker daemons on workstations and the k3s nodes need to treat it as an insecure registry. That's fine for my personal needs, as: + +* I don't store any secrets in the images +* I access the registry this way only via my LAN +* I may will change it later on... + +On my Fedora workstation where I build images: + +```sh +$ cat <<"EOF" | sudo tee /etc/docker/daemon.json >/dev/null +{ + "insecure-registries": [ + "r0.lan.buetow.org:30001", + "r1.lan.buetow.org:30001", + "r2.lan.buetow.org:30001" + ] +} +EOF +$ sudo systemctl restart docker +``` + +On each k3s node, make `registry.lan.buetow.org` resolve locally and point k3s at the NodePort: + +```sh +$ for node in r0 r1 r2; do +> ssh root@$node "echo '127.0.0.1 registry.lan.buetow.org' >> /etc/hosts" +> done + +$ for node in r0 r1 r2; do +> ssh root@$node "cat <<'EOF' > /etc/rancher/k3s/registries.yaml +mirrors: + "registry.lan.buetow.org:30001": + endpoint: + - "http://localhost:30001" +EOF +systemctl restart k3s" +> done +``` + +Thanks to the relayd configuration earlier in the post, the external hostnames (`f3s.foo.zone`, etc.) can already reach NodePort `30001`, so publishing the registry later to the outside world is just a matter of wiring the DNS the same way as the ingress hosts. But by default, that's not enabled for now due to security reasons. + +### Pushing and pulling images + +Tag any locally built image with one of the node IPs on port `30001`, then push it. I usually target whichever node is closest to me, but any of the three will do: + +```sh +$ docker tag my-app:latest r0.lan.buetow.org:30001/my-app:latest +$ docker push r0.lan.buetow.org:30001/my-app:latest +``` + +Inside the cluster (or from other nodes), reference the image via the service name that Helm created: + +``` +image: docker-registry-service:5000/my-app:latest +``` + +You can test the pull path straight away: + +```sh +$ kubectl run registry-test \ +> --image=docker-registry-service:5000/my-app:latest \ +> --restart=Never -n test --command -- sleep 300 +``` + +If the pod pulls successfully, the private registry is ready for use by the rest of the workloads. Note, that the commands above actually don't work, they are only for illustration purpose mentioned here. + +## Example: Anki Sync Server from the private registry + +One of the first workloads I migrated onto the k3s cluster after standing up the registry was my Anki sync server. The configuration repo ships everything in `examples/conf/f3s/anki-sync-server/`: a Docker build context plus a Helm chart that references the freshly built image. + +### Build and push the image + +The Dockerfile lives under `docker-image/` and takes the Anki release to compile as an `ANKI_VERSION` build argument. The accompanying `Justfile` wraps the steps, but the raw commands look like this: + +```sh +$ cd conf/f3s/examples/conf/f3s/anki-sync-server/docker-image +$ docker build -t anki-sync-server:25.07.5b --build-arg ANKI_VERSION=25.07.5 . +$ docker tag anki-sync-serv