From ec274d473332eceb14f626a56907404c42e9f971 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Sun, 13 Jul 2025 16:49:45 +0300 Subject: Update content for md --- about/resources.md | 200 +- ...6-04-09-jails-and-zfs-on-freebsd-with-puppet.md | 1 + ...2022-07-30-lets-encrypt-with-openbsd-and-rex.md | 1 + .../2024-01-13-one-reason-why-i-love-openbsd.md | 1 + ...24-04-01-KISS-high-availability-with-OpenBSD.md | 1 + ...024-11-17-f3s-kubernetes-with-freebsd-part-1.md | 2 + ...024-12-03-f3s-kubernetes-with-freebsd-part-2.md | 2 + ...025-02-01-f3s-kubernetes-with-freebsd-part-3.md | 2 + ...025-04-05-f3s-kubernetes-with-freebsd-part-4.md | 2 + ...025-05-11-f3s-kubernetes-with-freebsd-part-5.md | 6 +- ...025-07-14-f3s-kubernetes-with-freebsd-part-6.md | 1610 ++++++++++++ .../DRAFT-f3s-kubernetes-with-freebsd-part-6.md | 2702 -------------------- .../f3s-kubernetes-with-freebsd-part-6/zrepl.png | Bin 0 -> 166760 bytes gemfeed/index.md | 1 + index.md | 3 +- uptime-stats.md | 32 +- 16 files changed, 1746 insertions(+), 2820 deletions(-) create mode 100644 gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md delete mode 100644 gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.md create mode 100644 gemfeed/f3s-kubernetes-with-freebsd-part-6/zrepl.png diff --git a/about/resources.md b/about/resources.md index 19bad06e..0fabb703 100644 --- a/about/resources.md +++ b/about/resources.md @@ -35,105 +35,105 @@ You won't find any links on this site because, over time, the links will break. In random order: -* Modern Perl; Chromatic ; Onyx Neon Press -* Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers -* Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner -* Effective awk programming; Arnold Robbins; O'Reilly +* Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt +* Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly +* Perl New Features; Joshua McAdams, brian d foy; Perl School * 97 things every SRE should know; Emil Stolarsky, Jaime Woo; O'Reilly -* DNS and BIND; Cricket Liu; O'Reilly -* Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly -* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly -* Data Science at the Command Line; Jeroen Janssens; O'Reilly +* Site Reliability Engineering; How Google runs production systems; O'Reilly +* Ultimate Go Notebook; Bill Kennedy +* The Pragmatic Programmer; David Thomas; Addison-Wesley +* Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf +* Java ist auch eine Insel; Christian Ullenboom; * Concurrency in Go; Katherine Cox-Buday; O'Reilly -* Perl New Features; Joshua McAdams, brian d foy; Perl School +* The Docker Book; James Turnbull; Kindle +* Leanring eBPF; Liz Rice; O'Reilly * The KCNA (Kubernetes and Cloud Native Associate) Book; Nigel Poulton -* Programming Ruby 3.3 (5th Edition); Noel Rappin, with Dave Thomas; The Pragmatic Bookshelf -* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible +* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications +* 21st Century C: C Tips from the New School; Ben Klemens; O'Reilly +* C++ Programming Language; Bjarne Stroustrup; * Pro Puppet; James Turnbull, Jeffrey McCune; Apress -* Raku Fundamentals; Moritz Lenz; Apress -* Ultimate Go Notebook; Bill Kennedy -* Hands-on Infrastructure Monitoring with Prometheus; Joel Bastos, Pedro Araujo; Packt -* Leanring eBPF; Liz Rice; O'Reilly -* Higher Order Perl; Mark Dominus; Morgan Kaufmann +* Programming Perl aka "The Camel Book"; Tom Christiansen, brian d foy, Larry Wall & Jon Orwant; O'Reilly +* Effective Java; Joshua Bloch; Addison-Wesley Professional +* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers +* Developing Games in Java; David Brackeen and others...; New Riders +* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly +* DevOps And Site Reliability Engineering Handbook; Stephen Fleming; Audible +* The Kubernetes Book; Nigel Poulton; Unabridged Audiobook +* Terraform Cookbook; Mikael Krief; Packt Publishing +* Go Brain Teasers - Exercise Your Mind; Miki Tebeka; The Pragmatic Programmers * Systems Performance Tuning; Gian-Paolo D. Musumeci and others...; O'Reilly +* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly * Polished Ruby Programming; Jeremy Evans; Packt Publishing -* Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press +* DNS and BIND; Cricket Liu; O'Reilly +* The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional * Systemprogrammierung in Go; Frank Müller; dpunkt -* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible -* The Docker Book; James Turnbull; Kindle -* The Kubernetes Book; Nigel Poulton; Unabridged Audiobook -* The Pragmatic Programmer; David Thomas; Addison-Wesley * Funktionale Programmierung; Peter Pepper; Springer -* Kubernetes Cookbook; Sameer Naik, Sébastien Goasguen, Jonathan Michaux; O'Reilly -* Clusterbau mit Linux-HA; Michael Schwartzkopff; O'Reilly -* Terraform Cookbook; Mikael Krief; Packt Publishing -* Site Reliability Engineering; How Google runs production systems; O'Reilly -* C++ Programming Language; Bjarne Stroustrup; +* Effective awk programming; Arnold Robbins; O'Reilly +* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson * The Practise of System and Network Administration; Thomas A. Limoncelli, Christina J. Hogan, Strata R. Chalup; Addison-Wesley Professional Pro Git; Scott Chacon, Ben Straub; Apress -* 100 Go Mistakes and How to Avoid Them; Teiva Harsanyi; Manning Publications -* Think Raku (aka Think Perl 6); Laurent Rosenfeld, Allen B. Downey; O'Reilly +* Raku Fundamentals; Moritz Lenz; Apress +* Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications +* Learn You a Haskell for Great Good!; Miran Lipovaca; No Starch Press +* Data Science at the Command Line; Jeroen Janssens; O'Reilly +* Object-Oriented Programming with ANSI-C; Axel-Tobias Schreiner +* The DevOps Handbook; Gene Kim, Jez Humble, Patrick Debois, John Willis; Audible * Raku Recipes; J.J. Merelo; Apress +* Higher Order Perl; Mark Dominus; Morgan Kaufmann * Learn You Some Erlang for Great Good; Fred Herbert; No Starch Press -* Distributed Systems: Principles and Paradigms; Andrew S. Tanenbaum; Pearson -* Effective Java; Joshua Bloch; Addison-Wesley Professional -* Java ist auch eine Insel; Christian Ullenboom; -* Tmux 2: Productive Mouse-free Development; Brain P. Hogan; The Pragmatic Programmers -* Developing Games in Java; David Brackeen and others...; New Riders -* The Go Programming Language; Alan A. A. Donovan; Addison-Wesley Professional -* Amazon Web Services in Action; Michael Wittig and Andreas Wittig; Manning Publications +* Modern Perl; Chromatic ; Onyx Neon Press ## Technical references I didn't read them from the beginning to the end, but I am using them to look up things. The books are in random order: -* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley -* Implementing Service Level Objectives; Alex Hidalgo; O'Reilly * The Linux Programming Interface; Michael Kerrisk; No Starch Press +* BPF Performance Tools - Linux System and Application Observability, Brendan Gregg; Addison Wesley +* Relayd and Httpd Mastery; Michael W Lucas * Go: Design Patterns for Real-World Projects; Mat Ryer; Packt +* Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly +* Implementing Service Level Objectives; Alex Hidalgo; O'Reilly * Groovy Kurz & Gut; Joerg Staudemeier; O'Reilly -* Relayd and Httpd Mastery; Michael W Lucas * Algorithms; Robert Sedgewick, Kevin Wayne; Addison Wesley -* Understanding the Linux Kernel; Daniel P. Bovet, Marco Cesati; O'Reilly ## Self-development and soft-skills books In random order: -* So Good They Can't Ignore You; Cal Newport; Business Plus -* Stop starting, start finishing; Arne Roock; Lean-Kanban University -* Psycho-Cybernetics; Maxwell Maltz; Perigee Books -* Influence without Authority; A. Cohen, D. Bradford; Wiley -* The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook -* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion +* Eat That Frog; Brian Tracy +* Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne * Ultralearning; Anna Laurent; Self-published via Amazon -* Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook -* Deep Work; Cal Newport; Piatkus +* The Joy of Missing Out; Christina Crook; New Society Publishers * Meditation for Mortals, Oliver Burkeman, Audiobook -* The Bullet Journal Method; Ryder Carroll; Fourth Estate -* Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook -* Atomic Habits; James Clear; Random House Business +* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business +* The Power of Now; Eckhard Tolle; Yellow Kite +* The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK +* Deep Work; Cal Newport; Piatkus +* Getting Things Done; David Allen +* The Complete Software Developer's Career Guide; John Sonmez; Unabridged Audiobook * The Obstacle Is The Way; Ryan Holiday; Profile Books Ltd -* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly +* The Good Enough Job; Simone Stolzoff; Ebury Edge * Soft Skills; John Sommez; Manning Publications +* So Good They Can't Ignore You; Cal Newport; Business Plus +* The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME) +* Eat That Frog!; Brian Tracy; Hodder Paperbacks +* Solve for Happy; Mo Gawdat (RE-READ 1ST TIME) +* 101 Essays that change the way you think; Brianna Wiest; Audiobook +* Atomic Habits; James Clear; Random House Business +* Digital Minimalism; Cal Newport; Portofolio Penguin * Consciousness: A Very Short Introduction; Susan Blackmore; Oxford Uiversity Press -* Getting Things Done; David Allen +* Psycho-Cybernetics; Maxwell Maltz; Perigee Books +* Ultralearning; Scott Young; Thorsons * Buddah and Einstein walk into a Bar; Guy Joseph Ale, Claire Bloom; Blackstone Publishing -* Eat That Frog; Brian Tracy -* The 7 Habits Of Highly Effective People; Stephen R. Covey; Simon & Schuster UK -* The Good Enough Job; Simone Stolzoff; Ebury Edge -* Digital Minimalism; Cal Newport; Portofolio Penguin -* Eat That Frog!; Brian Tracy; Hodder Paperbacks +* Influence without Authority; A. Cohen, D. Bradford; Wiley +* Staff Engineer: Leadership beyond the management track; Will Larson; Audiobook +* Coders at Work - Reflections on the craft of programming, Peter Seibel and Mitchell Dorian et al., Audiobook +* Who Moved My Cheese?; Dr. Spencer Johnson; Vermilion * The Daily Stoic; Ryan Holiday, Stephen Hanselman; Profile Books -* The Joy of Missing Out; Christina Crook; New Society Publishers -* The Power of Now; Eckhard Tolle; Yellow Kite -* The Off Switch; Mark Cropley; Virgin Books (RE-READ 1ST TIME) -* Slow Productivity; Cal Newport; Penguin Random House +* Stop starting, start finishing; Arne Roock; Lean-Kanban University +* The Bullet Journal Method; Ryder Carroll; Fourth Estate +* Time Management for System Administrators; Thomas A. Limoncelli; O'Reilly * The Phoenix Project - A Novel About IT, DevOps, and Helping your Business Win; Gene Kim and Kevin Behr; Trade Select -* Solve for Happy; Mo Gawdat (RE-READ 1ST TIME) -* Search Inside Yourself - The Unexpected path to Achieving Success, Happiness (and World Peace); Chade-Meng Tan, Daniel Goleman, Jon Kabat-Zinn; HarperOne -* Ultralearning; Scott Young; Thorsons -* Never Split the Difference; Chris Voss, Tahl Raz; Random House Business -* 101 Essays that change the way you think; Brianna Wiest; Audiobook +* Slow Productivity; Cal Newport; Penguin Random House [Here are notes of mine for some of the books](../notes/index.md) @@ -141,30 +141,30 @@ In random order: Some of these were in-person with exams; others were online learning lectures only. In random order: -* Apache Tomcat Best Practises; 3-day on-site training -* Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training -* Ultimate Go Programming; Bill Kennedy; O'Reilly Online -* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need) -* AWS Immersion Day; Amazon; 1-day interactive online training -* F5 Loadbalancers Training; 2-day on-site training; F5, Inc. * Protocol buffers; O'Reilly Online +* The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online +* Ultimate Go Programming; Bill Kennedy; O'Reilly Online * Functional programming lecture; Remote University of Hagen -* Structure and Interpretation of Computer Programs; Harold Abelson and more...; -* Developing IaC with Terraform (with Live Lessons); O'Reilly Online * The Ultimate Kubernetes Bootcamp; School of Devops; O'Reilly Online +* MySQL Deep Dive Workshop; 2-day on-site training +* AWS Immersion Day; Amazon; 1-day interactive online training +* Algorithms Video Lectures; Robert Sedgewick; O'Reilly Online +* Red Hat Certified System Administrator; Course + certification (Although I had the option, I decided not to take the next course as it is more effective to self learn what I need) +* F5 Loadbalancers Training; 2-day on-site training; F5, Inc. * Cloud Operations on AWS - Learn how to configure, deploy, maintain, and troubleshoot your AWS environments; 3-day online live training with labs; Amazon +* Apache Tomcat Best Practises; 3-day on-site training +* Linux Security and Isolation APIs Training; Michael Kerrisk; 3-day on-site training +* Structure and Interpretation of Computer Programs; Harold Abelson and more...; * Scripting Vim; Damian Conway; O'Reilly Online -* Algorithms Video Lectures; Robert Sedgewick; O'Reilly Online -* The Well-Grounded Rubyist Video Edition; David. A. Black; O'Reilly Online -* MySQL Deep Dive Workshop; 2-day on-site training +* Developing IaC with Terraform (with Live Lessons); O'Reilly Online ## Technical guides These are not whole books, but guides (smaller or larger) which I found very useful. in random order: -* Raku Guide at https://raku.guide * Advanced Bash-Scripting Guide * How CPUs work at https://cpu.land +* Raku Guide at https://raku.guide ## Podcasts @@ -172,57 +172,57 @@ These are not whole books, but guides (smaller or larger) which I found very use In random order: -* BSD Now [BSD] -* Modern Mentor -* Hidden Brain -* Backend Banter -* The Pragmatic Engineer Podcast -* Fork Around And Find Out * Fallthrough [Golang] * The Changelog Podcast(s) +* Backend Banter +* BSD Now [BSD] +* Dev Interrupted +* Fork Around And Find Out +* Modern Mentor * Cup o' Go [Golang] -* The ProdCast (Google SRE Podcast) +* Pratical AI * Maintainable +* Hidden Brain +* The ProdCast (Google SRE Podcast) * Deep Questions with Cal Newport -* Pratical AI -* Dev Interrupted +* The Pragmatic Engineer Podcast ### Podcasts I liked I liked them but am not listening to them anymore. The podcasts have either "finished" (no more episodes) or I stopped listening to them due to time constraints or a shift in my interests. -* Java Pub House -* Ship It (predecessor of Fork Around And Find Out) -* Go Time (predecessor of fallthrough) -* FLOSS weekly * CRE: Chaosradio Express [german] +* FLOSS weekly * Modern Mentor +* Ship It (predecessor of Fork Around And Find Out) +* Java Pub House +* Go Time (predecessor of fallthrough) ## Newsletters I like This is a mix of tech and non-tech newsletters I am subscribed to. In random order: +* Monospace Mentor +* The Imperfectionist +* Ruby Weekly * The Pragmatic Engineer -* Register Spill +* Golang Weekly * Andreas Brandhorst Newsletter (Sci-Fi author) -* The Imperfectionist +* Applied Go Weekly Newsletter +* byteSizeGo * The Valuable Dev -* Monospace Mentor * VK Newsletter +* Register Spill * Changelog News -* Golang Weekly -* Ruby Weekly -* byteSizeGo -* Applied Go Weekly Newsletter ## Magazines I like(d) This is a mix of tech I like(d). I may not be a current subscriber, but now and then, I buy an issue. In random order: -* LWN (online only) -* Linux Magazine * Linux User * freeX (not published anymore) +* Linux Magazine +* LWN (online only) # Formal education diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md index 4b609bc8..f2735c9a 100644 --- a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md +++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.md @@ -397,6 +397,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md index 8234a0e5..f70661ff 100644 --- a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md +++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.md @@ -676,6 +676,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) diff --git a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md index a6fdfb1c..aa53d5fa 100644 --- a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md +++ b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.md @@ -51,6 +51,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD related posts are: +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) diff --git a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md index de244403..adf3ef55 100644 --- a/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md +++ b/gemfeed/2024-04-01-KISS-high-availability-with-OpenBSD.md @@ -300,6 +300,7 @@ E-Mail your comments to `paul@nospam.buetow.org` :-) Other *BSD and KISS related posts are: +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) diff --git a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md index 0847feb6..05ba7e53 100644 --- a/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md +++ b/gemfeed/2024-11-17-f3s-kubernetes-with-freebsd-part-1.md @@ -13,6 +13,7 @@ These are all the posts so far: [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -161,6 +162,7 @@ Read the next post of this series: Other *BSD-related posts: +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) diff --git a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md index a63558ce..1b10965a 100644 --- a/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md +++ b/gemfeed/2024-12-03-f3s-kubernetes-with-freebsd-part-2.md @@ -13,6 +13,7 @@ These are all the posts so far: [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -300,6 +301,7 @@ Read the next post of this series: Other *BSD-related posts: +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) diff --git a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md index b1b44830..f11a637f 100644 --- a/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md +++ b/gemfeed/2025-02-01-f3s-kubernetes-with-freebsd-part-3.md @@ -9,6 +9,7 @@ This is the third blog post about my f3s series for my self-hosting demands in m [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts (You are currently reading this)](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -363,6 +364,7 @@ Read the next post of this series: Other BSD related posts are: +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts (You are currently reading this)](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) diff --git a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md index 53f10a57..9a80f5fe 100644 --- a/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md +++ b/gemfeed/2025-04-05-f3s-kubernetes-with-freebsd-part-4.md @@ -9,6 +9,7 @@ This is the fourth blog post about the f3s series for self-hosting demands in a [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs (You are currently reading this)](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -509,6 +510,7 @@ Read the next post of this series: Other *BSD-related posts: +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs (You are currently reading this)](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) diff --git a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md index 034b3fe8..1dca3454 100644 --- a/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md +++ b/gemfeed/2025-05-11-f3s-kubernetes-with-freebsd-part-5.md @@ -13,6 +13,7 @@ These are all the posts so far: [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network (You are currently reading this)](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) @@ -924,10 +925,13 @@ peer: 2htXdNcxzpI2FdPDJy4T4VGtm1wpMEQu1AkQHjNY6F8= Having a mesh network on our hosts is great for securing all the traffic between them for our future k3s setup. A self-managed WireGuard mesh network is better than Tailscale as it eliminates reliance on a third party and provides full control over the configuration. It reduces unnecessary abstraction and "magic," enabling easier debugging and ensuring full ownership of our network. -I look forward to the next blog post in this series. We may start setting up k3s or take a first look at the NFS server (for persistent storage) side of things. I hope you liked all the posts so far in this series. +Read the next post of this series: + +[f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) Other *BSD-related posts: +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) [2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network (You are currently reading this)](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) [2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) [2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) diff --git a/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md new file mode 100644 index 00000000..87a4f63c --- /dev/null +++ b/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.md @@ -0,0 +1,1610 @@ +# f3s: Kubernetes with FreeBSD - Part 6: Storage + +> Published at 2025-07-13T16:44:29+03:00 + +This is the sixth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines. + +[2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage](./2024-11-17-f3s-kubernetes-with-freebsd-part-1.md) +[2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation](./2024-12-03-f3s-kubernetes-with-freebsd-part-2.md) +[2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts](./2025-02-01-f3s-kubernetes-with-freebsd-part-3.md) +[2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs](./2025-04-05-f3s-kubernetes-with-freebsd-part-4.md) +[2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network](./2025-05-11-f3s-kubernetes-with-freebsd-part-5.md) +[2025-07-14 f3s: Kubernetes with FreeBSD - Part 6: Storage (You are currently reading this)](./2025-07-14-f3s-kubernetes-with-freebsd-part-6.md) + +[![f3s logo](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png "f3s logo")](./f3s-kubernetes-with-freebsd-part-1/f3slogo.png) + +## Table of Contents + +* [⇢ f3s: Kubernetes with FreeBSD - Part 6: Storage](#f3s-kubernetes-with-freebsd---part-6-storage) +* [⇢ ⇢ Introduction](#introduction) +* [⇢ ⇢ Additional storage capacity](#additional-storage-capacity) +* [⇢ ⇢ ZFS encryption keys](#zfs-encryption-keys) +* [⇢ ⇢ ⇢ UFS on USB keys](#ufs-on-usb-keys) +* [⇢ ⇢ ⇢ Generating encryption keys](#generating-encryption-keys) +* [⇢ ⇢ ⇢ Configuring `zdata` ZFS pool encryption](#configuring-zdata-zfs-pool-encryption) +* [⇢ ⇢ ⇢ Migrating Bhyve VMs to an encrypted `bhyve` ZFS volume](#migrating-bhyve-vms-to-an-encrypted-bhyve-zfs-volume) +* [⇢ ⇢ ZFS Replication with `zrepl`](#zfs-replication-with-zrepl) +* [⇢ ⇢ ⇢ Understanding Replication Requirements](#understanding-replication-requirements) +* [⇢ ⇢ ⇢ Installing `zrepl`](#installing-zrepl) +* [⇢ ⇢ ⇢ Configuring `zrepl` on `f1` (sink)](#configuring-zrepl-on-f1-sink) +* [⇢ ⇢ ⇢ Enabling and starting `zrepl` services](#enabling-and-starting-zrepl-services) +* [⇢ ⇢ ⇢ Monitoring replication](#monitoring-replication) +* [⇢ ⇢ ⇢ Verifying replication after reboot](#verifying-replication-after-reboot) +* [⇢ ⇢ ⇢ Understanding Failover Limitations and Design Decisions](#understanding-failover-limitations-and-design-decisions) +* [⇢ ⇢ ⇢ Mounting the NFS datasets](#mounting-the-nfs-datasets) +* [⇢ ⇢ ⇢ Troubleshooting: Files not appearing in replication](#troubleshooting-files-not-appearing-in-replication) +* [⇢ ⇢ ⇢ Configuring automatic key loading on boot](#configuring-automatic-key-loading-on-boot) +* [⇢ ⇢ CARP (Common Address Redundancy Protocol)](#carp-common-address-redundancy-protocol) +* [⇢ ⇢ ⇢ How CARP Works](#how-carp-works) +* [⇢ ⇢ ⇢ Configuring CARP](#configuring-carp) +* [⇢ ⇢ ⇢ CARP State Change Notifications](#carp-state-change-notifications) +* [⇢ ⇢ NFS Server Configuration](#nfs-server-configuration) +* [⇢ ⇢ ⇢ Setting up NFS on `f0` (Primary)](#setting-up-nfs-on-f0-primary) +* [⇢ ⇢ ⇢ Configuring Stunnel for NFS Encryption with CARP Failover](#configuring-stunnel-for-nfs-encryption-with-carp-failover) +* [⇢ ⇢ ⇢ Creating a Certificate Authority for Client Authentication](#creating-a-certificate-authority-for-client-authentication) +* [⇢ ⇢ ⇢ Install and Configure Stunnel on `f0`](#install-and-configure-stunnel-on-f0) +* [⇢ ⇢ ⇢ Setting up NFS on `f1` (Standby)](#setting-up-nfs-on-f1-standby) +* [⇢ ⇢ ⇢ CARP Control Script for Clean Failover](#carp-control-script-for-clean-failover) +* [⇢ ⇢ ⇢ CARP Management Script](#carp-management-script) +* [⇢ ⇢ ⇢ Automatic Failback After Reboot](#automatic-failback-after-reboot) +* [⇢ ⇢ Client Configuration for Stunnel](#client-configuration-for-stunnel) +* [⇢ ⇢ ⇢ Configuring Rocky Linux Clients (`r0`, `r1`, `r2`)](#configuring-rocky-linux-clients-r0-r1-r2) +* [⇢ ⇢ ⇢ Testing NFS Mount with Stunnel](#testing-nfs-mount-with-stunnel) +* [⇢ ⇢ ⇢ Testing CARP Failover with mounted clients and stale file handles:](#testing-carp-failover-with-mounted-clients-and-stale-file-handles) +* [⇢ ⇢ ⇢ Complete Failover Test](#complete-failover-test) +* [⇢ ⇢ Conclusion](#conclusion) +* [⇢ ⇢ Future Storage Explorations](#future-storage-explorations) +* [⇢ ⇢ ⇢ MinIO for S3-Compatible Object Storage](#minio-for-s3-compatible-object-storage) +* [⇢ ⇢ ⇢ MooseFS for Distributed High Availability](#moosefs-for-distributed-high-availability) + +## Introduction + +In the previous posts, we set up a FreeBSD-based Kubernetes cluster using k3s. While the base system works well, Kubernetes workloads often require persistent storage for databases, configuration files, and application data. Local storage on each node has significant limitations: + +* No data sharing: Pods (once we run Kubernetes) on different nodes can't access the same data +* Pod mobility: If a pod moves to another node, it loses access to its data +* No redundancy: Hardware failure means data loss + +This post implements a robust storage solution using: + +* CARP: For high availability with automatic IP failover +* NFS over stunnel: For secure, encrypted network storage +* ZFS: For data integrity, encryption, and efficient snapshots +* `zrepl`: For continuous ZFS replication between nodes + +The result is a highly available, encrypted storage system that survives node failures while providing shared storage to all Kubernetes pods. + +Other than what was mentioned in the first post of this blog series, we aren't using HAST, but `zrepl` for data replication. Read more about it later in this blog post. + +## Additional storage capacity + +We add 1 TB of additional storage to each of the nodes (`f0`, `f1`, `f2`) in the form of an SSD drive. The Beelink mini PCs have enough space in the chassis for the extra space. + +[![./f3s-kubernetes-with-freebsd-part-6/drives.jpg](./f3s-kubernetes-with-freebsd-part-6/drives.jpg)](./f3s-kubernetes-with-freebsd-part-6/drives.jpg) + +Upgrading the storage was as easy as unscrewing, plugging the drive in, and then screwing it back together again. The procedure was uneventful! We're using two different SSD models (Samsung 870 EVO and Crucial BX500) to avoid simultaneous failures from the same manufacturing batch. + +We then create the `zdata` ZFS pool on all three nodes: + +```sh +paul@f0:~ % doas zpool create -m /data zdata /dev/ada1 +paul@f0:~ % zpool list +NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT +zdata 928G 12.1M 928G - - 0% 0% 1.00x ONLINE - +zroot 472G 29.0G 443G - - 0% 6% 1.00x ONLINE - + +paul@f0:/ % doas camcontrol devlist +<512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0) + at scbus1 target 0 lun 0 (pass1,ada1) +paul@f0:/ % +``` + +To verify that we have a different SSD on the second node (the third node has the same drive as the first): + +```sh +paul@f1:/ % doas camcontrol devlist +<512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0) + at scbus1 target 0 lun 0 (pass1,ada1) +``` + +## ZFS encryption keys + +ZFS native encryption requires encryption keys to unlock datasets. We need a secure method to store these keys that balances security with operational needs: + +* Security: Keys must not be stored on the same disks they encrypt +* Availability: Keys must be available at boot for automatic mounting +* Portability: Keys should be easily moved between systems for recovery + +Using USB flash drives as hardware key storage provides a convenient and elegant solution. The encrypted data is unreadable without physical access to the USB key, protecting against disk theft or improper disposal. In production environments, you may use enterprise key management systems; however, for a home lab, USB keys offer good security with minimal complexity. + +### UFS on USB keys + +We'll format the USB drives with UFS (Unix File System) rather than ZFS for simplicity. There is no need to use ZFS. + +Let's see the USB keys: + +[![USB keys](./f3s-kubernetes-with-freebsd-part-6/usbkeys1.jpg "USB keys")](./f3s-kubernetes-with-freebsd-part-6/usbkeys1.jpg) + +To verify that the USB key (flash disk) is there: + +``` +paul@f0:/ % doas camcontrol devlist +<512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0) + at scbus1 target 0 lun 0 (pass1,ada1) + at scbus2 target 0 lun 0 (da0,pass2) +paul@f0:/ % +``` + +Let's create the UFS file system and mount it (done on all three nodes `f0`, `f1` and `f2`): + +```sh +paul@f0:/ % doas newfs /dev/da0 +/dev/da0: 15000.0MB (30720000 sectors) block size 32768, fragment size 4096 + using 24 cylinder groups of 625.22MB, 20007 blks, 80128 inodes. + with soft updates +super-block backups (for fsck_ffs -b #) at: + 192, 1280640, 2561088, 3841536, 5121984, 6402432, 7682880, 8963328, 10243776, +11524224, 12804672, 14085120, 15365568, 16646016, 17926464, 19206912,k 20487360, +... + +paul@f0:/ % echo '/dev/da0 /keys ufs rw 0 2' | doas tee -a /etc/fstab +/dev/da0 /keys ufs rw 0 2 +paul@f0:/ % doas mkdir /keys +paul@f0:/ % doas mount /keys +paul@f0:/ % df | grep keys +/dev/da0 14877596 8 13687384 0% /keys +``` + +[![USB keys stuck in](./f3s-kubernetes-with-freebsd-part-6/usbkeys2.jpg "USB keys stuck in")](./f3s-kubernetes-with-freebsd-part-6/usbkeys2.jpg) + +### Generating encryption keys + +The following keys will later be used to encrypt the ZFS file systems. They will be stored on all three nodes, serving as a backup in case one of the keys is lost or corrupted. When we later replicate encrypted ZFS volumes from one node to another, the keys must also be available on the destination node. + +``` +paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:bhyve.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:bhyve.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:bhyve.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:zdata.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:zdata.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:zdata.key 32 +paul@f0:/keys % doas chown root * +paul@f0:/keys % doas chmod 400 * + +paul@f0:/keys % ls -l +total 20 +*r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:bhyve.key +*r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:bhyve.key +*r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:bhyve.key +*r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:zdata.key +*r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:zdata.key +*r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:zdata.key +```` + +After creation, these are copied to the other two nodes, `f1` and `f2`, into the `/keys` partition (I won't provide the commands here; create a tarball, copy it over, and extract it on the destination nodes). + +### Configuring `zdata` ZFS pool encryption + +Let's encrypt our `zdata` ZFS pool. We are not encrypting the whole pool, but everything within the `zdata/enc` data set: + +```sh +paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o \ + keylocation=file:///keys/`hostname`:zdata.key zdata/enc +paul@f0:/ % zfs list | grep zdata +zdata 836K 899G 96K /data +zdata/enc 200K 899G 200K /data/enc + +paul@f0:/keys % zfs get all zdata/enc | grep -E -i '(encryption|key)' +zdata/enc encryption aes-256-gcm - +zdata/enc keylocation file:///keys/f0.lan.buetow.org:zdata.key local +zdata/enc keyformat raw - +zdata/enc encryptionroot zdata/enc - +zdata/enc keystatus available - +```` + +All future data sets within `zdata/enc` will inherit the same encryption key. + +### Migrating Bhyve VMs to an encrypted `bhyve` ZFS volume + +We set up Bhyve VMs in a previous blog post. Their ZFS data sets rely on `zroot`, which is the default ZFS pool on the internal 512GB NVME drive. They aren't encrypted yet, so we encrypt the VM data sets as well now. To do so, we first shut down the VMs on all three nodes: + +```sh +paul@f0:/keys % doas vm stop rocky +Sending ACPI shutdown to rocky + +paul@f0:/keys % doas vm list +NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE +rocky default uefi 4 14G - Yes [1] Stopped +``` + +After this, we rename the unencrypted data set to `_old`, create a new encrypted data set, and also snapshot it as `@hamburger`. + +```sh +paul@f0:/keys % doas zfs rename zroot/bhyve zroot/bhyve_old +paul@f0:/keys % doas zfs set mountpoint=/mnt zroot/bhyve_old +paul@f0:/keys % doas zfs snapshot zroot/bhyve_old/rocky@hamburger + +paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o \ + keylocation=file:///keys/`hostname`:bhyve.key zroot/bhyve +paul@f0:/keys % doas zfs set mountpoint=/zroot/bhyve zroot/bhyve +paul@f0:/keys % doas zfs set mountpoint=/zroot/bhyve/rocky zroot/bhyve/rocky +``` + +Once done, we import the snapshot into the encrypted dataset and also copy some other metadata files from `vm-bhyve` back over. + +``` +paul@f0:/keys % doas zfs send zroot/bhyve_old/rocky@hamburger | \ + doas zfs recv zroot/bhyve/rocky +paul@f0:/keys % doas cp -Rp /mnt/.config /zroot/bhyve/ +paul@f0:/keys % doas cp -Rp /mnt/.img /zroot/bhyve/ +paul@f0:/keys % doas cp -Rp /mnt/.templates /zroot/bhyve/ +paul@f0:/keys % doas cp -Rp /mnt/.iso /zroot/bhyve/ +``` + +We also have to make encrypted ZFS data sets mount automatically on boot: + +```sh +paul@f0:/keys % doas sysrc zfskeys_enable=YES +zfskeys_enable: -> YES +paul@f0:/keys % doas vm init +paul@f0:/keys % doas reboot +. +. +. +paul@f0:~ % doas vm list +paul@f0:~ % doas vm list +NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE +rocky default uefi 4 14G 0.0.0.0:5900 Yes [1] Running (2265) +``` + +As you can see, the VM is running. This means the encrypted `zroot/bhyve` was mounted successfully after the reboot! Now we can destroy the old, unencrypted, and now unused bhyve dataset: + +```sh +paul@f0:~ % doas zfs destroy -R zroot/bhyve_old +``` + +To verify once again that `zroot/bhyve` and `zroot/bhyve/rocky` are now both encrypted, we run: + +```sh +paul@f0:~ % zfs get all zroot/bhyve | grep -E '(encryption|key)' +zroot/bhyve encryption aes-256-gcm - +zroot/bhyve keylocation file:///keys/f0.lan.buetow.org:bhyve.key local +zroot/bhyve keyformat raw - +zroot/bhyve encryptionroot zroot/bhyve - +zroot/bhyve keystatus available - + +paul@f0:~ % zfs get all zroot/bhyve/rocky | grep -E '(encryption|key)' +zroot/bhyve/rocky encryption aes-256-gcm - +zroot/bhyve/rocky keylocation none default +zroot/bhyve/rocky keyformat raw - +zroot/bhyve/rocky encryptionroot zroot/bhyve - +zroot/bhyve/rocky keystatus available - +``` + +## ZFS Replication with `zrepl` + +Data replication is the cornerstone of high availability. While CARP handles IP failover (see later in this post), we need continuous data replication to ensure the backup server has current data when it becomes active. Without replication, failover would result in data loss or require shared storage (like iSCSI), which introduces a single point of failure. + +### Understanding Replication Requirements + +Our storage system has different replication needs: + +* NFS data (`/data/nfs/k3svolumes`): Soon, it will contain active Kubernetes persistent volumes. Needs frequent replication (every minute) to minimise data loss during failover. +* VM data (`/zroot/bhyve/fedora`): Contains VM images that change less frequently. Can tolerate longer replication intervals (every 10 minutes). + +The 1-minute replication window is perfectly acceptable for my personal use cases. This isn't a high-frequency trading system or a real-time database—it's storage for personal projects, development work, and home lab experiments. Losing at most 1 minute of work in a disaster scenario is a reasonable trade-off for the reliability and simplicity of snapshot-based replication. Additionally, in the case of a "1 minute of data loss," I would likely still have the data available on the client side. + +Why use `zrepl` instead of HAST? While HAST (Highly Available Storage) is FreeBSD's native solution for high-availability storage and supports synchronous replication—thus eliminating the mentioned 1-minute window—I've chosen `zrepl` for several important reasons: + +* HAST can cause ZFS corruption: HAST operates at the block level and doesn't understand ZFS's transactional semantics. During failover, in-flight transactions can lead to corrupted zpools. I've experienced this firsthand (I am confident I have configured something wrong) - the automatic failover would trigger while ZFS was still writing, resulting in an unmountable pool. +* ZFS-aware replication: `zrepl` understands ZFS datasets and snapshots. It replicates at the dataset level, ensuring each snapshot is a consistent point-in-time copy. This is fundamentally safer than block-level replication. +* Snapshot history: With `zrepl`, you get multiple recovery points (every minute for NFS data in our setup). If corruption occurs, you can roll back to any previous snapshot. HAST only gives you the current state. +* Easier recovery: When something goes wrong with `zrepl`, you still have intact snapshots on both sides. With HAST, a corrupted primary often means a corrupted secondary as well. + +[FreeBSD HAST](https://wiki.freebsd.org/HighlyAvailableStorage) + +### Installing `zrepl` + +First, install `zrepl` on both hosts involved (we will replicate data from `f0` to `f1`): + +```sh +paul@f0:~ % doas pkg install -y zrepl +``` + +Then, we verify the pools and datasets on both hosts: + +```sh +# On f0 +paul@f0:~ % doas zpool list +NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT +zdata 928G 1.03M 928G - - 0% 0% 1.00x ONLINE - +zroot 472G 26.7G 445G - - 0% 5% 1.00x ONLINE - + +paul@f0:~ % doas zfs list -r zdata/enc +NAME USED AVAIL REFER MOUNTPOINT +zdata/enc 200K 899G 200K /data/enc + +# On f1 +paul@f1:~ % doas zpool list +NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT +zdata 928G 956K 928G - - 0% 0% 1.00x ONLINE - +zroot 472G 11.7G 460G - - 0% 2% 1.00x ONLINE - + +paul@f1:~ % doas zfs list -r zdata/enc +NAME USED AVAIL REFER MOUNTPOINT +zdata/enc 200K 899G 200K /data/enc +``` + +Since we have a WireGuard tunnel between `f0` and f1, we'll use TCP transport over the secure tunnel instead of SSH. First, check the WireGuard IP addresses: + +```sh +# Check WireGuard interface IPs +paul@f0:~ % ifconfig wg0 | grep inet + inet 192.168.2.130 netmask 0xffffff00 + +paul@f1:~ % ifconfig wg0 | grep inet + inet 192.168.2.131 netmask 0xffffff00 +``` + +Let's create a dedicated dataset for NFS data that will be replicated: + +```sh +# Create the nfsdata dataset that will hold all data exposed via NFS +paul@f0:~ % doas zfs create zdata/enc/nfsdata +``` + +Afterwards, we create the `zrepl` configuration on `f0`: + +```sh +paul@f0:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<'EOF' +global: + logging: + - type: stdout + level: info + format: human + +jobs: + - name: f0_to_f1_nfsdata + type: push + connect: + type: tcp + address: "192.168.2.131:8888" + filesystems: + "zdata/enc/nfsdata": true + send: + encrypted: true + snapshotting: + type: periodic + prefix: zrepl_ + interval: 1m + pruning: + keep_sender: + - type: last_n + count: 10 + keep_receiver: + - type: last_n + count: 10 + + - name: f0_to_f1_fedora + type: push + connect: + type: tcp + address: "192.168.2.131:8888" + filesystems: + "zroot/bhyve/fedora": true + send: + encrypted: true + snapshotting: + type: periodic + prefix: zrepl_ + interval: 10m + pruning: + keep_sender: + - type: last_n + count: 10 + keep_receiver: + - type: last_n + count: 10 +EOF +``` + + We're using two separate replication jobs with different intervals: + +* `f0_to_f1_nfsdata`: Replicates NFS data every minute for faster failover recovery +* `f0_to_f1_fedora`: Replicates Fedora VM every ten minutes (less critical) + +The Fedora VM is only used for development purposes, so it doesn't require as frequent replication as the NFS data. It's off-topic to this blog series, but it showcases, hows `zrepl`'s flexibility in handling different datasets with varying replication needs. + +Furthermore: + +* We're specifically replicating `zdata/enc/nfsdata` instead of the entire `zdata/enc` dataset. This dedicated dataset will contain all the data we later want to expose via NFS, keeping a clear separation between replicated NFS data and other local encrypted data. +* The `send: encrypted: false` option turns off ZFS native encryption for the replication stream. Since we're using a WireGuard tunnel between `f0` and `f1`, the data is already encrypted in transit. Disabling ZFS stream encryption reduces CPU overhead and improves replication performance. + +### Configuring `zrepl` on `f1` (sink) + +On `f1` (the sink, meaning it's the node receiving the replication data), we configure `zrepl` to receive the data as follows: + +```sh +# First, create a dedicated sink dataset +paul@f1:~ % doas zfs create zdata/sink + +paul@f1:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<'EOF' +global: + logging: + - type: stdout + level: info + format: human + +jobs: + - name: sink + type: sink + serve: + type: tcp + listen: "192.168.2.131:8888" + clients: + "192.168.2.130": "f0" + recv: + placeholder: + encryption: inherit + root_fs: "zdata/sink" +EOF +``` + +### Enabling and starting `zrepl` services + +We then enable and start `zrepl` on both hosts via: + +```sh +# On f0 +paul@f0:~ % doas sysrc zrepl_enable=YES +zrepl_enable: -> YES +paul@f0:~ % doas service `zrepl` start +Starting zrepl. + +# On f1 +paul@f1:~ % doas sysrc zrepl_enable=YES +zrepl_enable: -> YES +paul@f1:~ % doas service `zrepl` start +Starting zrepl. +``` + +To check the replication status, we run: + +```sh +# On f0, check `zrepl` status (use raw mode for non-tty) +paul@f0:~ % doas pkg install jq +paul@f0:~ % doas zrepl status --mode raw | grep -A2 "Replication" | jq . +"Replication":{"StartAt":"2025-07-01T22:31:48.712143123+03:00"... + +# Check if services are running +paul@f0:~ % doas service zrepl status +zrepl is running as pid 2649. + +paul@f1:~ % doas service zrepl status +zrepl is running as pid 2574. + +# Check for `zrepl` snapshots on source +paul@f0:~ % doas zfs list -t snapshot -r zdata/enc | grep zrepl +zdata/enc@zrepl_20250701_193148_000 0B - 176K - + +# On f1, verify the replicated datasets +paul@f1:~ % doas zfs list -r zdata | grep f0 +zdata/f0 576K 899G 200K none +zdata/f0/zdata 376K 899G 200K none +zdata/f0/zdata/enc 176K 899G 176K none + +# Check replicated snapshots on f1 +paul@f1:~ % doas zfs list -t snapshot -r zdata | grep zrepl +zdata/f0/zdata/enc@zrepl_20250701_193148_000 0B - 176K - +zdata/f0/zdata/enc@zrepl_20250701_194148_000 0B - 176K - +. +. +. +``` + +### Monitoring replication + +You can monitor the replication progress with: + +```sh +paul@f0:~ % doas zrepl status +``` + +[![zrepl status](./f3s-kubernetes-with-freebsd-part-6/zrepl.png "zrepl status")](./f3s-kubernetes-with-freebsd-part-6/zrepl.png) + +With this setup, both `zdata/enc/nfsdata` and `zroot/bhyve/fedora` on `f0` will be automatically replicated to `f1` every 1 minute (or 10 minutes in the case of the Fedora VM), with encrypted snapshots preserved on both sides. The pruning policy ensures that we keep the last 10 snapshots while managing disk space efficiently. + +The replicated data appears on `f1` under `zdata/sink/` with the source host and dataset hierarchy preserved: + +* `zdata/enc/nfsdata` → `zdata/sink/f0/zdata/enc/nfsdata` +* `zroot/bhyve/fedora` → `zdata/sink/f0/zroot/bhyve/fedora` + +This is by design - `zrepl` preserves the complete path from the source to ensure there are no conflicts when replicating from multiple sources. + +### Verifying replication after reboot + +The `zrepl` service is configured to start automatically at boot. After rebooting both hosts: + +```sh +paul@f0:~ % uptime +11:17PM up 1 min, 0 users, load averages: 0.16, 0.06, 0.02 + +paul@f0:~ % doas service `zrepl` status +zrepl is running as pid 2366. + +paul@f1:~ % doas service `zrepl` status +zrepl is running as pid 2309. + +# Check that new snapshots are being created and replicated +paul@f0:~ % doas zfs list -t snapshot | grep `zrepl` | tail -2 +zdata/enc/nfsdata@zrepl_20250701_202530_000 0B - 200K - +zroot/bhyve/fedora@zrepl_20250701_202530_000 0B - 2.97G - +. +. +. + +paul@f1:~ % doas zfs list -t snapshot -r zdata/sink | grep 202530 +zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250701_202530_000 0B - 176K - +zdata/sink/f0/zroot/bhyve/fedora@zrepl_20250701_202530_000 0B - 2.97G - +. +. +. +``` + +The timestamps confirm that replication resumed automatically after the reboot, ensuring continuous data protection. We can also write a test file to the NFS data directory on `f0` and verify whether it appears on `f1` after a minute. + +### Understanding Failover Limitations and Design Decisions + +Our system intentionally fails over to a read-only copy of the replica in the event of the primary's failure. This is due to the nature of `zrepl`, which only replicates data in one direction. If we mount the data set on the sink node in read-write mode, it would cause the ZFS dataset to diverge from the original, and the replication would break. It can still be mounted read-write on the sink node in case of a genuine issue on the primary node, but that step is left intentionally manual. Therefore, we don't need to fix the replication later on manually. + +So in summary: + +* Split-brain prevention: Automatic failover to a read-write copy can cause both nodes to become active simultaneously if network communication fails. This leads to data divergence that's extremely difficult to resolve. +* False positive protection: Temporary network issues or high load can trigger unwanted failovers. Manual intervention ensures that failovers occur only when truly necessary. +* Data integrity over availability: For storage systems, data consistency is paramount. A few minutes of downtime is preferable to data corruption in this specific use case. +* Simplified recovery: With manual failover, you always know which dataset is authoritative, making recovery more straightforward. + +### Mounting the NFS datasets + +To make the NFS data accessible on both nodes, we need to mount it. On `f0`, this is straightforward: + +```sh +# On f0 - set mountpoint for the primary nfsdata +paul@f0:~ % doas zfs set mountpoint=/data/nfs zdata/enc/nfsdata +paul@f0:~ % doas mkdir -p /data/nfs + +# Verify it's mounted +paul@f0:~ % df -h /data/nfs +Filesystem Size Used Avail Capacity Mounted on +zdata/enc/nfsdata 899G 204K 899G 0% /data/nfs +``` + +On `f1`, we need to handle the encryption key and mount the standby copy: + +```sh +# On f1 - first check encryption status +paul@f1:~ % doas zfs get keystatus zdata/sink/f0/zdata/enc/nfsdata +NAME PROPERTY VALUE SOURCE +zdata/sink/f0/zdata/enc/nfsdata keystatus unavailable - + +# Load the encryption key (using f0's key stored on the USB) +paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \ + zdata/sink/f0/zdata/enc/nfsdata + +# Set mountpoint and mount (same path as f0 for easier failover) +paul@f1:~ % doas mkdir -p /data/nfs +paul@f1:~ % doas zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata +paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata + +# Make it read-only to prevent accidental writes that would break replication +paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata + +# Verify +paul@f1:~ % df -h /data/nfs +Filesystem Size Used Avail Capacity Mounted on +zdata/sink/f0/zdata/enc/nfsdata 896G 204K 896G 0% /data/nfs +``` + +Note: The dataset is mounted at the same path (`/data/nfs`) on both hosts to simplify failover procedures. The dataset on `f1` is set to `readonly=on` to prevent accidental modifications, which, as mentioned earlier, would break replication. If we did, replication from `f0` to `f1` would fail like this: + +> cannot receive incremental stream: destination zdata/sink/f0/zdata/enc/nfsdata has been modified since most recent snapshot + +To fix a broken replication after accidental writes, we can do: + +```sh +# Option 1: Rollback to the last common snapshot (loses local changes) +paul@f1:~ % doas zfs rollback zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250701_204054_000 + +# Option 2: Make it read-only to prevent accidents again +paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata +``` + +And replication should work again! + +### Troubleshooting: Files not appearing in replication + +If you write files to `/data/nfs/` on `f0` but they don't appear on `f1`, check if the dataset is mounted on `f0`? + +```sh +paul@f0:~ % doas zfs list -o name,mountpoint,mounted | grep nfsdata +zdata/enc/nfsdata /data/nfs yes +``` + +If it shows `no`, the dataset isn't mounted! This means files are being written to the root filesystem, not ZFS. Next, we should check whether the encryption key is loaded: + +```sh +paul@f0:~ % doas zfs get keystatus zdata/enc/nfsdata +NAME PROPERTY VALUE SOURCE +zdata/enc/nfsdata keystatus available - +# If "unavailable", load the key: +paul@f0:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata +paul@f0:~ % doas zfs mount zdata/enc/nfsdata +``` + +You can also verify that files are in the snapshot (not just the directory): + +```sh +paul@f0:~ % ls -la /data/nfs/.zfs/snapshot/zrepl_*/ +``` + +This issue commonly occurs after a reboot if the encryption keys aren't configured to load automatically. + +### Configuring automatic key loading on boot + +To ensure all additional encrypted datasets are mounted automatically after reboot as well, we do: + +```sh +# On f0 - configure all encrypted datasets +paul@f0:~ % doas sysrc zfskeys_enable=YES +zfskeys_enable: YES -> YES +paul@f0:~ % doas sysrc zfskeys_datasets="zdata/enc zdata/enc/nfsdata zroot/bhyve" +zfskeys_datasets: -> zdata/enc zdata/enc/nfsdata zroot/bhyve + +# Set correct key locations for all datasets +paul@f0:~ % doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata + +# On f1 - include the replicated dataset +paul@f1:~ % doas sysrc zfskeys_enable=YES +zfskeys_enable: YES -> YES +paul@f1:~ % doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata" +zfskeys_datasets: -> zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata + +# Set key location for replicated dataset +paul@f1:~ % doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key zdata/sink/f0/zdata/enc/nfsdata +``` + +Important notes: + +* Each encryption root needs its own key load entry +* The replicated dataset on `f1` uses the same encryption key as the source on `f0` +* Always verify datasets are mounted after reboot with `zfs list -o name,mounted` +* Critical: Always ensure the replicated dataset on `f1` remains read-only with `doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata` + +## CARP (Common Address Redundancy Protocol) + +High availability is crucial for storage systems. If the storage server goes down, all NFS clients (which will also be Kubernetes pods later on in this series) lose access to their persistent data. CARP provides a solution by creating a virtual IP address that automatically migrates to a different server during failures. This means that clients point to that VIP for NFS mounts and are always contacting the current primary node. + +### How CARP Works + +In our case, CARP allows two hosts (`f0` and `f1`) to share a virtual IP address (VIP). The hosts communicate using multicast to elect a MASTER, while the other remain as BACKUP. When the MASTER fails, the BACKUP automatically promotes itself, and the VIP is reassigned to the new MASTER. This happens within seconds. + +Key benefits for our storage system: + +* Automatic failover: No manual intervention is required for basic failures, although there are a few limitations. The backup will have read-only access to the available data by default, as we have already learned. +* Transparent to clients: Pods continue using the same IP address +* Works with `stunnel`: Behind the VIP, there will be a `stunnel` process running, which ensures encrypted connections follow the active server. + +[FreeBSD CARP](https://docs-archive.freebsd.org/doc/13.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html) +[Stunnel](https://www.stunnel.org/) + +### Configuring CARP + +First, we add the CARP configuration to `/etc/rc.conf` on both `f0` and `f1`: + +```sh +# The virtual IP 192.168.1.138 will float between f0 and f1 +ifconfig_re0_alias0="inet vhid 1 pass testpass alias 192.168.1.138/32" +``` + +Whereas: + +* `vhid 1`: Virtual Host ID - must match on all CARP members +* `pass testpass`: Password for CARP authentication (if you follow this, use a different password!) +* `alias 192.168.1.138/32`: The virtual IP address with a /32 netmask + +Next, update `/etc/hosts` on all nodes (`f0`, `f1`, `f2`, `r0`, `r1`, `r2`) to resolve the VIP hostname: + +``` +192.168.1.138 f3s-storage-ha f3s-storage-ha.lan f3s-storage-ha.lan.buetow.org +``` + +This allows clients to connect to `f3s-storage-ha` regardless of which physical server is currently the MASTER. + +### CARP State Change Notifications + +To correctly manage services during failover, we need to detect CARP state changes. FreeBSD's devd system can notify us when CARP transitions between MASTER and BACKUP states. + +Add this to `/etc/devd.conf` on both `f0` and `f1`: + +```sh +paul@f0:~ % cat <