diff options
| author | Paul Buetow <paul@buetow.org> | 2025-07-12 22:46:46 +0300 |
|---|---|---|
| committer | Paul Buetow <paul@buetow.org> | 2025-07-12 22:46:46 +0300 |
| commit | e15aa5447150d31e86590da8b2ebd8e62acadcb8 (patch) | |
| tree | 083acb60cc9ef89ccc36cb99a72aabc27b6a3f2b /gemfeed | |
| parent | 943206251faf4b50c33a1230804206366a2bfa73 (diff) | |
Update content for html
Diffstat (limited to 'gemfeed')
| -rw-r--r-- | gemfeed/2025-06-22-task-samurai.html | 28 | ||||
| -rw-r--r-- | gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.html | 2606 | ||||
| -rw-r--r-- | gemfeed/atom.xml | 30 | ||||
| -rw-r--r-- | gemfeed/f3s-kubernetes-with-freebsd-part-6/drives.jpg | bin | 0 -> 114358 bytes | |||
| -rw-r--r-- | gemfeed/f3s-kubernetes-with-freebsd-part-6/usbkeys1.jpg | bin | 0 -> 301121 bytes | |||
| -rw-r--r-- | gemfeed/f3s-kubernetes-with-freebsd-part-6/usbkeys2.jpg | bin | 0 -> 356228 bytes |
6 files changed, 2418 insertions, 246 deletions
diff --git a/gemfeed/2025-06-22-task-samurai.html b/gemfeed/2025-06-22-task-samurai.html index 24c625d5..ffba945a 100644 --- a/gemfeed/2025-06-22-task-samurai.html +++ b/gemfeed/2025-06-22-task-samurai.html @@ -27,7 +27,7 @@ <li>⇢ <a href='#where-and-how-to-get-it'>Where and how to get it</a></li> <li>⇢ <a href='#lessons-learned-from-building-task-samurai-with-agentic-coding'>Lessons learned from building Task Samurai with agentic coding</a></li> <li>⇢ ⇢ <a href='#developer-workflow'>Developer workflow</a></li> -<li>⇢ ⇢ <a href='#how-it-went-down'>How it went down</a></li> +<li>⇢ ⇢ <a href='#how-it-went'>How it went</a></li> <li>⇢ ⇢ <a href='#what-went-wrong'>What went wrong</a></li> <li>⇢ ⇢ <a href='#patterns-that-helped'>Patterns that helped</a></li> <li>⇢ ⇢ <a href='#what-i-learned-using-agentic-coding'>What I learned using agentic coding</a></li> @@ -44,6 +44,7 @@ <h3 style='display: inline' id='why-does-this-exist'>Why does this exist?</h3><br /> <br /> <span>I wanted to tinker with agentic coding. This project was implemented entirely using OpenAI Codex. (After this blog post was published, I also used the Claude Code CLI.)</span><br /> +<br /> <ul> <li>I wanted a faster UI for Taskwarrior than other options, like Vit, which is Python-based.</li> <li>I wanted something built with Bubble Tea, but I never had time to dive deep into it.</li> @@ -71,17 +72,19 @@ <br /> <h3 style='display: inline' id='developer-workflow'>Developer workflow</h3><br /> <br /> -<span>I was trying out OpenAI Codex because I regularly run out of Claude Code CLI (another agentic coding tool I am trying out currently) credits (it still happens!), but Codex was still available to me. So, I seized the opportunity to push agentic coding a bit more using another platform.</span><br /> +<span>I was trying out OpenAI Codex because I regularly run out of Claude Code CLI (another agentic coding tool I am currently trying out) credits (it still happens!), but Codex was still available to me. So, I took the opportunity to push agentic coding a bit further with another platform.</span><br /> <br /> <span>I didn't really love the web UI you have to use for Codex, as I usually live in the terminal. But this is all I have for Codex for now, and I thought I'd give it a try regardless. The web UI is simple and pretty straightforward. There's also a Codex CLI one could use directly in the terminal, but I didn't get it working. I will try again soon.</span><br /> <br /> +<span class='quote'>Update: Codex CLI now works for me, after OpenAI released a new version!</span><br /> +<br /> <span>For every task given to Codex, it spins up its own container. From there, you can drill down and watch what it is doing. At the end, the result (in the form of a code diff) will be presented. From there, you can make suggestions about what else to change in the codebase. What I found inconvenient is that for every additional change, there's an overhead because Codex has to spin up a container and bootstrap the entire development environment again, which adds extra delay. That could be eliminated by setting up predefined custom containers, but that feature still seems somewhat limited.</span><br /> <br /> -<span>Once satisfied, you can ask Codex to create a GitHub PR; from there, you can merge it and then pull it to your local laptop or workstation to test the changes again. I found myself looping a lot around the Codex UI, GitHub PRs, and local checkouts.</span><br /> +<span>Once satisfied, you can ask Codex to create a GitHub PR (too bad only GitHub is supported and no other Git hosters); from there, you can merge it and then pull it to your local laptop or workstation to test the changes again. I found myself looping a lot around the Codex UI, GitHub PRs, and local checkouts. </span><br /> <br /> -<h3 style='display: inline' id='how-it-went-down'>How it went down</h3><br /> +<h3 style='display: inline' id='how-it-went'>How it went</h3><br /> <br /> -<span>Task Samurai's codebase came together quickly: the entire Git history spans from June 19 to 22, 2025, culminating in 179 commits. Here are the broad strokes:</span><br /> +<span>Task Samurai's codebase came together quickly: the entire Git history spans from June 19 to 22, 2025, culminating in 179 commits:</span><br /> <br /> <ul> <li>June 19: Scaffolded the Go boilerplate, set up tests, integrated the Bubble Tea UI framework, and got the first table views showing up.</li> @@ -95,7 +98,7 @@ <br /> <h3 style='display: inline' id='what-went-wrong'>What went wrong</h3><br /> <br /> -<span>Going agentic isn't all smooth sailing. Here are the hiccups I ran into, plus a few hard-earned lessons:</span><br /> +<span>Going agentic isn't all smooth. Here are the hiccups I ran into, plus a few lessons:</span><br /> <br /> <ul> <li>Merge Floods: Every minor feature or fix existed on its branch, so merging was a constant process. It kept progress flowing but also drowned the committed history in noise and the occasional conflict. I found this to be an issue with OpenAI's Codex in particular. Not so much with other agentic coding tools like Claude Code CLI (not covered in this blog post.)</li> @@ -111,29 +114,28 @@ <li>Tests Matter: A solid base of unit tests for task manipulations kept things from breaking entirely when experimenting.</li> <li>Live Documentation: Documentation, such as the README, is updated regularly to reflect all the hotkey and feature changes.</li> </ul><br /> +<span>Maybe a better approach would have been to design the whole application from scratch before letting Codix do any of the coding. I will try that with my next toy project.</span><br /> <br /> <h3 style='display: inline' id='what-i-learned-using-agentic-coding'>What I learned using agentic coding</h3><br /> <br /> -<span>Stepping into agentic coding with Codex as my "pair programmer" was a genuine shift. I learned a lot—not just about automating code generation, but also about how you have to tightly steer, guide, and audit every line as things move at breakneck speed. I must admit, I sometimes lost track of what all the generated code was actually doing. But as the features seemed to work after a few iterations, I was satisfied—which is a bit concerning. Imagine if I approved a PR for a production-grade deployment without fully understanding what it was doing (and not a toy project like in this post).</span><br /> -<br /> -<span>Discussing requirements with Codex forced me to clarify features and spot logical pitfalls earlier. All those fast iterations meant I was constantly coaxing more helpful, less ambiguous code out of the model—making me rethink how to break features into clear, testable steps.</span><br /> +<span>Stepping into agentic coding with Codex as my "pair programmer" was a big shift. I learned a lot—not just about automating code generation, but also about how you have to tightly steer, guide, and audit every line as things move at high speed. I must admit, I sometimes lost track of what all the generated code was actually doing. But as the features seemed to work after a few iterations, I was satisfied—which is a bit concerning. Imagine if I approved a PR for a production-grade deployment without fully understanding what it was doing (and not a toy project like in this post).</span><br /> <br /> <h3 style='display: inline' id='how-much-time-did-i-save'>how much time did I save?</h3><br /> <br /> -<span>Did it buy me speed? Let's do some back-of-the-envelope math:</span><br /> +<span>Did it buy me speed? </span><br /> <br /> <ul> <li>Say each commit takes Codex 5 minutes to generate, and you need to review/guide 179 commits = about _6 hours of active development_.</li> <li>If you coded it all yourself, including all the bug fixes, features, design, and documentation, you might spend _10–20 hours_.</li> -<li>That's a couple of days potential savings.</li> +<li>That's a couple of days of potential savings—and I am by no means an expert in agentic coding, since this was my first completed agentic coding project.</li> </ul><br /> <h2 style='display: inline' id='conclusion'>Conclusion</h2><br /> <br /> -<span>Building Task Samurai with agentic coding was a wild ride—rapid feature growth, plenty of churns, countless fast fixes, and more merge commits I'd expected. Keep the iterations short (or maybe in my next experiment, much larger, with better and more complete design before generating a single line of code), keep tests and documentation concise, and review and refine for final polish at the end. Even with the bumps along the way, shipping a polished terminal UI in days instead of weeks is a testament to the raw power (and some hazards) of agentic development.</span><br /> +<span>Building Task Samurai with agentic coding was a wild ride—rapid feature growth, countless fast fixes, and more merge commits I'd expected. Keep the iterations short (or maybe in my next experiment, much larger, with better and more complete design before generating a single line of code), keep tests and documentation concise, and review and refine for final polish at the end. Even with the bumps along the way, shipping a polished terminal UI in days instead of weeks is a testament to the power of agentic development.</span><br /> <br /> <span>Am I an agentic coding expert now? I don't think so. There are still many things to learn, and the landscape is constantly evolving.</span><br /> <br /> -<span>While working on Task Samurai, there were times I genuinely missed manual coding and the satisfaction that comes from writing every line yourself, debugging issues manually, and crafting solutions from scratch. However, this is the direction in which the industry seems to be shifting, unfortunately. If applied correctly, AI will boost performance, and if you don't use AI, your next performance review may be awkward.</span><br /> +<span>While working on Task Samurai, there were times I missed manual coding and the satisfaction that comes from writing every line yourself, debugging issues manually, and crafting solutions from scratch. However, this is the direction in which the industry seems to be shifting, unfortunately. If applied correctly, AI will boost performance, and if you don't use AI, your next performance review may be awkward.</span><br /> <br /> <span>Personally, I am not sure whether I like where the industry is going with agentic coding. I love "traditional" coding, and with agentic coding you operate at a higher level and don't interact directly with code as often, which I would miss. I think that in the future, designing, reviewing, and being able to read and understand code will be more important than writing code by hand.</span><br /> <br /> diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.html b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.html index 6d4f42d9..44e4e743 100644 --- a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.html +++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.html @@ -30,40 +30,188 @@ <ul> <li><a href='#f3s-kubernetes-with-freebsd---part-6-storage'>f3s: Kubernetes with FreeBSD - Part 6: Storage</a></li> <li>⇢ <a href='#introduction'>Introduction</a></li> +<li>⇢ <a href='#additional-storage-capacity'>Additional storage capacity</a></li> <li>⇢ <a href='#zfs-encryption-keys'>ZFS encryption keys</a></li> <li>⇢ ⇢ <a href='#ufs-on-usb-keys'>UFS on USB keys</a></li> <li>⇢ ⇢ <a href='#generating-encryption-keys'>Generating encryption keys</a></li> -<li>⇢ ⇢ <a href='#configuring-zdata-zfs-pool-and-encryption'>Configuring <span class='inlinecode'>zdata</span> ZFS pool and encryption</a></li> +<li>⇢ ⇢ <a href='#configuring-zdata-zfs-pool-encryption'>Configuring <span class='inlinecode'>zdata</span> ZFS pool encryption</a></li> <li>⇢ ⇢ <a href='#migrating-bhyve-vms-to-encrypted-bhyve-zfs-volume'>Migrating Bhyve VMs to encrypted <span class='inlinecode'>bhyve</span> ZFS volume</a></li> -<li>⇢ <a href='#carp'>CARP</a></li> <li>⇢ <a href='#zfs-replication-with-zrepl'>ZFS Replication with zrepl</a></li> -<li>⇢ ⇢ <a href='#why-zrepl-instead-of-hast'>Why zrepl instead of HAST?</a></li> +<li>⇢ ⇢ <a href='#understanding-replication-requirements'>Understanding Replication Requirements</a></li> +<li>⇢ ⇢ <a href='#why-zrepl-instead-of-hast'>Why <span class='inlinecode'>zrepl</span> instead of HAST?</a></li> <li>⇢ ⇢ <a href='#installing-zrepl'>Installing zrepl</a></li> <li>⇢ ⇢ <a href='#checking-zfs-pools'>Checking ZFS pools</a></li> -<li>⇢ ⇢ <a href='#configuring-zrepl-with-wireguard-tunnel'>Configuring zrepl with WireGuard tunnel</a></li> -<li>⇢ ⇢ <a href='#configuring-zrepl-on-f0-source'>Configuring zrepl on f0 (source)</a></li> -<li>⇢ ⇢ <a href='#configuring-zrepl-on-f1-sink'>Configuring zrepl on f1 (sink)</a></li> -<li>⇢ ⇢ <a href='#enabling-and-starting-zrepl-services'>Enabling and starting zrepl services</a></li> +<li>⇢ ⇢ <a href='#configuring-zrepl-with-wireguard-tunnel'>Configuring <span class='inlinecode'>zrepl</span> with WireGuard tunnel</a></li> +<li>⇢ ⇢ <a href='#configuring-zrepl-on-f0-source'>Configuring <span class='inlinecode'>zrepl</span> on f0 (source)</a></li> +<li>⇢ ⇢ <a href='#configuring-zrepl-on-f1-sink'>Configuring <span class='inlinecode'>zrepl</span> on <span class='inlinecode'>f1</span> (sink)</a></li> +<li>⇢ ⇢ <a href='#enabling-and-starting-zrepl-services'>Enabling and starting <span class='inlinecode'>zrepl</span> services</a></li> <li>⇢ ⇢ <a href='#verifying-replication'>Verifying replication</a></li> <li>⇢ ⇢ <a href='#monitoring-replication'>Monitoring replication</a></li> -<li>⇢ ⇢ <a href='#a-note-about-the-bhyve-vm-replication'>A note about the Bhyve VM replication</a></li> -<li>⇢ ⇢ <a href='#quick-status-check-commands'>Quick status check commands</a></li> <li>⇢ ⇢ <a href='#verifying-replication-after-reboot'>Verifying replication after reboot</a></li> -<li>⇢ ⇢ <a href='#important-note-about-failover-limitations'>Important note about failover limitations</a></li> +<li>⇢ ⇢ <a href='#understanding-failover-limitations-and-design-decisions'>Understanding Failover Limitations and Design Decisions</a></li> +<li>⇢ <a href='#-why-manual-failover'>⇢# Why Manual Failover?</a></li> +<li>⇢ <a href='#-current-failover-process'>⇢# Current Failover Process</a></li> <li>⇢ ⇢ <a href='#mounting-the-nfs-datasets'>Mounting the NFS datasets</a></li> <li>⇢ ⇢ <a href='#failback-scenario-syncing-changes-from-f1-back-to-f0'>Failback scenario: Syncing changes from f1 back to f0</a></li> <li>⇢ ⇢ <a href='#testing-the-failback-scenario'>Testing the failback scenario</a></li> +<li>⇢ ⇢ <a href='#troubleshooting-files-not-appearing-in-replication'>Troubleshooting: Files not appearing in replication</a></li> +<li>⇢ ⇢ <a href='#configuring-automatic-key-loading-on-boot'>Configuring automatic key loading on boot</a></li> +<li>⇢ ⇢ <a href='#troubleshooting-replication-broken-due-to-modified-destination'>Troubleshooting: Replication broken due to modified destination</a></li> +<li>⇢ ⇢ <a href='#forcing-a-full-resync'>Forcing a full resync</a></li> +<li>⇢ <a href='#carp-common-address-redundancy-protocol'>CARP (Common Address Redundancy Protocol)</a></li> +<li>⇢ ⇢ <a href='#how-carp-works'>How CARP Works</a></li> +<li>⇢ ⇢ <a href='#configuring-carp'>Configuring CARP</a></li> +<li>⇢ ⇢ <a href='#carp-state-change-notifications'>CARP State Change Notifications</a></li> +<li>⇢ <a href='#future-storage-explorations'>Future Storage Explorations</a></li> +<li>⇢ ⇢ <a href='#minio-for-s3-compatible-object-storage'>MinIO for S3-Compatible Object Storage</a></li> +<li>⇢ ⇢ <a href='#moosefs-for-distributed-high-availability'>MooseFS for Distributed High Availability</a></li> +<li>⇢ <a href='#nfs-server-configuration'>NFS Server Configuration</a></li> +<li>⇢ ⇢ <a href='#setting-up-nfs-on-f0-primary'>Setting up NFS on f0 (Primary)</a></li> +<li>⇢ ⇢ <a href='#configuring-stunnel-for-nfs-encryption-with-carp-failover'>Configuring Stunnel for NFS Encryption with CARP Failover</a></li> +<li>⇢ <a href='#-why-not-native-nfs-over-tls'>⇢# Why Not Native NFS over TLS?</a></li> +<li>⇢ <a href='#-stunnel-architecture-with-carp'>⇢# Stunnel Architecture with CARP</a></li> +<li>⇢ <a href='#-creating-a-certificate-authority-for-client-authentication'>⇢# Creating a Certificate Authority for Client Authentication</a></li> +<li>⇢ <a href='#-install-and-configure-stunnel-on-f0'>⇢# Install and Configure Stunnel on f0</a></li> +<li>⇢ ⇢ <a href='#setting-up-nfs-on-f1-standby'>Setting up NFS on f1 (Standby)</a></li> +<li>⇢ ⇢ <a href='#how-stunnel-works-with-carp'>How Stunnel Works with CARP</a></li> +<li>⇢ ⇢ <a href='#carp-control-script-for-clean-failover'>CARP Control Script for Clean Failover</a></li> +<li>⇢ ⇢ <a href='#carp-management-script'>CARP Management Script</a></li> +<li>⇢ ⇢ <a href='#automatic-failback-after-reboot'>Automatic Failback After Reboot</a></li> +<li>⇢ <a href='#-why-automatic-failback'>⇢# Why Automatic Failback?</a></li> +<li>⇢ <a href='#-the-auto-failback-script'>⇢# The Auto-Failback Script</a></li> +<li>⇢ <a href='#-setting-up-the-marker-file'>⇢# Setting Up the Marker File</a></li> +<li>⇢ <a href='#-configuring-cron'>⇢# Configuring Cron</a></li> +<li>⇢ <a href='#-managing-automatic-failback'>⇢# Managing Automatic Failback</a></li> +<li>⇢ <a href='#-how-it-works'>⇢# How It Works</a></li> +<li>⇢ ⇢ <a href='#verifying-stunnel-and-carp-status'>Verifying Stunnel and CARP Status</a></li> +<li>⇢ ⇢ <a href='#verifying-nfs-exports'>Verifying NFS Exports</a></li> +<li>⇢ ⇢ <a href='#client-configuration-for-stunnel'>Client Configuration for Stunnel</a></li> +<li>⇢ <a href='#-preparing-client-certificates'>⇢# Preparing Client Certificates</a></li> +<li>⇢ <a href='#-configuring-rocky-linux-clients-r0-r1-r2'>⇢# Configuring Rocky Linux Clients (r0, r1, r2)</a></li> +<li>⇢ ⇢ <a href='#testing-nfs-mount-with-stunnel'>Testing NFS Mount with Stunnel</a></li> +<li>⇢ ⇢ <a href='#important-encryption-keys-for-replicated-datasets'>Important: Encryption Keys for Replicated Datasets</a></li> +<li>⇢ ⇢ <a href='#nfs-failover-with-carp-and-stunnel'>NFS Failover with CARP and Stunnel</a></li> +<li>⇢ ⇢ <a href='#testing-carp-failover'>Testing CARP Failover</a></li> +<li>⇢ ⇢ <a href='#handling-stale-file-handles-after-failover'>Handling Stale File Handles After Failover</a></li> +<li>⇢ ⇢ <a href='#complete-failover-test'>Complete Failover Test</a></li> +<li>⇢ ⇢ <a href='#verifying-replication-status'>Verifying Replication Status</a></li> +<li>⇢ ⇢ <a href='#post-reboot-verification'>Post-Reboot Verification</a></li> +<li>⇢ ⇢ <a href='#integration-with-kubernetes'>Integration with Kubernetes</a></li> +<li>⇢ ⇢ <a href='#security-benefits-of-stunnel-with-client-certificates'>Security Benefits of Stunnel with Client Certificates</a></li> +<li>⇢ ⇢ <a href='#laptopworkstation-access'>Laptop/Workstation Access</a></li> +<li>⇢ <a href='#-important-nfsv4-and-stunnel-on-newer-linux-clients'>⇢# Important: NFSv4 and Stunnel on Newer Linux Clients</a></li> +<li>⇢ <a href='#mounting-nfs-on-rocky-linux-9'>Mounting NFS on Rocky Linux 9</a></li> +<li>⇢ ⇢ <a href='#installing-and-configuring-nfs-clients-on-r0-r1-and-r2'>Installing and Configuring NFS Clients on r0, r1, and r2</a></li> +<li>⇢ ⇢ <a href='#configuring-stunnel-client-on-all-nodes'>Configuring Stunnel Client on All Nodes</a></li> +<li>⇢ ⇢ <a href='#setting-up-nfs-mounts'>Setting Up NFS Mounts</a></li> +<li>⇢ ⇢ <a href='#comprehensive-nfs-mount-testing'>Comprehensive NFS Mount Testing</a></li> +<li>⇢ <a href='#-test-1-verify-mount-status-on-all-nodes'>⇢# Test 1: Verify Mount Status on All Nodes</a></li> +<li>⇢ <a href='#-test-2-verify-stunnel-connectivity'>⇢# Test 2: Verify Stunnel Connectivity</a></li> +<li>⇢ <a href='#-test-3-file-creation-and-visibility-test'>⇢# Test 3: File Creation and Visibility Test</a></li> +<li>⇢ <a href='#-test-4-verify-files-on-storage-servers'>⇢# Test 4: Verify Files on Storage Servers</a></li> +<li>⇢ <a href='#-test-5-performance-and-concurrent-access-test'>⇢# Test 5: Performance and Concurrent Access Test</a></li> +<li>⇢ <a href='#-test-6-directory-operations-test'>⇢# Test 6: Directory Operations Test</a></li> +<li>⇢ <a href='#-test-7-permission-and-ownership-test'>⇢# Test 7: Permission and Ownership Test</a></li> +<li>⇢ <a href='#-test-8-failover-test-optional-but-recommended'>⇢# Test 8: Failover Test (Optional but Recommended)</a></li> +<li>⇢ ⇢ <a href='#troubleshooting-common-issues'>Troubleshooting Common Issues</a></li> +<li>⇢ <a href='#-mount-hangs-or-times-out'>⇢# Mount Hangs or Times Out</a></li> +<li>⇢ <a href='#-permission-denied-errors'>⇢# Permission Denied Errors</a></li> +<li>⇢ <a href='#-files-not-visible-across-nodes'>⇢# Files Not Visible Across Nodes</a></li> +<li>⇢ <a href='#-io-errors-when-accessing-nfs-mount'>⇢# I/O Errors When Accessing NFS Mount</a></li> +<li>⇢ ⇢ <a href='#comprehensive-production-test-results'>Comprehensive Production Test Results</a></li> +<li>⇢ <a href='#-test-scenario-full-system-reboot-and-failover'>⇢# Test Scenario: Full System Reboot and Failover</a></li> +<li>⇢ <a href='#-key-findings'>⇢# Key Findings</a></li> +<li>⇢ <a href='#performance-considerations'>Performance Considerations</a></li> +<li>⇢ ⇢ <a href='#encryption-overhead'>Encryption Overhead</a></li> +<li>⇢ ⇢ <a href='#replication-bandwidth'>Replication Bandwidth</a></li> +<li>⇢ ⇢ <a href='#nfs-tuning'>NFS Tuning</a></li> +<li>⇢ ⇢ <a href='#zfs-tuning'>ZFS Tuning</a></li> +<li>⇢ ⇢ <a href='#monitoring'>Monitoring</a></li> +<li>⇢ ⇢ <a href='#cleanup-after-testing'>Cleanup After Testing</a></li> +<li>⇢ <a href='#conclusion'>Conclusion</a></li> +<li>⇢ ⇢ <a href='#what-we-achieved'>What We Achieved</a></li> +<li>⇢ ⇢ <a href='#architecture-benefits'>Architecture Benefits</a></li> +<li>⇢ ⇢ <a href='#lessons-learned'>Lessons Learned</a></li> +<li>⇢ ⇢ <a href='#next-steps'>Next Steps</a></li> +<li>⇢ ⇢ <a href='#references'>References</a></li> </ul><br /> <h2 style='display: inline' id='introduction'>Introduction</h2><br /> <br /> -<span>In this blog post, we are going to extend the Beelinks with some additional storage.</span><br /> +<span>In the previous posts, we set up a FreeBSD-based Kubernetes cluster using k3s. While the base system works well, Kubernetes workloads often require persistent storage for databases, configuration files, and application data. Local storage on each node has significant limitations:</span><br /> <br /> -<span>Some photos here, describe why there are 2 different models of SSD drives (replication etc)</span><br /> +<ul> +<li>No data sharing: Pods on different nodes can't access the same data</li> +<li>Pod mobility: If a pod moves to another node, it loses access to its data</li> +<li>No redundancy: Hardware failure means data loss</li> +</ul><br /> +<span>This post implements a robust storage solution using:</span><br /> +<br /> +<ul> +<li>CARP: For high availability with automatic IP failover</li> +<li>NFS over stunnel: For secure, encrypted network storage</li> +<li>ZFS: For data integrity, encryption, and efficient snapshots</li> +<li>zrepl: For continuous ZFS replication between nodes</li> +</ul><br /> +<span>The end result is a highly available, encrypted storage system that survives node failures while providing shared storage to all Kubernetes pods.</span><br /> +<br /> +<h2 style='display: inline' id='additional-storage-capacity'>Additional storage capacity</h2><br /> +<br /> +<span>We add to each of the nodes (<span class='inlinecode'>f0</span>, <span class='inlinecode'>f1</span>, <span class='inlinecode'>f2</span>) additional 1TB storage in form of an SSD drive. The Beelink mini PCs have enough space in the chassis for the additional space.</span><br /> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-6/drives.jpg'><img src='./f3s-kubernetes-with-freebsd-part-6/drives.jpg' /></a><br /> +<br /> +<span>Upgrading the storage was as easy as unscrewing, plugging the drive in, and then screwing it together again. So the procedure was pretty uneventful! We're using two different SSD models (Samsung 870 EVO and Crucial BX500) to avoid simultaneous failures from the same manufacturing batch.</span><br /> +<br /> +<span>We then create the <span class='inlinecode'>zdata</span> ZFS pool on all three nodes:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:~ % doas zpool create -m /data zdata /dev/ada<font color="#000000">1</font> +paul@f0:~ % zpool list +NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT +zdata 928G <font color="#000000">12</font>.1M 928G - - <font color="#000000">0</font>% <font color="#000000">0</font>% <font color="#000000">1</font>.00x ONLINE - +zroot 472G <font color="#000000">29</font>.0G 443G - - <font color="#000000">0</font>% <font color="#000000">6</font>% <font color="#000000">1</font>.00x ONLINE - + +paul@f0:/ % doas camcontrol devlist +<512GB SSD D910R170> at scbus0 target <font color="#000000">0</font> lun <font color="#000000">0</font> (pass0,ada0) +<Samsung SSD <font color="#000000">870</font> EVO 1TB SVT03B6Q> at scbus1 target <font color="#000000">0</font> lun <font color="#000000">0</font> (pass1,ada1) +paul@f0:/ % +</pre> +<br /> +<span>To verify that we have a different SSD on the second node (the third node has the same drive as the first):</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f1:/ % doas camcontrol devlist +<512GB SSD D910R170> at scbus0 target <font color="#000000">0</font> lun <font color="#000000">0</font> (pass0,ada0) +<CT1000BX500SSD1 M6CR072> at scbus1 target <font color="#000000">0</font> lun <font color="#000000">0</font> (pass1,ada1) +</pre> <br /> <h2 style='display: inline' id='zfs-encryption-keys'>ZFS encryption keys</h2><br /> <br /> +<span>ZFS native encryption requires encryption keys to unlock datasets. We need a secure method to store these keys that balances security with operational needs:</span><br /> +<br /> +<ul> +<li>Security: Keys must not be stored on the same disks they encrypt</li> +<li>Availability: Keys must be available at boot for automatic mounting</li> +<li>Portability: Keys should be easily moved between systems for recovery</li> +</ul><br /> +<span>Using USB flash drives as hardware key storage provides an elegant solution. The encrypted data is unreadable without physical access to the USB key, protecting against disk theft or improper disposal. In production environments, you might use enterprise key management systems, but for a home lab, USB keys offer good security with minimal complexity.</span><br /> +<br /> <h3 style='display: inline' id='ufs-on-usb-keys'>UFS on USB keys</h3><br /> <br /> +<span>We'll format the USB drives with UFS (Unix File System) rather than ZFS for simplicity. There is no need to use ZFS.</span><br /> +<br /> +<span>Let's see the USB keys:</span><br /> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-6/usbkeys1.jpg'><img alt='USB keys' title='USB keys' src='./f3s-kubernetes-with-freebsd-part-6/usbkeys1.jpg' /></a><br /> +<br /> +<span>To verify, that the USB key (flash disk) is there:</span><br /> +<br /> <pre> paul@f0:/ % doas camcontrol devlist <512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0) @@ -72,13 +220,7 @@ paul@f0:/ % doas camcontrol devlist paul@f0:/ % </pre> <br /> -<pre> -paul@f1:/ % doas camcontrol devlist -<512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0) -<CT1000BX500SSD1 M6CR072> at scbus1 target 0 lun 0 (pass1,ada1) -<Generic Flash Disk 8.07> at scbus2 target 0 lun 0 (da0,pass2) -paul@f1:/ % -</pre> +<span>Let's create the UFS file system and mount it (done on all 3 nodes <span class='inlinecode'>f0</span>, <span class='inlinecode'>f1</span> and <span class='inlinecode'>f2</span>):</span><br /> <br /> <!-- Generator: GNU source-highlight 3.1.9 by Lorenzo Bettini @@ -101,39 +243,47 @@ paul@f0:/ % df | grep keys /dev/da<font color="#000000">0</font> <font color="#000000">14877596</font> <font color="#000000">8</font> <font color="#000000">13687384</font> <font color="#000000">0</font>% /keys </pre> <br /> +<a href='./f3s-kubernetes-with-freebsd-part-6/usbkeys2.jpg'><img alt='USB keys sticked in' title='USB keys sticked in' src='./f3s-kubernetes-with-freebsd-part-6/usbkeys2.jpg' /></a><br /> +<br /> <h3 style='display: inline' id='generating-encryption-keys'>Generating encryption keys</h3><br /> <br /> -<span>paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:bhyve.key 32</span><br /> -<span>paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:bhyve.key 32</span><br /> -<span>paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:bhyve.key 32</span><br /> -<span>paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:zdata.key 32</span><br /> -<span>paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:zdata.key 32</span><br /> -<span>paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:zdata.key 32</span><br /> -<span>paul@f0:/keys % doas chown root *</span><br /> -<span>paul@f0:/keys % doas chmod 400 *</span><br /> +<span>The following keys will later be used to encrypt the ZFS file systems. They will be stored on all three nodes, serving as a backup in case one of the keys is lost. When we later replicate encrypted ZFS volumes from one node to another, the keys must also be available on the destination node.</span><br /> <br /> -<span>paul@f0:/keys % ls -l</span><br /> -<span>total 20</span><br /> -<span>-r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:bhyve.key</span><br /> -<span>-r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:bhyve.key</span><br /> -<span>-r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:bhyve.key</span><br /> -<span>-r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:zdata.key</span><br /> -<span>-r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:zdata.key</span><br /> -<span>-r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:zdata.key</span><br /> +<pre> +paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:bhyve.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:bhyve.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:bhyve.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:zdata.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:zdata.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:zdata.key 32 +paul@f0:/keys % doas chown root * +paul@f0:/keys % doas chmod 400 * + +paul@f0:/keys % ls -l +total 20 +*r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:bhyve.key +*r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:bhyve.key +*r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:bhyve.key +*r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:zdata.key +*r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:zdata.key +*r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:zdata.key +</pre> +<br /> +<span>After creation, these are copied to the other two nodes, <span class='inlinecode'>f1</span> and <span class='inlinecode'>f2</span>, into the <span class='inlinecode'>/keys</span> partition (I won't provide the commands here; just create a tarball, copy it over, and extract it on the destination nodes).</span><br /> <br /> -<span>Copy those to all 3 nodes to /keys</span><br /> +<h3 style='display: inline' id='configuring-zdata-zfs-pool-encryption'>Configuring <span class='inlinecode'>zdata</span> ZFS pool encryption</h3><br /> <br /> -<h3 style='display: inline' id='configuring-zdata-zfs-pool-and-encryption'>Configuring <span class='inlinecode'>zdata</span> ZFS pool and encryption</h3><br /> +<span>Let's encrypt our <span class='inlinecode'>zdata</span> ZFS pool. Actually, we are not encrypting the whole pool, but everythig within the <span class='inlinecode'>zdata/enc</span> data set:</span><br /> <br /> <!-- Generator: GNU source-highlight 3.1.9 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> -<pre>paul@f0:/keys % doas zpool create -m /data zdata /dev/ada<font color="#000000">1</font> -paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///keys/`hostname`:zdata.key zdata/enc +<pre>paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///keys/`hostname`:zdata.key zdata/enc paul@f0:/ % zfs list | grep zdata zdata 836K 899G 96K /data zdata/enc 200K 899G 200K /data/enc + paul@f0:/keys % zfs get all zdata/enc | grep -E -i <font color="#808080">'(encryption|key)'</font> zdata/enc encryption aes-<font color="#000000">256</font>-gcm - zdata/enc keylocation file:///keys/f<font color="#000000">0</font>.lan.buetow.org:zdata.key <b><u><font color="#000000">local</font></u></b> @@ -142,9 +292,11 @@ zdata/enc encryptionroot zdata/enc - zdata/enc keystatus available - </pre> <br /> +<span>All future data sets within <span class='inlinecode'>zdata/enc</span> will inherit the same encription key.</span><br /> +<br /> <h3 style='display: inline' id='migrating-bhyve-vms-to-encrypted-bhyve-zfs-volume'>Migrating Bhyve VMs to encrypted <span class='inlinecode'>bhyve</span> ZFS volume</h3><br /> <br /> -<span>Run on all 3 nodes</span><br /> +<span>We set up Bhyve VMs in one of the previous blog posts. Their ZFS data sets rely on <span class='inlinecode'>zroot</span>, which is the default ZFS pool on the internal 512GB NVME drive. They aren't encrypted yet, so we encrypt the VM data sets as well now. To do so, we first shut down the VMs on all 3 nodes:</span><br /> <br /> <!-- Generator: GNU source-highlight 3.1.9 by Lorenzo Bettini @@ -156,24 +308,42 @@ Sending ACPI shutdown to rocky paul@f0:/keys % doas vm list NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE rocky default uefi <font color="#000000">4</font> 14G - Yes [<font color="#000000">1</font>] Stopped - - -paul@f0:/keys % doas zfs rename zroot/bhyve zroot/bhyve_old +</pre> +<br /> +<span>After this, we rename the unencrypted data set to <span class='inlinecode'>_old</span> and create a new encrypted data set and we also snapshot it as <span class='inlinecode'>@hamburger</span>!</span><br /> +<span> </span><br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:/keys % doas zfs rename zroot/bhyve zroot/bhyve_old paul@f0:/keys % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/mnt zroot/bhyve_old paul@f0:/keys % doas zfs snapshot zroot/bhyve_old/rocky@hamburger - -paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///keys/`hostname`:bhyve.key zroot/bhyve +paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o \ + keylocation=file:///keys/`hostname`:bhyve.key zroot/bhyve paul@f0:/keys % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/zroot/bhyve zroot/bhyve paul@f0:/keys % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/zroot/bhyve/rocky zroot/bhyve/rocky - -paul@f0:/keys % doas zfs send zroot/bhyve_old/rocky@hamburger | doas zfs recv zroot/bhyve/rocky +</pre> +<br /> +<span>Once done, we import the snapshot into the encrypted dataset and also copy some other metadata files from <span class='inlinecode'>vm-bhyve</span> back over.</span><br /> +<br /> +<pre> +paul@f0:/keys % doas zfs send zroot/bhyve_old/rocky@hamburger | \ + doas zfs recv zroot/bhyve/rocky paul@f0:/keys % doas cp -Rp /mnt/.config /zroot/bhyve/ paul@f0:/keys % doas cp -Rp /mnt/.img /zroot/bhyve/ paul@f0:/keys % doas cp -Rp /mnt/.templates /zroot/bhyve/ paul@f0:/keys % doas cp -Rp /mnt/.iso /zroot/bhyve/ - -paul@f0:/keys % doas sysrc zfskeys_enable=YES +</pre> +<br /> +<span>We also have to make encrypted ZFS data sets mount automatically on boot:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:/keys % doas sysrc zfskeys_enable=YES zfskeys_enable: -> YES paul@f0:/keys % doas vm init paul@f0:/keys % doas reboot @@ -186,18 +356,28 @@ NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE rocky default uefi <font color="#000000">4</font> 14G <font color="#000000">0.0</font>.<font color="#000000">0.0</font>:<font color="#000000">5900</font> Yes [<font color="#000000">1</font>] Running (<font color="#000000">2265</font>) </pre> <br /> +<span>As you can see, the VM is running. This means the encrypted <span class='inlinecode'>zroot/bhyve</span> was mounted successfully after the reboot! Now we can destroy the old, unencrypted, and now unused bhyve dataset:</span><br /> +<br /> <!-- Generator: GNU source-highlight 3.1.9 by Lorenzo Bettini http://www.lorenzobettini.it http://www.gnu.org/software/src-highlite --> <pre>paul@f0:~ % doas zfs destroy -R zroot/bhyve_old - -paul@f0:~ % zfs get all zroot/bhyve | grep -E <font color="#808080">'(encryption|key)'</font> +</pre> +<br /> +<span>To verify once again that <span class='inlinecode'>zroot/bhyve</span> and <span class='inlinecode'>zroot/bhyve/rocky</span> are now both encrypted, we run:</span><br /> +<span> </span><br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:~ % zfs get all zroot/bhyve | grep -E <font color="#808080">'(encryption|key)'</font> zroot/bhyve encryption aes-<font color="#000000">256</font>-gcm - zroot/bhyve keylocation file:///keys/f<font color="#000000">0</font>.lan.buetow.org:bhyve.key <b><u><font color="#000000">local</font></u></b> zroot/bhyve keyformat raw - zroot/bhyve encryptionroot zroot/bhyve - zroot/bhyve keystatus available - + paul@f0:~ % zfs get all zroot/bhyve/rocky | grep -E <font color="#808080">'(encryption|key)'</font> zroot/bhyve/rocky encryption aes-<font color="#000000">256</font>-gcm - zroot/bhyve/rocky keylocation none default @@ -206,62 +386,41 @@ zroot/bhyve/rocky encry |