more about this

3 files changed, 350 insertions, 2265 deletions

diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.gmi b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.gmi
index df1d7b40..74c102e4 100644
--- a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.gmi
+++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.gmi
@@ -22,110 +22,46 @@ This is the sixth blog post about the f3s series for self-hosting demands in a h
 * ⇢ ⇢ ⇢ Generating encryption keys
 * ⇢ ⇢ ⇢ Configuring `zdata` ZFS pool encryption
 * ⇢ ⇢ ⇢ Migrating Bhyve VMs to encrypted `bhyve` ZFS volume
-* ⇢ ⇢ ZFS Replication with zrepl
+* ⇢ ⇢ ZFS Replication with `zrepl`
 * ⇢ ⇢ ⇢ Understanding Replication Requirements
-* ⇢ ⇢ ⇢ Why `zrepl` instead of HAST?
-* ⇢ ⇢ ⇢ Installing zrepl
-* ⇢ ⇢ ⇢ Checking ZFS pools
-* ⇢ ⇢ ⇢ Configuring `zrepl` with WireGuard tunnel
-* ⇢ ⇢ ⇢ Configuring `zrepl` on f0 (source)
+* ⇢ ⇢ ⇢ Installing `zrepl`
 * ⇢ ⇢ ⇢ Configuring `zrepl` on `f1` (sink)
 * ⇢ ⇢ ⇢ Enabling and starting `zrepl` services
-* ⇢ ⇢ ⇢ Verifying replication
 * ⇢ ⇢ ⇢ Monitoring replication
 * ⇢ ⇢ ⇢ Verifying replication after reboot
 * ⇢ ⇢ ⇢ Understanding Failover Limitations and Design Decisions
-* ⇢ ⇢ ⇢# Why Manual Failover?
-* ⇢ ⇢ ⇢# Current Failover Process
 * ⇢ ⇢ ⇢ Mounting the NFS datasets
-* ⇢ ⇢ ⇢ Failback scenario: Syncing changes from f1 back to f0
-* ⇢ ⇢ ⇢ Testing the failback scenario
 * ⇢ ⇢ ⇢ Troubleshooting: Files not appearing in replication
 * ⇢ ⇢ ⇢ Configuring automatic key loading on boot
-* ⇢ ⇢ ⇢ Troubleshooting: Replication broken due to modified destination
-* ⇢ ⇢ ⇢ Forcing a full resync
 * ⇢ ⇢ CARP (Common Address Redundancy Protocol)
 * ⇢ ⇢ ⇢ How CARP Works
 * ⇢ ⇢ ⇢ Configuring CARP
 * ⇢ ⇢ ⇢ CARP State Change Notifications
-* ⇢ ⇢ Future Storage Explorations
-* ⇢ ⇢ ⇢ MinIO for S3-Compatible Object Storage
-* ⇢ ⇢ ⇢ MooseFS for Distributed High Availability
 * ⇢ ⇢ NFS Server Configuration
-* ⇢ ⇢ ⇢ Setting up NFS on f0 (Primary)
+* ⇢ ⇢ ⇢ Setting up NFS on `f0` (Primary)
 * ⇢ ⇢ ⇢ Configuring Stunnel for NFS Encryption with CARP Failover
-* ⇢ ⇢ ⇢# Why Not Native NFS over TLS?
-* ⇢ ⇢ ⇢# Stunnel Architecture with CARP
-* ⇢ ⇢ ⇢# Creating a Certificate Authority for Client Authentication
-* ⇢ ⇢ ⇢# Install and Configure Stunnel on f0
-* ⇢ ⇢ ⇢ Setting up NFS on f1 (Standby)
-* ⇢ ⇢ ⇢ How Stunnel Works with CARP
+* ⇢ ⇢ ⇢ Creating a Certificate Authority for Client Authentication
+* ⇢ ⇢ ⇢ Install and Configure Stunnel on `f0`
+* ⇢ ⇢ ⇢ Setting up NFS on `f1` (Standby)
 * ⇢ ⇢ ⇢ CARP Control Script for Clean Failover
 * ⇢ ⇢ ⇢ CARP Management Script
 * ⇢ ⇢ ⇢ Automatic Failback After Reboot
-* ⇢ ⇢ ⇢# Why Automatic Failback?
-* ⇢ ⇢ ⇢# The Auto-Failback Script
-* ⇢ ⇢ ⇢# Setting Up the Marker File
-* ⇢ ⇢ ⇢# Configuring Cron
-* ⇢ ⇢ ⇢# Managing Automatic Failback
-* ⇢ ⇢ ⇢# How It Works
-* ⇢ ⇢ ⇢ Verifying Stunnel and CARP Status
-* ⇢ ⇢ ⇢ Verifying NFS Exports
-* ⇢ ⇢ ⇢ Client Configuration for Stunnel
-* ⇢ ⇢ ⇢# Preparing Client Certificates
-* ⇢ ⇢ ⇢# Configuring Rocky Linux Clients (r0, r1, r2)
+* ⇢ ⇢ Client Configuration for Stunnel
+* ⇢ ⇢ ⇢ Preparing Client Certificates
+* ⇢ ⇢ ⇢ Configuring Rocky Linux Clients (r0, r1, r2)
 * ⇢ ⇢ ⇢ Testing NFS Mount with Stunnel
-* ⇢ ⇢ ⇢ Important: Encryption Keys for Replicated Datasets
-* ⇢ ⇢ ⇢ NFS Failover with CARP and Stunnel
-* ⇢ ⇢ ⇢ Testing CARP Failover
-* ⇢ ⇢ ⇢ Handling Stale File Handles After Failover
+* ⇢ ⇢ ⇢ Testing CARP Failover with mounted clients and stale file handles:
 * ⇢ ⇢ ⇢ Complete Failover Test
-* ⇢ ⇢ ⇢ Verifying Replication Status
-* ⇢ ⇢ ⇢ Post-Reboot Verification
-* ⇢ ⇢ ⇢ Integration with Kubernetes
-* ⇢ ⇢ ⇢ Security Benefits of Stunnel with Client Certificates
-* ⇢ ⇢ ⇢ Laptop/Workstation Access
-* ⇢ ⇢ ⇢# Important: NFSv4 and Stunnel on Newer Linux Clients
-* ⇢ ⇢ Mounting NFS on Rocky Linux 9
-* ⇢ ⇢ ⇢ Installing and Configuring NFS Clients on r0, r1, and r2
-* ⇢ ⇢ ⇢ Configuring Stunnel Client on All Nodes
-* ⇢ ⇢ ⇢ Setting Up NFS Mounts
-* ⇢ ⇢ ⇢ Comprehensive NFS Mount Testing
-* ⇢ ⇢ ⇢# Test 1: Verify Mount Status on All Nodes
-* ⇢ ⇢ ⇢# Test 2: Verify Stunnel Connectivity
-* ⇢ ⇢ ⇢# Test 3: File Creation and Visibility Test
-* ⇢ ⇢ ⇢# Test 4: Verify Files on Storage Servers
-* ⇢ ⇢ ⇢# Test 5: Performance and Concurrent Access Test
-* ⇢ ⇢ ⇢# Test 6: Directory Operations Test
-* ⇢ ⇢ ⇢# Test 7: Permission and Ownership Test
-* ⇢ ⇢ ⇢# Test 8: Failover Test (Optional but Recommended)
-* ⇢ ⇢ ⇢ Troubleshooting Common Issues
-* ⇢ ⇢ ⇢# Mount Hangs or Times Out
-* ⇢ ⇢ ⇢# Permission Denied Errors
-* ⇢ ⇢ ⇢# Files Not Visible Across Nodes
-* ⇢ ⇢ ⇢# I/O Errors When Accessing NFS Mount
-* ⇢ ⇢ ⇢ Comprehensive Production Test Results
-* ⇢ ⇢ ⇢# Test Scenario: Full System Reboot and Failover
-* ⇢ ⇢ ⇢# Key Findings
-* ⇢ ⇢ Performance Considerations
-* ⇢ ⇢ ⇢ Encryption Overhead
-* ⇢ ⇢ ⇢ Replication Bandwidth
-* ⇢ ⇢ ⇢ NFS Tuning
-* ⇢ ⇢ ⇢ ZFS Tuning
-* ⇢ ⇢ ⇢ Monitoring
-* ⇢ ⇢ ⇢ Cleanup After Testing
 * ⇢ ⇢ Conclusion
-* ⇢ ⇢ ⇢ What We Achieved
-* ⇢ ⇢ ⇢ Architecture Benefits
-* ⇢ ⇢ ⇢ Lessons Learned
-* ⇢ ⇢ ⇢ Next Steps
-* ⇢ ⇢ ⇢ References
+* ⇢ ⇢ Future Storage Explorations
+* ⇢ ⇢ ⇢ MinIO for S3-Compatible Object Storage
+* ⇢ ⇢ ⇢ MooseFS for Distributed High Availability
 
 ## Introduction
 
 In the previous posts, we set up a FreeBSD-based Kubernetes cluster using k3s. While the base system works well, Kubernetes workloads often require persistent storage for databases, configuration files, and application data. Local storage on each node has significant limitations:
 
-* No data sharing: Pods on different nodes can't access the same data
+* No data sharing: Pods (once we run Kubernetes) on different nodes can't access the same data
 * Pod mobility: If a pod moves to another node, it loses access to its data
 * No redundancy: Hardware failure means data loss
 
@@ -134,10 +70,12 @@ This post implements a robust storage solution using:
 * CARP: For high availability with automatic IP failover
 * NFS over stunnel: For secure, encrypted network storage
 * ZFS: For data integrity, encryption, and efficient snapshots
-* zrepl: For continuous ZFS replication between nodes
+* `zrepl`: For continuous ZFS replication between nodes
 
 The end result is a highly available, encrypted storage system that survives node failures while providing shared storage to all Kubernetes pods.
 
+Other than what was mentioned in the first post of this blog series, we aren't using HAST, but `zrepl` for data replication. Read more about it later in this blog post.
+
 ## Additional storage capacity
 
 We add to each of the nodes (`f0`, `f1`, `f2`) additional 1TB storage in form of an SSD drive. The Beelink mini PCs have enough space in the chassis for the additional space.
@@ -250,7 +188,8 @@ After creation, these are copied to the other two nodes, `f1` and `f2`, into the
 Let's encrypt our `zdata` ZFS pool. Actually, we are not encrypting the whole pool, but everythig within the `zdata/enc` data set:
 
 ```sh
-paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o keylocation=file:///keys/`hostname`:zdata.key zdata/enc
+paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o \
+  keylocation=file:///keys/`hostname`:zdata.key zdata/enc
 paul@f0:/ % zfs list | grep zdata
 zdata                                          836K   899G    96K  /data
 zdata/enc                                      200K   899G   200K  /data/enc
@@ -342,7 +281,7 @@ zroot/bhyve/rocky  encryptionroot        zroot/bhyve            -
 zroot/bhyve/rocky  keystatus             available              -
 ```
 
-## ZFS Replication with zrepl
+## ZFS Replication with `zrepl`
 
 Data replication is the cornerstone of high availability. While CARP handles IP failover (see later in this post), we need continuous data replication to ensure the backup server has current data when it becomes active. Without replication, failover would result in data loss or require shared storage (like iSCSI), which introduces a single point of failure.
 
@@ -350,23 +289,21 @@ Data replication is the cornerstone of high availability. While CARP handles IP
 
 Our storage system has different replication needs:
 
-* NFS data (`/data/nfs/k3svolumes`): Contains active Kubernetes persistent volumes. Needs frequent replication (every minute) to minimize data loss during failover.
+* NFS data (`/data/nfs/k3svolumes`): Soon, contains active Kubernetes persistent volumes. Needs frequent replication (every minute) to minimize data loss during failover.
 * VM data (`/zroot/bhyve/fedora`): Contains VM images that change less frequently. Can tolerate longer replication intervals (every 10 minutes).
 
-The replication frequency determines your Recovery Point Objective (RPO) - the maximum acceptable data loss. With 1-minute replication, you lose at most 1 minute of changes during an unplanned failover.
-
-### Why `zrepl` instead of HAST?
+The 1-minute replication window is perfectly acceptable for my personal use cases. This isn't a high-frequency trading system or a real-time database—it's storage for personal projects, development work, and home lab experiments. Losing at most 1 minute of work in a disaster scenario is a reasonable trade-off for the reliability and simplicity of snapshot-based replication. Also, in the case of "1 minute of data loss," I would very likely still have the data available on the client side.
 
-While HAST (Highly Available Storage) is FreeBSD's native solution for high-availability storage, I've chosen `zrepl` for several important reasons:
+Why use `zrepl` instead of HAST? While HAST (Highly Available Storage) is FreeBSD's native solution for high-availability storage and supports synchronous replication—thus eliminating the mentioned 1-minute window—I've chosen `zrepl` for several important reasons:
 
-* HAST can cause ZFS corruption: HAST operates at the block level and doesn't understand ZFS's transactional semantics. During failover, in-flight transactions can lead to corrupted zpools. I've experienced this firsthand - the automatic failover would trigger while ZFS was still writing, resulting in an unmountable pool.
+* HAST can cause ZFS corruption: HAST operates at the block level and doesn't understand ZFS's transactional semantics. During failover, in-flight transactions can lead to corrupted zpools. I've experienced this firsthand (I am sure I might have configured something wrong) - the automatic failover would trigger while ZFS was still writing, resulting in an unmountable pool.
 * ZFS-aware replication: `zrepl` understands ZFS datasets and snapshots. It replicates at the dataset level, ensuring each snapshot is a consistent point-in-time copy. This is fundamentally safer than block-level replication.
-* Snapshot history: With zrepl, you get multiple recovery points (every minute for NFS data in our setup). If corruption occurs, you can roll back to any previous snapshot. HAST only gives you the current state.
-* Easier recovery: When something goes wrong with zrepl, you still have intact snapshots on both sides. With HAST, a corrupted primary often means a corrupted secondary too.
+* Snapshot history: With `zrepl`, you get multiple recovery points (every minute for NFS data in our setup). If corruption occurs, you can roll back to any previous snapshot. HAST only gives you the current state.
+* Easier recovery: When something goes wrong with `zrepl`, you still have intact snapshots on both sides. With HAST, a corrupted primary often means a corrupted secondary too.
 
-The 1-minute replication window is perfectly acceptable for my personal use cases. This isn't a high-frequency trading system or a real-time database—it's storage for personal projects, development work, and home lab experiments. Losing at most 1 minute of work in a disaster scenario is a reasonable trade-off for the reliability and simplicity of snapshot-based replication. Also, in the case of "1 minute of data loss," I would very likely still have the data available on the client side.
+=> https://wiki.freebsd.org/HighlyAvailableStorage FreeBSD HAST
 
-### Installing zrepl
+### Installing `zrepl`
 
 First, install `zrepl` on both hosts involved (we will replicate data from `f0` to `f1`):
 
@@ -374,9 +311,7 @@ First, install `zrepl` on both hosts involved (we will replicate data from `f0`
 paul@f0:~ % doas pkg install -y zrepl
 ```
 
-### Checking ZFS pools
-
-Verify the pools and datasets on both hosts:
+Then, we verify the pools and datasets on both hosts:
 
 ```sh
 # On f0
@@ -400,9 +335,7 @@ NAME        USED  AVAIL  REFER  MOUNTPOINT
 zdata/enc   200K   899G   200K  /data/enc
 ```
 
-### Configuring `zrepl` with WireGuard tunnel
-
-Since we have a WireGuard tunnel between f0 and f1, we'll use TCP transport over the secure tunnel instead of SSH. First, check the WireGuard IP addresses:
+Since we have a WireGuard tunnel between `f0` and f1, we'll use TCP transport over the secure tunnel instead of SSH. First, check the WireGuard IP addresses:
 
 ```sh
 # Check WireGuard interface IPs
@@ -413,16 +346,14 @@ paul@f1:~ % ifconfig wg0 | grep inet
 	inet 192.168.2.131 netmask 0xffffff00
 ```
 
-### Configuring `zrepl` on f0 (source)
-
-First, create a dedicated dataset for NFS data that will be replicated:
+Let's create a dedicated dataset for NFS data that will be replicated:
 
 ```sh
 # Create the nfsdata dataset that will hold all data exposed via NFS
 paul@f0:~ % doas zfs create zdata/enc/nfsdata
 ```
 
-Create the `zrepl` configuration on f0:
+Afterwards, we create the `zrepl` configuration on `f0`:
 
 ```sh
 paul@f0:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<'EOF'
@@ -480,18 +411,18 @@ EOF
  We're using two separate replication jobs with different intervals:
 
 * `f0_to_f1_nfsdata`: Replicates NFS data every minute for faster failover recovery
-* `f0_to_f1_fedora`: Replicates Fedora VM every 10 minutes (less critical for NFS operations)
+* `f0_to_f1_fedora`: Replicates Fedora VM every ten minutes (less critical)
 
-The Fedora is only used for development purposes, so it doesn't require as frequent replication as the NFS data. It's off-topic to this blog series, but it showcases, hows zrepl's flexibility in handling different datasets with varying replication needs.
+The Fedora VM is only used for development purposes, so it doesn't require as frequent replication as the NFS data. It's off-topic to this blog series, but it showcases, hows `zrepl`'s flexibility in handling different datasets with varying replication needs.
 
 Furthermore:
 
 * We're specifically replicating `zdata/enc/nfsdata` instead of the entire `zdata/enc` dataset. This dedicated dataset will contain all the data we later want to expose via NFS, keeping a clear separation between replicated NFS data and other local encrypted data.
-* The `send: encrypted: false` option disables ZFS native encryption for the replication stream. Since we're using a WireGuard tunnel between f0 and f1, the data is already encrypted in transit. Disabling ZFS stream encryption reduces CPU overhead and improves replication performance.
+* The `send: encrypted: false` option disables ZFS native encryption for the replication stream. Since we're using a WireGuard tunnel between `f0` and `f1`, the data is already encrypted in transit. Disabling ZFS stream encryption reduces CPU overhead and improves replication performance.
 
 ### Configuring `zrepl` on `f1` (sink)
 
-On `f1` we configure `zrepl` to receive the data as follows:
+On `f1` (the sink, meaning it's the node receiving the replication data), we configure `zrepl` to receive the data as follows:
 
 ```sh
 # First create a dedicated sink dataset
@@ -521,7 +452,7 @@ EOF
 
 ### Enabling and starting `zrepl` services
 
-Enable and start `zrepl` on both hosts:
+We then enable and start `zrepl` on both hosts via:
 
 ```sh
 # On f0
@@ -537,20 +468,19 @@ paul@f1:~ % doas service `zrepl` start
 Starting zrepl.
 ```
 
-### Verifying replication
-
 To check the replication status, we run:
 
 ```sh
 # On f0, check `zrepl` status (use raw mode for non-tty)
-paul@f0:~ % doas `zrepl` status --mode raw | grep -A2 "Replication"
+paul@f0:~ % doas pkg install jq
+paul@f0:~ % doas zrepl status --mode raw | grep -A2 "Replication" | jq .
 "Replication":{"StartAt":"2025-07-01T22:31:48.712143123+03:00"...
 
 # Check if services are running
-paul@f0:~ % doas service `zrepl` status
+paul@f0:~ % doas service zrepl status
 zrepl is running as pid 2649.
 
-paul@f1:~ % doas service `zrepl` status
+paul@f1:~ % doas service zrepl status
 zrepl is running as pid 2574.
 
 # Check for `zrepl` snapshots on source
@@ -567,6 +497,9 @@ zdata/f0/zdata/enc   176K   899G   176K  none
 paul@f1:~ % doas zfs list -t snapshot -r zdata | grep zrepl
 zdata/f0/zdata/enc@zrepl_20250701_193148_000     0B      -   176K  -
 zdata/f0/zdata/enc@zrepl_20250701_194148_000     0B      -   176K  -
+.
+.
+.
 ```
 
 ### Monitoring replication
@@ -574,21 +507,19 @@ zdata/f0/zdata/enc@zrepl_20250701_194148_000     0B      -   176K  -
 You can monitor the replication progress with:
 
 ```sh
-# Real-time status
-paul@f0:~ % doas `zrepl` status --mode interactive
-
-# Check specific job details
-paul@f0:~ % doas `zrepl` status --job f0_to_f1
+paul@f0:~ % doas zrepl status
 ```
 
-With this setup, both `zdata/enc/nfsdata` and `zroot/bhyve/fedora` on f0 will be automatically replicated to f1 every 1 (or 10 in case of the Fedora VM) minutes, with encrypted snapshots preserved on both sides. The pruning policy ensures that we keep the last 10 snapshots while managing disk space efficiently.
+=> ./f3s-kubernetes-with-freebsd-part-6/zrepl.png zrepl status
 
-The replicated data appears on f1 under `zdata/sink/` with the source host and dataset hierarchy preserved:
+With this setup, both `zdata/enc/nfsdata` and `zroot/bhyve/fedora` on `f0` will be automatically replicated to `f1` every 1 (or 10 in case of the Fedora VM) minutes, with encrypted snapshots preserved on both sides. The pruning policy ensures that we keep the last 10 snapshots while managing disk space efficiently.
+
+The replicated data appears on `f1` under `zdata/sink/` with the source host and dataset hierarchy preserved:
 
 * `zdata/enc/nfsdata` → `zdata/sink/f0/zdata/enc/nfsdata`
 * `zroot/bhyve/fedora` → `zdata/sink/f0/zroot/bhyve/fedora`
 
-This is by design - `zrepl` preserves the complete path from the source to ensure there are no conflicts when replicating from multiple sources. The replication uses the WireGuard tunnel for secure, encrypted transport between nodes.
+This is by design - `zrepl` preserves the complete path from the source to ensure there are no conflicts when replicating from multiple sources.
 
 ### Verifying replication after reboot
 
@@ -608,64 +539,34 @@ zrepl is running as pid 2309.
 paul@f0:~ % doas zfs list -t snapshot | grep `zrepl` | tail -2
 zdata/enc/nfsdata@zrepl_20250701_202530_000                0B      -   200K  -
 zroot/bhyve/fedora@zrepl_20250701_202530_000               0B      -  2.97G  -
+.
+.
+.
 
 paul@f1:~ % doas zfs list -t snapshot -r zdata/sink | grep 202530
 zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250701_202530_000      0B      -   176K  -
 zdata/sink/f0/zroot/bhyve/fedora@zrepl_20250701_202530_000     0B      -  2.97G  -
+.
+.
+.
 ```
 
-The timestamps confirm that replication resumed automatically after the reboot, ensuring continuous data protection.
+The timestamps confirm that replication resumed automatically after the reboot, ensuring continuous data protection. We can also write a test file to the NFS data directory on `f0` and see whether it shows up on `f1` after a minute.
 
 ### Understanding Failover Limitations and Design Decisions
 
+Our system intentionally failovers to a read-only copy of the replica in case the primary goes down. This is due to the nature that `zrepl` only replicates the data in one direction and if we mounted the data set on the sink node read-write, it would make the ZFS data-set diverge from the original and the replication would break. It can still be mounted read-write on the sink node in case of a real issue on the primary node, but that step is left intentional manualy. So we don't need to manually fix the replication later on.
 
+So in summary:
 
-#### Why Manual Failover?
-
-This storage system intentionally uses manual failover rather than automatic failover. This might seem counterintuitive for a "high availability" system, but it's a deliberate design choice based on real-world experience:
-
-1. Split-brain prevention: Automatic failover can cause both nodes to become active simultaneously if network communication fails. This leads to data divergence that's extremely difficult to resolve.
-
-2. False positive protection: Temporary network issues or high load can trigger unwanted failovers. Manual intervention ensures failovers only occur when truly necessary.
-
-3. Data integrity over availability: For storage systems, data consistency is paramount. A few minutes of downtime is preferable to data corruption or loss.
-
-4. Simplified recovery: With manual failover, you always know which dataset is authoritative, making recovery straightforward.
-
-#### Current Failover Process
-
-The replicated datasets on f1 are intentionally not mounted (`mountpoint=none`). In case f0 fails:
-
-```sh
-# Manual steps needed on f1 to activate the replicated data:
-paul@f1:~ % doas zfs set mountpoint=/data/nfsdata zdata/sink/f0/zdata/enc/nfsdata
-paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
-```
-
-However, this creates a split-brain problem: when f0 comes back online, both systems would have diverged data. Resolving this requires careful manual intervention to:
-
-1. Stop the original replication
-2. Sync changes from f1 back to f0
-3. Re-establish normal replication
-
-For true high-availability NFS, you might consider:
-
-* Shared storage (like iSCSI) with proper clustering
-* GlusterFS or similar distributed filesystems
-* Manual failover with ZFS replication (as we have here)
-
-Note: While HAST+CARP is often suggested for HA storage, it can cause filesystem corruption in practice, especially with ZFS. The block-level replication of HAST doesn't understand ZFS's transactional model, leading to inconsistent states during failover. 
-
-The current `zrepl` setup, despite requiring manual intervention, is actually safer because:
-
-* ZFS snapshots are always consistent
-* Replication is ZFS-aware (not just block-level)
-* You have full control over the failover process
-* No risk of split-brain corruption
+* Split-brain prevention: Automatic failover to a read-write copy can cause both nodes to become active simultaneously if network communication fails. This leads to data divergence that's extremely difficult to resolve.
+* False positive protection: Temporary network issues or high load can trigger unwanted failovers. Manual intervention ensures failovers only occur when truly necessary.
+* Data integrity over availability: For storage systems, data consistency is paramount. A few minutes of downtime is preferable to data corruption in this specific use case.
+* Simplified recovery: With manual failover, you always know which dataset is authoritative, making recovery more straightforward.
 
 ### Mounting the NFS datasets
 
-To make the nfsdata accessible on both nodes, we need to mount them. On f0, this is straightforward:
+To make the NFS data accessible on both nodes, we need to mount them. On `f0`, this is straightforward:
 
 ```sh
 # On f0 - set mountpoint for the primary nfsdata
@@ -678,7 +579,7 @@ Filesystem           Size    Used   Avail Capacity  Mounted on
 zdata/enc/nfsdata    899G    204K    899G     0%    /data/nfs
 ```
 
-On f1, we need to handle the encryption key and mount the standby copy:
+On `f1`, we need to handle the encryption key and mount the standby copy:
 
 ```sh
 # On f1 - first check encryption status
@@ -704,188 +605,45 @@ Filesystem                         Size    Used   Avail Capacity  Mounted on
 zdata/sink/f0/zdata/enc/nfsdata    896G    204K    896G     0%    /data/nfs
 ```
 
-Note: The dataset is mounted at the same path (`/data/nfs`) on both hosts to simplify failover procedures. The dataset on f1 is set to `readonly=on` to prevent accidental modifications that would break replication.
+Note: The dataset is mounted at the same path (`/data/nfs`) on both hosts to simplify failover procedures. The dataset on `f1` is set to `readonly=on` to prevent accidental modifications that, as mentioned already, would break replication. If we did, replication from `f0` to `f1` would fail like this:
 
-CRITICAL WARNING: Do NOT write to `/data/nfs/` on f1! Any modifications will break the replication. If you accidentally write to it, you'll see this error:
+> cannot receive incremental stream: destination zdata/sink/f0/zdata/enc/nfsdata has been modified since most recent snapshot 
 
-```
-cannot receive incremental stream: destination zdata/sink/f0/zdata/enc/nfsdata has been modified
-since most recent snapshot
-```
+To fix a broken replication after accidental writes, we can do:
 
-To fix a broken replication after accidental writes:
 ```sh
 # Option 1: Rollback to the last common snapshot (loses local changes)
 paul@f1:~ % doas zfs rollback zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250701_204054_000
 
-# Option 2: Make it read-only to prevent accidents
+# Option 2: Make it read-only to prevent accidents again
 paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
 ```
 
-### Failback scenario: Syncing changes from f1 back to f0
-
-In a disaster recovery scenario where f0 has failed and f1 has taken over, you'll need to sync changes back when f0 returns. Here's how to failback:
-
-```sh
-# On f1: First, make the dataset writable (if it was readonly)
-paul@f1:~ % doas zfs set readonly=off zdata/sink/f0/zdata/enc/nfsdata
-
-# Create a snapshot of the current state
-paul@f1:~ % doas zfs snapshot zdata/sink/f0/zdata/enc/nfsdata@failback
-
-# On f0: Stop any services using the dataset
-paul@f0:~ % doas service nfsd stop  # If NFS is running
-
-# Send the snapshot from f1 to f0, forcing a rollback
-# This WILL DESTROY any data on f0 that's not on f1!
-paul@f1:~ % doas zfs send -R zdata/sink/f0/zdata/enc/nfsdata@failback | \
-    ssh f0 "doas zfs recv -F zdata/enc/nfsdata"
-
-# Alternative: If you want to see what would be received first
-paul@f1:~ % doas zfs send -R zdata/sink/f0/zdata/enc/nfsdata@failback | \
-    ssh f0 "doas zfs recv -nv -F zdata/enc/nfsdata"
-
-# After successful sync, on f0:
-paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@failback
-
-# On f1: Make it readonly again and destroy the failback snapshot
-paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
-paul@f1:~ % doas zfs destroy zdata/sink/f0/zdata/enc/nfsdata@failback
-
-# Stop `zrepl` services first - CRITICAL!
-paul@f0:~ % doas service `zrepl` stop
-paul@f1:~ % doas service `zrepl` stop
-
-# Clean up any `zrepl` snapshots on f0
-paul@f0:~ % doas zfs list -t snapshot -r zdata/enc/nfsdata | grep `zrepl` | \
-    awk '{print $1}' | xargs -I {} doas zfs destroy {}
-
-# Clean up and destroy the entire replicated structure on f1
-# First release any holds
-paul@f1:~ % doas zfs holds -r zdata/sink/f0 | grep -v NAME | \
-    awk '{print $2, $1}' | while read tag snap; do 
-        doas zfs release "$tag" "$snap"
-    done
-
-# Then destroy the entire f0 tree
-paul@f1:~ % doas zfs destroy -rf zdata/sink/f0
-
-# Create parent dataset structure on f1
-paul@f1:~ % doas zfs create -p zdata/sink/f0/zdata/enc
-
-# Create a fresh manual snapshot to establish baseline
-paul@f0:~ % doas zfs snapshot zdata/enc/nfsdata@manual_baseline
-
-# Send this snapshot to f1
-paul@f0:~ % doas zfs send -w zdata/enc/nfsdata@manual_baseline | \
-    ssh f1 "doas zfs recv zdata/sink/f0/zdata/enc/nfsdata"
-
-# Clean up the manual snapshot
-paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@manual_baseline
-paul@f1:~ % doas zfs destroy zdata/sink/f0/zdata/enc/nfsdata@manual_baseline
-
-# Set mountpoint and make readonly on f1
-paul@f1:~ % doas zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata
-paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
-
-# Load encryption key and mount on f1
-paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \
-    zdata/sink/f0/zdata/enc/nfsdata
-paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
-
-# Now restart `zrepl` services
-paul@f0:~ % doas service `zrepl` start
-paul@f1:~ % doas service `zrepl` start
-
-# Verify replication is working
-paul@f0:~ % doas `zrepl` status --job f0_to_f1
-```
-
-Important notes about failback:
-
-* The `-F` flag forces a rollback on f0, destroying any local changes
-* Replication often won't resume automatically after a forced receive
-* You must clean up old `zrepl` snapshots on both sides
-* Creating a manual snapshot helps re-establish the replication relationship
-* Always verify replication status after the failback procedure
-* The first replication after failback will be a full send of the current state
-
-### Testing the failback scenario
-
-Here's a real test of the failback procedure:
-
-```sh
-# Simulate failure: Stop replication on f0
-paul@f0:~ % doas service `zrepl` stop
-
-# On f1: Take over by making the dataset writable
-paul@f1:~ % doas zfs set readonly=off zdata/sink/f0/zdata/enc/nfsdata
-
-# Write some data on f1 during the "outage"
-paul@f1:~ % echo 'Data written on f1 during failover' | doas tee /data/nfs/failover-data.txt
-Data written on f1 during failover
-
-# Now perform failback when f0 comes back online
-# Create snapshot on f1
-paul@f1:~ % doas zfs snapshot zdata/sink/f0/zdata/enc/nfsdata@failback
-
-# Send data back to f0 (note: we had to send to a temporary dataset due to holds)
-paul@f1:~ % doas zfs send -Rw zdata/sink/f0/zdata/enc/nfsdata@failback | \
-    ssh f0 "doas zfs recv -F zdata/enc/nfsdata_temp"
-
-# On f0: Rename datasets to complete failback
-paul@f0:~ % doas zfs set mountpoint=none zdata/enc/nfsdata
-paul@f0:~ % doas zfs rename zdata/enc/nfsdata zdata/enc/nfsdata_old
-paul@f0:~ % doas zfs rename zdata/enc/nfsdata_temp zdata/enc/nfsdata
-
-# Load encryption key and mount
-paul@f0:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata
-paul@f0:~ % doas zfs mount zdata/enc/nfsdata
-
-# Verify the data from f1 is now on f0
-paul@f0:~ % ls -la /data/nfs/
-total 18
-drwxr-xr-x  2 root wheel  4 Jul  2 00:01 .
-drwxr-xr-x  4 root wheel  4 Jul  1 23:41 ..
-*rw-r--r--  1 root wheel 35 Jul  2 00:01 failover-data.txt
-*rw-r--r--  1 root wheel 12 Jul  1 23:34 hello.txt
-```
-
-Success! The failover data from f1 is now on f0. To resume normal replication, you would need to:
-
-1. Clean up old snapshots on both sides
-2. Create a new manual baseline snapshot
-3. Restart `zrepl` services
-
-Key learnings from the test:
-
-* The `-w` flag is essential for encrypted datasets
-* Dataset holds can complicate the process (consider sending to a temporary dataset)
-* The encryption key must be loaded after receiving the dataset
-* Always verify data integrity before resuming normal operations
+And replication should work again!
 
 ### Troubleshooting: Files not appearing in replication
 
-If you write files to `/data/nfs/` on f0 but they don't appear on f1, check:
+If you write files to `/data/nfs/` on `f0` but they don't appear on f1, check Is the dataset actually mounted on f0?
 
 ```sh
-# 1. Is the dataset actually mounted on f0?
 paul@f0:~ % doas zfs list -o name,mountpoint,mounted | grep nfsdata
 zdata/enc/nfsdata                             /data/nfs             yes
+```
 
-# If it shows "no", the dataset isn't mounted!
-# This means files are being written to the root filesystem, not ZFS
+If it shows `no`, the dataset isn't mounted! This means files are being written to the root filesystem, not ZFS. Next, we should chcek whether the encryption key is loaded:
 
-# 2. Check if encryption key is loaded
+```sh
 paul@f0:~ % doas zfs get keystatus zdata/enc/nfsdata
 NAME               PROPERTY   VALUE        SOURCE
 zdata/enc/nfsdata  keystatus  available    -
-
 # If "unavailable", load the key:
 paul@f0:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata
 paul@f0:~ % doas zfs mount zdata/enc/nfsdata
+```
+
+You can also verify that files are in the snapshot (not just the directory):
 
-# 3. Verify files are in the snapshot (not just the directory)
+```sh
 paul@f0:~ % ls -la /data/nfs/.zfs/snapshot/zrepl_*/
 ```
 
@@ -893,12 +651,12 @@ This issue commonly occurs after reboot if the encryption keys aren't configured
 
 ### Configuring automatic key loading on boot
 
-To ensure all encrypted datasets are mounted automatically after reboot:
+To ensure all additional encrypted datasets are mounted automatically after reboot as well, we do:
 
 ```sh
 # On f0 - configure all encrypted datasets
 paul@f0:~ % doas sysrc zfskeys_enable=YES
-zfskeys_enable: NO -> YES
+zfskeys_enable: YES -> YES
 paul@f0:~ % doas sysrc zfskeys_datasets="zdata/enc zdata/enc/nfsdata zroot/bhyve"
 zfskeys_datasets:  -> zdata/enc zdata/enc/nfsdata zroot/bhyve
 
@@ -907,7 +665,7 @@ paul@f0:~ % doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key zd
 
 # On f1 - include the replicated dataset
 paul@f1:~ % doas sysrc zfskeys_enable=YES
-zfskeys_enable: NO -> YES
+zfskeys_enable: YES -> YES
 paul@f1:~ % doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata"
 zfskeys_datasets:  -> zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata
 
@@ -916,93 +674,32 @@ paul@f1:~ % doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key zd
 ```
 
 Important notes:
-* Each encryption root needs its own key load entry - child datasets don't inherit key loading
-* The replicated dataset on f1 uses the same encryption key as the source on f0
-* Always verify datasets are mounted after reboot with `zfs list -o name,mounted`
-* Critical: Always ensure the replicated dataset on f1 remains read-only with `doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata`
-
-### Troubleshooting: Replication broken due to modified destination
-
-If you see the error "cannot receive incremental stream: destination has been modified since most recent snapshot", it means the read-only flag was accidentally removed on f1. To fix without a full resync:
-
-```sh
-# Stop `zrepl` on both servers
-paul@f0:~ % doas service `zrepl` stop
-paul@f1:~ % doas service `zrepl` stop
-
-# Find the last common snapshot
-paul@f0:~ % doas zfs list -t snapshot -o name,creation zdata/enc/nfsdata
-paul@f1:~ % doas zfs list -t snapshot -o name,creation zdata/sink/f0/zdata/enc/nfsdata
-
-# Rollback f1 to the last common snapshot (example: @zrepl_20250705_000007_000)
-paul@f1:~ % doas zfs rollback -r zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250705_000007_000
-
-# Ensure the dataset is read-only
-paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
-
-# Restart zrepl
-paul@f0:~ % doas service `zrepl` start
-paul@f1:~ % doas service `zrepl` start
-```
-
-### Forcing a full resync
-
-If replication gets out of sync and incremental updates fail:
-
-```sh
-# Stop services
-paul@f0:~ % doas service `zrepl` stop
-paul@f1:~ % doas service `zrepl` stop
-
-# On f1: Release holds and destroy the dataset
-paul@f1:~ % doas zfs holds -r zdata/sink/f0/zdata/enc/nfsdata | \
-    grep -v NAME | awk '{print $2, $1}' | \
-    while read tag snap; do doas zfs release "$tag" "$snap"; done
-paul@f1:~ % doas zfs destroy -rf zdata/sink/f0/zdata/enc/nfsdata
-
-# On f0: Create fresh snapshot
-paul@f0:~ % doas zfs snapshot zdata/enc/nfsdata@resync
 
-# Send full dataset
-paul@f0:~ % doas zfs send -Rw zdata/enc/nfsdata@resync | \
-    ssh f1 "doas zfs recv zdata/sink/f0/zdata/enc/nfsdata"
-
-# Configure f1
-paul@f1:~ % doas zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata
-paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
-paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \
-    zdata/sink/f0/zdata/enc/nfsdata
-paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
-
-# Clean up and restart
-paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@resync
-paul@f1:~ % doas zfs destroy zdata/sink/f0/zdata/enc/nfsdata@resync
-paul@f0:~ % doas service `zrepl` start
-paul@f1:~ % doas service `zrepl` start
-```
-
-ZFS auto scrubbing....~?
-
-Backup of the keys on the key locations (all keys on all 3 USB keys)
+* Each encryption root needs its own key load entry
+* The replicated dataset on `f1` uses the same encryption key as the source on `f0`
+* Always verify datasets are mounted after reboot with `zfs list -o name,mounted`
+* Critical: Always ensure the replicated dataset on `f1` remains read-only with `doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata`
 
 ## CARP (Common Address Redundancy Protocol)
 
-High availability is crucial for storage systems. If the storage server goes down, all pods lose access to their persistent data. CARP provides a solution by creating a virtual IP address that automatically moves between servers during failures.
+High availability is crucial for storage systems. If the storage server goes down, all NFS clients (which will also be Kubernetes pods later on in this series) lose access to their persistent data. CARP provides a solution by creating a virtual IP address that automatically moves between servers during failures. This means that clients point to that VIP for NFS mounts and are always contacting the current master node.
 
 ### How CARP Works
 
-CARP allows two hosts to share a virtual IP address (VIP). The hosts communicate using multicast to elect a MASTER, while the other remain as BACKUP. When the MASTER fails, a BACKUP automatically promotes itself, and the VIP moves to the new MASTER. This happens within seconds.
+In our case, CARP allows two hosts (`f0` and `f1`) to share a virtual IP address (VIP). The hosts communicate using multicast to elect a MASTER, while the other remain as BACKUP. When the MASTER fails, a BACKUP automatically promotes itself, and the VIP moves to the new MASTER. This happens within seconds.
 
 Key benefits for our storage system:
 
-* Automatic failover: No manual intervention is required for basic failures, although there are a few limitations. The backup will only have read-only access to the available data, as we will learn later. However, we could manually promote it to read-write if needed.
+* Automatic failover: No manual intervention is required for basic failures, although there are a few limitations. The backup will only have read-only access by default to the available data, as we learned already.
 * Transparent to clients: Pods continue using the same IP address
-* Works with stunnel: Behind the VIP there will be a `stunnel` process running, which ensures encrypted connections follow the active server
-* Simple configuration
+* Works with `stunnel`: Behind the VIP there will be a `stunnel` process running, which ensures encrypted connections follow the active server.
+
+=> https://docs-archive.freebsd.org/doc/13.0-RELEASE/usr/local/share/doc/freebsd/en/books/handbook/carp.html FreeBSD CARP
+=> https://www.stunnel.org/ Stunnel
 
 ### Configuring CARP
 
-First, add the CARP configuration to `/etc/rc.conf` on both f0 and f1:
+First, we add the CARP configuration to `/etc/rc.conf` on both `f0` and `f1`:
 
 ```sh
 # The virtual IP 192.168.1.138 will float between f0 and f1
@@ -1027,8 +724,9 @@ This allows clients to connect to `f3s-storage-ha` regardless of which physical
 
 To properly manage services during failover, we need to detect CARP state changes. FreeBSD's devd system can notify us when CARP transitions between MASTER and BACKUP states.
 
-Add this to `/etc/devd.conf` on both f0 and f1:
+Add this to `/etc/devd.conf` on both `f0` and `f1`:
 
+```sh
 paul@f0:~ % cat <<END | doas tee -a /etc/devd.conf
 notify 0 {
         match "system"          "CARP";
@@ -1038,7 +736,10 @@ notify 0 {
 };
 END
 
-Next, create the CARP control script that will restart stunnel when CARP state changes:
+paul@f0:~ % doas service devd restart
+```
+
+Next, we create the CARP control script that will restart stunnel when CARP state changes:
 
 ```sh
 paul@f0:~ % doas tee /usr/local/bin/carpcontrol.sh <<'EOF'
@@ -1048,20 +749,9 @@ paul@f0:~ % doas tee /usr/local/bin/carpcontrol.sh <<'EOF'
 case "$1" in
     MASTER)
         logger "CARP state changed to MASTER, starting services"
-        service rpcbind start >/dev/null 2>&1
-        service mountd start >/dev/null 2>&1
-        service nfsd start >/dev/null 2>&1
-        service nfsuserd start >/dev/null 2>&1