add showcases

2 files changed, 1995 insertions, 79 deletions

diff --git a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.gmi b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.gmi
index 086179e6..3f1885eb 100644
--- a/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.gmi
+++ b/gemfeed/DRAFT-f3s-kubernetes-with-freebsd-part-6.gmi
@@ -21,8 +21,12 @@ This is the sixth blog post about the f3s series for self-hosting demands in a h
 * ⇢ ⇢ ⇢ Generating encryption keys
 * ⇢ ⇢ ⇢ Configuring `zdata` ZFS pool and encryption
 * ⇢ ⇢ ⇢ Migrating Bhyve VMs to encrypted `bhyve` ZFS volume
-* ⇢ ⇢ CARP
+* ⇢ ⇢ CARP (Common Address Redundancy Protocol)
+* ⇢ ⇢ ⇢ How CARP Works
+* ⇢ ⇢ ⇢ Configuring CARP
+* ⇢ ⇢ ⇢ CARP State Change Notifications
 * ⇢ ⇢ ZFS Replication with zrepl
+* ⇢ ⇢ ⇢ Understanding Replication Requirements
 * ⇢ ⇢ ⇢ Why zrepl instead of HAST?
 * ⇢ ⇢ ⇢ Installing zrepl
 * ⇢ ⇢ ⇢ Checking ZFS pools
@@ -35,21 +39,128 @@ This is the sixth blog post about the f3s series for self-hosting demands in a h
 * ⇢ ⇢ ⇢ A note about the Bhyve VM replication
 * ⇢ ⇢ ⇢ Quick status check commands
 * ⇢ ⇢ ⇢ Verifying replication after reboot
-* ⇢ ⇢ ⇢ Important note about failover limitations
+* ⇢ ⇢ ⇢ Understanding Failover Limitations and Design Decisions
+* ⇢ ⇢ ⇢# Why Manual Failover?
+* ⇢ ⇢ ⇢# Current Failover Process
 * ⇢ ⇢ ⇢ Mounting the NFS datasets
 * ⇢ ⇢ ⇢ Failback scenario: Syncing changes from f1 back to f0
 * ⇢ ⇢ ⇢ Testing the failback scenario
+* ⇢ ⇢ ⇢ Troubleshooting: Files not appearing in replication
+* ⇢ ⇢ ⇢ Configuring automatic key loading on boot
+* ⇢ ⇢ ⇢ Troubleshooting: Replication broken due to modified destination
+* ⇢ ⇢ ⇢ Forcing a full resync
+* ⇢ ⇢ Future Storage Explorations
+* ⇢ ⇢ ⇢ MinIO for S3-Compatible Object Storage
+* ⇢ ⇢ ⇢ MooseFS for Distributed High Availability
+* ⇢ ⇢ NFS Server Configuration
+* ⇢ ⇢ ⇢ Setting up NFS on f0 (Primary)
+* ⇢ ⇢ ⇢ Configuring Stunnel for NFS Encryption with CARP Failover
+* ⇢ ⇢ ⇢# Why Not Native NFS over TLS?
+* ⇢ ⇢ ⇢# Stunnel Architecture with CARP
+* ⇢ ⇢ ⇢# Creating a Certificate Authority for Client Authentication
+* ⇢ ⇢ ⇢# Install and Configure Stunnel on f0
+* ⇢ ⇢ ⇢ Setting up NFS on f1 (Standby)
+* ⇢ ⇢ ⇢ How Stunnel Works with CARP
+* ⇢ ⇢ ⇢ CARP Control Script for Clean Failover
+* ⇢ ⇢ ⇢ CARP Management Script
+* ⇢ ⇢ ⇢ Automatic Failback After Reboot
+* ⇢ ⇢ ⇢# Why Automatic Failback?
+* ⇢ ⇢ ⇢# The Auto-Failback Script
+* ⇢ ⇢ ⇢# Setting Up the Marker File
+* ⇢ ⇢ ⇢# Configuring Cron
+* ⇢ ⇢ ⇢# Managing Automatic Failback
+* ⇢ ⇢ ⇢# How It Works
+* ⇢ ⇢ ⇢ Verifying Stunnel and CARP Status
+* ⇢ ⇢ ⇢ Verifying NFS Exports
+* ⇢ ⇢ ⇢ Client Configuration for Stunnel
+* ⇢ ⇢ ⇢# Preparing Client Certificates
+* ⇢ ⇢ ⇢# Configuring Rocky Linux Clients (r0, r1, r2)
+* ⇢ ⇢ ⇢ Testing NFS Mount with Stunnel
+* ⇢ ⇢ ⇢ Important: Encryption Keys for Replicated Datasets
+* ⇢ ⇢ ⇢ NFS Failover with CARP and Stunnel
+* ⇢ ⇢ ⇢ Testing CARP Failover
+* ⇢ ⇢ ⇢ Handling Stale File Handles After Failover
+* ⇢ ⇢ ⇢ Complete Failover Test
+* ⇢ ⇢ ⇢ Verifying Replication Status
+* ⇢ ⇢ ⇢ Post-Reboot Verification
+* ⇢ ⇢ ⇢ Integration with Kubernetes
+* ⇢ ⇢ ⇢ Security Benefits of Stunnel with Client Certificates
+* ⇢ ⇢ ⇢ Laptop/Workstation Access
+* ⇢ ⇢ ⇢# Important: NFSv4 and Stunnel on Newer Linux Clients
+* ⇢ ⇢ Mounting NFS on Rocky Linux 9
+* ⇢ ⇢ ⇢ Installing and Configuring NFS Clients on r0, r1, and r2
+* ⇢ ⇢ ⇢ Configuring Stunnel Client on All Nodes
+* ⇢ ⇢ ⇢ Setting Up NFS Mounts
+* ⇢ ⇢ ⇢ Comprehensive NFS Mount Testing
+* ⇢ ⇢ ⇢# Test 1: Verify Mount Status on All Nodes
+* ⇢ ⇢ ⇢# Test 2: Verify Stunnel Connectivity
+* ⇢ ⇢ ⇢# Test 3: File Creation and Visibility Test
+* ⇢ ⇢ ⇢# Test 4: Verify Files on Storage Servers
+* ⇢ ⇢ ⇢# Test 5: Performance and Concurrent Access Test
+* ⇢ ⇢ ⇢# Test 6: Directory Operations Test
+* ⇢ ⇢ ⇢# Test 7: Permission and Ownership Test
+* ⇢ ⇢ ⇢# Test 8: Failover Test (Optional but Recommended)
+* ⇢ ⇢ ⇢ Troubleshooting Common Issues
+* ⇢ ⇢ ⇢# Mount Hangs or Times Out
+* ⇢ ⇢ ⇢# Permission Denied Errors
+* ⇢ ⇢ ⇢# Files Not Visible Across Nodes
+* ⇢ ⇢ ⇢# I/O Errors When Accessing NFS Mount
+* ⇢ ⇢ ⇢ Comprehensive Production Test Results
+* ⇢ ⇢ ⇢# Test Scenario: Full System Reboot and Failover
+* ⇢ ⇢ ⇢# Key Findings
+* ⇢ ⇢ Performance Considerations
+* ⇢ ⇢ ⇢ Encryption Overhead
+* ⇢ ⇢ ⇢ Replication Bandwidth
+* ⇢ ⇢ ⇢ NFS Tuning
+* ⇢ ⇢ ⇢ ZFS Tuning
+* ⇢ ⇢ ⇢ Monitoring
+* ⇢ ⇢ ⇢ Cleanup After Testing
+* ⇢ ⇢ Conclusion
+* ⇢ ⇢ ⇢ What We Achieved
+* ⇢ ⇢ ⇢ Architecture Benefits
+* ⇢ ⇢ ⇢ Lessons Learned
+* ⇢ ⇢ ⇢ Next Steps
+* ⇢ ⇢ ⇢ References
 
 ## Introduction
 
-In this blog post, we are going to extend the Beelinks with some additional storage.
+In the previous posts, we set up a FreeBSD-based Kubernetes cluster using k3s. While the base system works well, Kubernetes workloads often require persistent storage for databases, configuration files, and application data. Local storage on each node has significant limitations:
 
-Some photos here, describe why there are 2 different models of SSD drives (replication etc)
+* No data sharing: Pods on different nodes can't access the same data
+* Pod mobility: If a pod moves to another node, it loses access to its data
+* No redundancy: Hardware failure means data loss
+* Limited capacity: Individual nodes have finite storage
+
+This post implements a robust storage solution using:
+
+* ZFS: For data integrity, encryption, and efficient snapshots
+* CARP: For high availability with automatic IP failover
+* NFS over stunnel: For secure, encrypted network storage
+ zrepl: For continuous replication between nodes
+
+The end result is a highly available, encrypted storage system that survives node failures while providing shared storage to all Kubernetes pods. We're using two different SSD models (Samsung 870 EVO and Crucial BX500) to avoid simultaneous failures from the same manufacturing batch.
 
 ## ZFS encryption keys
 
+ZFS native encryption requires encryption keys to unlock datasets. We need a secure method to store these keys that balances security with operational needs:
+
+* Security: Keys must not be stored on the same disks they encrypt
+* Availability: Keys must be available at boot for automatic mounting
+* Portability: Keys should be easily moved between systems for recovery
+
+Using USB flash drives as hardware key storage provides an elegant solution. The encrypted data is unreadable without physical access to the USB key, protecting against disk theft or improper disposal. In production environments, you might use enterprise key management systems, but for a home lab, USB keys offer good security with minimal complexity.
+
 ### UFS on USB keys
 
+We'll format the USB drives with UFS (Unix File System) rather than ZFS for several reasons:
+
+* Simplicity: UFS has less overhead for small, removable media
+* Reliability: No ZFS pool import/export issues with removable devices
+
+Let's see the USB keys:
+
+TODO: Insert photos here
+
 ```
 paul@f0:/ % doas camcontrol devlist
 <512GB SSD D910R170>               at scbus0 target 0 lun 0 (pass0,ada0)
@@ -66,6 +177,8 @@ paul@f1:/ % doas camcontrol devlist
 paul@f1:/ %
 ```
 
+Let's create the UFS file system and mount it (done on all 3 nodes `f0`, `f1` and `f2`):
+
 ```sh
 paul@f0:/ % doas newfs /dev/da0
 /dev/da0: 15000.0MB (30720000 sectors) block size 32768, fragment size 4096
@@ -86,6 +199,9 @@ paul@f0:/ % df | grep keys
 
 ### Generating encryption keys
 
+The following keys will later be used to encrypt the ZFS file systems:
+
+```
 paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:bhyve.key 32
 paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:bhyve.key 32
 paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:bhyve.key 32
@@ -97,14 +213,15 @@ paul@f0:/keys % doas chmod 400 *
 
 paul@f0:/keys % ls -l
 total 20
--r--------  1 root wheel 32 May 25 13:07 f0.lan.buetow.org:bhyve.key
--r--------  1 root wheel 32 May 25 13:07 f1.lan.buetow.org:bhyve.key
--r--------  1 root wheel 32 May 25 13:07 f2.lan.buetow.org:bhyve.key
--r--------  1 root wheel 32 May 25 13:07 f0.lan.buetow.org:zdata.key
--r--------  1 root wheel 32 May 25 13:07 f1.lan.buetow.org:zdata.key
--r--------  1 root wheel 32 May 25 13:07 f2.lan.buetow.org:zdata.key
+*r--------  1 root wheel 32 May 25 13:07 f0.lan.buetow.org:bhyve.key
+*r--------  1 root wheel 32 May 25 13:07 f1.lan.buetow.org:bhyve.key
+*r--------  1 root wheel 32 May 25 13:07 f2.lan.buetow.org:bhyve.key
+*r--------  1 root wheel 32 May 25 13:07 f0.lan.buetow.org:zdata.key
+*r--------  1 root wheel 32 May 25 13:07 f1.lan.buetow.org:zdata.key
+*r--------  1 root wheel 32 May 25 13:07 f2.lan.buetow.org:zdata.key
+````
 
-Copy those to all 3 nodes to /keys
+After creation, those are copied to the other two nodes `f1` and `f2` to the `/keys` partition.
 
 ### Configuring `zdata` ZFS pool and encryption
 
@@ -180,16 +297,48 @@ zroot/bhyve/rocky  encryptionroot        zroot/bhyve            -
 zroot/bhyve/rocky  keystatus             available              -
 ```
 
-## CARP
+## CARP (Common Address Redundancy Protocol)
+
+High availability is crucial for storage systems. If the NFS server goes down, all pods lose access to their persistent data. CARP provides a solution by creating a virtual IP address that automatically moves between servers during failures.
+
+### How CARP Works
+
+CARP allows multiple hosts to share a virtual IP address (VIP). The hosts communicate using multicast to elect a MASTER, while others remain as BACKUP. When the MASTER fails, a BACKUP automatically promotes itself, and the VIP moves to the new MASTER. This happens within seconds, minimizing downtime.
+
+Key benefits for our storage system:
+* Automatic failover: No manual intervention required for basic failures
+* Transparent to clients: Pods continue using the same IP address
+* Works with stunnel: The VIP ensures encrypted connections follow the active server
+* Simple configuration: Just a single line in rc.conf
+
+### Configuring CARP
 
-adding to /etc/rc.conf on f0 and f1:
+First, add the CARP configuration to `/etc/rc.conf` on both f0 and f1:
+
+```sh
+# The virtual IP 192.168.1.138 will float between f0 and f1
 ifconfig_re0_alias0="inet vhid 1 pass testpass alias 192.168.1.138/32"
+```
 
-adding to /etc/hosts:
+Parameters explained:
+* `vhid 1`: Virtual Host ID - must match on all CARP members
+* `pass testpass`: Password for CARP authentication (use a stronger password in production)
+* `alias 192.168.1.138/32`: The virtual IP address with a /32 netmask
 
+Next, update `/etc/hosts` on all nodes (n0, n1, n2, r0, r1, r2) to resolve the VIP hostname:
+
+```
 192.168.1.138 f3s-storage-ha f3s-storage-ha.lan f3s-storage-ha.lan.buetow.org
+192.168.2.138 f3s-storage-ha f3s-storage-ha.wg0 f3s-storage-ha.wg0.wan.buetow.org
+```
+
+This allows clients to connect to `f3s-storage-ha` regardless of which physical server is currently the MASTER.
+
+### CARP State Change Notifications
+
+To properly manage services during failover, we need to detect CARP state changes. FreeBSD's devd system can notify us when CARP transitions between MASTER and BACKUP states.
 
-Adding on f0 and f1:
+Add this to `/etc/devd.conf` on both f0 and f1:
 
 paul@f0:~ % cat <<END | doas tee -a /etc/devd.conf
 notify 0 {
@@ -200,29 +349,78 @@ notify 0 {
 };
 END
 
-next, copied that script /usr/local/bin/carpcontrol.sh and adjusted the disk to storage
+Next, create the CARP control script that will restart stunnel when CARP state changes:
 
-/boot/loader.conf add carp_load="YES"
-reboot or run doas kldload carp0 
+```sh
+paul@f0:~ % doas tee /usr/local/bin/carpcontrol.sh <<'EOF'
+#!/bin/sh
+# CARP state change handler for storage failover
+
+subsystem=$1
+state=$2
+
+logger "CARP state change: $subsystem is now $state"
+
+case "$state" in
+    MASTER)
+        # Restart stunnel to bind to the VIP
+        service stunnel restart
+        logger "Restarted stunnel for MASTER state"
+        ;;
+    BACKUP)
+        # Stop stunnel since we can't bind to VIP as BACKUP
+        service stunnel stop
+        logger "Stopped stunnel for BACKUP state"
+        ;;
+esac
+EOF
+
+paul@f0:~ % doas chmod +x /usr/local/bin/carpcontrol.sh
+
+# Copy the same script to f1
+paul@f0:~ % scp /usr/local/bin/carpcontrol.sh f1:/tmp/
+paul@f1:~ % doas mv /tmp/carpcontrol.sh /usr/local/bin/
+paul@f1:~ % doas chmod +x /usr/local/bin/carpcontrol.sh
+```
+
+Enable CARP in /boot/loader.conf:
+
+```sh
+paul@f0:~ % echo 'carp_load="YES"' | doas tee -a /boot/loader.conf
+carp_load="YES"
+paul@f1:~ % echo 'carp_load="YES"' | doas tee -a /boot/loader.conf  
+carp_load="YES"
+```
+
+Then reboot both hosts or run `doas kldload carp` to load the module immediately. 
 
 
 ## ZFS Replication with zrepl
 
-In this section, we'll set up automatic ZFS replication from f0 to f1 using zrepl. This ensures our data is replicated across nodes for redundancy.
+Data replication is the cornerstone of high availability. While CARP handles IP failover, we need continuous data replication to ensure the backup server has current data when it becomes active. Without replication, failover would result in data loss or require shared storage (like iSCSI), which introduces a single point of failure.
+
+### Understanding Replication Requirements
+
+Our storage system has different replication needs:
+
+* NFS data (`/data/nfs/k3svolumes`): Contains active Kubernetes persistent volumes. Needs frequent replication (every minute) to minimize data loss during failover.
+* VM data (`/zroot/bhyve/fedora`): Contains VM images that change less frequently. Can tolerate longer replication intervals (every 10 minutes).
+
+The replication frequency determines your Recovery Point Objective (RPO) - the maximum acceptable data loss. With 1-minute replication, you lose at most 1 minute of changes during an unplanned failover.
 
 ### Why zrepl instead of HAST?
 
 While HAST (Highly Available Storage) is FreeBSD's native solution for high-availability storage, I've chosen zrepl for several important reasons:
 
-1. **HAST can cause ZFS corruption**: HAST operates at the block level and doesn't understand ZFS's transactional semantics. During failover, in-flight transactions can lead to corrupted zpools. I've experienced this firsthand - the automatic failover would trigger while ZFS was still writing, resulting in an unmountable pool.
+1. HAST can cause ZFS corruption: HAST operates at the block level and doesn't understand ZFS's transactional semantics. During failover, in-flight transactions can lead to corrupted zpools. I've experienced this firsthand - the automatic failover would trigger while ZFS was still writing, resulting in an unmountable pool.
 
-2. **ZFS-aware replication**: zrepl understands ZFS datasets and snapshots. It replicates at the dataset level, ensuring each snapshot is a consistent point-in-time copy. This is fundamentally safer than block-level replication.
+2. ZFS-aware replication: zrepl understands ZFS datasets and snapshots. It replicates at the dataset level, ensuring each snapshot is a consistent point-in-time copy. This is fundamentally safer than block-level replication.
 
-3. **Snapshot history**: With zrepl, you get multiple recovery points (every 5 minutes in our setup). If corruption occurs, you can roll back to any previous snapshot. HAST only gives you the current state.
+3. Snapshot history: With zrepl, you get multiple recovery points (every minute for NFS data in our setup). If corruption occurs, you can roll back to any previous snapshot. HAST only gives you the current state.
 
-4. **Easier recovery**: When something goes wrong with zrepl, you still have intact snapshots on both sides. With HAST, a corrupted primary often means a corrupted secondary too.
+4. Easier recovery: When something goes wrong with zrepl, you still have intact snapshots on both sides. With HAST, a corrupted primary often means a corrupted secondary too.
 
-5. **Network flexibility**: zrepl works over any TCP connection (in our case, WireGuard), while HAST requires dedicated network configuration.
+5. Network flexibility: zrepl works over any TCP connection (in our case, WireGuard), while HAST requires dedicated network configuration.
 
 The 5-minute replication window is perfectly acceptable for my personal use cases. This isn't a high-frequency trading system or a real-time database - it's storage for personal projects, development work, and home lab experiments. Losing at most 5 minutes of work in a disaster scenario is a reasonable trade-off for the reliability and simplicity of snapshot-based replication.
 
@@ -297,20 +495,40 @@ global:
       format: human
 
 jobs:
-  - name: f0_to_f1
+  - name: f0_to_f1_nfsdata
     type: push
     connect:
       type: tcp
       address: "192.168.2.131:8888"
     filesystems:
       "zdata/enc/nfsdata": true
+    send:
+      encrypted: false
+    snapshotting:
+      type: periodic
+      prefix: zrepl_
+      interval: 1m
+    pruning:
+      keep_sender:
+        - type: last_n
+          count: 10
+      keep_receiver:
+        - type: last_n
+          count: 10
+
+  - name: f0_to_f1_fedora
+    type: push
+    connect:
+      type: tcp
+      address: "192.168.2.131:8888"
+    filesystems:
       "zroot/bhyve/fedora": true
     send:
-      encrypted: true
+      encrypted: false
     snapshotting:
       type: periodic
       prefix: zrepl_
-      interval: 5m
+      interval: 10m
     pruning:
       keep_sender:
         - type: last_n
@@ -321,7 +539,12 @@ jobs:
 EOF
 ```
 
-Note: We're specifically replicating `zdata/enc/nfsdata` instead of the entire `zdata/enc` dataset. This dedicated dataset will contain all the data we later want to expose via NFS, keeping a clear separation between replicated NFS data and other local encrypted data.
+Key configuration notes:
+* We're using two separate replication jobs with different intervals:
+  - `f0_to_f1_nfsdata`: Replicates NFS data every minute for faster failover recovery
+  - `f0_to_f1_fedora`: Replicates Fedora VM every 10 minutes (less critical for NFS operations)
+* We're specifically replicating `zdata/enc/nfsdata` instead of the entire `zdata/enc` dataset. This dedicated dataset will contain all the data we later want to expose via NFS, keeping a clear separation between replicated NFS data and other local encrypted data.
+* The `send: encrypted: false` option disables ZFS native encryption for the replication stream. Since we're using a WireGuard tunnel between f0 and f1, the data is already encrypted in transit. Disabling ZFS stream encryption reduces CPU overhead and improves replication performance.
 
 ### Configuring zrepl on f1 (sink)
 
@@ -504,9 +727,23 @@ zdata/sink/f0/zroot/bhyve/fedora@zrepl_20250701_202530_000     0B      -  2.97G
 
 The timestamps confirm that replication resumed automatically after the reboot, ensuring continuous data protection.
 
-### Important note about failover limitations
+### Understanding Failover Limitations and Design Decisions
+
+#### Why Manual Failover?
+
+This storage system intentionally uses manual failover rather than automatic failover. This might seem counterintuitive for a "high availability" system, but it's a deliberate design choice based on real-world experience:
+
+1. Split-brain prevention: Automatic failover can cause both nodes to become active simultaneously if network communication fails. This leads to data divergence that's extremely difficult to resolve.
+
+2. False positive protection: Temporary network issues or high load can trigger unwanted failovers. Manual intervention ensures failovers only occur when truly necessary.
+
+3. Data integrity over availability: For storage systems, data consistency is paramount. A few minutes of downtime is preferable to data corruption or loss.
 
-The current zrepl setup provides **backup/disaster recovery** but not automatic failover. The replicated datasets on f1 are not mounted by default (`mountpoint=none`). In case f0 fails:
+4. Simplified recovery: With manual failover, you always know which dataset is authoritative, making recovery straightforward.
+
+#### Current Failover Process
+
+The replicated datasets on f1 are intentionally not mounted (`mountpoint=none`). In case f0 fails:
 
 ```sh
 # Manual steps needed on f1 to activate the replicated data:
@@ -514,7 +751,7 @@ paul@f1:~ % doas zfs set mountpoint=/data/nfsdata zdata/sink/f0/zdata/enc/nfsdat
 paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
 ```
 
-However, this creates a **split-brain problem**: when f0 comes back online, both systems would have diverged data. Resolving this requires careful manual intervention to:
+However, this creates a split-brain problem: when f0 comes back online, both systems would have diverged data. Resolving this requires careful manual intervention to:
 
 1. Stop the original replication
 2. Sync changes from f1 back to f0
@@ -522,9 +759,9 @@ However, this creates a **split-brain problem**: when f0 comes back online, both
 
 For true high-availability NFS, you might consider:
 
-* **Shared storage** (like iSCSI) with proper clustering
-* **GlusterFS** or similar distributed filesystems
-* **Manual failover with ZFS replication** (as we have here)
+* Shared storage (like iSCSI) with proper clustering
+* GlusterFS or similar distributed filesystems
+* Manual failover with ZFS replication (as we have here)
 
 Note: While HAST+CARP is often suggested for HA storage, it can cause filesystem corruption in practice, especially with ZFS. The block-level replication of HAST doesn't understand ZFS's transactional model, leading to inconsistent states during failover. 
 
@@ -578,7 +815,7 @@ zdata/sink/f0/zdata/enc/nfsdata    896G    204K    896G     0%    /data/nfs
 
 Note: The dataset is mounted at the same path (`/data/nfs`) on both hosts to simplify failover procedures. The dataset on f1 is set to `readonly=on` to prevent accidental modifications that would break replication.
 
-**CRITICAL WARNING**: Do NOT write to `/data/nfs/` on f1! Any modifications will break the replication. If you accidentally write to it, you'll see this error:
+CRITICAL WARNING: Do NOT write to `/data/nfs/` on f1! Any modifications will break the replication. If you accidentally write to it, you'll see this error:
 
 ```
 cannot receive incremental stream: destination zdata/sink/f0/zdata/enc/nfsdata has been modified
@@ -594,11 +831,6 @@ paul@f1:~ % doas zfs rollback zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250701_204
 paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
 ```
 
-To ensure the encryption key is loaded automatically after reboot on f1:
-```sh
-paul@f1:~ % doas sysrc zfskeys_datasets="zdata/sink/f0/zdata/enc/nfsdata"
-```
-
 ### Failback scenario: Syncing changes from f1 back to f0
 
 In a disaster recovery scenario where f0 has failed and f1 has taken over, you'll need to sync changes back when f0 returns. Here's how to failback:
@@ -678,7 +910,7 @@ paul@f1:~ % doas service zrepl start
 paul@f0:~ % doas zrepl status --job f0_to_f1
 ```
 
-**Important notes about failback**:
+Important notes about failback:
 
 * The `-F` flag forces a rollback on f0, destroying any local changes
 * Replication often won't resume automatically after a forced receive
@@ -724,8 +956,8 @@ paul@f0:~ % ls -la /data/nfs/
 total 18
 drwxr-xr-x  2 root wheel  4 Jul  2 00:01 .
 drwxr-xr-x  4 root wheel  4 Jul  1 23:41 ..
--rw-r--r--  1 root wheel 35 Jul  2 00:01 failover-data.txt
--rw-r--r--  1 root wheel 12 Jul  1 23:34 hello.txt
+*rw-r--r--  1 root wheel 35 Jul  2 00:01 failover-data.txt
+*rw-r--r--  1 root wheel 12 Jul  1 23:34 hello.txt
 ```
 
 Success! The failover data from f1 is now on f0. To resume normal replication, you would need to:
@@ -734,17 +966,1728 @@ Success! The failover data from f1 is now on f0. To resume normal replication, y
 2. Create a new manual baseline snapshot
 3. Restart zrepl services
 
-**Key learnings from the test**:
+Key learnings from the test:
 
 * The `-w` flag is essential for encrypted datasets
 * Dataset holds can complicate the process (consider sending to a temporary dataset)
 * The encryption key must be loaded after receiving the dataset
 * Always verify data integrity before resuming normal operations
 
+### Troubleshooting: Files not appearing in replication
+
+If you write files to `/data/nfs/` on f0 but they don't appear on f1, check:
+
+```sh
+# 1. Is the dataset actually mounted on f0?
+paul@f0:~ % doas zfs list -o name,mountpoint,mounted | grep nfsdata
+zdata/enc/nfsdata                             /data/nfs             yes
+
+# If it shows "no", the dataset isn't mounted!
+# This means files are being written to the root filesystem, not ZFS
+
+# 2. Check if encryption key is loaded
+paul@f0:~ % doas zfs get keystatus zdata/enc/nfsdata
+NAME               PROPERTY   VALUE        SOURCE
+zdata/enc/nfsdata  keystatus  available    -
+
+# If "unavailable", load the key:
+paul@f0:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata
+paul@f0:~ % doas zfs mount zdata/enc/nfsdata
+
+# 3. Verify files are in the snapshot (not just the directory)
+paul@f0:~ % ls -la /data/nfs/.zfs/snapshot/zrepl_*/
+```
+
+This issue commonly occurs after reboot if the encryption keys aren't configured to load automatically.
+
+### Configuring automatic key loading on boot
+
+To ensure all encrypted datasets are mounted automatically after reboot:
+
+```sh
+# On f0 - configure all encrypted datasets
+paul@f0:~ % doas sysrc zfskeys_enable=YES
+zfskeys_enable: NO -> YES
+paul@f0:~ % doas sysrc zfskeys_datasets="zdata/enc zdata/enc/nfsdata zroot/bhyve"
+zfskeys_datasets:  -> zdata/enc zdata/enc/nfsdata zroot/bhyve
+
+# Set correct key locations for all datasets
+paul@f0:~ % doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key zdata/enc/nfsdata
+
+# On f1 - include the replicated dataset
+paul@f1:~ % doas sysrc zfskeys_enable=YES
+zfskeys_enable: NO -> YES
+paul@f1:~ % doas sysrc zfskeys_datasets="zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata"
+zfskeys_datasets:  -> zdata/enc zroot/bhyve zdata/sink/f0/zdata/enc/nfsdata
+
+# Set key location for replicated dataset
+paul@f1:~ % doas zfs set keylocation=file:///keys/f0.lan.buetow.org:zdata.key zdata/sink/f0/zdata/enc/nfsdata
+```
+
+Important notes:
+* Each encryption root needs its own key load entry - child datasets don't inherit key loading
+* The replicated dataset on f1 uses the same encryption key as the source on f0
+* Always verify datasets are mounted after reboot with `zfs list -o name,mounted`
+* Critical: Always ensure the replicated dataset on f1 remains read-only with `doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata`
+
+### Troubleshooting: Replication broken due to modified destination
+
+If you see the error "cannot receive incremental stream: destination has been modified since most recent snapshot", it means the read-only flag was accidentally removed on f1. To fix without a full resync:
+
+```sh
+# Stop zrepl on both servers
+paul@f0:~ % doas service zrepl stop
+paul@f1:~ % doas service zrepl stop
+
+# Find the last common snapshot
+paul@f0:~ % doas zfs list -t snapshot -o name,creation zdata/enc/nfsdata
+paul@f1:~ % doas zfs list -t snapshot -o name,creation zdata/sink/f0/zdata/enc/nfsdata
+
+# Rollback f1 to the last common snapshot (example: @zrepl_20250705_000007_000)
+paul@f1:~ % doas zfs rollback -r zdata/sink/f0/zdata/enc/nfsdata@zrepl_20250705_000007_000
+
+# Ensure the dataset is read-only
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+
+# Restart zrepl
+paul@f0:~ % doas service zrepl start
+paul@f1:~ % doas service zrepl start
+```
+
+### Forcing a full resync
+
+If replication gets out of sync and incremental updates fail:
+
+```sh
+# Stop services
+paul@f0:~ % doas service zrepl stop
+paul@f1:~ % doas service zrepl stop
+
+# On f1: Release holds and destroy the dataset
+paul@f1:~ % doas zfs holds -r zdata/sink/f0/zdata/enc/nfsdata | \
+    grep -v NAME | awk '{print $2, $1}' | \
+    while read tag snap; do doas zfs release "$tag" "$snap"; done
+paul@f1:~ % doas zfs destroy -rf zdata/sink/f0/zdata/enc/nfsdata
+
+# On f0: Create fresh snapshot
+paul@f0:~ % doas zfs snapshot zdata/enc/nfsdata@resync
+
+# Send full dataset
+paul@f0:~ % doas zfs send -Rw zdata/enc/nfsdata@resync | \
+    ssh f1 "doas zfs recv zdata/sink/f0/zdata/enc/nfsdata"
+
+# Configure f1
+paul@f1:~ % doas zfs set mountpoint=/data/nfs zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs set readonly=on zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs load-key -L file:///keys/f0.lan.buetow.org:zdata.key \
+    zdata/sink/f0/zdata/enc/nfsdata
+paul@f1:~ % doas zfs mount zdata/sink/f0/zdata/enc/nfsdata
+
+# Clean up and restart
+paul@f0:~ % doas zfs destroy zdata/enc/nfsdata@resync
+paul@f1:~ % doas zfs destroy zdata/sink/f0/zdata/enc/nfsdata@resync
+paul@f0:~ % doas service zrepl start
+paul@f1:~ % doas service zrepl start
+```
+
 ZFS auto scrubbing....~?
 
 Backup of the keys on the key locations (all keys on all 3 USB keys)
 
+## Future Storage Explorations
+
+While zrepl provides excellent snapshot-based replication for disaster recovery, there are other storage technologies worth exploring for the f3s project:
+
+### MinIO for S3-Compatible Object Storage
+
+MinIO is a high-performance, S3-compatible object storage system that could complement our ZFS-based storage. Some potential use cases:
+
+* S3 API compatibility: Many modern applications expect S3-style object storage APIs. MinIO could provide this interface while using our ZFS storage as the backend.
+* Multi-site replication: MinIO supports active-active replication across multiple sites, which could work well with our f0/f1/f2 node setup.
+* Kubernetes native: MinIO has excellent Kubernetes integration with operators and CSI drivers, making it ideal for the f3s k3s environment.
+
+### MooseFS for Distributed High Availability
+
+MooseFS is a fault-tolerant, distributed file system that could provide true high-availability storage:
+
+* True HA: Unlike our current setup which requires manual failover, MooseFS provides automatic failover with no single point of failure.
+* POSIX compliance: Applications can use MooseFS like any regular filesystem, no code changes needed.
+* Flexible redundancy: Configure different replication levels per directory or file, optimizing storage efficiency.
+* FreeBSD support: MooseFS has native FreeBSD support, making it a natural fit for the f3s project.
+
+Both technologies could potentially run on top of our encrypted ZFS volumes, combining ZFS's data integrity and encryption features with distributed storage capabilities. This would be particularly interesting for workloads that need either S3-compatible APIs (MinIO) or transparent distributed POSIX storage (MooseFS).
+
+## NFS Server Configuration
+
+With ZFS replication in place, we can now set up NFS servers on both f0 and f1 to export the replicated data. Since native NFS over TLS (RFC 9289) has compatibility issues between Linux and FreeBSD, we'll use stunnel to provide encryption.
+
+### Setting up NFS on f0 (Primary)
+
+First, enable the NFS services in rc.conf:
+
+```sh
+paul@f0:~ % doas sysrc nfs_server_enable=YES
+nfs_server_enable: YES -> YES
+paul@f0:~ % doas sysrc nfsv4_server_enable=YES
+nfsv4_server_enable: YES -> YES
+paul@f0:~ % doas sysrc nfsuserd_enable=YES
+nfsuserd_enable: YES -> YES
+paul@f0:~ % doas sysrc mountd_enable=YES
+mountd_enable: NO -> YES
+paul@f0:~ % doas sysrc rpcbind_enable=YES
+rpcbind_enable: NO -> YES
+```
+
+Create a dedicated directory for Kubernetes volumes:
+
+```sh
+# First ensure the dataset is mounted
+paul@f0:~ % doas zfs get mounted zdata/enc/nfsdata
+NAME               PROPERTY  VALUE    SOURCE
+zdata/enc/nfsdata  mounted   yes      -
+
+# Create the k3svolumes directory
+paul@f0:~ % doas mkdir -p /data/nfs/k3svolumes
+paul@f0:~ % doas chmod 755 /data/nfs/k3svolumes
+
+# This directory will be replicated to f1 automatically
+```
+
+Create the /etc/exports file. Since we're using stunnel for encryption, ALL clients must connect through stunnel, which appears as localhost (127.0.0.1) to the NFS server:
+
+```sh
+paul@f0:~ % doas tee /etc/exports <<'EOF'
+V4: /data/nfs -sec=sys
+/data/nfs -alldirs -maproot=root -network 127.0.0.1 -mask 255.255.255.255
+EOF
+```
+
+The exports configuration:
+
+* `V4: /data/nfs -sec=sys`: Sets the NFSv4 root directory to /data/nfs
+* `/data/nfs -alldirs`: Allows mounting any subdirectory under /data/nfs
+* `-maproot=root`: Maps root user from client to root on server (needed for Kubernetes and ownership changes)
+* `-network 127.0.0.1`: Only accepts connections from localhost (stunnel)
+
+Note: 
+* ALL clients (r0, r1, r2, laptop) must connect through stunnel for encryption
+* Stunnel proxies connections through localhost, so only 127.0.0.1 needs access
+* With NFSv4, clients mount using relative paths (e.g., `/k3svolumes` instead of `/data/nfs/k3svolumes`)
+
+Start the NFS services:
+
+```sh
+paul@f0:~ % doas service rpcbind start
+Starting rpcbind.
+paul@f0:~ % doas service mountd start
+Starting mountd.
+paul@f0:~ % doas service nfsd start
+Starting nfsd.
+paul@f0:~ % doas service nfsuserd start
+Starting nfsuserd.
+```
+
+### Configuring Stunnel for NFS Encryption with CARP Failover
+
+#### Why Not Native NFS over TLS?
+
+FreeBSD 13+ supports native NFS over TLS (RFC 9289), which would be the ideal solution. However, there are significant compatibility challenges:
+
+* Linux client support is incomplete: Most Linux distributions don't fully support NFS over TLS yet
+* Certificate management differs: FreeBSD and Linux handle TLS certificates differently for NFS
+* Kernel module requirements: Requires specific kernel modules that may not be available
+
+Stunnel provides a more compatible solution that works reliably across all operating systems while offering equivalent security.
+
+#### Stunnel Architecture with CARP
+
+Stunnel integrates seamlessly with our CARP setup:
+
+```
+                    CARP VIP (192.168.1.138)
+                           |
+    f0 (MASTER) ←---------→|←---------→ f1 (BACKUP)
+    stunnel:2323           |           stunnel:stopped
+    nfsd:2049              |           nfsd:stopped
+                           |
+                    Clients connect here
+```
+
+The key insight is that stunnel binds to the CARP VIP. When CARP fails over, the VIP moves to the new MASTER, and stunnel starts there automatically. Clients maintain their connection to the same IP throughout.
+
+#### Creating a Certificate Authority for Client Authentication
+
+First, create a CA to sign both server and client certificates:
+
+```sh
+# On f0 - Create CA
+paul@f0:~ % doas mkdir -p /usr/local/etc/stunnel/ca
+paul@f0:~ % cd /usr/local/etc/stunnel/ca
+paul@f0:~ % doas openssl genrsa -out ca-key.pem 4096
+paul@f0:~ % doas openssl req -new -x509 -days 3650 -key ca-key.pem -out ca-cert.pem \
+  -subj '/C=US/ST=State/L=City/O=F3S Storage/CN=F3S Stunnel CA'
+
+# Create server certificate
+paul@f0:~ % cd /usr/local/etc/stunnel
+paul@f0:~ % doas openssl genrsa -out server-key.pem 4096
+paul@f0:~ % doas openssl req -new -key server-key.pem -out server.csr \
+  -subj '/C=US/ST=State/L=City/O=F3S Storage/CN=f3s-storage-ha.lan'
+paul@f0:~ % doas openssl x509 -req -days 3650 -in server.csr -CA ca/ca-cert.pem \
+  -CAkey ca/ca-key.pem -CAcreateserial -out server-cert.pem
+
+# Create client certificates for authorized clients
+paul@f0:~ % cd /usr/local/etc/stunnel/ca
+paul@f0:~ % doas sh -c 'for client in r0 r1 r2 earth; do 
+  openssl genrsa -out ${client}-key.pem 4096
+  openssl req -new -key ${client}-key.pem -out ${client}.csr \
+    -subj "/C=US/ST=State/L=City/O=F3S Storage/CN=${client}.lan.buetow.org"
+  openssl x509 -req -days 3650 -in ${client}.csr -CA ca-cert.pem \
+    -CAkey ca-key.pem -CAcreateserial -out ${client}-cert.pem
+done'
+```
+
+#### Install and Configure Stunnel on f0
+
+```sh
+# Install stunnel
+paul@f0:~ % doas pkg install -y stunnel
+
+# Configure stunnel server with client certificate authentication
+paul@f0:~ % doas tee /usr/local/etc/stunnel/stunnel.conf <<'EOF'
+cert = /usr/local/etc/stunnel/server-cert.pem
+key = /usr/local/etc/stunnel/server-key.pem
+
+setuid = stunnel
+setgid = stunnel
+
+[nfs-tls]
+accept = 192.168.1.138:2323
+connect = 127.0.0.1:2049
+CAfile = /usr/local/etc/stunnel/ca/ca-cert.pem
+verify = 2
+requireCert = yes
+EOF
+
+# Enable and start stunnel
+paul@f0:~ % doas sysrc stunnel_enable=YES
+stunnel_enable:  -> YES
+paul@f0:~ % doas service stunnel start
+Starting stunnel.
+
+# Restart stunnel to apply the CARP VIP binding
+paul@f0:~ % doas service stunnel restart
+Stopping stunnel.
+Starting stunnel.
+```
+
+The configuration includes:
+* `verify = 2`: Verify client certificate and fail if not provided
+* `requireCert = yes`: Client must present a valid certificate
+* `CAfile`: Path to the CA certificate that signed the client certificates
+
+### Setting up NFS on f1 (Standby)
+
+Repeat the same configuration on f1:
+
+```sh
+paul@f1:~ % doas sysrc nfs_server_enable=YES
+nfs_server_enable: NO -> YES
+paul@f1:~ % doas sysrc nfsv4_server_enable=YES
+nfsv4_server_enable: NO -> YES
+paul@f1:~ % doas sysrc nfsuserd_enable=YES
+nfsuserd_enable: NO -> YES
+paul@f1:~ % doas sysrc mountd_enable=YES
+mountd_enable: NO -> YES
+paul@f1:~ % doas sysrc rpcbind_enable=YES
+rpcbind_enable: NO -> YES
+
+paul@f1:~ % doas tee /etc/exports <<'EOF'
+V4: /data/nfs -sec=sys
+/data/nfs -alldirs -maproot=root -network 127.0.0.1 -mask 255.255.255.255
+EOF
+
+paul@f1:~ % doas service rpcbind start
+Starting rpcbind.
+paul@f1:~ % doas service mountd start
+Starting mountd.
+paul@f1:~ % doas service nfsd start
+Starting nfsd.
+paul@f1:~ % doas service nfsuserd start
+Starting nfsuserd.
+```
+
+Configure stunnel on f1:
+
+```sh
+# Install stunnel
+paul@f1:~ % doas pkg install -y stunnel
+
+# Copy certificates from f0
+paul@f0:~ % doas tar -cf /tmp/stunnel-certs.tar -C /usr/local/etc/stunnel server-cert.pem server-key.pem ca
+paul@f0:~ % scp /tmp/stunnel-certs.tar f1:/tmp/
+paul@f1:~ % cd /usr/local/etc/stunnel && doas tar -xf /tmp/stunnel-certs.tar
+
+# Configure stunnel server on f1 with client certificate authentication
+paul@f1:~ % doas tee /usr/local/etc/stunnel/stunnel.conf <<'EOF'
+cert = /usr/local/etc/stunnel/server-cert.pem
+key = /usr/local/etc/stunnel/server-key.pem
+
+setuid = stunnel
+setgid = stunnel
+
+[nfs-tls]
+accept = 192.168.1.138:2323
+connect = 127.0.0.1:2049
+CAfile = /usr/local/etc/stunnel/ca/ca-cert.pem
+verify = 2
+requireCert = yes
+EOF
+
+# Enable and start stunnel
+paul@f1:~ % doas sysrc stunnel_enable=YES
+stunnel_enable:  ->