diff options
Diffstat (limited to 'gemfeed/atom.xml')
| -rw-r--r-- | gemfeed/atom.xml | 1998 |
1 files changed, 1819 insertions, 179 deletions
diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index e7e59869..3bced798 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,12 +1,1827 @@ <?xml version="1.0" encoding="utf-8"?> <feed xmlns="http://www.w3.org/2005/Atom"> - <updated>2025-07-12T22:45:27+03:00</updated> + <updated>2025-07-13T16:44:30+03:00</updated> <title>foo.zone feed</title> <subtitle>To be in the .zone!</subtitle> <link href="gemini://foo.zone/gemfeed/atom.xml" rel="self" /> <link href="gemini://foo.zone/" /> <id>gemini://foo.zone/</id> <entry> + <title>f3s: Kubernetes with FreeBSD - Part 6: Storage</title> + <link href="gemini://foo.zone/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi" /> + <id>gemini://foo.zone/gemfeed/2025-07-14-f3s-kubernetes-with-freebsd-part-6.gmi</id> + <updated>2025-07-13T16:44:29+03:00</updated> + <author> + <name>Paul Buetow aka snonux</name> + <email>paul@dev.buetow.org</email> + </author> + <summary>This is the sixth blog post about the f3s series for self-hosting demands in a home lab. f3s? The 'f' stands for FreeBSD, and the '3s' stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.</summary> + <content type="xhtml"> + <div xmlns="http://www.w3.org/1999/xhtml"> + <h1 style='display: inline' id='f3s-kubernetes-with-freebsd---part-6-storage'>f3s: Kubernetes with FreeBSD - Part 6: Storage</h1><br /> +<br /> +<span>This is the sixth blog post about the f3s series for self-hosting demands in a home lab. f3s? The "f" stands for FreeBSD, and the "3s" stands for k3s, the Kubernetes distribution used on FreeBSD-based physical machines.</span><br /> +<br /> +<a class='textlink' href='./2024-11-17-f3s-kubernetes-with-freebsd-part-1.html'>2024-11-17 f3s: Kubernetes with FreeBSD - Part 1: Setting the stage</a><br /> +<a class='textlink' href='./2024-12-03-f3s-kubernetes-with-freebsd-part-2.html'>2024-12-03 f3s: Kubernetes with FreeBSD - Part 2: Hardware and base installation</a><br /> +<a class='textlink' href='./2025-02-01-f3s-kubernetes-with-freebsd-part-3.html'>2025-02-01 f3s: Kubernetes with FreeBSD - Part 3: Protecting from power cuts</a><br /> +<a class='textlink' href='./2025-04-05-f3s-kubernetes-with-freebsd-part-4.html'>2025-04-05 f3s: Kubernetes with FreeBSD - Part 4: Rocky Linux Bhyve VMs</a><br /> +<a class='textlink' href='./2025-05-11-f3s-kubernetes-with-freebsd-part-5.html'>2025-05-11 f3s: Kubernetes with FreeBSD - Part 5: WireGuard mesh network</a><br /> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png'><img alt='f3s logo' title='f3s logo' src='./f3s-kubernetes-with-freebsd-part-1/f3slogo.png' /></a><br /> +<br /> +<h2 style='display: inline' id='table-of-contents'>Table of Contents</h2><br /> +<br /> +<ul> +<li><a href='#f3s-kubernetes-with-freebsd---part-6-storage'>f3s: Kubernetes with FreeBSD - Part 6: Storage</a></li> +<li>⇢ <a href='#introduction'>Introduction</a></li> +<li>⇢ <a href='#additional-storage-capacity'>Additional storage capacity</a></li> +<li>⇢ <a href='#zfs-encryption-keys'>ZFS encryption keys</a></li> +<li>⇢ ⇢ <a href='#ufs-on-usb-keys'>UFS on USB keys</a></li> +<li>⇢ ⇢ <a href='#generating-encryption-keys'>Generating encryption keys</a></li> +<li>⇢ ⇢ <a href='#configuring-zdata-zfs-pool-encryption'>Configuring <span class='inlinecode'>zdata</span> ZFS pool encryption</a></li> +<li>⇢ ⇢ <a href='#migrating-bhyve-vms-to-an-encrypted-bhyve-zfs-volume'>Migrating Bhyve VMs to an encrypted <span class='inlinecode'>bhyve</span> ZFS volume</a></li> +<li>⇢ <a href='#zfs-replication-with-zrepl'>ZFS Replication with <span class='inlinecode'>zrepl</span></a></li> +<li>⇢ ⇢ <a href='#understanding-replication-requirements'>Understanding Replication Requirements</a></li> +<li>⇢ ⇢ <a href='#installing-zrepl'>Installing <span class='inlinecode'>zrepl</span></a></li> +<li>⇢ ⇢ <a href='#configuring-zrepl-on-f1-sink'>Configuring <span class='inlinecode'>zrepl</span> on <span class='inlinecode'>f1</span> (sink)</a></li> +<li>⇢ ⇢ <a href='#enabling-and-starting-zrepl-services'>Enabling and starting <span class='inlinecode'>zrepl</span> services</a></li> +<li>⇢ ⇢ <a href='#monitoring-replication'>Monitoring replication</a></li> +<li>⇢ ⇢ <a href='#verifying-replication-after-reboot'>Verifying replication after reboot</a></li> +<li>⇢ ⇢ <a href='#understanding-failover-limitations-and-design-decisions'>Understanding Failover Limitations and Design Decisions</a></li> +<li>⇢ ⇢ <a href='#mounting-the-nfs-datasets'>Mounting the NFS datasets</a></li> +<li>⇢ ⇢ <a href='#troubleshooting-files-not-appearing-in-replication'>Troubleshooting: Files not appearing in replication</a></li> +<li>⇢ ⇢ <a href='#configuring-automatic-key-loading-on-boot'>Configuring automatic key loading on boot</a></li> +<li>⇢ <a href='#carp-common-address-redundancy-protocol'>CARP (Common Address Redundancy Protocol)</a></li> +<li>⇢ ⇢ <a href='#how-carp-works'>How CARP Works</a></li> +<li>⇢ ⇢ <a href='#configuring-carp'>Configuring CARP</a></li> +<li>⇢ ⇢ <a href='#carp-state-change-notifications'>CARP State Change Notifications</a></li> +<li>⇢ <a href='#nfs-server-configuration'>NFS Server Configuration</a></li> +<li>⇢ ⇢ <a href='#setting-up-nfs-on-f0-primary'>Setting up NFS on <span class='inlinecode'>f0</span> (Primary)</a></li> +<li>⇢ ⇢ <a href='#configuring-stunnel-for-nfs-encryption-with-carp-failover'>Configuring Stunnel for NFS Encryption with CARP Failover</a></li> +<li>⇢ ⇢ <a href='#creating-a-certificate-authority-for-client-authentication'>Creating a Certificate Authority for Client Authentication</a></li> +<li>⇢ ⇢ <a href='#install-and-configure-stunnel-on-f0'>Install and Configure Stunnel on <span class='inlinecode'>f0</span></a></li> +<li>⇢ ⇢ <a href='#setting-up-nfs-on-f1-standby'>Setting up NFS on <span class='inlinecode'>f1</span> (Standby)</a></li> +<li>⇢ ⇢ <a href='#carp-control-script-for-clean-failover'>CARP Control Script for Clean Failover</a></li> +<li>⇢ ⇢ <a href='#carp-management-script'>CARP Management Script</a></li> +<li>⇢ ⇢ <a href='#automatic-failback-after-reboot'>Automatic Failback After Reboot</a></li> +<li>⇢ <a href='#client-configuration-for-stunnel'>Client Configuration for Stunnel</a></li> +<li>⇢ ⇢ <a href='#configuring-rocky-linux-clients-r0-r1-r2'>Configuring Rocky Linux Clients (<span class='inlinecode'>r0</span>, <span class='inlinecode'>r1</span>, <span class='inlinecode'>r2</span>)</a></li> +<li>⇢ ⇢ <a href='#testing-nfs-mount-with-stunnel'>Testing NFS Mount with Stunnel</a></li> +<li>⇢ ⇢ <a href='#testing-carp-failover-with-mounted-clients-and-stale-file-handles'>Testing CARP Failover with mounted clients and stale file handles:</a></li> +<li>⇢ ⇢ <a href='#complete-failover-test'>Complete Failover Test</a></li> +<li>⇢ <a href='#conclusion'>Conclusion</a></li> +<li>⇢ <a href='#future-storage-explorations'>Future Storage Explorations</a></li> +<li>⇢ ⇢ <a href='#minio-for-s3-compatible-object-storage'>MinIO for S3-Compatible Object Storage</a></li> +<li>⇢ ⇢ <a href='#moosefs-for-distributed-high-availability'>MooseFS for Distributed High Availability</a></li> +</ul><br /> +<h2 style='display: inline' id='introduction'>Introduction</h2><br /> +<br /> +<span>In the previous posts, we set up a FreeBSD-based Kubernetes cluster using k3s. While the base system works well, Kubernetes workloads often require persistent storage for databases, configuration files, and application data. Local storage on each node has significant limitations:</span><br /> +<br /> +<ul> +<li>No data sharing: Pods (once we run Kubernetes) on different nodes can't access the same data</li> +<li>Pod mobility: If a pod moves to another node, it loses access to its data</li> +<li>No redundancy: Hardware failure means data loss</li> +</ul><br /> +<span>This post implements a robust storage solution using:</span><br /> +<br /> +<ul> +<li>CARP: For high availability with automatic IP failover</li> +<li>NFS over stunnel: For secure, encrypted network storage</li> +<li>ZFS: For data integrity, encryption, and efficient snapshots</li> +<li><span class='inlinecode'>zrepl</span>: For continuous ZFS replication between nodes</li> +</ul><br /> +<span>The result is a highly available, encrypted storage system that survives node failures while providing shared storage to all Kubernetes pods.</span><br /> +<br /> +<span>Other than what was mentioned in the first post of this blog series, we aren't using HAST, but <span class='inlinecode'>zrepl</span> for data replication. Read more about it later in this blog post.</span><br /> +<br /> +<h2 style='display: inline' id='additional-storage-capacity'>Additional storage capacity</h2><br /> +<br /> +<span>We add 1 TB of additional storage to each of the nodes (<span class='inlinecode'>f0</span>, <span class='inlinecode'>f1</span>, <span class='inlinecode'>f2</span>) in the form of an SSD drive. The Beelink mini PCs have enough space in the chassis for the extra space.</span><br /> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-6/drives.jpg'><img src='./f3s-kubernetes-with-freebsd-part-6/drives.jpg' /></a><br /> +<br /> +<span>Upgrading the storage was as easy as unscrewing, plugging the drive in, and then screwing it back together again. The procedure was uneventful! We're using two different SSD models (Samsung 870 EVO and Crucial BX500) to avoid simultaneous failures from the same manufacturing batch.</span><br /> +<br /> +<span>We then create the <span class='inlinecode'>zdata</span> ZFS pool on all three nodes:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:~ % doas zpool create -m /data zdata /dev/ada<font color="#000000">1</font> +paul@f0:~ % zpool list +NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT +zdata 928G <font color="#000000">12</font>.1M 928G - - <font color="#000000">0</font>% <font color="#000000">0</font>% <font color="#000000">1</font>.00x ONLINE - +zroot 472G <font color="#000000">29</font>.0G 443G - - <font color="#000000">0</font>% <font color="#000000">6</font>% <font color="#000000">1</font>.00x ONLINE - + +paul@f0:/ % doas camcontrol devlist +<512GB SSD D910R170> at scbus0 target <font color="#000000">0</font> lun <font color="#000000">0</font> (pass0,ada0) +<Samsung SSD <font color="#000000">870</font> EVO 1TB SVT03B6Q> at scbus1 target <font color="#000000">0</font> lun <font color="#000000">0</font> (pass1,ada1) +paul@f0:/ % +</pre> +<br /> +<span>To verify that we have a different SSD on the second node (the third node has the same drive as the first):</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f1:/ % doas camcontrol devlist +<512GB SSD D910R170> at scbus0 target <font color="#000000">0</font> lun <font color="#000000">0</font> (pass0,ada0) +<CT1000BX500SSD1 M6CR072> at scbus1 target <font color="#000000">0</font> lun <font color="#000000">0</font> (pass1,ada1) +</pre> +<br /> +<h2 style='display: inline' id='zfs-encryption-keys'>ZFS encryption keys</h2><br /> +<br /> +<span>ZFS native encryption requires encryption keys to unlock datasets. We need a secure method to store these keys that balances security with operational needs:</span><br /> +<br /> +<ul> +<li>Security: Keys must not be stored on the same disks they encrypt</li> +<li>Availability: Keys must be available at boot for automatic mounting</li> +<li>Portability: Keys should be easily moved between systems for recovery</li> +</ul><br /> +<span>Using USB flash drives as hardware key storage provides a convenient and elegant solution. The encrypted data is unreadable without physical access to the USB key, protecting against disk theft or improper disposal. In production environments, you may use enterprise key management systems; however, for a home lab, USB keys offer good security with minimal complexity.</span><br /> +<br /> +<h3 style='display: inline' id='ufs-on-usb-keys'>UFS on USB keys</h3><br /> +<br /> +<span>We'll format the USB drives with UFS (Unix File System) rather than ZFS for simplicity. There is no need to use ZFS.</span><br /> +<br /> +<span>Let's see the USB keys:</span><br /> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-6/usbkeys1.jpg'><img alt='USB keys' title='USB keys' src='./f3s-kubernetes-with-freebsd-part-6/usbkeys1.jpg' /></a><br /> +<br /> +<span>To verify that the USB key (flash disk) is there:</span><br /> +<br /> +<pre> +paul@f0:/ % doas camcontrol devlist +<512GB SSD D910R170> at scbus0 target 0 lun 0 (pass0,ada0) +<Samsung SSD 870 EVO 1TB SVT03B6Q> at scbus1 target 0 lun 0 (pass1,ada1) +<Generic Flash Disk 8.07> at scbus2 target 0 lun 0 (da0,pass2) +paul@f0:/ % +</pre> +<br /> +<span>Let's create the UFS file system and mount it (done on all three nodes <span class='inlinecode'>f0</span>, <span class='inlinecode'>f1</span> and <span class='inlinecode'>f2</span>):</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:/ % doas newfs /dev/da<font color="#000000">0</font> +/dev/da<font color="#000000">0</font>: <font color="#000000">15000</font>.0MB (<font color="#000000">30720000</font> sectors) block size <font color="#000000">32768</font>, fragment size <font color="#000000">4096</font> + using <font color="#000000">24</font> cylinder groups of <font color="#000000">625</font>.22MB, <font color="#000000">20007</font> blks, <font color="#000000">80128</font> inodes. + with soft updates +super-block backups (<b><u><font color="#000000">for</font></u></b> fsck_ffs -b <i><font color="silver">#) at:</font></i> + <font color="#000000">192</font>, <font color="#000000">1280640</font>, <font color="#000000">2561088</font>, <font color="#000000">3841536</font>, <font color="#000000">5121984</font>, <font color="#000000">6402432</font>, <font color="#000000">7682880</font>, <font color="#000000">8963328</font>, <font color="#000000">10243776</font>, +<font color="#000000">11524224</font>, <font color="#000000">12804672</font>, <font color="#000000">14085120</font>, <font color="#000000">15365568</font>, <font color="#000000">16646016</font>, <font color="#000000">17926464</font>, <font color="#000000">19206912</font>,k <font color="#000000">20487360</font>, +... + +paul@f0:/ % echo <font color="#808080">'/dev/da0 /keys ufs rw 0 2'</font> | doas tee -a /etc/fstab +/dev/da<font color="#000000">0</font> /keys ufs rw <font color="#000000">0</font> <font color="#000000">2</font> +paul@f0:/ % doas mkdir /keys +paul@f0:/ % doas mount /keys +paul@f0:/ % df | grep keys +/dev/da<font color="#000000">0</font> <font color="#000000">14877596</font> <font color="#000000">8</font> <font color="#000000">13687384</font> <font color="#000000">0</font>% /keys +</pre> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-6/usbkeys2.jpg'><img alt='USB keys stuck in' title='USB keys stuck in' src='./f3s-kubernetes-with-freebsd-part-6/usbkeys2.jpg' /></a><br /> +<br /> +<h3 style='display: inline' id='generating-encryption-keys'>Generating encryption keys</h3><br /> +<br /> +<span>The following keys will later be used to encrypt the ZFS file systems. They will be stored on all three nodes, serving as a backup in case one of the keys is lost or corrupted. When we later replicate encrypted ZFS volumes from one node to another, the keys must also be available on the destination node.</span><br /> +<br /> +<pre> +paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:bhyve.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:bhyve.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:bhyve.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f0.lan.buetow.org:zdata.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f1.lan.buetow.org:zdata.key 32 +paul@f0:/keys % doas openssl rand -out /keys/f2.lan.buetow.org:zdata.key 32 +paul@f0:/keys % doas chown root * +paul@f0:/keys % doas chmod 400 * + +paul@f0:/keys % ls -l +total 20 +*r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:bhyve.key +*r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:bhyve.key +*r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:bhyve.key +*r-------- 1 root wheel 32 May 25 13:07 f0.lan.buetow.org:zdata.key +*r-------- 1 root wheel 32 May 25 13:07 f1.lan.buetow.org:zdata.key +*r-------- 1 root wheel 32 May 25 13:07 f2.lan.buetow.org:zdata.key +</pre> +<br /> +<span>After creation, these are copied to the other two nodes, <span class='inlinecode'>f1</span> and <span class='inlinecode'>f2</span>, into the <span class='inlinecode'>/keys</span> partition (I won't provide the commands here; create a tarball, copy it over, and extract it on the destination nodes).</span><br /> +<br /> +<h3 style='display: inline' id='configuring-zdata-zfs-pool-encryption'>Configuring <span class='inlinecode'>zdata</span> ZFS pool encryption</h3><br /> +<br /> +<span>Let's encrypt our <span class='inlinecode'>zdata</span> ZFS pool. We are not encrypting the whole pool, but everything within the <span class='inlinecode'>zdata/enc</span> data set:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o \ + keylocation=file:///keys/`hostname`:zdata.key zdata/enc +paul@f0:/ % zfs list | grep zdata +zdata 836K 899G 96K /data +zdata/enc 200K 899G 200K /data/enc + +paul@f0:/keys % zfs get all zdata/enc | grep -E -i <font color="#808080">'(encryption|key)'</font> +zdata/enc encryption aes-<font color="#000000">256</font>-gcm - +zdata/enc keylocation file:///keys/f<font color="#000000">0</font>.lan.buetow.org:zdata.key <b><u><font color="#000000">local</font></u></b> +zdata/enc keyformat raw - +zdata/enc encryptionroot zdata/enc - +zdata/enc keystatus available - +</pre> +<br /> +<span>All future data sets within <span class='inlinecode'>zdata/enc</span> will inherit the same encryption key.</span><br /> +<br /> +<h3 style='display: inline' id='migrating-bhyve-vms-to-an-encrypted-bhyve-zfs-volume'>Migrating Bhyve VMs to an encrypted <span class='inlinecode'>bhyve</span> ZFS volume</h3><br /> +<br /> +<span>We set up Bhyve VMs in a previous blog post. Their ZFS data sets rely on <span class='inlinecode'>zroot</span>, which is the default ZFS pool on the internal 512GB NVME drive. They aren't encrypted yet, so we encrypt the VM data sets as well now. To do so, we first shut down the VMs on all three nodes:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:/keys % doas vm stop rocky +Sending ACPI shutdown to rocky + +paul@f0:/keys % doas vm list +NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE +rocky default uefi <font color="#000000">4</font> 14G - Yes [<font color="#000000">1</font>] Stopped +</pre> +<br /> +<span>After this, we rename the unencrypted data set to <span class='inlinecode'>_old</span>, create a new encrypted data set, and also snapshot it as <span class='inlinecode'>@hamburger</span>.</span><br /> +<span> </span><br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:/keys % doas zfs rename zroot/bhyve zroot/bhyve_old +paul@f0:/keys % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/mnt zroot/bhyve_old +paul@f0:/keys % doas zfs snapshot zroot/bhyve_old/rocky@hamburger + +paul@f0:/keys % doas zfs create -o encryption=on -o keyformat=raw -o \ + keylocation=file:///keys/`hostname`:bhyve.key zroot/bhyve +paul@f0:/keys % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/zroot/bhyve zroot/bhyve +paul@f0:/keys % doas zfs <b><u><font color="#000000">set</font></u></b> mountpoint=/zroot/bhyve/rocky zroot/bhyve/rocky +</pre> +<br /> +<span>Once done, we import the snapshot into the encrypted dataset and also copy some other metadata files from <span class='inlinecode'>vm-bhyve</span> back over.</span><br /> +<br /> +<pre> +paul@f0:/keys % doas zfs send zroot/bhyve_old/rocky@hamburger | \ + doas zfs recv zroot/bhyve/rocky +paul@f0:/keys % doas cp -Rp /mnt/.config /zroot/bhyve/ +paul@f0:/keys % doas cp -Rp /mnt/.img /zroot/bhyve/ +paul@f0:/keys % doas cp -Rp /mnt/.templates /zroot/bhyve/ +paul@f0:/keys % doas cp -Rp /mnt/.iso /zroot/bhyve/ +</pre> +<br /> +<span>We also have to make encrypted ZFS data sets mount automatically on boot:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:/keys % doas sysrc zfskeys_enable=YES +zfskeys_enable: -> YES +paul@f0:/keys % doas vm init +paul@f0:/keys % doas reboot +. +. +. +paul@f0:~ % doas vm list +paul@f0:~ % doas vm list +NAME DATASTORE LOADER CPU MEMORY VNC AUTO STATE +rocky default uefi <font color="#000000">4</font> 14G <font color="#000000">0.0</font>.<font color="#000000">0.0</font>:<font color="#000000">5900</font> Yes [<font color="#000000">1</font>] Running (<font color="#000000">2265</font>) +</pre> +<br /> +<span>As you can see, the VM is running. This means the encrypted <span class='inlinecode'>zroot/bhyve</span> was mounted successfully after the reboot! Now we can destroy the old, unencrypted, and now unused bhyve dataset:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:~ % doas zfs destroy -R zroot/bhyve_old +</pre> +<br /> +<span>To verify once again that <span class='inlinecode'>zroot/bhyve</span> and <span class='inlinecode'>zroot/bhyve/rocky</span> are now both encrypted, we run:</span><br /> +<span> </span><br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:~ % zfs get all zroot/bhyve | grep -E <font color="#808080">'(encryption|key)'</font> +zroot/bhyve encryption aes-<font color="#000000">256</font>-gcm - +zroot/bhyve keylocation file:///keys/f<font color="#000000">0</font>.lan.buetow.org:bhyve.key <b><u><font color="#000000">local</font></u></b> +zroot/bhyve keyformat raw - +zroot/bhyve encryptionroot zroot/bhyve - +zroot/bhyve keystatus available - + +paul@f0:~ % zfs get all zroot/bhyve/rocky | grep -E <font color="#808080">'(encryption|key)'</font> +zroot/bhyve/rocky encryption aes-<font color="#000000">256</font>-gcm - +zroot/bhyve/rocky keylocation none default +zroot/bhyve/rocky keyformat raw - +zroot/bhyve/rocky encryptionroot zroot/bhyve - +zroot/bhyve/rocky keystatus available - +</pre> +<br /> +<h2 style='display: inline' id='zfs-replication-with-zrepl'>ZFS Replication with <span class='inlinecode'>zrepl</span></h2><br /> +<br /> +<span>Data replication is the cornerstone of high availability. While CARP handles IP failover (see later in this post), we need continuous data replication to ensure the backup server has current data when it becomes active. Without replication, failover would result in data loss or require shared storage (like iSCSI), which introduces a single point of failure.</span><br /> +<br /> +<h3 style='display: inline' id='understanding-replication-requirements'>Understanding Replication Requirements</h3><br /> +<br /> +<span>Our storage system has different replication needs:</span><br /> +<br /> +<ul> +<li>NFS data (<span class='inlinecode'>/data/nfs/k3svolumes</span>): Soon, it will contain active Kubernetes persistent volumes. Needs frequent replication (every minute) to minimise data loss during failover.</li> +<li>VM data (<span class='inlinecode'>/zroot/bhyve/fedora</span>): Contains VM images that change less frequently. Can tolerate longer replication intervals (every 10 minutes).</li> +</ul><br /> +<span>The 1-minute replication window is perfectly acceptable for my personal use cases. This isn't a high-frequency trading system or a real-time database—it's storage for personal projects, development work, and home lab experiments. Losing at most 1 minute of work in a disaster scenario is a reasonable trade-off for the reliability and simplicity of snapshot-based replication. Additionally, in the case of a "1 minute of data loss," I would likely still have the data available on the client side.</span><br /> +<br /> +<span>Why use <span class='inlinecode'>zrepl</span> instead of HAST? While HAST (Highly Available Storage) is FreeBSD's native solution for high-availability storage and supports synchronous replication—thus eliminating the mentioned 1-minute window—I've chosen <span class='inlinecode'>zrepl</span> for several important reasons:</span><br /> +<br /> +<ul> +<li>HAST can cause ZFS corruption: HAST operates at the block level and doesn't understand ZFS's transactional semantics. During failover, in-flight transactions can lead to corrupted zpools. I've experienced this firsthand (I am confident I have configured something wrong) - the automatic failover would trigger while ZFS was still writing, resulting in an unmountable pool.</li> +<li>ZFS-aware replication: <span class='inlinecode'>zrepl</span> understands ZFS datasets and snapshots. It replicates at the dataset level, ensuring each snapshot is a consistent point-in-time copy. This is fundamentally safer than block-level replication.</li> +<li>Snapshot history: With <span class='inlinecode'>zrepl</span>, you get multiple recovery points (every minute for NFS data in our setup). If corruption occurs, you can roll back to any previous snapshot. HAST only gives you the current state.</li> +<li>Easier recovery: When something goes wrong with <span class='inlinecode'>zrepl</span>, you still have intact snapshots on both sides. With HAST, a corrupted primary often means a corrupted secondary as well.</li> +</ul><br /> +<a class='textlink' href='https://wiki.freebsd.org/HighlyAvailableStorage'>FreeBSD HAST</a><br /> +<br /> +<h3 style='display: inline' id='installing-zrepl'>Installing <span class='inlinecode'>zrepl</span></h3><br /> +<br /> +<span>First, install <span class='inlinecode'>zrepl</span> on both hosts involved (we will replicate data from <span class='inlinecode'>f0</span> to <span class='inlinecode'>f1</span>):</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:~ % doas pkg install -y zrepl +</pre> +<br /> +<span>Then, we verify the pools and datasets on both hosts:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="silver"># On f0</font></i> +paul@f0:~ % doas zpool list +NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT +zdata 928G <font color="#000000">1</font>.03M 928G - - <font color="#000000">0</font>% <font color="#000000">0</font>% <font color="#000000">1</font>.00x ONLINE - +zroot 472G <font color="#000000">26</font>.7G 445G - - <font color="#000000">0</font>% <font color="#000000">5</font>% <font color="#000000">1</font>.00x ONLINE - + +paul@f0:~ % doas zfs list -r zdata/enc +NAME USED AVAIL REFER MOUNTPOINT +zdata/enc 200K 899G 200K /data/enc + +<i><font color="silver"># On f1</font></i> +paul@f1:~ % doas zpool list +NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT +zdata 928G 956K 928G - - <font color="#000000">0</font>% <font color="#000000">0</font>% <font color="#000000">1</font>.00x ONLINE - +zroot 472G <font color="#000000">11</font>.7G 460G - - <font color="#000000">0</font>% <font color="#000000">2</font>% <font color="#000000">1</font>.00x ONLINE - + +paul@f1:~ % doas zfs list -r zdata/enc +NAME USED AVAIL REFER MOUNTPOINT +zdata/enc 200K 899G 200K /data/enc +</pre> +<br /> +<span>Since we have a WireGuard tunnel between <span class='inlinecode'>f0</span> and f1, we'll use TCP transport over the secure tunnel instead of SSH. First, check the WireGuard IP addresses:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="silver"># Check WireGuard interface IPs</font></i> +paul@f0:~ % ifconfig wg0 | grep inet + inet <font color="#000000">192.168</font>.<font color="#000000">2.130</font> netmask <font color="#000000">0xffffff00</font> + +paul@f1:~ % ifconfig wg0 | grep inet + inet <font color="#000000">192.168</font>.<font color="#000000">2.131</font> netmask <font color="#000000">0xffffff00</font> +</pre> +<br /> +<span>Let's create a dedicated dataset for NFS data that will be replicated:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="silver"># Create the nfsdata dataset that will hold all data exposed via NFS</font></i> +paul@f0:~ % doas zfs create zdata/enc/nfsdata +</pre> +<br /> +<span>Afterwards, we create the <span class='inlinecode'>zrepl</span> configuration on <span class='inlinecode'>f0</span>:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<<font color="#808080">'EOF'</font> +global: + logging: + - <b><u><font color="#000000">type</font></u></b>: stdout + level: info + format: human + +<b><u><font color="#000000">jobs</font></u></b>: + - name: f0_to_f1_nfsdata + <b><u><font color="#000000">type</font></u></b>: push + connect: + <b><u><font color="#000000">type</font></u></b>: tcp + address: <font color="#808080">"192.168.2.131:8888"</font> + filesystems: + <font color="#808080">"zdata/enc/nfsdata"</font>: <b><u><font color="#000000">true</font></u></b> + send: + encrypted: <b><u><font color="#000000">true</font></u></b> + snapshotting: + <b><u><font color="#000000">type</font></u></b>: periodic + prefix: zrepl_ + interval: 1m + pruning: + keep_sender: + - <b><u><font color="#000000">type</font></u></b>: last_n + count: <font color="#000000">10</font> + keep_receiver: + - <b><u><font color="#000000">type</font></u></b>: last_n + count: <font color="#000000">10</font> + + - name: f0_to_f1_fedora + <b><u><font color="#000000">type</font></u></b>: push + connect: + <b><u><font color="#000000">type</font></u></b>: tcp + address: <font color="#808080">"192.168.2.131:8888"</font> + filesystems: + <font color="#808080">"zroot/bhyve/fedora"</font>: <b><u><font color="#000000">true</font></u></b> + send: + encrypted: <b><u><font color="#000000">true</font></u></b> + snapshotting: + <b><u><font color="#000000">type</font></u></b>: periodic + prefix: zrepl_ + interval: 10m + pruning: + keep_sender: + - <b><u><font color="#000000">type</font></u></b>: last_n + count: <font color="#000000">10</font> + keep_receiver: + - <b><u><font color="#000000">type</font></u></b>: last_n + count: <font color="#000000">10</font> +EOF +</pre> +<br /> +<span> We're using two separate replication jobs with different intervals:</span><br /> +<br /> +<ul> +<li><span class='inlinecode'>f0_to_f1_nfsdata</span>: Replicates NFS data every minute for faster failover recovery</li> +<li><span class='inlinecode'>f0_to_f1_fedora</span>: Replicates Fedora VM every ten minutes (less critical)</li> +</ul><br /> +<span>The Fedora VM is only used for development purposes, so it doesn't require as frequent replication as the NFS data. It's off-topic to this blog series, but it showcases, hows <span class='inlinecode'>zrepl</span>'s flexibility in handling different datasets with varying replication needs.</span><br /> +<br /> +<span>Furthermore:</span><br /> +<br /> +<ul> +<li>We're specifically replicating <span class='inlinecode'>zdata/enc/nfsdata</span> instead of the entire <span class='inlinecode'>zdata/enc</span> dataset. This dedicated dataset will contain all the data we later want to expose via NFS, keeping a clear separation between replicated NFS data and other local encrypted data.</li> +<li>The <span class='inlinecode'>send: encrypted: false</span> option turns off ZFS native encryption for the replication stream. Since we're using a WireGuard tunnel between <span class='inlinecode'>f0</span> and <span class='inlinecode'>f1</span>, the data is already encrypted in transit. Disabling ZFS stream encryption reduces CPU overhead and improves replication performance.</li> +</ul><br /> +<h3 style='display: inline' id='configuring-zrepl-on-f1-sink'>Configuring <span class='inlinecode'>zrepl</span> on <span class='inlinecode'>f1</span> (sink)</h3><br /> +<br /> +<span>On <span class='inlinecode'>f1</span> (the sink, meaning it's the node receiving the replication data), we configure <span class='inlinecode'>zrepl</span> to receive the data as follows:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="silver"># First, create a dedicated sink dataset</font></i> +paul@f1:~ % doas zfs create zdata/sink + +paul@f1:~ % doas tee /usr/local/etc/zrepl/zrepl.yml <<<font color="#808080">'EOF'</font> +global: + logging: + - <b><u><font color="#000000">type</font></u></b>: stdout + level: info + format: human + +<b><u><font color="#000000">jobs</font></u></b>: + - name: sink + <b><u><font color="#000000">type</font></u></b>: sink + serve: + <b><u><font color="#000000">type</font></u></b>: tcp + listen: <font color="#808080">"192.168.2.131:8888"</font> + clients: + <font color="#808080">"192.168.2.130"</font>: <font color="#808080">"f0"</font> + recv: + placeholder: + encryption: inherit + root_fs: <font color="#808080">"zdata/sink"</font> +EOF +</pre> +<br /> +<h3 style='display: inline' id='enabling-and-starting-zrepl-services'>Enabling and starting <span class='inlinecode'>zrepl</span> services</h3><br /> +<br /> +<span>We then enable and start <span class='inlinecode'>zrepl</span> on both hosts via:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="silver"># On f0</font></i> +paul@f0:~ % doas sysrc zrepl_enable=YES +zrepl_enable: -> YES +paul@f0:~ % doas service `zrepl` start +Starting zrepl. + +<i><font color="silver"># On f1</font></i> +paul@f1:~ % doas sysrc zrepl_enable=YES +zrepl_enable: -> YES +paul@f1:~ % doas service `zrepl` start +Starting zrepl. +</pre> +<br /> +<span>To check the replication status, we run:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre><i><font color="silver"># On f0, check `zrepl` status (use raw mode for non-tty)</font></i> +paul@f0:~ % doas pkg install jq +paul@f0:~ % doas zrepl status --mode raw | grep -A<font color="#000000">2</font> <font color="#808080">"Replication"</font> | jq . +<font color="#808080">"Replication"</font>:{<font color="#808080">"StartAt"</font>:<font color="#808080">"2025-07-01T22:31:48.712143123+03:00"</font>... + +<i><font color="silver"># Check if services are running</font></i> +paul@f0:~ % doas service zrepl status +zrepl is running as pid <font color="#000000">2649</font>. + +paul@f1:~ % doas service zrepl status +zrepl is running as pid <font color="#000000">2574</font>. + +<i><font color="silver"># Check for `zrepl` snapshots on source</font></i> +paul@f0:~ % doas zfs list -t snapshot -r zdata/enc | grep zrepl +zdata/enc@zrepl_20250701_193148_000 0B - 176K - + +<i><font color="silver"># On f1, verify the replicated datasets </font></i> +paul@f1:~ % doas zfs list -r zdata | grep f0 +zdata/f<font color="#000000">0</font> 576K 899G 200K none +zdata/f<font color="#000000">0</font>/zdata 376K 899G 200K none +zdata/f<font color="#000000">0</font>/zdata/enc 176K 899G 176K none + +<i><font color="silver"># Check replicated snapshots on f1</font></i> +paul@f1:~ % doas zfs list -t snapshot -r zdata | grep zrepl +zdata/f<font color="#000000">0</font>/zdata/enc@zrepl_20250701_193148_000 0B - 176K - +zdata/f<font color="#000000">0</font>/zdata/enc@zrepl_20250701_194148_000 0B - 176K - +. +. +. +</pre> +<br /> +<h3 style='display: inline' id='monitoring-replication'>Monitoring replication</h3><br /> +<br /> +<span>You can monitor the replication progress with:</span><br /> +<br /> +<!-- Generator: GNU source-highlight 3.1.9 +by Lorenzo Bettini +http://www.lorenzobettini.it +http://www.gnu.org/software/src-highlite --> +<pre>paul@f0:~ % doas zrepl status +</pre> +<br /> +<a href='./f3s-kubernetes-with-freebsd-part-6/zrepl.png'><img alt='zrepl status' title='zrepl status' src='./f3s-kubernetes-with-freebsd-part-6/zrepl.png' /></a><br /> +<br /> +<span>With this setup, both <span class='inlinecode'>zdata/enc/nfsdata</span> and <span class='inlinecode'>zroot/bhyve/fedora</span> on <span class='inlinecode'>f0</span> will be automatically replicated to <span class='inlinecode'>f1</span> every 1 minute (or 10 minutes in the case of the Fedora VM), with encrypted snapshots preserved on both sides. The pruning policy ensures that we keep the last 10 snapshots while managing disk space efficiently.</span><br /> +<br /> |