Update content for gemtext

11 files changed, 1971 insertions, 209 deletions

diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi
index 764e3d7d..c7099c30 100644
--- a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi
+++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi
@@ -385,6 +385,13 @@ Of course I am operating multiple Jails on the same host this way with Puppet:
 
 All done in a pretty automated manor. 
 
+Other *BSD related posts are:
+
+=> ./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi 2016-04-09 Jails and ZFS with Puppet on FreeBSD (You are currently reading this)
+=> ./2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi 2022-07-30 Let's Encrypt with OpenBSD and Rex
+=> ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD
+=> ./2024-01-13-one-reason-why-i-love-openbsd.gmi 2024-01-13 One reason why I love OpenBSD
+
 E-Mail your comments to `paul@nospam.buetow.org` :-)
 
 => ../ Back to the main site
diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi.tpl b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi.tpl
new file mode 100644
index 00000000..29cdc38d
--- /dev/null
+++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi.tpl
@@ -0,0 +1,394 @@
+# Jails and ZFS with Puppet on FreeBSD
+
+> Published at 2016-04-09T18:29:47+01:00
+
+```
+            __     __
+           (( \---/ ))
+            )__   __(
+           / ()___() \
+           \  /(_)\  /
+            \ \_|_/ /
+      _______>     <_______
+     //\      |>o<|      /\\
+     \\/___           ___\//
+           |         |
+           |         |
+           |         |
+           |         |
+           `--....---'
+             \     \
+              \     `.     hjw
+               \      `.
+```
+
+Over the last couple of years I wrote quite a few Puppet modules in order to manage my personal server infrastructure. One of them manages FreeBSD Jails and another one ZFS file systems. I thought I would give a brief overview in how it looks and feels.
+
+## ZFS
+
+The ZFS module is a pretty basic one. It does not manage ZFS pools yet as I am not creating them often enough which would justify implementing an automation. But let's see how we can create a ZFS file system (on an already given ZFS pool named ztank):
+
+Puppet snippet:
+
+```
+zfs::create { 'ztank/foo':
+  ensure     => present,
+  filesystem => '/srv/foo',
+
+  require => File['/srv'],
+}
+```
+
+Puppet run:
+
+```
+admin alphacentauri:/opt/git/server/puppet/manifests [1212]% puppet.apply
+Password:
+Info: Loading facts
+Info: Loading facts
+Info: Loading facts
+Info: Loading facts
+Notice: Compiled catalog for alphacentauri.home in environment production in 7.14 seconds
+Info: Applying configuration version '1460189837'
+Info: mount[files]: allowing * access
+Info: mount[restricted]: allowing * access
+Notice: /Stage[main]/Main/Node[alphacentauri]/Zfs::Create[ztank/foo]/Exec[ztank/foo_create]/returns: executed successfully
+Notice: Finished catalog run in 25.41 seconds
+admin alphacentauri:~ [1213]% zfs list | grep foo
+ztank/foo                     96K  1.13T    96K  /srv/foo
+admin alphacentauri:~ [1214]% df | grep foo
+ztank/foo                  1214493520        96 1214493424     0%    /srv/foo
+admin alphacentauri:~ [1215]% 
+```
+
+The destruction of the file system just requires to set "ensure" to "absent" in Puppet:
+
+```
+zfs::create { 'ztank/foo':
+  ensure     => absent,
+  filesystem => '/srv/foo',
+
+  require => File['/srv'],
+}¬
+```
+
+Puppet run:
+
+```
+admin alphacentauri:/opt/git/server/puppet/manifests [1220]% puppet.apply
+Password:
+Info: Loading facts
+Info: Loading facts
+Info: Loading facts
+Info: Loading facts
+Notice: Compiled catalog for alphacentauri.home in environment production in 6.14 seconds
+Info: Applying configuration version '1460190203'
+Info: mount[files]: allowing * access
+Info: mount[restricted]: allowing * access
+Notice: /Stage[main]/Main/Node[alphacentauri]/Zfs::Create[ztank/foo]/Exec[zfs destroy -r ztank/foo]/returns: executed successfully
+Notice: Finished catalog run in 22.72 seconds
+admin alphacentauri:/opt/git/server/puppet/manifests [1221]% zfs list | grep foo
+zsh: done       zfs list | 
+zsh: exit 1     grep foo
+admin alphacentauri:/opt/git/server/puppet/manifests [1222:1]% df | grep foo
+zsh: done       df | 
+zsh: exit 1     grep foo
+```
+
+## Jails
+
+Here is an example in how a FreeBSD Jail can be created. The Jail will have its own public IPv6 address. And it will have its own internal IPv4 address with IPv4 NAT to the internet (this is due to the limitation that the host server only got one public IPv4 address which requires sharing between all the Jails).
+
+Furthermore, Puppet will ensure that the Jail will have its own ZFS file system (internally it is using the ZFS module). Please notice that the NAT requires the packet filter to be setup correctly (not covered in this blog post).
+
+```
+include jail::freebsd
+
+# Cloned interface for Jail IPv4 NAT
+freebsd::rc_config { 'cloned_interfaces':
+  value => 'lo1',
+}
+freebsd::rc_config { 'ipv4_addrs_lo1':
+  value => '192.168.0.1-24/24'
+}
+
+freebsd::ipalias { '2a01:4f8:120:30e8::17':
+  ensure    => up,
+  proto     => 'inet6',
+  preflen   => '64',
+  interface => 're0',
+  aliasnum  => '8',
+}
+
+class { 'jail':
+  ensure              => present,
+  jails_config        => {
+    sync                     => {
+      '_ensure'             => present,
+      '_type'               => 'freebsd',
+      '_mirror'             => 'ftp://ftp.de.freebsd.org',
+      '_remote_path'        => 'FreeBSD/releases/amd64/10.1-RELEASE',
+      '_dists'              => [ 'base.txz', 'doc.txz', ],
+      '_ensure_directories' => [ '/opt', '/opt/enc' ],
+      '_ensure_zfs'         => [ '/sync' ],
+      'host.hostname'       => "'sync.ian.buetow.org'",
+      'ip4.addr'            => '192.168.0.17',
+      'ip6.addr'            => '2a01:4f8:120:30e8::17',
+    },
+  }
+}
+```
+
+This is how the result looks like:
+
+```
+admin sun:/etc [1939]% puppet.apply
+Info: Loading facts
+Info: Loading facts
+Info: Loading facts
+Info: Loading facts
+Notice: Compiled catalog for sun.ian.buetow.org in environment production in 1.80 seconds
+Info: Applying configuration version '1460190986'
+Notice: /Stage[main]/Jail/File[/etc/jail.conf]/ensure: created
+Info: mount[files]: allowing * access
+Info: mount[restricted]: allowing * access
+Info: Computing checksum on file /etc/motd
+Info: /Stage[main]/Motd/File[/etc/motd]: Filebucketed /etc/motd to puppet with sum fced1b6e89f50ef2c40b0d7fba9defe8
+Notice: /Stage[main]/Jail/Jail::Create[sync]/File[/jail/sync]/ensure: created
+Notice: /Stage[main]/Jail/Jail::Create[sync]/Zfs::Create[zroot/jail/sync]/Exec[zroot/jail/sync_create]/returns: executed successfully
+Notice: /Stage[main]/Jail/Jail::Create[sync]/File[/jail/sync/opt]/ensure: created
+Notice: /Stage[main]/Jail/Jail::Create[sync]/File[/jail/sync/opt/enc]/ensure: created
+Notice: /Stage[main]/Jail/Jail::Create[sync]/Jail::Ensure_zfs[/sync]/Zfs::Create[zroot/jail/sync/sync]/Exec[zroot/jail/sync/sync_create]/returns: executed successfully
+Notice: /Stage[main]/Jail/Jail::Create[sync]/Jail::Freebsd::Create[sync]/File[/jail/sync/.jailbootstrap]/ensure: created
+Notice: /Stage[main]/Jail/Jail::Create[sync]/Jail::Freebsd::Create[sync]/File[/etc/fstab.jail.sync]/ensure: created
+Notice: /Stage[main]/Jail/Jail::Create[sync]/Jail::Freebsd::Create[sync]/File[/jail/sync/.jailbootstrap/bootstrap.sh]/ensure: created
+Notice: /Stage[main]/Jail/Jail::Create[sync]/Jail::Freebsd::Create[sync]/Exec[sync_bootstrap]/returns: executed successfully
+Notice: Finished catalog run in 49.72 seconds
+admin sun:/etc [1942]% ls -l /jail/sync
+total 154
+-r--r--r--   1 root  wheel  6198 11 Nov  2014 COPYRIGHT
+drwxr-xr-x   2 root  wheel    47 11 Nov  2014 bin
+drwxr-xr-x   7 root  wheel    43 11 Nov  2014 boot
+dr-xr-xr-x   2 root  wheel     2 11 Nov  2014 dev
+drwxr-xr-x  23 root  wheel   101  9 Apr 10:37 etc
+drwxr-xr-x   3 root  wheel    50 11 Nov  2014 lib
+drwxr-xr-x   3 root  wheel     4 11 Nov  2014 libexec
+drwxr-xr-x   2 root  wheel     2 11 Nov  2014 media
+drwxr-xr-x   2 root  wheel     2 11 Nov  2014 mnt
+drwxr-xr-x   3 root  wheel     3  9 Apr 10:36 opt
+dr-xr-xr-x   2 root  wheel     2 11 Nov  2014 proc
+drwxr-xr-x   2 root  wheel   143 11 Nov  2014 rescue
+drwxr-xr-x   2 root  wheel     6 11 Nov  2014 root
+drwxr-xr-x   2 root  wheel   132 11 Nov  2014 sbin
+drwxr-xr-x   2 root  wheel     2  9 Apr 10:36 sync
+lrwxr-xr-x   1 root  wheel    11 11 Nov  2014 sys -> usr/src/sys
+drwxrwxrwt   2 root  wheel     2 11 Nov  2014 tmp
+drwxr-xr-x  14 root  wheel    14 11 Nov  2014 usr
+drwxr-xr-x  24 root  wheel    24 11 Nov  2014 var
+admin sun:/etc [1943]% zfs list | grep sync;df | grep sync
+zroot/jail/sync                 162M   343G   162M  /jail/sync
+zroot/jail/sync/sync            144K   343G   144K  /jail/sync/sync
+/opt/enc                                                 5061624     84248    4572448     2%    /jail/sync/opt/enc
+zroot/jail/sync                                        360214972    166372  360048600     0%    /jail/sync
+zroot/jail/sync/sync                                   360048744       144  360048600     0%    /jail/sync/sync
+admin sun:/etc [1944]% cat /etc/fstab.jail.sync
+# Generated by Puppet for a Jail.
+# Can contain file systems to be mounted curing jail start.
+admin sun:/etc [1945]% cat /etc/jail.conf
+# Generated by Puppet
+
+allow.chflags = true;
+exec.start = '/bin/sh /etc/rc';
+exec.stop = '/bin/sh /etc/rc.shutdown';
+mount.devfs = true;
+mount.fstab = "/etc/fstab.jail.$name";
+path = "/jail/$name";
+
+sync {
+      host.hostname = 'sync.ian.buetow.org';
+      ip4.addr = 192.168.0.17;
+      ip6.addr = 2a01:4f8:120:30e8::17;
+}
+admin sun:/etc [1955]% sudo service jail start sync
+Password:
+Starting jails: sync.
+admin sun:/etc [1956]% jls | grep sync
+   103  192.168.0.17    sync.ian.buetow.org           /jail/sync
+admin sun:/etc [1957]% sudo jexec 103 /bin/csh
+root@sync:/ # ifconfig -a
+re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
+     options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
+     ether 50:46:5d:9f:fd:1e
+     inet6 2a01:4f8:120:30e8::17 prefixlen 64 
+     nd6 options=8021<PERFORMNUD,AUTO_LINKLOCAL,DEFAULTIF>
+     media: Ethernet autoselect (1000baseT <full-duplex>)
+     status: active
+lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
+     options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
+     nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
+     lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
+     options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
+     inet 192.168.0.17 netmask 0xffffffff 
+     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
+```
+
+## Inside-Jail Puppet
+
+To automatically setup the applications running in the Jail I am using Puppet as well. I wrote a few scripts which bootstrap Puppet inside of a newly created Jail. It is doing the following:
+
+* Mounts an encrypted container (containing a secret Puppet manifests [git repository])
+* Activates "pkg-ng", the FreeBSD binary package manager, in the Jail
+* Installs Puppet plus all dependencies in the Jail
+* Updates the Jail via "freebsd-update" to the latest version
+* Restarts the Jail and invokes Puppet.
+* Puppet then also schedules a periodic cron job for the next Puppet runs.
+
+```
+admin sun:~ [1951]% sudo /opt/snonux/local/etc/init.d/enc activate sync
+Starting jails: dns.
+The package management tool is not yet installed on your system.
+Do you want to fetch and install it now? [y/N]: y
+Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/freebsd:10:x86:64/latest, please wait...
+Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
+[sync.ian.buetow.org] Installing pkg-1.7.2...
+[sync.ian.buetow.org] Extracting pkg-1.7.2: 100%
+Updating FreeBSD repository catalogue...
+[sync.ian.buetow.org] Fetching meta.txz: 100%    944 B   0.9kB/s    00:01    
+[sync.ian.buetow.org] Fetching packagesite.txz: 100%    5 MiB   5.6MB/s    00:01   
+Processing entries: 100%
+FreeBSD repository update completed. 25091 packages processed.
+Updating database digests format: 100%
+The following 20 package(s) will be affected (of 0 checked):
+
+  New packages to be INSTALLED:
+          git: 2.7.4_1
+          expat: 2.1.0_3
+          python27: 2.7.11_1
+          libffi: 3.2.1
+          indexinfo: 0.2.4
+          gettext-runtime: 0.19.7
+          p5-Error: 0.17024
+          perl5: 5.20.3_9
+          cvsps: 2.1_1
+          p5-Authen-SASL: 2.16_1
+          p5-Digest-HMAC: 1.03_1
+          p5-GSSAPI: 0.28_1
+          curl: 7.48.0_1
+          ca_root_nss: 3.22.2
+          p5-Net-SMTP-SSL: 1.03
+          p5-IO-Socket-SSL: 2.024
+          p5-Net-SSLeay: 1.72
+          p5-IO-Socket-IP: 0.37
+          p5-Socket: 2.021
+          p5-Mozilla-CA: 20160104
+
+          The process will require 144 MiB more space.
+          30 MiB to be downloaded.
+[sync.ian.buetow.org] Fetching git-2.7.4_1.txz: 100%    4 MiB   3.7MB/s    00:01    
+[sync.ian.buetow.org] Fetching expat-2.1.0_3.txz: 100%   98 KiB 100.2kB/s    00:01    
+[sync.ian.buetow.org] Fetching python27-2.7.11_1.txz: 100%   10 MiB  10.7MB/s    00:01    
+[sync.ian.buetow.org] Fetching libffi-3.2.1.txz: 100%   35 KiB  36.2kB/s    00:01    
+[sync.ian.buetow.org] Fetching indexinfo-0.2.4.txz: 100%    5 KiB   5.0kB/s    00:01    
+[sync.ian.buetow.org] Fetching gettext-runtime-0.19.7.txz: 100%  148 KiB 151.1kB/s    00:01    
+[sync.ian.buetow.org] Fetching p5-Error-0.17024.txz: 100%   24 KiB  24.8kB/s    00:01    
+[sync.ian.buetow.org] Fetching perl5-5.20.3_9.txz: 100%   13 MiB   6.9MB/s    00:02    
+[sync.ian.buetow.org] Fetching cvsps-2.1_1.txz: 100%   41 KiB  42.1kB/s    00:01    
+[sync.ian.buetow.org] Fetching p5-Authen-SASL-2.16_1.txz: 100%   44 KiB  45.1kB/s    00:01    
+[sync.ian.buetow.org] Fetching p5-Digest-HMAC-1.03_1.txz: 100%    9 KiB   9.5kB/s    00:01    
+[sync.ian.buetow.org] Fetching p5-GSSAPI-0.28_1.txz: 100%   41 KiB  41.7kB/s    00:01    
+[sync.ian.buetow.org] Fetching curl-7.48.0_1.txz: 100%    2 MiB   2.2MB/s    00:01    
+[sync.ian.buetow.org] Fetching ca_root_nss-3.22.2.txz: 100%  324 KiB 331.4kB/s    00:01    
+[sync.ian.buetow.org] Fetching p5-Net-SMTP-SSL-1.03.txz: 100%   11 KiB  10.8kB/s    00:01    
+[sync.ian.buetow.org] Fetching p5-IO-Socket-SSL-2.024.txz: 100%  153 KiB 156.4kB/s    00:01    
+[sync.ian.buetow.org] Fetching p5-Net-SSLeay-1.72.txz: 100%  234 KiB 239.3kB/s    00:01    
+[sync.ian.buetow.org] Fetching p5-IO-Socket-IP-0.37.txz: 100%   27 KiB  27.4kB/s    00:01    
+[sync.ian.buetow.org] Fetching p5-Socket-2.021.txz: 100%   37 KiB  38.0kB/s    00:01    
+[sync.ian.buetow.org] Fetching p5-Mozilla-CA-20160104.txz: 100%  147 KiB 150.8kB/s    00:01    
+Checking integrity...
+[sync.ian.buetow.org] [1/12] Installing libyaml-0.1.6_2...
+[sync.ian.buetow.org] [1/12] Extracting libyaml-0.1.6_2: 100%
+[sync.ian.buetow.org] [2/12] Installing libedit-3.1.20150325_2...
+[sync.ian.buetow.org] [2/12] Extracting libedit-3.1.20150325_2: 100%
+[sync.ian.buetow.org] [3/12] Installing ruby-2.2.4,1...
+[sync.ian.buetow.org] [3/12] Extracting ruby-2.2.4,1: 100%
+[sync.ian.buetow.org] [4/12] Installing ruby22-gems-2.6.2...
+[sync.ian.buetow.org] [4/12] Extracting ruby22-gems-2.6.2: 100%
+[sync.ian.buetow.org] [5/12] Installing libxml2-2.9.3...
+[sync.ian.buetow.org] [5/12] Extracting libxml2-2.9.3: 100%
+[sync.ian.buetow.org] [6/12] Installing dmidecode-3.0...
+[sync.ian.buetow.org] [6/12] Extracting dmidecode-3.0: 100%
+[sync.ian.buetow.org] [7/12] Installing rubygem-json_pure-1.8.3...
+[sync.ian.buetow.org] [7/12] Extracting rubygem-json_pure-1.8.3: 100%
+[sync.ian.buetow.org] [8/12] Installing augeas-1.4.0...
+[sync.ian.buetow.org] [8/12] Extracting augeas-1.4.0: 100%
+[sync.ian.buetow.org] [9/12] Installing rubygem-facter-2.4.4...
+[sync.ian.buetow.org] [9/12] Extracting rubygem-facter-2.4.4: 100%
+[sync.ian.buetow.org] [10/12] Installing rubygem-hiera1-1.3.4_1...
+[sync.ian.buetow.org] [10/12] Extracting rubygem-hiera1-1.3.4_1: 100%
+[sync.ian.buetow.org] [11/12] Installing rubygem-ruby-augeas-0.5.0_2...
+[sync.ian.buetow.org] [11/12] Extracting rubygem-ruby-augeas-0.5.0_2: 100%
+[sync.ian.buetow.org] [12/12] Installing puppet38-3.8.4_1...
+===> Creating users and/or groups.
+Creating group 'puppet' with gid '814'.
+Creating user 'puppet' with uid '814'.
+[sync.ian.buetow.org] [12/12] Extracting puppet38-3.8.4_1: 100%
+.
+.
+.
+.
+.
+Looking up update.FreeBSD.org mirrors... 4 mirrors found.
+Fetching public key from update4.freebsd.org... done.
+Fetching metadata signature for 10.1-RELEASE from update4.freebsd.org... done.
+Fetching metadata index... done.
+Fetching 2 metadata files... done.
+Inspecting system... done.
+Preparing to download files... done.
+Fetching 874 patches.....10....20....30....
+.
+.
+.
+Applying patches... done.
+Fetching 1594 files... 
+Installing updates...
+done.
+Info: Loading facts
+Info: Loading facts
+Info: Loading facts
+Info: Loading facts
+Could not retrieve fact='pkgng_version', resolution='<anonymous>': undefined method `pkgng_enabled' for Facter:Module
+Warning: Config file /usr/local/etc/puppet/hiera.yaml not found, using Hiera defaults
+Notice: Compiled catalog for sync.ian.buetow.org in environment production in 1.31 seconds
+Warning: Found multiple default providers for package: pkgng, gem, pip; using pkgng
+Info: Applying configuration version '1460192563'
+Notice: /Stage[main]/S_base_freebsd/User[root]/shell: shell changed '/bin/csh' to '/bin/tcsh'
+Notice: /Stage[main]/S_user::Root_files/S_user::All_files[root_user]/File[/root/user]/ensure: created
+Notice: /Stage[main]/S_user::Root_files/S_user::My_files[root]/File[/root/userfiles]/ensure: created
+Notice: /Stage[main]/S_user::Root_files/S_user::My_files[root]/File[/root/.task]/ensure: created
+.
+.
+.
+.
+Notice: Finished catalog run in 206.09 seconds
+```
+
+## Managing multiple Jails
+
+Of course I am operating multiple Jails on the same host this way with Puppet:
+
+* A Jail for the MTA
+* A Jail for the Webserver
+* A Jail for BIND DNS server
+* A Jail for syncing data forth and back between various servers
+* A Jail for other personal (experimental) use
+* ...etc
+
+All done in a pretty automated manor. 
+
+Other *BSD related posts are:
+
+<< template::inline::index bsd
+
+E-Mail your comments to `paul@nospam.buetow.org` :-)
+
+=> ../ Back to the main site
diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi
index 41af24e3..1c9ea500 100644
--- a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi
+++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi
@@ -656,6 +656,13 @@ OpenBSD suits perfectly here as all the tools are already part of the base insta
 
 Why re-inventing the wheel? I love that a `Rexfile` is just a Perl DSL. Also, OpenBSD comes with Perl in the base system. So no new programming language had to be added to my mix for the configuration management system. Also, the `acme.sh` shell script is not a Bash but a standard Bourne shell script, so I didn't have to install an additional shell as OpenBSD does not come with the Bash pre-installed.
 
+Other *BSD related posts are:
+
+=> ./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi 2016-04-09 Jails and ZFS with Puppet on FreeBSD
+=> ./2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi 2022-07-30 Let's Encrypt with OpenBSD and Rex (You are currently reading this)
+=> ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD
+=> ./2024-01-13-one-reason-why-i-love-openbsd.gmi 2024-01-13 One reason why I love OpenBSD
+
 E-Mail your comments to `paul@nospam.buetow.org` :-)
 
 => ../ Back to the main site
diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi.tpl b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi.tpl
new file mode 100644
index 00000000..1a83ef67
--- /dev/null
+++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi.tpl
@@ -0,0 +1,665 @@
+# Let's Encrypt with OpenBSD and Rex
+
+> Published at 2022-07-30T12:14:31+01:00
+
+```
+                                               /    _    \
+  The Hebern Machine                            \ ." ". /
+                                  ___            /     \
+                              ..""   ""..       |   O   |
+                             /           \      |       |
+                            /             \     |       |
+                          ---------------------------------
+                        _/  o     (O)     o   _            |
+                      _/                    ." ".          |
+                    I/    _________________/     \         |
+                  _/I   ."                        |        |
+          =====  /  I  /                         /         |
+     =====  | | |   \ |       _________________."          |
+=====  | |  | | |   /  \     /  _|_|__|_|_          __     |
+  | |  | |  | | |   \   "._."  /  o    o  \       ."  ".   |
+  |  --|  --|  -|   /          \         _/      /      \  |
+   \____\____\__|   \  ______   |       /       |        | |
+               --------      ---       /        |        | |
+              ( )        (O)          /          \      /  |
+               -----------------------            ".__."   |
+               _|__________________________________________|_
+              /                                              \
+             /________________________________________________\
+                                 ASCII Art by John Savard
+```
+
+I was amazed at how easy it is to automatically generate and update Let's Encrypt certificates with OpenBSD.
+
+## What's Let's Encrypt?
+
+> Let's Encrypt is a non-profit certificate authority run by Internet Security Research Group that provides X.509 certificates for Transport Layer Security (TLS) encryption at no charge. It is the world's largest certificate authority, used by more than 265 million websites, with the goal of all websites being secure and using HTTPS.
+
+=> https://en.wikipedia.org/wiki/Let's_Encrypt Source: Wikipedia
+
+In short, it gives away TLS certificates for your website - for free! The catch is, that the certificates are only valid for three months. So it is better to automate certificate generation and renewals.
+
+## Meet `acme-client`
+
+`acme-client` is the default Automatic Certifcate Management Environment (ACME) client on OpenBSD and part of the OpenBSD base system. 
+
+When invoked, the client first checks whether certificates actually require to be generated.
+
+* It first checks whether a certificate already exists; if not, it will attempt to generate a new one.
+* If the certificate already exists but expires within the next 30 days, it will renew it.
+* Otherwise, `acme-client` won't do anything.
+
+Oversimplified, the following steps are undertaken by `acme-client` for generating a new certificate:
+
+* Reading its config file `/etc/acme-client.conf` for a list of hosts (and their alternative names) to generate certificates. So it means you can also have certificates for arbitrary subdomains!
+* Automatic generation of the private certificate part (the certificate key) and the certificate signing request (CSR) to `/etc/ssl/...`.
+* Requesting Let's Encrypt to sign the certificate. This also includes providing a set of temporary files requested by Let's Encrypt in the next step for verification.
+* Let's Encrypt then will contact the hostname for the certificate through a particular URL (e.g. `http://foo.zone/.well-known/acme-challenge/...`) to verify that the requester is the valid owner of the host.
+* Let's Encrypt generates a certificate, which then is downloaded to `/etc/ssl/...`.
+
+## Configuration
+
+There is some (but easy) configuration required to make that all work on OpenBSD.
+
+### acme-client.conf
+
+This is how my `/etc/acme-client.conf` looks like (I copied a template from `/etc/examples/acme-client.conf` to `/etc/acme-client.conf` and added my domains to the bottom:
+
+```
+#
+# $OpenBSD: acme-client.conf,v 1.4 2020/09/17 09:13:06 florian Exp $
+#
+authority letsencrypt {
+    api url "https://acme-v02.api.letsencrypt.org/directory"
+    account key "/etc/acme/letsencrypt-privkey.pem"
+}
+
+authority letsencrypt-staging {
+    api url "https://acme-staging-v02.api.letsencrypt.org/directory"
+    account key "/etc/acme/letsencrypt-staging-privkey.pem"
+}
+
+authority buypass {
+    api url "https://api.buypass.com/acme/directory"
+    account key "/etc/acme/buypass-privkey.pem"
+    contact "mailto:me@example.com"
+}
+
+authority buypass-test {
+    api url "https://api.test4.buypass.no/acme/directory"
+    account key "/etc/acme/buypass-test-privkey.pem"
+    contact "mailto:me@example.com"
+}
+
+domain buetow.org {
+    alternative names { www.buetow.org paul.buetow.org }
+    domain key "/etc/ssl/private/buetow.org.key"
+    domain full chain certificate "/etc/ssl/buetow.org.fullchain.pem"
+    sign with letsencrypt
+}
+
+domain dtail.dev {
+    alternative names { www.dtail.dev }
+    domain key "/etc/ssl/private/dtail.dev.key"
+    domain full chain certificate "/etc/ssl/dtail.dev.fullchain.pem"
+    sign with letsencrypt
+}
+
+domain foo.zone {
+    alternative names { www.foo.zone }
+    domain key "/etc/ssl/private/foo.zone.key"
+    domain full chain certificate "/etc/ssl/foo.zone.fullchain.pem"
+    sign with letsencrypt
+}
+
+domain irregular.ninja {
+    alternative names { www.irregular.ninja }
+    domain key "/etc/ssl/private/irregular.ninja.key"
+    domain full chain certificate "/etc/ssl/irregular.ninja.fullchain.pem"
+    sign with letsencrypt
+}
+
+domain snonux.land {
+    alternative names { www.snonux.land }
+    domain key "/etc/ssl/private/snonux.land.key"
+    domain full chain certificate "/etc/ssl/snonux.land.fullchain.pem"
+    sign with letsencrypt
+}
+```
+
+### httpd.conf
+
+For ACME to work, you will need to configure the HTTP daemon so that the "special" ACME requests from Let's Encrypt are served correctly. I am using the standard OpenBSD `httpd` here. These are the snippets I use for the `foo.zone` host in `/etc/httpd.conf` (of course, you need a similar setup for all other hosts as well):
+
+```
+server "foo.zone" {
+  listen on * port 80
+  location "/.well-known/acme-challenge/*" {
+    root "/acme"
+    request strip 2
+  }
+  location * {
+    block return 302 "https://$HTTP_HOST$REQUEST_URI"
+  }
+}
+
+server "foo.zone" {
+  listen on * tls port 443
+  tls {
+    certificate "/etc/ssl/foo.zone.fullchain.pem"
+    key "/etc/ssl/private/foo.zone.key"
+  }
+  location * {
+    root "/htdocs/gemtexter/foo.zone"
+    directory auto index
+  }
+}
+```
+
+As you see, plain HTTP only serves the ACME challenge path. Otherwise, it redirects the requests to TLS. The TLS section then attempts to use the Let's Encrypt certificates.
+
+It is worth noticing that `httpd` will start without the certificates being present. This will cause a certificate error when you try to reach the HTTPS endpoint, but it helps to bootstrap Let's Encrypt. As you saw in the config snippet above, Let's Encrypt only requests the plain HTTP endpoint for the verification process, so HTTPS doesn't need to be operational yet at this stage. But once the certificates are generated, you will have to reload or restart `httpd` to use any new certificate.
+
+### CRON job
+
+You could now run `doas acme-client foo.zone` to generate the certificate or to renew it. Or you could automate it with CRON.
+
+I have created a script `/usr/local/bin/acme.sh` for that for all of my domains:
+
+```
+#!/bin/sh
+
+function handle_cert {
+    host=$1
+    # Create symlink, so that relayd also can read it.
+    crt_path=/etc/ssl/$host
+    if [ -e $crt_path.crt ]; then
+        rm $crt_path.crt
+    fi
+    ln -s $crt_path.fullchain.pem $crt_path.crt
+    # Requesting and renewing certificate.
+    /usr/sbin/acme-client -v $host
+}
+
+has_update=no
+handle_cert www.buetow.org
+if [ $? -eq 0 ]; then
+    has_update=yes
+fi
+handle_cert www.paul.buetow.org
+if [ $? -eq 0 ]; then
+    has_update=yes
+fi
+handle_cert www.tmp.buetow.org
+if [ $? -eq 0 ]; then
+    has_update=yes
+fi
+handle_cert www.dtail.dev
+if [ $? -eq 0 ]; then
+    has_update=yes
+fi
+handle_cert www.foo.zone
+if [ $? -eq 0 ]; then
+    has_update=yes
+fi
+handle_cert www.irregular.ninja
+if [ $? -eq 0 ]; then
+    has_update=yes
+fi
+handle_cert www.snonux.land
+if [ $? -eq 0 ]; then
+    has_update=yes
+fi
+
+# Pick up the new certs.
+if [ $has_update = yes ]; then
+    /usr/sbin/rcctl reload httpd
+    /usr/sbin/rcctl reload relayd
+    /usr/sbin/rcctl restart smtpd
+fi
+```
+
+And added the following line to `/etc/daily.local` to run the script once daily so that certificates will be renewed fully automatically:
+
+```
+/usr/local/bin/acme.sh
+```
+
+I am receiving a daily output via E-Mail like this now:
+
+```
+Running daily.local:
+acme-client: /etc/ssl/buetow.org.fullchain.pem: certificate valid: 80 days left
+acme-client: /etc/ssl/paul.buetow.org.fullchain.pem: certificate valid: 80 days left
+acme-client: /etc/ssl/tmp.buetow.org.fullchain.pem: certificate valid: 80 days left
+acme-client: /etc/ssl/dtail.dev.fullchain.pem: certificate valid: 80 days left
+acme-client: /etc/ssl/foo.zone.fullchain.pem: certificate valid: 80 days left
+acme-client: /etc/ssl/irregular.ninja.fullchain.pem: certificate valid: 80 days left
+acme-client: /etc/ssl/snonux.land.fullchain.pem: certificate valid: 79 days left
+```
+
+## relayd.conf and smtpd.conf
+
+Besides `httpd`, `relayd` (mainly for Gemini) and `smtpd` (for mail, of course) also use TLS certificates. And as you can see in `acme.sh`, the services are reloaded or restarted (`smtpd` doesn't support reload) whenever a certificate is generated or updated.
+
+## Rexification
+
+I didn't write all these configuration files by hand. As a matter of fact, everything is automated with the Rex configuration management system.
+
+=> https://www.rexify.org
+
+At the top of the `Rexfile` I define all my hosts:
+
+```
+our @acme_hosts = qw/buetow.org paul.buetow.org tmp.buetow.org dtail.dev foo.zone irregular.ninja snonux.land/;
+```
+
+### General ACME client configuration
+
+ACME will be installed into the frontend group of hosts. Here, blowfish is the primary, and twofish is the secondary OpenBSD box.
+
+```
+group frontends => 'blowfish.buetow.org', 'twofish.buetow.org';
+```
+
+This is my Rex task for the general ACME configuration:
+
+```
+desc 'Configure ACME client';
+task 'acme', group => 'frontends',
+  sub {
+    file '/etc/acme-client.conf',
+      content => template('./etc/acme-client.conf.tpl',
+        acme_hosts => \@acme_hosts,
+        is_primary => $is_primary),
+      owner => 'root',
+      group => 'wheel',
+      mode => '644';
+
+    file '/usr/local/bin/acme.sh',
+      content => template('./scripts/acme.sh.tpl',
+        acme_hosts => \@acme_hosts,
+        is_primary => $is_primary),
+      owner => 'root',
+      group => 'wheel',
+      mode => '744';
+
+    file '/etc/daily.local',
+      ensure => 'present',
+      owner => 'root',
+      group => 'wheel',
+      mode => '644';
+
+    append_if_no_such_line '/etc/daily.local', '/usr/local/bin/acme.sh';
+  };
+```
+
+And there is also a Rex task just to run the ACME script remotely:
+
+```
+desc 'Invoke ACME client';
+task 'acme_invoke', group => 'frontends',
+  sub {
+    say run '/usr/local/bin/acme.sh';
+  };
+
+```
+
+Furthermore, this snippet (also at the top of the Rexfile) helps to determine whether the current server is the primary server (all hosts will be without the `www.` prefix) or the secondary server (all hosts will be with the `www.` prefix):
+
+```
+# Bootstrapping the FQDN based on the server IP as the hostname and domain
+# facts aren't set yet due to the myname file in the first place.
+our $fqdns = sub {
+  my $ipv4 = shift;
+  return 'blowfish.buetow.org' if $ipv4 eq '23.88.35.144';
+  return 'twofish.buetow.org' if $ipv4 eq '108.160.134.135';
+  Rex::Logger::info("Unable to determine hostname for $ipv4", 'error');
+  return 'HOSTNAME-UNKNOWN.buetow.org';
+};
+
+# To determine whether the server is the primary or the secondary.
+our $is_primary = sub {
+  my $ipv4 = shift;
+  $fqdns->($ipv4) eq 'blowfish.buetow.org';
+};
+```
+
+The following is the `acme-client.conf.tpl` Rex template file used for the automation. You see that the `www.` prefix isn't sent for the primary server. E.g. `foo.zone` will be served by the primary server (in my case, a server located in Germany) and `www.foo.zone` by the secondary server (in my case, a server located in Japan):
+
+```
+#
+# $OpenBSD: acme-client.conf,v 1.4 2020/09/17 09:13:06 florian Exp $
+#
+authority letsencrypt {
+	api url "https://acme-v02.api.letsencrypt.org/directory"
+	account key "/etc/acme/letsencrypt-privkey.pem"
+}
+
+authority letsencrypt-staging {
+	api url "https://acme-staging-v02.api.letsencrypt.org/directory"
+	account key "/etc/acme/letsencrypt-staging-privkey.pem"
+}
+
+authority buypass {
+	api url "https://api.buypass.com/acme/directory"
+	account key "/etc/acme/buypass-privkey.pem"
+	contact "mailto:me@example.com"
+}
+
+authority buypass-test {
+	api url "https://api.test4.buypass.no/acme/directory"
+	account key "/etc/acme/buypass-test-privkey.pem"
+	contact "mailto:me@example.com"
+}
+
+<%
+  our $primary = $is_primary->($vio0_ip);
+  our $prefix = $primary ? '' : 'www.';
+%>
+
+<% for my $host (@$acme_hosts) { %>
+domain <%= $prefix.$host %> {
+	domain key "/etc/ssl/private/<%= $prefix.$host %>.key"
+	domain full chain certificate "/etc/ssl/<%= $prefix.$host %>.fullchain.pem"
+	sign with letsencrypt
+}
+<% } %>
+
+```
+
+And this is the `acme.sh.tpl`:
+
+```
+#!/bin/sh
+
+<%
+  our $primary = $is_primary->($vio0_ip);
+  our $prefix = $primary ? '' : 'www.';
+-%>
+
+function handle_cert {
+    host=$1
+    # Create symlink, so that relayd also can read it.
+    crt_path=/etc/ssl/$host
+    if [ -e $crt_path.crt ]; then
+        rm $crt_path.crt
+    fi
+    ln -s $crt_path.fullchain.pem $crt_path.crt
+    # Requesting and renewing certificate.
+    /usr/sbin/acme-client -v $host
+}
+
+has_update=no
+<% for my $host (@$acme_hosts) { -%>
+handle_cert <%= $prefix.$host %>
+if [ $? -eq 0 ]; then
+    has_update=yes
+fi
+<% } -%>
+
+# Pick up the new certs.
+if [ $has_update = yes ]; then
+    /usr/sbin/rcctl reload httpd
+    /usr/sbin/rcctl reload relayd
+    /usr/sbin/rcctl restart smtpd
+fi
+```
+
+### Service rexification 
+
+These are the Rex tasks setting up `httpd`, `relayd` and `smtpd` services:
+
+```
+desc 'Setup httpd';
+task 'httpd', group => 'frontends',
+  sub {
+    append_if_no_such_line '/etc/rc.conf.local', 'httpd_flags=';
+
+    file '/etc/