From 03a2cf8147eb2b06404be42314a3134bf835bad9 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Sat, 13 Jan 2024 23:08:14 +0200 Subject: Update content for gemtext --- ...-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi | 7 + ...09-jails-and-zfs-on-freebsd-with-puppet.gmi.tpl | 394 ++++++++++++ ...022-07-30-lets-encrypt-with-openbsd-and-rex.gmi | 7 + ...07-30-lets-encrypt-with-openbsd-and-rex.gmi.tpl | 665 +++++++++++++++++++++ .../2024-01-13-one-reason-why-i-love-openbsd.gmi | 71 +++ ...024-01-13-one-reason-why-i-love-openbsd.gmi.tpl | 68 +++ gemfeed/atom.xml | 307 ++++------ gemfeed/atom.xml.tmp | 655 ++++++++++++++++++++ gemfeed/index.gmi | 1 + 9 files changed, 1968 insertions(+), 207 deletions(-) create mode 100644 gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi.tpl create mode 100644 gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi.tpl create mode 100644 gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi create mode 100644 gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi.tpl create mode 100644 gemfeed/atom.xml.tmp (limited to 'gemfeed') diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi index 764e3d7d..c7099c30 100644 --- a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi +++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi @@ -385,6 +385,13 @@ Of course I am operating multiple Jails on the same host this way with Puppet: All done in a pretty automated manor. +Other *BSD related posts are: + +=> ./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi 2016-04-09 Jails and ZFS with Puppet on FreeBSD (You are currently reading this) +=> ./2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi 2022-07-30 Let's Encrypt with OpenBSD and Rex +=> ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD +=> ./2024-01-13-one-reason-why-i-love-openbsd.gmi 2024-01-13 One reason why I love OpenBSD + E-Mail your comments to `paul@nospam.buetow.org` :-) => ../ Back to the main site diff --git a/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi.tpl b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi.tpl new file mode 100644 index 00000000..29cdc38d --- /dev/null +++ b/gemfeed/2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi.tpl @@ -0,0 +1,394 @@ +# Jails and ZFS with Puppet on FreeBSD + +> Published at 2016-04-09T18:29:47+01:00 + +``` + __ __ + (( \---/ )) + )__ __( + / ()___() \ + \ /(_)\ / + \ \_|_/ / + _______> <_______ + //\ |>o<| /\\ + \\/___ ___\// + | | + | | + | | + | | + `--....---' + \ \ + \ `. hjw + \ `. +``` + +Over the last couple of years I wrote quite a few Puppet modules in order to manage my personal server infrastructure. One of them manages FreeBSD Jails and another one ZFS file systems. I thought I would give a brief overview in how it looks and feels. + +## ZFS + +The ZFS module is a pretty basic one. It does not manage ZFS pools yet as I am not creating them often enough which would justify implementing an automation. But let's see how we can create a ZFS file system (on an already given ZFS pool named ztank): + +Puppet snippet: + +``` +zfs::create { 'ztank/foo': + ensure => present, + filesystem => '/srv/foo', + + require => File['/srv'], +} +``` + +Puppet run: + +``` +admin alphacentauri:/opt/git/server/puppet/manifests [1212]% puppet.apply +Password: +Info: Loading facts +Info: Loading facts +Info: Loading facts +Info: Loading facts +Notice: Compiled catalog for alphacentauri.home in environment production in 7.14 seconds +Info: Applying configuration version '1460189837' +Info: mount[files]: allowing * access +Info: mount[restricted]: allowing * access +Notice: /Stage[main]/Main/Node[alphacentauri]/Zfs::Create[ztank/foo]/Exec[ztank/foo_create]/returns: executed successfully +Notice: Finished catalog run in 25.41 seconds +admin alphacentauri:~ [1213]% zfs list | grep foo +ztank/foo 96K 1.13T 96K /srv/foo +admin alphacentauri:~ [1214]% df | grep foo +ztank/foo 1214493520 96 1214493424 0% /srv/foo +admin alphacentauri:~ [1215]% +``` + +The destruction of the file system just requires to set "ensure" to "absent" in Puppet: + +``` +zfs::create { 'ztank/foo': + ensure => absent, + filesystem => '/srv/foo', + + require => File['/srv'], +}¬ +``` + +Puppet run: + +``` +admin alphacentauri:/opt/git/server/puppet/manifests [1220]% puppet.apply +Password: +Info: Loading facts +Info: Loading facts +Info: Loading facts +Info: Loading facts +Notice: Compiled catalog for alphacentauri.home in environment production in 6.14 seconds +Info: Applying configuration version '1460190203' +Info: mount[files]: allowing * access +Info: mount[restricted]: allowing * access +Notice: /Stage[main]/Main/Node[alphacentauri]/Zfs::Create[ztank/foo]/Exec[zfs destroy -r ztank/foo]/returns: executed successfully +Notice: Finished catalog run in 22.72 seconds +admin alphacentauri:/opt/git/server/puppet/manifests [1221]% zfs list | grep foo +zsh: done zfs list | +zsh: exit 1 grep foo +admin alphacentauri:/opt/git/server/puppet/manifests [1222:1]% df | grep foo +zsh: done df | +zsh: exit 1 grep foo +``` + +## Jails + +Here is an example in how a FreeBSD Jail can be created. The Jail will have its own public IPv6 address. And it will have its own internal IPv4 address with IPv4 NAT to the internet (this is due to the limitation that the host server only got one public IPv4 address which requires sharing between all the Jails). + +Furthermore, Puppet will ensure that the Jail will have its own ZFS file system (internally it is using the ZFS module). Please notice that the NAT requires the packet filter to be setup correctly (not covered in this blog post). + +``` +include jail::freebsd + +# Cloned interface for Jail IPv4 NAT +freebsd::rc_config { 'cloned_interfaces': + value => 'lo1', +} +freebsd::rc_config { 'ipv4_addrs_lo1': + value => '192.168.0.1-24/24' +} + +freebsd::ipalias { '2a01:4f8:120:30e8::17': + ensure => up, + proto => 'inet6', + preflen => '64', + interface => 're0', + aliasnum => '8', +} + +class { 'jail': + ensure => present, + jails_config => { + sync => { + '_ensure' => present, + '_type' => 'freebsd', + '_mirror' => 'ftp://ftp.de.freebsd.org', + '_remote_path' => 'FreeBSD/releases/amd64/10.1-RELEASE', + '_dists' => [ 'base.txz', 'doc.txz', ], + '_ensure_directories' => [ '/opt', '/opt/enc' ], + '_ensure_zfs' => [ '/sync' ], + 'host.hostname' => "'sync.ian.buetow.org'", + 'ip4.addr' => '192.168.0.17', + 'ip6.addr' => '2a01:4f8:120:30e8::17', + }, + } +} +``` + +This is how the result looks like: + +``` +admin sun:/etc [1939]% puppet.apply +Info: Loading facts +Info: Loading facts +Info: Loading facts +Info: Loading facts +Notice: Compiled catalog for sun.ian.buetow.org in environment production in 1.80 seconds +Info: Applying configuration version '1460190986' +Notice: /Stage[main]/Jail/File[/etc/jail.conf]/ensure: created +Info: mount[files]: allowing * access +Info: mount[restricted]: allowing * access +Info: Computing checksum on file /etc/motd +Info: /Stage[main]/Motd/File[/etc/motd]: Filebucketed /etc/motd to puppet with sum fced1b6e89f50ef2c40b0d7fba9defe8 +Notice: /Stage[main]/Jail/Jail::Create[sync]/File[/jail/sync]/ensure: created +Notice: /Stage[main]/Jail/Jail::Create[sync]/Zfs::Create[zroot/jail/sync]/Exec[zroot/jail/sync_create]/returns: executed successfully +Notice: /Stage[main]/Jail/Jail::Create[sync]/File[/jail/sync/opt]/ensure: created +Notice: /Stage[main]/Jail/Jail::Create[sync]/File[/jail/sync/opt/enc]/ensure: created +Notice: /Stage[main]/Jail/Jail::Create[sync]/Jail::Ensure_zfs[/sync]/Zfs::Create[zroot/jail/sync/sync]/Exec[zroot/jail/sync/sync_create]/returns: executed successfully +Notice: /Stage[main]/Jail/Jail::Create[sync]/Jail::Freebsd::Create[sync]/File[/jail/sync/.jailbootstrap]/ensure: created +Notice: /Stage[main]/Jail/Jail::Create[sync]/Jail::Freebsd::Create[sync]/File[/etc/fstab.jail.sync]/ensure: created +Notice: /Stage[main]/Jail/Jail::Create[sync]/Jail::Freebsd::Create[sync]/File[/jail/sync/.jailbootstrap/bootstrap.sh]/ensure: created +Notice: /Stage[main]/Jail/Jail::Create[sync]/Jail::Freebsd::Create[sync]/Exec[sync_bootstrap]/returns: executed successfully +Notice: Finished catalog run in 49.72 seconds +admin sun:/etc [1942]% ls -l /jail/sync +total 154 +-r--r--r-- 1 root wheel 6198 11 Nov 2014 COPYRIGHT +drwxr-xr-x 2 root wheel 47 11 Nov 2014 bin +drwxr-xr-x 7 root wheel 43 11 Nov 2014 boot +dr-xr-xr-x 2 root wheel 2 11 Nov 2014 dev +drwxr-xr-x 23 root wheel 101 9 Apr 10:37 etc +drwxr-xr-x 3 root wheel 50 11 Nov 2014 lib +drwxr-xr-x 3 root wheel 4 11 Nov 2014 libexec +drwxr-xr-x 2 root wheel 2 11 Nov 2014 media +drwxr-xr-x 2 root wheel 2 11 Nov 2014 mnt +drwxr-xr-x 3 root wheel 3 9 Apr 10:36 opt +dr-xr-xr-x 2 root wheel 2 11 Nov 2014 proc +drwxr-xr-x 2 root wheel 143 11 Nov 2014 rescue +drwxr-xr-x 2 root wheel 6 11 Nov 2014 root +drwxr-xr-x 2 root wheel 132 11 Nov 2014 sbin +drwxr-xr-x 2 root wheel 2 9 Apr 10:36 sync +lrwxr-xr-x 1 root wheel 11 11 Nov 2014 sys -> usr/src/sys +drwxrwxrwt 2 root wheel 2 11 Nov 2014 tmp +drwxr-xr-x 14 root wheel 14 11 Nov 2014 usr +drwxr-xr-x 24 root wheel 24 11 Nov 2014 var +admin sun:/etc [1943]% zfs list | grep sync;df | grep sync +zroot/jail/sync 162M 343G 162M /jail/sync +zroot/jail/sync/sync 144K 343G 144K /jail/sync/sync +/opt/enc 5061624 84248 4572448 2% /jail/sync/opt/enc +zroot/jail/sync 360214972 166372 360048600 0% /jail/sync +zroot/jail/sync/sync 360048744 144 360048600 0% /jail/sync/sync +admin sun:/etc [1944]% cat /etc/fstab.jail.sync +# Generated by Puppet for a Jail. +# Can contain file systems to be mounted curing jail start. +admin sun:/etc [1945]% cat /etc/jail.conf +# Generated by Puppet + +allow.chflags = true; +exec.start = '/bin/sh /etc/rc'; +exec.stop = '/bin/sh /etc/rc.shutdown'; +mount.devfs = true; +mount.fstab = "/etc/fstab.jail.$name"; +path = "/jail/$name"; + +sync { + host.hostname = 'sync.ian.buetow.org'; + ip4.addr = 192.168.0.17; + ip6.addr = 2a01:4f8:120:30e8::17; +} +admin sun:/etc [1955]% sudo service jail start sync +Password: +Starting jails: sync. +admin sun:/etc [1956]% jls | grep sync + 103 192.168.0.17 sync.ian.buetow.org /jail/sync +admin sun:/etc [1957]% sudo jexec 103 /bin/csh +root@sync:/ # ifconfig -a +re0: flags=8843 metric 0 mtu 1500 + options=8209b + ether 50:46:5d:9f:fd:1e + inet6 2a01:4f8:120:30e8::17 prefixlen 64 + nd6 options=8021 + media: Ethernet autoselect (1000baseT ) + status: active +lo0: flags=8049 metric 0 mtu 16384 + options=600003 + nd6 options=21 + lo1: flags=8049 metric 0 mtu 16384 + options=600003 + inet 192.168.0.17 netmask 0xffffffff + nd6 options=29 +``` + +## Inside-Jail Puppet + +To automatically setup the applications running in the Jail I am using Puppet as well. I wrote a few scripts which bootstrap Puppet inside of a newly created Jail. It is doing the following: + +* Mounts an encrypted container (containing a secret Puppet manifests [git repository]) +* Activates "pkg-ng", the FreeBSD binary package manager, in the Jail +* Installs Puppet plus all dependencies in the Jail +* Updates the Jail via "freebsd-update" to the latest version +* Restarts the Jail and invokes Puppet. +* Puppet then also schedules a periodic cron job for the next Puppet runs. + +``` +admin sun:~ [1951]% sudo /opt/snonux/local/etc/init.d/enc activate sync +Starting jails: dns. +The package management tool is not yet installed on your system. +Do you want to fetch and install it now? [y/N]: y +Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/freebsd:10:x86:64/latest, please wait... +Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done +[sync.ian.buetow.org] Installing pkg-1.7.2... +[sync.ian.buetow.org] Extracting pkg-1.7.2: 100% +Updating FreeBSD repository catalogue... +[sync.ian.buetow.org] Fetching meta.txz: 100% 944 B 0.9kB/s 00:01 +[sync.ian.buetow.org] Fetching packagesite.txz: 100% 5 MiB 5.6MB/s 00:01 +Processing entries: 100% +FreeBSD repository update completed. 25091 packages processed. +Updating database digests format: 100% +The following 20 package(s) will be affected (of 0 checked): + + New packages to be INSTALLED: + git: 2.7.4_1 + expat: 2.1.0_3 + python27: 2.7.11_1 + libffi: 3.2.1 + indexinfo: 0.2.4 + gettext-runtime: 0.19.7 + p5-Error: 0.17024 + perl5: 5.20.3_9 + cvsps: 2.1_1 + p5-Authen-SASL: 2.16_1 + p5-Digest-HMAC: 1.03_1 + p5-GSSAPI: 0.28_1 + curl: 7.48.0_1 + ca_root_nss: 3.22.2 + p5-Net-SMTP-SSL: 1.03 + p5-IO-Socket-SSL: 2.024 + p5-Net-SSLeay: 1.72 + p5-IO-Socket-IP: 0.37 + p5-Socket: 2.021 + p5-Mozilla-CA: 20160104 + + The process will require 144 MiB more space. + 30 MiB to be downloaded. +[sync.ian.buetow.org] Fetching git-2.7.4_1.txz: 100% 4 MiB 3.7MB/s 00:01 +[sync.ian.buetow.org] Fetching expat-2.1.0_3.txz: 100% 98 KiB 100.2kB/s 00:01 +[sync.ian.buetow.org] Fetching python27-2.7.11_1.txz: 100% 10 MiB 10.7MB/s 00:01 +[sync.ian.buetow.org] Fetching libffi-3.2.1.txz: 100% 35 KiB 36.2kB/s 00:01 +[sync.ian.buetow.org] Fetching indexinfo-0.2.4.txz: 100% 5 KiB 5.0kB/s 00:01 +[sync.ian.buetow.org] Fetching gettext-runtime-0.19.7.txz: 100% 148 KiB 151.1kB/s 00:01 +[sync.ian.buetow.org] Fetching p5-Error-0.17024.txz: 100% 24 KiB 24.8kB/s 00:01 +[sync.ian.buetow.org] Fetching perl5-5.20.3_9.txz: 100% 13 MiB 6.9MB/s 00:02 +[sync.ian.buetow.org] Fetching cvsps-2.1_1.txz: 100% 41 KiB 42.1kB/s 00:01 +[sync.ian.buetow.org] Fetching p5-Authen-SASL-2.16_1.txz: 100% 44 KiB 45.1kB/s 00:01 +[sync.ian.buetow.org] Fetching p5-Digest-HMAC-1.03_1.txz: 100% 9 KiB 9.5kB/s 00:01 +[sync.ian.buetow.org] Fetching p5-GSSAPI-0.28_1.txz: 100% 41 KiB 41.7kB/s 00:01 +[sync.ian.buetow.org] Fetching curl-7.48.0_1.txz: 100% 2 MiB 2.2MB/s 00:01 +[sync.ian.buetow.org] Fetching ca_root_nss-3.22.2.txz: 100% 324 KiB 331.4kB/s 00:01 +[sync.ian.buetow.org] Fetching p5-Net-SMTP-SSL-1.03.txz: 100% 11 KiB 10.8kB/s 00:01 +[sync.ian.buetow.org] Fetching p5-IO-Socket-SSL-2.024.txz: 100% 153 KiB 156.4kB/s 00:01 +[sync.ian.buetow.org] Fetching p5-Net-SSLeay-1.72.txz: 100% 234 KiB 239.3kB/s 00:01 +[sync.ian.buetow.org] Fetching p5-IO-Socket-IP-0.37.txz: 100% 27 KiB 27.4kB/s 00:01 +[sync.ian.buetow.org] Fetching p5-Socket-2.021.txz: 100% 37 KiB 38.0kB/s 00:01 +[sync.ian.buetow.org] Fetching p5-Mozilla-CA-20160104.txz: 100% 147 KiB 150.8kB/s 00:01 +Checking integrity... +[sync.ian.buetow.org] [1/12] Installing libyaml-0.1.6_2... +[sync.ian.buetow.org] [1/12] Extracting libyaml-0.1.6_2: 100% +[sync.ian.buetow.org] [2/12] Installing libedit-3.1.20150325_2... +[sync.ian.buetow.org] [2/12] Extracting libedit-3.1.20150325_2: 100% +[sync.ian.buetow.org] [3/12] Installing ruby-2.2.4,1... +[sync.ian.buetow.org] [3/12] Extracting ruby-2.2.4,1: 100% +[sync.ian.buetow.org] [4/12] Installing ruby22-gems-2.6.2... +[sync.ian.buetow.org] [4/12] Extracting ruby22-gems-2.6.2: 100% +[sync.ian.buetow.org] [5/12] Installing libxml2-2.9.3... +[sync.ian.buetow.org] [5/12] Extracting libxml2-2.9.3: 100% +[sync.ian.buetow.org] [6/12] Installing dmidecode-3.0... +[sync.ian.buetow.org] [6/12] Extracting dmidecode-3.0: 100% +[sync.ian.buetow.org] [7/12] Installing rubygem-json_pure-1.8.3... +[sync.ian.buetow.org] [7/12] Extracting rubygem-json_pure-1.8.3: 100% +[sync.ian.buetow.org] [8/12] Installing augeas-1.4.0... +[sync.ian.buetow.org] [8/12] Extracting augeas-1.4.0: 100% +[sync.ian.buetow.org] [9/12] Installing rubygem-facter-2.4.4... +[sync.ian.buetow.org] [9/12] Extracting rubygem-facter-2.4.4: 100% +[sync.ian.buetow.org] [10/12] Installing rubygem-hiera1-1.3.4_1... +[sync.ian.buetow.org] [10/12] Extracting rubygem-hiera1-1.3.4_1: 100% +[sync.ian.buetow.org] [11/12] Installing rubygem-ruby-augeas-0.5.0_2... +[sync.ian.buetow.org] [11/12] Extracting rubygem-ruby-augeas-0.5.0_2: 100% +[sync.ian.buetow.org] [12/12] Installing puppet38-3.8.4_1... +===> Creating users and/or groups. +Creating group 'puppet' with gid '814'. +Creating user 'puppet' with uid '814'. +[sync.ian.buetow.org] [12/12] Extracting puppet38-3.8.4_1: 100% +. +. +. +. +. +Looking up update.FreeBSD.org mirrors... 4 mirrors found. +Fetching public key from update4.freebsd.org... done. +Fetching metadata signature for 10.1-RELEASE from update4.freebsd.org... done. +Fetching metadata index... done. +Fetching 2 metadata files... done. +Inspecting system... done. +Preparing to download files... done. +Fetching 874 patches.....10....20....30.... +. +. +. +Applying patches... done. +Fetching 1594 files... +Installing updates... +done. +Info: Loading facts +Info: Loading facts +Info: Loading facts +Info: Loading facts +Could not retrieve fact='pkgng_version', resolution='': undefined method `pkgng_enabled' for Facter:Module +Warning: Config file /usr/local/etc/puppet/hiera.yaml not found, using Hiera defaults +Notice: Compiled catalog for sync.ian.buetow.org in environment production in 1.31 seconds +Warning: Found multiple default providers for package: pkgng, gem, pip; using pkgng +Info: Applying configuration version '1460192563' +Notice: /Stage[main]/S_base_freebsd/User[root]/shell: shell changed '/bin/csh' to '/bin/tcsh' +Notice: /Stage[main]/S_user::Root_files/S_user::All_files[root_user]/File[/root/user]/ensure: created +Notice: /Stage[main]/S_user::Root_files/S_user::My_files[root]/File[/root/userfiles]/ensure: created +Notice: /Stage[main]/S_user::Root_files/S_user::My_files[root]/File[/root/.task]/ensure: created +. +. +. +. +Notice: Finished catalog run in 206.09 seconds +``` + +## Managing multiple Jails + +Of course I am operating multiple Jails on the same host this way with Puppet: + +* A Jail for the MTA +* A Jail for the Webserver +* A Jail for BIND DNS server +* A Jail for syncing data forth and back between various servers +* A Jail for other personal (experimental) use +* ...etc + +All done in a pretty automated manor. + +Other *BSD related posts are: + +<< template::inline::index bsd + +E-Mail your comments to `paul@nospam.buetow.org` :-) + +=> ../ Back to the main site diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi index 41af24e3..1c9ea500 100644 --- a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi +++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi @@ -656,6 +656,13 @@ OpenBSD suits perfectly here as all the tools are already part of the base insta Why re-inventing the wheel? I love that a `Rexfile` is just a Perl DSL. Also, OpenBSD comes with Perl in the base system. So no new programming language had to be added to my mix for the configuration management system. Also, the `acme.sh` shell script is not a Bash but a standard Bourne shell script, so I didn't have to install an additional shell as OpenBSD does not come with the Bash pre-installed. +Other *BSD related posts are: + +=> ./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi 2016-04-09 Jails and ZFS with Puppet on FreeBSD +=> ./2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi 2022-07-30 Let's Encrypt with OpenBSD and Rex (You are currently reading this) +=> ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD +=> ./2024-01-13-one-reason-why-i-love-openbsd.gmi 2024-01-13 One reason why I love OpenBSD + E-Mail your comments to `paul@nospam.buetow.org` :-) => ../ Back to the main site diff --git a/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi.tpl b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi.tpl new file mode 100644 index 00000000..1a83ef67 --- /dev/null +++ b/gemfeed/2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi.tpl @@ -0,0 +1,665 @@ +# Let's Encrypt with OpenBSD and Rex + +> Published at 2022-07-30T12:14:31+01:00 + +``` + / _ \ + The Hebern Machine \ ." ". / + ___ / \ + .."" "".. | O | + / \ | | + / \ | | + --------------------------------- + _/ o (O) o _ | + _/ ." ". | + I/ _________________/ \ | + _/I ." | | + ===== / I / / | + ===== | | | \ | _________________." | +===== | | | | | / \ / _|_|__|_|_ __ | + | | | | | | | \ "._." / o o \ ." ". | + | --| --| -| / \ _/ / \ | + \____\____\__| \ ______ | / | | | + -------- --- / | | | + ( ) (O) / \ / | + ----------------------- ".__." | + _|__________________________________________|_ + / \ + /________________________________________________\ + ASCII Art by John Savard +``` + +I was amazed at how easy it is to automatically generate and update Let's Encrypt certificates with OpenBSD. + +## What's Let's Encrypt? + +> Let's Encrypt is a non-profit certificate authority run by Internet Security Research Group that provides X.509 certificates for Transport Layer Security (TLS) encryption at no charge. It is the world's largest certificate authority, used by more than 265 million websites, with the goal of all websites being secure and using HTTPS. + +=> https://en.wikipedia.org/wiki/Let's_Encrypt Source: Wikipedia + +In short, it gives away TLS certificates for your website - for free! The catch is, that the certificates are only valid for three months. So it is better to automate certificate generation and renewals. + +## Meet `acme-client` + +`acme-client` is the default Automatic Certifcate Management Environment (ACME) client on OpenBSD and part of the OpenBSD base system. + +When invoked, the client first checks whether certificates actually require to be generated. + +* It first checks whether a certificate already exists; if not, it will attempt to generate a new one. +* If the certificate already exists but expires within the next 30 days, it will renew it. +* Otherwise, `acme-client` won't do anything. + +Oversimplified, the following steps are undertaken by `acme-client` for generating a new certificate: + +* Reading its config file `/etc/acme-client.conf` for a list of hosts (and their alternative names) to generate certificates. So it means you can also have certificates for arbitrary subdomains! +* Automatic generation of the private certificate part (the certificate key) and the certificate signing request (CSR) to `/etc/ssl/...`. +* Requesting Let's Encrypt to sign the certificate. This also includes providing a set of temporary files requested by Let's Encrypt in the next step for verification. +* Let's Encrypt then will contact the hostname for the certificate through a particular URL (e.g. `http://foo.zone/.well-known/acme-challenge/...`) to verify that the requester is the valid owner of the host. +* Let's Encrypt generates a certificate, which then is downloaded to `/etc/ssl/...`. + +## Configuration + +There is some (but easy) configuration required to make that all work on OpenBSD. + +### acme-client.conf + +This is how my `/etc/acme-client.conf` looks like (I copied a template from `/etc/examples/acme-client.conf` to `/etc/acme-client.conf` and added my domains to the bottom: + +``` +# +# $OpenBSD: acme-client.conf,v 1.4 2020/09/17 09:13:06 florian Exp $ +# +authority letsencrypt { + api url "https://acme-v02.api.letsencrypt.org/directory" + account key "/etc/acme/letsencrypt-privkey.pem" +} + +authority letsencrypt-staging { + api url "https://acme-staging-v02.api.letsencrypt.org/directory" + account key "/etc/acme/letsencrypt-staging-privkey.pem" +} + +authority buypass { + api url "https://api.buypass.com/acme/directory" + account key "/etc/acme/buypass-privkey.pem" + contact "mailto:me@example.com" +} + +authority buypass-test { + api url "https://api.test4.buypass.no/acme/directory" + account key "/etc/acme/buypass-test-privkey.pem" + contact "mailto:me@example.com" +} + +domain buetow.org { + alternative names { www.buetow.org paul.buetow.org } + domain key "/etc/ssl/private/buetow.org.key" + domain full chain certificate "/etc/ssl/buetow.org.fullchain.pem" + sign with letsencrypt +} + +domain dtail.dev { + alternative names { www.dtail.dev } + domain key "/etc/ssl/private/dtail.dev.key" + domain full chain certificate "/etc/ssl/dtail.dev.fullchain.pem" + sign with letsencrypt +} + +domain foo.zone { + alternative names { www.foo.zone } + domain key "/etc/ssl/private/foo.zone.key" + domain full chain certificate "/etc/ssl/foo.zone.fullchain.pem" + sign with letsencrypt +} + +domain irregular.ninja { + alternative names { www.irregular.ninja } + domain key "/etc/ssl/private/irregular.ninja.key" + domain full chain certificate "/etc/ssl/irregular.ninja.fullchain.pem" + sign with letsencrypt +} + +domain snonux.land { + alternative names { www.snonux.land } + domain key "/etc/ssl/private/snonux.land.key" + domain full chain certificate "/etc/ssl/snonux.land.fullchain.pem" + sign with letsencrypt +} +``` + +### httpd.conf + +For ACME to work, you will need to configure the HTTP daemon so that the "special" ACME requests from Let's Encrypt are served correctly. I am using the standard OpenBSD `httpd` here. These are the snippets I use for the `foo.zone` host in `/etc/httpd.conf` (of course, you need a similar setup for all other hosts as well): + +``` +server "foo.zone" { + listen on * port 80 + location "/.well-known/acme-challenge/*" { + root "/acme" + request strip 2 + } + location * { + block return 302 "https://$HTTP_HOST$REQUEST_URI" + } +} + +server "foo.zone" { + listen on * tls port 443 + tls { + certificate "/etc/ssl/foo.zone.fullchain.pem" + key "/etc/ssl/private/foo.zone.key" + } + location * { + root "/htdocs/gemtexter/foo.zone" + directory auto index + } +} +``` + +As you see, plain HTTP only serves the ACME challenge path. Otherwise, it redirects the requests to TLS. The TLS section then attempts to use the Let's Encrypt certificates. + +It is worth noticing that `httpd` will start without the certificates being present. This will cause a certificate error when you try to reach the HTTPS endpoint, but it helps to bootstrap Let's Encrypt. As you saw in the config snippet above, Let's Encrypt only requests the plain HTTP endpoint for the verification process, so HTTPS doesn't need to be operational yet at this stage. But once the certificates are generated, you will have to reload or restart `httpd` to use any new certificate. + +### CRON job + +You could now run `doas acme-client foo.zone` to generate the certificate or to renew it. Or you could automate it with CRON. + +I have created a script `/usr/local/bin/acme.sh` for that for all of my domains: + +``` +#!/bin/sh + +function handle_cert { + host=$1 + # Create symlink, so that relayd also can read it. + crt_path=/etc/ssl/$host + if [ -e $crt_path.crt ]; then + rm $crt_path.crt + fi + ln -s $crt_path.fullchain.pem $crt_path.crt + # Requesting and renewing certificate. + /usr/sbin/acme-client -v $host +} + +has_update=no +handle_cert www.buetow.org +if [ $? -eq 0 ]; then + has_update=yes +fi +handle_cert www.paul.buetow.org +if [ $? -eq 0 ]; then + has_update=yes +fi +handle_cert www.tmp.buetow.org +if [ $? -eq 0 ]; then + has_update=yes +fi +handle_cert www.dtail.dev +if [ $? -eq 0 ]; then + has_update=yes +fi +handle_cert www.foo.zone +if [ $? -eq 0 ]; then + has_update=yes +fi +handle_cert www.irregular.ninja +if [ $? -eq 0 ]; then + has_update=yes +fi +handle_cert www.snonux.land +if [ $? -eq 0 ]; then + has_update=yes +fi + +# Pick up the new certs. +if [ $has_update = yes ]; then + /usr/sbin/rcctl reload httpd + /usr/sbin/rcctl reload relayd + /usr/sbin/rcctl restart smtpd +fi +``` + +And added the following line to `/etc/daily.local` to run the script once daily so that certificates will be renewed fully automatically: + +``` +/usr/local/bin/acme.sh +``` + +I am receiving a daily output via E-Mail like this now: + +``` +Running daily.local: +acme-client: /etc/ssl/buetow.org.fullchain.pem: certificate valid: 80 days left +acme-client: /etc/ssl/paul.buetow.org.fullchain.pem: certificate valid: 80 days left +acme-client: /etc/ssl/tmp.buetow.org.fullchain.pem: certificate valid: 80 days left +acme-client: /etc/ssl/dtail.dev.fullchain.pem: certificate valid: 80 days left +acme-client: /etc/ssl/foo.zone.fullchain.pem: certificate valid: 80 days left +acme-client: /etc/ssl/irregular.ninja.fullchain.pem: certificate valid: 80 days left +acme-client: /etc/ssl/snonux.land.fullchain.pem: certificate valid: 79 days left +``` + +## relayd.conf and smtpd.conf + +Besides `httpd`, `relayd` (mainly for Gemini) and `smtpd` (for mail, of course) also use TLS certificates. And as you can see in `acme.sh`, the services are reloaded or restarted (`smtpd` doesn't support reload) whenever a certificate is generated or updated. + +## Rexification + +I didn't write all these configuration files by hand. As a matter of fact, everything is automated with the Rex configuration management system. + +=> https://www.rexify.org + +At the top of the `Rexfile` I define all my hosts: + +``` +our @acme_hosts = qw/buetow.org paul.buetow.org tmp.buetow.org dtail.dev foo.zone irregular.ninja snonux.land/; +``` + +### General ACME client configuration + +ACME will be installed into the frontend group of hosts. Here, blowfish is the primary, and twofish is the secondary OpenBSD box. + +``` +group frontends => 'blowfish.buetow.org', 'twofish.buetow.org'; +``` + +This is my Rex task for the general ACME configuration: + +``` +desc 'Configure ACME client'; +task 'acme', group => 'frontends', + sub { + file '/etc/acme-client.conf', + content => template('./etc/acme-client.conf.tpl', + acme_hosts => \@acme_hosts, + is_primary => $is_primary), + owner => 'root', + group => 'wheel', + mode => '644'; + + file '/usr/local/bin/acme.sh', + content => template('./scripts/acme.sh.tpl', + acme_hosts => \@acme_hosts, + is_primary => $is_primary), + owner => 'root', + group => 'wheel', + mode => '744'; + + file '/etc/daily.local', + ensure => 'present', + owner => 'root', + group => 'wheel', + mode => '644'; + + append_if_no_such_line '/etc/daily.local', '/usr/local/bin/acme.sh'; + }; +``` + +And there is also a Rex task just to run the ACME script remotely: + +``` +desc 'Invoke ACME client'; +task 'acme_invoke', group => 'frontends', + sub { + say run '/usr/local/bin/acme.sh'; + }; + +``` + +Furthermore, this snippet (also at the top of the Rexfile) helps to determine whether the current server is the primary server (all hosts will be without the `www.` prefix) or the secondary server (all hosts will be with the `www.` prefix): + +``` +# Bootstrapping the FQDN based on the server IP as the hostname and domain +# facts aren't set yet due to the myname file in the first place. +our $fqdns = sub { + my $ipv4 = shift; + return 'blowfish.buetow.org' if $ipv4 eq '23.88.35.144'; + return 'twofish.buetow.org' if $ipv4 eq '108.160.134.135'; + Rex::Logger::info("Unable to determine hostname for $ipv4", 'error'); + return 'HOSTNAME-UNKNOWN.buetow.org'; +}; + +# To determine whether the server is the primary or the secondary. +our $is_primary = sub { + my $ipv4 = shift; + $fqdns->($ipv4) eq 'blowfish.buetow.org'; +}; +``` + +The following is the `acme-client.conf.tpl` Rex template file used for the automation. You see that the `www.` prefix isn't sent for the primary server. E.g. `foo.zone` will be served by the primary server (in my case, a server located in Germany) and `www.foo.zone` by the secondary server (in my case, a server located in Japan): + +``` +# +# $OpenBSD: acme-client.conf,v 1.4 2020/09/17 09:13:06 florian Exp $ +# +authority letsencrypt { + api url "https://acme-v02.api.letsencrypt.org/directory" + account key "/etc/acme/letsencrypt-privkey.pem" +} + +authority letsencrypt-staging { + api url "https://acme-staging-v02.api.letsencrypt.org/directory" + account key "/etc/acme/letsencrypt-staging-privkey.pem" +} + +authority buypass { + api url "https://api.buypass.com/acme/directory" + account key "/etc/acme/buypass-privkey.pem" + contact "mailto:me@example.com" +} + +authority buypass-test { + api url "https://api.test4.buypass.no/acme/directory" + account key "/etc/acme/buypass-test-privkey.pem" + contact "mailto:me@example.com" +} + +<% + our $primary = $is_primary->($vio0_ip); + our $prefix = $primary ? '' : 'www.'; +%> + +<% for my $host (@$acme_hosts) { %> +domain <%= $prefix.$host %> { + domain key "/etc/ssl/private/<%= $prefix.$host %>.key" + domain full chain certificate "/etc/ssl/<%= $prefix.$host %>.fullchain.pem" + sign with letsencrypt +} +<% } %> + +``` + +And this is the `acme.sh.tpl`: + +``` +#!/bin/sh + +<% + our $primary = $is_primary->($vio0_ip); + our $prefix = $primary ? '' : 'www.'; +-%> + +function handle_cert { + host=$1 + # Create symlink, so that relayd also can read it. + crt_path=/etc/ssl/$host + if [ -e $crt_path.crt ]; then + rm $crt_path.crt + fi + ln -s $crt_path.fullchain.pem $crt_path.crt + # Requesting and renewing certificate. + /usr/sbin/acme-client -v $host +} + +has_update=no +<% for my $host (@$acme_hosts) { -%> +handle_cert <%= $prefix.$host %> +if [ $? -eq 0 ]; then + has_update=yes +fi +<% } -%> + +# Pick up the new certs. +if [ $has_update = yes ]; then + /usr/sbin/rcctl reload httpd + /usr/sbin/rcctl reload relayd + /usr/sbin/rcctl restart smtpd +fi +``` + +### Service rexification + +These are the Rex tasks setting up `httpd`, `relayd` and `smtpd` services: + +``` +desc 'Setup httpd'; +task 'httpd', group => 'frontends', + sub { + append_if_no_such_line '/etc/rc.conf.local', 'httpd_flags='; + + file '/etc/httpd.conf', + content => template('./etc/httpd.conf.tpl', + acme_hosts => \@acme_hosts, + is_primary => $is_primary), + owner => 'root', + group => 'wheel', + mode => '644', + on_change => sub { service 'httpd' => 'restart' }; + + service 'httpd', ensure => 'started'; + }; + +desc 'Setup relayd'; +task 'relayd', group => 'frontends', + sub { + append_if_no_such_line '/etc/rc.conf.local', 'relayd_flags='; + + file '/etc/relayd.conf', + content => template('./etc/relayd.conf.tpl', + ipv6address => $ipv6address, + is_primary => $is_primary), + owner => 'root', + group => 'wheel', + mode => '600', + on_change => sub { service 'relayd' => 'restart' }; + + service 'relayd', ensure => 'started'; + }; + +desc 'Setup OpenSMTPD'; +task 'smtpd', group => 'frontends', + sub { + Rex::Logger::info('Dealing with mail aliases'); + file '/etc/mail/aliases', + source => './etc/mail/aliases', + owner => 'root', + group => 'wheel', + mode => '644', + on_change => sub { say run 'newaliases' }; + + Rex::Logger::info('Dealing with mail virtual domains'); + file '/etc/mail/virtualdomains', + source => './etc/mail/virtualdomains', + owner => 'root', + group => 'wheel', + mode => '644', + on_change => sub { service 'smtpd' => 'restart' }; + + Rex::Logger::info('Dealing with mail virtual users'); + file '/etc/mail/virtualusers', + source => './etc/mail/virtualusers', + owner => 'root', + group => 'wheel', + mode => '644', + on_change => sub { service 'smtpd' => 'restart' }; + + Rex::Logger::info('Dealing with smtpd.conf'); + file '/etc/mail/smtpd.conf', + content => template('./etc/mail/smtpd.conf.tpl', + is_primary => $is_primary), + owner => 'root', + group => 'wheel', + mode => '644', + on_change => sub { service 'smtpd' => 'restart' }; + + service 'smtpd', ensure => 'started'; + }; + +``` + +This is the `httpd.conf.tpl`: + +``` +<% + our $primary = $is_primary->($vio0_ip); + our $prefix = $primary ? '' : 'www.'; +%> + +# Plain HTTP for ACME and HTTPS redirect +<% for my $host (@$acme_hosts) { %> +server "<%= $prefix.$host %>" { + listen on * port 80 + location "/.well-known/acme-challenge/*" { + root "/acme" + request strip 2 + } + location * { + block return 302 "https://$HTTP_HOST$REQUEST_URI" + } +} +<% } %> + +# Gemtexter hosts +<% for my $host (qw/foo.zone snonux.land/) { %> +server "<%= $prefix.$host %>" { + listen on * tls port 443 + tls { + certificate "/etc/ssl/<%= $prefix.$host %>.fullchain.pem" + key "/etc/ssl/private/<%= $prefix.$host %>.key" + } + location * { + root "/htdocs/gemtexter/<%= $host %>" + directory auto index + } +} +<% } %> + +# DTail special host +server "<%= $prefix %>dtail.dev" { + listen on * tls port 443 + tls { + certificate "/etc/ssl/<%= $prefix %>dtail.dev.fullchain.pem" + key "/etc/ssl/private/<%= $prefix %>dtail.dev.key" + } + location * { + block return 302 "https://github.dtail.dev$REQUEST_URI" + } +} + +# Irregular Ninja special host +server "<%= $prefix %>irregular.ninja" { + listen on * tls port 443 + tls { + certificate "/etc/ssl/<%= $prefix %>irregular.ninja.fullchain.pem" + key "/etc/ssl/private/<%= $prefix %>irregular.ninja.key" + } + location * { + root "/htdocs/irregular.ninja" + directory auto index + } +} + +# buetow.org special host. +server "<%= $prefix %>buetow.org" { + listen on * tls port 443 + tls { + certificate "/etc/ssl/<%= $prefix %>buetow.org.fullchain.pem" + key "/etc/ssl/private/<%= $prefix %>buetow.org.key" + } + block return 302 "https://paul.buetow.org" +} + +server "<%= $prefix %>paul.buetow.org" { + listen on * tls port 443 + tls { + certificate "/etc/ssl/<%= $prefix %>paul.buetow.org.fullchain.pem" + key "/etc/ssl/private/<%= $prefix %>paul.buetow.org.key" + } + block return 302 "https://foo.zone/contact-information.html" +} + +server "<%= $prefix %>tmp.buetow.org" { + listen on * tls port 443 + tls { + certificate "/etc/ssl/<%= $prefix %>tmp.buetow.org.fullchain.pem" + key "/etc/ssl/private/<%= $prefix %>tmp.buetow.org.key" + } + root "/htdocs/buetow.org/tmp" + directory auto index +} +``` + +and this the `relayd.conf.tpl`: + +``` +<% + our $primary = $is_primary->($vio0_ip); + our $prefix = $primary ? '' : 'www.'; +%> + +log connection + +tcp protocol "gemini" { + tls keypair <%= $prefix %>foo.zone + tls keypair <%= $prefix %>buetow.org +} + +relay "gemini4" { + listen on <%= $vio0_ip %> port 1965 tls + protocol "gemini" + forward to 127.0.0.1 port 11965 +} + +relay "gemini6" { + listen on <%= $ipv6address->($hostname) %> port 1965 tls + protocol "gemini" + forward to 127.0.0.1 port 11965 +} +``` + +And last but not least, this is the `smtpd.conf.tpl`: + +``` +<% + our $primary = $is_primary->($vio0_ip); + our $prefix = $primary ? '' : 'www.'; +%> + +pki "buetow_org_tls" cert "/etc/ssl/<%= $prefix %>buetow.org.fullchain.pem" +pki "buetow_org_tls" key "/etc/ssl/private/<%= $prefix %>buetow.org.key" + +table aliases file:/etc/mail/aliases +table virtualdomains file:/etc/mail/virtualdomains +table virtualusers file:/etc/mail/virtualusers + +listen on socket +listen on all tls pki "buetow_org_tls" hostname "<%= $prefix %>buetow.org" +#listen on all + +action localmail mbox alias +action receive mbox virtual +action outbound relay + +match from any for domain action receive +match from local for local action localmail +match from local for any action outbound +``` + +## All pieces together + +For the complete `Rexfile` example and all the templates, please look at the Git repository: + +=> https://codeberg.org/snonux/rexfiles + +Besides ACME, other things, such as DNS servers, are also rexified. The following command will run all the Rex tasks and configure everything on my frontend machines automatically: + +``` +rex commons +``` + +The `commons` is a group of tasks I specified which combines a set of common tasks I always want to execute on all frontend machines. This also includes the ACME tasks mentioned in this article! + +## Conclusion + +ACME and Let's Encrypt greatly help reduce recurring manual maintenance work (creating and renewing certificates). Furthermore, all the certificates are free of cost! I love to use OpenBSD and Rex to automate all of this. + +OpenBSD suits perfectly here as all the tools are already part of the base installation. But I like underdogs. Rex is not as powerful and popular as other configuration management systems (e.g. Puppet, Chef, SALT or even Ansible). It is more of an underdog, and the community is small. + +Why re-inventing the wheel? I love that a `Rexfile` is just a Perl DSL. Also, OpenBSD comes with Perl in the base system. So no new programming language had to be added to my mix for the configuration management system. Also, the `acme.sh` shell script is not a Bash but a standard Bourne shell script, so I didn't have to install an additional shell as OpenBSD does not come with the Bash pre-installed. + +Other *BSD related posts are: + +<< template::inline::index bsd + +E-Mail your comments to `paul@nospam.buetow.org` :-) + +=> ../ Back to the main site diff --git a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi new file mode 100644 index 00000000..a29ec709 --- /dev/null +++ b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi @@ -0,0 +1,71 @@ +# One reason why I love OpenBSD + +> Published at 2024-01-13T22:55:33+02:00 + +``` + . + A ; + | ,--,-/ \---,-/| , + _|\,'. /| /| `/|-. + \`.' /| , `;. + ,'\ A A A A _ /| `.; + ,/ _ A _ / _ /| ; + /\ / \ , , A / / `/| + /_| | _ \ , , ,/ \ + // | |/ `.\ ,- , , ,/ ,/ \/ + / @| |@ / /' \ \ , > /| ,--. + |\_/ \_/ / | | , ,/ \ ./' __:.. + | __ __ | | | .--. , > > |-' / ` + ,/| / ' \ | | | \ , | / + / |<--.__,->| | | . `. > > / ( +/_,' \\ ^ / \ / / `. >-- /^\ | + \\___/ \ / / \__' \ \ \/ \ | + `. |/ , , /`\ \ ) + \ ' |/ , V \ / `-\ + `|/ ' V V \ \.' \_ + '`-. V V \./'\ + `|/-. \ / \ /,---`\ kat + n + / `._____V_____V' + ' ' +``` + +I just upgraded my OpenBSD's from `7.3` to `7.4` by following the unattended upgrade guide: + +=> https://www.openbsd.org/faq/upgrade74.html + +```shell +doas installboot sd0 # Update the bootloader (not for every upgrade required) +doas sysupgrade # Update all binaries (including Kernel) +``` + +`sysupgrade` downloaded and upgraded to the next release and rebooted the system. After the reboot, I run: + +```shell +doas sysmerge # Update system configuration files +doas pkg_add -u # Update all packages +doas reboot # Just in case, reboot one more time +``` + +That's it! Took me around 5 minutes in total! No issues, only these few comands, only 5 minutes! It just works! No problems, no conflicts, no tons (actually none) config file merge conflicts. + +I followed the same procedure the previous times and never encountered any difficulties with any OpenBSD upgrades. + +I have seen upgrades of other Operating Systems either take a long time or break the system (which takes manual steps to repair). That's just one of many reasons why I love OpenBSD! There appear never to be any problems. It just gets its job done! + +=> https://www.openbsd.org The OpenBSD Project + +BTW: are you looking for an opinionated OpenBSD VM hoster? OpenBSD Amsterdam may be for you. They rock (I am having a VM there, too)! + +=> https://openbsd.amsterdam + +Other *BSD related posts are: + +=> ./2016-04-09-jails-and-zfs-on-freebsd-with-puppet.gmi 2016-04-09 Jails and ZFS with Puppet on FreeBSD +=> ./2022-07-30-lets-encrypt-with-openbsd-and-rex.gmi 2022-07-30 Let's Encrypt with OpenBSD and Rex +=> ./2022-10-30-installing-dtail-on-openbsd.gmi 2022-10-30 Installing DTail on OpenBSD +=> ./2024-01-13-one-reason-why-i-love-openbsd.gmi 2024-01-13 One reason why I love OpenBSD (You are currently reading this) + +E-Mail your comments to `paul@nospam.buetow.org` :-) + +=> ../ Back to the main site diff --git a/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi.tpl b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi.tpl new file mode 100644 index 00000000..1d9c71a1 --- /dev/null +++ b/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi.tpl @@ -0,0 +1,68 @@ +# One reason why I love OpenBSD + +> Published at 2024-01-13T22:55:33+02:00 + +``` + . + A ; + | ,--,-/ \---,-/| , + _|\,'. /| /| `/|-. + \`.' /| , `;. + ,'\ A A A A _ /| `.; + ,/ _ A _ / _ /| ; + /\ / \ , , A / / `/| + /_| | _ \ , , ,/ \ + // | |/ `.\ ,- , , ,/ ,/ \/ + / @| |@ / /' \ \ , > /| ,--. + |\_/ \_/ / | | , ,/ \ ./' __:.. + | __ __ | | | .--. , > > |-' / ` + ,/| / ' \ | | | \ , | / + / |<--.__,->| | | . `. > > / ( +/_,' \\ ^ / \ / / `. >-- /^\ | + \\___/ \ / / \__' \ \ \/ \ | + `. |/ , , /`\ \ ) + \ ' |/ , V \ / `-\ + `|/ ' V V \ \.' \_ + '`-. V V \./'\ + `|/-. \ / \ /,---`\ kat + n + / `._____V_____V' + ' ' +``` + +I just upgraded my OpenBSD's from `7.3` to `7.4` by following the unattended upgrade guide: + +=> https://www.openbsd.org/faq/upgrade74.html + +```shell +doas installboot sd0 # Update the bootloader (not for every upgrade required) +doas sysupgrade # Update all binaries (including Kernel) +``` + +`sysupgrade` downloaded and upgraded to the next release and rebooted the system. After the reboot, I run: + +```shell +doas sysmerge # Update system configuration files +doas pkg_add -u # Update all packages +doas reboot # Just in case, reboot one more time +``` + +That's it! Took me around 5 minutes in total! No issues, only these few comands, only 5 minutes! It just works! No problems, no conflicts, no tons (actually none) config file merge conflicts. + +I followed the same procedure the previous times and never encountered any difficulties with any OpenBSD upgrades. + +I have seen upgrades of other Operating Systems either take a long time or break the system (which takes manual steps to repair). That's just one of many reasons why I love OpenBSD! There appear never to be any problems. It just gets its job done! + +=> https://www.openbsd.org The OpenBSD Project + +BTW: are you looking for an opinionated OpenBSD VM hoster? OpenBSD Amsterdam may be for you. They rock (I am having a VM there, too)! + +=> https://openbsd.amsterdam + +Other *BSD related posts are: + +<< template::inline::index bsd + +E-Mail your comments to `paul@nospam.buetow.org` :-) + +=> ../ Back to the main site diff --git a/gemfeed/atom.xml b/gemfeed/atom.xml index e22d2d0f..1ad8513c 100644 --- a/gemfeed/atom.xml +++ b/gemfeed/atom.xml @@ -1,11 +1,103 @@ - 2024-01-09T18:45:17+02:00 + 2024-01-13T23:06:21+02:00 foo.zone feed To be in the .zone! gemini://foo.zone/ + + One reason why I love OpenBSD + + gemini://foo.zone/gemfeed/2024-01-13-one-reason-why-i-love-openbsd.gmi + 2024-01-13T22:55:33+02:00 + + Paul Buetow aka snonux + paul@dev.buetow.org + + I just upgraded my OpenBSD's from `7.3` to `7.4` by following the unattended upgrade guide: + +
+

One reason why I love OpenBSD


+
+Published at 2024-01-13T22:55:33+02:00
+
+
+                         .
+                          A       ;
+                |   ,--,-/ \---,-/|  ,
+               _|\,'. /|      /|   `/|-.
+           \`.'    /|      ,            `;.
+          ,'\   A     A         A   A _ /| `.;
+        ,/  _              A       _  / _   /|  ;
+       /\  / \   ,  ,           A  /    /     `/|
+      /_| | _ \         ,     ,             ,/  \
+     // | |/ `.\  ,-      ,       ,   ,/ ,/      \/
+     / @| |@  / /'   \  \      ,              >  /|    ,--.
+    |\_/   \_/ /      |  |           ,  ,/        \  ./' __:..
+    |  __ __  |       |  | .--.  ,         >  >   |-'   /     `
+  ,/| /  '  \ |       |  |     \      ,           |    /
+ /  |<--.__,->|       |  | .    `.        >  >    /   (
+/_,' \\  ^  /  \     /  /   `.    >--            /^\   |
+      \\___/    \   /  /      \__'     \   \   \/   \  |
+       `.   |/          ,  ,                  /`\    \  )
+         \  '  |/    ,       V    \          /        `-\
+          `|/  '  V      V           \    \.'            \_
+           '`-.       V       V        \./'\
+               `|/-.      \ /   \ /,---`\         kat
+                n
+                /   `._____V_____V'
+                           '     '
+
+
+I just upgraded my OpenBSD's from 7.3 to 7.4 by following the unattended upgrade guide:
+
+https://www.openbsd.org/faq/upgrade74.html
+
+ +
doas installboot sd0 # Update the bootloader (not for every upgrade required)
+doas sysupgrade # Update all binaries (including Kernel)
+
+
+sysupgrade downloaded and upgraded to the next release and rebooted the system. After the reboot, I run:
+
+ +
doas sysmerge # Update system configuration files
+doas pkg_add -u # Update all packages
+doas reboot # Just in case, reboot one more time
+
+
+That's it! Took me around 5 minutes in total! No issues, only these few comands, only 5 minutes! It just works! No problems, no conflicts, no tons (actually none) config file merge conflicts.
+
+I followed the same procedure the previous times and never encountered any difficulties with any OpenBSD upgrades.
+
+I have seen upgrades of other Operating Systems either take a long time or break the system (which takes manual steps to repair). That's just one of many reasons why I love OpenBSD! There appear never to be any problems. It just gets its job done!
+
+The OpenBSD Project
+
+BTW: are you looking for an opinionated OpenBSD VM hoster? OpenBSD Amsterdam may be for you. They rock (I am having a VM there, too)!
+
+https://openbsd.amsterdam
+
+Other *BSD related posts are:
+
+2016-04-09 Jails and ZFS with Puppet on FreeBSD
+2022-07-30 Let's Encrypt with OpenBSD and Rex
+2022-10-30 Installing DTail on OpenBSD
+2024-01-13 One reason why I love OpenBSD (You are currently reading this)
+
+E-Mail your comments to paul@nospam.buetow.org :-)
+
+Back to the main site
+
+ + Site Reliability Engineering - Part 3: On-Call Culture and the Human Aspect @@ -4899,6 +4991,13 @@ rex commons
Why re-inventing the wheel? I love that a Rexfile is just a Perl DSL. Also, OpenBSD comes with Perl in the base system. So no new programming language had to be added to my mix for the configuration management system. Also, the acme.sh shell script is not a Bash but a standard Bourne shell script, so I didn't have to install an additional shell as OpenBSD does not come with the Bash pre-installed.

+Other *BSD related posts are:
+
+2016-04-09 Jails and ZFS with Puppet on FreeBSD
+2022-07-30 Let's Encrypt with OpenBSD and Rex (You are currently reading this)
+2022-10-30 Installing DTail on OpenBSD
+2024-01-13 One reason why I love OpenBSD
+
E-Mail your comments to paul@nospam.buetow.org :-)

Back to the main site
@@ -8772,212 +8871,6 @@ dtail –servers serverlist.txt –files ‘/var/log/*.log’ –regex ‘(?i:er
E-Mail your comments to paul@nospam.buetow.org :-)

-Back to the main site
- - - - - Realistic load testing with I/O Riot for Linux - - gemini://foo.zone/gemfeed/2018-06-01-realistic-load-testing-with-ioriot-for-linux.gmi - 2018-06-01T14:50:29+01:00 - - Paul Buetow aka snonux - paul@dev.buetow.org - - This text first was published in the german IT-Administrator computer Magazine. 3 years have passed since and I decided to publish it on my blog too. - -
-

Realistic load testing with I/O Riot for Linux


-
-Published at 2018-06-01T14:50:29+01:00; Updated at 2021-05-08
-
-
-       .---.
-      /     \
-      \.@-@./
-      /`\_/`\
-     //  _  \\
-    | \     )|_
-   /`\_`>  <_/ \
-jgs\__/'---'\__/
-
-
-

Foreword


-
-This text first was published in the german IT-Administrator computer Magazine. 3 years have passed since and I decided to publish it on my blog too.
-
-https://www.admin-magazin.de/Das-Heft/2018/06/Realistische-Lasttests-mit-I-O-Riot
-
-I havn't worked on I/O Riot for some time now, but all what is written here is still valid. I am still using I/O Riot to debug I/O issues and pattern once in a while, so by all means the tool is not obsolete yet. The tool even helped to resolve a major production incident at work caused by disk I/O.
-
-I am eagerly looking forward to revamp I/O Riot so that it uses the new BPF Linux capabilities instead of plain old Systemtap (or alternatively: Newer versions of Systemtap can also use BPF as the backend I have learned). Also, when I wrote I/O Riot initially, I didn't have any experience with the Go programming language yet and therefore I wrote it in C. Once it gets revamped I might consider using Go instead of C as it would spare me from many segmentation faults and headaches during development ;-). I might also just stick to C for plain performance reasons and just refactor the code dealing with concurrency.
-
-Pleace notice that some of the screenshots show the command "ioreplay" instead of "ioriot". That's because the name has changed after taking those.
-
-

The article


-
-With I/O Riot IT administrators can load test and optimize the I/O subsystem of Linux-based operating systems. The tool makes it possible to record I/O patterns and replay them at a later time as often as desired. This means bottlenecks can be reproduced and eradicated.
-
-When storing huge amounts of data, such as more than 200 billion archived emails at Mimecast, it's not only the available storage capacity that matters, but also the data throughput and latency. At the same time, operating costs must be kept as low as possible. The more systems involved, the more important it is to optimize the hardware, the operating system and the applications running on it.
-
-

Background: Existing Techniques


-
-Conventional I/O benchmarking: Administrators usually use open source benchmarking tools like IOZone and bonnie++. Available database systems such as Redis and MySQL come with their own benchmarking tools. The common problem with these tools is that they work with prescribed artificial I/O patterns. Although this can test both sequential and randomized data access, the patterns do not correspond to what can be found on production systems.
-
-Testing by load test environment: Another option is to use a separate load test environment in which, as far as possible, a production environment with all its dependencies is simulated. However, an environment consisting of many microservices is very complex. Microservices are usually managed by different teams, which means extra coordination effort for each load test. Another challenge is to generate the load as authentically as possible so that the patterns correspond to a productive environment. Such a load test environment can only handle as many requests as its weakest link can handle. For example, load generators send many read and write requests to a frontend microservice, whereby the frontend forwards the requests to a backend microservice responsible for storing the data. If the frontend service does not process the requests efficiently enough, the backend service is not well utilized in the first place. As a rule, all microservices are clustered across many servers, which makes everything even more complicated. Under all these conditions it is very difficult to test I/O of separate backend systems. Moreover, for many small and medium-sized companies, a separate load test environment would not be feasible for cost reasons.
-
-Testing in the production environment: For these reasons, benchmarks are often carried out in the production environment. In order to derive value from this such tests are especially performed during peak hours when systems are under high load. However, testing on production systems is associated with risks and can lead to failure or loss of data without adequate protection.
-
-

Benchmarking the Email Cloud at Mimecast


-
-For email archiving, Mimecast uses an internally developed microservice, which is operated directly on Linux-based storage systems. A storage cluster is divided into several replication volumes. Data is always replicated three times across two secure data centers. Customer data is automatically allocated to one or more volumes, depending on throughput, so that all volumes are automatically assigned the same load. Customer data is archived on conventional, but inexpensive hard disks with several terabytes of storage capacity each. I/O benchmarking proved difficult for all the reasons mentioned above. Furthermore, there are no ready-made tools for this purpose in the case of self-developed software. The service operates on many block devices simultaneously, which can make the RAID controller a bottleneck. None of the freely available benchmarking tools can test several block devices at the same time without extra effort. In addition, emails typically consist of many small files. Randomized access to many small files is particularly inefficient. In addition to many software adaptations, the hardware and operating system must also be optimized.
-
-Mimecast encourages employees to be innovative and pursue their own ideas in the form of an internal competition, Pet Project. The goal of the pet project I/O Riot was to simplify OS and hardwa