From 78efc18d07396be18649de1b873da5a89d484bd7 Mon Sep 17 00:00:00 2001 From: Paul Buetow Date: Wed, 7 Jan 2026 21:14:36 +0200 Subject: Update --- frontends/AGENT.md | 253 ---------------------------------------------------- frontends/AGENTS.md | 253 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 253 insertions(+), 253 deletions(-) delete mode 100644 frontends/AGENT.md create mode 100644 frontends/AGENTS.md (limited to 'frontends') diff --git a/frontends/AGENT.md b/frontends/AGENT.md deleted file mode 100644 index d989eca..0000000 --- a/frontends/AGENT.md +++ /dev/null @@ -1,253 +0,0 @@ -# Frontend Infrastructure Knowledge - -## Architecture Overview - -### Request Flow -``` -Internet → relayd (port 443) → routing decision → httpd (port 8080) or f3s cluster (port 80) -``` - -### Key Components - -**relayd** - Reverse proxy that: -- Terminates TLS on port 443 (IPv4 and IPv6) -- Routes requests based on Host header matching -- Has two backend pools: - - `` (127.0.0.1, ::1) - Routes to local httpd on port 8080 - - `` (192.168.2.120-122) - Routes to f3s k3s cluster on port 80 -- Falls back to f3s cluster when no explicit routing match exists - -**httpd** - OpenBSD httpd that: -- Listens on port 8080 (behind relayd) -- Listens on port 80 for ACME challenges and HTTP→HTTPS redirects -- Serves static content for various domains -- Has server-specific blocks for each server's own hostname - -**Rexfile** - Configuration management using Rex (Perl): -- Defines configuration arrays (`@acme_hosts`, `@f3s_hosts`, etc.) -- Templates use these arrays to generate httpd and relayd configs -- Deploys to both blowfish and fishfinger servers in parallel -- Each server receives templates processed with its own `$hostname` value - -## Configuration Arrays - -### @acme_hosts -Controls which hosts get: -- ACME certificate requests -- HTTP port 80 server blocks for ACME challenges -- Explicit routing rules in relayd to `` - -**Critical**: Hosts NOT in `@acme_hosts` will fall through to f3s cluster backends in relayd. - -### @f3s_hosts -Hosts served by the f3s k3s cluster: -- Get fallback page served by httpd -- Special routing rules in relayd to f3s backends - -### @prefixes -Array: `('', 'www.', 'standby.')` - -Used in loops to create hostname variants: -- `foo.zone` -- `www.foo.zone` -- `standby.foo.zone` - -## Template Processing - -Rex processes `.tpl` files using embedded Perl: - -```perl -<% ... -%> # Perl code (- suppresses trailing newline) -<%= $var %> # Print variable value -``` - -Templates are processed **per-server** with different values: -- `$hostname` = "blowfish" or "fishfinger" -- `$domain` = "buetow.org" -- `$hostname.$domain` = "blowfish.buetow.org" or "fishfinger.buetow.org" - -## Routing Configuration - -### Explicit Routing Rules (relayd.conf.tpl:45-50) - -```perl -<% for my $host (@$acme_hosts) { - next if grep { $_ eq $host } @$f3s_hosts; - for my $prefix (@prefixes) { -%> -match request header "Host" value "<%= $prefix.$host -%>" forward to -``` - -- Only hosts in `@acme_hosts` get explicit routing to `` -- Excludes f3s hosts (they have separate routing) -- Creates rules for all prefixes ('', 'www.', 'standby.') - -### Routing Logic - -**Routing is explicit, not implicit**: Just because httpd has a server block doesn't mean relayd will route to it. The routing decision happens in relayd based on: - -1. Explicit Host header match → route to specified backend -2. No match → fall through to default relay backends (f3s cluster first, then localhost) - -## TLS Certificate Management - -### Certificate Loading (relayd.conf.tpl:24-31) - -```perl -http protocol "https" { - <% for my $host (@$acme_hosts) { -%> - tls keypair <%= $host %> - tls keypair standby.<%= $host %> - <% } -%> - tls keypair <%= $hostname.'.'.$domain -%> -``` - -**Critical insight**: In multi-server deployments, each server only has its own TLS certificate. - -- blowfish has: `blowfish.buetow.org.crt` (NOT fishfinger's cert) -- fishfinger has: `fishfinger.buetow.org.crt` (NOT blowfish's cert) - -When the template runs on blowfish, it tries to load certs for ALL hosts in `@acme_hosts`. If fishfinger.buetow.org is in the array, relayd will fail to start because that cert doesn't exist on blowfish. - -**Solution pattern**: Skip server-specific hostnames in the loop, use dedicated keypair line: -```perl -<% for my $host (@$acme_hosts) { - next if $host eq 'blowfish.buetow.org' or $host eq 'fishfinger.buetow.org'; -%> -``` - -The line `tls keypair <%= $hostname.'.'.$domain -%>` loads the correct cert for each server. - -## Server Block Management - -### httpd.conf.tpl Patterns - -**ACME and redirect blocks (port 80)**: -```perl -<% for my $host (@$acme_hosts) { - next if $host eq "$hostname.$domain"; # Skip current server - for my $prefix (@prefixes) { -%> -server "<%= $prefix.$host %>" { - listen on * port 80 -``` - -**Why skip current server**: Each server has a dedicated "Current server's FQDN" block: - -```perl -server "<%= "$hostname.$domain" %>" { - listen on * port 80 - ... -} -``` - -Without the skip, adding server hostnames to `@acme_hosts` creates duplicate server blocks, causing httpd to fail with "server defined twice" error. - -### Content Serving Blocks (port 8080) - -Different patterns based on content type: -- **Gemtexter sites**: Serve from `/htdocs/gemtexter/` -- **Server self**: Serve from `/htdocs/buetow.org/self` -- **Special hosts**: Custom root paths (e.g., gogios, joern, dory) -- **f3s fallback**: Rewrite all to `/index.html` for cluster-down message - -## Server-Specific vs. Shared Configuration - -### Shared Hosts (Service Domains) -Examples: foo.zone, irregular.ninja, f3s.buetow.org - -- Same content/routing on both servers -- Both servers have TLS certs -- Include in `@acme_hosts` without guards -- Create with prefix loops for www/standby variants - -### Server-Specific Hosts (Server FQDNs) -Examples: blowfish.buetow.org, fishfinger.buetow.org - -- Different per server -- Each server has ONLY its own cert -- Include in `@acme_hosts` for routing -- **Must skip in template loops** -- Use dedicated server blocks and keypair lines - -### Pattern for Adding Server FQDNs - -1. **Routing**: Add to `@acme_hosts` (relayd needs routing rules) -2. **ACME loop**: Skip with `next if $host eq "$hostname.$domain"` -3. **TLS loop**: Skip with `next if $host eq 'blowfish.buetow.org' or $host eq 'fishfinger.buetow.org'` -4. **Server blocks**: Use existing dedicated "Current server's FQDN" block - -## Deployment Process - -```bash -rex httpd relayd # Deploy to both servers -``` - -Process: -1. Rex connects to both blowfish and fishfinger in parallel -2. For each server, processes templates with server-specific `$hostname` -3. Generates `/etc/httpd.conf` and `/etc/relayd.conf` -4. Writes files and restarts services via `on_change` handlers -5. Each server gets identical config structure but different hostname values - -## Monitoring System (Gogios) - -### Configuration -- Runs as user `_gogios` -- Config: `/etc/gogios.json` (generated from `etc/gogios.json.tpl`) -- Output: `/var/www/htdocs/buetow.org/self/gogios/index.html` -- State: `/var/run/gogios/state.json` -- Cron schedule: Every 5 minutes between 08:00-22:00 -- Check intervals: Independent from cron (e.g., TLS checks every 3600s) - -**Important**: Check intervals (`RunInterval`) are independent from cron schedule. A check with 3600s interval won't re-run just because cron triggered, it runs only when interval expires. - -### Template Pattern (gogios.json.tpl) - -**Dedicated server checks (lines 29-46)**: Bare hostnames only -```perl -<% for my $host (qw(fishfinger blowfish)) { -%> -"Check TLS Certificate <%= $host %>.buetow.org": { ... } -``` - -Creates checks for: -- `fishfinger.buetow.org` (bare hostname) -- `blowfish.buetow.org` (bare hostname) - -**Service domain checks (lines 47-66)**: All prefix variants -```perl -<% for my $host (@$acme_hosts) { - # Skip server hostnames - they have dedicated checks above without www/standby variants - next if $host eq 'blowfish.buetow.org' or $host eq 'fishfinger.buetow.org'; -%> -<% for my $prefix ('', 'standby.', 'www.') { -%> -``` - -Creates checks for: -- `foo.zone`, `standby.foo.zone`, `www.foo.zone` (all variants) -- **NOT** `www.blowfish.buetow.org`, `standby.blowfish.buetow.org` (no DNS records) - -### Why Skip Server Hostnames - -Server FQDNs like blowfish.buetow.org and fishfinger.buetow.org: -- Don't have DNS records for www/standby prefixes -- Already monitored via dedicated checks without prefixes -- Including them in `@acme_hosts` loop creates false alerts - -Without the skip, gogios generates checks for non-existent: -- `www.blowfish.buetow.org` → DNS failure -- `standby.blowfish.buetow.org` → DNS failure -- `www.fishfinger.buetow.org` → DNS failure -- `standby.fishfinger.buetow.org` → DNS failure - -Result: 12 false CRITICAL alerts (3 checks × 4 hostnames) - -## Configuration Testing - -Before deploying: -```bash -ssh rex@server "doas httpd -n" # Test httpd config syntax -ssh rex@server "doas relayd -n" # Test relayd config syntax -``` - -After deploying: -```bash -ssh rex@server "doas rcctl check httpd" -ssh rex@server "doas rcctl check relayd" -``` diff --git a/frontends/AGENTS.md b/frontends/AGENTS.md new file mode 100644 index 0000000..d989eca --- /dev/null +++ b/frontends/AGENTS.md @@ -0,0 +1,253 @@ +# Frontend Infrastructure Knowledge + +## Architecture Overview + +### Request Flow +``` +Internet → relayd (port 443) → routing decision → httpd (port 8080) or f3s cluster (port 80) +``` + +### Key Components + +**relayd** - Reverse proxy that: +- Terminates TLS on port 443 (IPv4 and IPv6) +- Routes requests based on Host header matching +- Has two backend pools: + - `` (127.0.0.1, ::1) - Routes to local httpd on port 8080 + - `` (192.168.2.120-122) - Routes to f3s k3s cluster on port 80 +- Falls back to f3s cluster when no explicit routing match exists + +**httpd** - OpenBSD httpd that: +- Listens on port 8080 (behind relayd) +- Listens on port 80 for ACME challenges and HTTP→HTTPS redirects +- Serves static content for various domains +- Has server-specific blocks for each server's own hostname + +**Rexfile** - Configuration management using Rex (Perl): +- Defines configuration arrays (`@acme_hosts`, `@f3s_hosts`, etc.) +- Templates use these arrays to generate httpd and relayd configs +- Deploys to both blowfish and fishfinger servers in parallel +- Each server receives templates processed with its own `$hostname` value + +## Configuration Arrays + +### @acme_hosts +Controls which hosts get: +- ACME certificate requests +- HTTP port 80 server blocks for ACME challenges +- Explicit routing rules in relayd to `` + +**Critical**: Hosts NOT in `@acme_hosts` will fall through to f3s cluster backends in relayd. + +### @f3s_hosts +Hosts served by the f3s k3s cluster: +- Get fallback page served by httpd +- Special routing rules in relayd to f3s backends + +### @prefixes +Array: `('', 'www.', 'standby.')` + +Used in loops to create hostname variants: +- `foo.zone` +- `www.foo.zone` +- `standby.foo.zone` + +## Template Processing + +Rex processes `.tpl` files using embedded Perl: + +```perl +<% ... -%> # Perl code (- suppresses trailing newline) +<%= $var %> # Print variable value +``` + +Templates are processed **per-server** with different values: +- `$hostname` = "blowfish" or "fishfinger" +- `$domain` = "buetow.org" +- `$hostname.$domain` = "blowfish.buetow.org" or "fishfinger.buetow.org" + +## Routing Configuration + +### Explicit Routing Rules (relayd.conf.tpl:45-50) + +```perl +<% for my $host (@$acme_hosts) { + next if grep { $_ eq $host } @$f3s_hosts; + for my $prefix (@prefixes) { -%> +match request header "Host" value "<%= $prefix.$host -%>" forward to +``` + +- Only hosts in `@acme_hosts` get explicit routing to `` +- Excludes f3s hosts (they have separate routing) +- Creates rules for all prefixes ('', 'www.', 'standby.') + +### Routing Logic + +**Routing is explicit, not implicit**: Just because httpd has a server block doesn't mean relayd will route to it. The routing decision happens in relayd based on: + +1. Explicit Host header match → route to specified backend +2. No match → fall through to default relay backends (f3s cluster first, then localhost) + +## TLS Certificate Management + +### Certificate Loading (relayd.conf.tpl:24-31) + +```perl +http protocol "https" { + <% for my $host (@$acme_hosts) { -%> + tls keypair <%= $host %> + tls keypair standby.<%= $host %> + <% } -%> + tls keypair <%= $hostname.'.'.$domain -%> +``` + +**Critical insight**: In multi-server deployments, each server only has its own TLS certificate. + +- blowfish has: `blowfish.buetow.org.crt` (NOT fishfinger's cert) +- fishfinger has: `fishfinger.buetow.org.crt` (NOT blowfish's cert) + +When the template runs on blowfish, it tries to load certs for ALL hosts in `@acme_hosts`. If fishfinger.buetow.org is in the array, relayd will fail to start because that cert doesn't exist on blowfish. + +**Solution pattern**: Skip server-specific hostnames in the loop, use dedicated keypair line: +```perl +<% for my $host (@$acme_hosts) { + next if $host eq 'blowfish.buetow.org' or $host eq 'fishfinger.buetow.org'; -%> +``` + +The line `tls keypair <%= $hostname.'.'.$domain -%>` loads the correct cert for each server. + +## Server Block Management + +### httpd.conf.tpl Patterns + +**ACME and redirect blocks (port 80)**: +```perl +<% for my $host (@$acme_hosts) { + next if $host eq "$hostname.$domain"; # Skip current server + for my $prefix (@prefixes) { -%> +server "<%= $prefix.$host %>" { + listen on * port 80 +``` + +**Why skip current server**: Each server has a dedicated "Current server's FQDN" block: + +```perl +server "<%= "$hostname.$domain" %>" { + listen on * port 80 + ... +} +``` + +Without the skip, adding server hostnames to `@acme_hosts` creates duplicate server blocks, causing httpd to fail with "server defined twice" error. + +### Content Serving Blocks (port 8080) + +Different patterns based on content type: +- **Gemtexter sites**: Serve from `/htdocs/gemtexter/` +- **Server self**: Serve from `/htdocs/buetow.org/self` +- **Special hosts**: Custom root paths (e.g., gogios, joern, dory) +- **f3s fallback**: Rewrite all to `/index.html` for cluster-down message + +## Server-Specific vs. Shared Configuration + +### Shared Hosts (Service Domains) +Examples: foo.zone, irregular.ninja, f3s.buetow.org + +- Same content/routing on both servers +- Both servers have TLS certs +- Include in `@acme_hosts` without guards +- Create with prefix loops for www/standby variants + +### Server-Specific Hosts (Server FQDNs) +Examples: blowfish.buetow.org, fishfinger.buetow.org + +- Different per server +- Each server has ONLY its own cert +- Include in `@acme_hosts` for routing +- **Must skip in template loops** +- Use dedicated server blocks and keypair lines + +### Pattern for Adding Server FQDNs + +1. **Routing**: Add to `@acme_hosts` (relayd needs routing rules) +2. **ACME loop**: Skip with `next if $host eq "$hostname.$domain"` +3. **TLS loop**: Skip with `next if $host eq 'blowfish.buetow.org' or $host eq 'fishfinger.buetow.org'` +4. **Server blocks**: Use existing dedicated "Current server's FQDN" block + +## Deployment Process + +```bash +rex httpd relayd # Deploy to both servers +``` + +Process: +1. Rex connects to both blowfish and fishfinger in parallel +2. For each server, processes templates with server-specific `$hostname` +3. Generates `/etc/httpd.conf` and `/etc/relayd.conf` +4. Writes files and restarts services via `on_change` handlers +5. Each server gets identical config structure but different hostname values + +## Monitoring System (Gogios) + +### Configuration +- Runs as user `_gogios` +- Config: `/etc/gogios.json` (generated from `etc/gogios.json.tpl`) +- Output: `/var/www/htdocs/buetow.org/self/gogios/index.html` +- State: `/var/run/gogios/state.json` +- Cron schedule: Every 5 minutes between 08:00-22:00 +- Check intervals: Independent from cron (e.g., TLS checks every 3600s) + +**Important**: Check intervals (`RunInterval`) are independent from cron schedule. A check with 3600s interval won't re-run just because cron triggered, it runs only when interval expires. + +### Template Pattern (gogios.json.tpl) + +**Dedicated server checks (lines 29-46)**: Bare hostnames only +```perl +<% for my $host (qw(fishfinger blowfish)) { -%> +"Check TLS Certificate <%= $host %>.buetow.org": { ... } +``` + +Creates checks for: +- `fishfinger.buetow.org` (bare hostname) +- `blowfish.buetow.org` (bare hostname) + +**Service domain checks (lines 47-66)**: All prefix variants +```perl +<% for my $host (@$acme_hosts) { + # Skip server hostnames - they have dedicated checks above without www/standby variants + next if $host eq 'blowfish.buetow.org' or $host eq 'fishfinger.buetow.org'; -%> +<% for my $prefix ('', 'standby.', 'www.') { -%> +``` + +Creates checks for: +- `foo.zone`, `standby.foo.zone`, `www.foo.zone` (all variants) +- **NOT** `www.blowfish.buetow.org`, `standby.blowfish.buetow.org` (no DNS records) + +### Why Skip Server Hostnames + +Server FQDNs like blowfish.buetow.org and fishfinger.buetow.org: +- Don't have DNS records for www/standby prefixes +- Already monitored via dedicated checks without prefixes +- Including them in `@acme_hosts` loop creates false alerts + +Without the skip, gogios generates checks for non-existent: +- `www.blowfish.buetow.org` → DNS failure +- `standby.blowfish.buetow.org` → DNS failure +- `www.fishfinger.buetow.org` → DNS failure +- `standby.fishfinger.buetow.org` → DNS failure + +Result: 12 false CRITICAL alerts (3 checks × 4 hostnames) + +## Configuration Testing + +Before deploying: +```bash +ssh rex@server "doas httpd -n" # Test httpd config syntax +ssh rex@server "doas relayd -n" # Test relayd config syntax +``` + +After deploying: +```bash +ssh rex@server "doas rcctl check httpd" +ssh rex@server "doas rcctl check relayd" +``` -- cgit v1.2.3