path: root/f3s/docs/freebsd-relayd-lan-access.md

# FreeBSD relayd Configuration for LAN Access to k3s Services

This document describes how to configure FreeBSD relayd on the CARP VIP to provide LAN access to k3s services with TLS termination.

## Architecture

```
LAN Client
   ↓
192.168.1.138:443 (CARP VIP - f0/f1)
   ↓
relayd (TLS termination)
   ↓
k3s Traefik NodePort :80 (r0/r1/r2)
   ↓
Service (Navidrome, etc.)
```

## Current CARP Setup

- **CARP VIP**: 192.168.1.138 (vhid 1)
- **Master**: f0 (192.168.1.130) - advskew 0
- **Backup**: f1 (192.168.1.131) - advskew 100
- **Existing Service**: stunnel on port 2323 (NFS-over-TLS)

## Installation

### 1. Install relayd

On both f0 and f1:

```bash
doas pkg install relayd
```

### 2. Configure relayd

Create `/usr/local/etc/relayd.conf` on both f0 and f1:

```
# k3s nodes backend table
table <k3s_nodes> { 192.168.1.120 192.168.1.121 192.168.1.122 }

# HTTP protocol (pass-through to Traefik)
http protocol "lan_http" {
    # Pass all requests to backend
    pass request quick
    pass response quick
}

# HTTPS protocol with TLS termination
http protocol "lan_https" {
    # TLS configuration
    tls keypair "f3s.lan.buetow.org"
    
    # Pass decrypted HTTP to Traefik
    pass request quick
    pass response quick
}

# HTTP relay (port 80)
relay "lan_http" {
    listen on 192.168.1.138 port 80
    protocol "lan_http"
    forward to <k3s_nodes> port 80 check tcp
}

# HTTPS relay (port 443) with TLS
relay "lan_https" {
    listen on 192.168.1.138 port 443 tls
    protocol "lan_https"
    forward to <k3s_nodes> port 80 check tcp
}
```

### 3. Install TLS Certificates

Certificates are generated by cert-manager in k3s and must be exported to FreeBSD.

#### Export from k3s

On a k3s node or workstation with kubectl access:

```bash
cd /home/paul/git/conf/f3s/cert-manager
just export-certs
```

This creates:
- `/tmp/f3s-lan-cert.pem` - Certificate
- `/tmp/f3s-lan-key.pem` - Private key

#### Copy to FreeBSD Hosts

```bash
# Copy to f0
scp /tmp/f3s-lan-cert.pem paul@192.168.1.130:/tmp/
scp /tmp/f3s-lan-key.pem paul@192.168.1.130:/tmp/

# Copy to f1
scp /tmp/f3s-lan-cert.pem paul@192.168.1.131:/tmp/
scp /tmp/f3s-lan-key.pem paul@192.168.1.131:/tmp/
```

#### Install on FreeBSD

On both f0 and f1:

```bash
# Create certificate directory
doas mkdir -p /usr/local/etc/ssl/relayd

# Move certificates
doas mv /tmp/f3s-lan-cert.pem /usr/local/etc/ssl/relayd/f3s.lan.buetow.org.crt
doas mv /tmp/f3s-lan-key.pem /usr/local/etc/ssl/relayd/f3s.lan.buetow.org.key

# Combine cert and key for relayd (relayd expects combined PEM)
doas sh -c 'cat /usr/local/etc/ssl/relayd/f3s.lan.buetow.org.crt /usr/local/etc/ssl/relayd/f3s.lan.buetow.org.key > /usr/local/etc/ssl/relayd/f3s.lan.buetow.org.pem'

# Set permissions
doas chmod 600 /usr/local/etc/ssl/relayd/*
doas chown root:wheel /usr/local/etc/ssl/relayd/*
```

### 4. Enable and Start relayd

On both f0 and f1:

```bash
# Enable relayd service
doas sysrc relayd_enable=YES

# Start relayd
doas service relayd start

# Check status
doas service relayd status
```

### 5. Verify relayd is Running

```bash
# Check relayd process
doas ps aux | grep relayd

# Check listening ports on CARP VIP
doas sockstat -4 -l | grep 192.168.1.138
```

Expected output:
```
stunnel  stunnel     1546 8   tcp4   192.168.1.138:2323    *:*
relayd   relayd      2101 3   tcp4   192.168.1.138:80      *:*
relayd   relayd      2101 4   tcp4   192.168.1.138:443     *:*
```

## Testing

### Test HTTP Access

```bash
curl -v http://navidrome.f3s.lan.buetow.org
```

### Test HTTPS Access

Without CA trust (expect certificate warning):
```bash
curl -k https://navidrome.f3s.lan.buetow.org
```

With CA trust installed:
```bash
curl https://navidrome.f3s.lan.buetow.org
```

### Test CARP Failover

1. On f0 (current master), disable the interface:
   ```bash
   doas ifconfig re0 down
   ```

2. Check CARP status on f1 (should become MASTER):
   ```bash
   ifconfig re0 | grep carp
   ```

3. Test access (should still work):
   ```bash
   curl https://navidrome.f3s.lan.buetow.org
   ```

4. Re-enable f0:
   ```bash
   doas ifconfig re0 up
   ```

## Certificate Renewal

Certificates from cert-manager are renewed automatically every 75 days (15 days before 90-day expiration).

After renewal:

1. Export new certificates from k3s
2. Copy to f0 and f1
3. Reload relayd on both hosts:
   ```bash
   doas service relayd reload
   ```

## Troubleshooting

### Check relayd Configuration

```bash
doas relayd -n
```

### View relayd Logs

```bash
# Real-time logs
doas tail -f /var/log/daemon.log | grep relayd

# Recent logs
doas grep relayd /var/log/daemon.log | tail -50
```

### Check Backend Health

relayd performs TCP health checks on k3s nodes. View status:

```bash
doas relayctl show summary
doas relayctl show hosts
```

### Connection Test from FreeBSD

```bash
# Test Traefik on k3s nodes directly
curl -H "Host: navidrome.f3s.lan.buetow.org" http://192.168.1.120
curl -H "Host: navidrome.f3s.lan.buetow.org" http://192.168.1.121
curl -H "Host: navidrome.f3s.lan.buetow.org" http://192.168.1.122
```

### Common Issues

**relayd fails to start:**
- Check configuration: `doas relayd -n`
- Verify certificates exist and have correct permissions
- Check `/var/log/daemon.log` for errors

**Certificate errors:**
- Ensure certificate and key are combined in one PEM file
- Verify filename matches `tls keypair` directive in config
- Check certificate validity: `openssl x509 -in /usr/local/etc/ssl/relayd/f3s.lan.buetow.org.pem -text -noout`

**No response from services:**
- Check k3s Traefik is running: `kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik`
- Verify k3s nodes are reachable from FreeBSD
- Check backend health: `doas relayctl show hosts`

## Security Considerations

- Certificates are self-signed; clients must trust the CA
- relayd runs as unprivileged user after binding to ports
- Only ports 80 and 443 are exposed on CARP VIP for HTTP/HTTPS
- Existing stunnel NFS service (port 2323) is unaffected

## Adding More Services

To add LAN access to another service:

1. Create ingress in service's helm chart with `host: service.f3s.lan.buetow.org`
2. Add DNS entry: `192.168.1.138  service.f3s.lan.buetow.org`
3. No relayd changes needed - it forwards all traffic to Traefik

## References

- [relayd(8) man page](https://man.freebsd.org/cgi/man.cgi?query=relayd)
- [CARP Configuration on FreeBSD](https://docs.freebsd.org/en/books/handbook/advanced-networking/#carp)
- [cert-manager Documentation](https://cert-manager.io/docs/)