1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
|
# Backups and Local-Path Storage
## AWS S3 Glacier Deep Archive Backups
Encrypted incremental ZFS snapshots from `zdata` pool backed up daily to **AWS S3 Glacier Deep Archive** via cron. Scripts adapted from FreeBSD Home NAS setup. Also performs periodic zpool scrubbing.
The **`zusb` quarterly backup** (`/opt/snonux/bin/backup/backup`, which travels on the `zusb` pool — see [USB Key Mounting](usb-keys.md) → "Removable backup pool (`zusb`)") also uploads to the same S3 Glacier Deep Archive bucket (`s3://org-buetow-backup/<host>/`). Both workflows need the AWS CLI on the host that runs them.
### AWS CLI setup on a FreeBSD host
Install the `awscli` package. The Python flavor depends on the FreeBSD version:
```sh
# FreeBSD 14.x (e.g. t450)
# py39-awscli-1.29.81
# FreeBSD 15.x (e.g. f1, f-hosts)
# py312-awscli-1.42.44
sudo pkg install -y py312-awscli # adjust py3XX to what pkg search -q awscli shows
```
Wire `/root/.aws` (the backup script runs `aws` as root). The **credentials ride on the `zusb` pool** at `/opt/snonux/secrets/aws.credentials` (INI: `[default]` + `aws_access_key_id` + `aws_secret_access_key`), so on any host that has `zusb` imported (i.e. `/opt` mounted) you only need the config file and a symlink — the secret is not duplicated on host disks and is not in git:
```sh
sudo mkdir -p /root/.aws && sudo chmod 700 /root/.aws
printf '[default]\nregion = eu-central-1\n' | sudo tee /root/.aws/config >/dev/null
sudo chmod 600 /root/.aws/config
sudo ln -sf /opt/snonux/secrets/aws.credentials /root/.aws/credentials
```
Because the credentials are a symlink into `/opt` (`zusb/data/opt`), `aws` only resolves them while `zusb` is imported on that host. That is fine for the quarterly backup workflow (load `zusb` → run backup → export `zusb`); it is **not** suitable for the `zdata` daily-cron S3 backup on an f-host that does not normally have `zusb` imported — that host would need its own credentials copy (out of scope here).
Verify (read-only):
```sh
aws --version
aws sts get-caller-identity # expect Arn arn:aws:iam::634617747016:user/org-buetow-backup-user
aws s3 ls s3://org-buetow-backup/ # expect the per-host prefixes (e.g. t450/)
```
Installed 2026-07-20 on f1 (`py312-awscli-1.42.44`), matching the t450 setup (`py39-awscli-1.29.81`, same `/root/.aws/config` region and the same credentials symlink).
## Local-Path Storage for SQLite Workloads
Some k3s workloads use `local-path` (k3s default storageClass) instead of NFS for
their data volumes. This is appropriate when:
- The application uses SQLite: NFS file-lock semantics cause `fcntl()` races on
pod restarts, and `Recreate` strategy only reduces (not eliminates) the risk.
- Cache-heavy workloads: NFS over stunnel adds TLS round-trip latency to every
cache read. Navidrome's image/background cache init took ~19s over NFS; it
takes ~25ms from local disk.
**Trade-off**: a local-path PV lives on one specific node. If that node is down,
the pod reschedules elsewhere but finds no data volume — it starts with an empty DB,
losing play history, scrobble queue, etc. For a home server this is acceptable.
The deployment must pin the pod to the same node via `nodeSelector` so the local
PV is always reachable.
### Workloads using local-path
| App | Node | Path on node |
|-----|------|--------------|
| navidrome `/data` (DB + cache) | r1 | `/var/lib/rancher/k3s/storage/pvc-*_services_navidrome-data-pvc` |
### Migrating NFS hostPath → local-path
1. Disable ArgoCD auto-sync: `kubectl patch application <app> -n cicd --type=json -p='[{"op":"replace","path":"/spec/syncPolicy","value":{}}]'`
2. Scale deployment to 0: `kubectl scale deployment <app> -n services --replicas=0`
3. Delete old PVC and static PV.
4. Create new PVC with `storageClassName: local-path`.
5. Create a migration pod pinned to the target node that mounts both the NFS hostPath
(source) and the new PVC (target); copy data with `cp -av /src/. /dst/`.
6. Delete migration pod, apply updated deployment (with `nodeSelector`), scale back up.
7. Re-enable ArgoCD auto-sync and push manifests to git; push to in-cluster git-server
(`git push r0 master`) so ArgoCD picks up the new storageClass spec.
|
