# Remote in-place NetBSD install on a Raspberry Pi 3B+ (no SD removal) Runbook + scripts to replace **Rocky Linux 9** with **NetBSD 10.1** on an f3s Raspberry Pi **3 Model B+**, entirely over SSH, **without pulling the microSD card**. Proven on `pi0` (2026-07-03); this document is written so the same can be done to `pi1` (or a rebuild of `pi0`) by changing two values. > **TL;DR of the hard-won lesson:** `kexec` is compiled out of the Rocky RPi > kernel and the systemd shutdown-pivot does not fire on these boxes, so the > only remote RAM-flasher that works is one loaded by the **Pi firmware** as an > `initramfs` via a one-shot `config.txt`. See [Why the obvious methods > fail](#why-the-obvious-methods-fail). --- ## Contents - [Outcome / definition of done](#outcome--definition-of-done) - [Hardware & facts](#hardware--facts) - [Why the obvious methods fail](#why-the-obvious-methods-fail) - [How it works (the method that does)](#how-it-works-the-method-that-does) - [Prerequisites](#prerequisites) - [Stage A — bake the golden image (on `earth`)](#stage-a--bake-the-golden-image-on-earth) - [Stage B — remote in-place flash (on the Pi)](#stage-b--remote-in-place-flash-on-the-pi) - [Doing pi1 specifically](#doing-pi1-specifically) - [Verification](#verification) - [Rollback / recovery](#rollback--recovery) - [Troubleshooting & gotchas](#troubleshooting--gotchas) - [Cleanup](#cleanup) - [Post-install (deferred)](#post-install-deferred) - [File manifest](#file-manifest) --- ## Outcome / definition of done `ssh paul@` lands on a **NetBSD 10.1 (GENERIC64) evbarm/aarch64** system: - login as `paul` via **SSH key** (member of `wheel`); password auth also on as a backup - hostname correct (`piN.lan.buetow.org`) - **static IP on `mue0`** (the 3B+ onboard LAN78xx NIC in NetBSD), default route + DNS working - root filesystem auto-resized to the whole card on first boot ## Hardware & facts | Item | pi0 | pi1 | |------|-----|-----| | Board | Raspberry Pi 3 Model B+ | Raspberry Pi 3 Model B+ | | Was running | Rocky Linux 9.7 aarch64 | Rocky Linux 9.x aarch64 | | LAN IP | 192.168.1.125 | **192.168.1.126** | | Hostname | pi0.lan.buetow.org | **pi1.lan.buetow.org** | | WireGuard | 192.168.2.203 | 192.168.2.204 | | RAM | 909 MiB | 909 MiB | | SD | `/dev/mmcblk0` (30 GB): p1 `/boot` vfat, p2 swap, p3 `/` ext4 | same | | NIC (Linux → NetBSD) | `lan78xx` → **`mue0`** | `lan78xx` → **`mue0`** | | Gateway / DNS | 192.168.1.1 | 192.168.1.1 | Confirmed on the RPi Rocky kernel (`6.1.31-v8.1.el9.altarch`): `CONFIG_KEXEC` disabled, `CONFIG_KEXEC_FILE` unset; **boots with no initramfs** (kernel mounts the ext4 root directly); `ext4`, `vfat`, `nls_cp437`, `nls_ascii`, `mmc_block`, `sdhci`, `tmpfs` are all **built-in** (`=y`) — so a flasher initramfs needs no extra modules to mount `/boot`, the root, or a tmpfs. NetBSD base image used: **NetBSD 10.1 evbarm-aarch64 gzimg** `https://cdn.netbsd.org/pub/NetBSD/NetBSD-10.1/evbarm-aarch64/binary/gzimg/arm64.img.gz` (GPT: EFI System partition + NetBSD FFS root; boots RPi 3/4/5). ## Why the obvious methods fail Three RAM-flasher mechanisms were tried on `pi0`. Only the third works here: 1. **`kexec` into a RAM flasher** (what most guides assume). ❌ Impossible: `kexec_load(2)` returns `ENOSYS` and `kexec_file_load(2)` is not built (`CONFIG_KEXEC_FILE` unset). `kexec-tools` is installed but the syscalls are not in the kernel. 2. **systemd shutdown-pivot** (`/run/initramfs/shutdown` switch-root at reboot). ❌ Does **not fire** on these boxes — proven with a 3-channel evidence probe over several test reboots; `systemd-shutdown` never switch-roots into `/run/initramfs` here (the box was not booted from an initrd, and `dracut-shutdown` is a no-op because `.need_shutdown` is never created). *Also_ a red herring along the way: mounting a **separate tmpfs** at `/run/initramfs` gets unmounted during shutdown — always populate it as a plain dir on `/run` if you ever revisit this. 3. **Pi-firmware `initramfs`** (this runbook). ✅ Works. The VideoCore firmware loads a small dracut initramfs right after the kernel; a dracut *pre-mount* hook runs in RAM before the real root is mounted and does the flash. ## How it works (the method that does) ``` EARTH (x86_64 laptop) piN (RPi 3B+, Rocky) ───────────────────── ───────────────────── Stage A: bake golden image Stage B: remote flash qemu-system-aarch64 -M virt (TCG) 1. build flasher.img (dracut, boots stock NetBSD arm64.img pre-mount hook) -> /boot -> configure headless (sshd, 2. arm: write /boot/config.txt static mue0, user+key, pw) = "initramfs flasher.img -> sync + poweroff followkernel" + trigger gzip -> netbsd-piN-golden.img.gz file /boot/netbsd-flash-mode │ 3. systemctl reboot └── scp to piN:/home/paul/ ───────────► ── firmware loads flasher.img ──┐ ▼ dracut pre-mount hook (95netbsdflash): - DISARM: rm /boot/config.txt + trigger (so any later failure self-heals to Rocky) - mount ext4 root ro, copy golden gz -> tmpfs (RAM) - dryrun: gunzip|wc -c (validate, reboot to Rocky) - real: gunzip|dd of=/dev/mmcblk0 (reboot to NetBSD) ▼ NetBSD 10.1 boots headless, static .12x, sshd ▼ ssh paul@ ✅ ACCEPTANCE ``` Key safety property: the hook **removes `config.txt` before touching anything else**, so the flasher is strictly one-shot and any failure *after the hook starts* leaves a card that boots normally (Rocky, or — once flashed — NetBSD). The **dry-run** proves the entire boot+stage+decompress path non-destructively before the real `dd`. ## Prerequisites On **earth** (the flashing workstation): - `qemu-system-aarch64`, `qemu-img`, `edk2` AAVMF firmware, `expect`, passwordless `sudo` (`sudo dnf install -y qemu-system-aarch64 qemu-img edk2-aarch64 expect`). - Note: earth is x86_64, so the aarch64 guest runs under **TCG emulation** (slow but fine — a bake is a handful of minutes). - Your SSH public key (`~/.ssh/id_rsa.pub`) — it gets baked into the image. On the **Pi**: - Reachable over SSH as `paul` with **passwordless sudo** (the f3s default). - `dracut` present (it is on Rocky), `/boot` is the vfat firmware partition, ~500 MB free on `/` to stage the image. - Its twin Pi still serving the shared role (so downtime is a non-event). ## Stage A — bake the golden image (on `earth`) Scripts: [`bake/`](bake). Editing the NetBSD FFS root from Linux is unsafe, so we configure the image **from inside a real NetBSD** running under qemu. 1. **Edit the two per-host values** at the top of [`bake/setup.sh`](bake/setup.sh): `HOSTNAME` and `IPADDR` (see [Doing pi1](#doing-pi1-specifically)). 2. Run the orchestrator: ```bash cd f3s/pi-netbsd/bake ./bake-golden.sh piN # e.g. ./bake-golden.sh pi1 ``` It downloads `arm64.img.gz` (if absent), serves `setup.sh` over a localhost HTTP server, boots the image in qemu, blind-drives the console via `config.exp` to fetch+run `setup.sh`, verifies the on-disk result, then `sync`+powers off and produces `netbsd-piN-golden.img.gz`. 3. A random backup password is generated and printed (also saved to `piN-cred.txt`). The **SSH key is the primary login**; change the password after install. What `setup.sh` configures inside the image: `sshd=YES`, `hostname`, `ifconfig_mue0="inet netmask 0xffffff00"`, `defaultroute`, `dhcpcd=NO`, `/etc/resolv.conf`, user `paul` (+`wheel`) with your `authorized_keys` and an argon2id password (root too), `sshd_config` pubkey+password auth, and an `/etc/rc.local` fallback that puts the static IP on the first real ethernet interface if `mue0` is ever named differently. > **The qemu-automation gotchas** (already handled in the scripts, documented so > you understand them): add a **virtio-rng** device or NetBSD stalls on entropy > and never generates ssh host keys; run `expect` under **`LC_ALL=C`** or Tcl > chokes on the serial control bytes; **blind-drive** the login/commands with > fixed sleeps and judge success by markers written to the log rather than > matching the flaky console; and **`sync` then wait for a clean poweroff** — > killing qemu before the FFS is flushed corrupts `rc.conf`/`pwd.db`. ## Stage B — remote in-place flash (on the Pi) Scripts: [`flash/`](flash). Copy this whole `pi-netbsd/` tree (or at least `flash/`) to the Pi first, e.g. `scp -r f3s/pi-netbsd paul@piN.lan.buetow.org:`. 1. **Stage the golden image on the Pi** (on its ext4 root): ```bash scp netbsd-piN-golden.img.gz paul@piN.lan.buetow.org:/home/paul/ ``` The flasher hook globs for **`/home/paul/netbsd-*-golden.img.gz`** (or under `/root/`), so any `netbsd--golden.img.gz` name works — no rename needed. Verify integrity: `gzip -t` and compare `sha256sum` against earth. 2. **Build the flasher initramfs** (module name is `netbsdflash`, no numeric prefix): ```bash ssh paul@piN.lan.buetow.org 'sudo /home/paul/pi-netbsd/flash/build-flasher.sh' ``` Produces `/boot/flasher.img` (~38 MB) and prints an `lsinitrd` sanity check. 3. **Dry-run first (non-destructive):** ```bash ssh paul@piN.lan.buetow.org ' sudo rm -f /boot/flash-evidence.txt printf "initramfs flasher.img followkernel\n" | sudo tee /boot/config.txt printf "dryrun\n" | sudo tee /boot/netbsd-flash-mode sync; sudo systemctl reboot' ``` The Pi goes down ~90 s (staging + decompress), then returns as **Rocky**. Read the proof: ```bash ssh paul@piN.lan.buetow.org 'sudo cat /boot/flash-evidence.txt' # EXPECT: # FLASHER-RAN mode=dryrun up=... # DRYRUN-RESULT bytes=1586495488 rc=0 ``` `bytes=1586495488 rc=0` and `config.txt` gone ⇒ the whole path works. **Do not proceed to the real flash unless the dry-run shows this.** 4. **Real flash (destructive, irreversible):** ```bash ssh paul@piN.lan.buetow.org ' sudo rm -f /boot/flash-evidence.txt printf "initramfs flasher.img followkernel\n" | sudo tee /boot/config.txt printf "real\n" | sudo tee /boot/netbsd-flash-mode sync; sudo systemctl reboot' ``` The Pi stages to RAM, `dd`s the 1.59 GB image onto `/dev/mmcblk0`, then reboots into **NetBSD**. Allow several minutes (SD write + first boot + root resize). ## Doing pi1 specifically Only two things differ from pi0. In [`bake/setup.sh`](bake/setup.sh): ```sh HOSTNAME="pi1.lan.buetow.org" IPADDR="192.168.1.126" ``` Everything else is identical: same board, same NIC (`mue0`), same gateway/DNS, same flasher. WireGuard peer for pi1 is `192.168.2.204` (configured later, out of scope here). Then run Stage A (`./bake-golden.sh pi1`) and Stage B as above against `pi1.lan.buetow.org` / `192.168.1.126`. > Keep pi0 (or the other twin) up while flashing pi1 so the static > `f3s.buetow.org` backend stays served. ## Verification ```bash ssh -i ~/.ssh/id_rsa -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null paul@192.168.1.126 ' uname -a # NetBSD piN.lan.buetow.org 10.1 ... evbarm hostname # piN.lan.buetow.org id # uid=1000(paul) ... groups ... wheel ifconfig mue0 # inet 192.168.1.126 netstat -rn -f inet | grep default # default 192.168.1.1 ... mue0 host cdn.netbsd.org # DNS resolves df -h / # root grown to ~29G' ``` NetBSD regenerates its own SSH **host keys**, so your `known_hosts` entry for the Pi will change — remove the old line or use the relaxed flags above. ## Rollback / recovery - **Before the real flash:** nothing is destroyed. Any failure once the hook has started leaves the card booting normally (the hook removes `config.txt` first). If the initramfs never runs the hook (rare — a malformed `flasher.img`), the Pi boot-loops on the missing/failed initramfs until you fix it physically: pull the SD, delete `config.txt`, reinsert. Mitigation is the dry-run, which proves the initramfs boots. - **After the real flash:** the card is NetBSD. To go back to Rocky, reflash the Rocky image (keep a known-good one handy) — physically or by the same method in reverse. **Keep a Rocky SD image before starting** as the ultimate fallback (physical access is "inconvenient but possible" for these boxes). ## Troubleshooting & gotchas - **Dry-run returns fast (~20 s) with no `flash-evidence.txt`:** the initramfs did not run the hook — check `/boot/config.txt` really says `initramfs flasher.img followkernel`, that `/boot/flasher.img` exists, and that the trigger file `/boot/netbsd-flash-mode` contains exactly `dryrun`. - **`dracut module 'netbsdflash' cannot be found`:** use `--add netbsdflash` (module name = directory name **without** the `95` prefix). - **`bytes=` is not `1586495488` or `rc!=0`:** the staged/decompressed image is bad — re-check the `scp`/`sha256sum` of the golden gz on the Pi. - **NetBSD boots but is unreachable:** the NIC came up under a name other than `mue0`; the `rc.local` fallback should still put `.12x` on the first real ethernet iface — attach a console (HDMI/serial) to inspect if needed. - **qemu bake stalls / login never proceeds:** see the Stage A gotchas box (virtio-rng, `LC_ALL=C`, blind-drive, sync-before-poweroff). ## Cleanup On the Pi (once NetBSD is confirmed, these are on the wiped card anyway for a real flash; relevant only if you leave a Pi on Rocky after dry-runs): ```bash sudo rm -f /boot/config.txt /boot/netbsd-flash-mode /boot/flash-evidence.txt \ /boot/flasher.img /home/paul/netbsd-pi0-golden.img.gz # optional: restore the untouched initramfs backup sudo mv /boot/initramfs-*.img.orig /boot/initramfs-$(uname -r).img # if present ``` On earth: stop any leftover `python3 -m http.server` from the bake; the `*.img`/`*.img.gz` work files can be deleted or kept for the next Pi. ## Post-install (deferred) Base OS + networking + SSH is the whole scope of this runbook. Re-provisioning the Pi's role on NetBSD is separate work: - **lighttpd static site** (`f3s.buetow.org`): `pkgin install lighttpd`, recreate `/var/www` + the Host-based vhost config. relayd backends keep the same IPs, so the OpenBSD frontends need no change. - **DTail `dserver`**: needs a **`netbsd/arm64`** build (the Pis previously ran `linux/arm64`). - **WireGuard** peer, monitoring agents, etc. ## File manifest ``` f3s/pi-netbsd/ ├── README.md # this runbook ├── bake/ │ ├── bake-golden.sh # earth: orchestrate the qemu bake │ ├── setup.sh # image customization (EDIT HOSTNAME/IPADDR) │ └── config.exp # earth: expect driver for the qemu console └── flash/ ├── build-flasher.sh # Pi: build /boot/flasher.img via dracut └── 95netbsdflash/ # dracut module (name: "netbsdflash") ├── module-setup.sh └── flash.sh # the pre-mount flasher hook ```