Rebuild complete: all 20 nodes verified, planck020 EEPROM fixed, postmortem notes

docs/PLAN.md

@@ -58,7 +58,45 @@ Fully unattended network reimage, no per-node physical access:
 - UniFi MFA blocks API access; DHCP reservations deferred — mDNS
   (`planck0NN.local`) works across subnets today.
 
-## Current debug state (paused here, swap to stronger model)
+## REBUILD COMPLETE (2026-09-19)
+
+All 20 nodes re-imaged with Raspberry Pi OS Lite (Debian 13 trixie), via
+pull-and-image on the laptop (`scripts/flash-one.sh`, one NVMe stick at a
+time). Every node verified: correct hostname, SSH key auth, passwordless
+sudo, 235G root filesystem, timezone America/Los_Angeles.
+
+- Node password: `configs/node-password.txt` (gitignored)
+- planck019 + planck020 boot chips updated to 2026-05-17 firmware
+- planck020 boot chip cleaned of the netboot-test config (BOOT_ORDER=0xf14,
+  no TFTP settings)
+- Other 18 nodes: 2023-01-11 firmware, fine; 2026 firmware is staged on
+  their boot partitions and will apply on their next natural reboot
+- The netboot/TFTP experiment is retired: planck-tftp service on the
+  Synology is stopped+disabled; planck-netboot (HTTP helper, :8000) still
+  running and harmless
+
+### Postmortem notes (for future reflashes)
+
+1. The OS image ships a locked `pi` user (uid 1000). Always remove or
+   rename it before adding your own user, or sudo breaks silently.
+2. The desktop auto-mounter (udisks2) grabs partitions mid-flash; it also
+   reactivates itself when stopped. Mask it (`systemctl mask udisks2`)
+   during flashing sessions.
+3. Stick device names bounce between sda/sdb — always look up the device
+   by size/model, never assume.
+4. One stick locked up its JMicron USB bridge mid-write (e2fsck hung, 0
+   IOPS, kernel I/O errors). Unplug/replug fixed it. If a stick hangs the
+   flash, reseat it first, suspect the stick second.
+5. Boot-chip updates (EEPROM) have a timestamp guard: a staged update
+   older-or-equal to what's already flashed is silently ignored. Also,
+   network-delivered EEPROM updates change the FIRMWARE but preserve the
+   CONFIG. `bootloader_update=1` in config.txt is required for the
+   bootloader to look for updates on the boot medium.
+6. If a node ever 7-blinks its green LED: kernel not found — check for
+   leftover `recovery.bin`/`pieeprom.upd`/`pieeprom.sig` on the boot
+   partition (staged-but-rejected updates block boot).
+
+## Old debug state (superseded)
 
 First live test on planck020 (EEPROM install-mode staged, works — confirmed
 via `vcgencmd bootloader_config` showing BOOT_ORDER=0xf142, TFTP_IP,