pi-swarm/docs/PLAN.md

Rebuild plan & decisions log

Where we started (2026-09-19)

• Old cluster: Docker Swarm (manager planck001) with Portainer, Traefik, and what appears to be CapRover remnants. Broken: workers stuck in "pending" because the manager's IP changed (workers dial stale 192.168.1.173:2377).
• Decision: wipe everything and rebuild clean rather than repair.

Hardware

• 20× Raspberry Pi 4 Model B Rev 1.4, 8GB RAM, 4 cores
• Boot: 238.5GB USB SSD on every node (no SD cards installed)
• Bootloader EEPROM: 2023/01/11 on 19 nodes; planck019 has Feb 2021 (update it)
• All nodes healthy: temps 43–52°C, disks ~4% used

Network

• Ubiquiti UDM Pro, 2 switches, all nodes on the same router
• IPs currently drift across 192.168.0/1/2.x subnets via DHCP — planck001 sits on 192.168.30.x (different subnet, cause unknown, doesn't matter: we're rebuilding)
• Decision: dedicated static addressing for the cluster via UniFi DHCP reservations (MAC table in docs/MACS.txt)

Storage

• Synology NAS on same LAN — shared storage target (NFS) for cluster workloads

Rebuild architecture (decided 2026-09-19)

Fully unattended network reimage, no per-node physical access:

• Synology (192.168.1.157) hosts three services (all auto-start, see configs/):
  • planck-tftp.service — custom python TFTP server on :69 serving /volume1/plancknetboot/tftp/ (firmware, kernel, installer initramfs)
  • planck-netboot.service — python HTTP helper on :8000 (GET rootfs.tar template, GET /events/<name> progress log, POST = upload)
  • NFS read-only export of the share for future use
• Installer initramfs (configs/installer-init.sh): busybox + sfdisk + e2fsprogs. Identifies the node by CPU serial (map in configs/hostmap), wipes /dev/sda, downloads the Debian 13 Raspberry Pi OS Lite template over HTTP, installs it, sets hostname/ssh keys/user, POSTs pieeprom-revert.upd/.sig to tftp/<serial>/, reboots.
• Bootloader flow: nodes' EEPROM is flashed (via staged pieeprom.upd from the old OS) to install-mode: BOOT_ORDER=0xf142 (network first), TFTP_IP=192.168.1.157, TFTP_PREFIX=1 (per-serial dirs), ENABLE_SELF_UPDATE=1. After install, the TFTP-served bootloader_update=1 config + <serial>/pieeprom.upd reverts the node to BOOT_ORDER=0xf14 (USB first). Fresh OS boots from SSD; firstboot service tidies up.
• Reinstall a node anytime: stage install-mode EEPROM again + reboot (delete tftp/<serial>/pieeprom.* on the Synology first if it exists).
• Node password: see configs/node-password.txt (gitignored). SSH is key-based; password exists for console/sudo recovery.
• UniFi MFA blocks API access; DHCP reservations deferred — mDNS (planck0NN.local) works across subnets today.

Phases

• Phase 0 — Access + inventory (SSH keys on all 20, docs/)
• Phase 1 — Network design: static IPs / reservations in UniFi
• Phase 2 — Reimage all 20 nodes with fresh Raspberry Pi OS
• Phase 3 — Cluster runtime install (candidates: k3s vs Docker Swarm)
• Phase 4 — Storage integration with Synology (NFS CSI)
• Phase 5 — Apps: monitoring, Portainer/UI, whatever the lab is for

Open decisions

1. Reimage method — options: a. PXE/network boot: EEPROM boot-order set to netboot; DHCP next-server points at a TFTP/HTTP installer. Zero-touch per node, reusable forever. Installer server can run on the Synology (Docker) or a laptop. b. SD-card bootstrap installer: flash one SD image, insert per node, it wipes + installs the USB SSD, removes itself. Simple, physical walk. c. Pull SSDs, image on a PC with Raspberry Pi Imager ×20. Most manual.
2. Cluster runtime — k3s (lighter, k8s-compatible, we manage it with automation) vs Docker Swarm + Portainer (simplest).