adamcarr/pi-swarm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
3df157a774b11b5dced85d13c326b7c90edc2b0e
pi-swarm/docs/PLAN.md

9.8 KiB
Raw Blame History

Rebuild plan & decisions log

Where we started (2026-09-19)

  • Old cluster: Docker Swarm (manager planck001) with Portainer, Traefik, and what appears to be CapRover remnants. Broken: workers stuck in "pending" because the manager's IP changed (workers dial stale 192.168.1.173:2377).
  • Decision: wipe everything and rebuild clean rather than repair.

Hardware

  • 20× Raspberry Pi 4 Model B Rev 1.4, 8GB RAM, 4 cores
  • Boot: 238.5GB USB SSD on every node (no SD cards installed)
  • Bootloader EEPROM: 2023/01/11 on 19 nodes; planck019 has Feb 2021 (update it)
  • All nodes healthy: temps 43–52°C, disks ~4% used

Network

  • Ubiquiti UDM Pro, 2 switches, all nodes on the same router
  • IPs currently drift across 192.168.0/1/2.x subnets via DHCP — planck001 sits on 192.168.30.x (different subnet, cause unknown, doesn't matter: we're rebuilding)
  • Decision: dedicated static addressing for the cluster via UniFi DHCP reservations (MAC table in docs/MACS.txt)

Storage

  • Synology NAS on same LAN — shared storage target (NFS) for cluster workloads

Rebuild architecture (decided 2026-09-19)

Fully unattended network reimage, no per-node physical access:

  • Synology (192.168.1.157) hosts three services (all auto-start, see configs/):
    • planck-tftp.service — custom python TFTP server on :69 serving /volume1/plancknetboot/tftp/ (firmware, kernel, installer initramfs)
    • planck-netboot.service — python HTTP helper on :8000 (GET rootfs.tar template, GET /events/<name> progress log, POST = upload)
    • NFS read-only export of the share for future use
  • Installer initramfs (configs/installer-init.sh): busybox + sfdisk + e2fsprogs. Identifies the node by CPU serial (map in configs/hostmap), wipes /dev/sda, downloads the Debian 13 Raspberry Pi OS Lite template over HTTP, installs it, sets hostname/ssh keys/user, POSTs pieeprom-revert.upd/.sig to tftp/<serial>/, reboots.
  • Bootloader flow: nodes' EEPROM is flashed (via staged pieeprom.upd from the old OS) to install-mode: BOOT_ORDER=0xf142 (network first), TFTP_IP=192.168.1.157, TFTP_PREFIX=1 (per-serial dirs), ENABLE_SELF_UPDATE=1. After install, the TFTP-served bootloader_update=1 config + <serial>/pieeprom.upd reverts the node to BOOT_ORDER=0xf14 (USB first). Fresh OS boots from SSD; firstboot service tidies up.
  • Reinstall a node anytime: stage install-mode EEPROM again + reboot (delete tftp/<serial>/pieeprom.* on the Synology first if it exists).
  • Node password: see configs/node-password.txt (gitignored). SSH is key-based; password exists for console/sudo recovery.
  • UniFi MFA blocks API access; DHCP reservations deferred — mDNS (planck0NN.local) works across subnets today.

K3S CLUSTER LIVE (2026-09-19)

  • 3 servers (planck002/003/004, embedded etcd HA) + 17 workers
  • Kubernetes v1.36.4+k3s1, all 20 nodes Ready, static IPs
  • kubectl on the laptop: ~/.local/bin/kubectl, kubeconfig at ~/.kube/config (server URL https://192.168.0.110:6443). k3s node-token lives on planck002 at /var/lib/rancher/k3s/server/node-token.
  • Join flow: scripts/k3s-join.sh (needs K3S_TOKEN env)
  • Traefik ingress + local-path storage ship with k3s; default
  • NOTE: Pi firmware injects cgroup_disable=memory — the node cmdline.txt overrides with cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory (already applied fleet-wide; required for k3s).
  • Domain: carr.pub is LIVE — DNSimple apex A record (id 84723206) → 50.46.44.67, UDM port-forwards 80/443 → planck002, Traefik ingress, cert-manager Let's Encrypt issuer letsencrypt-prod, DDNS CronJob ddns-carr-pub keeps DNS fresh. DNSimple token: k8s secret dnsimple-token in namespace monitoring
  • Next up: monitoring (light Prometheus+Grafana), Synology NFS storageclass, carr.pub DNS + cert-manager, load-test rig, agent harness namespaces

REBUILD COMPLETE (2026-09-19)

All 20 nodes re-imaged with Raspberry Pi OS Lite (Debian 13 trixie), via pull-and-image on the laptop (scripts/flash-one.sh, one NVMe stick at a time). Every node verified: correct hostname, SSH key auth, passwordless sudo, 235G root filesystem, timezone America/Los_Angeles.

  • Node password: configs/node-password.txt (gitignored)
  • planck019 + planck020 boot chips updated to 2026-05-17 firmware
  • planck020 boot chip cleaned of the netboot-test config (BOOT_ORDER=0xf14, no TFTP settings)
  • Other 18 nodes: 2023-01-11 firmware, fine; 2026 firmware is staged on their boot partitions and will apply on their next natural reboot
  • The netboot/TFTP experiment is retired: planck-tftp service on the Synology is stopped+disabled; planck-netboot (HTTP helper, :8000) still running and harmless

Postmortem notes (for future reflashes)

  1. The OS image ships a locked pi user (uid 1000). Always remove or rename it before adding your own user, or sudo breaks silently.
  2. The desktop auto-mounter (udisks2) grabs partitions mid-flash; it also reactivates itself when stopped. Mask it (systemctl mask udisks2) during flashing sessions.
  3. Stick device names bounce between sda/sdb — always look up the device by size/model, never assume.
  4. One stick locked up its JMicron USB bridge mid-write (e2fsck hung, 0 IOPS, kernel I/O errors). Unplug/replug fixed it. If a stick hangs the flash, reseat it first, suspect the stick second.
  5. Boot-chip updates (EEPROM) have a timestamp guard: a staged update older-or-equal to what's already flashed is silently ignored. Also, network-delivered EEPROM updates change the FIRMWARE but preserve the CONFIG. bootloader_update=1 in config.txt is required for the bootloader to look for updates on the boot medium.
  6. If a node ever 7-blinks its green LED: kernel not found — check for leftover recovery.bin/pieeprom.upd/pieeprom.sig on the boot partition (staged-but-rejected updates block boot).

Old debug state (superseded)

First live test on planck020 (EEPROM install-mode staged, works — confirmed via vcgencmd bootloader_config showing BOOT_ORDER=0xf142, TFTP_IP, TFTP_PREFIX=1):

  • TFTP chain works: bootloader + firmware fetch firmware, config.txt, cmdline.txt, kernel8.img, and the initramfs (initramfs8 via auto_initramfs=1 — the explicit initramfs <file> followkernel syntax did NOT work on this bootloader; use auto_initramfs + file named initramfs8). All transfers show DONE on the Synology TFTP server (journalctl -u planck-tftp). "STALLED" log lines are harmless — they're duplicate/speculative client sessions that never get ACKed.
  • Custom TFTP server gotcha: firmware requires tsize in the OACK — without it transfers stall. Fixed in configs/planck-tftp-server.py (deployed version on Synology is current).
  • Kernel boots (node pings ~25s after reboot with a DHCP-assigned IP — init reached DHCP), but no HTTP requests ever arrive at the planck-netboot helper (no /events/ GETs, no template GET, journal always empty). Then the node reboots into the OLD OS (~2-3 min cycle). So initramfs /init is either crashing before/inside wget, or HTTP egress from the initramfs fails silently, and the test-mode poweroff -f acts like a reboot (node returns on old OS — actually likely USB fallback after a failed netboot pass).
  • planck020 EEPROM is still in install-mode; every reboot retries netboot then falls back to old OS. To restore it: stage revert via rpi-eeprom-config --config revert.boot.conf flow from the old OS (see EEPROM configs in git history / regenerate from /lib/firmware/raspberrypi/bootloader/stable/pieeprom-2023-01-11.bin).

Next steps for whoever continues

  1. Get console visibility on the netbooted node: add per-log-line HTTP reporting in configs/installer-init.sh (each log() also does wget -q -O /dev/null "$HTTP/events/<serial>-<msg>" once DHCP is up), plus flush a tmpfs log file after DHCP. Alternatively boot with console=serial0,115200 and attach a USB-serial cable to planck020.
  2. Suspects for the silent init death: busybox wget HTTP behavior in initramfs (try wget -O- verbose, test with numeric IP + port 8000 — verified reachable from other nodes), or udhcpc script path issues. Consider testing initramfs content standalone (gunzip + cpio listing).
  3. Template tarball + all services are already on the Synology (/volume1/plancknetboot/): rootfs.tar (sha256 e37837bee03ae2b3...), tftp/ (firmware, initramfs8, auto_initramfs config), HTTP helper :8000, TFTP :69. Credentials for Synology SSH are session-only (not stored).
  4. Once one node completes end-to-end, the rest is a loop: stage install-mode EEPROM per node + reboot (script this in scripts/).

Phases

  • Phase 0 — Access + inventory (SSH keys on all 20, docs/)
  • Phase 1 — Network design: static IPs / reservations in UniFi
  • Phase 2 — Reimage all 20 nodes with fresh Raspberry Pi OS
  • Phase 3 — Cluster runtime install (k3s — decided and live)
  • Phase 4 — Fixed IPs pinned on nodes (Synology NFS storage class still open)
  • Phase 5 — Monitoring (Prometheus/Grafana), load-test rig, https://carr.pub

Resolved decisions (were "open" on day one)

  1. Reimage method — pull-and-image on the laptop won (flash-one.sh). The network-boot experiment is retired but documented below as a postmortem; its Synology services linger (tftp disabled by design).
  2. Cluster runtime — k3s, 3 servers (002-004) + 17 workers.

Open items (next sessions)

  • Synology NFS StorageClass (survives node swaps; local-path is current default)
  • Grafana dashboards worth looking at (stack up, dashboards not built)
  • Rotate the Grafana admin password (planck-lab-admin is a committed placeholder in manifests/monitoring/grafana.yaml)
  • UniFi DHCP reservations for the 20 node IPs (see docs/IPS.md)