183 lines
9.4 KiB
Markdown
183 lines
9.4 KiB
Markdown
# Rebuild plan & decisions log
|
||
|
||
## Where we started (2026-09-19)
|
||
|
||
- Old cluster: Docker Swarm (manager planck001) with Portainer, Traefik, and
|
||
what appears to be CapRover remnants. Broken: workers stuck in "pending"
|
||
because the manager's IP changed (workers dial stale `192.168.1.173:2377`).
|
||
- Decision: **wipe everything and rebuild clean** rather than repair.
|
||
|
||
## Hardware
|
||
|
||
- 20× Raspberry Pi 4 Model B Rev 1.4, 8GB RAM, 4 cores
|
||
- Boot: 238.5GB USB SSD on every node (no SD cards installed)
|
||
- Bootloader EEPROM: 2023/01/11 on 19 nodes; planck019 has Feb 2021 (update it)
|
||
- All nodes healthy: temps 43–52°C, disks ~4% used
|
||
|
||
## Network
|
||
|
||
- Ubiquiti UDM Pro, 2 switches, all nodes on the same router
|
||
- IPs currently drift across 192.168.0/1/2.x subnets via DHCP — planck001 sits
|
||
on 192.168.30.x (different subnet, cause unknown, doesn't matter: we're
|
||
rebuilding)
|
||
- Decision: **dedicated static addressing for the cluster** via UniFi DHCP
|
||
reservations (MAC table in `docs/MACS.txt`)
|
||
|
||
## Storage
|
||
|
||
- Synology NAS on same LAN — shared storage target (NFS) for cluster
|
||
workloads
|
||
|
||
## Rebuild architecture (decided 2026-09-19)
|
||
|
||
Fully unattended network reimage, no per-node physical access:
|
||
|
||
- **Synology** (`192.168.1.157`) hosts three services (all auto-start, see
|
||
`configs/`):
|
||
- `planck-tftp.service` — custom python TFTP server on :69 serving
|
||
`/volume1/plancknetboot/tftp/` (firmware, kernel, installer initramfs)
|
||
- `planck-netboot.service` — python HTTP helper on :8000 (GET `rootfs.tar`
|
||
template, GET `/events/<name>` progress log, POST = upload)
|
||
- NFS read-only export of the share for future use
|
||
- **Installer initramfs** (`configs/installer-init.sh`): busybox + sfdisk +
|
||
e2fsprogs. Identifies the node by CPU serial (map in
|
||
`configs/hostmap`), wipes `/dev/sda`, downloads the Debian 13 Raspberry Pi
|
||
OS Lite template over HTTP, installs it, sets hostname/ssh keys/user,
|
||
POSTs `pieeprom-revert.upd/.sig` to `tftp/<serial>/`, reboots.
|
||
- **Bootloader flow**: nodes' EEPROM is flashed (via staged `pieeprom.upd`
|
||
from the old OS) to install-mode: `BOOT_ORDER=0xf142` (network first),
|
||
`TFTP_IP=192.168.1.157`, `TFTP_PREFIX=1` (per-serial dirs),
|
||
`ENABLE_SELF_UPDATE=1`. After install, the TFTP-served
|
||
`bootloader_update=1` config + `<serial>/pieeprom.upd` reverts the node to
|
||
`BOOT_ORDER=0xf14` (USB first). Fresh OS boots from SSD; firstboot service
|
||
tidies up.
|
||
- **Reinstall a node anytime**: stage install-mode EEPROM again + reboot
|
||
(delete `tftp/<serial>/pieeprom.*` on the Synology first if it exists).
|
||
- Node password: see `configs/node-password.txt` (gitignored). SSH is
|
||
key-based; password exists for console/sudo recovery.
|
||
- UniFi MFA blocks API access; DHCP reservations deferred — mDNS
|
||
(`planck0NN.local`) works across subnets today.
|
||
|
||
## K3S CLUSTER LIVE (2026-09-19)
|
||
|
||
- **3 servers** (planck002/003/004, embedded etcd HA) + **17 workers**
|
||
- Kubernetes v1.36.4+k3s1, all 20 nodes Ready, static IPs
|
||
- kubectl on the laptop: `~/.local/bin/kubectl`, kubeconfig at `~/.kube/config`
|
||
(server URL https://192.168.0.110:6443). k3s node-token lives on planck002
|
||
at /var/lib/rancher/k3s/server/node-token.
|
||
- Join flow: `scripts/k3s-join.sh` (needs K3S_TOKEN env)
|
||
- Traefik ingress + local-path storage ship with k3s; default
|
||
- NOTE: Pi firmware injects `cgroup_disable=memory` — the node cmdline.txt
|
||
overrides with `cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory`
|
||
(already applied fleet-wide; required for k3s).
|
||
- Domain for websites: carr.pub (DNS not yet pointed; pending)
|
||
- Next up: monitoring (light Prometheus+Grafana), Synology NFS storageclass,
|
||
carr.pub DNS + cert-manager, load-test rig, agent harness namespaces
|
||
|
||
## REBUILD COMPLETE (2026-09-19)
|
||
|
||
All 20 nodes re-imaged with Raspberry Pi OS Lite (Debian 13 trixie), via
|
||
pull-and-image on the laptop (`scripts/flash-one.sh`, one NVMe stick at a
|
||
time). Every node verified: correct hostname, SSH key auth, passwordless
|
||
sudo, 235G root filesystem, timezone America/Los_Angeles.
|
||
|
||
- Node password: `configs/node-password.txt` (gitignored)
|
||
- planck019 + planck020 boot chips updated to 2026-05-17 firmware
|
||
- planck020 boot chip cleaned of the netboot-test config (BOOT_ORDER=0xf14,
|
||
no TFTP settings)
|
||
- Other 18 nodes: 2023-01-11 firmware, fine; 2026 firmware is staged on
|
||
their boot partitions and will apply on their next natural reboot
|
||
- The netboot/TFTP experiment is retired: planck-tftp service on the
|
||
Synology is stopped+disabled; planck-netboot (HTTP helper, :8000) still
|
||
running and harmless
|
||
|
||
### Postmortem notes (for future reflashes)
|
||
|
||
1. The OS image ships a locked `pi` user (uid 1000). Always remove or
|
||
rename it before adding your own user, or sudo breaks silently.
|
||
2. The desktop auto-mounter (udisks2) grabs partitions mid-flash; it also
|
||
reactivates itself when stopped. Mask it (`systemctl mask udisks2`)
|
||
during flashing sessions.
|
||
3. Stick device names bounce between sda/sdb — always look up the device
|
||
by size/model, never assume.
|
||
4. One stick locked up its JMicron USB bridge mid-write (e2fsck hung, 0
|
||
IOPS, kernel I/O errors). Unplug/replug fixed it. If a stick hangs the
|
||
flash, reseat it first, suspect the stick second.
|
||
5. Boot-chip updates (EEPROM) have a timestamp guard: a staged update
|
||
older-or-equal to what's already flashed is silently ignored. Also,
|
||
network-delivered EEPROM updates change the FIRMWARE but preserve the
|
||
CONFIG. `bootloader_update=1` in config.txt is required for the
|
||
bootloader to look for updates on the boot medium.
|
||
6. If a node ever 7-blinks its green LED: kernel not found — check for
|
||
leftover `recovery.bin`/`pieeprom.upd`/`pieeprom.sig` on the boot
|
||
partition (staged-but-rejected updates block boot).
|
||
|
||
## Old debug state (superseded)
|
||
|
||
First live test on planck020 (EEPROM install-mode staged, works — confirmed
|
||
via `vcgencmd bootloader_config` showing BOOT_ORDER=0xf142, TFTP_IP,
|
||
TFTP_PREFIX=1):
|
||
|
||
- **TFTP chain works**: bootloader + firmware fetch firmware, config.txt,
|
||
cmdline.txt, kernel8.img, and the initramfs (`initramfs8` via
|
||
`auto_initramfs=1` — the explicit `initramfs <file> followkernel` syntax
|
||
did NOT work on this bootloader; use auto_initramfs + file named
|
||
`initramfs8`). All transfers show DONE on the Synology TFTP server
|
||
(`journalctl -u planck-tftp`). "STALLED" log lines are harmless — they're
|
||
duplicate/speculative client sessions that never get ACKed.
|
||
- **Custom TFTP server gotcha**: firmware requires `tsize` in the OACK —
|
||
without it transfers stall. Fixed in
|
||
`configs/planck-tftp-server.py` (deployed version on Synology is current).
|
||
- **Kernel boots** (node pings ~25s after reboot with a DHCP-assigned IP —
|
||
init reached DHCP), but **no HTTP requests ever arrive** at the
|
||
planck-netboot helper (no `/events/` GETs, no template GET, journal
|
||
always empty). Then the node reboots into the OLD OS (~2-3 min cycle).
|
||
So initramfs `/init` is either crashing before/inside wget, or HTTP egress
|
||
from the initramfs fails silently, and the test-mode `poweroff -f` acts
|
||
like a reboot (node returns on old OS — actually likely USB fallback after
|
||
a failed netboot pass).
|
||
- planck020 EEPROM is still in install-mode; every reboot retries netboot
|
||
then falls back to old OS. To restore it: stage revert via
|
||
`rpi-eeprom-config --config revert.boot.conf` flow from the old OS (see
|
||
EEPROM configs in git history / regenerate from
|
||
`/lib/firmware/raspberrypi/bootloader/stable/pieeprom-2023-01-11.bin`).
|
||
|
||
### Next steps for whoever continues
|
||
|
||
1. Get console visibility on the netbooted node: add per-log-line HTTP
|
||
reporting in `configs/installer-init.sh` (each `log()` also does
|
||
`wget -q -O /dev/null "$HTTP/events/<serial>-<msg>"` once DHCP is up),
|
||
plus flush a tmpfs log file after DHCP. Alternatively boot with
|
||
`console=serial0,115200` and attach a USB-serial cable to planck020.
|
||
2. Suspects for the silent init death: busybox `wget` HTTP behavior in
|
||
initramfs (try `wget -O-` verbose, test with numeric IP + port 8000 —
|
||
verified reachable from other nodes), or udhcpc script path issues.
|
||
Consider testing initramfs content standalone (gunzip + cpio listing).
|
||
3. Template tarball + all services are already on the Synology
|
||
(`/volume1/plancknetboot/`): rootfs.tar (sha256 e37837bee03ae2b3...),
|
||
tftp/ (firmware, initramfs8, auto_initramfs config), HTTP helper :8000,
|
||
TFTP :69. Credentials for Synology SSH are session-only (not stored).
|
||
4. Once one node completes end-to-end, the rest is a loop: stage
|
||
install-mode EEPROM per node + reboot (script this in `scripts/`).
|
||
|
||
## Phases
|
||
|
||
- [x] Phase 0 — Access + inventory (SSH keys on all 20, docs/)
|
||
- [ ] Phase 1 — Network design: static IPs / reservations in UniFi
|
||
- [ ] Phase 2 — Reimage all 20 nodes with fresh Raspberry Pi OS
|
||
- [ ] Phase 3 — Cluster runtime install (candidates: k3s vs Docker Swarm)
|
||
- [ ] Phase 4 — Storage integration with Synology (NFS CSI)
|
||
- [ ] Phase 5 — Apps: monitoring, Portainer/UI, whatever the lab is for
|
||
|
||
## Open decisions
|
||
|
||
1. **Reimage method** — options:
|
||
a. PXE/network boot: EEPROM boot-order set to netboot; DHCP next-server
|
||
points at a TFTP/HTTP installer. Zero-touch per node, reusable forever.
|
||
Installer server can run on the Synology (Docker) or a laptop.
|
||
b. SD-card bootstrap installer: flash one SD image, insert per node, it
|
||
wipes + installs the USB SSD, removes itself. Simple, physical walk.
|
||
c. Pull SSDs, image on a PC with Raspberry Pi Imager ×20. Most manual.
|
||
2. **Cluster runtime** — k3s (lighter, k8s-compatible, we manage it with
|
||
automation) vs Docker Swarm + Portainer (simplest).
|