Backups complete: Glacier offsite tier covers plancknetboot

Backups: etcd snapshots (6h/14d), NAS btrfs daily snapshots, secrets off-laptop
Portal: link to DSM directly
2026-09-20 17:03:57 -07:00 · 2026-09-20 16:58:45 -07:00 · 2026-09-20 16:46:22 -07:00 · 2026-09-20 16:11:36 -07:00 · 2026-09-20 14:42:46 -07:00 · 2026-09-20 14:33:50 -07:00
7 changed files with 333 additions and 21 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -3,3 +3,4 @@ configs/node-password.txt
 configs/admin-password.txt
 configs/authelia-session-secret.txt
 configs/authelia-jwt-secret.txt
+configs/gitea-password.txt
--- a/docs/IPS.md
+++ b/docs/IPS.md
@@ -3,29 +3,34 @@
 All nodes: static IP configured on the node (NetworkManager, "Wired
 connection 1", manual, /22, gateway+DNS 192.168.1.1 = UDM Pro).

+Every node is also on the owner's Tailscale tailnet (always-on) — the second
+IP is reachable from anywhere via the tailnet, including the laptop
+kubeconfig (API: https://100.124.112.13:6443 = planck002's Tailscale IP,
+added to k3s tls-san on all three servers).
+
 RECOMMENDED follow-up: add these as DHCP reservations in UniFi too, so the
 router never leases these addresses to other devices. (UniFi UI: each client
 device → Fixed IP. API is blocked by MFA.)

-| Node | IP | MAC |
+| Node | LAN IP | Tailscale IP | MAC |
 |------|----|----|
-| planck001 | 192.168.2.245 | dc:a6:32:e9:7f:75 |
-| planck002 | 192.168.0.110 | dc:a6:32:e9:7f:de |
-| planck003 | 192.168.1.109 | dc:a6:32:e9:7d:fa |
-| planck004 | 192.168.2.250 | dc:a6:32:e9:7f:7b |
-| planck005 | 192.168.2.178 | dc:a6:32:e9:7f:33 |
-| planck006 | 192.168.1.2 | dc:a6:32:e9:7d:8b |
-| planck007 | 192.168.1.48 | dc:a6:32:e9:7d:bd |
-| planck008 | 192.168.2.43 | dc:a6:32:e9:7e:b2 |
-| planck009 | 192.168.1.82 | dc:a6:32:e9:7d:df |
-| planck010 | 192.168.2.91 | dc:a6:32:e9:7e:e2 |
-| planck011 | 192.168.0.185 | dc:a6:32:e9:7d:46 |
-| planck012 | 192.168.0.235 | dc:a6:32:e9:7d:78 |
-| planck013 | 192.168.2.230 | dc:a6:32:e9:7f:67 |
-| planck014 | 192.168.2.4 | dc:a6:32:e9:78:d1 |
-| planck015 | 192.168.2.31 | dc:a6:32:e9:78:ec |
-| planck016 | 192.168.2.94 | dc:a6:32:e9:7e:e5 |
-| planck017 | 192.168.1.94 | dc:a6:32:e9:7d:eb |
-| planck018 | 192.168.2.102 | dc:a6:32:e9:7e:ed |
-| planck019 | 192.168.2.32 | dc:a6:32:e9:7e:a6 |
-| planck020 | 192.168.1.23 | dc:a6:32:e9:7d:a4 |
+| planck001 | 192.168.2.245 | 100.95.205.28 | dc:a6:32:e9:7f:75 |
+| planck002 | 192.168.0.110 | 100.124.112.13 | dc:a6:32:e9:7f:de |
+| planck003 | 192.168.1.109 | 100.98.49.13 | dc:a6:32:e9:7d:fa |
+| planck004 | 192.168.2.250 | 100.93.201.48 | dc:a6:32:e9:7f:7b |
+| planck005 | 192.168.2.178 | 100.98.22.50 | dc:a6:32:e9:7f:33 |
+| planck006 | 192.168.1.2 | 100.117.72.42 | dc:a6:32:e9:7d:8b |
+| planck007 | 192.168.1.48 | 100.116.208.66 | dc:a6:32:e9:7d:bd |
+| planck008 | 192.168.2.43 | 100.125.106.86 | dc:a6:32:e9:7e:b2 |
+| planck009 | 192.168.1.82 | 100.91.196.17 | dc:a6:32:e9:7d:df |
+| planck010 | 192.168.2.91 | 100.65.93.90 | dc:a6:32:e9:7e:e2 |
+| planck011 | 192.168.0.185 | 100.68.40.111 | dc:a6:32:e9:7d:46 |
+| planck012 | 192.168.0.235 | 100.98.229.124 | dc:a6:32:e9:7d:78 |
+| planck013 | 192.168.2.230 | 100.121.93.41 | dc:a6:32:e9:7f:67 |
+| planck014 | 192.168.2.4 | 100.68.35.118 | dc:a6:32:e9:78:d1 |
+| planck015 | 192.168.2.31 | 100.108.69.67 | dc:a6:32:e9:78:ec |
+| planck016 | 192.168.2.94 | 100.88.118.95 | dc:a6:32:e9:7e:e5 |
+| planck017 | 192.168.1.94 | 100.72.37.121 | dc:a6:32:e9:7d:eb |
+| planck018 | 192.168.2.102 | 100.114.149.34 | dc:a6:32:e9:7e:ed |
+| planck019 | 192.168.2.32 | 100.113.193.126 | dc:a6:32:e9:7e:a6 |
+| planck020 | 192.168.1.23 | 100.104.220.31 | dc:a6:32:e9:7d:a4 |
--- a/docs/PLAN.md
+++ b/docs/PLAN.md
@@ -180,6 +180,35 @@ TFTP_PREFIX=1):
   postmortem; its Synology services linger (tftp disabled by design).
 2. **Cluster runtime** — k3s, 3 servers (002-004) + 17 workers.

+## Backups (established 2026-09-20)
+
+Three layers:
+
+1. **etcd snapshots** (cluster state — all k3s objects, secrets, config):
+   every 6h on each server node, 14-day retention
+   (`/etc/rancher/k3s/config.yaml`: `etcd-snapshot-schedule-cron: "0 */6 * * *"`).
+   Files at `/var/lib/rancher/k3s/server/db/snapshots/`. Manual test snapshot:
+   `sudo k3s etcd-snapshot save --name test`. Restore: stop k3s on all
+   servers, `sudo k3s server --cluster-init --cluster-reset
+   --cluster-reset-restore-path=<snapshot>` on one, then start all.
+2. **NAS btrfs snapshots** (Gitea repos, k8s-volumes, netboot artifacts):
+   `/usr/local/bin/planck-backup.sh` on the Synology, daily at 3am, 14-day
+   retention, read-only btrfs snapshots of the `plancknetboot` share
+   (`plancknetboot@daily-YYYY-MM-DD`). Manual run: the script itself.
+   Restore: `btrfs subvolume snapshot /volume1/plancknetboot@daily-<date> /volume1/restore`.
+3. **Laptop-only secret files** (node/gitea/admin passwords, authelia
+   session/jwt secrets — the gitignored ones): tarball at
+   `/volume1/plancknetboot/cluster-secrets/secrets.tar.gz` (root-only).
+   Refresh whenever a password is created/changed.
+
+4. **Offsite**: Glacier Backup to AWS now includes the `plancknetboot` share
+   (owner verified the task 2026-09-20). Slow-restore tier for real
+   disasters; layers 1-2 are the fast-recovery tiers.
+
+Grafana dashboards and all manifests live in git (Gitea + laptop), so they
+are covered by layers 1-3 transitively. Prometheus history (15d) is
+deliberately not backed up — monitoring data is expendable.
+
 ## Open items (next sessions)

 - Synology NFS StorageClass (survives node swaps; local-path is current default)
--- a/manifests/admin/portal.yaml
+++ b/manifests/admin/portal.yaml
@@ -23,6 +23,8 @@ data:
      <body><div class="card">
        <h1>planck</h1>
        <a href="https://grafana.carr.pub">Grafana</a>
+        <a href="https://git.carr.pub">Gitea (git + CI)</a>
+        <a href="https://100.123.214.83:5001">Synology DSM + files</a>
        <a href="https://auth.carr.pub">Account &amp; 2FA settings</a>
        <p>20 nodes · k3s · carr.pub</p>
      </div></body>
--- a/manifests/git/act-runner.yaml
+++ b/manifests/git/act-runner.yaml
@@ -0,0 +1,75 @@
+# act_runner x3 with docker-in-docker sidecars, so Gitea Actions jobs
+# run as containers on the worker nodes (ARM64).
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: act-runner
+  namespace: git
+spec:
+  replicas: 3
+  selector:
+    matchLabels: {app: act-runner}
+  template:
+    metadata:
+      labels: {app: act-runner}
+    spec:
+      containers:
+        - name: runner
+          image: gitea/act_runner:latest
+          env:
+            - {name: GITEA_INSTANCE_URL, value: "http://gitea.git.svc:3000"}
+            - {name: GITEA_RUNNER_REGISTRATION_TOKEN, valueFrom: {secretKeyRef: {name: act-runner-token, key: token}}}
+            - {name: DOCKER_HOST, value: "tcp://127.0.0.1:2375"}
+            - {name: CONFIG_FILE, value: /config/config.yaml}
+          resources:
+            requests: {cpu: 50m, memory: 64Mi}
+            limits: {memory: 256Mi}
+          volumeMounts:
+            - {name: work, mountPath: /data}
+            - {name: config, mountPath: /config}
+        - name: dind
+          image: docker:27-dind
+          securityContext:
+            privileged: true
+          env:
+            - {name: DOCKER_TLS_CERTDIR, value: ""}
+          resources:
+            requests: {cpu: 100m, memory: 256Mi}
+            limits: {memory: 1Gi}
+          volumeMounts:
+            - {name: work, mountPath: /data}
+      volumes:
+        - name: work
+          emptyDir: {}
+        - name: config
+          configMap: {name: act-runner-config}
+      affinity:
+        podAntiAffinity:
+          preferredDuringSchedulingIgnoredDuringExecution:
+            - weight: 100
+              podAffinityTerm:
+                labelSelector:
+                  matchLabels: {app: act-runner}
+                topologyKey: kubernetes.io/hostname
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: act-runner-config
+  namespace: git
+data:
+  config.yaml: |
+    runner:
+      capacity: 4
+      timeout: 30m
+      insecure: false
+      fetch_timeout: 5s
+      labels:
+        - "arm64:host"
+        - "ubuntu-latest:docker://node:22-bookworm"
+    cache:
+      enabled: false
+    container:
+      privileged: false
+      options: ""
+      valid_volumes: []
--- a/manifests/git/gitea.yaml
+++ b/manifests/git/gitea.yaml
@@ -0,0 +1,86 @@
+# Gitea: git hosting + CI, with repo data on Synology NFS.
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: git
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
+  name: gitea-data
+  namespace: git
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: nfs-synology
+  resources:
+    requests: {storage: 20Gi}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: gitea
+  namespace: git
+spec:
+  replicas: 1
+  strategy:
+    type: Recreate
+  selector:
+    matchLabels: {app: gitea}
+  template:
+    metadata:
+      labels: {app: gitea}
+    spec:
+      containers:
+        - name: gitea
+          image: gitea/gitea:1.24
+          env:
+            - {name: USER_UID, value: "1000"}
+            - {name: USER_GID, value: "1000"}
+            - {name: GITEA__database__DB_TYPE, value: sqlite3}
+            - {name: GITEA__server__DOMAIN, value: git.carr.pub}
+            - {name: GITEA__server__ROOT_URL, value: "https://git.carr.pub/"}
+            - {name: GITEA__server__HTTP_PORT, value: "3000"}
+            - {name: GITEA__actions__ENABLED, value: "true"}
+            - {name: GITEA__service__DISABLE_REGISTRATION, value: "true"}
+          ports: [{containerPort: 3000}]
+          resources:
+            requests: {cpu: 100m, memory: 256Mi}
+            limits: {memory: 1Gi}
+          volumeMounts:
+            - {name: data, mountPath: /data}
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: gitea-data}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: gitea
+  namespace: git
+spec:
+  selector: {app: gitea}
+  ports:
+    - port: 3000
+      targetPort: 3000
+---
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: gitea
+  namespace: git
+  annotations:
+    cert-manager.io/cluster-issuer: letsencrypt-prod
+    # NOTE: intentionally NOT behind the SSO gate - git clients and CI
+    # need direct token auth; Gitea has its own accounts
+spec:
+  tls:
+    - hosts: [git.carr.pub]
+      secretName: git-carr-pub-tls
+  rules:
+    - host: git.carr.pub
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service: {name: gitea, port: {number: 3000}}
--- a/manifests/monitoring/nfs-provisioner.yaml
+++ b/manifests/monitoring/nfs-provisioner.yaml
@@ -0,0 +1,114 @@
+# NFS dynamic provisioning from the Synology (192.168.1.157).
+# Base dir /volume1/plancknetboot/k8s-volumes; each PVC gets a subdir.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: nfs-provisioner
+  namespace: monitoring
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: nfs-provisioner
+rules:
+  - apiGroups: [""]
+    resources: [persistentvolumes]
+    verbs: [get, list, watch, create, delete]
+  - apiGroups: [""]
+    resources: [persistentvolumeclaims]
+    verbs: [get, list, watch, update]
+  - apiGroups: ["storage.k8s.io"]
+    resources: [storageclasses, volumeattributesclasses]
+    verbs: [get, list, watch]
+  - apiGroups: [""]
+    resources: [events]
+    verbs: [create, update, patch]
+  - apiGroups: ["storage.k8s.io"]
+    resources: [csidrivers]
+    verbs: [get, list]
+  - apiGroups: [""]
+    resources: [nodes]
+    verbs: [get, list, watch]
+  - apiGroups: ["storage.k8s.io"]
+    resources: [volumeattachments]
+    verbs: [get, list, watch, delete]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: nfs-provisioner
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: nfs-provisioner
+subjects:
+  - kind: ServiceAccount
+    name: nfs-provisioner
+    namespace: monitoring
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: nfs-provisioner
+  namespace: monitoring
+rules:
+  - apiGroups: [""]
+    resources: [endpoints]
+    verbs: [get, list, watch, create, update, patch]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: nfs-provisioner
+  namespace: monitoring
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: nfs-provisioner
+subjects:
+  - kind: ServiceAccount
+    name: nfs-provisioner
+    namespace: monitoring
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nfs-provisioner
+  namespace: monitoring
+spec:
+  replicas: 1
+  selector:
+    matchLabels: {app: nfs-provisioner}
+  template:
+    metadata:
+      labels: {app: nfs-provisioner}
+    spec:
+      serviceAccountName: nfs-provisioner
+      containers:
+        - name: nfs-provisioner
+          image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2
+          volumeMounts:
+            - {name: nfs, mountPath: /persistentvolumes}
+          env:
+            - {name: PROVISIONER_NAME, value: nfs-synology}
+            - {name: NFS_SERVER, value: "192.168.1.157"}
+            - {name: NFS_PATH, value: /volume1/plancknetboot/k8s-volumes}
+            - {name: ENABLE_LEADER_ELECTION, value: "false"}
+          resources:
+            requests: {cpu: 20m, memory: 32Mi}
+            limits: {memory: 128Mi}
+      volumes:
+        - name: nfs
+          nfs:
+            server: "192.168.1.157"
+            path: /volume1/plancknetboot/k8s-volumes
+---
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+  name: nfs-synology
+provisioner: nfs-synology
+reclaimPolicy: Delete
+volumeBindingMode: Immediate
+parameters:
+  archiveOnDelete: "true"
Author	SHA1	Message	Date
Adam Carr	258284a0c7	Backups complete: Glacier offsite tier covers plancknetboot	2026-09-20 17:03:57 -07:00
Adam Carr	e25d9add77	Backups: etcd snapshots (6h/14d), NAS btrfs daily snapshots, secrets off-laptop	2026-09-20 16:58:45 -07:00
Adam Carr	3df157a774	Portal: link to DSM directly	2026-09-20 16:46:22 -07:00
Adam Carr	a7a841988f	Tailscale on Synology: NAS + File Station reachable remotely via tailnet	2026-09-20 16:11:36 -07:00
Adam Carr	e6d7b265f8	Tailscale on all 20 nodes: remote kubectl + SSH from anywhere (docs/IPS.md)	2026-09-20 14:42:46 -07:00
Adam Carr	b2878f43c6	act-runner: move registration token to k8s secret	2026-09-20 14:33:50 -07:00
Adam Carr	de0bb0ad7d	Gitea + CI: git.carr.pub, NFS storage class, 3 runners	2026-09-20 14:33:39 -07:00