feat: add homelab topology documentation, docker deployments, and authentik sso configuration scripts

2026-10-30 12:56:53 -04:00
parent e07e2204e3
commit 6b12646a0f
24 changed files with 3902 additions and 1 deletions
--- a/docs/power_resilience_plan.md
+++ b/docs/power_resilience_plan.md
@@ -0,0 +1,263 @@
+# Snarfnet Homelab Power Resilience & UPS Automation Plan
+
+This document details the power resilience strategy, tiered graceful shutdown sequence, and integration architecture for the **snarfnet** homelab centered around the primary **NUT (Network UPS Tools) Master Server**.
+
+---
+
+## 1. Core Architecture & IP Assignments
+
+The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
+
+| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description |
+|---|---|---|---|---|
+| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. |
+| **NUT Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Compute / Swarm) | Peanut container UI consuming `14.4.4.44:3493`. |
+| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for alerts/triggers. |
+| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493`. |
+| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. |
+| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. |
+
+---
+
+## 2. Power Resilience Topology Map
+
+```mermaid
+flowchart TD
+    subgraph Grid ["Utility Power Grid"]
+        Mains["Main AC Power Supply"]
+    end
+
+    subgraph PowerHardware ["Power Protection"]
+        UPS["Central UPS Hardware"]
+    end
+
+    subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"]
+        UPSD["upsd Daemon (Port 3493)"]
+        UPSMON_M["upsmon (Master)"]
+    end
+
+    subgraph Dashboards ["Monitoring & Alerting"]
+        Peanut["Peanut Dashboard\n(peanut.snarfnet.net:8050)"]
+        HA["Home Assistant\n(14.4.4.20)"]
+        Zabbix["Zabbix Server\n(140.44.4.61)"]
+    end
+
+    subgraph Slaves ["UPS Slaves (upsmon Client Nodes)"]
+        Harvester["Spacedock HCI Nodes\n(140.44.4.81 - 83)"]
+        K3sControl["K3s Control Planes\n(Aerodrome & Spaceport)"]
+        K3sWorkers["K3s Worker Nodes"]
+        Swarm["Shipyard Swarm Nodes\n(140.44.4.71 - 75)"]
+        NAS["Synology NAS Arrays\n(14.10.10.71 / 72)"]
+    end
+
+    Mains --> UPS
+    UPS -- USB / SNMP --> UPSD
+    UPSD --> UPSMON_M
+    
+    UPSD -- TCP 3493 --> Peanut
+    UPSD -- TCP 3493 --> HA
+    UPSD -- TCP 3493 --> Zabbix
+    
+    UPSD -- TCP 3493 (Slave Broadcast) --> Harvester
+    UPSD -- TCP 3493 (Slave Broadcast) --> K3sControl
+    UPSD -- TCP 3493 (Slave Broadcast) --> K3sWorkers
+    UPSD -- TCP 3493 (Slave Broadcast) --> Swarm
+    UPSD -- TCP 3493 (Slave Broadcast) --> NAS
+```
+
+---
+
+## 3. Tiered Shutdown Hierarchy & Execution Timeline
+
+To prevent data corruption and unexpected hard cutoffs, hosts shutdown in staged tiers based on power outage duration and battery level.
+
+```
+-------------------------------------------------------------------------+
+| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT       |
+-------------------------------------------------------------------------+
+                                    |
+                                    v
+-------------------------------------------------------------------------+
+| [T+3m / Batt <= 80%] TIER 1: Non-essential Workloads                    |
+| - Dev VMs, transcoders, test containers in Swarm & K3s drained          |
+-------------------------------------------------------------------------+
+                                    |
+                                    v
+-------------------------------------------------------------------------+
+| [T+7m / Batt <= 50%] TIER 2: Hypervisors, Compute Workers & Storage     |
+| - Harvester spacedock VMs gracefully shut down via virsh/API            |
+| - Aerodrome & Spaceport worker nodes gracefully halt                    |
+| - Synology NAS arrays (14.10.10.71/72) enter Safe Mode & unmount pools |
+-------------------------------------------------------------------------+
+                                    |
+                                    v
+-------------------------------------------------------------------------+
+| [T+12m / Batt <= 30%] TIER 3: Control Planes & Management Stack         |
+| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Control Planes|
+| - Swarm Manager (140.44.4.71) & AI Nodes (140.44.4.7-8) halt             |
+-------------------------------------------------------------------------+
+                                    |
+                                    v
+-------------------------------------------------------------------------+
+| [T+15m / Batt Low] TIER 4: Forced Shutdown (FSD) & Hardware Cutoff      |
+| - NUT Master (14.4.4.44) issues FSD signal to remaining clients         |
+| - NUT Master issues `upsdrvctl shutdown` (cuts UPS load outlet)         |
+| - Network Core (UDM Pro 14.44.4.1) stays alive until final battery drop  |
+-------------------------------------------------------------------------+
+```
+
+---
+
+## 4. Configuration Blueprint & Service Integrations
+
+### A. NUT Master Server (`14.4.4.44`) Configuration Files
+
+#### `/etc/nut/ups.conf`
+```ini
+[snarf-ups]
+    driver = usbhid-ups
+    port = auto
+    desc = "Snarfnet Main Server Rack UPS"
+    pollinterval = 5
+```
+
+#### `/etc/nut/upsd.conf`
+```ini
+LISTEN 14.4.4.44 3493
+LISTEN 127.0.0.1 3493
+MAXAGE 15
+```
+
+#### `/etc/nut/upsd.users`
+```ini
+[admin]
+    password = SUPER_SECRET_ADMIN_PASS
+    actions = SET
+    instcmds = ALL
+
+[upsmon_master]
+    password = MASTER_MONITOR_PASS
+    upsmon master
+
+[upsmon_slave]
+    password = SLAVE_MONITOR_PASS
+    upsmon slave
+
+[peanut]
+    password = PEANUT_MONITOR_PASS
+    upsmon slave
+```
+
+#### `/etc/nut/upsmon.conf`
+```ini
+MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master
+MINSUPPLIES 1
+SHUTDOWNCMD "/sbin/shutdown -h +0"
+NOTIFYCMD /usr/local/bin/nut-notify.sh
+POWERDOWNFLAG /etc/killpower
+
+NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC
+NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC
+NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC
+NOTIFYFLAG FSD      SYSLOG+WALL+EXEC
+NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC
+NOTIFYFLAG COMMOK   SYSLOG+WALL+EXEC
+```
+
+---
+
+### B. Client Node Configuration (`upsmon` Slave)
+
+Applies to Harvester hypervisors, K3s master/worker hosts, Docker Swarm hosts, and Debian Linux nodes.
+
+#### `/etc/nut/upsmon.conf` (Slave Mode)
+```ini
+MONITOR snarf-ups@14.4.4.44 1 upsmon_slave SLAVE_MONITOR_PASS slave
+MINSUPPLIES 1
+SHUTDOWNCMD "/sbin/shutdown -h +0"
+POWERDOWNFLAG /etc/killpower
+HOSTSYNC 15
+DEADTIME 15
+RBWARNTIME 43200
+NOCOMMWARNTIME 300
+FINALDELAY 5
+```
+
+---
+
+### C. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`)
+
+1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**.
+2. Check **Enable UPS Support**.
+3. Select UPS Type: **Synology UPS Server / Network NUT Server**.
+4. Network UPS server IP address: **`14.4.4.44`**.
+5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `5 minutes` (or `Upon low battery`).
+
+---
+
+### D. Peanut Web Dashboard (`peanut.snarfnet.net`)
+
+Peanut runs on Docker Swarm (`140.44.4.70:8050`). Ensure the Peanut service configuration references the NUT server at `14.4.4.44`:
+
+#### Updated `docker-deployments/peanut.yml`
+```yaml
+services:
+  peanut:
+    image: brandawg93/peanut:latest
+    ports:
+      - "8050:8050"
+    environment:
+      - TZ=America/New_York
+      - NUT_HOST=14.4.4.44
+      - NUT_PORT=3493
+      - NUT_USER=peanut
+      - NUT_PASSWORD=PEANUT_MONITOR_PASS
+    volumes:
+      - peanut_config:/config
+
+volumes:
+  peanut_config:
+    driver: local
+    driver_opts:
+      type: none
+      o: bind
+      device: /mnt/swarm_shared/legacy_volumes/peanut/
+```
+
+---
+
+### E. Home Assistant (`14.4.4.20`) Integration & Automation
+
+1. Navigate to **Settings** $\rightarrow$ **Devices & Services** $\rightarrow$ **Add Integration** $\rightarrow$ **Network UPS Tools (NUT)**.
+2. Host: **`14.4.4.44`**, Port: **`3493`**.
+3. Username: **`upsmon_slave`**, Password: **`SLAVE_MONITOR_PASS`**.
+
+#### Automation Snippet (`automations.yaml`)
+```yaml
+alias: "Power Resilience: Grid Power Lost Alert"
+trigger:
+  - platform: state
+    entity_id: sensor.snarf_ups_status
+    to: "On Battery"
+action:
+  - service: notify.gotify
+    data:
+      title: "⚡ Power Outage Alert"
+      message: "Grid power failure detected! Homelab is running on UPS battery (14.4.4.44). Tiered shutdown in progress."
+      priority: 8
+```
+
+---
+
+## 5. Power Restoration & Auto-Boot Sequence
+
+When grid power returns, systems auto-recover in reverse order:
+
+1. **UPS AC Mains Restored**: Physical UPS re-enables output outlets once battery reaches safe threshold ($\ge 20\%$).
+2. **Phase 1 - Gateway & Edge**: UDM Pro (`14.44.4.1`) & Core switches boot up and establish internal VLAN routes.
+3. **Phase 2 - NUT Master**: Host at `14.4.4.44` boots (BIOS `Restore AC Power` set to `Always On`). `upsd` service starts.
+4. **Phase 3 - Storage NAS**: Synology NAS nodes (`14.10.10.71`, `14.10.10.72`) auto-boot upon power recovery.
+5. **Phase 4 - Control Planes & Hypervisors**:
+   - Harvester `spacedock` hosts (`140.44.4.81-83`) boot via Wake-on-LAN / IPMI / BIOS settings.
+   - K3s master nodes (`aerodrome` & `spaceport`) boot, resume etcd quorum.
+6. **Phase 5 - Worker Compute & Workloads**: Worker nodes boot; Kubernetes and Docker Swarm automatically reschedule workloads.