# Snarfnet Homelab Power Resilience & UPS Automation Plan This document details the power resilience strategy, tiered graceful shutdown sequence, and integration architecture for the **snarfnet** homelab centered around the primary **NUT (Network UPS Tools) Master Server**. --- ## 1. Core Architecture & IP Assignments The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs. | Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description | |---|---|---|---|---| | **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. | | **NUT Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Compute / Swarm) | Peanut container UI consuming `14.4.4.44:3493`. | | **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for alerts/triggers. | | **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493`. | | **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. | | **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. | --- ## 2. Power Resilience Topology Map ```mermaid flowchart TD subgraph Grid ["Utility Power Grid"] Mains["Main AC Power Supply"] end subgraph PowerHardware ["Power Protection"] UPS["Central UPS Hardware"] end subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"] UPSD["upsd Daemon (Port 3493)"] UPSMON_M["upsmon (Master)"] end subgraph Dashboards ["Monitoring & Alerting"] Peanut["Peanut Dashboard\n(peanut.snarfnet.net:8050)"] HA["Home Assistant\n(14.4.4.20)"] Zabbix["Zabbix Server\n(140.44.4.61)"] end subgraph Slaves ["UPS Slaves (upsmon Client Nodes)"] Harvester["Spacedock HCI Nodes\n(140.44.4.81 - 83)"] K3sControl["K3s Control Planes\n(Aerodrome & Spaceport)"] K3sWorkers["K3s Worker Nodes"] Swarm["Shipyard Swarm Nodes\n(140.44.4.71 - 75)"] NAS["Synology NAS Arrays\n(14.10.10.71 / 72)"] end Mains --> UPS UPS -- USB / SNMP --> UPSD UPSD --> UPSMON_M UPSD -- TCP 3493 --> Peanut UPSD -- TCP 3493 --> HA UPSD -- TCP 3493 --> Zabbix UPSD -- TCP 3493 (Slave Broadcast) --> Harvester UPSD -- TCP 3493 (Slave Broadcast) --> K3sControl UPSD -- TCP 3493 (Slave Broadcast) --> K3sWorkers UPSD -- TCP 3493 (Slave Broadcast) --> Swarm UPSD -- TCP 3493 (Slave Broadcast) --> NAS ``` --- ## 3. Tiered Shutdown Hierarchy & Execution Timeline To prevent data corruption and unexpected hard cutoffs, hosts shutdown in staged tiers based on power outage duration and battery level. ``` +-------------------------------------------------------------------------+ | [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT | +-------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------+ | [T+3m / Batt <= 80%] TIER 1: Non-essential Workloads | | - Dev VMs, transcoders, test containers in Swarm & K3s drained | +-------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------+ | [T+7m / Batt <= 50%] TIER 2: Hypervisors, Compute Workers & Storage | | - Harvester spacedock VMs gracefully shut down via virsh/API | | - Aerodrome & Spaceport worker nodes gracefully halt | | - Synology NAS arrays (14.10.10.71/72) enter Safe Mode & unmount pools | +-------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------+ | [T+12m / Batt <= 30%] TIER 3: Control Planes & Management Stack | | - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Control Planes| | - Swarm Manager (140.44.4.71) & AI Nodes (140.44.4.7-8) halt | +-------------------------------------------------------------------------+ | v +-------------------------------------------------------------------------+ | [T+15m / Batt Low] TIER 4: Forced Shutdown (FSD) & Hardware Cutoff | | - NUT Master (14.4.4.44) issues FSD signal to remaining clients | | - NUT Master issues `upsdrvctl shutdown` (cuts UPS load outlet) | | - Network Core (UDM Pro 14.44.4.1) stays alive until final battery drop | +-------------------------------------------------------------------------+ ``` --- ## 4. Configuration Blueprint & Service Integrations ### A. NUT Master Server (`14.4.4.44`) Configuration Files #### `/etc/nut/ups.conf` ```ini [snarf-ups] driver = usbhid-ups port = auto desc = "Snarfnet Main Server Rack UPS" pollinterval = 5 ``` #### `/etc/nut/upsd.conf` ```ini LISTEN 14.4.4.44 3493 LISTEN 127.0.0.1 3493 MAXAGE 15 ``` #### `/etc/nut/upsd.users` ```ini [admin] password = SUPER_SECRET_ADMIN_PASS actions = SET instcmds = ALL [upsmon_master] password = MASTER_MONITOR_PASS upsmon master [upsmon_slave] password = SLAVE_MONITOR_PASS upsmon slave [peanut] password = PEANUT_MONITOR_PASS upsmon slave ``` #### `/etc/nut/upsmon.conf` ```ini MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master MINSUPPLIES 1 SHUTDOWNCMD "/sbin/shutdown -h +0" NOTIFYCMD /usr/local/bin/nut-notify.sh POWERDOWNFLAG /etc/killpower NOTIFYFLAG ONLINE SYSLOG+WALL+EXEC NOTIFYFLAG ONBATT SYSLOG+WALL+EXEC NOTIFYFLAG LOWBATT SYSLOG+WALL+EXEC NOTIFYFLAG FSD SYSLOG+WALL+EXEC NOTIFYFLAG COMMBAD SYSLOG+WALL+EXEC NOTIFYFLAG COMMOK SYSLOG+WALL+EXEC ``` --- ### B. Client Node Configuration (`upsmon` Slave) Applies to Harvester hypervisors, K3s master/worker hosts, Docker Swarm hosts, and Debian Linux nodes. #### `/etc/nut/upsmon.conf` (Slave Mode) ```ini MONITOR snarf-ups@14.4.4.44 1 upsmon_slave SLAVE_MONITOR_PASS slave MINSUPPLIES 1 SHUTDOWNCMD "/sbin/shutdown -h +0" POWERDOWNFLAG /etc/killpower HOSTSYNC 15 DEADTIME 15 RBWARNTIME 43200 NOCOMMWARNTIME 300 FINALDELAY 5 ``` --- ### C. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`) 1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**. 2. Check **Enable UPS Support**. 3. Select UPS Type: **Synology UPS Server / Network NUT Server**. 4. Network UPS server IP address: **`14.4.4.44`**. 5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `5 minutes` (or `Upon low battery`). --- ### D. Peanut Web Dashboard (`peanut.snarfnet.net`) Peanut runs on Docker Swarm (`140.44.4.70:8050`). Ensure the Peanut service configuration references the NUT server at `14.4.4.44`: #### Updated `docker-deployments/peanut.yml` ```yaml services: peanut: image: brandawg93/peanut:latest ports: - "8050:8050" environment: - TZ=America/New_York - NUT_HOST=14.4.4.44 - NUT_PORT=3493 - NUT_USER=peanut - NUT_PASSWORD=PEANUT_MONITOR_PASS volumes: - peanut_config:/config volumes: peanut_config: driver: local driver_opts: type: none o: bind device: /mnt/swarm_shared/legacy_volumes/peanut/ ``` --- ### E. Home Assistant (`14.4.4.20`) Integration & Automation 1. Navigate to **Settings** $\rightarrow$ **Devices & Services** $\rightarrow$ **Add Integration** $\rightarrow$ **Network UPS Tools (NUT)**. 2. Host: **`14.4.4.44`**, Port: **`3493`**. 3. Username: **`upsmon_slave`**, Password: **`SLAVE_MONITOR_PASS`**. #### Automation Snippet (`automations.yaml`) ```yaml alias: "Power Resilience: Grid Power Lost Alert" trigger: - platform: state entity_id: sensor.snarf_ups_status to: "On Battery" action: - service: notify.gotify data: title: "⚡ Power Outage Alert" message: "Grid power failure detected! Homelab is running on UPS battery (14.4.4.44). Tiered shutdown in progress." priority: 8 ``` --- ## 5. Power Restoration & Auto-Boot Sequence When grid power returns, systems auto-recover in reverse order: 1. **UPS AC Mains Restored**: Physical UPS re-enables output outlets once battery reaches safe threshold ($\ge 20\%$). 2. **Phase 1 - Gateway & Edge**: UDM Pro (`14.44.4.1`) & Core switches boot up and establish internal VLAN routes. 3. **Phase 2 - NUT Master**: Host at `14.4.4.44` boots (BIOS `Restore AC Power` set to `Always On`). `upsd` service starts. 4. **Phase 3 - Storage NAS**: Synology NAS nodes (`14.10.10.71`, `14.10.10.72`) auto-boot upon power recovery. 5. **Phase 4 - Control Planes & Hypervisors**: - Harvester `spacedock` hosts (`140.44.4.81-83`) boot via Wake-on-LAN / IPMI / BIOS settings. - K3s master nodes (`aerodrome` & `spaceport`) boot, resume etcd quorum. 6. **Phase 5 - Worker Compute & Workloads**: Worker nodes boot; Kubernetes and Docker Swarm automatically reschedule workloads.