264 lines
9.4 KiB
Markdown
264 lines
9.4 KiB
Markdown
# Snarfnet Homelab Power Resilience & UPS Automation Plan
|
|
|
|
This document details the power resilience strategy, tiered graceful shutdown sequence, and integration architecture for the **snarfnet** homelab centered around the primary **NUT (Network UPS Tools) Master Server**.
|
|
|
|
---
|
|
|
|
## 1. Core Architecture & IP Assignments
|
|
|
|
The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
|
|
|
|
| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description |
|
|
|---|---|---|---|---|
|
|
| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. |
|
|
| **NUT Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Compute / Swarm) | Peanut container UI consuming `14.4.4.44:3493`. |
|
|
| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for alerts/triggers. |
|
|
| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493`. |
|
|
| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. |
|
|
| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. |
|
|
|
|
---
|
|
|
|
## 2. Power Resilience Topology Map
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
subgraph Grid ["Utility Power Grid"]
|
|
Mains["Main AC Power Supply"]
|
|
end
|
|
|
|
subgraph PowerHardware ["Power Protection"]
|
|
UPS["Central UPS Hardware"]
|
|
end
|
|
|
|
subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"]
|
|
UPSD["upsd Daemon (Port 3493)"]
|
|
UPSMON_M["upsmon (Master)"]
|
|
end
|
|
|
|
subgraph Dashboards ["Monitoring & Alerting"]
|
|
Peanut["Peanut Dashboard\n(peanut.snarfnet.net:8050)"]
|
|
HA["Home Assistant\n(14.4.4.20)"]
|
|
Zabbix["Zabbix Server\n(140.44.4.61)"]
|
|
end
|
|
|
|
subgraph Slaves ["UPS Slaves (upsmon Client Nodes)"]
|
|
Harvester["Spacedock HCI Nodes\n(140.44.4.81 - 83)"]
|
|
K3sControl["K3s Control Planes\n(Aerodrome & Spaceport)"]
|
|
K3sWorkers["K3s Worker Nodes"]
|
|
Swarm["Shipyard Swarm Nodes\n(140.44.4.71 - 75)"]
|
|
NAS["Synology NAS Arrays\n(14.10.10.71 / 72)"]
|
|
end
|
|
|
|
Mains --> UPS
|
|
UPS -- USB / SNMP --> UPSD
|
|
UPSD --> UPSMON_M
|
|
|
|
UPSD -- TCP 3493 --> Peanut
|
|
UPSD -- TCP 3493 --> HA
|
|
UPSD -- TCP 3493 --> Zabbix
|
|
|
|
UPSD -- TCP 3493 (Slave Broadcast) --> Harvester
|
|
UPSD -- TCP 3493 (Slave Broadcast) --> K3sControl
|
|
UPSD -- TCP 3493 (Slave Broadcast) --> K3sWorkers
|
|
UPSD -- TCP 3493 (Slave Broadcast) --> Swarm
|
|
UPSD -- TCP 3493 (Slave Broadcast) --> NAS
|
|
```
|
|
|
|
---
|
|
|
|
## 3. Tiered Shutdown Hierarchy & Execution Timeline
|
|
|
|
To prevent data corruption and unexpected hard cutoffs, hosts shutdown in staged tiers based on power outage duration and battery level.
|
|
|
|
```
|
|
+-------------------------------------------------------------------------+
|
|
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT |
|
|
+-------------------------------------------------------------------------+
|
|
|
|
|
v
|
|
+-------------------------------------------------------------------------+
|
|
| [T+3m / Batt <= 80%] TIER 1: Non-essential Workloads |
|
|
| - Dev VMs, transcoders, test containers in Swarm & K3s drained |
|
|
+-------------------------------------------------------------------------+
|
|
|
|
|
v
|
|
+-------------------------------------------------------------------------+
|
|
| [T+7m / Batt <= 50%] TIER 2: Hypervisors, Compute Workers & Storage |
|
|
| - Harvester spacedock VMs gracefully shut down via virsh/API |
|
|
| - Aerodrome & Spaceport worker nodes gracefully halt |
|
|
| - Synology NAS arrays (14.10.10.71/72) enter Safe Mode & unmount pools |
|
|
+-------------------------------------------------------------------------+
|
|
|
|
|
v
|
|
+-------------------------------------------------------------------------+
|
|
| [T+12m / Batt <= 30%] TIER 3: Control Planes & Management Stack |
|
|
| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Control Planes|
|
|
| - Swarm Manager (140.44.4.71) & AI Nodes (140.44.4.7-8) halt |
|
|
+-------------------------------------------------------------------------+
|
|
|
|
|
v
|
|
+-------------------------------------------------------------------------+
|
|
| [T+15m / Batt Low] TIER 4: Forced Shutdown (FSD) & Hardware Cutoff |
|
|
| - NUT Master (14.4.4.44) issues FSD signal to remaining clients |
|
|
| - NUT Master issues `upsdrvctl shutdown` (cuts UPS load outlet) |
|
|
| - Network Core (UDM Pro 14.44.4.1) stays alive until final battery drop |
|
|
+-------------------------------------------------------------------------+
|
|
```
|
|
|
|
---
|
|
|
|
## 4. Configuration Blueprint & Service Integrations
|
|
|
|
### A. NUT Master Server (`14.4.4.44`) Configuration Files
|
|
|
|
#### `/etc/nut/ups.conf`
|
|
```ini
|
|
[snarf-ups]
|
|
driver = usbhid-ups
|
|
port = auto
|
|
desc = "Snarfnet Main Server Rack UPS"
|
|
pollinterval = 5
|
|
```
|
|
|
|
#### `/etc/nut/upsd.conf`
|
|
```ini
|
|
LISTEN 14.4.4.44 3493
|
|
LISTEN 127.0.0.1 3493
|
|
MAXAGE 15
|
|
```
|
|
|
|
#### `/etc/nut/upsd.users`
|
|
```ini
|
|
[admin]
|
|
password = SUPER_SECRET_ADMIN_PASS
|
|
actions = SET
|
|
instcmds = ALL
|
|
|
|
[upsmon_master]
|
|
password = MASTER_MONITOR_PASS
|
|
upsmon master
|
|
|
|
[upsmon_slave]
|
|
password = SLAVE_MONITOR_PASS
|
|
upsmon slave
|
|
|
|
[peanut]
|
|
password = PEANUT_MONITOR_PASS
|
|
upsmon slave
|
|
```
|
|
|
|
#### `/etc/nut/upsmon.conf`
|
|
```ini
|
|
MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master
|
|
MINSUPPLIES 1
|
|
SHUTDOWNCMD "/sbin/shutdown -h +0"
|
|
NOTIFYCMD /usr/local/bin/nut-notify.sh
|
|
POWERDOWNFLAG /etc/killpower
|
|
|
|
NOTIFYFLAG ONLINE SYSLOG+WALL+EXEC
|
|
NOTIFYFLAG ONBATT SYSLOG+WALL+EXEC
|
|
NOTIFYFLAG LOWBATT SYSLOG+WALL+EXEC
|
|
NOTIFYFLAG FSD SYSLOG+WALL+EXEC
|
|
NOTIFYFLAG COMMBAD SYSLOG+WALL+EXEC
|
|
NOTIFYFLAG COMMOK SYSLOG+WALL+EXEC
|
|
```
|
|
|
|
---
|
|
|
|
### B. Client Node Configuration (`upsmon` Slave)
|
|
|
|
Applies to Harvester hypervisors, K3s master/worker hosts, Docker Swarm hosts, and Debian Linux nodes.
|
|
|
|
#### `/etc/nut/upsmon.conf` (Slave Mode)
|
|
```ini
|
|
MONITOR snarf-ups@14.4.4.44 1 upsmon_slave SLAVE_MONITOR_PASS slave
|
|
MINSUPPLIES 1
|
|
SHUTDOWNCMD "/sbin/shutdown -h +0"
|
|
POWERDOWNFLAG /etc/killpower
|
|
HOSTSYNC 15
|
|
DEADTIME 15
|
|
RBWARNTIME 43200
|
|
NOCOMMWARNTIME 300
|
|
FINALDELAY 5
|
|
```
|
|
|
|
---
|
|
|
|
### C. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`)
|
|
|
|
1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**.
|
|
2. Check **Enable UPS Support**.
|
|
3. Select UPS Type: **Synology UPS Server / Network NUT Server**.
|
|
4. Network UPS server IP address: **`14.4.4.44`**.
|
|
5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `5 minutes` (or `Upon low battery`).
|
|
|
|
---
|
|
|
|
### D. Peanut Web Dashboard (`peanut.snarfnet.net`)
|
|
|
|
Peanut runs on Docker Swarm (`140.44.4.70:8050`). Ensure the Peanut service configuration references the NUT server at `14.4.4.44`:
|
|
|
|
#### Updated `docker-deployments/peanut.yml`
|
|
```yaml
|
|
services:
|
|
peanut:
|
|
image: brandawg93/peanut:latest
|
|
ports:
|
|
- "8050:8050"
|
|
environment:
|
|
- TZ=America/New_York
|
|
- NUT_HOST=14.4.4.44
|
|
- NUT_PORT=3493
|
|
- NUT_USER=peanut
|
|
- NUT_PASSWORD=PEANUT_MONITOR_PASS
|
|
volumes:
|
|
- peanut_config:/config
|
|
|
|
volumes:
|
|
peanut_config:
|
|
driver: local
|
|
driver_opts:
|
|
type: none
|
|
o: bind
|
|
device: /mnt/swarm_shared/legacy_volumes/peanut/
|
|
```
|
|
|
|
---
|
|
|
|
### E. Home Assistant (`14.4.4.20`) Integration & Automation
|
|
|
|
1. Navigate to **Settings** $\rightarrow$ **Devices & Services** $\rightarrow$ **Add Integration** $\rightarrow$ **Network UPS Tools (NUT)**.
|
|
2. Host: **`14.4.4.44`**, Port: **`3493`**.
|
|
3. Username: **`upsmon_slave`**, Password: **`SLAVE_MONITOR_PASS`**.
|
|
|
|
#### Automation Snippet (`automations.yaml`)
|
|
```yaml
|
|
alias: "Power Resilience: Grid Power Lost Alert"
|
|
trigger:
|
|
- platform: state
|
|
entity_id: sensor.snarf_ups_status
|
|
to: "On Battery"
|
|
action:
|
|
- service: notify.gotify
|
|
data:
|
|
title: "⚡ Power Outage Alert"
|
|
message: "Grid power failure detected! Homelab is running on UPS battery (14.4.4.44). Tiered shutdown in progress."
|
|
priority: 8
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Power Restoration & Auto-Boot Sequence
|
|
|
|
When grid power returns, systems auto-recover in reverse order:
|
|
|
|
1. **UPS AC Mains Restored**: Physical UPS re-enables output outlets once battery reaches safe threshold ($\ge 20\%$).
|
|
2. **Phase 1 - Gateway & Edge**: UDM Pro (`14.44.4.1`) & Core switches boot up and establish internal VLAN routes.
|
|
3. **Phase 2 - NUT Master**: Host at `14.4.4.44` boots (BIOS `Restore AC Power` set to `Always On`). `upsd` service starts.
|
|
4. **Phase 3 - Storage NAS**: Synology NAS nodes (`14.10.10.71`, `14.10.10.72`) auto-boot upon power recovery.
|
|
5. **Phase 4 - Control Planes & Hypervisors**:
|
|
- Harvester `spacedock` hosts (`140.44.4.81-83`) boot via Wake-on-LAN / IPMI / BIOS settings.
|
|
- K3s master nodes (`aerodrome` & `spaceport`) boot, resume etcd quorum.
|
|
6. **Phase 5 - Worker Compute & Workloads**: Worker nodes boot; Kubernetes and Docker Swarm automatically reschedule workloads.
|