Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
6b12646a0f682bd04882660734dc001dbfba0933
SnarfCode/docs/power_resilience_plan.md

9.4 KiB
Raw Blame History

Snarfnet Homelab Power Resilience & UPS Automation Plan

This document details the power resilience strategy, tiered graceful shutdown sequence, and integration architecture for the snarfnet homelab centered around the primary NUT (Network UPS Tools) Master Server.

1. Core Architecture & IP Assignments

The power resilience architecture uses a centralized NUT Master Server host that polls connected UPS hardware and broadcasts status alerts (ONBATT, LOWBATT, ONLINE, FSD) across all homelab VLANs.

Role Hostname / Identifier Target IP Subnet / VLAN Description
NUT Master Server nutserver.snarfnet.net 14.4.4.44 VLAN 50 (Smart Home) Primary NUT upsd daemon polling physical UPS via USB/Serial/SNMP.
NUT Web Dashboard peanut.snarfnet.net 140.44.4.70:8050 VLAN 20 (Compute / Swarm) Peanut container UI consuming 14.4.4.44:3493.
Smart Home Automation homeassistant.snarfnet.net 14.4.4.20 VLAN 50 (Smart Home) Home Assistant integration monitoring 14.4.4.44:3493 for alerts/triggers.
Centralized Monitoring zabbix.snarfnet.net 140.44.4.61 VLAN 20 (Spaceport K3s) Zabbix Agent 2 / NUT exporter polling 14.4.4.44:3493.
Primary Storage 1 sn-nas.snarfnet.net 14.10.10.71 VLAN 40 (Storage) Synology DSM UPS client polling 14.4.4.44.
Primary Storage 2 snarfnas.snarfnet.net 14.10.10.72 VLAN 40 (Storage) Synology DSM UPS client polling 14.4.4.44.

2. Power Resilience Topology Map

flowchart TD
    subgraph Grid ["Utility Power Grid"]
        Mains["Main AC Power Supply"]
    end

    subgraph PowerHardware ["Power Protection"]
        UPS["Central UPS Hardware"]
    end

    subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"]
        UPSD["upsd Daemon (Port 3493)"]
        UPSMON_M["upsmon (Master)"]
    end

    subgraph Dashboards ["Monitoring & Alerting"]
        Peanut["Peanut Dashboard\n(peanut.snarfnet.net:8050)"]
        HA["Home Assistant\n(14.4.4.20)"]
        Zabbix["Zabbix Server\n(140.44.4.61)"]
    end

    subgraph Slaves ["UPS Slaves (upsmon Client Nodes)"]
        Harvester["Spacedock HCI Nodes\n(140.44.4.81 - 83)"]
        K3sControl["K3s Control Planes\n(Aerodrome & Spaceport)"]
        K3sWorkers["K3s Worker Nodes"]
        Swarm["Shipyard Swarm Nodes\n(140.44.4.71 - 75)"]
        NAS["Synology NAS Arrays\n(14.10.10.71 / 72)"]
    end

    Mains --> UPS
    UPS -- USB / SNMP --> UPSD
    UPSD --> UPSMON_M
    
    UPSD -- TCP 3493 --> Peanut
    UPSD -- TCP 3493 --> HA
    UPSD -- TCP 3493 --> Zabbix
    
    UPSD -- TCP 3493 (Slave Broadcast) --> Harvester
    UPSD -- TCP 3493 (Slave Broadcast) --> K3sControl
    UPSD -- TCP 3493 (Slave Broadcast) --> K3sWorkers
    UPSD -- TCP 3493 (Slave Broadcast) --> Swarm
    UPSD -- TCP 3493 (Slave Broadcast) --> NAS

3. Tiered Shutdown Hierarchy & Execution Timeline

To prevent data corruption and unexpected hard cutoffs, hosts shutdown in staged tiers based on power outage duration and battery level.

+-------------------------------------------------------------------------+
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT       |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| [T+3m / Batt <= 80%] TIER 1: Non-essential Workloads                    |
| - Dev VMs, transcoders, test containers in Swarm & K3s drained          |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| [T+7m / Batt <= 50%] TIER 2: Hypervisors, Compute Workers & Storage     |
| - Harvester spacedock VMs gracefully shut down via virsh/API            |
| - Aerodrome & Spaceport worker nodes gracefully halt                    |
| - Synology NAS arrays (14.10.10.71/72) enter Safe Mode & unmount pools |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| [T+12m / Batt <= 30%] TIER 3: Control Planes & Management Stack         |
| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Control Planes|
| - Swarm Manager (140.44.4.71) & AI Nodes (140.44.4.7-8) halt             |
+-------------------------------------------------------------------------+
                                    |
                                    v
+-------------------------------------------------------------------------+
| [T+15m / Batt Low] TIER 4: Forced Shutdown (FSD) & Hardware Cutoff      |
| - NUT Master (14.4.4.44) issues FSD signal to remaining clients         |
| - NUT Master issues `upsdrvctl shutdown` (cuts UPS load outlet)         |
| - Network Core (UDM Pro 14.44.4.1) stays alive until final battery drop  |
+-------------------------------------------------------------------------+

4. Configuration Blueprint & Service Integrations

A. NUT Master Server (14.4.4.44) Configuration Files

/etc/nut/ups.conf

[snarf-ups]
    driver = usbhid-ups
    port = auto
    desc = "Snarfnet Main Server Rack UPS"
    pollinterval = 5

/etc/nut/upsd.conf

LISTEN 14.4.4.44 3493
LISTEN 127.0.0.1 3493
MAXAGE 15

/etc/nut/upsd.users

[admin]
    password = SUPER_SECRET_ADMIN_PASS
    actions = SET
    instcmds = ALL

[upsmon_master]
    password = MASTER_MONITOR_PASS
    upsmon master

[upsmon_slave]
    password = SLAVE_MONITOR_PASS
    upsmon slave

[peanut]
    password = PEANUT_MONITOR_PASS
    upsmon slave

/etc/nut/upsmon.conf

MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
NOTIFYCMD /usr/local/bin/nut-notify.sh
POWERDOWNFLAG /etc/killpower

NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC
NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC
NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC
NOTIFYFLAG FSD      SYSLOG+WALL+EXEC
NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC
NOTIFYFLAG COMMOK   SYSLOG+WALL+EXEC

B. Client Node Configuration (upsmon Slave)

Applies to Harvester hypervisors, K3s master/worker hosts, Docker Swarm hosts, and Debian Linux nodes.

/etc/nut/upsmon.conf (Slave Mode)

MONITOR snarf-ups@14.4.4.44 1 upsmon_slave SLAVE_MONITOR_PASS slave
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POWERDOWNFLAG /etc/killpower
HOSTSYNC 15
DEADTIME 15
RBWARNTIME 43200
NOCOMMWARNTIME 300
FINALDELAY 5

C. Synology NAS Integration (14.10.10.71 & 14.10.10.72)

  1. Log into DSM Console \rightarrow Control Panel \rightarrow Hardware & Power \rightarrow UPS.
  2. Check Enable UPS Support.
  3. Select UPS Type: Synology UPS Server / Network NUT Server.
  4. Network UPS server IP address: 14.4.4.44.
  5. Set Time before Synology NAS enters Standby Mode: Customize time \rightarrow 5 minutes (or Upon low battery).

D. Peanut Web Dashboard (peanut.snarfnet.net)

Peanut runs on Docker Swarm (140.44.4.70:8050). Ensure the Peanut service configuration references the NUT server at 14.4.4.44:

Updated docker-deployments/peanut.yml

services:
  peanut:
    image: brandawg93/peanut:latest
    ports:
      - "8050:8050"
    environment:
      - TZ=America/New_York
      - NUT_HOST=14.4.4.44
      - NUT_PORT=3493
      - NUT_USER=peanut
      - NUT_PASSWORD=PEANUT_MONITOR_PASS
    volumes:
      - peanut_config:/config

volumes:
  peanut_config:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/swarm_shared/legacy_volumes/peanut/

E. Home Assistant (14.4.4.20) Integration & Automation

  1. Navigate to Settings \rightarrow Devices & Services \rightarrow Add Integration \rightarrow Network UPS Tools (NUT).
  2. Host: 14.4.4.44, Port: 3493.
  3. Username: upsmon_slave, Password: SLAVE_MONITOR_PASS.

Automation Snippet (automations.yaml)

alias: "Power Resilience: Grid Power Lost Alert"
trigger:
  - platform: state
    entity_id: sensor.snarf_ups_status
    to: "On Battery"
action:
  - service: notify.gotify
    data:
      title: "⚡ Power Outage Alert"
      message: "Grid power failure detected! Homelab is running on UPS battery (14.4.4.44). Tiered shutdown in progress."
      priority: 8

5. Power Restoration & Auto-Boot Sequence

When grid power returns, systems auto-recover in reverse order:

  1. UPS AC Mains Restored: Physical UPS re-enables output outlets once battery reaches safe threshold (\ge 20\%).
  2. Phase 1 - Gateway & Edge: UDM Pro (14.44.4.1) & Core switches boot up and establish internal VLAN routes.
  3. Phase 2 - NUT Master: Host at 14.4.4.44 boots (BIOS Restore AC Power set to Always On). upsd service starts.
  4. Phase 3 - Storage NAS: Synology NAS nodes (14.10.10.71, 14.10.10.72) auto-boot upon power recovery.
  5. Phase 4 - Control Planes & Hypervisors:
    • Harvester spacedock hosts (140.44.4.81-83) boot via Wake-on-LAN / IPMI / BIOS settings.
    • K3s master nodes (aerodrome & spaceport) boot, resume etcd quorum.
  6. Phase 5 - Worker Compute & Workloads: Worker nodes boot; Kubernetes and Docker Swarm automatically reschedule workloads.
Reference in New Issue View Git Blame Copy Permalink