Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
c492afc377e310a75948d5c954c726c9bbdbddcd
SnarfCode/docs/power_resilience_plan.md

20 KiB
Raw Blame History

Snarfnet Homelab Power Resilience & UPS Automation Plan

This document details the comprehensive power resilience strategy, tiered graceful shutdown sequence, and automated recovery orchestration for the snarfnet homelab. The system is centered around a dedicated NUT (Network UPS Tools) Master Server at 14.4.4.44.

1. Core Architecture & Component Roles

The power resilience architecture uses a centralized NUT Master Server host that polls connected UPS hardware and broadcasts status alerts (ONBATT, LOWBATT, ONLINE, FSD) across all homelab VLANs.

Role Hostname / Identifier Target IP Subnet / VLAN Description & Failure Protocol
NUT Master Server nutserver.snarfnet.net 14.4.4.44 VLAN 50 (Smart Home) Primary NUT upsd daemon polling physical UPS via USB/Serial/SNMP. Runs upsmon master.
Edge Ingress / Reverse Proxy nginx.snarfnet.net 140.44.4.74 VLAN 20 (Swarm 04) NPM running in PublishMode: host on ports 80/443/81. Dependent on Synology NFS; must be stopped prior to NAS Safe Mode.
Authentication & SSO authentik.snarfnet.net 140.44.4.29:444 VLAN 20 (Aerodrome K3s) Handles Forward Auth for Peanut, Gotify, Homebox, WLED. Runs on Longhorn storage in Aerodrome.
UPS Web Dashboard peanut.snarfnet.net 140.44.4.70:8050 VLAN 20 (Swarm VIP) Peanut container UI consuming 14.4.4.44:3493. Protected by Authentik Forward Auth externally.
Emergency Push Notifications gotify.snarfnet.net 140.44.4.70:8090 VLAN 20 (Swarm VIP) Gotify server. Internal outage alerts MUST use direct IP (http://140.44.4.70:8090) to bypass Authentik during shutdown.
Smart Home Automation homeassistant.snarfnet.net 14.4.4.20 VLAN 50 (Smart Home) Home Assistant integration monitoring 14.4.4.44:3493 for state changes, critical push alerts, and automated shedding.
Centralized Monitoring zabbix.snarfnet.net 140.44.4.61 VLAN 20 (Spaceport K3s) Zabbix Agent 2 / NUT exporter polling 14.4.4.44:3493 for battery voltage, load %, and runtime metrics.
Primary Storage 1 sn-nas.snarfnet.net 14.10.10.71 VLAN 40 (Storage) Synology DSM UPS client polling 14.4.4.44. Hosts NPM NFS shares (/volume1/docker/nginx/data).
Primary Storage 2 snarfnas.snarfnet.net 14.10.10.72 VLAN 40 (Storage) Synology DSM UPS client polling 14.4.4.44. Hosts secondary backup shares.
Distributed Block Storage longhorn.aerodrome.snarfnet.net 140.44.4.36:80 VLAN 20 (Aerodrome K3s) Longhorn CSI storage engine powering Authentik, AWX, Grafana PVCs. Protected with recurring snapshots (snapshot-4h, snapshot-daily).

Important

Emergency Alerting Path Warning: peanut.snarfnet.net and gotify.snarfnet.net are protected externally behind Authentik Forward Auth (oauth2-proxy). During a power outage, when K3s aerodrome shuts down in Tier 3, Authentik goes offline. Therefore, emergency outage alerts and NUT notification scripts must communicate directly with Gotify via internal IP (http://140.44.4.70:8090/message?token=...) or native Home Assistant push notifications, bypassing NPM and Authentik.

2. Power Resilience Topology Map

flowchart TD
    subgraph Grid ["Utility Power Grid"]
        Mains["Main AC Utility Power"]
    end

    subgraph PowerHardware ["Power Hardware & Protection"]
        UPS["Central UPS Hardware (Battery Bank)"]
    end

    subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"]
        UPSD["upsd Daemon (Port 3493)"]
        UPSMON_M["upsmon (Master Daemon)"]
        NOTIFY["nut-notify.sh (Direct Alert Engine)"]
    end

    subgraph OutageAlerts ["Emergency Notification Channels"]
        HA_Alert["Home Assistant (14.4.4.20)\nLocal Push Notification"]
        Gotify_Direct["Gotify Internal Direct\nhttp://140.44.4.70:8090\n(Bypasses Authentik / NPM)"]
        Syslog["Wall / Syslog Broadcast"]
    end

    subgraph MonitoringPanels ["Monitoring & Management"]
        Peanut["Peanut Dashboard\n(peanut.snarfnet.net)"]
        Zabbix["Zabbix Server\n(140.44.4.61)"]
    end

    subgraph InfrastructureConsumers ["Homelab Tiered Consumers"]
        SwarmNFS["Docker Swarm (140.44.4.71-75)\nNPM Host Mode (140.44.4.74)"]
        K8sCompute["Aerodrome & Spaceport K3s\nLonghorn Engine & Worker Nodes"]
        NAS["Synology NAS Arrays\n(14.10.10.71 / 14.10.10.72)"]
        K8sControl["K3s Control Planes & etcd"]
        Harvester["Spacedock HCI Hypervisors\n(140.44.4.81 - 83)"]
        CoreEdge["UDM Pro Gateway (14.44.4.1)\nCore PoE Switches"]
    end

    Mains --> UPS
    UPS -- USB / SNMP --> UPSD
    UPSD --> UPSMON_M
    UPSMON_M --> NOTIFY

    NOTIFY -- Direct API Token --> Gotify_Direct
    NOTIFY -- TCP 3493 Event --> HA_Alert
    NOTIFY --> Syslog

    UPSD -- TCP 3493 --> Peanut
    UPSD -- TCP 3493 --> Zabbix

    UPSMON_M -. Tiered Signal .-> SwarmNFS
    UPSMON_M -. Tiered Signal .-> K8sCompute
    UPSMON_M -. Tiered Signal .-> NAS
    UPSMON_M -. Tiered Signal .-> K8sControl
    UPSMON_M -. Tiered Signal .-> Harvester
    UPSMON_M -. Final Cut .-> CoreEdge

3. Tiered Graceful Shutdown Hierarchy

To eliminate data corruption, prevent stale NFS file handles, and ensure Longhorn volume detach cleanly, the homelab executes a strict 6-stage progressive shutdown sequence.

+--------------------------------------------------------------------------------------------------+
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT                                |
| - Broadcasts emergency alert via Home Assistant push & Gotify direct API (http://140.44.4.70:8090)|
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads                                  |
| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped      |
| - Media transcoders (Plex active sessions signaled)                                              |
| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted                         |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM / Peanut)                    |
| - Stop `nginx-proxy-manager_app` & `nginx-proxy-manager_backup` on snshpd4lnxswm04 (140.44.4.74) |
| - Stop `peanut` container on Swarm                                                               |
| - Flushes open SQLite databases (database.sqlite) and unmounts NFS mounts to 14.10.10.71        |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach             |
| - Harvester guest VMs (harvester.spacedock.snarfnet.net 140.44.4.90) gracefully shut down via API|
| - Cordon & scale down stateful pods on Aerodrome (Authentik PostgreSQL, AWX Postgres, Grafana)   |
| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes     |
| - Aerodrome (140.44.4.21-25) & Spaceport (140.44.4.51-56) worker nodes execute shutdown          |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode)                            |
| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode via NUT client trigger      |
| - Unmounts all Btrfs volume pools, terminates NFS/iSCSI daemon threads, parks drive read heads    |
| - Zero active network clients remain connected to storage                                        |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors                             |
| - Aerodrome K3s Control Planes (140.44.4.11-13) take etcd snapshot and halt                     |
| - Spaceport K3s Control Planes (140.44.4.41-44) take etcd snapshot and halt                     |
| - Swarm Manager nodes (140.44.4.71-75) halt                                                      |
| - Harvester baremetal hypervisors (140.44.4.81-83) halt via IPMI/SSH                             |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut                               |
| - NUT Master (14.4.4.44) issues final FSD broadcast to all remaining clients                     |
| - NUT Master issues `upsdrvctl shutdown` with /etc/killpower flag (commands UPS load drop)        |
| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops   |
+--------------------------------------------------------------------------------------------------+

4. Configuration Blueprints & Integration Scripts

A. NUT Master Server (14.4.4.44)

/etc/nut/ups.conf

[snarf-ups]
    driver = usbhid-ups
    port = auto
    desc = "Snarfnet Main Rack UPS"
    pollinterval = 5

/etc/nut/upsd.conf

LISTEN 14.4.4.44 3493
LISTEN 127.0.0.1 3493
MAXAGE 15

/etc/nut/upsd.users

[admin]
    password = ADMIN_PASSWORD_CHANGE_ME
    actions = SET
    instcmds = ALL

[upsmon_master]
    password = MASTER_MONITOR_PASS
    upsmon master

[upsmon_slave]
    password = SLAVE_MONITOR_PASS
    upsmon slave

[peanut]
    password = PEANUT_MONITOR_PASS
    upsmon slave

/etc/nut/upsmon.conf

MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master
MINSUPPLIES 1
SHUTDOWNCMD "/usr/local/bin/homelab_power_shutdown.sh"
NOTIFYCMD /usr/local/bin/nut-notify.sh
POWERDOWNFLAG /etc/killpower

NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC
NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC
NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC
NOTIFYFLAG FSD      SYSLOG+WALL+EXEC
NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC
NOTIFYFLAG COMMOK   SYSLOG+WALL+EXEC

B. Outage Notification Engine (/usr/local/bin/nut-notify.sh)

This script runs on the NUT Master (14.4.4.44) whenever a power state event triggers. It sends push notifications directly to Gotify (http://140.44.4.70:8090) and Home Assistant (http://14.4.4.20:8123) without touching NPM or Authentik.

#!/bin/bash
# /usr/local/bin/nut-notify.sh
# Emergency power notification script for Network UPS Tools
set -euo pipefail

EVENT_TYPE="${1:-$NOTIFYTYPE}"
UPS_NAME="${UPSNAME:-snarf-ups}"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

GOTIFY_INTERNAL_URL="http://140.44.4.70:8090/message"
GOTIFY_APP_TOKEN="A4uN4fEmErGeNcYtOkEn" # Replace with valid Gotify App Token

send_gotify() {
    local title="$1"
    local message="$2"
    local priority="$3"
    
    curl -s -m 5 -X POST "${GOTIFY_INTERNAL_URL}?token=${GOTIFY_APP_TOKEN}" \
        -H "Content-Type: application/json" \
        -d "{\"title\": \"${title}\", \"message\": \"${message}\", \"priority\": ${priority}}" >/dev/null 2>&1 || true
}

case "$EVENT_TYPE" in
    ONBATT)
        send_gotify "⚡ Grid Power Lost" "[$TIMESTAMP] Homelab is running on UPS battery (${UPS_NAME}). Staged shutdown countdown started." 8
        logger -t nut-notify "Grid power failure detected - running on UPS battery."
        ;;
    ONLINE)
        send_gotify "✅ Grid Power Restored" "[$TIMESTAMP] Utility AC power restored to ${UPS_NAME}. Battery recharging." 5
        logger -t nut-notify "AC utility power restored."
        ;;
    LOWBATT)
        send_gotify "🚨 UPS Battery Critical" "[$TIMESTAMP] Battery level critical on ${UPS_NAME}! Final Tier 4-6 shutdown initiated." 10
        logger -t nut-notify "CRITICAL: UPS battery low. Enforcing emergency shutdown."
        ;;
    FSD)
        send_gotify "🛑 Forced Shutdown Engaged" "[$TIMESTAMP] NUT Master has commanded immediate power cutoff across all clusters." 10
        logger -t nut-notify "FSD signal received. Halting systems."
        ;;
    *)
        logger -t nut-notify "Unhandled UPS event: ${EVENT_TYPE}"
        ;;
esac

C. Orchestrated Tiered Shutdown Script (/usr/local/bin/homelab_power_shutdown.sh)

This script is invoked by upsmon on 14.4.4.44 when an outage exceeds the battery threshold:

#!/bin/bash
# /usr/local/bin/homelab_power_shutdown.sh
# Staged graceful homelab shutdown orchestrator
set -uo pipefail

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [POWER-SHUTDOWN] $*" | tee -a /var/log/power-shutdown.log; }
SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes"

log "=== INITIATING HOMELAB TIERED SHUTDOWN ==="

# ------------------------------------------------------------------
# TIER 1: Non-Essential Compute & AI Pis
# ------------------------------------------------------------------
log "Stage 1: Halting AI Raspberry Pi stack (140.44.4.7, 140.44.4.8)..."
ssh $SSH_OPTS root@140.44.4.7 "shutdown -h now" 2>/dev/null || true
ssh $SSH_OPTS root@140.44.4.8 "shutdown -h now" 2>/dev/null || true

# ------------------------------------------------------------------
# TIER 2: Docker Swarm NFS Workloads (NPM & Peanut)
# ------------------------------------------------------------------
log "Stage 2: Stopping Swarm services mounting Synology NFS (NPM, Peanut)..."
ssh $SSH_OPTS root@140.44.4.71 "docker service scale nginx-proxy-manager_app=0 nginx-proxy-manager_backup=0 peanut_peanut=0" 2>/dev/null || true
sleep 5

# ------------------------------------------------------------------
# TIER 3: Harvester Guest VMs & Kubernetes Worker Nodes
# ------------------------------------------------------------------
log "Stage 3: Shutting down Harvester guest VMs and K3s workers..."
# Gracefully signal Harvester VMs
ssh $SSH_OPTS root@140.44.4.81 "virsh list --name | xargs -r -n1 virsh shutdown" 2>/dev/null || true

# Stop K3s agent on Aerodrome workers (clean Longhorn PVC detach)
for ip in 140.44.4.22 140.44.4.23 140.44.4.24 140.44.4.25; do
    ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
done

# Stop K3s agent on Spaceport workers
for ip in 140.44.4.51 140.44.4.52 140.44.4.53 140.44.4.54 140.44.4.55 140.44.4.56; do
    ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
done
sleep 15

# ------------------------------------------------------------------
# TIER 4: Storage Arrays (Synology DSM Safe Mode)
# ------------------------------------------------------------------
log "Stage 4: Triggering Synology NAS Safe Mode (14.10.10.71, 14.10.10.72)..."
# Synology enters safe mode automatically via NUT upsd client signal (port 3493)
# Explicit API / SSH command to flush pools if configured
ssh $SSH_OPTS admin@14.10.10.71 "synopoweroff" 2>/dev/null || true
ssh $SSH_OPTS admin@14.10.10.72 "synopoweroff" 2>/dev/null || true
sleep 20

# ------------------------------------------------------------------
# TIER 5: Control Planes & Swarm Managers
# ------------------------------------------------------------------
log "Stage 5: Halting K3s control planes and Swarm managers..."
# Spaceport Masters (140.44.4.41-44)
for ip in 140.44.4.41 140.44.4.42 140.44.4.43 140.44.4.44; do
    ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
done

# Aerodrome Masters (140.44.4.11-13)
for ip in 140.44.4.11 140.44.4.12 140.44.4.13; do
    ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
done

# Swarm Managers (140.44.4.71, 72, 74)
for ip in 140.44.4.71 140.44.4.72 140.44.4.74; do
    ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
done

# Harvester Baremetal Hypervisors (140.44.4.81-83)
for ip in 140.44.4.81 140.44.4.82 140.44.4.83; do
    ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
done

# ------------------------------------------------------------------
# TIER 6: Master Node Self-Cutoff (FSD)
# ------------------------------------------------------------------
log "Stage 6: Executing final local halt and arming killpower..."
touch /etc/killpower
/sbin/shutdown -h +0

D. Synology NAS Integration (14.10.10.71 & 14.10.10.72)

  1. Log into DSM Console \rightarrow Control Panel \rightarrow Hardware & Power \rightarrow UPS.
  2. Check Enable UPS Support.
  3. Select UPS Type: Synology UPS Server / Network NUT Server.
  4. Network UPS server IP address: 14.4.4.44.
  5. Set Time before Synology NAS enters Standby Mode: Customize time \rightarrow 6 minutes (guaranteeing that Tier 2 Swarm NFS workloads have cleanly scaled down before disk pools unmount).
  6. Check Shut down UPS when the system enters Standby Mode (disabled, managed centrally by 14.4.4.44).

E. Peanut Web Dashboard Configuration

Peanut runs on Docker Swarm (140.44.4.70:8050). It connects directly to the NUT master server over internal TCP 3493:

services:
  peanut:
    image: brandawg93/peanut:latest
    ports:
      - "8050:8050"
    environment:
      - TZ=America/New_York
      - NUT_HOST=14.4.4.44
      - NUT_PORT=3493
      - NUT_USER=peanut
      - NUT_PASSWORD=PEANUT_MONITOR_PASS
    volumes:
      - peanut_config:/config

volumes:
  peanut_config:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/swarm_shared/legacy_volumes/peanut/

5. Automated Power Restoration & Boot Sequence

When utility grid power returns, the homelab automatically recovers in a strict bottom-up reverse sequence to guarantee all storage and network prerequisites are ready before compute services attempt to start.

flowchart TD
    G_On["1. Grid AC Restored\nUPS Battery Reaches >= 30%"] --> P1["2. Phase 1: Core Network & Edge\nUDM Pro (14.44.4.1) & Core PoE Switches\nEstablish VLANs, DHCP, Inter-VLAN Routing"]
    P1 --> P2["3. Phase 2: NUT Master\nNode at 14.4.4.44 Boots (BIOS: Always On)\nupsd daemon starts, monitors line quality"]
    P2 --> P3["4. Phase 3: Storage Foundation\nSynology SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) Boot\nMount Btrfs pools, start NFS & iSCSI daemons"]
    P3 --> P4["5. Phase 4: Control Planes & Hypervisors\n- Harvester spacedock (140.44.4.81-83) boots via IPMI/WOL\n- Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) K3s masters boot\n- etcd clusters re-establish quorum"]
    P4 --> P5["6. Phase 5: Storage CSI & Compute Workers\n- Aerodrome & Spaceport worker nodes boot\n- Longhorn CSI engine initializes, verifies volume health, mounts PVCs"]
    P5 --> P6["7. Phase 6: Ingress & Workloads\n- Docker Swarm (140.44.4.71-75) boots\n- NPM host-mode container binds 80/443/81 on 140.44.4.74\n- Authentik SSO & Forward Auth come online\n- Applications (Peanut, Gotify, Homebox, etc.) become reachable"]

Critical Health Verification Gates:

  1. Storage Gate (Phase 3 \rightarrow Phase 4):
    • Synology NAS NFS daemons must be responding on 14.10.10.71:2049 before Docker Swarm starts, ensuring NPM does not encounter stale NFS mounts or SQLite locking.
  2. Database Gate (Phase 5 \rightarrow Phase 6):
    • Longhorn volume data-authentik-postgresql-0 on Aerodrome must reach Healthy state before Authentik server pods start.
  3. Ingress Gate (Phase 6):
    • Nginx Proxy Manager on snshpd4lnxswm04 (140.44.4.74) confirms host-mode ports 80 and 443 are bound and forwards traffic to internal VIPs.
Reference in New Issue View Git Blame Copy Permalink