# Snarfnet Homelab Power Resilience & UPS Automation Plan

This document details the comprehensive power resilience strategy, tiered graceful shutdown sequence, and automated recovery orchestration for the **snarfnet** homelab. The system is centered around a dedicated **NUT (Network UPS Tools) Master Server** at `14.4.4.44`.

---

## 1. Core Architecture & Component Roles

The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.

| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol |
|---|---|---|---|---|
| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master`. |
| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; must be stopped prior to NAS Safe Mode. |
| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth for Peanut, Gotify, Homebox, WLED. Runs on Longhorn storage in Aerodrome. |
| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. |
| **Emergency Push Notifications** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server. **Internal outage alerts MUST use direct IP (`http://140.44.4.70:8090`)** to bypass Authentik during shutdown. |
| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and automated shedding. |
| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493` for battery voltage, load %, and runtime metrics. |
| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts NPM NFS shares (`/volume1/docker/nginx/data`). |
| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts secondary backup shares. |
| **Distributed Block Storage** | `longhorn.aerodrome.snarfnet.net` | `140.44.4.36:80` | VLAN 20 (Aerodrome K3s) | Longhorn CSI storage engine powering Authentik, AWX, Grafana PVCs. Protected with recurring snapshots (`snapshot-4h`, `snapshot-daily`). |

> [!IMPORTANT]
> **Emergency Alerting Path Warning:**
> `peanut.snarfnet.net` and `gotify.snarfnet.net` are protected externally behind Authentik Forward Auth (`oauth2-proxy`). During a power outage, when K3s `aerodrome` shuts down in Tier 3, Authentik goes offline. Therefore, **emergency outage alerts and NUT notification scripts must communicate directly with Gotify via internal IP (`http://140.44.4.70:8090/message?token=...`)** or native Home Assistant push notifications, bypassing NPM and Authentik.

---

## 2. Power Resilience Topology Map

```mermaid
flowchart TD
    subgraph Grid ["Utility Power Grid"]
        Mains["Main AC Utility Power"]
    end

    subgraph PowerHardware ["Power Hardware & Protection"]
        UPS["Central UPS Hardware (Battery Bank)"]
    end

    subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"]
        UPSD["upsd Daemon (Port 3493)"]
        UPSMON_M["upsmon (Master Daemon)"]
        NOTIFY["nut-notify.sh (Direct Alert Engine)"]
    end

    subgraph OutageAlerts ["Emergency Notification Channels"]
        HA_Alert["Home Assistant (14.4.4.20)\nLocal Push Notification"]
        Gotify_Direct["Gotify Internal Direct\nhttp://140.44.4.70:8090\n(Bypasses Authentik / NPM)"]
        Syslog["Wall / Syslog Broadcast"]
    end

    subgraph MonitoringPanels ["Monitoring & Management"]
        Peanut["Peanut Dashboard\n(peanut.snarfnet.net)"]
        Zabbix["Zabbix Server\n(140.44.4.61)"]
    end

    subgraph InfrastructureConsumers ["Homelab Tiered Consumers"]
        SwarmNFS["Docker Swarm (140.44.4.71-75)\nNPM Host Mode (140.44.4.74)"]
        K8sCompute["Aerodrome & Spaceport K3s\nLonghorn Engine & Worker Nodes"]
        NAS["Synology NAS Arrays\n(14.10.10.71 / 14.10.10.72)"]
        K8sControl["K3s Control Planes & etcd"]
        Harvester["Spacedock HCI Hypervisors\n(140.44.4.81 - 83)"]
        CoreEdge["UDM Pro Gateway (14.44.4.1)\nCore PoE Switches"]
    end

    Mains --> UPS
    UPS -- USB / SNMP --> UPSD
    UPSD --> UPSMON_M
    UPSMON_M --> NOTIFY

    NOTIFY -- Direct API Token --> Gotify_Direct
    NOTIFY -- TCP 3493 Event --> HA_Alert
    NOTIFY --> Syslog

    UPSD -- TCP 3493 --> Peanut
    UPSD -- TCP 3493 --> Zabbix

    UPSMON_M -. Tiered Signal .-> SwarmNFS
    UPSMON_M -. Tiered Signal .-> K8sCompute
    UPSMON_M -. Tiered Signal .-> NAS
    UPSMON_M -. Tiered Signal .-> K8sControl
    UPSMON_M -. Tiered Signal .-> Harvester
    UPSMON_M -. Final Cut .-> CoreEdge
```

---

## 3. Tiered Graceful Shutdown Hierarchy

To eliminate data corruption, prevent stale NFS file handles, and ensure Longhorn volume detach cleanly, the homelab executes a **strict 6-stage progressive shutdown sequence**.

```
+--------------------------------------------------------------------------------------------------+
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT                                |
| - Broadcasts emergency alert via Home Assistant push & Gotify direct API (http://140.44.4.70:8090)|
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads                                  |
| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped      |
| - Media transcoders (Plex active sessions signaled)                                              |
| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted                         |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM / Peanut)                    |
| - Stop `nginx-proxy-manager_app` & `nginx-proxy-manager_backup` on snshpd4lnxswm04 (140.44.4.74) |
| - Stop `peanut` container on Swarm                                                               |
| - Flushes open SQLite databases (database.sqlite) and unmounts NFS mounts to 14.10.10.71        |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach             |
| - Harvester guest VMs (harvester.spacedock.snarfnet.net 140.44.4.90) gracefully shut down via API|
| - Cordon & scale down stateful pods on Aerodrome (Authentik PostgreSQL, AWX Postgres, Grafana)   |
| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes     |
| - Aerodrome (140.44.4.21-25) & Spaceport (140.44.4.51-56) worker nodes execute shutdown          |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode)                            |
| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode via NUT client trigger      |
| - Unmounts all Btrfs volume pools, terminates NFS/iSCSI daemon threads, parks drive read heads    |
| - Zero active network clients remain connected to storage                                        |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors                             |
| - Aerodrome K3s Control Planes (140.44.4.11-13) take etcd snapshot and halt                     |
| - Spaceport K3s Control Planes (140.44.4.41-44) take etcd snapshot and halt                     |
| - Swarm Manager nodes (140.44.4.71-75) halt                                                      |
| - Harvester baremetal hypervisors (140.44.4.81-83) halt via IPMI/SSH                             |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut                               |
| - NUT Master (14.4.4.44) issues final FSD broadcast to all remaining clients                     |
| - NUT Master issues `upsdrvctl shutdown` with /etc/killpower flag (commands UPS load drop)        |
| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops   |
+--------------------------------------------------------------------------------------------------+
```

---

## 4. Configuration Blueprints & Integration Scripts

### A. NUT Master Server (`14.4.4.44`)

#### `/etc/nut/ups.conf`
```ini
[snarf-ups]
    driver = usbhid-ups
    port = auto
    desc = "Snarfnet Main Rack UPS"
    pollinterval = 5
```

#### `/etc/nut/upsd.conf`
```ini
LISTEN 14.4.4.44 3493
LISTEN 127.0.0.1 3493
MAXAGE 15
```

#### `/etc/nut/upsd.users`
```ini
[admin]
    password = ADMIN_PASSWORD_CHANGE_ME
    actions = SET
    instcmds = ALL

[upsmon_master]
    password = MASTER_MONITOR_PASS
    upsmon master

[upsmon_slave]
    password = SLAVE_MONITOR_PASS
    upsmon slave

[peanut]
    password = PEANUT_MONITOR_PASS
    upsmon slave
```

#### `/etc/nut/upsmon.conf`
```ini
MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master
MINSUPPLIES 1
SHUTDOWNCMD "/usr/local/bin/homelab_power_shutdown.sh"
NOTIFYCMD /usr/local/bin/nut-notify.sh
POWERDOWNFLAG /etc/killpower

NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC
NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC
NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC
NOTIFYFLAG FSD      SYSLOG+WALL+EXEC
NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC
NOTIFYFLAG COMMOK   SYSLOG+WALL+EXEC
```

---

### B. Outage Notification Engine (`/usr/local/bin/nut-notify.sh`)

This script runs on the NUT Master (`14.4.4.44`) whenever a power state event triggers. It sends push notifications directly to Gotify (`http://140.44.4.70:8090`) and Home Assistant (`http://14.4.4.20:8123`) without touching NPM or Authentik.

```bash
#!/bin/bash
# /usr/local/bin/nut-notify.sh
# Emergency power notification script for Network UPS Tools
set -euo pipefail

EVENT_TYPE="${1:-$NOTIFYTYPE}"
UPS_NAME="${UPSNAME:-snarf-ups}"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

GOTIFY_INTERNAL_URL="http://140.44.4.70:8090/message"
GOTIFY_APP_TOKEN="A4uN4fEmErGeNcYtOkEn" # Replace with valid Gotify App Token

send_gotify() {
    local title="$1"
    local message="$2"
    local priority="$3"
    
    curl -s -m 5 -X POST "${GOTIFY_INTERNAL_URL}?token=${GOTIFY_APP_TOKEN}" \
        -H "Content-Type: application/json" \
        -d "{\"title\": \"${title}\", \"message\": \"${message}\", \"priority\": ${priority}}" >/dev/null 2>&1 || true
}

case "$EVENT_TYPE" in
    ONBATT)
        send_gotify "⚡ Grid Power Lost" "[$TIMESTAMP] Homelab is running on UPS battery (${UPS_NAME}). Staged shutdown countdown started." 8
        logger -t nut-notify "Grid power failure detected - running on UPS battery."
        ;;
    ONLINE)
        send_gotify "✅ Grid Power Restored" "[$TIMESTAMP] Utility AC power restored to ${UPS_NAME}. Battery recharging." 5
        logger -t nut-notify "AC utility power restored."
        ;;
    LOWBATT)
        send_gotify "🚨 UPS Battery Critical" "[$TIMESTAMP] Battery level critical on ${UPS_NAME}! Final Tier 4-6 shutdown initiated." 10
        logger -t nut-notify "CRITICAL: UPS battery low. Enforcing emergency shutdown."
        ;;
    FSD)
        send_gotify "🛑 Forced Shutdown Engaged" "[$TIMESTAMP] NUT Master has commanded immediate power cutoff across all clusters." 10
        logger -t nut-notify "FSD signal received. Halting systems."
        ;;
    *)
        logger -t nut-notify "Unhandled UPS event: ${EVENT_TYPE}"
        ;;
esac
```

---

### C. Orchestrated Tiered Shutdown Script (`/usr/local/bin/homelab_power_shutdown.sh`)

This script is invoked by `upsmon` on `14.4.4.44` when an outage exceeds the battery threshold:

```bash
#!/bin/bash
# /usr/local/bin/homelab_power_shutdown.sh
# Staged graceful homelab shutdown orchestrator
set -uo pipefail

log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [POWER-SHUTDOWN] $*" | tee -a /var/log/power-shutdown.log; }
SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes"

log "=== INITIATING HOMELAB TIERED SHUTDOWN ==="

# ------------------------------------------------------------------
# TIER 1: Non-Essential Compute & AI Pis
# ------------------------------------------------------------------
log "Stage 1: Halting AI Raspberry Pi stack (140.44.4.7, 140.44.4.8)..."
ssh $SSH_OPTS root@140.44.4.7 "shutdown -h now" 2>/dev/null || true
ssh $SSH_OPTS root@140.44.4.8 "shutdown -h now" 2>/dev/null || true

# ------------------------------------------------------------------
# TIER 2: Docker Swarm NFS Workloads (NPM & Peanut)
# ------------------------------------------------------------------
log "Stage 2: Stopping Swarm services mounting Synology NFS (NPM, Peanut)..."
ssh $SSH_OPTS root@140.44.4.71 "docker service scale nginx-proxy-manager_app=0 nginx-proxy-manager_backup=0 peanut_peanut=0" 2>/dev/null || true
sleep 5

# ------------------------------------------------------------------
# TIER 3: Harvester Guest VMs & Kubernetes Worker Nodes
# ------------------------------------------------------------------
log "Stage 3: Shutting down Harvester guest VMs and K3s workers..."
# Gracefully signal Harvester VMs
ssh $SSH_OPTS root@140.44.4.81 "virsh list --name | xargs -r -n1 virsh shutdown" 2>/dev/null || true

# Stop K3s agent on Aerodrome workers (clean Longhorn PVC detach)
for ip in 140.44.4.22 140.44.4.23 140.44.4.24 140.44.4.25; do
    ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
done

# Stop K3s agent on Spaceport workers
for ip in 140.44.4.51 140.44.4.52 140.44.4.53 140.44.4.54 140.44.4.55 140.44.4.56; do
    ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
done
sleep 15

# ------------------------------------------------------------------
# TIER 4: Storage Arrays (Synology DSM Safe Mode)
# ------------------------------------------------------------------
log "Stage 4: Triggering Synology NAS Safe Mode (14.10.10.71, 14.10.10.72)..."
# Synology enters safe mode automatically via NUT upsd client signal (port 3493)
# Explicit API / SSH command to flush pools if configured
ssh $SSH_OPTS admin@14.10.10.71 "synopoweroff" 2>/dev/null || true
ssh $SSH_OPTS admin@14.10.10.72 "synopoweroff" 2>/dev/null || true
sleep 20

# ------------------------------------------------------------------
# TIER 5: Control Planes & Swarm Managers
# ------------------------------------------------------------------
log "Stage 5: Halting K3s control planes and Swarm managers..."
# Spaceport Masters (140.44.4.41-44)
for ip in 140.44.4.41 140.44.4.42 140.44.4.43 140.44.4.44; do
    ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
done

# Aerodrome Masters (140.44.4.11-13)
for ip in 140.44.4.11 140.44.4.12 140.44.4.13; do
    ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
done

# Swarm Managers (140.44.4.71, 72, 74)
for ip in 140.44.4.71 140.44.4.72 140.44.4.74; do
    ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
done

# Harvester Baremetal Hypervisors (140.44.4.81-83)
for ip in 140.44.4.81 140.44.4.82 140.44.4.83; do
    ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
done

# ------------------------------------------------------------------
# TIER 6: Master Node Self-Cutoff (FSD)
# ------------------------------------------------------------------
log "Stage 6: Executing final local halt and arming killpower..."
touch /etc/killpower
/sbin/shutdown -h +0
```

---

### D. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`)

1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**.
2. Check **Enable UPS Support**.
3. Select UPS Type: **Synology UPS Server / Network NUT Server**.
4. Network UPS server IP address: **`14.4.4.44`**.
5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `6 minutes` (guaranteeing that Tier 2 Swarm NFS workloads have cleanly scaled down before disk pools unmount).
6. Check **Shut down UPS when the system enters Standby Mode** (disabled, managed centrally by `14.4.4.44`).

---

### E. Peanut Web Dashboard Configuration

Peanut runs on Docker Swarm (`140.44.4.70:8050`). It connects directly to the NUT master server over internal TCP 3493:

```yaml
services:
  peanut:
    image: brandawg93/peanut:latest
    ports:
      - "8050:8050"
    environment:
      - TZ=America/New_York
      - NUT_HOST=14.4.4.44
      - NUT_PORT=3493
      - NUT_USER=peanut
      - NUT_PASSWORD=PEANUT_MONITOR_PASS
    volumes:
      - peanut_config:/config

volumes:
  peanut_config:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: /mnt/swarm_shared/legacy_volumes/peanut/
```

---

## 5. Automated Power Restoration & Boot Sequence

When utility grid power returns, the homelab automatically recovers in a **strict bottom-up reverse sequence** to guarantee all storage and network prerequisites are ready before compute services attempt to start.

```mermaid
flowchart TD
    G_On["1. Grid AC Restored\nUPS Battery Reaches >= 30%"] --> P1["2. Phase 1: Core Network & Edge\nUDM Pro (14.44.4.1) & Core PoE Switches\nEstablish VLANs, DHCP, Inter-VLAN Routing"]
    P1 --> P2["3. Phase 2: NUT Master\nNode at 14.4.4.44 Boots (BIOS: Always On)\nupsd daemon starts, monitors line quality"]
    P2 --> P3["4. Phase 3: Storage Foundation\nSynology SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) Boot\nMount Btrfs pools, start NFS & iSCSI daemons"]
    P3 --> P4["5. Phase 4: Control Planes & Hypervisors\n- Harvester spacedock (140.44.4.81-83) boots via IPMI/WOL\n- Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) K3s masters boot\n- etcd clusters re-establish quorum"]
    P4 --> P5["6. Phase 5: Storage CSI & Compute Workers\n- Aerodrome & Spaceport worker nodes boot\n- Longhorn CSI engine initializes, verifies volume health, mounts PVCs"]
    P5 --> P6["7. Phase 6: Ingress & Workloads\n- Docker Swarm (140.44.4.71-75) boots\n- NPM host-mode container binds 80/443/81 on 140.44.4.74\n- Authentik SSO & Forward Auth come online\n- Applications (Peanut, Gotify, Homebox, etc.) become reachable"]
```

### Critical Health Verification Gates:

1. **Storage Gate (Phase 3 $\rightarrow$ Phase 4):**
   - Synology NAS NFS daemons must be responding on `14.10.10.71:2049` before Docker Swarm starts, ensuring NPM does not encounter stale NFS mounts or SQLite locking.
2. **Database Gate (Phase 5 $\rightarrow$ Phase 6):**
   - Longhorn volume `data-authentik-postgresql-0` on Aerodrome must reach `Healthy` state before Authentik server pods start.
3. **Ingress Gate (Phase 6):**
   - Nginx Proxy Manager on `snshpd4lnxswm04` (`140.44.4.74`) confirms host-mode ports `80` and `443` are bound and forwards traffic to internal VIPs.