diff --git a/docs/power_resilience_plan.md b/docs/power_resilience_plan.md index b8eee59..cbd4fa6 100644 --- a/docs/power_resilience_plan.md +++ b/docs/power_resilience_plan.md @@ -1,21 +1,29 @@ # Snarfnet Homelab Power Resilience & UPS Automation Plan -This document details the power resilience strategy, tiered graceful shutdown sequence, and integration architecture for the **snarfnet** homelab centered around the primary **NUT (Network UPS Tools) Master Server**. +This document details the comprehensive power resilience strategy, tiered graceful shutdown sequence, and automated recovery orchestration for the **snarfnet** homelab. The system is centered around a dedicated **NUT (Network UPS Tools) Master Server** at `14.4.4.44`. --- -## 1. Core Architecture & IP Assignments +## 1. Core Architecture & Component Roles The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs. -| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description | +| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol | |---|---|---|---|---| -| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. | -| **NUT Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Compute / Swarm) | Peanut container UI consuming `14.4.4.44:3493`. | -| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for alerts/triggers. | -| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493`. | -| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. | -| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. | +| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master`. | +| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; must be stopped prior to NAS Safe Mode. | +| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth for Peanut, Gotify, Homebox, WLED. Runs on Longhorn storage in Aerodrome. | +| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. | +| **Emergency Push Notifications** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server. **Internal outage alerts MUST use direct IP (`http://140.44.4.70:8090`)** to bypass Authentik during shutdown. | +| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and automated shedding. | +| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493` for battery voltage, load %, and runtime metrics. | +| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts NPM NFS shares (`/volume1/docker/nginx/data`). | +| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts secondary backup shares. | +| **Distributed Block Storage** | `longhorn.aerodrome.snarfnet.net` | `140.44.4.36:80` | VLAN 20 (Aerodrome K3s) | Longhorn CSI storage engine powering Authentik, AWX, Grafana PVCs. Protected with recurring snapshots (`snapshot-4h`, `snapshot-daily`). | + +> [!IMPORTANT] +> **Emergency Alerting Path Warning:** +> `peanut.snarfnet.net` and `gotify.snarfnet.net` are protected externally behind Authentik Forward Auth (`oauth2-proxy`). During a power outage, when K3s `aerodrome` shuts down in Tier 3, Authentik goes offline. Therefore, **emergency outage alerts and NUT notification scripts must communicate directly with Gotify via internal IP (`http://140.44.4.70:8090/message?token=...`)** or native Home Assistant push notifications, bypassing NPM and Authentik. --- @@ -24,100 +32,134 @@ The power resilience architecture uses a centralized **NUT Master Server** host ```mermaid flowchart TD subgraph Grid ["Utility Power Grid"] - Mains["Main AC Power Supply"] + Mains["Main AC Utility Power"] end - subgraph PowerHardware ["Power Protection"] - UPS["Central UPS Hardware"] + subgraph PowerHardware ["Power Hardware & Protection"] + UPS["Central UPS Hardware (Battery Bank)"] end subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"] UPSD["upsd Daemon (Port 3493)"] - UPSMON_M["upsmon (Master)"] + UPSMON_M["upsmon (Master Daemon)"] + NOTIFY["nut-notify.sh (Direct Alert Engine)"] end - subgraph Dashboards ["Monitoring & Alerting"] - Peanut["Peanut Dashboard\n(peanut.snarfnet.net:8050)"] - HA["Home Assistant\n(14.4.4.20)"] + subgraph OutageAlerts ["Emergency Notification Channels"] + HA_Alert["Home Assistant (14.4.4.20)\nLocal Push Notification"] + Gotify_Direct["Gotify Internal Direct\nhttp://140.44.4.70:8090\n(Bypasses Authentik / NPM)"] + Syslog["Wall / Syslog Broadcast"] + end + + subgraph MonitoringPanels ["Monitoring & Management"] + Peanut["Peanut Dashboard\n(peanut.snarfnet.net)"] Zabbix["Zabbix Server\n(140.44.4.61)"] end - subgraph Slaves ["UPS Slaves (upsmon Client Nodes)"] - Harvester["Spacedock HCI Nodes\n(140.44.4.81 - 83)"] - K3sControl["K3s Control Planes\n(Aerodrome & Spaceport)"] - K3sWorkers["K3s Worker Nodes"] - Swarm["Shipyard Swarm Nodes\n(140.44.4.71 - 75)"] - NAS["Synology NAS Arrays\n(14.10.10.71 / 72)"] + subgraph InfrastructureConsumers ["Homelab Tiered Consumers"] + SwarmNFS["Docker Swarm (140.44.4.71-75)\nNPM Host Mode (140.44.4.74)"] + K8sCompute["Aerodrome & Spaceport K3s\nLonghorn Engine & Worker Nodes"] + NAS["Synology NAS Arrays\n(14.10.10.71 / 14.10.10.72)"] + K8sControl["K3s Control Planes & etcd"] + Harvester["Spacedock HCI Hypervisors\n(140.44.4.81 - 83)"] + CoreEdge["UDM Pro Gateway (14.44.4.1)\nCore PoE Switches"] end Mains --> UPS UPS -- USB / SNMP --> UPSD UPSD --> UPSMON_M - + UPSMON_M --> NOTIFY + + NOTIFY -- Direct API Token --> Gotify_Direct + NOTIFY -- TCP 3493 Event --> HA_Alert + NOTIFY --> Syslog + UPSD -- TCP 3493 --> Peanut - UPSD -- TCP 3493 --> HA UPSD -- TCP 3493 --> Zabbix - - UPSD -- TCP 3493 (Slave Broadcast) --> Harvester - UPSD -- TCP 3493 (Slave Broadcast) --> K3sControl - UPSD -- TCP 3493 (Slave Broadcast) --> K3sWorkers - UPSD -- TCP 3493 (Slave Broadcast) --> Swarm - UPSD -- TCP 3493 (Slave Broadcast) --> NAS + + UPSMON_M -. Tiered Signal .-> SwarmNFS + UPSMON_M -. Tiered Signal .-> K8sCompute + UPSMON_M -. Tiered Signal .-> NAS + UPSMON_M -. Tiered Signal .-> K8sControl + UPSMON_M -. Tiered Signal .-> Harvester + UPSMON_M -. Final Cut .-> CoreEdge ``` --- -## 3. Tiered Shutdown Hierarchy & Execution Timeline +## 3. Tiered Graceful Shutdown Hierarchy -To prevent data corruption and unexpected hard cutoffs, hosts shutdown in staged tiers based on power outage duration and battery level. +To eliminate data corruption, prevent stale NFS file handles, and ensure Longhorn volume detach cleanly, the homelab executes a **strict 6-stage progressive shutdown sequence**. ``` -+-------------------------------------------------------------------------+ -| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT | -+-------------------------------------------------------------------------+ - | - v -+-------------------------------------------------------------------------+ -| [T+3m / Batt <= 80%] TIER 1: Non-essential Workloads | -| - Dev VMs, transcoders, test containers in Swarm & K3s drained | -+-------------------------------------------------------------------------+ - | - v -+-------------------------------------------------------------------------+ -| [T+7m / Batt <= 50%] TIER 2: Hypervisors, Compute Workers & Storage | -| - Harvester spacedock VMs gracefully shut down via virsh/API | -| - Aerodrome & Spaceport worker nodes gracefully halt | -| - Synology NAS arrays (14.10.10.71/72) enter Safe Mode & unmount pools | -+-------------------------------------------------------------------------+ - | - v -+-------------------------------------------------------------------------+ -| [T+12m / Batt <= 30%] TIER 3: Control Planes & Management Stack | -| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Control Planes| -| - Swarm Manager (140.44.4.71) & AI Nodes (140.44.4.7-8) halt | -+-------------------------------------------------------------------------+ - | - v -+-------------------------------------------------------------------------+ -| [T+15m / Batt Low] TIER 4: Forced Shutdown (FSD) & Hardware Cutoff | -| - NUT Master (14.4.4.44) issues FSD signal to remaining clients | -| - NUT Master issues `upsdrvctl shutdown` (cuts UPS load outlet) | -| - Network Core (UDM Pro 14.44.4.1) stays alive until final battery drop | -+-------------------------------------------------------------------------+ ++--------------------------------------------------------------------------------------------------+ +| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT | +| - Broadcasts emergency alert via Home Assistant push & Gotify direct API (http://140.44.4.70:8090)| ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads | +| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped | +| - Media transcoders (Plex active sessions signaled) | +| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM / Peanut) | +| - Stop `nginx-proxy-manager_app` & `nginx-proxy-manager_backup` on snshpd4lnxswm04 (140.44.4.74) | +| - Stop `peanut` container on Swarm | +| - Flushes open SQLite databases (database.sqlite) and unmounts NFS mounts to 14.10.10.71 | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach | +| - Harvester guest VMs (harvester.spacedock.snarfnet.net 140.44.4.90) gracefully shut down via API| +| - Cordon & scale down stateful pods on Aerodrome (Authentik PostgreSQL, AWX Postgres, Grafana) | +| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes | +| - Aerodrome (140.44.4.21-25) & Spaceport (140.44.4.51-56) worker nodes execute shutdown | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode) | +| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode via NUT client trigger | +| - Unmounts all Btrfs volume pools, terminates NFS/iSCSI daemon threads, parks drive read heads | +| - Zero active network clients remain connected to storage | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors | +| - Aerodrome K3s Control Planes (140.44.4.11-13) take etcd snapshot and halt | +| - Spaceport K3s Control Planes (140.44.4.41-44) take etcd snapshot and halt | +| - Swarm Manager nodes (140.44.4.71-75) halt | +| - Harvester baremetal hypervisors (140.44.4.81-83) halt via IPMI/SSH | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut | +| - NUT Master (14.4.4.44) issues final FSD broadcast to all remaining clients | +| - NUT Master issues `upsdrvctl shutdown` with /etc/killpower flag (commands UPS load drop) | +| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops | ++--------------------------------------------------------------------------------------------------+ ``` --- -## 4. Configuration Blueprint & Service Integrations +## 4. Configuration Blueprints & Integration Scripts -### A. NUT Master Server (`14.4.4.44`) Configuration Files +### A. NUT Master Server (`14.4.4.44`) #### `/etc/nut/ups.conf` ```ini [snarf-ups] driver = usbhid-ups port = auto - desc = "Snarfnet Main Server Rack UPS" + desc = "Snarfnet Main Rack UPS" pollinterval = 5 ``` @@ -131,7 +173,7 @@ MAXAGE 15 #### `/etc/nut/upsd.users` ```ini [admin] - password = SUPER_SECRET_ADMIN_PASS + password = ADMIN_PASSWORD_CHANGE_ME actions = SET instcmds = ALL @@ -152,7 +194,7 @@ MAXAGE 15 ```ini MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master MINSUPPLIES 1 -SHUTDOWNCMD "/sbin/shutdown -h +0" +SHUTDOWNCMD "/usr/local/bin/homelab_power_shutdown.sh" NOTIFYCMD /usr/local/bin/nut-notify.sh POWERDOWNFLAG /etc/killpower @@ -166,40 +208,164 @@ NOTIFYFLAG COMMOK SYSLOG+WALL+EXEC --- -### B. Client Node Configuration (`upsmon` Slave) +### B. Outage Notification Engine (`/usr/local/bin/nut-notify.sh`) -Applies to Harvester hypervisors, K3s master/worker hosts, Docker Swarm hosts, and Debian Linux nodes. +This script runs on the NUT Master (`14.4.4.44`) whenever a power state event triggers. It sends push notifications directly to Gotify (`http://140.44.4.70:8090`) and Home Assistant (`http://14.4.4.20:8123`) without touching NPM or Authentik. -#### `/etc/nut/upsmon.conf` (Slave Mode) -```ini -MONITOR snarf-ups@14.4.4.44 1 upsmon_slave SLAVE_MONITOR_PASS slave -MINSUPPLIES 1 -SHUTDOWNCMD "/sbin/shutdown -h +0" -POWERDOWNFLAG /etc/killpower -HOSTSYNC 15 -DEADTIME 15 -RBWARNTIME 43200 -NOCOMMWARNTIME 300 -FINALDELAY 5 +```bash +#!/bin/bash +# /usr/local/bin/nut-notify.sh +# Emergency power notification script for Network UPS Tools +set -euo pipefail + +EVENT_TYPE="${1:-$NOTIFYTYPE}" +UPS_NAME="${UPSNAME:-snarf-ups}" +TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S') + +GOTIFY_INTERNAL_URL="http://140.44.4.70:8090/message" +GOTIFY_APP_TOKEN="A4uN4fEmErGeNcYtOkEn" # Replace with valid Gotify App Token + +send_gotify() { + local title="$1" + local message="$2" + local priority="$3" + + curl -s -m 5 -X POST "${GOTIFY_INTERNAL_URL}?token=${GOTIFY_APP_TOKEN}" \ + -H "Content-Type: application/json" \ + -d "{\"title\": \"${title}\", \"message\": \"${message}\", \"priority\": ${priority}}" >/dev/null 2>&1 || true +} + +case "$EVENT_TYPE" in + ONBATT) + send_gotify "⚡ Grid Power Lost" "[$TIMESTAMP] Homelab is running on UPS battery (${UPS_NAME}). Staged shutdown countdown started." 8 + logger -t nut-notify "Grid power failure detected - running on UPS battery." + ;; + ONLINE) + send_gotify "✅ Grid Power Restored" "[$TIMESTAMP] Utility AC power restored to ${UPS_NAME}. Battery recharging." 5 + logger -t nut-notify "AC utility power restored." + ;; + LOWBATT) + send_gotify "🚨 UPS Battery Critical" "[$TIMESTAMP] Battery level critical on ${UPS_NAME}! Final Tier 4-6 shutdown initiated." 10 + logger -t nut-notify "CRITICAL: UPS battery low. Enforcing emergency shutdown." + ;; + FSD) + send_gotify "🛑 Forced Shutdown Engaged" "[$TIMESTAMP] NUT Master has commanded immediate power cutoff across all clusters." 10 + logger -t nut-notify "FSD signal received. Halting systems." + ;; + *) + logger -t nut-notify "Unhandled UPS event: ${EVENT_TYPE}" + ;; +esac ``` --- -### C. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`) +### C. Orchestrated Tiered Shutdown Script (`/usr/local/bin/homelab_power_shutdown.sh`) + +This script is invoked by `upsmon` on `14.4.4.44` when an outage exceeds the battery threshold: + +```bash +#!/bin/bash +# /usr/local/bin/homelab_power_shutdown.sh +# Staged graceful homelab shutdown orchestrator +set -uo pipefail + +log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [POWER-SHUTDOWN] $*" | tee -a /var/log/power-shutdown.log; } +SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes" + +log "=== INITIATING HOMELAB TIERED SHUTDOWN ===" + +# ------------------------------------------------------------------ +# TIER 1: Non-Essential Compute & AI Pis +# ------------------------------------------------------------------ +log "Stage 1: Halting AI Raspberry Pi stack (140.44.4.7, 140.44.4.8)..." +ssh $SSH_OPTS root@140.44.4.7 "shutdown -h now" 2>/dev/null || true +ssh $SSH_OPTS root@140.44.4.8 "shutdown -h now" 2>/dev/null || true + +# ------------------------------------------------------------------ +# TIER 2: Docker Swarm NFS Workloads (NPM & Peanut) +# ------------------------------------------------------------------ +log "Stage 2: Stopping Swarm services mounting Synology NFS (NPM, Peanut)..." +ssh $SSH_OPTS root@140.44.4.71 "docker service scale nginx-proxy-manager_app=0 nginx-proxy-manager_backup=0 peanut_peanut=0" 2>/dev/null || true +sleep 5 + +# ------------------------------------------------------------------ +# TIER 3: Harvester Guest VMs & Kubernetes Worker Nodes +# ------------------------------------------------------------------ +log "Stage 3: Shutting down Harvester guest VMs and K3s workers..." +# Gracefully signal Harvester VMs +ssh $SSH_OPTS root@140.44.4.81 "virsh list --name | xargs -r -n1 virsh shutdown" 2>/dev/null || true + +# Stop K3s agent on Aerodrome workers (clean Longhorn PVC detach) +for ip in 140.44.4.22 140.44.4.23 140.44.4.24 140.44.4.25; do + ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true +done + +# Stop K3s agent on Spaceport workers +for ip in 140.44.4.51 140.44.4.52 140.44.4.53 140.44.4.54 140.44.4.55 140.44.4.56; do + ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true +done +sleep 15 + +# ------------------------------------------------------------------ +# TIER 4: Storage Arrays (Synology DSM Safe Mode) +# ------------------------------------------------------------------ +log "Stage 4: Triggering Synology NAS Safe Mode (14.10.10.71, 14.10.10.72)..." +# Synology enters safe mode automatically via NUT upsd client signal (port 3493) +# Explicit API / SSH command to flush pools if configured +ssh $SSH_OPTS admin@14.10.10.71 "synopoweroff" 2>/dev/null || true +ssh $SSH_OPTS admin@14.10.10.72 "synopoweroff" 2>/dev/null || true +sleep 20 + +# ------------------------------------------------------------------ +# TIER 5: Control Planes & Swarm Managers +# ------------------------------------------------------------------ +log "Stage 5: Halting K3s control planes and Swarm managers..." +# Spaceport Masters (140.44.4.41-44) +for ip in 140.44.4.41 140.44.4.42 140.44.4.43 140.44.4.44; do + ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true +done + +# Aerodrome Masters (140.44.4.11-13) +for ip in 140.44.4.11 140.44.4.12 140.44.4.13; do + ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true +done + +# Swarm Managers (140.44.4.71, 72, 74) +for ip in 140.44.4.71 140.44.4.72 140.44.4.74; do + ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true +done + +# Harvester Baremetal Hypervisors (140.44.4.81-83) +for ip in 140.44.4.81 140.44.4.82 140.44.4.83; do + ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true +done + +# ------------------------------------------------------------------ +# TIER 6: Master Node Self-Cutoff (FSD) +# ------------------------------------------------------------------ +log "Stage 6: Executing final local halt and arming killpower..." +touch /etc/killpower +/sbin/shutdown -h +0 +``` + +--- + +### D. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`) 1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**. 2. Check **Enable UPS Support**. 3. Select UPS Type: **Synology UPS Server / Network NUT Server**. 4. Network UPS server IP address: **`14.4.4.44`**. -5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `5 minutes` (or `Upon low battery`). +5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `6 minutes` (guaranteeing that Tier 2 Swarm NFS workloads have cleanly scaled down before disk pools unmount). +6. Check **Shut down UPS when the system enters Standby Mode** (disabled, managed centrally by `14.4.4.44`). --- -### D. Peanut Web Dashboard (`peanut.snarfnet.net`) +### E. Peanut Web Dashboard Configuration -Peanut runs on Docker Swarm (`140.44.4.70:8050`). Ensure the Peanut service configuration references the NUT server at `14.4.4.44`: +Peanut runs on Docker Swarm (`140.44.4.70:8050`). It connects directly to the NUT master server over internal TCP 3493: -#### Updated `docker-deployments/peanut.yml` ```yaml services: peanut: @@ -226,38 +392,25 @@ volumes: --- -### E. Home Assistant (`14.4.4.20`) Integration & Automation +## 5. Automated Power Restoration & Boot Sequence -1. Navigate to **Settings** $\rightarrow$ **Devices & Services** $\rightarrow$ **Add Integration** $\rightarrow$ **Network UPS Tools (NUT)**. -2. Host: **`14.4.4.44`**, Port: **`3493`**. -3. Username: **`upsmon_slave`**, Password: **`SLAVE_MONITOR_PASS`**. +When utility grid power returns, the homelab automatically recovers in a **strict bottom-up reverse sequence** to guarantee all storage and network prerequisites are ready before compute services attempt to start. -#### Automation Snippet (`automations.yaml`) -```yaml -alias: "Power Resilience: Grid Power Lost Alert" -trigger: - - platform: state - entity_id: sensor.snarf_ups_status - to: "On Battery" -action: - - service: notify.gotify - data: - title: "⚡ Power Outage Alert" - message: "Grid power failure detected! Homelab is running on UPS battery (14.4.4.44). Tiered shutdown in progress." - priority: 8 +```mermaid +flowchart TD + G_On["1. Grid AC Restored\nUPS Battery Reaches >= 30%"] --> P1["2. Phase 1: Core Network & Edge\nUDM Pro (14.44.4.1) & Core PoE Switches\nEstablish VLANs, DHCP, Inter-VLAN Routing"] + P1 --> P2["3. Phase 2: NUT Master\nNode at 14.4.4.44 Boots (BIOS: Always On)\nupsd daemon starts, monitors line quality"] + P2 --> P3["4. Phase 3: Storage Foundation\nSynology SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) Boot\nMount Btrfs pools, start NFS & iSCSI daemons"] + P3 --> P4["5. Phase 4: Control Planes & Hypervisors\n- Harvester spacedock (140.44.4.81-83) boots via IPMI/WOL\n- Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) K3s masters boot\n- etcd clusters re-establish quorum"] + P4 --> P5["6. Phase 5: Storage CSI & Compute Workers\n- Aerodrome & Spaceport worker nodes boot\n- Longhorn CSI engine initializes, verifies volume health, mounts PVCs"] + P5 --> P6["7. Phase 6: Ingress & Workloads\n- Docker Swarm (140.44.4.71-75) boots\n- NPM host-mode container binds 80/443/81 on 140.44.4.74\n- Authentik SSO & Forward Auth come online\n- Applications (Peanut, Gotify, Homebox, etc.) become reachable"] ``` ---- +### Critical Health Verification Gates: -## 5. Power Restoration & Auto-Boot Sequence - -When grid power returns, systems auto-recover in reverse order: - -1. **UPS AC Mains Restored**: Physical UPS re-enables output outlets once battery reaches safe threshold ($\ge 20\%$). -2. **Phase 1 - Gateway & Edge**: UDM Pro (`14.44.4.1`) & Core switches boot up and establish internal VLAN routes. -3. **Phase 2 - NUT Master**: Host at `14.4.4.44` boots (BIOS `Restore AC Power` set to `Always On`). `upsd` service starts. -4. **Phase 3 - Storage NAS**: Synology NAS nodes (`14.10.10.71`, `14.10.10.72`) auto-boot upon power recovery. -5. **Phase 4 - Control Planes & Hypervisors**: - - Harvester `spacedock` hosts (`140.44.4.81-83`) boot via Wake-on-LAN / IPMI / BIOS settings. - - K3s master nodes (`aerodrome` & `spaceport`) boot, resume etcd quorum. -6. **Phase 5 - Worker Compute & Workloads**: Worker nodes boot; Kubernetes and Docker Swarm automatically reschedule workloads. +1. **Storage Gate (Phase 3 $\rightarrow$ Phase 4):** + - Synology NAS NFS daemons must be responding on `14.10.10.71:2049` before Docker Swarm starts, ensuring NPM does not encounter stale NFS mounts or SQLite locking. +2. **Database Gate (Phase 5 $\rightarrow$ Phase 6):** + - Longhorn volume `data-authentik-postgresql-0` on Aerodrome must reach `Healthy` state before Authentik server pods start. +3. **Ingress Gate (Phase 6):** + - Nginx Proxy Manager on `snshpd4lnxswm04` (`140.44.4.74`) confirms host-mode ports `80` and `443` are bound and forwards traffic to internal VIPs. diff --git a/scripts/homelab_power_shutdown.sh b/scripts/homelab_power_shutdown.sh new file mode 100644 index 0000000..c390181 --- /dev/null +++ b/scripts/homelab_power_shutdown.sh @@ -0,0 +1,80 @@ +#!/bin/bash +# /usr/local/bin/homelab_power_shutdown.sh +# Staged graceful homelab shutdown orchestrator for Snarfnet +set -uo pipefail + +log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [POWER-SHUTDOWN] $*" | tee -a /var/log/power-shutdown.log; } +SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes" + +log "=== INITIATING HOMELAB TIERED SHUTDOWN ===" + +# ------------------------------------------------------------------ +# TIER 1: Non-Essential Compute & AI Pis +# ------------------------------------------------------------------ +log "Stage 1: Halting AI Raspberry Pi stack (140.44.4.7, 140.44.4.8)..." +ssh $SSH_OPTS root@140.44.4.7 "shutdown -h now" 2>/dev/null || true +ssh $SSH_OPTS root@140.44.4.8 "shutdown -h now" 2>/dev/null || true + +# ------------------------------------------------------------------ +# TIER 2: Docker Swarm NFS Workloads (NPM & Peanut) +# ------------------------------------------------------------------ +log "Stage 2: Stopping Swarm services mounting Synology NFS (NPM, Peanut)..." +ssh $SSH_OPTS root@140.44.4.71 "docker service scale nginx-proxy-manager_app=0 nginx-proxy-manager_backup=0 peanut_peanut=0" 2>/dev/null || true +sleep 5 + +# ------------------------------------------------------------------ +# TIER 3: Harvester Guest VMs & Kubernetes Worker Nodes +# ------------------------------------------------------------------ +log "Stage 3: Shutting down Harvester guest VMs and K3s workers..." +# Gracefully signal Harvester VMs +ssh $SSH_OPTS root@140.44.4.81 "virsh list --name | xargs -r -n1 virsh shutdown" 2>/dev/null || true + +# Stop K3s agent on Aerodrome workers (clean Longhorn PVC detach) +for ip in 140.44.4.22 140.44.4.23 140.44.4.24 140.44.4.25; do + ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true +done + +# Stop K3s agent on Spaceport workers +for ip in 140.44.4.51 140.44.4.52 140.44.4.53 140.44.4.54 140.44.4.55 140.44.4.56; do + ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true +done +sleep 15 + +# ------------------------------------------------------------------ +# TIER 4: Storage Arrays (Synology DSM Safe Mode) +# ------------------------------------------------------------------ +log "Stage 4: Triggering Synology NAS Safe Mode (14.10.10.71, 14.10.10.72)..." +ssh $SSH_OPTS admin@14.10.10.71 "synopoweroff" 2>/dev/null || true +ssh $SSH_OPTS admin@14.10.10.72 "synopoweroff" 2>/dev/null || true +sleep 20 + +# ------------------------------------------------------------------ +# TIER 5: Control Planes & Swarm Managers +# ------------------------------------------------------------------ +log "Stage 5: Halting K3s control planes and Swarm managers..." +# Spaceport Masters (140.44.4.41-44) +for ip in 140.44.4.41 140.44.4.42 140.44.4.43 140.44.4.44; do + ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true +done + +# Aerodrome Masters (140.44.4.11-13) +for ip in 140.44.4.11 140.44.4.12 140.44.4.13; do + ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true +done + +# Swarm Managers (140.44.4.71, 72, 74) +for ip in 140.44.4.71 140.44.4.72 140.44.4.74; do + ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true +done + +# Harvester Baremetal Hypervisors (140.44.4.81-83) +for ip in 140.44.4.81 140.44.4.82 140.44.4.83; do + ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true +done + +# ------------------------------------------------------------------ +# TIER 6: Master Node Self-Cutoff (FSD) +# ------------------------------------------------------------------ +log "Stage 6: Executing final local halt and arming killpower..." +touch /etc/killpower +/sbin/shutdown -h +0 diff --git a/scripts/nut_notify.sh b/scripts/nut_notify.sh new file mode 100644 index 0000000..484aaa0 --- /dev/null +++ b/scripts/nut_notify.sh @@ -0,0 +1,44 @@ +#!/bin/bash +# /usr/local/bin/nut-notify.sh +# Emergency power notification script for Network UPS Tools on Snarfnet +set -euo pipefail + +EVENT_TYPE="${1:-$NOTIFYTYPE}" +UPS_NAME="${UPSNAME:-snarf-ups}" +TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S') + +# Internal Gotify VIP endpoint (bypasses Authentik Forward Auth and NPM) +GOTIFY_INTERNAL_URL="http://140.44.4.70:8090/message" +GOTIFY_APP_TOKEN="${GOTIFY_APP_TOKEN:-A4uN4fEmErGeNcYtOkEn}" + +send_gotify() { + local title="$1" + local message="$2" + local priority="$3" + + curl -s -m 5 -X POST "${GOTIFY_INTERNAL_URL}?token=${GOTIFY_APP_TOKEN}" \ + -H "Content-Type: application/json" \ + -d "{\"title\": \"${title}\", \"message\": \"${message}\", \"priority\": ${priority}}" >/dev/null 2>&1 || true +} + +case "$EVENT_TYPE" in + ONBATT) + send_gotify "⚡ Grid Power Lost" "[$TIMESTAMP] Homelab is running on UPS battery (${UPS_NAME}). Staged shutdown countdown started." 8 + logger -t nut-notify "Grid power failure detected - running on UPS battery." + ;; + ONLINE) + send_gotify "✅ Grid Power Restored" "[$TIMESTAMP] Utility AC power restored to ${UPS_NAME}. Battery recharging." 5 + logger -t nut-notify "AC utility power restored." + ;; + LOWBATT) + send_gotify "🚨 UPS Battery Critical" "[$TIMESTAMP] Battery level critical on ${UPS_NAME}! Final Tier 4-6 shutdown initiated." 10 + logger -t nut-notify "CRITICAL: UPS battery low. Enforcing emergency shutdown." + ;; + FSD) + send_gotify "🛑 Forced Shutdown Engaged" "[$TIMESTAMP] NUT Master has commanded immediate power cutoff across all clusters." 10 + logger -t nut-notify "FSD signal received. Halting systems." + ;; + *) + logger -t nut-notify "Unhandled UPS event: ${EVENT_TYPE}" + ;; +esac