docs: modernize power resilience plan and shutdown orchestration with NFS-safe tiering, direct Gotify alerts, and Longhorn clean detach
This commit is contained in:
@@ -1,21 +1,29 @@
|
|||||||
# Snarfnet Homelab Power Resilience & UPS Automation Plan
|
# Snarfnet Homelab Power Resilience & UPS Automation Plan
|
||||||
|
|
||||||
This document details the power resilience strategy, tiered graceful shutdown sequence, and integration architecture for the **snarfnet** homelab centered around the primary **NUT (Network UPS Tools) Master Server**.
|
This document details the comprehensive power resilience strategy, tiered graceful shutdown sequence, and automated recovery orchestration for the **snarfnet** homelab. The system is centered around a dedicated **NUT (Network UPS Tools) Master Server** at `14.4.4.44`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Core Architecture & IP Assignments
|
## 1. Core Architecture & Component Roles
|
||||||
|
|
||||||
The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
|
The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
|
||||||
|
|
||||||
| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description |
|
| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol |
|
||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. |
|
| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master`. |
|
||||||
| **NUT Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Compute / Swarm) | Peanut container UI consuming `14.4.4.44:3493`. |
|
| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; must be stopped prior to NAS Safe Mode. |
|
||||||
| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for alerts/triggers. |
|
| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth for Peanut, Gotify, Homebox, WLED. Runs on Longhorn storage in Aerodrome. |
|
||||||
| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493`. |
|
| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. |
|
||||||
| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. |
|
| **Emergency Push Notifications** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server. **Internal outage alerts MUST use direct IP (`http://140.44.4.70:8090`)** to bypass Authentik during shutdown. |
|
||||||
| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. |
|
| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and automated shedding. |
|
||||||
|
| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493` for battery voltage, load %, and runtime metrics. |
|
||||||
|
| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts NPM NFS shares (`/volume1/docker/nginx/data`). |
|
||||||
|
| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts secondary backup shares. |
|
||||||
|
| **Distributed Block Storage** | `longhorn.aerodrome.snarfnet.net` | `140.44.4.36:80` | VLAN 20 (Aerodrome K3s) | Longhorn CSI storage engine powering Authentik, AWX, Grafana PVCs. Protected with recurring snapshots (`snapshot-4h`, `snapshot-daily`). |
|
||||||
|
|
||||||
|
> [!IMPORTANT]
|
||||||
|
> **Emergency Alerting Path Warning:**
|
||||||
|
> `peanut.snarfnet.net` and `gotify.snarfnet.net` are protected externally behind Authentik Forward Auth (`oauth2-proxy`). During a power outage, when K3s `aerodrome` shuts down in Tier 3, Authentik goes offline. Therefore, **emergency outage alerts and NUT notification scripts must communicate directly with Gotify via internal IP (`http://140.44.4.70:8090/message?token=...`)** or native Home Assistant push notifications, bypassing NPM and Authentik.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -24,100 +32,134 @@ The power resilience architecture uses a centralized **NUT Master Server** host
|
|||||||
```mermaid
|
```mermaid
|
||||||
flowchart TD
|
flowchart TD
|
||||||
subgraph Grid ["Utility Power Grid"]
|
subgraph Grid ["Utility Power Grid"]
|
||||||
Mains["Main AC Power Supply"]
|
Mains["Main AC Utility Power"]
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph PowerHardware ["Power Protection"]
|
subgraph PowerHardware ["Power Hardware & Protection"]
|
||||||
UPS["Central UPS Hardware"]
|
UPS["Central UPS Hardware (Battery Bank)"]
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"]
|
subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"]
|
||||||
UPSD["upsd Daemon (Port 3493)"]
|
UPSD["upsd Daemon (Port 3493)"]
|
||||||
UPSMON_M["upsmon (Master)"]
|
UPSMON_M["upsmon (Master Daemon)"]
|
||||||
|
NOTIFY["nut-notify.sh (Direct Alert Engine)"]
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph Dashboards ["Monitoring & Alerting"]
|
subgraph OutageAlerts ["Emergency Notification Channels"]
|
||||||
Peanut["Peanut Dashboard\n(peanut.snarfnet.net:8050)"]
|
HA_Alert["Home Assistant (14.4.4.20)\nLocal Push Notification"]
|
||||||
HA["Home Assistant\n(14.4.4.20)"]
|
Gotify_Direct["Gotify Internal Direct\nhttp://140.44.4.70:8090\n(Bypasses Authentik / NPM)"]
|
||||||
|
Syslog["Wall / Syslog Broadcast"]
|
||||||
|
end
|
||||||
|
|
||||||
|
subgraph MonitoringPanels ["Monitoring & Management"]
|
||||||
|
Peanut["Peanut Dashboard\n(peanut.snarfnet.net)"]
|
||||||
Zabbix["Zabbix Server\n(140.44.4.61)"]
|
Zabbix["Zabbix Server\n(140.44.4.61)"]
|
||||||
end
|
end
|
||||||
|
|
||||||
subgraph Slaves ["UPS Slaves (upsmon Client Nodes)"]
|
subgraph InfrastructureConsumers ["Homelab Tiered Consumers"]
|
||||||
Harvester["Spacedock HCI Nodes\n(140.44.4.81 - 83)"]
|
SwarmNFS["Docker Swarm (140.44.4.71-75)\nNPM Host Mode (140.44.4.74)"]
|
||||||
K3sControl["K3s Control Planes\n(Aerodrome & Spaceport)"]
|
K8sCompute["Aerodrome & Spaceport K3s\nLonghorn Engine & Worker Nodes"]
|
||||||
K3sWorkers["K3s Worker Nodes"]
|
NAS["Synology NAS Arrays\n(14.10.10.71 / 14.10.10.72)"]
|
||||||
Swarm["Shipyard Swarm Nodes\n(140.44.4.71 - 75)"]
|
K8sControl["K3s Control Planes & etcd"]
|
||||||
NAS["Synology NAS Arrays\n(14.10.10.71 / 72)"]
|
Harvester["Spacedock HCI Hypervisors\n(140.44.4.81 - 83)"]
|
||||||
|
CoreEdge["UDM Pro Gateway (14.44.4.1)\nCore PoE Switches"]
|
||||||
end
|
end
|
||||||
|
|
||||||
Mains --> UPS
|
Mains --> UPS
|
||||||
UPS -- USB / SNMP --> UPSD
|
UPS -- USB / SNMP --> UPSD
|
||||||
UPSD --> UPSMON_M
|
UPSD --> UPSMON_M
|
||||||
|
UPSMON_M --> NOTIFY
|
||||||
|
|
||||||
|
NOTIFY -- Direct API Token --> Gotify_Direct
|
||||||
|
NOTIFY -- TCP 3493 Event --> HA_Alert
|
||||||
|
NOTIFY --> Syslog
|
||||||
|
|
||||||
UPSD -- TCP 3493 --> Peanut
|
UPSD -- TCP 3493 --> Peanut
|
||||||
UPSD -- TCP 3493 --> HA
|
|
||||||
UPSD -- TCP 3493 --> Zabbix
|
UPSD -- TCP 3493 --> Zabbix
|
||||||
|
|
||||||
UPSD -- TCP 3493 (Slave Broadcast) --> Harvester
|
UPSMON_M -. Tiered Signal .-> SwarmNFS
|
||||||
UPSD -- TCP 3493 (Slave Broadcast) --> K3sControl
|
UPSMON_M -. Tiered Signal .-> K8sCompute
|
||||||
UPSD -- TCP 3493 (Slave Broadcast) --> K3sWorkers
|
UPSMON_M -. Tiered Signal .-> NAS
|
||||||
UPSD -- TCP 3493 (Slave Broadcast) --> Swarm
|
UPSMON_M -. Tiered Signal .-> K8sControl
|
||||||
UPSD -- TCP 3493 (Slave Broadcast) --> NAS
|
UPSMON_M -. Tiered Signal .-> Harvester
|
||||||
|
UPSMON_M -. Final Cut .-> CoreEdge
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 3. Tiered Shutdown Hierarchy & Execution Timeline
|
## 3. Tiered Graceful Shutdown Hierarchy
|
||||||
|
|
||||||
To prevent data corruption and unexpected hard cutoffs, hosts shutdown in staged tiers based on power outage duration and battery level.
|
To eliminate data corruption, prevent stale NFS file handles, and ensure Longhorn volume detach cleanly, the homelab executes a **strict 6-stage progressive shutdown sequence**.
|
||||||
|
|
||||||
```
|
```
|
||||||
+-------------------------------------------------------------------------+
|
+--------------------------------------------------------------------------------------------------+
|
||||||
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT |
|
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT |
|
||||||
+-------------------------------------------------------------------------+
|
| - Broadcasts emergency alert via Home Assistant push & Gotify direct API (http://140.44.4.70:8090)|
|
||||||
|
|
+--------------------------------------------------------------------------------------------------+
|
||||||
v
|
|
|
||||||
+-------------------------------------------------------------------------+
|
v
|
||||||
| [T+3m / Batt <= 80%] TIER 1: Non-essential Workloads |
|
+--------------------------------------------------------------------------------------------------+
|
||||||
| - Dev VMs, transcoders, test containers in Swarm & K3s drained |
|
| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads |
|
||||||
+-------------------------------------------------------------------------+
|
| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped |
|
||||||
|
|
| - Media transcoders (Plex active sessions signaled) |
|
||||||
v
|
| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted |
|
||||||
+-------------------------------------------------------------------------+
|
+--------------------------------------------------------------------------------------------------+
|
||||||
| [T+7m / Batt <= 50%] TIER 2: Hypervisors, Compute Workers & Storage |
|
|
|
||||||
| - Harvester spacedock VMs gracefully shut down via virsh/API |
|
v
|
||||||
| - Aerodrome & Spaceport worker nodes gracefully halt |
|
+--------------------------------------------------------------------------------------------------+
|
||||||
| - Synology NAS arrays (14.10.10.71/72) enter Safe Mode & unmount pools |
|
| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM / Peanut) |
|
||||||
+-------------------------------------------------------------------------+
|
| - Stop `nginx-proxy-manager_app` & `nginx-proxy-manager_backup` on snshpd4lnxswm04 (140.44.4.74) |
|
||||||
|
|
| - Stop `peanut` container on Swarm |
|
||||||
v
|
| - Flushes open SQLite databases (database.sqlite) and unmounts NFS mounts to 14.10.10.71 |
|
||||||
+-------------------------------------------------------------------------+
|
+--------------------------------------------------------------------------------------------------+
|
||||||
| [T+12m / Batt <= 30%] TIER 3: Control Planes & Management Stack |
|
|
|
||||||
| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Control Planes|
|
v
|
||||||
| - Swarm Manager (140.44.4.71) & AI Nodes (140.44.4.7-8) halt |
|
+--------------------------------------------------------------------------------------------------+
|
||||||
+-------------------------------------------------------------------------+
|
| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach |
|
||||||
|
|
| - Harvester guest VMs (harvester.spacedock.snarfnet.net 140.44.4.90) gracefully shut down via API|
|
||||||
v
|
| - Cordon & scale down stateful pods on Aerodrome (Authentik PostgreSQL, AWX Postgres, Grafana) |
|
||||||
+-------------------------------------------------------------------------+
|
| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes |
|
||||||
| [T+15m / Batt Low] TIER 4: Forced Shutdown (FSD) & Hardware Cutoff |
|
| - Aerodrome (140.44.4.21-25) & Spaceport (140.44.4.51-56) worker nodes execute shutdown |
|
||||||
| - NUT Master (14.4.4.44) issues FSD signal to remaining clients |
|
+--------------------------------------------------------------------------------------------------+
|
||||||
| - NUT Master issues `upsdrvctl shutdown` (cuts UPS load outlet) |
|
|
|
||||||
| - Network Core (UDM Pro 14.44.4.1) stays alive until final battery drop |
|
v
|
||||||
+-------------------------------------------------------------------------+
|
+--------------------------------------------------------------------------------------------------+
|
||||||
|
| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode) |
|
||||||
|
| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode via NUT client trigger |
|
||||||
|
| - Unmounts all Btrfs volume pools, terminates NFS/iSCSI daemon threads, parks drive read heads |
|
||||||
|
| - Zero active network clients remain connected to storage |
|
||||||
|
+--------------------------------------------------------------------------------------------------+
|
||||||
|
|
|
||||||
|
v
|
||||||
|
+--------------------------------------------------------------------------------------------------+
|
||||||
|
| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors |
|
||||||
|
| - Aerodrome K3s Control Planes (140.44.4.11-13) take etcd snapshot and halt |
|
||||||
|
| - Spaceport K3s Control Planes (140.44.4.41-44) take etcd snapshot and halt |
|
||||||
|
| - Swarm Manager nodes (140.44.4.71-75) halt |
|
||||||
|
| - Harvester baremetal hypervisors (140.44.4.81-83) halt via IPMI/SSH |
|
||||||
|
+--------------------------------------------------------------------------------------------------+
|
||||||
|
|
|
||||||
|
v
|
||||||
|
+--------------------------------------------------------------------------------------------------+
|
||||||
|
| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut |
|
||||||
|
| - NUT Master (14.4.4.44) issues final FSD broadcast to all remaining clients |
|
||||||
|
| - NUT Master issues `upsdrvctl shutdown` with /etc/killpower flag (commands UPS load drop) |
|
||||||
|
| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops |
|
||||||
|
+--------------------------------------------------------------------------------------------------+
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 4. Configuration Blueprint & Service Integrations
|
## 4. Configuration Blueprints & Integration Scripts
|
||||||
|
|
||||||
### A. NUT Master Server (`14.4.4.44`) Configuration Files
|
### A. NUT Master Server (`14.4.4.44`)
|
||||||
|
|
||||||
#### `/etc/nut/ups.conf`
|
#### `/etc/nut/ups.conf`
|
||||||
```ini
|
```ini
|
||||||
[snarf-ups]
|
[snarf-ups]
|
||||||
driver = usbhid-ups
|
driver = usbhid-ups
|
||||||
port = auto
|
port = auto
|
||||||
desc = "Snarfnet Main Server Rack UPS"
|
desc = "Snarfnet Main Rack UPS"
|
||||||
pollinterval = 5
|
pollinterval = 5
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -131,7 +173,7 @@ MAXAGE 15
|
|||||||
#### `/etc/nut/upsd.users`
|
#### `/etc/nut/upsd.users`
|
||||||
```ini
|
```ini
|
||||||
[admin]
|
[admin]
|
||||||
password = SUPER_SECRET_ADMIN_PASS
|
password = ADMIN_PASSWORD_CHANGE_ME
|
||||||
actions = SET
|
actions = SET
|
||||||
instcmds = ALL
|
instcmds = ALL
|
||||||
|
|
||||||
@@ -152,7 +194,7 @@ MAXAGE 15
|
|||||||
```ini
|
```ini
|
||||||
MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master
|
MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master
|
||||||
MINSUPPLIES 1
|
MINSUPPLIES 1
|
||||||
SHUTDOWNCMD "/sbin/shutdown -h +0"
|
SHUTDOWNCMD "/usr/local/bin/homelab_power_shutdown.sh"
|
||||||
NOTIFYCMD /usr/local/bin/nut-notify.sh
|
NOTIFYCMD /usr/local/bin/nut-notify.sh
|
||||||
POWERDOWNFLAG /etc/killpower
|
POWERDOWNFLAG /etc/killpower
|
||||||
|
|
||||||
@@ -166,40 +208,164 @@ NOTIFYFLAG COMMOK SYSLOG+WALL+EXEC
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### B. Client Node Configuration (`upsmon` Slave)
|
### B. Outage Notification Engine (`/usr/local/bin/nut-notify.sh`)
|
||||||
|
|
||||||
Applies to Harvester hypervisors, K3s master/worker hosts, Docker Swarm hosts, and Debian Linux nodes.
|
This script runs on the NUT Master (`14.4.4.44`) whenever a power state event triggers. It sends push notifications directly to Gotify (`http://140.44.4.70:8090`) and Home Assistant (`http://14.4.4.20:8123`) without touching NPM or Authentik.
|
||||||
|
|
||||||
#### `/etc/nut/upsmon.conf` (Slave Mode)
|
```bash
|
||||||
```ini
|
#!/bin/bash
|
||||||
MONITOR snarf-ups@14.4.4.44 1 upsmon_slave SLAVE_MONITOR_PASS slave
|
# /usr/local/bin/nut-notify.sh
|
||||||
MINSUPPLIES 1
|
# Emergency power notification script for Network UPS Tools
|
||||||
SHUTDOWNCMD "/sbin/shutdown -h +0"
|
set -euo pipefail
|
||||||
POWERDOWNFLAG /etc/killpower
|
|
||||||
HOSTSYNC 15
|
EVENT_TYPE="${1:-$NOTIFYTYPE}"
|
||||||
DEADTIME 15
|
UPS_NAME="${UPSNAME:-snarf-ups}"
|
||||||
RBWARNTIME 43200
|
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
|
||||||
NOCOMMWARNTIME 300
|
|
||||||
FINALDELAY 5
|
GOTIFY_INTERNAL_URL="http://140.44.4.70:8090/message"
|
||||||
|
GOTIFY_APP_TOKEN="A4uN4fEmErGeNcYtOkEn" # Replace with valid Gotify App Token
|
||||||
|
|
||||||
|
send_gotify() {
|
||||||
|
local title="$1"
|
||||||
|
local message="$2"
|
||||||
|
local priority="$3"
|
||||||
|
|
||||||
|
curl -s -m 5 -X POST "${GOTIFY_INTERNAL_URL}?token=${GOTIFY_APP_TOKEN}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "{\"title\": \"${title}\", \"message\": \"${message}\", \"priority\": ${priority}}" >/dev/null 2>&1 || true
|
||||||
|
}
|
||||||
|
|
||||||
|
case "$EVENT_TYPE" in
|
||||||
|
ONBATT)
|
||||||
|
send_gotify "⚡ Grid Power Lost" "[$TIMESTAMP] Homelab is running on UPS battery (${UPS_NAME}). Staged shutdown countdown started." 8
|
||||||
|
logger -t nut-notify "Grid power failure detected - running on UPS battery."
|
||||||
|
;;
|
||||||
|
ONLINE)
|
||||||
|
send_gotify "✅ Grid Power Restored" "[$TIMESTAMP] Utility AC power restored to ${UPS_NAME}. Battery recharging." 5
|
||||||
|
logger -t nut-notify "AC utility power restored."
|
||||||
|
;;
|
||||||
|
LOWBATT)
|
||||||
|
send_gotify "🚨 UPS Battery Critical" "[$TIMESTAMP] Battery level critical on ${UPS_NAME}! Final Tier 4-6 shutdown initiated." 10
|
||||||
|
logger -t nut-notify "CRITICAL: UPS battery low. Enforcing emergency shutdown."
|
||||||
|
;;
|
||||||
|
FSD)
|
||||||
|
send_gotify "🛑 Forced Shutdown Engaged" "[$TIMESTAMP] NUT Master has commanded immediate power cutoff across all clusters." 10
|
||||||
|
logger -t nut-notify "FSD signal received. Halting systems."
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
logger -t nut-notify "Unhandled UPS event: ${EVENT_TYPE}"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### C. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`)
|
### C. Orchestrated Tiered Shutdown Script (`/usr/local/bin/homelab_power_shutdown.sh`)
|
||||||
|
|
||||||
|
This script is invoked by `upsmon` on `14.4.4.44` when an outage exceeds the battery threshold:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
# /usr/local/bin/homelab_power_shutdown.sh
|
||||||
|
# Staged graceful homelab shutdown orchestrator
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [POWER-SHUTDOWN] $*" | tee -a /var/log/power-shutdown.log; }
|
||||||
|
SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes"
|
||||||
|
|
||||||
|
log "=== INITIATING HOMELAB TIERED SHUTDOWN ==="
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 1: Non-Essential Compute & AI Pis
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 1: Halting AI Raspberry Pi stack (140.44.4.7, 140.44.4.8)..."
|
||||||
|
ssh $SSH_OPTS root@140.44.4.7 "shutdown -h now" 2>/dev/null || true
|
||||||
|
ssh $SSH_OPTS root@140.44.4.8 "shutdown -h now" 2>/dev/null || true
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 2: Docker Swarm NFS Workloads (NPM & Peanut)
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 2: Stopping Swarm services mounting Synology NFS (NPM, Peanut)..."
|
||||||
|
ssh $SSH_OPTS root@140.44.4.71 "docker service scale nginx-proxy-manager_app=0 nginx-proxy-manager_backup=0 peanut_peanut=0" 2>/dev/null || true
|
||||||
|
sleep 5
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 3: Harvester Guest VMs & Kubernetes Worker Nodes
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 3: Shutting down Harvester guest VMs and K3s workers..."
|
||||||
|
# Gracefully signal Harvester VMs
|
||||||
|
ssh $SSH_OPTS root@140.44.4.81 "virsh list --name | xargs -r -n1 virsh shutdown" 2>/dev/null || true
|
||||||
|
|
||||||
|
# Stop K3s agent on Aerodrome workers (clean Longhorn PVC detach)
|
||||||
|
for ip in 140.44.4.22 140.44.4.23 140.44.4.24 140.44.4.25; do
|
||||||
|
ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# Stop K3s agent on Spaceport workers
|
||||||
|
for ip in 140.44.4.51 140.44.4.52 140.44.4.53 140.44.4.54 140.44.4.55 140.44.4.56; do
|
||||||
|
ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
sleep 15
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 4: Storage Arrays (Synology DSM Safe Mode)
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 4: Triggering Synology NAS Safe Mode (14.10.10.71, 14.10.10.72)..."
|
||||||
|
# Synology enters safe mode automatically via NUT upsd client signal (port 3493)
|
||||||
|
# Explicit API / SSH command to flush pools if configured
|
||||||
|
ssh $SSH_OPTS admin@14.10.10.71 "synopoweroff" 2>/dev/null || true
|
||||||
|
ssh $SSH_OPTS admin@14.10.10.72 "synopoweroff" 2>/dev/null || true
|
||||||
|
sleep 20
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 5: Control Planes & Swarm Managers
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 5: Halting K3s control planes and Swarm managers..."
|
||||||
|
# Spaceport Masters (140.44.4.41-44)
|
||||||
|
for ip in 140.44.4.41 140.44.4.42 140.44.4.43 140.44.4.44; do
|
||||||
|
ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# Aerodrome Masters (140.44.4.11-13)
|
||||||
|
for ip in 140.44.4.11 140.44.4.12 140.44.4.13; do
|
||||||
|
ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# Swarm Managers (140.44.4.71, 72, 74)
|
||||||
|
for ip in 140.44.4.71 140.44.4.72 140.44.4.74; do
|
||||||
|
ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# Harvester Baremetal Hypervisors (140.44.4.81-83)
|
||||||
|
for ip in 140.44.4.81 140.44.4.82 140.44.4.83; do
|
||||||
|
ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 6: Master Node Self-Cutoff (FSD)
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 6: Executing final local halt and arming killpower..."
|
||||||
|
touch /etc/killpower
|
||||||
|
/sbin/shutdown -h +0
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### D. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`)
|
||||||
|
|
||||||
1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**.
|
1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**.
|
||||||
2. Check **Enable UPS Support**.
|
2. Check **Enable UPS Support**.
|
||||||
3. Select UPS Type: **Synology UPS Server / Network NUT Server**.
|
3. Select UPS Type: **Synology UPS Server / Network NUT Server**.
|
||||||
4. Network UPS server IP address: **`14.4.4.44`**.
|
4. Network UPS server IP address: **`14.4.4.44`**.
|
||||||
5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `5 minutes` (or `Upon low battery`).
|
5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `6 minutes` (guaranteeing that Tier 2 Swarm NFS workloads have cleanly scaled down before disk pools unmount).
|
||||||
|
6. Check **Shut down UPS when the system enters Standby Mode** (disabled, managed centrally by `14.4.4.44`).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### D. Peanut Web Dashboard (`peanut.snarfnet.net`)
|
### E. Peanut Web Dashboard Configuration
|
||||||
|
|
||||||
Peanut runs on Docker Swarm (`140.44.4.70:8050`). Ensure the Peanut service configuration references the NUT server at `14.4.4.44`:
|
Peanut runs on Docker Swarm (`140.44.4.70:8050`). It connects directly to the NUT master server over internal TCP 3493:
|
||||||
|
|
||||||
#### Updated `docker-deployments/peanut.yml`
|
|
||||||
```yaml
|
```yaml
|
||||||
services:
|
services:
|
||||||
peanut:
|
peanut:
|
||||||
@@ -226,38 +392,25 @@ volumes:
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### E. Home Assistant (`14.4.4.20`) Integration & Automation
|
## 5. Automated Power Restoration & Boot Sequence
|
||||||
|
|
||||||
1. Navigate to **Settings** $\rightarrow$ **Devices & Services** $\rightarrow$ **Add Integration** $\rightarrow$ **Network UPS Tools (NUT)**.
|
When utility grid power returns, the homelab automatically recovers in a **strict bottom-up reverse sequence** to guarantee all storage and network prerequisites are ready before compute services attempt to start.
|
||||||
2. Host: **`14.4.4.44`**, Port: **`3493`**.
|
|
||||||
3. Username: **`upsmon_slave`**, Password: **`SLAVE_MONITOR_PASS`**.
|
|
||||||
|
|
||||||
#### Automation Snippet (`automations.yaml`)
|
```mermaid
|
||||||
```yaml
|
flowchart TD
|
||||||
alias: "Power Resilience: Grid Power Lost Alert"
|
G_On["1. Grid AC Restored\nUPS Battery Reaches >= 30%"] --> P1["2. Phase 1: Core Network & Edge\nUDM Pro (14.44.4.1) & Core PoE Switches\nEstablish VLANs, DHCP, Inter-VLAN Routing"]
|
||||||
trigger:
|
P1 --> P2["3. Phase 2: NUT Master\nNode at 14.4.4.44 Boots (BIOS: Always On)\nupsd daemon starts, monitors line quality"]
|
||||||
- platform: state
|
P2 --> P3["4. Phase 3: Storage Foundation\nSynology SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) Boot\nMount Btrfs pools, start NFS & iSCSI daemons"]
|
||||||
entity_id: sensor.snarf_ups_status
|
P3 --> P4["5. Phase 4: Control Planes & Hypervisors\n- Harvester spacedock (140.44.4.81-83) boots via IPMI/WOL\n- Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) K3s masters boot\n- etcd clusters re-establish quorum"]
|
||||||
to: "On Battery"
|
P4 --> P5["6. Phase 5: Storage CSI & Compute Workers\n- Aerodrome & Spaceport worker nodes boot\n- Longhorn CSI engine initializes, verifies volume health, mounts PVCs"]
|
||||||
action:
|
P5 --> P6["7. Phase 6: Ingress & Workloads\n- Docker Swarm (140.44.4.71-75) boots\n- NPM host-mode container binds 80/443/81 on 140.44.4.74\n- Authentik SSO & Forward Auth come online\n- Applications (Peanut, Gotify, Homebox, etc.) become reachable"]
|
||||||
- service: notify.gotify
|
|
||||||
data:
|
|
||||||
title: "⚡ Power Outage Alert"
|
|
||||||
message: "Grid power failure detected! Homelab is running on UPS battery (14.4.4.44). Tiered shutdown in progress."
|
|
||||||
priority: 8
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
### Critical Health Verification Gates:
|
||||||
|
|
||||||
## 5. Power Restoration & Auto-Boot Sequence
|
1. **Storage Gate (Phase 3 $\rightarrow$ Phase 4):**
|
||||||
|
- Synology NAS NFS daemons must be responding on `14.10.10.71:2049` before Docker Swarm starts, ensuring NPM does not encounter stale NFS mounts or SQLite locking.
|
||||||
When grid power returns, systems auto-recover in reverse order:
|
2. **Database Gate (Phase 5 $\rightarrow$ Phase 6):**
|
||||||
|
- Longhorn volume `data-authentik-postgresql-0` on Aerodrome must reach `Healthy` state before Authentik server pods start.
|
||||||
1. **UPS AC Mains Restored**: Physical UPS re-enables output outlets once battery reaches safe threshold ($\ge 20\%$).
|
3. **Ingress Gate (Phase 6):**
|
||||||
2. **Phase 1 - Gateway & Edge**: UDM Pro (`14.44.4.1`) & Core switches boot up and establish internal VLAN routes.
|
- Nginx Proxy Manager on `snshpd4lnxswm04` (`140.44.4.74`) confirms host-mode ports `80` and `443` are bound and forwards traffic to internal VIPs.
|
||||||
3. **Phase 2 - NUT Master**: Host at `14.4.4.44` boots (BIOS `Restore AC Power` set to `Always On`). `upsd` service starts.
|
|
||||||
4. **Phase 3 - Storage NAS**: Synology NAS nodes (`14.10.10.71`, `14.10.10.72`) auto-boot upon power recovery.
|
|
||||||
5. **Phase 4 - Control Planes & Hypervisors**:
|
|
||||||
- Harvester `spacedock` hosts (`140.44.4.81-83`) boot via Wake-on-LAN / IPMI / BIOS settings.
|
|
||||||
- K3s master nodes (`aerodrome` & `spaceport`) boot, resume etcd quorum.
|
|
||||||
6. **Phase 5 - Worker Compute & Workloads**: Worker nodes boot; Kubernetes and Docker Swarm automatically reschedule workloads.
|
|
||||||
|
|||||||
80
scripts/homelab_power_shutdown.sh
Normal file
80
scripts/homelab_power_shutdown.sh
Normal file
@@ -0,0 +1,80 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# /usr/local/bin/homelab_power_shutdown.sh
|
||||||
|
# Staged graceful homelab shutdown orchestrator for Snarfnet
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [POWER-SHUTDOWN] $*" | tee -a /var/log/power-shutdown.log; }
|
||||||
|
SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes"
|
||||||
|
|
||||||
|
log "=== INITIATING HOMELAB TIERED SHUTDOWN ==="
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 1: Non-Essential Compute & AI Pis
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 1: Halting AI Raspberry Pi stack (140.44.4.7, 140.44.4.8)..."
|
||||||
|
ssh $SSH_OPTS root@140.44.4.7 "shutdown -h now" 2>/dev/null || true
|
||||||
|
ssh $SSH_OPTS root@140.44.4.8 "shutdown -h now" 2>/dev/null || true
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 2: Docker Swarm NFS Workloads (NPM & Peanut)
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 2: Stopping Swarm services mounting Synology NFS (NPM, Peanut)..."
|
||||||
|
ssh $SSH_OPTS root@140.44.4.71 "docker service scale nginx-proxy-manager_app=0 nginx-proxy-manager_backup=0 peanut_peanut=0" 2>/dev/null || true
|
||||||
|
sleep 5
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 3: Harvester Guest VMs & Kubernetes Worker Nodes
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 3: Shutting down Harvester guest VMs and K3s workers..."
|
||||||
|
# Gracefully signal Harvester VMs
|
||||||
|
ssh $SSH_OPTS root@140.44.4.81 "virsh list --name | xargs -r -n1 virsh shutdown" 2>/dev/null || true
|
||||||
|
|
||||||
|
# Stop K3s agent on Aerodrome workers (clean Longhorn PVC detach)
|
||||||
|
for ip in 140.44.4.22 140.44.4.23 140.44.4.24 140.44.4.25; do
|
||||||
|
ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# Stop K3s agent on Spaceport workers
|
||||||
|
for ip in 140.44.4.51 140.44.4.52 140.44.4.53 140.44.4.54 140.44.4.55 140.44.4.56; do
|
||||||
|
ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
sleep 15
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 4: Storage Arrays (Synology DSM Safe Mode)
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 4: Triggering Synology NAS Safe Mode (14.10.10.71, 14.10.10.72)..."
|
||||||
|
ssh $SSH_OPTS admin@14.10.10.71 "synopoweroff" 2>/dev/null || true
|
||||||
|
ssh $SSH_OPTS admin@14.10.10.72 "synopoweroff" 2>/dev/null || true
|
||||||
|
sleep 20
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 5: Control Planes & Swarm Managers
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 5: Halting K3s control planes and Swarm managers..."
|
||||||
|
# Spaceport Masters (140.44.4.41-44)
|
||||||
|
for ip in 140.44.4.41 140.44.4.42 140.44.4.43 140.44.4.44; do
|
||||||
|
ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# Aerodrome Masters (140.44.4.11-13)
|
||||||
|
for ip in 140.44.4.11 140.44.4.12 140.44.4.13; do
|
||||||
|
ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# Swarm Managers (140.44.4.71, 72, 74)
|
||||||
|
for ip in 140.44.4.71 140.44.4.72 140.44.4.74; do
|
||||||
|
ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# Harvester Baremetal Hypervisors (140.44.4.81-83)
|
||||||
|
for ip in 140.44.4.81 140.44.4.82 140.44.4.83; do
|
||||||
|
ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
|
||||||
|
done
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# TIER 6: Master Node Self-Cutoff (FSD)
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
log "Stage 6: Executing final local halt and arming killpower..."
|
||||||
|
touch /etc/killpower
|
||||||
|
/sbin/shutdown -h +0
|
||||||
44
scripts/nut_notify.sh
Normal file
44
scripts/nut_notify.sh
Normal file
@@ -0,0 +1,44 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# /usr/local/bin/nut-notify.sh
|
||||||
|
# Emergency power notification script for Network UPS Tools on Snarfnet
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
EVENT_TYPE="${1:-$NOTIFYTYPE}"
|
||||||
|
UPS_NAME="${UPSNAME:-snarf-ups}"
|
||||||
|
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
|
||||||
|
|
||||||
|
# Internal Gotify VIP endpoint (bypasses Authentik Forward Auth and NPM)
|
||||||
|
GOTIFY_INTERNAL_URL="http://140.44.4.70:8090/message"
|
||||||
|
GOTIFY_APP_TOKEN="${GOTIFY_APP_TOKEN:-A4uN4fEmErGeNcYtOkEn}"
|
||||||
|
|
||||||
|
send_gotify() {
|
||||||
|
local title="$1"
|
||||||
|
local message="$2"
|
||||||
|
local priority="$3"
|
||||||
|
|
||||||
|
curl -s -m 5 -X POST "${GOTIFY_INTERNAL_URL}?token=${GOTIFY_APP_TOKEN}" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "{\"title\": \"${title}\", \"message\": \"${message}\", \"priority\": ${priority}}" >/dev/null 2>&1 || true
|
||||||
|
}
|
||||||
|
|
||||||
|
case "$EVENT_TYPE" in
|
||||||
|
ONBATT)
|
||||||
|
send_gotify "⚡ Grid Power Lost" "[$TIMESTAMP] Homelab is running on UPS battery (${UPS_NAME}). Staged shutdown countdown started." 8
|
||||||
|
logger -t nut-notify "Grid power failure detected - running on UPS battery."
|
||||||
|
;;
|
||||||
|
ONLINE)
|
||||||
|
send_gotify "✅ Grid Power Restored" "[$TIMESTAMP] Utility AC power restored to ${UPS_NAME}. Battery recharging." 5
|
||||||
|
logger -t nut-notify "AC utility power restored."
|
||||||
|
;;
|
||||||
|
LOWBATT)
|
||||||
|
send_gotify "🚨 UPS Battery Critical" "[$TIMESTAMP] Battery level critical on ${UPS_NAME}! Final Tier 4-6 shutdown initiated." 10
|
||||||
|
logger -t nut-notify "CRITICAL: UPS battery low. Enforcing emergency shutdown."
|
||||||
|
;;
|
||||||
|
FSD)
|
||||||
|
send_gotify "🛑 Forced Shutdown Engaged" "[$TIMESTAMP] NUT Master has commanded immediate power cutoff across all clusters." 10
|
||||||
|
logger -t nut-notify "FSD signal received. Halting systems."
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
logger -t nut-notify "Unhandled UPS event: ${EVENT_TYPE}"
|
||||||
|
;;
|
||||||
|
esac
|
||||||
Reference in New Issue
Block a user