Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: modernize power resilience plan and shutdown orchestration with NFS-safe tiering, direct Gotify alerts, and Longhorn clean detach

Browse Source
This commit is contained in:
Snarf
2026-10-30 15:21:22 -04:00
parent d34852075a
commit c492afc377
3 changed files with 394 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files

387
docs/power_resilience_plan.md
View File

@@ -1,21 +1,29 @@
# Snarfnet Homelab Power Resilience & UPS Automation Plan
This document details the power resilience strategy, tiered graceful shutdown sequence, and integration architecture for the **snarfnet** homelab centered around the primary **NUT (Network UPS Tools) Master Server**.
This document details the comprehensive power resilience strategy, tiered graceful shutdown sequence, and automated recovery orchestration for the **snarfnet** homelab. The system is centered around a dedicated **NUT (Network UPS Tools) Master Server** at `14.4.4.44`.
---
## 1. Core Architecture & IP Assignments
## 1. Core Architecture & Component Roles
The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description |
| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol |
|---|---|---|---|---|
| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. |
| **NUT Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Compute / Swarm) | Peanut container UI consuming `14.4.4.44:3493`. |
| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for alerts/triggers. |
| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493`. |
| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. |
| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. |
| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master`. |
| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; must be stopped prior to NAS Safe Mode. |
| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth for Peanut, Gotify, Homebox, WLED. Runs on Longhorn storage in Aerodrome. |
| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. |
| **Emergency Push Notifications** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server. **Internal outage alerts MUST use direct IP (`http://140.44.4.70:8090`)** to bypass Authentik during shutdown. |
| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and automated shedding. |
| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493` for battery voltage, load %, and runtime metrics. |
| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts NPM NFS shares (`/volume1/docker/nginx/data`). |
| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts secondary backup shares. |
| **Distributed Block Storage** | `longhorn.aerodrome.snarfnet.net` | `140.44.4.36:80` | VLAN 20 (Aerodrome K3s) | Longhorn CSI storage engine powering Authentik, AWX, Grafana PVCs. Protected with recurring snapshots (`snapshot-4h`, `snapshot-daily`). |
> [!IMPORTANT]
> **Emergency Alerting Path Warning:**
> `peanut.snarfnet.net` and `gotify.snarfnet.net` are protected externally behind Authentik Forward Auth (`oauth2-proxy`). During a power outage, when K3s `aerodrome` shuts down in Tier 3, Authentik goes offline. Therefore, **emergency outage alerts and NUT notification scripts must communicate directly with Gotify via internal IP (`http://140.44.4.70:8090/message?token=...`)** or native Home Assistant push notifications, bypassing NPM and Authentik.
---
@@ -24,100 +32,134 @@ The power resilience architecture uses a centralized **NUT Master Server** host
```mermaid
flowchart TD
subgraph Grid ["Utility Power Grid"]
Mains["Main AC Power Supply"]
Mains["Main AC Utility Power"]
end
subgraph PowerHardware ["Power Protection"]
UPS["Central UPS Hardware"]
subgraph PowerHardware ["Power Hardware & Protection"]
UPS["Central UPS Hardware (Battery Bank)"]
end
subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"]
UPSD["upsd Daemon (Port 3493)"]
UPSMON_M["upsmon (Master)"]
UPSMON_M["upsmon (Master Daemon)"]
NOTIFY["nut-notify.sh (Direct Alert Engine)"]
end
subgraph Dashboards ["Monitoring & Alerting"]
Peanut["Peanut Dashboard\n(peanut.snarfnet.net:8050)"]
HA["Home Assistant\n(14.4.4.20)"]
subgraph OutageAlerts ["Emergency Notification Channels"]
HA_Alert["Home Assistant (14.4.4.20)\nLocal Push Notification"]
Gotify_Direct["Gotify Internal Direct\nhttp://140.44.4.70:8090\n(Bypasses Authentik / NPM)"]
Syslog["Wall / Syslog Broadcast"]
end
subgraph MonitoringPanels ["Monitoring & Management"]
Peanut["Peanut Dashboard\n(peanut.snarfnet.net)"]
Zabbix["Zabbix Server\n(140.44.4.61)"]
end
subgraph Slaves ["UPS Slaves (upsmon Client Nodes)"]
Harvester["Spacedock HCI Nodes\n(140.44.4.81 - 83)"]
K3sControl["K3s Control Planes\n(Aerodrome & Spaceport)"]
K3sWorkers["K3s Worker Nodes"]
Swarm["Shipyard Swarm Nodes\n(140.44.4.71 - 75)"]
NAS["Synology NAS Arrays\n(14.10.10.71 / 72)"]
subgraph InfrastructureConsumers ["Homelab Tiered Consumers"]
SwarmNFS["Docker Swarm (140.44.4.71-75)\nNPM Host Mode (140.44.4.74)"]
K8sCompute["Aerodrome & Spaceport K3s\nLonghorn Engine & Worker Nodes"]
NAS["Synology NAS Arrays\n(14.10.10.71 / 14.10.10.72)"]
K8sControl["K3s Control Planes & etcd"]
Harvester["Spacedock HCI Hypervisors\n(140.44.4.81 - 83)"]
CoreEdge["UDM Pro Gateway (14.44.4.1)\nCore PoE Switches"]
end
Mains --> UPS
UPS -- USB / SNMP --> UPSD
UPSD --> UPSMON_M
UPSMON_M --> NOTIFY
NOTIFY -- Direct API Token --> Gotify_Direct
NOTIFY -- TCP 3493 Event --> HA_Alert
NOTIFY --> Syslog
UPSD -- TCP 3493 --> Peanut
UPSD -- TCP 3493 --> HA
UPSD -- TCP 3493 --> Zabbix
UPSD -- TCP 3493 (Slave Broadcast) --> Harvester
UPSD -- TCP 3493 (Slave Broadcast) --> K3sControl
UPSD -- TCP 3493 (Slave Broadcast) --> K3sWorkers
UPSD -- TCP 3493 (Slave Broadcast) --> Swarm
UPSD -- TCP 3493 (Slave Broadcast) --> NAS
UPSMON_M -. Tiered Signal .-> SwarmNFS
UPSMON_M -. Tiered Signal .-> K8sCompute
UPSMON_M -. Tiered Signal .-> NAS
UPSMON_M -. Tiered Signal .-> K8sControl
UPSMON_M -. Tiered Signal .-> Harvester
UPSMON_M -. Final Cut .-> CoreEdge
```
---
## 3. Tiered Shutdown Hierarchy & Execution Timeline
## 3. Tiered Graceful Shutdown Hierarchy
To prevent data corruption and unexpected hard cutoffs, hosts shutdown in staged tiers based on power outage duration and battery level.
To eliminate data corruption, prevent stale NFS file handles, and ensure Longhorn volume detach cleanly, the homelab executes a **strict 6-stage progressive shutdown sequence**.
```
+-------------------------------------------------------------------------+
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| [T+3m / Batt <= 80%] TIER 1: Non-essential Workloads |
| - Dev VMs, transcoders, test containers in Swarm & K3s drained |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| [T+7m / Batt <= 50%] TIER 2: Hypervisors, Compute Workers & Storage |
| - Harvester spacedock VMs gracefully shut down via virsh/API |
| - Aerodrome & Spaceport worker nodes gracefully halt |
| - Synology NAS arrays (14.10.10.71/72) enter Safe Mode & unmount pools |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| [T+12m / Batt <= 30%] TIER 3: Control Planes & Management Stack |
| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Control Planes|
| - Swarm Manager (140.44.4.71) & AI Nodes (140.44.4.7-8) halt |
+-------------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------------+
| [T+15m / Batt Low] TIER 4: Forced Shutdown (FSD) & Hardware Cutoff |
| - NUT Master (14.4.4.44) issues FSD signal to remaining clients |
| - NUT Master issues `upsdrvctl shutdown` (cuts UPS load outlet) |
| - Network Core (UDM Pro 14.44.4.1) stays alive until final battery drop |
+-------------------------------------------------------------------------+
+--------------------------------------------------------------------------------------------------+
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT |
| - Broadcasts emergency alert via Home Assistant push & Gotify direct API (http://140.44.4.70:8090)|
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads |
| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped |
| - Media transcoders (Plex active sessions signaled) |
| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM / Peanut) |
| - Stop `nginx-proxy-manager_app` & `nginx-proxy-manager_backup` on snshpd4lnxswm04 (140.44.4.74) |
| - Stop `peanut` container on Swarm |
| - Flushes open SQLite databases (database.sqlite) and unmounts NFS mounts to 14.10.10.71 |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach |
| - Harvester guest VMs (harvester.spacedock.snarfnet.net 140.44.4.90) gracefully shut down via API|
| - Cordon & scale down stateful pods on Aerodrome (Authentik PostgreSQL, AWX Postgres, Grafana) |
| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes |
| - Aerodrome (140.44.4.21-25) & Spaceport (140.44.4.51-56) worker nodes execute shutdown |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode) |
| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode via NUT client trigger |
| - Unmounts all Btrfs volume pools, terminates NFS/iSCSI daemon threads, parks drive read heads |
| - Zero active network clients remain connected to storage |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors |
| - Aerodrome K3s Control Planes (140.44.4.11-13) take etcd snapshot and halt |
| - Spaceport K3s Control Planes (140.44.4.41-44) take etcd snapshot and halt |
| - Swarm Manager nodes (140.44.4.71-75) halt |
| - Harvester baremetal hypervisors (140.44.4.81-83) halt via IPMI/SSH |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut |
| - NUT Master (14.4.4.44) issues final FSD broadcast to all remaining clients |
| - NUT Master issues `upsdrvctl shutdown` with /etc/killpower flag (commands UPS load drop) |
| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops |
+--------------------------------------------------------------------------------------------------+
```
---
## 4. Configuration Blueprint & Service Integrations
## 4. Configuration Blueprints & Integration Scripts
### A. NUT Master Server (`14.4.4.44`) Configuration Files
### A. NUT Master Server (`14.4.4.44`)
#### `/etc/nut/ups.conf`
```ini
[snarf-ups]
driver = usbhid-ups
port = auto
desc = "Snarfnet Main Server Rack UPS"
desc = "Snarfnet Main Rack UPS"
pollinterval = 5
```
@@ -131,7 +173,7 @@ MAXAGE 15
#### `/etc/nut/upsd.users`
```ini
[admin]
password = SUPER_SECRET_ADMIN_PASS
password = ADMIN_PASSWORD_CHANGE_ME
actions = SET
instcmds = ALL
@@ -152,7 +194,7 @@ MAXAGE 15
```ini
MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
SHUTDOWNCMD "/usr/local/bin/homelab_power_shutdown.sh"
NOTIFYCMD /usr/local/bin/nut-notify.sh
POWERDOWNFLAG /etc/killpower
@@ -166,40 +208,164 @@ NOTIFYFLAG COMMOK SYSLOG+WALL+EXEC
---
### B. Client Node Configuration (`upsmon` Slave)
### B. Outage Notification Engine (`/usr/local/bin/nut-notify.sh`)
Applies to Harvester hypervisors, K3s master/worker hosts, Docker Swarm hosts, and Debian Linux nodes.
This script runs on the NUT Master (`14.4.4.44`) whenever a power state event triggers. It sends push notifications directly to Gotify (`http://140.44.4.70:8090`) and Home Assistant (`http://14.4.4.20:8123`) without touching NPM or Authentik.
#### `/etc/nut/upsmon.conf` (Slave Mode)
```ini
MONITOR snarf-ups@14.4.4.44 1 upsmon_slave SLAVE_MONITOR_PASS slave
MINSUPPLIES 1
SHUTDOWNCMD "/sbin/shutdown -h +0"
POWERDOWNFLAG /etc/killpower
HOSTSYNC 15
DEADTIME 15
RBWARNTIME 43200
NOCOMMWARNTIME 300
FINALDELAY 5
```bash
#!/bin/bash
# /usr/local/bin/nut-notify.sh
# Emergency power notification script for Network UPS Tools
set -euo pipefail
EVENT_TYPE="${1:-$NOTIFYTYPE}"
UPS_NAME="${UPSNAME:-snarf-ups}"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
GOTIFY_INTERNAL_URL="http://140.44.4.70:8090/message"
GOTIFY_APP_TOKEN="A4uN4fEmErGeNcYtOkEn" # Replace with valid Gotify App Token
send_gotify() {
local title="$1"
local message="$2"
local priority="$3"
curl -s -m 5 -X POST "${GOTIFY_INTERNAL_URL}?token=${GOTIFY_APP_TOKEN}" \
-H "Content-Type: application/json" \
-d "{\"title\": \"${title}\", \"message\": \"${message}\", \"priority\": ${priority}}" >/dev/null 2>&1 || true
}
case "$EVENT_TYPE" in
ONBATT)
send_gotify "⚡ Grid Power Lost" "[$TIMESTAMP] Homelab is running on UPS battery (${UPS_NAME}). Staged shutdown countdown started." 8
logger -t nut-notify "Grid power failure detected - running on UPS battery."
;;
ONLINE)
send_gotify "✅ Grid Power Restored" "[$TIMESTAMP] Utility AC power restored to ${UPS_NAME}. Battery recharging." 5
logger -t nut-notify "AC utility power restored."
;;
LOWBATT)
send_gotify "🚨 UPS Battery Critical" "[$TIMESTAMP] Battery level critical on ${UPS_NAME}! Final Tier 4-6 shutdown initiated." 10
logger -t nut-notify "CRITICAL: UPS battery low. Enforcing emergency shutdown."
;;
FSD)
send_gotify "🛑 Forced Shutdown Engaged" "[$TIMESTAMP] NUT Master has commanded immediate power cutoff across all clusters." 10
logger -t nut-notify "FSD signal received. Halting systems."
;;
*)
logger -t nut-notify "Unhandled UPS event: ${EVENT_TYPE}"
;;
esac
```
---
### C. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`)
### C. Orchestrated Tiered Shutdown Script (`/usr/local/bin/homelab_power_shutdown.sh`)
This script is invoked by `upsmon` on `14.4.4.44` when an outage exceeds the battery threshold:
```bash
#!/bin/bash
# /usr/local/bin/homelab_power_shutdown.sh
# Staged graceful homelab shutdown orchestrator
set -uo pipefail
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [POWER-SHUTDOWN] $*" | tee -a /var/log/power-shutdown.log; }
SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes"
log "=== INITIATING HOMELAB TIERED SHUTDOWN ==="
# ------------------------------------------------------------------
# TIER 1: Non-Essential Compute & AI Pis
# ------------------------------------------------------------------
log "Stage 1: Halting AI Raspberry Pi stack (140.44.4.7, 140.44.4.8)..."
ssh $SSH_OPTS root@140.44.4.7 "shutdown -h now" 2>/dev/null || true
ssh $SSH_OPTS root@140.44.4.8 "shutdown -h now" 2>/dev/null || true
# ------------------------------------------------------------------
# TIER 2: Docker Swarm NFS Workloads (NPM & Peanut)
# ------------------------------------------------------------------
log "Stage 2: Stopping Swarm services mounting Synology NFS (NPM, Peanut)..."
ssh $SSH_OPTS root@140.44.4.71 "docker service scale nginx-proxy-manager_app=0 nginx-proxy-manager_backup=0 peanut_peanut=0" 2>/dev/null || true
sleep 5
# ------------------------------------------------------------------
# TIER 3: Harvester Guest VMs & Kubernetes Worker Nodes
# ------------------------------------------------------------------
log "Stage 3: Shutting down Harvester guest VMs and K3s workers..."
# Gracefully signal Harvester VMs
ssh $SSH_OPTS root@140.44.4.81 "virsh list --name | xargs -r -n1 virsh shutdown" 2>/dev/null || true
# Stop K3s agent on Aerodrome workers (clean Longhorn PVC detach)
for ip in 140.44.4.22 140.44.4.23 140.44.4.24 140.44.4.25; do
ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
done
# Stop K3s agent on Spaceport workers
for ip in 140.44.4.51 140.44.4.52 140.44.4.53 140.44.4.54 140.44.4.55 140.44.4.56; do
ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
done
sleep 15
# ------------------------------------------------------------------
# TIER 4: Storage Arrays (Synology DSM Safe Mode)
# ------------------------------------------------------------------
log "Stage 4: Triggering Synology NAS Safe Mode (14.10.10.71, 14.10.10.72)..."
# Synology enters safe mode automatically via NUT upsd client signal (port 3493)
# Explicit API / SSH command to flush pools if configured
ssh $SSH_OPTS admin@14.10.10.71 "synopoweroff" 2>/dev/null || true
ssh $SSH_OPTS admin@14.10.10.72 "synopoweroff" 2>/dev/null || true
sleep 20
# ------------------------------------------------------------------
# TIER 5: Control Planes & Swarm Managers
# ------------------------------------------------------------------
log "Stage 5: Halting K3s control planes and Swarm managers..."
# Spaceport Masters (140.44.4.41-44)
for ip in 140.44.4.41 140.44.4.42 140.44.4.43 140.44.4.44; do
ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
done
# Aerodrome Masters (140.44.4.11-13)
for ip in 140.44.4.11 140.44.4.12 140.44.4.13; do
ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
done
# Swarm Managers (140.44.4.71, 72, 74)
for ip in 140.44.4.71 140.44.4.72 140.44.4.74; do
ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
done
# Harvester Baremetal Hypervisors (140.44.4.81-83)
for ip in 140.44.4.81 140.44.4.82 140.44.4.83; do
ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
done
# ------------------------------------------------------------------
# TIER 6: Master Node Self-Cutoff (FSD)
# ------------------------------------------------------------------
log "Stage 6: Executing final local halt and arming killpower..."
touch /etc/killpower
/sbin/shutdown -h +0
```
---
### D. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`)
1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**.
2. Check **Enable UPS Support**.
3. Select UPS Type: **Synology UPS Server / Network NUT Server**.
4. Network UPS server IP address: **`14.4.4.44`**.
5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `5 minutes` (or `Upon low battery`).
5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `6 minutes` (guaranteeing that Tier 2 Swarm NFS workloads have cleanly scaled down before disk pools unmount).
6. Check **Shut down UPS when the system enters Standby Mode** (disabled, managed centrally by `14.4.4.44`).
---
### D. Peanut Web Dashboard (`peanut.snarfnet.net`)
### E. Peanut Web Dashboard Configuration
Peanut runs on Docker Swarm (`140.44.4.70:8050`). Ensure the Peanut service configuration references the NUT server at `14.4.4.44`:
Peanut runs on Docker Swarm (`140.44.4.70:8050`). It connects directly to the NUT master server over internal TCP 3493:
#### Updated `docker-deployments/peanut.yml`
```yaml
services:
peanut:
@@ -226,38 +392,25 @@ volumes:
---
### E. Home Assistant (`14.4.4.20`) Integration & Automation
## 5. Automated Power Restoration & Boot Sequence
1. Navigate to **Settings** $\rightarrow$ **Devices & Services** $\rightarrow$ **Add Integration** $\rightarrow$ **Network UPS Tools (NUT)**.
2. Host: **`14.4.4.44`**, Port: **`3493`**.
3. Username: **`upsmon_slave`**, Password: **`SLAVE_MONITOR_PASS`**.
When utility grid power returns, the homelab automatically recovers in a **strict bottom-up reverse sequence** to guarantee all storage and network prerequisites are ready before compute services attempt to start.
#### Automation Snippet (`automations.yaml`)
```yaml
alias: "Power Resilience: Grid Power Lost Alert"
trigger:
- platform: state
entity_id: sensor.snarf_ups_status
to: "On Battery"
action:
- service: notify.gotify
data:
title: "⚡ Power Outage Alert"
message: "Grid power failure detected! Homelab is running on UPS battery (14.4.4.44). Tiered shutdown in progress."
priority: 8
```mermaid
flowchart TD
G_On["1. Grid AC Restored\nUPS Battery Reaches >= 30%"] --> P1["2. Phase 1: Core Network & Edge\nUDM Pro (14.44.4.1) & Core PoE Switches\nEstablish VLANs, DHCP, Inter-VLAN Routing"]
P1 --> P2["3. Phase 2: NUT Master\nNode at 14.4.4.44 Boots (BIOS: Always On)\nupsd daemon starts, monitors line quality"]
P2 --> P3["4. Phase 3: Storage Foundation\nSynology SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) Boot\nMount Btrfs pools, start NFS & iSCSI daemons"]
P3 --> P4["5. Phase 4: Control Planes & Hypervisors\n- Harvester spacedock (140.44.4.81-83) boots via IPMI/WOL\n- Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) K3s masters boot\n- etcd clusters re-establish quorum"]
P4 --> P5["6. Phase 5: Storage CSI & Compute Workers\n- Aerodrome & Spaceport worker nodes boot\n- Longhorn CSI engine initializes, verifies volume health, mounts PVCs"]
P5 --> P6["7. Phase 6: Ingress & Workloads\n- Docker Swarm (140.44.4.71-75) boots\n- NPM host-mode container binds 80/443/81 on 140.44.4.74\n- Authentik SSO & Forward Auth come online\n- Applications (Peanut, Gotify, Homebox, etc.) become reachable"]
```
---
### Critical Health Verification Gates:
## 5. Power Restoration & Auto-Boot Sequence
When grid power returns, systems auto-recover in reverse order:
1. **UPS AC Mains Restored**: Physical UPS re-enables output outlets once battery reaches safe threshold ($\ge 20\%$).
2. **Phase 1 - Gateway & Edge**: UDM Pro (`14.44.4.1`) & Core switches boot up and establish internal VLAN routes.
3. **Phase 2 - NUT Master**: Host at `14.4.4.44` boots (BIOS `Restore AC Power` set to `Always On`). `upsd` service starts.
4. **Phase 3 - Storage NAS**: Synology NAS nodes (`14.10.10.71`, `14.10.10.72`) auto-boot upon power recovery.
5. **Phase 4 - Control Planes & Hypervisors**:
- Harvester `spacedock` hosts (`140.44.4.81-83`) boot via Wake-on-LAN / IPMI / BIOS settings.
- K3s master nodes (`aerodrome` & `spaceport`) boot, resume etcd quorum.
6. **Phase 5 - Worker Compute & Workloads**: Worker nodes boot; Kubernetes and Docker Swarm automatically reschedule workloads.
1. **Storage Gate (Phase 3 $\rightarrow$ Phase 4):**
- Synology NAS NFS daemons must be responding on `14.10.10.71:2049` before Docker Swarm starts, ensuring NPM does not encounter stale NFS mounts or SQLite locking.
2. **Database Gate (Phase 5 $\rightarrow$ Phase 6):**
- Longhorn volume `data-authentik-postgresql-0` on Aerodrome must reach `Healthy` state before Authentik server pods start.
3. **Ingress Gate (Phase 6):**
- Nginx Proxy Manager on `snshpd4lnxswm04` (`140.44.4.74`) confirms host-mode ports `80` and `443` are bound and forwards traffic to internal VIPs.