Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(power): complete Phase 7 Power Resilience & UPS Automation deployment and verification

Browse Source
This commit is contained in:
Snarf
2026-11-02 21:58:49 -05:00
parent 7ca3d5acb8
commit d3e2e1f291
8 changed files with 216 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
docs/infrastructure_improvement_plan.md
View File

@@ -14,7 +14,7 @@ Following Site Reliability Engineering (SRE) best practices, the nine infrastruc
4. **Phase 4 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.* **[100% Completed & Verified]**
5. **Phase 5 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates. **[100% Completed & Verified]**
6. **Phase 6 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD App-of-Apps (`spaceport` & `aerodrome`) for automated drift remediation. **[100% Completed & Verified]**
7. **Phase 7 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment. **[In Progress / Next]**
7. **Phase 7 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment. **[100% Completed & Verified]**
```mermaid
flowchart LR
@@ -374,7 +374,23 @@ flowchart TD
---
### Phase 7: Power Resilience & UPS Automation Plan (Updated Core Pillar)
### Phase 7: Power Resilience & UPS Automation Plan (Updated Core Pillar) — [100% Completed & Verified]
#### Architecture & Implementation Steps (Executed & Verified)
1. **NUT Master Server Configuration (`14.4.4.44`):**
- Configured `upsmon.conf` on `nutserver` with `NOTIFYCMD /usr/local/bin/nut-notify.sh` and `SHUTDOWNCMD /usr/local/bin/homelab_power_shutdown.sh`.
- Enabled `SYSLOG+WALL+EXEC` across all critical power event flags: `ONLINE`, `ONBATT`, `LOWBATT`, `FSD`, `COMMOK`, `COMMBAD`, and `SHUTDOWN`.
- Verified `nut-monitor.service` is active, connected to `server-closet@localhost (primary)`.
2. **Notification Pipeline Verification (`scripts/nut_notify.sh`):**
- Verified automated dispatch to Gotify VIP (`http://140.44.4.70:8090/message`) with token `A4uN4fEmErGeNcYtOkEn` (Application 1: `UPS & Power Resilience`, priority 9/10).
- Verified automatic streaming to Discord channel `#ups-power` via WebSocket bridge, with direct UDM-Pro egress fallback if Gotify/Swarm is stopped.
3. **Graceful 6-Tier Shutdown Automation (`scripts/homelab_power_shutdown.sh`):**
- Tier 1: Stops transient AI inference Pis (`140.44.4.7`, `140.44.4.8`).
- Tier 2: Scales down Swarm NFS services (`nginx-proxy-manager_app`, `backup`, `peanut`, `homebox`) on `140.44.4.71` to cleanly flush databases and release NFS file locks.
- Tier 3: Stops K3s agents on all 7 Aerodrome workers (`140.44.4.21-27`) and 6 Spaceport workers (`140.44.4.51-56`), cleanly syncing and detaching Longhorn CSI persistent volumes.
- Tier 4: Signals Synology NAS arrays (`14.10.10.71`, `14.10.10.72`) into DSM Safe Mode via native NUT slave daemon (`synopoweroff`).
- Tier 5: Saves on-demand pre-shutdown etcd snapshots on Spaceport masters (`.41-.44`) and Aerodrome masters (`.11-.13`), stops K3s service, and halts master nodes and Swarm managers (`.71-.75`).
- Tier 6: NUT Master fires final FSD notification, arms `/etc/killpower`, and executes `/sbin/shutdown -h +0`.
#### 6.1 Core Power Architecture & IP Assignments