Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rename power_resilience_plan.md to infrastructure_improvement_plan.md and add 6-phase master homelab improvement roadmap

Browse Source
This commit is contained in:
Snarf
2026-10-30 22:10:36 -04:00
parent c492afc377
commit 1d58791dbf
2 changed files with 341 additions and 416 deletions
Download Patch File Download Diff File Expand all files Collapse all files

341
docs/infrastructure_improvement_plan.md Normal file
View File

@@ -0,0 +1,341 @@
# Snarfnet Master Infrastructure Improvement & Resilience Plan
This master architectural plan defines the prioritized roadmap, implementation blueprints, failure-mode analysis, and power resilience strategy for the **snarfnet** homelab. All solutions are designed to execute **locally on self-hosted infrastructure** (`aerodrome`, `spaceport`, `spacedock`, `shipyard`, and Synology NAS) with zero external SaaS dependencies outside of outbound Discord webhook delivery.
---
## 1. Executive Summary & Engineering Prioritization
Following Site Reliability Engineering (SRE) best practices, the eight infrastructure initiatives and the power resilience architecture are ordered into a **6-Phase Dependency Graph**:
1. **Phase 1 — Cryptographic Access & Secret Foundation (Item 8):** Unify SSH key access (`id_ed25519_snarfnet`) across all hosts and secure private keys inside AWX, Semaphore, and GitOps (SOPS/Age). *Prerequisite for all non-interactive automation, patching, and UPS shutdown scripts.*
2. **Phase 2 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3):** Resolve dual-default StorageClasses, clear stuck `Terminating` PVCs, add Longhorn recurring snapshots on `spaceport`, configure Synology NAS (`14.10.10.71`) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. *Prerequisite before upgrading kernels or control planes.*
3. **Phase 3 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.*
4. **Phase 4 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates.
5. **Phase 5 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using `iac-reverse` Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD ApplicationSets (`spaceport` & `aerodrome`) for automated drift remediation.
6. **Phase 6 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment.
```mermaid
flowchart LR
P1["Phase 1: Unified SSH Key\n& Secret Vault (Item 8)"] --> P2["Phase 2: Storage & NFS\nStabilization (Item 3)"]
P1 --> P3["Phase 3: Gotify, Discord &\nZabbix Sensors (Items 1, 2, 7)"]
P2 --> P4["Phase 4: AWX & Rolling\nInfra Upgrades (Items 5, 6)"]
P3 --> P4
P4 --> P5["Phase 5: Terraform &\nArgoCD GitOps (Item 4)"]
P1 --> P6["Phase 6: Power Resilience\n& UPS Shutdown"]
P2 --> P6
P3 --> P6
```
---
## 2. Live Infrastructure Baseline Audit (October 30, 2026)
Our automated live probe across the homelab APIs identified the following baseline state that directly informs this plan:
| Platform / Service | Current Version / State | Key Audit Findings & Action Required |
|---|---|---|
| **Aerodrome K3s (`local`)** | Masters (`ctl01-03`): `v1.26.10+k3s1`<br>Workers (`wrk01-07`): `v1.30.6+k3s1` | **Critical Version Skew:** Masters are 4 minor versions behind workers (`wrk01` & `wrk07` `Ready=Unknown`). Dual default StorageClasses (`local-path` & `longhorn`). 2 PVCs stuck `Terminating` (`pihole`, `technitium`). Longhorn `backup-target` is empty (`''`). |
| **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: `v1.31.4+k3s1` (All 10 `Ready`) | Dual default StorageClasses (`local-path` & `longhorn`). 1 PVC stuck `Terminating` (`technitium-longhorn-pvc`). **Zero Longhorn recurring snapshot jobs (`[]`)** and empty `backup-target` (`''`) despite hosting Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. |
| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. 2 active PVCs in `snarfai` (`longhorn-pvc-20tb`, `open-webui-pvc`). Not yet monitored in Zabbix. |
| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); `.73` & `.75` offline | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. Legacy stacks (`gotify`, `homebox`, `peanut`, `pihole`) use `/mnt/swarm_shared` (GlusterFS across `.71-.73`), which is degraded when `.73` is offline. |
| **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 22 hosts registered. **Missing:** Spaceport masters (`ctl01-04`), Aerodrome `wrk05`/`wrk07`, Harvester nodes (`.81-.83`), Synology NAS (`.71`/`.72`), AI Pis (`.7`/`.8`), UDM Pro (`.1`), Home Assistant (`.20`). **Enabled MediaTypes: `[]` (None)**; **Problem Action: Disabled (`status=1`)**. |
| **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | Healthy behind Authentik Forward Auth externally; direct internal VIP (`http://140.44.4.70:8090`) reachable for low-level webhooks. |
| **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Running in `awx` namespace (`awx-web`, `awx-task`, `awx-postgres-15-0`). Ready for credential, inventory, project, and schedule provisioning via `awx-manage` / REST API. |
| **ArgoCD (`argo.snarfnet.net`)** | Version `v3.4.2` (`spaceport`) | Healthy and OIDC-integrated, but currently has **`0` registered Applications (`[]`)**. |
---
## 3. Detailed Phase Execution Plans & Roadblock Analysis
### Phase 1: Single SSH Key & Cryptographic Secret Management (Item 8)
#### Architecture & Implementation Steps
1. **Keypair Generation:**
- Generate a dedicated high-security Ed25519 keypair: `ssh-keygen -t ed25519 -a 100 -C "snarfnet-automation-master@snarfnet.net" -f id_ed25519_snarfnet`.
2. **Standardized Automation Account (`snarf-admin` / `root`):**
- Bootstrap all 30+ hosts across VLANs 1, 2, 20, 40, and 50 with a single enrollment script that:
- Installs the public key into `/root/.ssh/authorized_keys` and `/home/snarf/.ssh/authorized_keys` (plus `rancher` on Harvester nodes and `admin` on Synology DSM).
- Configures `/etc/sudoers.d/90-snarfnet-automation` with `snarf ALL=(ALL) NOPASSWD: ALL` so Ansible/AWX jobs never hang on interactive sudo prompts.
- Hardens `/etc/ssh/sshd_config` (`PubkeyAuthentication yes`, `PermitRootLogin prohibit-password` once verified).
3. **Secure Key Storage (Zero Plaintext in Git):**
- **AWX Credential Vault:** Inject `id_ed25519_snarfnet` as an AWX `Machine` credential (encrypted at rest in `awx-postgres-15` using AWX's `SECRET_KEY`).
- **Semaphore Key Store:** Register the key in Semaphore (`140.44.4.58:100`) as a secondary runner fallback.
- **NUT Master (`14.4.4.44`):** Store key at `/root/.ssh/id_ed25519_snarfnet` (`chmod 600`) for `homelab_power_shutdown.sh` to execute passwordless tiered shutdowns.
- **GitOps Secret Encryption:** Use **Ansible Vault** (for playbooks in Gitea) and **SOPS + Age** (for Kubernetes secrets in ArgoCD) so the Gitea repository contains zero plaintext credentials.
#### Predicted Roadblocks & Mitigations
- **Roadblock 1.1 — Harvester HCI Immutable Root (`/home/rancher`):** Harvester v1.8.0 uses an elemental immutable filesystem; writing to `/etc/ssh` directly can be wiped on upgrade.
- *Mitigation:* Persist the SSH public key via Harvester's `/oem/99_custom.yaml` cloud-init configuration and the `Harvester Setting -> ssh-keys` CRD in Kubernetes (`spacedock`).
- **Roadblock 1.2 — Synology DSM Home Directory Permissions:** Synology DSM (`14.10.10.71` & `.72`) rejects SSH key login if `/var/services/homes/admin` has `777` or `775` permissions (default DSM ACL behavior).
- *Mitigation:* Run `chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys && chmod 755 ~` on both NAS appliances during enrollment.
- **Roadblock 1.3 — Currently Offline Nodes (`140.44.4.21`, `.25`/`.27`, `.73`, `.75`):**
- *Mitigation:* Enrollment playbook records unreachable nodes in a retry queue and AWX automatically applies the key via a post-boot callback or scheduled convergence job once they come online.
---
### Phase 2: Persistent Storage Stabilization & NAS NFS Architecture (Item 3)
#### Architecture & Implementation Steps
1. **Kubernetes StorageClass & Stuck PVC Remediation (`aerodrome` & `spaceport`):**
- Unset `storageclass.kubernetes.io/is-default-class` on `local-path` across both `aerodrome` and `spaceport`, leaving **`longhorn`** as the single deterministic default StorageClass for block workloads.
- Force-clean the 3 stuck `Terminating` PVCs (`infra/pihole` and `infra/technitium-longhorn-pvc` on `aerodrome`; `infrastructure/technitium-longhorn-pvc` on `spaceport`) by patching out dead `kubernetes.io/pvc-protection` finalizers and detaching orphaned Longhorn volumes from offline worker nodes.
- Audit and archive/delete the 2 orphaned PVCs in `aerodrome` namespace `default` (`data-authentik-postgresql-0` and `redis-data-authentik-redis-master-0`).
2. **Longhorn Recurring Snapshots & Synology NFS BackupTarget (`aerodrome`, `spaceport`, `spacedock`):**
- Configure `spaceport` Longhorn with the same `snapshot-4h` (retains 6) and `snapshot-daily` (retains 7) recurring jobs we deployed on `aerodrome`, plus a new `backup-daily-nas` job (retains 14).
- Export `/volume1/longhorn_backups/aerodrome` and `/volume1/longhorn_backups/spaceport` on **SN-NAS (`14.10.10.71`)** (or replicate to **SnarfNAS `14.10.10.72`**) via NFSv4.
- Set Longhorn's `backup-target` setting on both clusters to `nfs://14.10.10.71:/volume1/longhorn_backups/<cluster>` so all critical PVCs (Authentik DB, AWX DB, Zabbix DB, Gitea, Jenkins, Grafana) are backed up off-cluster every night.
3. **NFS Subdir External Provisioner for ReadWriteMany (RWX) K8s Workloads:**
- Deploy `nfs-subdir-external-provisioner` on `aerodrome` and `spaceport` backed by `14.10.10.71:/volume1/k8s_shared` as StorageClass `synology-nfs`.
- Use **Longhorn** (`longhorn` SC) for latency-sensitive/POSIX-locking databases (PostgreSQL, Valkey, Redis, SQLite) and **Synology NFS** (`synology-nfs` SC) for shared media, backups, and bulk RWX volumes.
4. **Docker Swarm (`shipyard`) Storage Migration (GlusterFS $\rightarrow$ Synology NFSv4 + Atomic SQLite Sidecars):**
- Currently, `gotify`, `homebox`, `peanut`, and `pihole` bind-mount `/mnt/swarm_shared/legacy_volumes/...` (a 3-node GlusterFS volume across `.71`, `.72`, `.73`). With `.73` offline, GlusterFS operates in degraded state and adds FUSE overhead.
- Migrate Swarm persistent volumes to structured NFSv4 shares on `14.10.10.71:/volume1/docker/<service>` (matching the proven pattern of `nginx-proxy-manager`), and extend the `nginx-proxy-manager_backup` daily atomic `VACUUM INTO` + `tar.gz` backup daemon to also snapshot Gotify (`gotify.db`) and Homebox (`homebox.db`).
#### Predicted Roadblocks & Mitigations
- **Roadblock 2.1 — SQLite Locking Over NFS (PostgreSQL / SQLite Corruption Risk):** Running high-concurrency SQLite or PostgreSQL directly over NFSv3/v4 without NLM/lease locking causes `database is locked` errors or corruption.
- *Mitigation:* Never move K8s PostgreSQL PVCs (`authentik-postgresql`, `awx-postgres`, `zabbix-postgresql`, `gitea-postgresql`) onto raw NFS; keep them on **Longhorn block storage (iSCSI-backed ext4)** and use NFS *strictly* as the Longhorn `backup-target` and for RWX file shares. For Swarm SQLite apps (`npm`, `gotify`, `homebox`), enforce `replicas: 1`, `update_config.order: stop-first`, and NFSv4 (`nfsvers=4.1,hard,timeo=600,retrans=2`).
- **Roadblock 2.2 — Synology NFS Squash Permissions (`root_squash`):** By default, Synology DSM maps container `root` (UID 0) to `nobody` (UID 65534), breaking container startup `chown`.
- *Mitigation:* Configure Synology NFS rules for `140.44.4.0/25` with `Squash: Map all users to admin` (or `No mapping` for Longhorn backup target) and enable `Allow connections from non-privileged ports (>1024)`.
---
### Phase 3: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7)
#### Architecture & Implementation Steps
```mermaid
flowchart TD
subgraph Sources ["Homelab Event & Alert Sources"]
ZBX["Zabbix 7.0.16 Server\n(140.44.4.61 / spaceport)"]
HA["Home Assistant\n(14.4.4.20 / VLAN 50)"]
NUT["NUT UPS Master\n(14.4.4.44 / nut-notify.sh)"]
AWX_N["AWX / ArgoCD / Backup Jobs"]
end
subgraph GotifyHub ["Central Local Notification Hub"]
GTF["Gotify Server v2.9.1\nInternal VIP: http://140.44.4.70:8090\nExternal: https://gotify.snarfnet.net"]
BRIDGE["gotify-discord-bridge\n(WebSocket Listener Container)"]
end
subgraphRecipients ["Important Devices & External Channels"]
Mobile["Mobile Devices (Android/iOS)\n& Desktop Browser Push"]
Homepage["Homepage Dashboard Widget"]
Discord["Discord Server Channels\n(#infra-alerts, #smart-home, #ups-power)"]
end
ZBX -- "1. Native Gotify MediaType" --> GTF
ZBX -- "2. Native Discord Webhook (Direct Fallback)" --> Discord
HA -- "1. REST notify.gotify" --> GTF
HA -- "2. Native Discord Integration" --> Discord
NUT -- "Direct HTTP (Bypasses SSO)" --> GTF
AWX_N -- "App Token Webhook" --> GTF
GTF -- "WebSocket Stream (Client Tokens)" --> Mobile
GTF -- "Client Token API" --> Homepage
GTF -- "ws://140.44.4.70:8090/stream" --> BRIDGE
BRIDGE -- "Formatted Discord Embeds" --> Discord
```
1. **Gotify Broadcast Channels & Device Enrollment (Item 1):**
- Automated Python script (`scripts/setup_gotify_channels.py`) provisions 5 structured Gotify Applications via `http://140.44.4.70:8090/application`:
- `UPS & Power Resilience` (Default Priority: `9`)
- `Zabbix Infrastructure Alerts` (Priority mapped `1-10` by severity)
- `Home Assistant Smart Home` (Default Priority: `5`)
- `AWX & GitOps Automation` (Default Priority: `4`)
- `Backups & Storage Health` (Default Priority: `6`)
- Provision dedicated **Client Tokens** (`http://140.44.4.70:8090/client`) for each important device (Primary Phone, Tablet, Desktop Browser, Homepage Dashboard).
2. **Discord Forwarding Architecture (Item 2):**
- **Gotify $\rightarrow$ Discord Bridge:** Deploy a lightweight, self-healing container service (`gotify-discord-bridge`) on Shipyard Swarm alongside Gotify. It connects internally to `ws://gotify:80/stream?token=<CLIENT_TOKEN>`, transforms every incoming Gotify message into a rich, color-coded Discord Embed (Red for priority $\ge 8$, Orange for $5-7$, Blue for $\le 4$), and POSTs it to your Discord Webhook URL.
- **Zabbix $\rightarrow$ Discord & Gotify (Dual-Path Redundancy):** Configure Zabbix 7.0.16 via JSON-RPC API to enable both the native **Discord** webhook media type and a **Gotify** webhook media type, linked to the `Admin` user, and **enable** Trigger Action `'Report problems to Zabbix administrators'` (`status=0`, currently disabled!). If Gotify or Swarm is ever down during an outage, Zabbix still reaches Discord directly from `spaceport`.
- **Home Assistant $\rightarrow$ Discord & Gotify:** Add a unified `notify.snarfnet_broadcast` group in Home Assistant (`14.4.4.20`) combining `notify.gotify` and the native Discord webhook integration.
3. **Complete Zabbix Sensors, Thresholds, & Templates (Item 7):**
- **Fix Missing Hosts (100% Coverage):**
- Deploy `zabbix-agent2` with shared PSK (`PSK_autoregister`) via Ansible across all missing Linux nodes: Spaceport Masters (`140.44.4.41-44`), Aerodrome Workers (`140.44.4.25`, `.27`), AI Pis (`140.44.4.7`, `.8`), and Harvester nodes (`140.44.4.81-83`).
- Enable SNMPv2c/v3 on **SN-NAS (`14.10.10.71`)**, **SnarfNAS (`14.10.10.72`)**, and **UDM Pro (`14.44.4.1`)**, registering them in Zabbix with templates `Synology DSM by SNMP` and `Ubiquiti AirOS/UniFi by SNMP`.
- **Custom Sensors & Threshold Alarms:**
- **NUT UPS Template (`nutserver` `14.4.4.44`):** Custom `UserParameter=ups.metric[*],upsc snarf-ups@127.0.0.1 $1` polling `ups.status` (Trigger: `!= OL` $\rightarrow$ High), `battery.charge` (`< 80%` Warning, `< 50%` High, `< 25%` Disaster), `ups.load` (`> 85%` High), and `battery.runtime` (`< 600s` High).
- **Kubernetes & Longhorn Health:** Link `Kubernetes nodes by HTTP` and `Kubernetes cluster state by HTTP` to `aerodrome` and `spaceport`; alert on Pod `CrashLoopBackOff`, Node `NotReady` $> 3\text{m}$, and Longhorn volume robustness `Degraded`/`Faulted`.
- **Hardware & Thermal Thresholds:** CPU temperature $> 75^\circ\text{C}$ (Warning) / $> 82^\circ\text{C}$ (High) for Raspberry Pi 5 nodes in `spaceport`; NVMe/HDD SMART health and Btrfs pool utilization $> 80\%$ (Warning) / $> 90\%$ (High).
- **TLS Certificate & Web Scenarios:** Zabbix Web Scenarios checking HTTP 200 and SSL certificate expiry ($< 14$ days Warning, $< 7$ days High) across all 35 `*.snarfnet.net` proxy hosts.
#### Predicted Roadblocks & Mitigations
- **Roadblock 3.1 — Authentik Forward Auth Blocking Gotify Mobile App:** The Android/iOS Gotify app uses its own `X-Gotify-Key` header and cannot complete an interactive Authentik OAuth2 browser redirect flow when connecting externally to `https://gotify.snarfnet.net`.
- *Mitigation:* In NPM Proxy Host #32 (`gotify.snarfnet.net`), bypass Authentik `auth_request` specifically when a valid `X-Gotify-Key` header or `/message` / `/stream` API path is requested, while enforcing Authentik Forward Auth on the web UI root (`/`, `/static/*`)!
- **Roadblock 3.2 — Discord Webhook Rate Limiting During Alert Storms:** During a power outage or switch flap, 25+ hosts firing simultaneously can hit Discord's `30 requests / 60 seconds` webhook rate limit (`HTTP 429`).
- *Mitigation:* Implement a 2-second batching queue with exponential backoff on `HTTP 429` (`Retry-After`) in the `gotify-discord-bridge` and configure Zabbix host dependency rules (all VLAN 20 hosts depend on UDM Pro `14.44.4.1` ICMP ping) so downstream alerts are suppressed if a core switch or gateway drops.
---
### Phase 4: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6)
#### Architecture & Implementation Steps
1. **AWX Provisioning (`https://awx.snarfnet.net` on `aerodrome`):**
- Execute an automated bootstrap script via `kubectl exec` into `awx-task` (or AWX REST API `/api/v2/`) to create:
- **Organization:** `Snarfnet Homelab`
- **Credentials:**
1. `Snarfnet Master SSH Key` (`Machine` credential with `id_ed25519_snarfnet` + passwordless sudo)
2. `Gitea Automation Token` (`Source Control` credential for pulling playbooks from `http://140.44.4.62:90/snarf/SnarfCode.git`)
3. `Authentik & Zabbix API Vault` (`Vault` credential)
- **Project:** `Snarfnet Infrastructure Playbooks` synced from local **Gitea** (`git.snarfnet.net`).
- **Inventory (`Snarfnet Master Inventory`):** Structured into groups:
- `k3s_aerodrome_masters` (`140.44.4.11-13`) & `k3s_aerodrome_workers` (`140.44.4.21-27`)
- `k3s_spaceport_masters` (`140.44.4.41-44`) & `k3s_spaceport_workers` (`140.44.4.51-56`)
- `harvester_spacedock` (`140.44.4.81-83`)
- `swarm_shipyard` (`140.44.4.71-75`)
- `ai_pi_cluster` (`140.44.4.7-8`)
- `smart_home_nodes` (`14.4.4.20`, `14.4.4.44`, `14.44.4.14`)
- `synology_nas` (`14.10.10.71-72`)
2. **Remediation of `aerodrome` K3s Control Plane Version Skew (Priority Upgrade):**
- `aerodrome` masters (`140.44.4.11-13`) are at `v1.26.10+k3s1` while workers are at `v1.30.6+k3s1`. Kubernetes strictly forbids skipping minor versions on control-plane upgrades.
- Create `ansible/playbooks/upgrade_k3s_aerodrome.yml` with `serial: 1`:
- Step 1: Take an `etcd-snapshot save pre-upgrade-1.26` on `.11`.
- Step 2: Sequentially step all 3 masters through each minor release channel: `v1.27.16+k3s1` $\rightarrow$ `v1.28.15+k3s1` $\rightarrow$ `v1.29.10+k3s1` $\rightarrow$ `v1.30.6+k3s1` (and optionally align with `spaceport` at `v1.31.4+k3s1`), waiting for `kubectl get nodes` `Ready` and etcd quorum health (`etcdctl endpoint health`) between every single node and minor step.
3. **AWX Job Templates & Scheduled Maintenance Windows:**
- **Template 1 — `Weekly OS & Security Patching` (Cron: Sundays 03:00 AM):**
- Runs `apt-get update && apt-get dist-upgrade -y` with `serial: 1`.
- Checks `/var/run/reboot-required`. If a kernel reboot is needed on a K8s or Swarm node, it runs `kubectl drain <node> --ignore-daemonsets --delete-emptydir-data` (or `docker node update --availability drain`), reboots, waits for SSH + kubelet `Ready` + Longhorn volume health (`Healthy`), and runs `kubectl uncordon <node>`.
- **Template 2 — `Monthly K3s & Container Stack Update` (Cron: 1st Sunday 04:00 AM):**
- Updates K3s binaries and pulls latest pinned container digests for Shipyard Swarm stacks, followed by HTTP 200 health verification across all 35 NPM endpoints.
- **Template 3 — `Daily Drift & Agent Health Audit` (Cron: Daily 06:00 AM):**
- Verifies `zabbix-agent2`, `upsmon` slave daemons, NFS mounts, and SSH `authorized_keys` across all nodes, sending a summary report to Gotify/Discord.
#### Predicted Roadblocks & Mitigations
- **Roadblock 4.1 — Longhorn Replica Rebuilding During Rolling Node Reboots:** If Ansible reboots node B immediately after node A finishes rebooting, Longhorn volumes may still be in `Degraded` (rebuilding replicas) on node A; rebooting node B can take the last healthy replica offline!
- *Mitigation:* Add an explicit Ansible `uri` / `command` pre-task gate in the rolling reboot playbook that queries the Longhorn API (`volumes` list) and pauses the next node's drain until **zero attached volumes are in `degraded` state**.
- **Roadblock 4.2 — AWX Running Inside the Cluster It Is Upgrading (`aerodrome`):** When AWX upgrades or reboots the `aerodrome` node currently hosting the `awx-task` pod, the active AWX job itself will be killed mid-execution!
- *Mitigation:* Pin `awx-task` and `awx-web` to `sninfr2lnxwrk06` during `aerodrome` master upgrades, and execute `aerodrome` worker reboots either via a Workflow Job Template that upgrades `wrk06` last, or trigger `aerodrome` node reboots from **Semaphore** (`140.44.4.58` on `spaceport`) so the orchestrator lives on a separate physical cluster!
---
### Phase 5: Terraform Plans & ArgoCD GitOps Integrity (Item 4)
#### Architecture & Implementation Steps
1. **Local GitOps Repository Structure on Gitea (`https://git.snarfnet.net/snarf/homelab-gitops`):**
```text
homelab-gitops/
├── terraform/
│ ├── authentik/ # Providers, Applications, Outposts, Conditional MFA Policies
│ ├── harvester/ # Spacedock VM definitions, VLAN networks, SSH keys
│ ├── shipyard-swarm/ # Docker Swarm stacks (NPM, Gotify, Homebox, Peanut, Pihole)
│ └── synology/ # Shared folders, NFS rules
└── argocd/
├── bootstrap/ # Root App-of-Apps for spaceport & aerodrome
├── spaceport/ # Gitea, Zabbix, Jenkins, Semaphore, Coder, Homepage, Traefik
└── aerodrome/ # Authentik, AWX, Grafana, Uptime Kuma, Longhorn settings
```
2. **Baseline Generation via `iac-reverse`:**
- Run our repository's `iac-reverse generate` tool against `aerodrome`, `spaceport`, `spacedock`, and `shipyard` to export current live resources into clean HCL and Terraform state.
- Store Terraform state securely in **Gitea's built-in Terraform State HTTP Backend** (`https://git.snarfnet.net/api/packages/snarf/terraform/state/...`) or on Synology NAS (`14.10.10.71`), backed up daily.
3. **ArgoCD Multi-Cluster Registration (`https://argo.snarfnet.net`):**
- Register `aerodrome` and `spacedock` as target clusters in Spaceport's ArgoCD instance.
- Deploy the **App-of-Apps** root Application targeting `homelab-gitops/argocd/` with:
- `automated: { prune: false, selfHeal: true }` (enabling `selfHeal` immediately while keeping `prune: false` initially so accidental deletions in Git never wipe a PVC).
- ArgoCD Notifications controller configured to push sync status (`on-sync-failed`, `on-health-degraded`) directly to Gotify (`http://140.44.4.70:8090`).
#### Predicted Roadblocks & Mitigations
- **Roadblock 5.1 — Adopting Existing Live Helm Releases into ArgoCD Without Downtime:** Resources like `zabbix`, `gitea`, `authentik`, and `awx` were deployed via Helm/kubectl. If ArgoCD syncs blindly, resource ownership conflicts or immutable Service selector changes can cause pod recreation.
- *Mitigation:* Use `ServerSideApply=true` and `Replace=false` sync options in ArgoCD, and test every Application in **Diff / Manual Sync mode** (`kubectl diff`) until ArgoCD reports `Synced / Healthy` with zero destructive diffs before flipping `selfHeal: true`.
- **Roadblock 5.2 — Circular Dependency if Gitea or ArgoCD Goes Down:**
- *Mitigation:* Mirror the `homelab-gitops` repository automatically to a bare Git repository on Synology NAS (`/volume1/git_mirror/homelab-gitops.git`) and your workstation (`SnarfCode`), ensuring Terraform and `kubectl apply` can always run directly from local disk if `spaceport` is offline.
---
### Phase 6: Power Resilience & UPS Automation Plan (Updated Core Pillar)
#### 6.1 Core Power Architecture & IP Assignments
The power resilience architecture uses a centralized **NUT Master Server** (`14.4.4.44`) that polls the physical UPS and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol |
|---|---|---|---|---|
| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master` and holds `/root/.ssh/id_ed25519_snarfnet`. |
| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; stopped in Tier 2 prior to NAS Safe Mode. |
| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth & Conditional MFA. Runs on Longhorn storage in Aerodrome. |
| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. |
| **Emergency Push & Discord** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server + `gotify-discord-bridge`. **Internal outage alerts use `http://140.44.4.70:8090`** (+ direct Discord webhook fallback) to bypass Authentik during shutdown. |
| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and non-essential load shedding. |
| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix 7.0.16 polling `14.4.4.44` via Zabbix Agent 2 `upsc` UserParameters. |
| **Primary Storage 1 & 2** | `sn-nas` / `snarfnas` | `14.10.10.71` / `.72` | VLAN 40 (Storage) | Synology DSM UPS clients polling `14.4.4.44`. Host NFSv4 shares and Longhorn BackupTargets. |
#### 6.2 Tiered Graceful Shutdown Hierarchy (6 Stages)
```text
+--------------------------------------------------------------------------------------------------+
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT |
| - Broadcasts emergency alert to Gotify (http://140.44.4.70:8090), Discord Webhook, & HA Push |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads |
| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped |
| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted via SSH key |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM, Gotify, Homebox, Peanut) |
| - Send final pre-shutdown notification to Gotify & Discord |
| - Scale down Swarm NFS services (`nginx-proxy-manager_app`, `backup`, `peanut`, `homebox`) |
| - Flushes open SQLite databases and cleanly releases all NFS file locks on 14.10.10.71 |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach |
| - Harvester guest VMs (140.44.4.81-83) gracefully shut down via `virsh shutdown` / API |
| - Stop K3s agent on Aerodrome (140.44.4.21-27) & Spaceport (140.44.4.51-56) worker nodes |
| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode) |
| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode (`synopoweroff` / NUT) |
| - Unmounts Btrfs volume pools, terminates NFS/iSCSI daemons, parks drive heads |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors |
| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Masters save etcd snapshot & halt |
| - Remaining Swarm Managers (140.44.4.71-75) & Harvester Hypervisors (140.44.4.81-83) halt |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut |
| - NUT Master (14.4.4.44) sends direct Discord webhook ("Final FSD Cut") & issues `upsdrvctl` |
| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops |
+--------------------------------------------------------------------------------------------------+
```
#### 6.3 Automated Power Restoration & Boot Verification Gates
1. **Stage 1 (Grid AC Restored, Battery $\ge 30\%$):** UPS re-energizes load outlets.
2. **Stage 2 (Network Core):** UDM Pro (`14.44.4.1`) & Core switches boot, establishing VLANs 1–60 and inter-VLAN routing.
3. **Stage 3 (NUT Master):** `14.4.4.44` boots (`BIOS: Restore on AC Power Loss = Always On`), starts `upsd`, and sends `ONLINE` alert to Discord/Gotify.
4. **Stage 4 (Storage Gate — Must Complete Before Compute):** Synology NAS arrays (`14.10.10.71` & `14.10.10.72`) boot, mount Btrfs pools, and expose NFS (`:2049`).
5. **Stage 5 (Control Planes & Hypervisors):** Harvester (`140.44.4.81-83`), K3s Masters (`aerodrome` & `spaceport`), and Swarm Managers (`140.44.4.71-75`) boot and establish etcd/Raft quorum.
6. **Stage 6 (Compute Workers & Ingress):** K3s workers boot, Longhorn reattaches PVCs (`data-authentik-postgresql-0`, `awx-postgres`, `zabbix-postgresql`), and NPM on `snshpd4lnxswm04` (`140.44.4.74`) binds host ports `80/443/81`.
---
## 4. Access & Inputs Required to Begin Automation
We already have verified administrative API access to:
- **Authentik API** (`https://authentik.snarfnet.net`)
- **Rancher & Kubernetes APIs** (`aerodrome`, `spaceport`, and `spacedock` Harvester)
- **Portainer & Docker Swarm API** (`https://shipyard.snarfnet.net`)
- **Zabbix 7.0.16 API** (`https://zabbix.snarfnet.net/api_jsonrpc.php`)
To automate the remaining items end-to-end, we will only need the following when we reach each corresponding task:
1. **Discord Webhook URL(s) (For Phase 3 — Item 2):**
- One Discord Webhook URL (or separate URLs if you prefer separate channels like `#infra-alerts`, `#smart-home`, and `#ups-power`).
2. **Gotify Admin Credentials (For Phase 3 — Item 1):**
- Current Gotify admin username/password (or we can inspect/bootstrap a token directly via the Gotify SQLite database on Shipyard Swarm).
3. **Home Assistant Long-Lived Access Token or SSH (For Phase 3 — Item 2):**
- To inject the Gotify/Discord notification configuration into `14.4.4.20` (`configuration.yaml` / `automations.yaml`).
4. **Host SSH / Synology DSM Credentials (For Phase 1 — Item 8 & Phase 2 — Item 3):**
- Confirmation of whether the existing `snarf` / `Katzir476!` credential (from `setup_storage_cluster.sh`) is valid across the Debian/RPi nodes, plus SSH/DSM credentials for Synology NAS (`14.10.10.71` / `14.10.10.72`) and Harvester nodes (`140.44.4.81-83`) so we can enroll the unified `id_ed25519_snarfnet` key.

416
docs/power_resilience_plan.md
View File

@@ -1,416 +0,0 @@
# Snarfnet Homelab Power Resilience & UPS Automation Plan
This document details the comprehensive power resilience strategy, tiered graceful shutdown sequence, and automated recovery orchestration for the **snarfnet** homelab. The system is centered around a dedicated **NUT (Network UPS Tools) Master Server** at `14.4.4.44`.
---
## 1. Core Architecture & Component Roles
The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol |
|---|---|---|---|---|
| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master`. |
| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; must be stopped prior to NAS Safe Mode. |
| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth for Peanut, Gotify, Homebox, WLED. Runs on Longhorn storage in Aerodrome. |
| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. |
| **Emergency Push Notifications** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server. **Internal outage alerts MUST use direct IP (`http://140.44.4.70:8090`)** to bypass Authentik during shutdown. |
| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and automated shedding. |
| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493` for battery voltage, load %, and runtime metrics. |
| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts NPM NFS shares (`/volume1/docker/nginx/data`). |
| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts secondary backup shares. |
| **Distributed Block Storage** | `longhorn.aerodrome.snarfnet.net` | `140.44.4.36:80` | VLAN 20 (Aerodrome K3s) | Longhorn CSI storage engine powering Authentik, AWX, Grafana PVCs. Protected with recurring snapshots (`snapshot-4h`, `snapshot-daily`). |
> [!IMPORTANT]
> **Emergency Alerting Path Warning:**
> `peanut.snarfnet.net` and `gotify.snarfnet.net` are protected externally behind Authentik Forward Auth (`oauth2-proxy`). During a power outage, when K3s `aerodrome` shuts down in Tier 3, Authentik goes offline. Therefore, **emergency outage alerts and NUT notification scripts must communicate directly with Gotify via internal IP (`http://140.44.4.70:8090/message?token=...`)** or native Home Assistant push notifications, bypassing NPM and Authentik.
---
## 2. Power Resilience Topology Map
```mermaid
flowchart TD
subgraph Grid ["Utility Power Grid"]
Mains["Main AC Utility Power"]
end
subgraph PowerHardware ["Power Hardware & Protection"]
UPS["Central UPS Hardware (Battery Bank)"]
end
subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"]
UPSD["upsd Daemon (Port 3493)"]
UPSMON_M["upsmon (Master Daemon)"]
NOTIFY["nut-notify.sh (Direct Alert Engine)"]
end
subgraph OutageAlerts ["Emergency Notification Channels"]
HA_Alert["Home Assistant (14.4.4.20)\nLocal Push Notification"]
Gotify_Direct["Gotify Internal Direct\nhttp://140.44.4.70:8090\n(Bypasses Authentik / NPM)"]
Syslog["Wall / Syslog Broadcast"]
end
subgraph MonitoringPanels ["Monitoring & Management"]
Peanut["Peanut Dashboard\n(peanut.snarfnet.net)"]
Zabbix["Zabbix Server\n(140.44.4.61)"]
end
subgraph InfrastructureConsumers ["Homelab Tiered Consumers"]
SwarmNFS["Docker Swarm (140.44.4.71-75)\nNPM Host Mode (140.44.4.74)"]
K8sCompute["Aerodrome & Spaceport K3s\nLonghorn Engine & Worker Nodes"]
NAS["Synology NAS Arrays\n(14.10.10.71 / 14.10.10.72)"]
K8sControl["K3s Control Planes & etcd"]
Harvester["Spacedock HCI Hypervisors\n(140.44.4.81 - 83)"]
CoreEdge["UDM Pro Gateway (14.44.4.1)\nCore PoE Switches"]
end
Mains --> UPS
UPS -- USB / SNMP --> UPSD
UPSD --> UPSMON_M
UPSMON_M --> NOTIFY
NOTIFY -- Direct API Token --> Gotify_Direct
NOTIFY -- TCP 3493 Event --> HA_Alert
NOTIFY --> Syslog
UPSD -- TCP 3493 --> Peanut
UPSD -- TCP 3493 --> Zabbix
UPSMON_M -. Tiered Signal .-> SwarmNFS
UPSMON_M -. Tiered Signal .-> K8sCompute
UPSMON_M -. Tiered Signal .-> NAS
UPSMON_M -. Tiered Signal .-> K8sControl
UPSMON_M -. Tiered Signal .-> Harvester
UPSMON_M -. Final Cut .-> CoreEdge
```
---
## 3. Tiered Graceful Shutdown Hierarchy
To eliminate data corruption, prevent stale NFS file handles, and ensure Longhorn volume detach cleanly, the homelab executes a **strict 6-stage progressive shutdown sequence**.
```
+--------------------------------------------------------------------------------------------------+
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT |
| - Broadcasts emergency alert via Home Assistant push & Gotify direct API (http://140.44.4.70:8090)|
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads |
| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped |
| - Media transcoders (Plex active sessions signaled) |
| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM / Peanut) |
| - Stop `nginx-proxy-manager_app` & `nginx-proxy-manager_backup` on snshpd4lnxswm04 (140.44.4.74) |
| - Stop `peanut` container on Swarm |
| - Flushes open SQLite databases (database.sqlite) and unmounts NFS mounts to 14.10.10.71 |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach |
| - Harvester guest VMs (harvester.spacedock.snarfnet.net 140.44.4.90) gracefully shut down via API|
| - Cordon & scale down stateful pods on Aerodrome (Authentik PostgreSQL, AWX Postgres, Grafana) |
| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes |
| - Aerodrome (140.44.4.21-25) & Spaceport (140.44.4.51-56) worker nodes execute shutdown |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode) |
| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode via NUT client trigger |
| - Unmounts all Btrfs volume pools, terminates NFS/iSCSI daemon threads, parks drive read heads |
| - Zero active network clients remain connected to storage |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors |
| - Aerodrome K3s Control Planes (140.44.4.11-13) take etcd snapshot and halt |
| - Spaceport K3s Control Planes (140.44.4.41-44) take etcd snapshot and halt |
| - Swarm Manager nodes (140.44.4.71-75) halt |
| - Harvester baremetal hypervisors (140.44.4.81-83) halt via IPMI/SSH |
+--------------------------------------------------------------------------------------------------+
|
v
+--------------------------------------------------------------------------------------------------+
| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut |
| - NUT Master (14.4.4.44) issues final FSD broadcast to all remaining clients |
| - NUT Master issues `upsdrvctl shutdown` with /etc/killpower flag (commands UPS load drop) |
| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops |
+--------------------------------------------------------------------------------------------------+
```
---
## 4. Configuration Blueprints & Integration Scripts
### A. NUT Master Server (`14.4.4.44`)
#### `/etc/nut/ups.conf`
```ini
[snarf-ups]
driver = usbhid-ups
port = auto
desc = "Snarfnet Main Rack UPS"
pollinterval = 5
```
#### `/etc/nut/upsd.conf`
```ini
LISTEN 14.4.4.44 3493
LISTEN 127.0.0.1 3493
MAXAGE 15
```
#### `/etc/nut/upsd.users`
```ini
[admin]
password = ADMIN_PASSWORD_CHANGE_ME
actions = SET
instcmds = ALL
[upsmon_master]
password = MASTER_MONITOR_PASS
upsmon master
[upsmon_slave]
password = SLAVE_MONITOR_PASS
upsmon slave
[peanut]
password = PEANUT_MONITOR_PASS
upsmon slave
```
#### `/etc/nut/upsmon.conf`
```ini
MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master
MINSUPPLIES 1
SHUTDOWNCMD "/usr/local/bin/homelab_power_shutdown.sh"
NOTIFYCMD /usr/local/bin/nut-notify.sh
POWERDOWNFLAG /etc/killpower
NOTIFYFLAG ONLINE SYSLOG+WALL+EXEC
NOTIFYFLAG ONBATT SYSLOG+WALL+EXEC
NOTIFYFLAG LOWBATT SYSLOG+WALL+EXEC
NOTIFYFLAG FSD SYSLOG+WALL+EXEC
NOTIFYFLAG COMMBAD SYSLOG+WALL+EXEC
NOTIFYFLAG COMMOK SYSLOG+WALL+EXEC
```
---
### B. Outage Notification Engine (`/usr/local/bin/nut-notify.sh`)
This script runs on the NUT Master (`14.4.4.44`) whenever a power state event triggers. It sends push notifications directly to Gotify (`http://140.44.4.70:8090`) and Home Assistant (`http://14.4.4.20:8123`) without touching NPM or Authentik.
```bash
#!/bin/bash
# /usr/local/bin/nut-notify.sh
# Emergency power notification script for Network UPS Tools
set -euo pipefail
EVENT_TYPE="${1:-$NOTIFYTYPE}"
UPS_NAME="${UPSNAME:-snarf-ups}"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
GOTIFY_INTERNAL_URL="http://140.44.4.70:8090/message"
GOTIFY_APP_TOKEN="A4uN4fEmErGeNcYtOkEn" # Replace with valid Gotify App Token
send_gotify() {
local title="$1"
local message="$2"
local priority="$3"
curl -s -m 5 -X POST "${GOTIFY_INTERNAL_URL}?token=${GOTIFY_APP_TOKEN}" \
-H "Content-Type: application/json" \
-d "{\"title\": \"${title}\", \"message\": \"${message}\", \"priority\": ${priority}}" >/dev/null 2>&1 || true
}
case "$EVENT_TYPE" in
ONBATT)
send_gotify "⚡ Grid Power Lost" "[$TIMESTAMP] Homelab is running on UPS battery (${UPS_NAME}). Staged shutdown countdown started." 8
logger -t nut-notify "Grid power failure detected - running on UPS battery."
;;
ONLINE)
send_gotify "✅ Grid Power Restored" "[$TIMESTAMP] Utility AC power restored to ${UPS_NAME}. Battery recharging." 5
logger -t nut-notify "AC utility power restored."
;;
LOWBATT)
send_gotify "🚨 UPS Battery Critical" "[$TIMESTAMP] Battery level critical on ${UPS_NAME}! Final Tier 4-6 shutdown initiated." 10
logger -t nut-notify "CRITICAL: UPS battery low. Enforcing emergency shutdown."
;;
FSD)
send_gotify "🛑 Forced Shutdown Engaged" "[$TIMESTAMP] NUT Master has commanded immediate power cutoff across all clusters." 10
logger -t nut-notify "FSD signal received. Halting systems."
;;
*)
logger -t nut-notify "Unhandled UPS event: ${EVENT_TYPE}"
;;
esac
```
---
### C. Orchestrated Tiered Shutdown Script (`/usr/local/bin/homelab_power_shutdown.sh`)
This script is invoked by `upsmon` on `14.4.4.44` when an outage exceeds the battery threshold:
```bash
#!/bin/bash
# /usr/local/bin/homelab_power_shutdown.sh
# Staged graceful homelab shutdown orchestrator
set -uo pipefail
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [POWER-SHUTDOWN] $*" | tee -a /var/log/power-shutdown.log; }
SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes"
log "=== INITIATING HOMELAB TIERED SHUTDOWN ==="
# ------------------------------------------------------------------
# TIER 1: Non-Essential Compute & AI Pis
# ------------------------------------------------------------------
log "Stage 1: Halting AI Raspberry Pi stack (140.44.4.7, 140.44.4.8)..."
ssh $SSH_OPTS root@140.44.4.7 "shutdown -h now" 2>/dev/null || true
ssh $SSH_OPTS root@140.44.4.8 "shutdown -h now" 2>/dev/null || true
# ------------------------------------------------------------------
# TIER 2: Docker Swarm NFS Workloads (NPM & Peanut)
# ------------------------------------------------------------------
log "Stage 2: Stopping Swarm services mounting Synology NFS (NPM, Peanut)..."
ssh $SSH_OPTS root@140.44.4.71 "docker service scale nginx-proxy-manager_app=0 nginx-proxy-manager_backup=0 peanut_peanut=0" 2>/dev/null || true
sleep 5
# ------------------------------------------------------------------
# TIER 3: Harvester Guest VMs & Kubernetes Worker Nodes
# ------------------------------------------------------------------
log "Stage 3: Shutting down Harvester guest VMs and K3s workers..."
# Gracefully signal Harvester VMs
ssh $SSH_OPTS root@140.44.4.81 "virsh list --name | xargs -r -n1 virsh shutdown" 2>/dev/null || true
# Stop K3s agent on Aerodrome workers (clean Longhorn PVC detach)
for ip in 140.44.4.22 140.44.4.23 140.44.4.24 140.44.4.25; do
ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
done
# Stop K3s agent on Spaceport workers
for ip in 140.44.4.51 140.44.4.52 140.44.4.53 140.44.4.54 140.44.4.55 140.44.4.56; do
ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
done
sleep 15
# ------------------------------------------------------------------
# TIER 4: Storage Arrays (Synology DSM Safe Mode)
# ------------------------------------------------------------------
log "Stage 4: Triggering Synology NAS Safe Mode (14.10.10.71, 14.10.10.72)..."
# Synology enters safe mode automatically via NUT upsd client signal (port 3493)
# Explicit API / SSH command to flush pools if configured
ssh $SSH_OPTS admin@14.10.10.71 "synopoweroff" 2>/dev/null || true
ssh $SSH_OPTS admin@14.10.10.72 "synopoweroff" 2>/dev/null || true
sleep 20
# ------------------------------------------------------------------
# TIER 5: Control Planes & Swarm Managers
# ------------------------------------------------------------------
log "Stage 5: Halting K3s control planes and Swarm managers..."
# Spaceport Masters (140.44.4.41-44)
for ip in 140.44.4.41 140.44.4.42 140.44.4.43 140.44.4.44; do
ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
done
# Aerodrome Masters (140.44.4.11-13)
for ip in 140.44.4.11 140.44.4.12 140.44.4.13; do
ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
done
# Swarm Managers (140.44.4.71, 72, 74)
for ip in 140.44.4.71 140.44.4.72 140.44.4.74; do
ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
done
# Harvester Baremetal Hypervisors (140.44.4.81-83)
for ip in 140.44.4.81 140.44.4.82 140.44.4.83; do
ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
done
# ------------------------------------------------------------------
# TIER 6: Master Node Self-Cutoff (FSD)
# ------------------------------------------------------------------
log "Stage 6: Executing final local halt and arming killpower..."
touch /etc/killpower
/sbin/shutdown -h +0
```
---
### D. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`)
1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**.
2. Check **Enable UPS Support**.
3. Select UPS Type: **Synology UPS Server / Network NUT Server**.
4. Network UPS server IP address: **`14.4.4.44`**.
5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `6 minutes` (guaranteeing that Tier 2 Swarm NFS workloads have cleanly scaled down before disk pools unmount).
6. Check **Shut down UPS when the system enters Standby Mode** (disabled, managed centrally by `14.4.4.44`).
---
### E. Peanut Web Dashboard Configuration
Peanut runs on Docker Swarm (`140.44.4.70:8050`). It connects directly to the NUT master server over internal TCP 3493:
```yaml
services:
peanut:
image: brandawg93/peanut:latest
ports:
- "8050:8050"
environment:
- TZ=America/New_York
- NUT_HOST=14.4.4.44
- NUT_PORT=3493
- NUT_USER=peanut
- NUT_PASSWORD=PEANUT_MONITOR_PASS
volumes:
- peanut_config:/config
volumes:
peanut_config:
driver: local
driver_opts:
type: none
o: bind
device: /mnt/swarm_shared/legacy_volumes/peanut/
```
---
## 5. Automated Power Restoration & Boot Sequence
When utility grid power returns, the homelab automatically recovers in a **strict bottom-up reverse sequence** to guarantee all storage and network prerequisites are ready before compute services attempt to start.
```mermaid
flowchart TD
G_On["1. Grid AC Restored\nUPS Battery Reaches >= 30%"] --> P1["2. Phase 1: Core Network & Edge\nUDM Pro (14.44.4.1) & Core PoE Switches\nEstablish VLANs, DHCP, Inter-VLAN Routing"]
P1 --> P2["3. Phase 2: NUT Master\nNode at 14.4.4.44 Boots (BIOS: Always On)\nupsd daemon starts, monitors line quality"]
P2 --> P3["4. Phase 3: Storage Foundation\nSynology SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) Boot\nMount Btrfs pools, start NFS & iSCSI daemons"]
P3 --> P4["5. Phase 4: Control Planes & Hypervisors\n- Harvester spacedock (140.44.4.81-83) boots via IPMI/WOL\n- Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) K3s masters boot\n- etcd clusters re-establish quorum"]
P4 --> P5["6. Phase 5: Storage CSI & Compute Workers\n- Aerodrome & Spaceport worker nodes boot\n- Longhorn CSI engine initializes, verifies volume health, mounts PVCs"]
P5 --> P6["7. Phase 6: Ingress & Workloads\n- Docker Swarm (140.44.4.71-75) boots\n- NPM host-mode container binds 80/443/81 on 140.44.4.74\n- Authentik SSO & Forward Auth come online\n- Applications (Peanut, Gotify, Homebox, etc.) become reachable"]
```
### Critical Health Verification Gates:
1. **Storage Gate (Phase 3 $\rightarrow$ Phase 4):**
- Synology NAS NFS daemons must be responding on `14.10.10.71:2049` before Docker Swarm starts, ensuring NPM does not encounter stale NFS mounts or SQLite locking.
2. **Database Gate (Phase 5 $\rightarrow$ Phase 6):**
- Longhorn volume `data-authentik-postgresql-0` on Aerodrome must reach `Healthy` state before Authentik server pods start.
3. **Ingress Gate (Phase 6):**
- Nginx Proxy Manager on `snshpd4lnxswm04` (`140.44.4.74`) confirms host-mode ports `80` and `443` are bound and forwards traffic to internal VIPs.