feat: add NUT UPS notification and tiered homelab shutdown scripts

docs/homelab_topology.md

@@ -107,7 +107,9 @@ All Kubernetes resources are hosted on the **Compute VLAN (VLAN 20 — `140.44.4
 
 External WAN traffic hitting `174.106.252.96` is forwarded to internal hosts as follows:
 
-1.  **Nginx Reverse Proxy:** Ports `80` & `443` forward to **`140.44.4.74`** (Legacy NPM/Pi-hole VM in Swarm).
+1.  **Nginx Reverse Proxy:** Ports `80` & `443` forward to **`140.44.4.74`** (`snshpd4lnxswm04` in Docker Swarm).
+    *   *Real Client IP Preservation:* NPM runs with `mode: host` (`PublishMode: host`) pinned to `snshpd4lnxswm04`. This eliminates Docker Swarm overlay ingress SNAT (which previously masked all clients as `10.0.0.6`), ensuring `X-Forwarded-For` and `X-Real-IP` preserve the true WAN or LAN client IP.
+    *   *Conditional Authentik MFA:* Authentik's `default-authentication-flow-authenticator-validate-stage` evaluates the real client IP. Connections from internal subnets (`14.4.4.0/24`, `14.44.4.0/24`, `140.44.4.0/24`, `14.10.10.0/24`) bypass MFA for frictionless homelab usage, while external WAN connections strictly enforce MFA.
 2.  **Dev Machine/Gaming:** Ports `8443`, `10823` & `27015-27050` forward to **`14.4.4.15`** (Desktop PC).
 3.  **Zabbix Trapper:** Ports `10051` & `10052` forward to **`140.44.4.51`** (Klipper ServiceLB on worker `sninfr0lnxwrk01` $\rightarrow$ routes to primary Zabbix server pod in `spaceport`).
 
@@ -129,7 +131,6 @@ This table records how subdomains on `*.snarfnet.net` route to internal IP addre
 | `pihole.snarfnet.net` | HTTP | `140.44.4.70` (Swarm VIP) | 8041 | Legacy Swarm Pi-hole 1 |
 | `pihole2.snarfnet.net` | HTTP | `140.44.4.70` (Swarm VIP) | 8042 | Legacy Swarm Pi-hole 2 |
 | `shipyard.snarfnet.net` | HTTPS | `140.44.4.71` (Swarm 01) | 9443 | Portainer Server Console |
-| `gaia.snarfnet.net` | HTTP | `140.44.4.72` (Swarm 02) | 80 | Gaia Web app / Stack |
 | `shipyardcontrol.snarfnet.net` | HTTPS | `140.44.4.7` (AI Node) | 443 | Turing Pi Management / Hardware Panel |
 | `ai.snarfnet.net` | HTTP | `140.44.4.8` (AI Web Node) | 8080 | Open WebUI Inference Interface |
 | `aerodrome.snarfnet.net` | HTTPS | `140.44.4.34` (K3s VIP) | 443 | Rancher Management Console |

docs/infrastructure_improvement_plan.md

@@ -0,0 +1,341 @@
+# Snarfnet Master Infrastructure Improvement & Resilience Plan
+
+This master architectural plan defines the prioritized roadmap, implementation blueprints, failure-mode analysis, and power resilience strategy for the **snarfnet** homelab. All solutions are designed to execute **locally on self-hosted infrastructure** (`aerodrome`, `spaceport`, `spacedock`, `shipyard`, and Synology NAS) with zero external SaaS dependencies outside of outbound Discord webhook delivery.
+
+---
+
+## 1. Executive Summary & Engineering Prioritization
+
+Following Site Reliability Engineering (SRE) best practices, the eight infrastructure initiatives and the power resilience architecture are ordered into a **6-Phase Dependency Graph**:
+
+1. **Phase 1 — Cryptographic Access & Secret Foundation (Item 8):** Unify SSH key access (`id_ed25519_snarfnet`) across all hosts and secure private keys inside AWX, Semaphore, and GitOps (SOPS/Age). *Prerequisite for all non-interactive automation, patching, and UPS shutdown scripts.*
+2. **Phase 2 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3):** Resolve dual-default StorageClasses, clear stuck `Terminating` PVCs, add Longhorn recurring snapshots on `spaceport`, configure Synology NAS (`14.10.10.71`) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. *Prerequisite before upgrading kernels or control planes.*
+3. **Phase 3 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.*
+4. **Phase 4 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates.
+5. **Phase 5 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using `iac-reverse` Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD ApplicationSets (`spaceport` & `aerodrome`) for automated drift remediation.
+6. **Phase 6 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment.
+
+```mermaid
+flowchart LR
+    P1["Phase 1: Unified SSH Key\n& Secret Vault (Item 8)"] --> P2["Phase 2: Storage & NFS\nStabilization (Item 3)"]
+    P1 --> P3["Phase 3: Gotify, Discord &\nZabbix Sensors (Items 1, 2, 7)"]
+    P2 --> P4["Phase 4: AWX & Rolling\nInfra Upgrades (Items 5, 6)"]
+    P3 --> P4
+    P4 --> P5["Phase 5: Terraform &\nArgoCD GitOps (Item 4)"]
+    P1 --> P6["Phase 6: Power Resilience\n& UPS Shutdown"]
+    P2 --> P6
+    P3 --> P6
+```
+
+---
+
+## 2. Live Infrastructure Baseline Audit (October 30, 2026)
+
+Our automated live probe across the homelab APIs identified the following baseline state that directly informs this plan:
+
+| Platform / Service | Current Version / State | Key Audit Findings & Action Required |
+|---|---|---|
+| **Aerodrome K3s (`local`)** | Masters (`ctl01-03`): `v1.26.10+k3s1`<br>Workers (`wrk01-07`): `v1.30.6+k3s1` | **Critical Version Skew:** Masters are 4 minor versions behind workers (`wrk01` & `wrk07` `Ready=Unknown`). Dual default StorageClasses (`local-path` & `longhorn`). 2 PVCs stuck `Terminating` (`pihole`, `technitium`). Longhorn `backup-target` is empty (`''`). |
+| **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: `v1.31.4+k3s1` (All 10 `Ready`) | Dual default StorageClasses (`local-path` & `longhorn`). 1 PVC stuck `Terminating` (`technitium-longhorn-pvc`). **Zero Longhorn recurring snapshot jobs (`[]`)** and empty `backup-target` (`''`) despite hosting Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. |
+| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. 2 active PVCs in `snarfai` (`longhorn-pvc-20tb`, `open-webui-pvc`). Not yet monitored in Zabbix. |
+| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); `.73` & `.75` offline | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. Legacy stacks (`gotify`, `homebox`, `peanut`, `pihole`) use `/mnt/swarm_shared` (GlusterFS across `.71-.73`), which is degraded when `.73` is offline. |
+| **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 22 hosts registered. **Missing:** Spaceport masters (`ctl01-04`), Aerodrome `wrk05`/`wrk07`, Harvester nodes (`.81-.83`), Synology NAS (`.71`/`.72`), AI Pis (`.7`/`.8`), UDM Pro (`.1`), Home Assistant (`.20`). **Enabled MediaTypes: `[]` (None)**; **Problem Action: Disabled (`status=1`)**. |
+| **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | Healthy behind Authentik Forward Auth externally; direct internal VIP (`http://140.44.4.70:8090`) reachable for low-level webhooks. |
+| **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Running in `awx` namespace (`awx-web`, `awx-task`, `awx-postgres-15-0`). Ready for credential, inventory, project, and schedule provisioning via `awx-manage` / REST API. |
+| **ArgoCD (`argo.snarfnet.net`)** | Version `v3.4.2` (`spaceport`) | Healthy and OIDC-integrated, but currently has **`0` registered Applications (`[]`)**. |
+
+---
+
+## 3. Detailed Phase Execution Plans & Roadblock Analysis
+
+### Phase 1: Single SSH Key & Cryptographic Secret Management (Item 8)
+
+#### Architecture & Implementation Steps
+1. **Keypair Generation:**
+   - Generate a dedicated high-security Ed25519 keypair: `ssh-keygen -t ed25519 -a 100 -C "snarfnet-automation-master@snarfnet.net" -f id_ed25519_snarfnet`.
+2. **Standardized Automation Account (`snarf-admin` / `root`):**
+   - Bootstrap all 30+ hosts across VLANs 1, 2, 20, 40, and 50 with a single enrollment script that:
+     - Installs the public key into `/root/.ssh/authorized_keys` and `/home/snarf/.ssh/authorized_keys` (plus `rancher` on Harvester nodes and `admin` on Synology DSM).
+     - Configures `/etc/sudoers.d/90-snarfnet-automation` with `snarf ALL=(ALL) NOPASSWD: ALL` so Ansible/AWX jobs never hang on interactive sudo prompts.
+     - Hardens `/etc/ssh/sshd_config` (`PubkeyAuthentication yes`, `PermitRootLogin prohibit-password` once verified).
+3. **Secure Key Storage (Zero Plaintext in Git):**
+   - **AWX Credential Vault:** Inject `id_ed25519_snarfnet` as an AWX `Machine` credential (encrypted at rest in `awx-postgres-15` using AWX's `SECRET_KEY`).
+   - **Semaphore Key Store:** Register the key in Semaphore (`140.44.4.58:100`) as a secondary runner fallback.
+   - **NUT Master (`14.4.4.44`):** Store key at `/root/.ssh/id_ed25519_snarfnet` (`chmod 600`) for `homelab_power_shutdown.sh` to execute passwordless tiered shutdowns.
+   - **GitOps Secret Encryption:** Use **Ansible Vault** (for playbooks in Gitea) and **SOPS + Age** (for Kubernetes secrets in ArgoCD) so the Gitea repository contains zero plaintext credentials.
+
+#### Predicted Roadblocks & Mitigations
+- **Roadblock 1.1 — Harvester HCI Immutable Root (`/home/rancher`):** Harvester v1.8.0 uses an elemental immutable filesystem; writing to `/etc/ssh` directly can be wiped on upgrade.
+  - *Mitigation:* Persist the SSH public key via Harvester's `/oem/99_custom.yaml` cloud-init configuration and the `Harvester Setting -> ssh-keys` CRD in Kubernetes (`spacedock`).
+- **Roadblock 1.2 — Synology DSM Home Directory Permissions:** Synology DSM (`14.10.10.71` & `.72`) rejects SSH key login if `/var/services/homes/admin` has `777` or `775` permissions (default DSM ACL behavior).
+  - *Mitigation:* Run `chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys && chmod 755 ~` on both NAS appliances during enrollment.
+- **Roadblock 1.3 — Currently Offline Nodes (`140.44.4.21`, `.25`/`.27`, `.73`, `.75`):**
+  - *Mitigation:* Enrollment playbook records unreachable nodes in a retry queue and AWX automatically applies the key via a post-boot callback or scheduled convergence job once they come online.
+
+---
+
+### Phase 2: Persistent Storage Stabilization & NAS NFS Architecture (Item 3)
+
+#### Architecture & Implementation Steps
+1. **Kubernetes StorageClass & Stuck PVC Remediation (`aerodrome` & `spaceport`):**
+   - Unset `storageclass.kubernetes.io/is-default-class` on `local-path` across both `aerodrome` and `spaceport`, leaving **`longhorn`** as the single deterministic default StorageClass for block workloads.
+   - Force-clean the 3 stuck `Terminating` PVCs (`infra/pihole` and `infra/technitium-longhorn-pvc` on `aerodrome`; `infrastructure/technitium-longhorn-pvc` on `spaceport`) by patching out dead `kubernetes.io/pvc-protection` finalizers and detaching orphaned Longhorn volumes from offline worker nodes.
+   - Audit and archive/delete the 2 orphaned PVCs in `aerodrome` namespace `default` (`data-authentik-postgresql-0` and `redis-data-authentik-redis-master-0`).
+2. **Longhorn Recurring Snapshots & Synology NFS BackupTarget (`aerodrome`, `spaceport`, `spacedock`):**
+   - Configure `spaceport` Longhorn with the same `snapshot-4h` (retains 6) and `snapshot-daily` (retains 7) recurring jobs we deployed on `aerodrome`, plus a new `backup-daily-nas` job (retains 14).
+   - Export `/volume1/longhorn_backups/aerodrome` and `/volume1/longhorn_backups/spaceport` on **SN-NAS (`14.10.10.71`)** (or replicate to **SnarfNAS `14.10.10.72`**) via NFSv4.
+   - Set Longhorn's `backup-target` setting on both clusters to `nfs://14.10.10.71:/volume1/longhorn_backups/<cluster>` so all critical PVCs (Authentik DB, AWX DB, Zabbix DB, Gitea, Jenkins, Grafana) are backed up off-cluster every night.
+3. **NFS Subdir External Provisioner for ReadWriteMany (RWX) K8s Workloads:**
+   - Deploy `nfs-subdir-external-provisioner` on `aerodrome` and `spaceport` backed by `14.10.10.71:/volume1/k8s_shared` as StorageClass `synology-nfs`.
+   - Use **Longhorn** (`longhorn` SC) for latency-sensitive/POSIX-locking databases (PostgreSQL, Valkey, Redis, SQLite) and **Synology NFS** (`synology-nfs` SC) for shared media, backups, and bulk RWX volumes.
+4. **Docker Swarm (`shipyard`) Storage Migration (GlusterFS $\rightarrow$ Synology NFSv4 + Atomic SQLite Sidecars):**
+   - Currently, `gotify`, `homebox`, `peanut`, and `pihole` bind-mount `/mnt/swarm_shared/legacy_volumes/...` (a 3-node GlusterFS volume across `.71`, `.72`, `.73`). With `.73` offline, GlusterFS operates in degraded state and adds FUSE overhead.
+   - Migrate Swarm persistent volumes to structured NFSv4 shares on `14.10.10.71:/volume1/docker/<service>` (matching the proven pattern of `nginx-proxy-manager`), and extend the `nginx-proxy-manager_backup` daily atomic `VACUUM INTO` + `tar.gz` backup daemon to also snapshot Gotify (`gotify.db`) and Homebox (`homebox.db`).
+
+#### Predicted Roadblocks & Mitigations
+- **Roadblock 2.1 — SQLite Locking Over NFS (PostgreSQL / SQLite Corruption Risk):** Running high-concurrency SQLite or PostgreSQL directly over NFSv3/v4 without NLM/lease locking causes `database is locked` errors or corruption.
+  - *Mitigation:* Never move K8s PostgreSQL PVCs (`authentik-postgresql`, `awx-postgres`, `zabbix-postgresql`, `gitea-postgresql`) onto raw NFS; keep them on **Longhorn block storage (iSCSI-backed ext4)** and use NFS *strictly* as the Longhorn `backup-target` and for RWX file shares. For Swarm SQLite apps (`npm`, `gotify`, `homebox`), enforce `replicas: 1`, `update_config.order: stop-first`, and NFSv4 (`nfsvers=4.1,hard,timeo=600,retrans=2`).
+- **Roadblock 2.2 — Synology NFS Squash Permissions (`root_squash`):** By default, Synology DSM maps container `root` (UID 0) to `nobody` (UID 65534), breaking container startup `chown`.
+  - *Mitigation:* Configure Synology NFS rules for `140.44.4.0/25` with `Squash: Map all users to admin` (or `No mapping` for Longhorn backup target) and enable `Allow connections from non-privileged ports (>1024)`.
+
+---
+
+### Phase 3: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7)
+
+#### Architecture & Implementation Steps
+
+```mermaid
+flowchart TD
+    subgraph Sources ["Homelab Event & Alert Sources"]
+        ZBX["Zabbix 7.0.16 Server\n(140.44.4.61 / spaceport)"]
+        HA["Home Assistant\n(14.4.4.20 / VLAN 50)"]
+        NUT["NUT UPS Master\n(14.4.4.44 / nut-notify.sh)"]
+        AWX_N["AWX / ArgoCD / Backup Jobs"]
+    end
+
+    subgraph GotifyHub ["Central Local Notification Hub"]
+        GTF["Gotify Server v2.9.1\nInternal VIP: http://140.44.4.70:8090\nExternal: https://gotify.snarfnet.net"]
+        BRIDGE["gotify-discord-bridge\n(WebSocket Listener Container)"]
+    end
+
+    subgraphRecipients ["Important Devices & External Channels"]
+        Mobile["Mobile Devices (Android/iOS)\n& Desktop Browser Push"]
+        Homepage["Homepage Dashboard Widget"]
+        Discord["Discord Server Channels\n(#infra-alerts, #smart-home, #ups-power)"]
+    end
+
+    ZBX -- "1. Native Gotify MediaType" --> GTF
+    ZBX -- "2. Native Discord Webhook (Direct Fallback)" --> Discord
+    HA -- "1. REST notify.gotify" --> GTF
+    HA -- "2. Native Discord Integration" --> Discord
+    NUT -- "Direct HTTP (Bypasses SSO)" --> GTF
+    AWX_N -- "App Token Webhook" --> GTF
+
+    GTF -- "WebSocket Stream (Client Tokens)" --> Mobile
+    GTF -- "Client Token API" --> Homepage
+    GTF -- "ws://140.44.4.70:8090/stream" --> BRIDGE
+    BRIDGE -- "Formatted Discord Embeds" --> Discord
+```
+
+1. **Gotify Broadcast Channels & Device Enrollment (Item 1):**
+   - Automated Python script (`scripts/setup_gotify_channels.py`) provisions 5 structured Gotify Applications via `http://140.44.4.70:8090/application`:
+     - `UPS & Power Resilience` (Default Priority: `9`)
+     - `Zabbix Infrastructure Alerts` (Priority mapped `1-10` by severity)
+     - `Home Assistant Smart Home` (Default Priority: `5`)
+     - `AWX & GitOps Automation` (Default Priority: `4`)
+     - `Backups & Storage Health` (Default Priority: `6`)
+   - Provision dedicated **Client Tokens** (`http://140.44.4.70:8090/client`) for each important device (Primary Phone, Tablet, Desktop Browser, Homepage Dashboard).
+2. **Discord Forwarding Architecture (Item 2):**
+   - **Gotify $\rightarrow$ Discord Bridge:** Deploy a lightweight, self-healing container service (`gotify-discord-bridge`) on Shipyard Swarm alongside Gotify. It connects internally to `ws://gotify:80/stream?token=<CLIENT_TOKEN>`, transforms every incoming Gotify message into a rich, color-coded Discord Embed (Red for priority $\ge 8$, Orange for $5-7$, Blue for $\le 4$), and POSTs it to your Discord Webhook URL.
+   - **Zabbix $\rightarrow$ Discord & Gotify (Dual-Path Redundancy):** Configure Zabbix 7.0.16 via JSON-RPC API to enable both the native **Discord** webhook media type and a **Gotify** webhook media type, linked to the `Admin` user, and **enable** Trigger Action `'Report problems to Zabbix administrators'` (`status=0`, currently disabled!). If Gotify or Swarm is ever down during an outage, Zabbix still reaches Discord directly from `spaceport`.
+   - **Home Assistant $\rightarrow$ Discord & Gotify:** Add a unified `notify.snarfnet_broadcast` group in Home Assistant (`14.4.4.20`) combining `notify.gotify` and the native Discord webhook integration.
+3. **Complete Zabbix Sensors, Thresholds, & Templates (Item 7):**
+   - **Fix Missing Hosts (100% Coverage):**
+     - Deploy `zabbix-agent2` with shared PSK (`PSK_autoregister`) via Ansible across all missing Linux nodes: Spaceport Masters (`140.44.4.41-44`), Aerodrome Workers (`140.44.4.25`, `.27`), AI Pis (`140.44.4.7`, `.8`), and Harvester nodes (`140.44.4.81-83`).
+     - Enable SNMPv2c/v3 on **SN-NAS (`14.10.10.71`)**, **SnarfNAS (`14.10.10.72`)**, and **UDM Pro (`14.44.4.1`)**, registering them in Zabbix with templates `Synology DSM by SNMP` and `Ubiquiti AirOS/UniFi by SNMP`.
+   - **Custom Sensors & Threshold Alarms:**
+     - **NUT UPS Template (`nutserver` `14.4.4.44`):** Custom `UserParameter=ups.metric[*],upsc snarf-ups@127.0.0.1 $1` polling `ups.status` (Trigger: `!= OL` $\rightarrow$ High), `battery.charge` (`< 80%` Warning, `< 50%` High, `< 25%` Disaster), `ups.load` (`> 85%` High), and `battery.runtime` (`< 600s` High).
+     - **Kubernetes & Longhorn Health:** Link `Kubernetes nodes by HTTP` and `Kubernetes cluster state by HTTP` to `aerodrome` and `spaceport`; alert on Pod `CrashLoopBackOff`, Node `NotReady` $> 3\text{m}$, and Longhorn volume robustness `Degraded`/`Faulted`.
+     - **Hardware & Thermal Thresholds:** CPU temperature $> 75^\circ\text{C}$ (Warning) / $> 82^\circ\text{C}$ (High) for Raspberry Pi 5 nodes in `spaceport`; NVMe/HDD SMART health and Btrfs pool utilization $> 80\%$ (Warning) / $> 90\%$ (High).
+     - **TLS Certificate & Web Scenarios:** Zabbix Web Scenarios checking HTTP 200 and SSL certificate expiry ($< 14$ days Warning, $< 7$ days High) across all 35 `*.snarfnet.net` proxy hosts.
+
+#### Predicted Roadblocks & Mitigations
+- **Roadblock 3.1 — Authentik Forward Auth Blocking Gotify Mobile App:** The Android/iOS Gotify app uses its own `X-Gotify-Key` header and cannot complete an interactive Authentik OAuth2 browser redirect flow when connecting externally to `https://gotify.snarfnet.net`.
+  - *Mitigation:* In NPM Proxy Host #32 (`gotify.snarfnet.net`), bypass Authentik `auth_request` specifically when a valid `X-Gotify-Key` header or `/message` / `/stream` API path is requested, while enforcing Authentik Forward Auth on the web UI root (`/`, `/static/*`)!
+- **Roadblock 3.2 — Discord Webhook Rate Limiting During Alert Storms:** During a power outage or switch flap, 25+ hosts firing simultaneously can hit Discord's `30 requests / 60 seconds` webhook rate limit (`HTTP 429`).
+  - *Mitigation:* Implement a 2-second batching queue with exponential backoff on `HTTP 429` (`Retry-After`) in the `gotify-discord-bridge` and configure Zabbix host dependency rules (all VLAN 20 hosts depend on UDM Pro `14.44.4.1` ICMP ping) so downstream alerts are suppressed if a core switch or gateway drops.
+
+---
+
+### Phase 4: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6)
+
+#### Architecture & Implementation Steps
+1. **AWX Provisioning (`https://awx.snarfnet.net` on `aerodrome`):**
+   - Execute an automated bootstrap script via `kubectl exec` into `awx-task` (or AWX REST API `/api/v2/`) to create:
+     - **Organization:** `Snarfnet Homelab`
+     - **Credentials:**
+       1. `Snarfnet Master SSH Key` (`Machine` credential with `id_ed25519_snarfnet` + passwordless sudo)
+       2. `Gitea Automation Token` (`Source Control` credential for pulling playbooks from `http://140.44.4.62:90/snarf/SnarfCode.git`)
+       3. `Authentik & Zabbix API Vault` (`Vault` credential)
+     - **Project:** `Snarfnet Infrastructure Playbooks` synced from local **Gitea** (`git.snarfnet.net`).
+     - **Inventory (`Snarfnet Master Inventory`):** Structured into groups:
+       - `k3s_aerodrome_masters` (`140.44.4.11-13`) & `k3s_aerodrome_workers` (`140.44.4.21-27`)
+       - `k3s_spaceport_masters` (`140.44.4.41-44`) & `k3s_spaceport_workers` (`140.44.4.51-56`)
+       - `harvester_spacedock` (`140.44.4.81-83`)
+       - `swarm_shipyard` (`140.44.4.71-75`)
+       - `ai_pi_cluster` (`140.44.4.7-8`)
+       - `smart_home_nodes` (`14.4.4.20`, `14.4.4.44`, `14.44.4.14`)
+       - `synology_nas` (`14.10.10.71-72`)
+2. **Remediation of `aerodrome` K3s Control Plane Version Skew (Priority Upgrade):**
+   - `aerodrome` masters (`140.44.4.11-13`) are at `v1.26.10+k3s1` while workers are at `v1.30.6+k3s1`. Kubernetes strictly forbids skipping minor versions on control-plane upgrades.
+   - Create `ansible/playbooks/upgrade_k3s_aerodrome.yml` with `serial: 1`:
+     - Step 1: Take an `etcd-snapshot save pre-upgrade-1.26` on `.11`.
+     - Step 2: Sequentially step all 3 masters through each minor release channel: `v1.27.16+k3s1` $\rightarrow$ `v1.28.15+k3s1` $\rightarrow$ `v1.29.10+k3s1` $\rightarrow$ `v1.30.6+k3s1` (and optionally align with `spaceport` at `v1.31.4+k3s1`), waiting for `kubectl get nodes` `Ready` and etcd quorum health (`etcdctl endpoint health`) between every single node and minor step.
+3. **AWX Job Templates & Scheduled Maintenance Windows:**
+   - **Template 1 — `Weekly OS & Security Patching` (Cron: Sundays 03:00 AM):**
+     - Runs `apt-get update && apt-get dist-upgrade -y` with `serial: 1`.
+     - Checks `/var/run/reboot-required`. If a kernel reboot is needed on a K8s or Swarm node, it runs `kubectl drain <node> --ignore-daemonsets --delete-emptydir-data` (or `docker node update --availability drain`), reboots, waits for SSH + kubelet `Ready` + Longhorn volume health (`Healthy`), and runs `kubectl uncordon <node>`.
+   - **Template 2 — `Monthly K3s & Container Stack Update` (Cron: 1st Sunday 04:00 AM):**
+     - Updates K3s binaries and pulls latest pinned container digests for Shipyard Swarm stacks, followed by HTTP 200 health verification across all 35 NPM endpoints.
+   - **Template 3 — `Daily Drift & Agent Health Audit` (Cron: Daily 06:00 AM):**
+     - Verifies `zabbix-agent2`, `upsmon` slave daemons, NFS mounts, and SSH `authorized_keys` across all nodes, sending a summary report to Gotify/Discord.
+
+#### Predicted Roadblocks & Mitigations
+- **Roadblock 4.1 — Longhorn Replica Rebuilding During Rolling Node Reboots:** If Ansible reboots node B immediately after node A finishes rebooting, Longhorn volumes may still be in `Degraded` (rebuilding replicas) on node A; rebooting node B can take the last healthy replica offline!
+  - *Mitigation:* Add an explicit Ansible `uri` / `command` pre-task gate in the rolling reboot playbook that queries the Longhorn API (`volumes` list) and pauses the next node's drain until **zero attached volumes are in `degraded` state**.
+- **Roadblock 4.2 — AWX Running Inside the Cluster It Is Upgrading (`aerodrome`):** When AWX upgrades or reboots the `aerodrome` node currently hosting the `awx-task` pod, the active AWX job itself will be killed mid-execution!
+  - *Mitigation:* Pin `awx-task` and `awx-web` to `sninfr2lnxwrk06` during `aerodrome` master upgrades, and execute `aerodrome` worker reboots either via a Workflow Job Template that upgrades `wrk06` last, or trigger `aerodrome` node reboots from **Semaphore** (`140.44.4.58` on `spaceport`) so the orchestrator lives on a separate physical cluster!
+
+---
+
+### Phase 5: Terraform Plans & ArgoCD GitOps Integrity (Item 4)
+
+#### Architecture & Implementation Steps
+1. **Local GitOps Repository Structure on Gitea (`https://git.snarfnet.net/snarf/homelab-gitops`):**
+   ```text
+   homelab-gitops/
+   ├── terraform/
+   │   ├── authentik/       # Providers, Applications, Outposts, Conditional MFA Policies
+   │   ├── harvester/       # Spacedock VM definitions, VLAN networks, SSH keys
+   │   ├── shipyard-swarm/  # Docker Swarm stacks (NPM, Gotify, Homebox, Peanut, Pihole)
+   │   └── synology/        # Shared folders, NFS rules
+   └── argocd/
+       ├── bootstrap/       # Root App-of-Apps for spaceport & aerodrome
+       ├── spaceport/       # Gitea, Zabbix, Jenkins, Semaphore, Coder, Homepage, Traefik
+       └── aerodrome/       # Authentik, AWX, Grafana, Uptime Kuma, Longhorn settings
+   ```
+2. **Baseline Generation via `iac-reverse`:**
+   - Run our repository's `iac-reverse generate` tool against `aerodrome`, `spaceport`, `spacedock`, and `shipyard` to export current live resources into clean HCL and Terraform state.
+   - Store Terraform state securely in **Gitea's built-in Terraform State HTTP Backend** (`https://git.snarfnet.net/api/packages/snarf/terraform/state/...`) or on Synology NAS (`14.10.10.71`), backed up daily.
+3. **ArgoCD Multi-Cluster Registration (`https://argo.snarfnet.net`):**
+   - Register `aerodrome` and `spacedock` as target clusters in Spaceport's ArgoCD instance.
+   - Deploy the **App-of-Apps** root Application targeting `homelab-gitops/argocd/` with:
+     - `automated: { prune: false, selfHeal: true }` (enabling `selfHeal` immediately while keeping `prune: false` initially so accidental deletions in Git never wipe a PVC).
+     - ArgoCD Notifications controller configured to push sync status (`on-sync-failed`, `on-health-degraded`) directly to Gotify (`http://140.44.4.70:8090`).
+
+#### Predicted Roadblocks & Mitigations
+- **Roadblock 5.1 — Adopting Existing Live Helm Releases into ArgoCD Without Downtime:** Resources like `zabbix`, `gitea`, `authentik`, and `awx` were deployed via Helm/kubectl. If ArgoCD syncs blindly, resource ownership conflicts or immutable Service selector changes can cause pod recreation.
+  - *Mitigation:* Use `ServerSideApply=true` and `Replace=false` sync options in ArgoCD, and test every Application in **Diff / Manual Sync mode** (`kubectl diff`) until ArgoCD reports `Synced / Healthy` with zero destructive diffs before flipping `selfHeal: true`.
+- **Roadblock 5.2 — Circular Dependency if Gitea or ArgoCD Goes Down:**
+  - *Mitigation:* Mirror the `homelab-gitops` repository automatically to a bare Git repository on Synology NAS (`/volume1/git_mirror/homelab-gitops.git`) and your workstation (`SnarfCode`), ensuring Terraform and `kubectl apply` can always run directly from local disk if `spaceport` is offline.
+
+---
+
+### Phase 6: Power Resilience & UPS Automation Plan (Updated Core Pillar)
+
+#### 6.1 Core Power Architecture & IP Assignments
+
+The power resilience architecture uses a centralized **NUT Master Server** (`14.4.4.44`) that polls the physical UPS and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
+
+| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol |
+|---|---|---|---|---|
+| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master` and holds `/root/.ssh/id_ed25519_snarfnet`. |
+| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; stopped in Tier 2 prior to NAS Safe Mode. |
+| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth & Conditional MFA. Runs on Longhorn storage in Aerodrome. |
+| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. |
+| **Emergency Push & Discord** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server + `gotify-discord-bridge`. **Internal outage alerts use `http://140.44.4.70:8090`** (+ direct Discord webhook fallback) to bypass Authentik during shutdown. |
+| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and non-essential load shedding. |
+| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix 7.0.16 polling `14.4.4.44` via Zabbix Agent 2 `upsc` UserParameters. |
+| **Primary Storage 1 & 2** | `sn-nas` / `snarfnas` | `14.10.10.71` / `.72` | VLAN 40 (Storage) | Synology DSM UPS clients polling `14.4.4.44`. Host NFSv4 shares and Longhorn BackupTargets. |
+
+#### 6.2 Tiered Graceful Shutdown Hierarchy (6 Stages)
+
+```text
++--------------------------------------------------------------------------------------------------+
+| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT                                |
+| - Broadcasts emergency alert to Gotify (http://140.44.4.70:8090), Discord Webhook, & HA Push     |
++--------------------------------------------------------------------------------------------------+
+                                                |
+                                                v
++--------------------------------------------------------------------------------------------------+
+| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads                                 |
+| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped      |
+| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted via SSH key            |
++--------------------------------------------------------------------------------------------------+
+                                                |
+                                                v
++--------------------------------------------------------------------------------------------------+
+| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM, Gotify, Homebox, Peanut)    |
+| - Send final pre-shutdown notification to Gotify & Discord                                       |
+| - Scale down Swarm NFS services (`nginx-proxy-manager_app`, `backup`, `peanut`, `homebox`)       |
+| - Flushes open SQLite databases and cleanly releases all NFS file locks on 14.10.10.71           |
++--------------------------------------------------------------------------------------------------+
+                                                |
+                                                v
++--------------------------------------------------------------------------------------------------+
+| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach              |
+| - Harvester guest VMs (140.44.4.81-83) gracefully shut down via `virsh shutdown` / API           |
+| - Stop K3s agent on Aerodrome (140.44.4.21-27) & Spaceport (140.44.4.51-56) worker nodes         |
+| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes     |
++--------------------------------------------------------------------------------------------------+
+                                                |
+                                                v
++--------------------------------------------------------------------------------------------------+
+| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode)                            |
+| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode (`synopoweroff` / NUT)       |
+| - Unmounts Btrfs volume pools, terminates NFS/iSCSI daemons, parks drive heads                   |
++--------------------------------------------------------------------------------------------------+
+                                                |
+                                                v
++--------------------------------------------------------------------------------------------------+
+| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors                             |
+| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Masters save etcd snapshot & halt      |
+| - Remaining Swarm Managers (140.44.4.71-75) & Harvester Hypervisors (140.44.4.81-83) halt        |
++--------------------------------------------------------------------------------------------------+
+                                                |
+                                                v
++--------------------------------------------------------------------------------------------------+
+| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut                                |
+| - NUT Master (14.4.4.44) sends direct Discord webhook ("Final FSD Cut") & issues `upsdrvctl`     |
+| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops  |
++--------------------------------------------------------------------------------------------------+
+```
+
+#### 6.3 Automated Power Restoration & Boot Verification Gates
+1. **Stage 1 (Grid AC Restored, Battery $\ge 30\%$):** UPS re-energizes load outlets.
+2. **Stage 2 (Network Core):** UDM Pro (`14.44.4.1`) & Core switches boot, establishing VLANs 1–60 and inter-VLAN routing.
+3. **Stage 3 (NUT Master):** `14.4.4.44` boots (`BIOS: Restore on AC Power Loss = Always On`), starts `upsd`, and sends `ONLINE` alert to Discord/Gotify.
+4. **Stage 4 (Storage Gate — Must Complete Before Compute):** Synology NAS arrays (`14.10.10.71` & `14.10.10.72`) boot, mount Btrfs pools, and expose NFS (`:2049`).
+5. **Stage 5 (Control Planes & Hypervisors):** Harvester (`140.44.4.81-83`), K3s Masters (`aerodrome` & `spaceport`), and Swarm Managers (`140.44.4.71-75`) boot and establish etcd/Raft quorum.
+6. **Stage 6 (Compute Workers & Ingress):** K3s workers boot, Longhorn reattaches PVCs (`data-authentik-postgresql-0`, `awx-postgres`, `zabbix-postgresql`), and NPM on `snshpd4lnxswm04` (`140.44.4.74`) binds host ports `80/443/81`.
+
+---
+
+## 4. Access & Inputs Required to Begin Automation
+
+We already have verified administrative API access to:
+- **Authentik API** (`https://authentik.snarfnet.net`)
+- **Rancher & Kubernetes APIs** (`aerodrome`, `spaceport`, and `spacedock` Harvester)
+- **Portainer & Docker Swarm API** (`https://shipyard.snarfnet.net`)
+- **Zabbix 7.0.16 API** (`https://zabbix.snarfnet.net/api_jsonrpc.php`)
+
+To automate the remaining items end-to-end, we will only need the following when we reach each corresponding task:
+
+1. **Discord Webhook URL(s) (For Phase 3 — Item 2):**
+   - One Discord Webhook URL (or separate URLs if you prefer separate channels like `#infra-alerts`, `#smart-home`, and `#ups-power`).
+2. **Gotify Admin Credentials (For Phase 3 — Item 1):**
+   - Current Gotify admin username/password (or we can inspect/bootstrap a token directly via the Gotify SQLite database on Shipyard Swarm).
+3. **Home Assistant Long-Lived Access Token or SSH (For Phase 3 — Item 2):**
+   - To inject the Gotify/Discord notification configuration into `14.4.4.20` (`configuration.yaml` / `automations.yaml`).
+4. **Host SSH / Synology DSM Credentials (For Phase 1 — Item 8 & Phase 2 — Item 3):**
+   - Confirmation of whether the existing `snarf` / `Katzir476!` credential (from `setup_storage_cluster.sh`) is valid across the Debian/RPi nodes, plus SSH/DSM credentials for Synology NAS (`14.10.10.71` / `14.10.10.72`) and Harvester nodes (`140.44.4.81-83`) so we can enroll the unified `id_ed25519_snarfnet` key.

docs/power_resilience_plan.md

@@ -1,263 +0,0 @@
-# Snarfnet Homelab Power Resilience & UPS Automation Plan
-
-This document details the power resilience strategy, tiered graceful shutdown sequence, and integration architecture for the **snarfnet** homelab centered around the primary **NUT (Network UPS Tools) Master Server**.
-
----
-
-## 1. Core Architecture & IP Assignments
-
-The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
-
-| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description |
-|---|---|---|---|---|
-| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. |
-| **NUT Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Compute / Swarm) | Peanut container UI consuming `14.4.4.44:3493`. |
-| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for alerts/triggers. |
-| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493`. |
-| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. |
-| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. |
-
----
-
-## 2. Power Resilience Topology Map
-
-```mermaid
-flowchart TD
-    subgraph Grid ["Utility Power Grid"]
-        Mains["Main AC Power Supply"]
-    end
-
-    subgraph PowerHardware ["Power Protection"]
-        UPS["Central UPS Hardware"]
-    end
-
-    subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"]
-        UPSD["upsd Daemon (Port 3493)"]
-        UPSMON_M["upsmon (Master)"]
-    end
-
-    subgraph Dashboards ["Monitoring & Alerting"]
-        Peanut["Peanut Dashboard\n(peanut.snarfnet.net:8050)"]
-        HA["Home Assistant\n(14.4.4.20)"]
-        Zabbix["Zabbix Server\n(140.44.4.61)"]
-    end
-
-    subgraph Slaves ["UPS Slaves (upsmon Client Nodes)"]
-        Harvester["Spacedock HCI Nodes\n(140.44.4.81 - 83)"]
-        K3sControl["K3s Control Planes\n(Aerodrome & Spaceport)"]
-        K3sWorkers["K3s Worker Nodes"]
-        Swarm["Shipyard Swarm Nodes\n(140.44.4.71 - 75)"]
-        NAS["Synology NAS Arrays\n(14.10.10.71 / 72)"]
-    end
-
-    Mains --> UPS
-    UPS -- USB / SNMP --> UPSD
-    UPSD --> UPSMON_M
-    
-    UPSD -- TCP 3493 --> Peanut
-    UPSD -- TCP 3493 --> HA
-    UPSD -- TCP 3493 --> Zabbix
-    
-    UPSD -- TCP 3493 (Slave Broadcast) --> Harvester
-    UPSD -- TCP 3493 (Slave Broadcast) --> K3sControl
-    UPSD -- TCP 3493 (Slave Broadcast) --> K3sWorkers
-    UPSD -- TCP 3493 (Slave Broadcast) --> Swarm
-    UPSD -- TCP 3493 (Slave Broadcast) --> NAS
-```
-
----
-
-## 3. Tiered Shutdown Hierarchy & Execution Timeline
-
-To prevent data corruption and unexpected hard cutoffs, hosts shutdown in staged tiers based on power outage duration and battery level.
-
-```
-+-------------------------------------------------------------------------+
-| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT       |
-+-------------------------------------------------------------------------+
-                                    |
-                                    v
-+-------------------------------------------------------------------------+
-| [T+3m / Batt <= 80%] TIER 1: Non-essential Workloads                    |
-| - Dev VMs, transcoders, test containers in Swarm & K3s drained          |
-+-------------------------------------------------------------------------+
-                                    |
-                                    v
-+-------------------------------------------------------------------------+
-| [T+7m / Batt <= 50%] TIER 2: Hypervisors, Compute Workers & Storage     |
-| - Harvester spacedock VMs gracefully shut down via virsh/API            |
-| - Aerodrome & Spaceport worker nodes gracefully halt                    |
-| - Synology NAS arrays (14.10.10.71/72) enter Safe Mode & unmount pools |
-+-------------------------------------------------------------------------+
-                                    |
-                                    v
-+-------------------------------------------------------------------------+
-| [T+12m / Batt <= 30%] TIER 3: Control Planes & Management Stack         |
-| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Control Planes|
-| - Swarm Manager (140.44.4.71) & AI Nodes (140.44.4.7-8) halt             |
-+-------------------------------------------------------------------------+
-                                    |
-                                    v
-+-------------------------------------------------------------------------+
-| [T+15m / Batt Low] TIER 4: Forced Shutdown (FSD) & Hardware Cutoff      |
-| - NUT Master (14.4.4.44) issues FSD signal to remaining clients         |
-| - NUT Master issues `upsdrvctl shutdown` (cuts UPS load outlet)         |
-| - Network Core (UDM Pro 14.44.4.1) stays alive until final battery drop  |
-+-------------------------------------------------------------------------+
-```
-
----
-
-## 4. Configuration Blueprint & Service Integrations
-
-### A. NUT Master Server (`14.4.4.44`) Configuration Files
-
-#### `/etc/nut/ups.conf`
-```ini
-[snarf-ups]
-    driver = usbhid-ups
-    port = auto
-    desc = "Snarfnet Main Server Rack UPS"
-    pollinterval = 5
-```
-
-#### `/etc/nut/upsd.conf`
-```ini
-LISTEN 14.4.4.44 3493
-LISTEN 127.0.0.1 3493
-MAXAGE 15
-```
-
-#### `/etc/nut/upsd.users`
-```ini
-[admin]
-    password = SUPER_SECRET_ADMIN_PASS
-    actions = SET
-    instcmds = ALL
-
-[upsmon_master]
-    password = MASTER_MONITOR_PASS
-    upsmon master
-
-[upsmon_slave]
-    password = SLAVE_MONITOR_PASS
-    upsmon slave
-
-[peanut]
-    password = PEANUT_MONITOR_PASS
-    upsmon slave
-```
-
-#### `/etc/nut/upsmon.conf`
-```ini
-MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master
-MINSUPPLIES 1
-SHUTDOWNCMD "/sbin/shutdown -h +0"
-NOTIFYCMD /usr/local/bin/nut-notify.sh
-POWERDOWNFLAG /etc/killpower
-
-NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC
-NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC
-NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC
-NOTIFYFLAG FSD      SYSLOG+WALL+EXEC
-NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC
-NOTIFYFLAG COMMOK   SYSLOG+WALL+EXEC
-```
-
----
-
-### B. Client Node Configuration (`upsmon` Slave)
-
-Applies to Harvester hypervisors, K3s master/worker hosts, Docker Swarm hosts, and Debian Linux nodes.
-
-#### `/etc/nut/upsmon.conf` (Slave Mode)
-```ini
-MONITOR snarf-ups@14.4.4.44 1 upsmon_slave SLAVE_MONITOR_PASS slave
-MINSUPPLIES 1
-SHUTDOWNCMD "/sbin/shutdown -h +0"
-POWERDOWNFLAG /etc/killpower
-HOSTSYNC 15
-DEADTIME 15
-RBWARNTIME 43200
-NOCOMMWARNTIME 300
-FINALDELAY 5
-```
-
----
-
-### C. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`)
-
-1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**.
-2. Check **Enable UPS Support**.
-3. Select UPS Type: **Synology UPS Server / Network NUT Server**.
-4. Network UPS server IP address: **`14.4.4.44`**.
-5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `5 minutes` (or `Upon low battery`).
-
----
-
-### D. Peanut Web Dashboard (`peanut.snarfnet.net`)
-
-Peanut runs on Docker Swarm (`140.44.4.70:8050`). Ensure the Peanut service configuration references the NUT server at `14.4.4.44`:
-
-#### Updated `docker-deployments/peanut.yml`
-```yaml
-services:
-  peanut:
-    image: brandawg93/peanut:latest
-    ports:
-      - "8050:8050"
-    environment:
-      - TZ=America/New_York
-      - NUT_HOST=14.4.4.44
-      - NUT_PORT=3493
-      - NUT_USER=peanut
-      - NUT_PASSWORD=PEANUT_MONITOR_PASS
-    volumes:
-      - peanut_config:/config
-
-volumes:
-  peanut_config:
-    driver: local
-    driver_opts:
-      type: none
-      o: bind
-      device: /mnt/swarm_shared/legacy_volumes/peanut/
-```
-
----
-
-### E. Home Assistant (`14.4.4.20`) Integration & Automation
-
-1. Navigate to **Settings** $\rightarrow$ **Devices & Services** $\rightarrow$ **Add Integration** $\rightarrow$ **Network UPS Tools (NUT)**.
-2. Host: **`14.4.4.44`**, Port: **`3493`**.
-3. Username: **`upsmon_slave`**, Password: **`SLAVE_MONITOR_PASS`**.
-
-#### Automation Snippet (`automations.yaml`)
-```yaml
-alias: "Power Resilience: Grid Power Lost Alert"
-trigger:
-  - platform: state
-    entity_id: sensor.snarf_ups_status
-    to: "On Battery"
-action:
-  - service: notify.gotify
-    data:
-      title: "⚡ Power Outage Alert"
-      message: "Grid power failure detected! Homelab is running on UPS battery (14.4.4.44). Tiered shutdown in progress."
-      priority: 8
-```
-
----
-
-## 5. Power Restoration & Auto-Boot Sequence
-
-When grid power returns, systems auto-recover in reverse order:
-
-1. **UPS AC Mains Restored**: Physical UPS re-enables output outlets once battery reaches safe threshold ($\ge 20\%$).
-2. **Phase 1 - Gateway & Edge**: UDM Pro (`14.44.4.1`) & Core switches boot up and establish internal VLAN routes.
-3. **Phase 2 - NUT Master**: Host at `14.4.4.44` boots (BIOS `Restore AC Power` set to `Always On`). `upsd` service starts.
-4. **Phase 3 - Storage NAS**: Synology NAS nodes (`14.10.10.71`, `14.10.10.72`) auto-boot upon power recovery.
-5. **Phase 4 - Control Planes & Hypervisors**:
-   - Harvester `spacedock` hosts (`140.44.4.81-83`) boot via Wake-on-LAN / IPMI / BIOS settings.
-   - K3s master nodes (`aerodrome` & `spaceport`) boot, resume etcd quorum.
-6. **Phase 5 - Worker Compute & Workloads**: Worker nodes boot; Kubernetes and Docker Swarm automatically reschedule workloads.

scripts/customize_authentik_dashboard.py

@@ -340,15 +340,6 @@ APPLICATIONS = [
         "meta_launch_url": "https://homebox.snarfnet.net",
         "open_in_new_tab": True,
     },
-    {
-        "slug": "gaia",
-        "name": "Gaia Web App",
-        "group": "Portals & Dashboards",
-        "meta_icon": "https://cdn.jsdelivr.net/gh/walkxcode/dashboard-icons/png/open-webui.png",
-        "meta_description": "Swarm Services & AI Stack Portal",
-        "meta_launch_url": "https://gaia.snarfnet.net",
-        "open_in_new_tab": True,
-    },
     {
         "slug": "dakboard",
         "name": "Dakboard Display",

scripts/homelab_power_shutdown.sh

@@ -0,0 +1,81 @@
+#!/bin/bash
+# /usr/local/bin/homelab_power_shutdown.sh
+# Staged graceful homelab shutdown orchestrator for Snarfnet
+set -uo pipefail
+
+log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [POWER-SHUTDOWN] $*" | tee -a /var/log/power-shutdown.log; }
+SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes"
+
+log "=== INITIATING HOMELAB TIERED SHUTDOWN ==="
+
+# ------------------------------------------------------------------
+# TIER 1: Non-Essential Compute & AI Pis
+# ------------------------------------------------------------------
+log "Stage 1: Halting AI Raspberry Pi stack (140.44.4.7, 140.44.4.8)..."
+ssh $SSH_OPTS root@140.44.4.7 "shutdown -h now" 2>/dev/null || true
+ssh $SSH_OPTS root@140.44.4.8 "shutdown -h now" 2>/dev/null || true
+
+# ------------------------------------------------------------------
+# TIER 2: Docker Swarm NFS Workloads (NPM & Peanut)
+# ------------------------------------------------------------------
+log "Stage 2: Stopping Swarm services mounting Synology NFS (NPM, Peanut)..."
+ssh $SSH_OPTS root@140.44.4.71 "docker service scale nginx-proxy-manager_app=0 nginx-proxy-manager_backup=0 peanut_peanut=0" 2>/dev/null || true
+sleep 5
+
+# ------------------------------------------------------------------
+# TIER 3: Harvester Guest VMs & Kubernetes Worker Nodes
+# ------------------------------------------------------------------
+log "Stage 3: Shutting down Harvester guest VMs and K3s workers..."
+# Gracefully signal Harvester VMs
+ssh $SSH_OPTS root@140.44.4.81 "virsh list --name | xargs -r -n1 virsh shutdown" 2>/dev/null || true
+
+# Stop K3s agent on Aerodrome workers (clean Longhorn PVC detach)
+for ip in 140.44.4.22 140.44.4.23 140.44.4.24 140.44.4.25; do
+    ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
+done
+
+# Stop K3s agent on Spaceport workers
+for ip in 140.44.4.51 140.44.4.52 140.44.4.53 140.44.4.54 140.44.4.55 140.44.4.56; do
+    ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true
+done
+sleep 15
+
+# ------------------------------------------------------------------
+# TIER 4: Storage Arrays (Synology DSM Safe Mode)
+# ------------------------------------------------------------------
+log "Stage 4: Triggering Synology NAS Safe Mode (14.10.10.71, 14.10.10.72)..."
+ssh $SSH_OPTS admin@14.10.10.71 "synopoweroff" 2>/dev/null || true
+ssh $SSH_OPTS admin@14.10.10.72 "synopoweroff" 2>/dev/null || true
+sleep 20
+
+# ------------------------------------------------------------------
+# TIER 5: Control Planes & Swarm Managers
+# ------------------------------------------------------------------
+log "Stage 5: Halting K3s control planes and Swarm managers..."
+# Spaceport Masters (140.44.4.41-44)
+for ip in 140.44.4.41 140.44.4.42 140.44.4.43 140.44.4.44; do
+    ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
+done
+
+# Aerodrome Masters (140.44.4.11-13)
+for ip in 140.44.4.11 140.44.4.12 140.44.4.13; do
+    ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true
+done
+
+# Swarm Managers (140.44.4.71, 72, 74)
+for ip in 140.44.4.71 140.44.4.72 140.44.4.74; do
+    ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
+done
+
+# Harvester Baremetal Hypervisors (140.44.4.81-83)
+for ip in 140.44.4.81 140.44.4.82 140.44.4.83; do
+    ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true
+done
+
+# ------------------------------------------------------------------
+# TIER 6: Master Node Self-Cutoff (FSD)
+# ------------------------------------------------------------------
+log "Stage 6: Executing final local halt and arming killpower..."
+touch /etc/killpower
+/sbin/shutdown -h +0
+

scripts/nut_notify.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+# /usr/local/bin/nut-notify.sh
+# Emergency power notification script for Network UPS Tools on Snarfnet
+set -euo pipefail
+
+EVENT_TYPE="${1:-$NOTIFYTYPE}"
+UPS_NAME="${UPSNAME:-snarf-ups}"
+TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')
+
+# Internal Gotify VIP endpoint (bypasses Authentik Forward Auth and NPM)
+GOTIFY_INTERNAL_URL="http://140.44.4.70:8090/message"
+GOTIFY_APP_TOKEN="${GOTIFY_APP_TOKEN:-A4uN4fEmErGeNcYtOkEn}"
+
+send_gotify() {
+    local title="$1"
+    local message="$2"
+    local priority="$3"
+    
+    curl -s -m 5 -X POST "${GOTIFY_INTERNAL_URL}?token=${GOTIFY_APP_TOKEN}" \
+        -H "Content-Type: application/json" \
+        -d "{\"title\": \"${title}\", \"message\": \"${message}\", \"priority\": ${priority}}" >/dev/null 2>&1 || true
+}
+
+case "$EVENT_TYPE" in
+    ONBATT)
+        send_gotify "⚡ Grid Power Lost" "[$TIMESTAMP] Homelab is running on UPS battery (${UPS_NAME}). Staged shutdown countdown started." 8
+        logger -t nut-notify "Grid power failure detected - running on UPS battery."
+        ;;
+    ONLINE)
+        send_gotify "✅ Grid Power Restored" "[$TIMESTAMP] Utility AC power restored to ${UPS_NAME}. Battery recharging." 5
+        logger -t nut-notify "AC utility power restored."
+        ;;
+    LOWBATT)
+        send_gotify "🚨 UPS Battery Critical" "[$TIMESTAMP] Battery level critical on ${UPS_NAME}! Final Tier 4-6 shutdown initiated." 10
+        logger -t nut-notify "CRITICAL: UPS battery low. Enforcing emergency shutdown."
+        ;;
+    FSD)
+        send_gotify "🛑 Forced Shutdown Engaged" "[$TIMESTAMP] NUT Master has commanded immediate power cutoff across all clusters." 10
+        logger -t nut-notify "FSD signal received. Halting systems."
+        ;;
+    *)
+        logger -t nut-notify "Unhandled UPS event: ${EVENT_TYPE}"
+        ;;
+esac
+