diff --git a/docs/infrastructure_improvement_plan.md b/docs/infrastructure_improvement_plan.md new file mode 100644 index 0000000..3965e80 --- /dev/null +++ b/docs/infrastructure_improvement_plan.md @@ -0,0 +1,341 @@ +# Snarfnet Master Infrastructure Improvement & Resilience Plan + +This master architectural plan defines the prioritized roadmap, implementation blueprints, failure-mode analysis, and power resilience strategy for the **snarfnet** homelab. All solutions are designed to execute **locally on self-hosted infrastructure** (`aerodrome`, `spaceport`, `spacedock`, `shipyard`, and Synology NAS) with zero external SaaS dependencies outside of outbound Discord webhook delivery. + +--- + +## 1. Executive Summary & Engineering Prioritization + +Following Site Reliability Engineering (SRE) best practices, the eight infrastructure initiatives and the power resilience architecture are ordered into a **6-Phase Dependency Graph**: + +1. **Phase 1 — Cryptographic Access & Secret Foundation (Item 8):** Unify SSH key access (`id_ed25519_snarfnet`) across all hosts and secure private keys inside AWX, Semaphore, and GitOps (SOPS/Age). *Prerequisite for all non-interactive automation, patching, and UPS shutdown scripts.* +2. **Phase 2 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3):** Resolve dual-default StorageClasses, clear stuck `Terminating` PVCs, add Longhorn recurring snapshots on `spaceport`, configure Synology NAS (`14.10.10.71`) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. *Prerequisite before upgrading kernels or control planes.* +3. **Phase 3 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.* +4. **Phase 4 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates. +5. **Phase 5 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using `iac-reverse` Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD ApplicationSets (`spaceport` & `aerodrome`) for automated drift remediation. +6. **Phase 6 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment. + +```mermaid +flowchart LR + P1["Phase 1: Unified SSH Key\n& Secret Vault (Item 8)"] --> P2["Phase 2: Storage & NFS\nStabilization (Item 3)"] + P1 --> P3["Phase 3: Gotify, Discord &\nZabbix Sensors (Items 1, 2, 7)"] + P2 --> P4["Phase 4: AWX & Rolling\nInfra Upgrades (Items 5, 6)"] + P3 --> P4 + P4 --> P5["Phase 5: Terraform &\nArgoCD GitOps (Item 4)"] + P1 --> P6["Phase 6: Power Resilience\n& UPS Shutdown"] + P2 --> P6 + P3 --> P6 +``` + +--- + +## 2. Live Infrastructure Baseline Audit (October 30, 2026) + +Our automated live probe across the homelab APIs identified the following baseline state that directly informs this plan: + +| Platform / Service | Current Version / State | Key Audit Findings & Action Required | +|---|---|---| +| **Aerodrome K3s (`local`)** | Masters (`ctl01-03`): `v1.26.10+k3s1`
Workers (`wrk01-07`): `v1.30.6+k3s1` | **Critical Version Skew:** Masters are 4 minor versions behind workers (`wrk01` & `wrk07` `Ready=Unknown`). Dual default StorageClasses (`local-path` & `longhorn`). 2 PVCs stuck `Terminating` (`pihole`, `technitium`). Longhorn `backup-target` is empty (`''`). | +| **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: `v1.31.4+k3s1` (All 10 `Ready`) | Dual default StorageClasses (`local-path` & `longhorn`). 1 PVC stuck `Terminating` (`technitium-longhorn-pvc`). **Zero Longhorn recurring snapshot jobs (`[]`)** and empty `backup-target` (`''`) despite hosting Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. | +| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. 2 active PVCs in `snarfai` (`longhorn-pvc-20tb`, `open-webui-pvc`). Not yet monitored in Zabbix. | +| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); `.73` & `.75` offline | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. Legacy stacks (`gotify`, `homebox`, `peanut`, `pihole`) use `/mnt/swarm_shared` (GlusterFS across `.71-.73`), which is degraded when `.73` is offline. | +| **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 22 hosts registered. **Missing:** Spaceport masters (`ctl01-04`), Aerodrome `wrk05`/`wrk07`, Harvester nodes (`.81-.83`), Synology NAS (`.71`/`.72`), AI Pis (`.7`/`.8`), UDM Pro (`.1`), Home Assistant (`.20`). **Enabled MediaTypes: `[]` (None)**; **Problem Action: Disabled (`status=1`)**. | +| **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | Healthy behind Authentik Forward Auth externally; direct internal VIP (`http://140.44.4.70:8090`) reachable for low-level webhooks. | +| **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Running in `awx` namespace (`awx-web`, `awx-task`, `awx-postgres-15-0`). Ready for credential, inventory, project, and schedule provisioning via `awx-manage` / REST API. | +| **ArgoCD (`argo.snarfnet.net`)** | Version `v3.4.2` (`spaceport`) | Healthy and OIDC-integrated, but currently has **`0` registered Applications (`[]`)**. | + +--- + +## 3. Detailed Phase Execution Plans & Roadblock Analysis + +### Phase 1: Single SSH Key & Cryptographic Secret Management (Item 8) + +#### Architecture & Implementation Steps +1. **Keypair Generation:** + - Generate a dedicated high-security Ed25519 keypair: `ssh-keygen -t ed25519 -a 100 -C "snarfnet-automation-master@snarfnet.net" -f id_ed25519_snarfnet`. +2. **Standardized Automation Account (`snarf-admin` / `root`):** + - Bootstrap all 30+ hosts across VLANs 1, 2, 20, 40, and 50 with a single enrollment script that: + - Installs the public key into `/root/.ssh/authorized_keys` and `/home/snarf/.ssh/authorized_keys` (plus `rancher` on Harvester nodes and `admin` on Synology DSM). + - Configures `/etc/sudoers.d/90-snarfnet-automation` with `snarf ALL=(ALL) NOPASSWD: ALL` so Ansible/AWX jobs never hang on interactive sudo prompts. + - Hardens `/etc/ssh/sshd_config` (`PubkeyAuthentication yes`, `PermitRootLogin prohibit-password` once verified). +3. **Secure Key Storage (Zero Plaintext in Git):** + - **AWX Credential Vault:** Inject `id_ed25519_snarfnet` as an AWX `Machine` credential (encrypted at rest in `awx-postgres-15` using AWX's `SECRET_KEY`). + - **Semaphore Key Store:** Register the key in Semaphore (`140.44.4.58:100`) as a secondary runner fallback. + - **NUT Master (`14.4.4.44`):** Store key at `/root/.ssh/id_ed25519_snarfnet` (`chmod 600`) for `homelab_power_shutdown.sh` to execute passwordless tiered shutdowns. + - **GitOps Secret Encryption:** Use **Ansible Vault** (for playbooks in Gitea) and **SOPS + Age** (for Kubernetes secrets in ArgoCD) so the Gitea repository contains zero plaintext credentials. + +#### Predicted Roadblocks & Mitigations +- **Roadblock 1.1 — Harvester HCI Immutable Root (`/home/rancher`):** Harvester v1.8.0 uses an elemental immutable filesystem; writing to `/etc/ssh` directly can be wiped on upgrade. + - *Mitigation:* Persist the SSH public key via Harvester's `/oem/99_custom.yaml` cloud-init configuration and the `Harvester Setting -> ssh-keys` CRD in Kubernetes (`spacedock`). +- **Roadblock 1.2 — Synology DSM Home Directory Permissions:** Synology DSM (`14.10.10.71` & `.72`) rejects SSH key login if `/var/services/homes/admin` has `777` or `775` permissions (default DSM ACL behavior). + - *Mitigation:* Run `chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys && chmod 755 ~` on both NAS appliances during enrollment. +- **Roadblock 1.3 — Currently Offline Nodes (`140.44.4.21`, `.25`/`.27`, `.73`, `.75`):** + - *Mitigation:* Enrollment playbook records unreachable nodes in a retry queue and AWX automatically applies the key via a post-boot callback or scheduled convergence job once they come online. + +--- + +### Phase 2: Persistent Storage Stabilization & NAS NFS Architecture (Item 3) + +#### Architecture & Implementation Steps +1. **Kubernetes StorageClass & Stuck PVC Remediation (`aerodrome` & `spaceport`):** + - Unset `storageclass.kubernetes.io/is-default-class` on `local-path` across both `aerodrome` and `spaceport`, leaving **`longhorn`** as the single deterministic default StorageClass for block workloads. + - Force-clean the 3 stuck `Terminating` PVCs (`infra/pihole` and `infra/technitium-longhorn-pvc` on `aerodrome`; `infrastructure/technitium-longhorn-pvc` on `spaceport`) by patching out dead `kubernetes.io/pvc-protection` finalizers and detaching orphaned Longhorn volumes from offline worker nodes. + - Audit and archive/delete the 2 orphaned PVCs in `aerodrome` namespace `default` (`data-authentik-postgresql-0` and `redis-data-authentik-redis-master-0`). +2. **Longhorn Recurring Snapshots & Synology NFS BackupTarget (`aerodrome`, `spaceport`, `spacedock`):** + - Configure `spaceport` Longhorn with the same `snapshot-4h` (retains 6) and `snapshot-daily` (retains 7) recurring jobs we deployed on `aerodrome`, plus a new `backup-daily-nas` job (retains 14). + - Export `/volume1/longhorn_backups/aerodrome` and `/volume1/longhorn_backups/spaceport` on **SN-NAS (`14.10.10.71`)** (or replicate to **SnarfNAS `14.10.10.72`**) via NFSv4. + - Set Longhorn's `backup-target` setting on both clusters to `nfs://14.10.10.71:/volume1/longhorn_backups/` so all critical PVCs (Authentik DB, AWX DB, Zabbix DB, Gitea, Jenkins, Grafana) are backed up off-cluster every night. +3. **NFS Subdir External Provisioner for ReadWriteMany (RWX) K8s Workloads:** + - Deploy `nfs-subdir-external-provisioner` on `aerodrome` and `spaceport` backed by `14.10.10.71:/volume1/k8s_shared` as StorageClass `synology-nfs`. + - Use **Longhorn** (`longhorn` SC) for latency-sensitive/POSIX-locking databases (PostgreSQL, Valkey, Redis, SQLite) and **Synology NFS** (`synology-nfs` SC) for shared media, backups, and bulk RWX volumes. +4. **Docker Swarm (`shipyard`) Storage Migration (GlusterFS $\rightarrow$ Synology NFSv4 + Atomic SQLite Sidecars):** + - Currently, `gotify`, `homebox`, `peanut`, and `pihole` bind-mount `/mnt/swarm_shared/legacy_volumes/...` (a 3-node GlusterFS volume across `.71`, `.72`, `.73`). With `.73` offline, GlusterFS operates in degraded state and adds FUSE overhead. + - Migrate Swarm persistent volumes to structured NFSv4 shares on `14.10.10.71:/volume1/docker/` (matching the proven pattern of `nginx-proxy-manager`), and extend the `nginx-proxy-manager_backup` daily atomic `VACUUM INTO` + `tar.gz` backup daemon to also snapshot Gotify (`gotify.db`) and Homebox (`homebox.db`). + +#### Predicted Roadblocks & Mitigations +- **Roadblock 2.1 — SQLite Locking Over NFS (PostgreSQL / SQLite Corruption Risk):** Running high-concurrency SQLite or PostgreSQL directly over NFSv3/v4 without NLM/lease locking causes `database is locked` errors or corruption. + - *Mitigation:* Never move K8s PostgreSQL PVCs (`authentik-postgresql`, `awx-postgres`, `zabbix-postgresql`, `gitea-postgresql`) onto raw NFS; keep them on **Longhorn block storage (iSCSI-backed ext4)** and use NFS *strictly* as the Longhorn `backup-target` and for RWX file shares. For Swarm SQLite apps (`npm`, `gotify`, `homebox`), enforce `replicas: 1`, `update_config.order: stop-first`, and NFSv4 (`nfsvers=4.1,hard,timeo=600,retrans=2`). +- **Roadblock 2.2 — Synology NFS Squash Permissions (`root_squash`):** By default, Synology DSM maps container `root` (UID 0) to `nobody` (UID 65534), breaking container startup `chown`. + - *Mitigation:* Configure Synology NFS rules for `140.44.4.0/25` with `Squash: Map all users to admin` (or `No mapping` for Longhorn backup target) and enable `Allow connections from non-privileged ports (>1024)`. + +--- + +### Phase 3: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7) + +#### Architecture & Implementation Steps + +```mermaid +flowchart TD + subgraph Sources ["Homelab Event & Alert Sources"] + ZBX["Zabbix 7.0.16 Server\n(140.44.4.61 / spaceport)"] + HA["Home Assistant\n(14.4.4.20 / VLAN 50)"] + NUT["NUT UPS Master\n(14.4.4.44 / nut-notify.sh)"] + AWX_N["AWX / ArgoCD / Backup Jobs"] + end + + subgraph GotifyHub ["Central Local Notification Hub"] + GTF["Gotify Server v2.9.1\nInternal VIP: http://140.44.4.70:8090\nExternal: https://gotify.snarfnet.net"] + BRIDGE["gotify-discord-bridge\n(WebSocket Listener Container)"] + end + + subgraphRecipients ["Important Devices & External Channels"] + Mobile["Mobile Devices (Android/iOS)\n& Desktop Browser Push"] + Homepage["Homepage Dashboard Widget"] + Discord["Discord Server Channels\n(#infra-alerts, #smart-home, #ups-power)"] + end + + ZBX -- "1. Native Gotify MediaType" --> GTF + ZBX -- "2. Native Discord Webhook (Direct Fallback)" --> Discord + HA -- "1. REST notify.gotify" --> GTF + HA -- "2. Native Discord Integration" --> Discord + NUT -- "Direct HTTP (Bypasses SSO)" --> GTF + AWX_N -- "App Token Webhook" --> GTF + + GTF -- "WebSocket Stream (Client Tokens)" --> Mobile + GTF -- "Client Token API" --> Homepage + GTF -- "ws://140.44.4.70:8090/stream" --> BRIDGE + BRIDGE -- "Formatted Discord Embeds" --> Discord +``` + +1. **Gotify Broadcast Channels & Device Enrollment (Item 1):** + - Automated Python script (`scripts/setup_gotify_channels.py`) provisions 5 structured Gotify Applications via `http://140.44.4.70:8090/application`: + - `UPS & Power Resilience` (Default Priority: `9`) + - `Zabbix Infrastructure Alerts` (Priority mapped `1-10` by severity) + - `Home Assistant Smart Home` (Default Priority: `5`) + - `AWX & GitOps Automation` (Default Priority: `4`) + - `Backups & Storage Health` (Default Priority: `6`) + - Provision dedicated **Client Tokens** (`http://140.44.4.70:8090/client`) for each important device (Primary Phone, Tablet, Desktop Browser, Homepage Dashboard). +2. **Discord Forwarding Architecture (Item 2):** + - **Gotify $\rightarrow$ Discord Bridge:** Deploy a lightweight, self-healing container service (`gotify-discord-bridge`) on Shipyard Swarm alongside Gotify. It connects internally to `ws://gotify:80/stream?token=`, transforms every incoming Gotify message into a rich, color-coded Discord Embed (Red for priority $\ge 8$, Orange for $5-7$, Blue for $\le 4$), and POSTs it to your Discord Webhook URL. + - **Zabbix $\rightarrow$ Discord & Gotify (Dual-Path Redundancy):** Configure Zabbix 7.0.16 via JSON-RPC API to enable both the native **Discord** webhook media type and a **Gotify** webhook media type, linked to the `Admin` user, and **enable** Trigger Action `'Report problems to Zabbix administrators'` (`status=0`, currently disabled!). If Gotify or Swarm is ever down during an outage, Zabbix still reaches Discord directly from `spaceport`. + - **Home Assistant $\rightarrow$ Discord & Gotify:** Add a unified `notify.snarfnet_broadcast` group in Home Assistant (`14.4.4.20`) combining `notify.gotify` and the native Discord webhook integration. +3. **Complete Zabbix Sensors, Thresholds, & Templates (Item 7):** + - **Fix Missing Hosts (100% Coverage):** + - Deploy `zabbix-agent2` with shared PSK (`PSK_autoregister`) via Ansible across all missing Linux nodes: Spaceport Masters (`140.44.4.41-44`), Aerodrome Workers (`140.44.4.25`, `.27`), AI Pis (`140.44.4.7`, `.8`), and Harvester nodes (`140.44.4.81-83`). + - Enable SNMPv2c/v3 on **SN-NAS (`14.10.10.71`)**, **SnarfNAS (`14.10.10.72`)**, and **UDM Pro (`14.44.4.1`)**, registering them in Zabbix with templates `Synology DSM by SNMP` and `Ubiquiti AirOS/UniFi by SNMP`. + - **Custom Sensors & Threshold Alarms:** + - **NUT UPS Template (`nutserver` `14.4.4.44`):** Custom `UserParameter=ups.metric[*],upsc snarf-ups@127.0.0.1 $1` polling `ups.status` (Trigger: `!= OL` $\rightarrow$ High), `battery.charge` (`< 80%` Warning, `< 50%` High, `< 25%` Disaster), `ups.load` (`> 85%` High), and `battery.runtime` (`< 600s` High). + - **Kubernetes & Longhorn Health:** Link `Kubernetes nodes by HTTP` and `Kubernetes cluster state by HTTP` to `aerodrome` and `spaceport`; alert on Pod `CrashLoopBackOff`, Node `NotReady` $> 3\text{m}$, and Longhorn volume robustness `Degraded`/`Faulted`. + - **Hardware & Thermal Thresholds:** CPU temperature $> 75^\circ\text{C}$ (Warning) / $> 82^\circ\text{C}$ (High) for Raspberry Pi 5 nodes in `spaceport`; NVMe/HDD SMART health and Btrfs pool utilization $> 80\%$ (Warning) / $> 90\%$ (High). + - **TLS Certificate & Web Scenarios:** Zabbix Web Scenarios checking HTTP 200 and SSL certificate expiry ($< 14$ days Warning, $< 7$ days High) across all 35 `*.snarfnet.net` proxy hosts. + +#### Predicted Roadblocks & Mitigations +- **Roadblock 3.1 — Authentik Forward Auth Blocking Gotify Mobile App:** The Android/iOS Gotify app uses its own `X-Gotify-Key` header and cannot complete an interactive Authentik OAuth2 browser redirect flow when connecting externally to `https://gotify.snarfnet.net`. + - *Mitigation:* In NPM Proxy Host #32 (`gotify.snarfnet.net`), bypass Authentik `auth_request` specifically when a valid `X-Gotify-Key` header or `/message` / `/stream` API path is requested, while enforcing Authentik Forward Auth on the web UI root (`/`, `/static/*`)! +- **Roadblock 3.2 — Discord Webhook Rate Limiting During Alert Storms:** During a power outage or switch flap, 25+ hosts firing simultaneously can hit Discord's `30 requests / 60 seconds` webhook rate limit (`HTTP 429`). + - *Mitigation:* Implement a 2-second batching queue with exponential backoff on `HTTP 429` (`Retry-After`) in the `gotify-discord-bridge` and configure Zabbix host dependency rules (all VLAN 20 hosts depend on UDM Pro `14.44.4.1` ICMP ping) so downstream alerts are suppressed if a core switch or gateway drops. + +--- + +### Phase 4: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6) + +#### Architecture & Implementation Steps +1. **AWX Provisioning (`https://awx.snarfnet.net` on `aerodrome`):** + - Execute an automated bootstrap script via `kubectl exec` into `awx-task` (or AWX REST API `/api/v2/`) to create: + - **Organization:** `Snarfnet Homelab` + - **Credentials:** + 1. `Snarfnet Master SSH Key` (`Machine` credential with `id_ed25519_snarfnet` + passwordless sudo) + 2. `Gitea Automation Token` (`Source Control` credential for pulling playbooks from `http://140.44.4.62:90/snarf/SnarfCode.git`) + 3. `Authentik & Zabbix API Vault` (`Vault` credential) + - **Project:** `Snarfnet Infrastructure Playbooks` synced from local **Gitea** (`git.snarfnet.net`). + - **Inventory (`Snarfnet Master Inventory`):** Structured into groups: + - `k3s_aerodrome_masters` (`140.44.4.11-13`) & `k3s_aerodrome_workers` (`140.44.4.21-27`) + - `k3s_spaceport_masters` (`140.44.4.41-44`) & `k3s_spaceport_workers` (`140.44.4.51-56`) + - `harvester_spacedock` (`140.44.4.81-83`) + - `swarm_shipyard` (`140.44.4.71-75`) + - `ai_pi_cluster` (`140.44.4.7-8`) + - `smart_home_nodes` (`14.4.4.20`, `14.4.4.44`, `14.44.4.14`) + - `synology_nas` (`14.10.10.71-72`) +2. **Remediation of `aerodrome` K3s Control Plane Version Skew (Priority Upgrade):** + - `aerodrome` masters (`140.44.4.11-13`) are at `v1.26.10+k3s1` while workers are at `v1.30.6+k3s1`. Kubernetes strictly forbids skipping minor versions on control-plane upgrades. + - Create `ansible/playbooks/upgrade_k3s_aerodrome.yml` with `serial: 1`: + - Step 1: Take an `etcd-snapshot save pre-upgrade-1.26` on `.11`. + - Step 2: Sequentially step all 3 masters through each minor release channel: `v1.27.16+k3s1` $\rightarrow$ `v1.28.15+k3s1` $\rightarrow$ `v1.29.10+k3s1` $\rightarrow$ `v1.30.6+k3s1` (and optionally align with `spaceport` at `v1.31.4+k3s1`), waiting for `kubectl get nodes` `Ready` and etcd quorum health (`etcdctl endpoint health`) between every single node and minor step. +3. **AWX Job Templates & Scheduled Maintenance Windows:** + - **Template 1 — `Weekly OS & Security Patching` (Cron: Sundays 03:00 AM):** + - Runs `apt-get update && apt-get dist-upgrade -y` with `serial: 1`. + - Checks `/var/run/reboot-required`. If a kernel reboot is needed on a K8s or Swarm node, it runs `kubectl drain --ignore-daemonsets --delete-emptydir-data` (or `docker node update --availability drain`), reboots, waits for SSH + kubelet `Ready` + Longhorn volume health (`Healthy`), and runs `kubectl uncordon `. + - **Template 2 — `Monthly K3s & Container Stack Update` (Cron: 1st Sunday 04:00 AM):** + - Updates K3s binaries and pulls latest pinned container digests for Shipyard Swarm stacks, followed by HTTP 200 health verification across all 35 NPM endpoints. + - **Template 3 — `Daily Drift & Agent Health Audit` (Cron: Daily 06:00 AM):** + - Verifies `zabbix-agent2`, `upsmon` slave daemons, NFS mounts, and SSH `authorized_keys` across all nodes, sending a summary report to Gotify/Discord. + +#### Predicted Roadblocks & Mitigations +- **Roadblock 4.1 — Longhorn Replica Rebuilding During Rolling Node Reboots:** If Ansible reboots node B immediately after node A finishes rebooting, Longhorn volumes may still be in `Degraded` (rebuilding replicas) on node A; rebooting node B can take the last healthy replica offline! + - *Mitigation:* Add an explicit Ansible `uri` / `command` pre-task gate in the rolling reboot playbook that queries the Longhorn API (`volumes` list) and pauses the next node's drain until **zero attached volumes are in `degraded` state**. +- **Roadblock 4.2 — AWX Running Inside the Cluster It Is Upgrading (`aerodrome`):** When AWX upgrades or reboots the `aerodrome` node currently hosting the `awx-task` pod, the active AWX job itself will be killed mid-execution! + - *Mitigation:* Pin `awx-task` and `awx-web` to `sninfr2lnxwrk06` during `aerodrome` master upgrades, and execute `aerodrome` worker reboots either via a Workflow Job Template that upgrades `wrk06` last, or trigger `aerodrome` node reboots from **Semaphore** (`140.44.4.58` on `spaceport`) so the orchestrator lives on a separate physical cluster! + +--- + +### Phase 5: Terraform Plans & ArgoCD GitOps Integrity (Item 4) + +#### Architecture & Implementation Steps +1. **Local GitOps Repository Structure on Gitea (`https://git.snarfnet.net/snarf/homelab-gitops`):** + ```text + homelab-gitops/ + ├── terraform/ + │ ├── authentik/ # Providers, Applications, Outposts, Conditional MFA Policies + │ ├── harvester/ # Spacedock VM definitions, VLAN networks, SSH keys + │ ├── shipyard-swarm/ # Docker Swarm stacks (NPM, Gotify, Homebox, Peanut, Pihole) + │ └── synology/ # Shared folders, NFS rules + └── argocd/ + ├── bootstrap/ # Root App-of-Apps for spaceport & aerodrome + ├── spaceport/ # Gitea, Zabbix, Jenkins, Semaphore, Coder, Homepage, Traefik + └── aerodrome/ # Authentik, AWX, Grafana, Uptime Kuma, Longhorn settings + ``` +2. **Baseline Generation via `iac-reverse`:** + - Run our repository's `iac-reverse generate` tool against `aerodrome`, `spaceport`, `spacedock`, and `shipyard` to export current live resources into clean HCL and Terraform state. + - Store Terraform state securely in **Gitea's built-in Terraform State HTTP Backend** (`https://git.snarfnet.net/api/packages/snarf/terraform/state/...`) or on Synology NAS (`14.10.10.71`), backed up daily. +3. **ArgoCD Multi-Cluster Registration (`https://argo.snarfnet.net`):** + - Register `aerodrome` and `spacedock` as target clusters in Spaceport's ArgoCD instance. + - Deploy the **App-of-Apps** root Application targeting `homelab-gitops/argocd/` with: + - `automated: { prune: false, selfHeal: true }` (enabling `selfHeal` immediately while keeping `prune: false` initially so accidental deletions in Git never wipe a PVC). + - ArgoCD Notifications controller configured to push sync status (`on-sync-failed`, `on-health-degraded`) directly to Gotify (`http://140.44.4.70:8090`). + +#### Predicted Roadblocks & Mitigations +- **Roadblock 5.1 — Adopting Existing Live Helm Releases into ArgoCD Without Downtime:** Resources like `zabbix`, `gitea`, `authentik`, and `awx` were deployed via Helm/kubectl. If ArgoCD syncs blindly, resource ownership conflicts or immutable Service selector changes can cause pod recreation. + - *Mitigation:* Use `ServerSideApply=true` and `Replace=false` sync options in ArgoCD, and test every Application in **Diff / Manual Sync mode** (`kubectl diff`) until ArgoCD reports `Synced / Healthy` with zero destructive diffs before flipping `selfHeal: true`. +- **Roadblock 5.2 — Circular Dependency if Gitea or ArgoCD Goes Down:** + - *Mitigation:* Mirror the `homelab-gitops` repository automatically to a bare Git repository on Synology NAS (`/volume1/git_mirror/homelab-gitops.git`) and your workstation (`SnarfCode`), ensuring Terraform and `kubectl apply` can always run directly from local disk if `spaceport` is offline. + +--- + +### Phase 6: Power Resilience & UPS Automation Plan (Updated Core Pillar) + +#### 6.1 Core Power Architecture & IP Assignments + +The power resilience architecture uses a centralized **NUT Master Server** (`14.4.4.44`) that polls the physical UPS and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs. + +| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol | +|---|---|---|---|---| +| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master` and holds `/root/.ssh/id_ed25519_snarfnet`. | +| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; stopped in Tier 2 prior to NAS Safe Mode. | +| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth & Conditional MFA. Runs on Longhorn storage in Aerodrome. | +| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. | +| **Emergency Push & Discord** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server + `gotify-discord-bridge`. **Internal outage alerts use `http://140.44.4.70:8090`** (+ direct Discord webhook fallback) to bypass Authentik during shutdown. | +| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and non-essential load shedding. | +| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix 7.0.16 polling `14.4.4.44` via Zabbix Agent 2 `upsc` UserParameters. | +| **Primary Storage 1 & 2** | `sn-nas` / `snarfnas` | `14.10.10.71` / `.72` | VLAN 40 (Storage) | Synology DSM UPS clients polling `14.4.4.44`. Host NFSv4 shares and Longhorn BackupTargets. | + +#### 6.2 Tiered Graceful Shutdown Hierarchy (6 Stages) + +```text ++--------------------------------------------------------------------------------------------------+ +| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT | +| - Broadcasts emergency alert to Gotify (http://140.44.4.70:8090), Discord Webhook, & HA Push | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads | +| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped | +| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted via SSH key | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM, Gotify, Homebox, Peanut) | +| - Send final pre-shutdown notification to Gotify & Discord | +| - Scale down Swarm NFS services (`nginx-proxy-manager_app`, `backup`, `peanut`, `homebox`) | +| - Flushes open SQLite databases and cleanly releases all NFS file locks on 14.10.10.71 | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach | +| - Harvester guest VMs (140.44.4.81-83) gracefully shut down via `virsh shutdown` / API | +| - Stop K3s agent on Aerodrome (140.44.4.21-27) & Spaceport (140.44.4.51-56) worker nodes | +| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode) | +| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode (`synopoweroff` / NUT) | +| - Unmounts Btrfs volume pools, terminates NFS/iSCSI daemons, parks drive heads | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors | +| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Masters save etcd snapshot & halt | +| - Remaining Swarm Managers (140.44.4.71-75) & Harvester Hypervisors (140.44.4.81-83) halt | ++--------------------------------------------------------------------------------------------------+ + | + v ++--------------------------------------------------------------------------------------------------+ +| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut | +| - NUT Master (14.4.4.44) sends direct Discord webhook ("Final FSD Cut") & issues `upsdrvctl` | +| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops | ++--------------------------------------------------------------------------------------------------+ +``` + +#### 6.3 Automated Power Restoration & Boot Verification Gates +1. **Stage 1 (Grid AC Restored, Battery $\ge 30\%$):** UPS re-energizes load outlets. +2. **Stage 2 (Network Core):** UDM Pro (`14.44.4.1`) & Core switches boot, establishing VLANs 1–60 and inter-VLAN routing. +3. **Stage 3 (NUT Master):** `14.4.4.44` boots (`BIOS: Restore on AC Power Loss = Always On`), starts `upsd`, and sends `ONLINE` alert to Discord/Gotify. +4. **Stage 4 (Storage Gate — Must Complete Before Compute):** Synology NAS arrays (`14.10.10.71` & `14.10.10.72`) boot, mount Btrfs pools, and expose NFS (`:2049`). +5. **Stage 5 (Control Planes & Hypervisors):** Harvester (`140.44.4.81-83`), K3s Masters (`aerodrome` & `spaceport`), and Swarm Managers (`140.44.4.71-75`) boot and establish etcd/Raft quorum. +6. **Stage 6 (Compute Workers & Ingress):** K3s workers boot, Longhorn reattaches PVCs (`data-authentik-postgresql-0`, `awx-postgres`, `zabbix-postgresql`), and NPM on `snshpd4lnxswm04` (`140.44.4.74`) binds host ports `80/443/81`. + +--- + +## 4. Access & Inputs Required to Begin Automation + +We already have verified administrative API access to: +- **Authentik API** (`https://authentik.snarfnet.net`) +- **Rancher & Kubernetes APIs** (`aerodrome`, `spaceport`, and `spacedock` Harvester) +- **Portainer & Docker Swarm API** (`https://shipyard.snarfnet.net`) +- **Zabbix 7.0.16 API** (`https://zabbix.snarfnet.net/api_jsonrpc.php`) + +To automate the remaining items end-to-end, we will only need the following when we reach each corresponding task: + +1. **Discord Webhook URL(s) (For Phase 3 — Item 2):** + - One Discord Webhook URL (or separate URLs if you prefer separate channels like `#infra-alerts`, `#smart-home`, and `#ups-power`). +2. **Gotify Admin Credentials (For Phase 3 — Item 1):** + - Current Gotify admin username/password (or we can inspect/bootstrap a token directly via the Gotify SQLite database on Shipyard Swarm). +3. **Home Assistant Long-Lived Access Token or SSH (For Phase 3 — Item 2):** + - To inject the Gotify/Discord notification configuration into `14.4.4.20` (`configuration.yaml` / `automations.yaml`). +4. **Host SSH / Synology DSM Credentials (For Phase 1 — Item 8 & Phase 2 — Item 3):** + - Confirmation of whether the existing `snarf` / `Katzir476!` credential (from `setup_storage_cluster.sh`) is valid across the Debian/RPi nodes, plus SSH/DSM credentials for Synology NAS (`14.10.10.71` / `14.10.10.72`) and Harvester nodes (`140.44.4.81-83`) so we can enroll the unified `id_ed25519_snarfnet` key. diff --git a/docs/power_resilience_plan.md b/docs/power_resilience_plan.md deleted file mode 100644 index cbd4fa6..0000000 --- a/docs/power_resilience_plan.md +++ /dev/null @@ -1,416 +0,0 @@ -# Snarfnet Homelab Power Resilience & UPS Automation Plan - -This document details the comprehensive power resilience strategy, tiered graceful shutdown sequence, and automated recovery orchestration for the **snarfnet** homelab. The system is centered around a dedicated **NUT (Network UPS Tools) Master Server** at `14.4.4.44`. - ---- - -## 1. Core Architecture & Component Roles - -The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs. - -| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol | -|---|---|---|---|---| -| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master`. | -| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; must be stopped prior to NAS Safe Mode. | -| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth for Peanut, Gotify, Homebox, WLED. Runs on Longhorn storage in Aerodrome. | -| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. | -| **Emergency Push Notifications** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server. **Internal outage alerts MUST use direct IP (`http://140.44.4.70:8090`)** to bypass Authentik during shutdown. | -| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and automated shedding. | -| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix Agent 2 / NUT exporter polling `14.4.4.44:3493` for battery voltage, load %, and runtime metrics. | -| **Primary Storage 1** | `sn-nas.snarfnet.net` | `14.10.10.71` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts NPM NFS shares (`/volume1/docker/nginx/data`). | -| **Primary Storage 2** | `snarfnas.snarfnet.net` | `14.10.10.72` | VLAN 40 (Storage) | Synology DSM UPS client polling `14.4.4.44`. Hosts secondary backup shares. | -| **Distributed Block Storage** | `longhorn.aerodrome.snarfnet.net` | `140.44.4.36:80` | VLAN 20 (Aerodrome K3s) | Longhorn CSI storage engine powering Authentik, AWX, Grafana PVCs. Protected with recurring snapshots (`snapshot-4h`, `snapshot-daily`). | - -> [!IMPORTANT] -> **Emergency Alerting Path Warning:** -> `peanut.snarfnet.net` and `gotify.snarfnet.net` are protected externally behind Authentik Forward Auth (`oauth2-proxy`). During a power outage, when K3s `aerodrome` shuts down in Tier 3, Authentik goes offline. Therefore, **emergency outage alerts and NUT notification scripts must communicate directly with Gotify via internal IP (`http://140.44.4.70:8090/message?token=...`)** or native Home Assistant push notifications, bypassing NPM and Authentik. - ---- - -## 2. Power Resilience Topology Map - -```mermaid -flowchart TD - subgraph Grid ["Utility Power Grid"] - Mains["Main AC Utility Power"] - end - - subgraph PowerHardware ["Power Hardware & Protection"] - UPS["Central UPS Hardware (Battery Bank)"] - end - - subgraph NUTMasterHost ["NUT Master Node (14.4.4.44)"] - UPSD["upsd Daemon (Port 3493)"] - UPSMON_M["upsmon (Master Daemon)"] - NOTIFY["nut-notify.sh (Direct Alert Engine)"] - end - - subgraph OutageAlerts ["Emergency Notification Channels"] - HA_Alert["Home Assistant (14.4.4.20)\nLocal Push Notification"] - Gotify_Direct["Gotify Internal Direct\nhttp://140.44.4.70:8090\n(Bypasses Authentik / NPM)"] - Syslog["Wall / Syslog Broadcast"] - end - - subgraph MonitoringPanels ["Monitoring & Management"] - Peanut["Peanut Dashboard\n(peanut.snarfnet.net)"] - Zabbix["Zabbix Server\n(140.44.4.61)"] - end - - subgraph InfrastructureConsumers ["Homelab Tiered Consumers"] - SwarmNFS["Docker Swarm (140.44.4.71-75)\nNPM Host Mode (140.44.4.74)"] - K8sCompute["Aerodrome & Spaceport K3s\nLonghorn Engine & Worker Nodes"] - NAS["Synology NAS Arrays\n(14.10.10.71 / 14.10.10.72)"] - K8sControl["K3s Control Planes & etcd"] - Harvester["Spacedock HCI Hypervisors\n(140.44.4.81 - 83)"] - CoreEdge["UDM Pro Gateway (14.44.4.1)\nCore PoE Switches"] - end - - Mains --> UPS - UPS -- USB / SNMP --> UPSD - UPSD --> UPSMON_M - UPSMON_M --> NOTIFY - - NOTIFY -- Direct API Token --> Gotify_Direct - NOTIFY -- TCP 3493 Event --> HA_Alert - NOTIFY --> Syslog - - UPSD -- TCP 3493 --> Peanut - UPSD -- TCP 3493 --> Zabbix - - UPSMON_M -. Tiered Signal .-> SwarmNFS - UPSMON_M -. Tiered Signal .-> K8sCompute - UPSMON_M -. Tiered Signal .-> NAS - UPSMON_M -. Tiered Signal .-> K8sControl - UPSMON_M -. Tiered Signal .-> Harvester - UPSMON_M -. Final Cut .-> CoreEdge -``` - ---- - -## 3. Tiered Graceful Shutdown Hierarchy - -To eliminate data corruption, prevent stale NFS file handles, and ensure Longhorn volume detach cleanly, the homelab executes a **strict 6-stage progressive shutdown sequence**. - -``` -+--------------------------------------------------------------------------------------------------+ -| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT | -| - Broadcasts emergency alert via Home Assistant push & Gotify direct API (http://140.44.4.70:8090)| -+--------------------------------------------------------------------------------------------------+ - | - v -+--------------------------------------------------------------------------------------------------+ -| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads | -| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped | -| - Media transcoders (Plex active sessions signaled) | -| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted | -+--------------------------------------------------------------------------------------------------+ - | - v -+--------------------------------------------------------------------------------------------------+ -| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM / Peanut) | -| - Stop `nginx-proxy-manager_app` & `nginx-proxy-manager_backup` on snshpd4lnxswm04 (140.44.4.74) | -| - Stop `peanut` container on Swarm | -| - Flushes open SQLite databases (database.sqlite) and unmounts NFS mounts to 14.10.10.71 | -+--------------------------------------------------------------------------------------------------+ - | - v -+--------------------------------------------------------------------------------------------------+ -| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach | -| - Harvester guest VMs (harvester.spacedock.snarfnet.net 140.44.4.90) gracefully shut down via API| -| - Cordon & scale down stateful pods on Aerodrome (Authentik PostgreSQL, AWX Postgres, Grafana) | -| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes | -| - Aerodrome (140.44.4.21-25) & Spaceport (140.44.4.51-56) worker nodes execute shutdown | -+--------------------------------------------------------------------------------------------------+ - | - v -+--------------------------------------------------------------------------------------------------+ -| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode) | -| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode via NUT client trigger | -| - Unmounts all Btrfs volume pools, terminates NFS/iSCSI daemon threads, parks drive read heads | -| - Zero active network clients remain connected to storage | -+--------------------------------------------------------------------------------------------------+ - | - v -+--------------------------------------------------------------------------------------------------+ -| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors | -| - Aerodrome K3s Control Planes (140.44.4.11-13) take etcd snapshot and halt | -| - Spaceport K3s Control Planes (140.44.4.41-44) take etcd snapshot and halt | -| - Swarm Manager nodes (140.44.4.71-75) halt | -| - Harvester baremetal hypervisors (140.44.4.81-83) halt via IPMI/SSH | -+--------------------------------------------------------------------------------------------------+ - | - v -+--------------------------------------------------------------------------------------------------+ -| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut | -| - NUT Master (14.4.4.44) issues final FSD broadcast to all remaining clients | -| - NUT Master issues `upsdrvctl shutdown` with /etc/killpower flag (commands UPS load drop) | -| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops | -+--------------------------------------------------------------------------------------------------+ -``` - ---- - -## 4. Configuration Blueprints & Integration Scripts - -### A. NUT Master Server (`14.4.4.44`) - -#### `/etc/nut/ups.conf` -```ini -[snarf-ups] - driver = usbhid-ups - port = auto - desc = "Snarfnet Main Rack UPS" - pollinterval = 5 -``` - -#### `/etc/nut/upsd.conf` -```ini -LISTEN 14.4.4.44 3493 -LISTEN 127.0.0.1 3493 -MAXAGE 15 -``` - -#### `/etc/nut/upsd.users` -```ini -[admin] - password = ADMIN_PASSWORD_CHANGE_ME - actions = SET - instcmds = ALL - -[upsmon_master] - password = MASTER_MONITOR_PASS - upsmon master - -[upsmon_slave] - password = SLAVE_MONITOR_PASS - upsmon slave - -[peanut] - password = PEANUT_MONITOR_PASS - upsmon slave -``` - -#### `/etc/nut/upsmon.conf` -```ini -MONITOR snarf-ups@14.4.4.44 1 upsmon_master MASTER_MONITOR_PASS master -MINSUPPLIES 1 -SHUTDOWNCMD "/usr/local/bin/homelab_power_shutdown.sh" -NOTIFYCMD /usr/local/bin/nut-notify.sh -POWERDOWNFLAG /etc/killpower - -NOTIFYFLAG ONLINE SYSLOG+WALL+EXEC -NOTIFYFLAG ONBATT SYSLOG+WALL+EXEC -NOTIFYFLAG LOWBATT SYSLOG+WALL+EXEC -NOTIFYFLAG FSD SYSLOG+WALL+EXEC -NOTIFYFLAG COMMBAD SYSLOG+WALL+EXEC -NOTIFYFLAG COMMOK SYSLOG+WALL+EXEC -``` - ---- - -### B. Outage Notification Engine (`/usr/local/bin/nut-notify.sh`) - -This script runs on the NUT Master (`14.4.4.44`) whenever a power state event triggers. It sends push notifications directly to Gotify (`http://140.44.4.70:8090`) and Home Assistant (`http://14.4.4.20:8123`) without touching NPM or Authentik. - -```bash -#!/bin/bash -# /usr/local/bin/nut-notify.sh -# Emergency power notification script for Network UPS Tools -set -euo pipefail - -EVENT_TYPE="${1:-$NOTIFYTYPE}" -UPS_NAME="${UPSNAME:-snarf-ups}" -TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S') - -GOTIFY_INTERNAL_URL="http://140.44.4.70:8090/message" -GOTIFY_APP_TOKEN="A4uN4fEmErGeNcYtOkEn" # Replace with valid Gotify App Token - -send_gotify() { - local title="$1" - local message="$2" - local priority="$3" - - curl -s -m 5 -X POST "${GOTIFY_INTERNAL_URL}?token=${GOTIFY_APP_TOKEN}" \ - -H "Content-Type: application/json" \ - -d "{\"title\": \"${title}\", \"message\": \"${message}\", \"priority\": ${priority}}" >/dev/null 2>&1 || true -} - -case "$EVENT_TYPE" in - ONBATT) - send_gotify "⚡ Grid Power Lost" "[$TIMESTAMP] Homelab is running on UPS battery (${UPS_NAME}). Staged shutdown countdown started." 8 - logger -t nut-notify "Grid power failure detected - running on UPS battery." - ;; - ONLINE) - send_gotify "✅ Grid Power Restored" "[$TIMESTAMP] Utility AC power restored to ${UPS_NAME}. Battery recharging." 5 - logger -t nut-notify "AC utility power restored." - ;; - LOWBATT) - send_gotify "🚨 UPS Battery Critical" "[$TIMESTAMP] Battery level critical on ${UPS_NAME}! Final Tier 4-6 shutdown initiated." 10 - logger -t nut-notify "CRITICAL: UPS battery low. Enforcing emergency shutdown." - ;; - FSD) - send_gotify "🛑 Forced Shutdown Engaged" "[$TIMESTAMP] NUT Master has commanded immediate power cutoff across all clusters." 10 - logger -t nut-notify "FSD signal received. Halting systems." - ;; - *) - logger -t nut-notify "Unhandled UPS event: ${EVENT_TYPE}" - ;; -esac -``` - ---- - -### C. Orchestrated Tiered Shutdown Script (`/usr/local/bin/homelab_power_shutdown.sh`) - -This script is invoked by `upsmon` on `14.4.4.44` when an outage exceeds the battery threshold: - -```bash -#!/bin/bash -# /usr/local/bin/homelab_power_shutdown.sh -# Staged graceful homelab shutdown orchestrator -set -uo pipefail - -log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] [POWER-SHUTDOWN] $*" | tee -a /var/log/power-shutdown.log; } -SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=3 -o BatchMode=yes" - -log "=== INITIATING HOMELAB TIERED SHUTDOWN ===" - -# ------------------------------------------------------------------ -# TIER 1: Non-Essential Compute & AI Pis -# ------------------------------------------------------------------ -log "Stage 1: Halting AI Raspberry Pi stack (140.44.4.7, 140.44.4.8)..." -ssh $SSH_OPTS root@140.44.4.7 "shutdown -h now" 2>/dev/null || true -ssh $SSH_OPTS root@140.44.4.8 "shutdown -h now" 2>/dev/null || true - -# ------------------------------------------------------------------ -# TIER 2: Docker Swarm NFS Workloads (NPM & Peanut) -# ------------------------------------------------------------------ -log "Stage 2: Stopping Swarm services mounting Synology NFS (NPM, Peanut)..." -ssh $SSH_OPTS root@140.44.4.71 "docker service scale nginx-proxy-manager_app=0 nginx-proxy-manager_backup=0 peanut_peanut=0" 2>/dev/null || true -sleep 5 - -# ------------------------------------------------------------------ -# TIER 3: Harvester Guest VMs & Kubernetes Worker Nodes -# ------------------------------------------------------------------ -log "Stage 3: Shutting down Harvester guest VMs and K3s workers..." -# Gracefully signal Harvester VMs -ssh $SSH_OPTS root@140.44.4.81 "virsh list --name | xargs -r -n1 virsh shutdown" 2>/dev/null || true - -# Stop K3s agent on Aerodrome workers (clean Longhorn PVC detach) -for ip in 140.44.4.22 140.44.4.23 140.44.4.24 140.44.4.25; do - ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true -done - -# Stop K3s agent on Spaceport workers -for ip in 140.44.4.51 140.44.4.52 140.44.4.53 140.44.4.54 140.44.4.55 140.44.4.56; do - ssh $SSH_OPTS root@$ip "systemctl stop k3s-agent && sync && shutdown -h now" 2>/dev/null || true -done -sleep 15 - -# ------------------------------------------------------------------ -# TIER 4: Storage Arrays (Synology DSM Safe Mode) -# ------------------------------------------------------------------ -log "Stage 4: Triggering Synology NAS Safe Mode (14.10.10.71, 14.10.10.72)..." -# Synology enters safe mode automatically via NUT upsd client signal (port 3493) -# Explicit API / SSH command to flush pools if configured -ssh $SSH_OPTS admin@14.10.10.71 "synopoweroff" 2>/dev/null || true -ssh $SSH_OPTS admin@14.10.10.72 "synopoweroff" 2>/dev/null || true -sleep 20 - -# ------------------------------------------------------------------ -# TIER 5: Control Planes & Swarm Managers -# ------------------------------------------------------------------ -log "Stage 5: Halting K3s control planes and Swarm managers..." -# Spaceport Masters (140.44.4.41-44) -for ip in 140.44.4.41 140.44.4.42 140.44.4.43 140.44.4.44; do - ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true -done - -# Aerodrome Masters (140.44.4.11-13) -for ip in 140.44.4.11 140.44.4.12 140.44.4.13; do - ssh $SSH_OPTS root@$ip "k3s etcd-snapshot save && systemctl stop k3s && shutdown -h now" 2>/dev/null || true -done - -# Swarm Managers (140.44.4.71, 72, 74) -for ip in 140.44.4.71 140.44.4.72 140.44.4.74; do - ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true -done - -# Harvester Baremetal Hypervisors (140.44.4.81-83) -for ip in 140.44.4.81 140.44.4.82 140.44.4.83; do - ssh $SSH_OPTS root@$ip "sync && shutdown -h now" 2>/dev/null || true -done - -# ------------------------------------------------------------------ -# TIER 6: Master Node Self-Cutoff (FSD) -# ------------------------------------------------------------------ -log "Stage 6: Executing final local halt and arming killpower..." -touch /etc/killpower -/sbin/shutdown -h +0 -``` - ---- - -### D. Synology NAS Integration (`14.10.10.71` & `14.10.10.72`) - -1. Log into **DSM Console** $\rightarrow$ **Control Panel** $\rightarrow$ **Hardware & Power** $\rightarrow$ **UPS**. -2. Check **Enable UPS Support**. -3. Select UPS Type: **Synology UPS Server / Network NUT Server**. -4. Network UPS server IP address: **`14.4.4.44`**. -5. Set **Time before Synology NAS enters Standby Mode**: `Customize time` $\rightarrow$ `6 minutes` (guaranteeing that Tier 2 Swarm NFS workloads have cleanly scaled down before disk pools unmount). -6. Check **Shut down UPS when the system enters Standby Mode** (disabled, managed centrally by `14.4.4.44`). - ---- - -### E. Peanut Web Dashboard Configuration - -Peanut runs on Docker Swarm (`140.44.4.70:8050`). It connects directly to the NUT master server over internal TCP 3493: - -```yaml -services: - peanut: - image: brandawg93/peanut:latest - ports: - - "8050:8050" - environment: - - TZ=America/New_York - - NUT_HOST=14.4.4.44 - - NUT_PORT=3493 - - NUT_USER=peanut - - NUT_PASSWORD=PEANUT_MONITOR_PASS - volumes: - - peanut_config:/config - -volumes: - peanut_config: - driver: local - driver_opts: - type: none - o: bind - device: /mnt/swarm_shared/legacy_volumes/peanut/ -``` - ---- - -## 5. Automated Power Restoration & Boot Sequence - -When utility grid power returns, the homelab automatically recovers in a **strict bottom-up reverse sequence** to guarantee all storage and network prerequisites are ready before compute services attempt to start. - -```mermaid -flowchart TD - G_On["1. Grid AC Restored\nUPS Battery Reaches >= 30%"] --> P1["2. Phase 1: Core Network & Edge\nUDM Pro (14.44.4.1) & Core PoE Switches\nEstablish VLANs, DHCP, Inter-VLAN Routing"] - P1 --> P2["3. Phase 2: NUT Master\nNode at 14.4.4.44 Boots (BIOS: Always On)\nupsd daemon starts, monitors line quality"] - P2 --> P3["4. Phase 3: Storage Foundation\nSynology SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) Boot\nMount Btrfs pools, start NFS & iSCSI daemons"] - P3 --> P4["5. Phase 4: Control Planes & Hypervisors\n- Harvester spacedock (140.44.4.81-83) boots via IPMI/WOL\n- Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) K3s masters boot\n- etcd clusters re-establish quorum"] - P4 --> P5["6. Phase 5: Storage CSI & Compute Workers\n- Aerodrome & Spaceport worker nodes boot\n- Longhorn CSI engine initializes, verifies volume health, mounts PVCs"] - P5 --> P6["7. Phase 6: Ingress & Workloads\n- Docker Swarm (140.44.4.71-75) boots\n- NPM host-mode container binds 80/443/81 on 140.44.4.74\n- Authentik SSO & Forward Auth come online\n- Applications (Peanut, Gotify, Homebox, etc.) become reachable"] -``` - -### Critical Health Verification Gates: - -1. **Storage Gate (Phase 3 $\rightarrow$ Phase 4):** - - Synology NAS NFS daemons must be responding on `14.10.10.71:2049` before Docker Swarm starts, ensuring NPM does not encounter stale NFS mounts or SQLite locking. -2. **Database Gate (Phase 5 $\rightarrow$ Phase 6):** - - Longhorn volume `data-authentik-postgresql-0` on Aerodrome must reach `Healthy` state before Authentik server pods start. -3. **Ingress Gate (Phase 6):** - - Nginx Proxy Manager on `snshpd4lnxswm04` (`140.44.4.74`) confirms host-mode ports `80` and `443` are bound and forwards traffic to internal VIPs.