# Snarfnet Master Infrastructure Improvement & Resilience Plan

This master architectural plan defines the prioritized roadmap, implementation blueprints, failure-mode analysis, and power resilience strategy for the **snarfnet** homelab. All solutions are designed to execute **locally on self-hosted infrastructure** (`aerodrome`, `spaceport`, `spacedock`, `shipyard`, and Synology NAS) with zero external SaaS dependencies outside of outbound Discord webhook delivery.

---

## 1. Executive Summary & Engineering Prioritization

Following Site Reliability Engineering (SRE) best practices, the eight infrastructure initiatives and the power resilience architecture are ordered into a **6-Phase Dependency Graph**:

1. **Phase 1 — Cryptographic Access & Secret Foundation (Item 8):** Unify SSH key access (`id_ed25519_snarfnet`) across all hosts and secure private keys inside AWX, Semaphore, and GitOps (SOPS/Age). *Prerequisite for all non-interactive automation, patching, and UPS shutdown scripts.*
2. **Phase 2 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3):** Resolve dual-default StorageClasses, clear stuck `Terminating` PVCs, add Longhorn recurring snapshots on `spaceport`, configure Synology NAS (`14.10.10.71`) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. *Prerequisite before upgrading kernels or control planes.*
3. **Phase 3 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.*
4. **Phase 4 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates.
5. **Phase 5 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using `iac-reverse` Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD ApplicationSets (`spaceport` & `aerodrome`) for automated drift remediation.
6. **Phase 6 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment.

```mermaid
flowchart LR
    P1["Phase 1: Unified SSH Key\n& Secret Vault (Item 8)"] --> P2["Phase 2: Storage & NFS\nStabilization (Item 3)"]
    P1 --> P3["Phase 3: Gotify, Discord &\nZabbix Sensors (Items 1, 2, 7)"]
    P2 --> P4["Phase 4: AWX & Rolling\nInfra Upgrades (Items 5, 6)"]
    P3 --> P4
    P4 --> P5["Phase 5: Terraform &\nArgoCD GitOps (Item 4)"]
    P1 --> P6["Phase 6: Power Resilience\n& UPS Shutdown"]
    P2 --> P6
    P3 --> P6
```

---

## 2. Live Infrastructure Baseline Audit (October 30, 2026)

Our automated live probe across the homelab APIs identified the following baseline state that directly informs this plan:

| Platform / Service | Current Version / State | Key Audit Findings & Action Required |
|---|---|---|
| **Aerodrome K3s (`local`)** | Masters (`ctl01-03`): `v1.26.10+k3s1`<br>Workers (`wrk01-07`): `v1.30.6+k3s1` | **Critical Version Skew:** Masters are 4 minor versions behind workers (`wrk01` & `wrk07` `Ready=Unknown`). Dual default StorageClasses (`local-path` & `longhorn`). 2 PVCs stuck `Terminating` (`pihole`, `technitium`). Longhorn `backup-target` is empty (`''`). |
| **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: `v1.31.4+k3s1` (All 10 `Ready`) | Dual default StorageClasses (`local-path` & `longhorn`). 1 PVC stuck `Terminating` (`technitium-longhorn-pvc`). **Zero Longhorn recurring snapshot jobs (`[]`)** and empty `backup-target` (`''`) despite hosting Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. |
| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. 2 active PVCs in `snarfai` (`longhorn-pvc-20tb`, `open-webui-pvc`). Not yet monitored in Zabbix. |
| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); `.73` & `.75` offline | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. Legacy stacks (`gotify`, `homebox`, `peanut`, `pihole`) use `/mnt/swarm_shared` (GlusterFS across `.71-.73`), which is degraded when `.73` is offline. |
| **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 22 hosts registered. **Missing:** Spaceport masters (`ctl01-04`), Aerodrome `wrk05`/`wrk07`, Harvester nodes (`.81-.83`), Synology NAS (`.71`/`.72`), AI Pis (`.7`/`.8`), UDM Pro (`.1`), Home Assistant (`.20`). **Enabled MediaTypes: `[]` (None)**; **Problem Action: Disabled (`status=1`)**. |
| **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | Healthy behind Authentik Forward Auth externally; direct internal VIP (`http://140.44.4.70:8090`) reachable for low-level webhooks. |
| **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Running in `awx` namespace (`awx-web`, `awx-task`, `awx-postgres-15-0`). Ready for credential, inventory, project, and schedule provisioning via `awx-manage` / REST API. |
| **ArgoCD (`argo.snarfnet.net`)** | Version `v3.4.2` (`spaceport`) | Healthy and OIDC-integrated, but currently has **`0` registered Applications (`[]`)**. |

---

## 3. Detailed Phase Execution Plans & Roadblock Analysis

### Phase 1: Single SSH Key & Cryptographic Secret Management (Item 8)

#### Architecture & Implementation Steps
1. **Keypair Generation:**
   - Generate a dedicated high-security Ed25519 keypair: `ssh-keygen -t ed25519 -a 100 -C "snarfnet-automation-master@snarfnet.net" -f id_ed25519_snarfnet`.
2. **Standardized Automation Account (`snarf-admin` / `root`):**
   - Bootstrap all 30+ hosts across VLANs 1, 2, 20, 40, and 50 with a single enrollment script that:
     - Installs the public key into `/root/.ssh/authorized_keys` and `/home/snarf/.ssh/authorized_keys` (plus `rancher` on Harvester nodes and `admin` on Synology DSM).
     - Configures `/etc/sudoers.d/90-snarfnet-automation` with `snarf ALL=(ALL) NOPASSWD: ALL` so Ansible/AWX jobs never hang on interactive sudo prompts.
     - Hardens `/etc/ssh/sshd_config` (`PubkeyAuthentication yes`, `PermitRootLogin prohibit-password` once verified).
3. **Secure Key Storage (Zero Plaintext in Git):**
   - **AWX Credential Vault:** Inject `id_ed25519_snarfnet` as an AWX `Machine` credential (encrypted at rest in `awx-postgres-15` using AWX's `SECRET_KEY`).
   - **Semaphore Key Store:** Register the key in Semaphore (`140.44.4.58:100`) as a secondary runner fallback.
   - **NUT Master (`14.4.4.44`):** Store key at `/root/.ssh/id_ed25519_snarfnet` (`chmod 600`) for `homelab_power_shutdown.sh` to execute passwordless tiered shutdowns.
   - **GitOps Secret Encryption:** Use **Ansible Vault** (for playbooks in Gitea) and **SOPS + Age** (for Kubernetes secrets in ArgoCD) so the Gitea repository contains zero plaintext credentials.

#### Predicted Roadblocks & Mitigations
- **Roadblock 1.1 — Harvester HCI Immutable Root (`/home/rancher`):** Harvester v1.8.0 uses an elemental immutable filesystem; writing to `/etc/ssh` directly can be wiped on upgrade.
  - *Mitigation:* Persist the SSH public key via Harvester's `/oem/99_custom.yaml` cloud-init configuration and the `Harvester Setting -> ssh-keys` CRD in Kubernetes (`spacedock`).
- **Roadblock 1.2 — Synology DSM Home Directory Permissions:** Synology DSM (`14.10.10.71` & `.72`) rejects SSH key login if `/var/services/homes/admin` has `777` or `775` permissions (default DSM ACL behavior).
  - *Mitigation:* Run `chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys && chmod 755 ~` on both NAS appliances during enrollment.
- **Roadblock 1.3 — Currently Offline Nodes (`140.44.4.21`, `.25`/`.27`, `.73`, `.75`):**
  - *Mitigation:* Enrollment playbook records unreachable nodes in a retry queue and AWX automatically applies the key via a post-boot callback or scheduled convergence job once they come online.

---

### Phase 2: Persistent Storage Stabilization & NAS NFS Architecture (Item 3)

#### Architecture & Implementation Steps
1. **Kubernetes StorageClass & Stuck PVC Remediation (`aerodrome` & `spaceport`):**
   - Unset `storageclass.kubernetes.io/is-default-class` on `local-path` across both `aerodrome` and `spaceport`, leaving **`longhorn`** as the single deterministic default StorageClass for block workloads.
   - Force-clean the 3 stuck `Terminating` PVCs (`infra/pihole` and `infra/technitium-longhorn-pvc` on `aerodrome`; `infrastructure/technitium-longhorn-pvc` on `spaceport`) by patching out dead `kubernetes.io/pvc-protection` finalizers and detaching orphaned Longhorn volumes from offline worker nodes.
   - Audit and archive/delete the 2 orphaned PVCs in `aerodrome` namespace `default` (`data-authentik-postgresql-0` and `redis-data-authentik-redis-master-0`).
2. **Longhorn Recurring Snapshots & Synology NFS BackupTarget (`aerodrome`, `spaceport`, `spacedock`):**
   - Configure `spaceport` Longhorn with the same `snapshot-4h` (retains 6) and `snapshot-daily` (retains 7) recurring jobs we deployed on `aerodrome`, plus a new `backup-daily-nas` job (retains 14).
   - Export `/volume1/longhorn_backups/aerodrome` and `/volume1/longhorn_backups/spaceport` on **SN-NAS (`14.10.10.71`)** (or replicate to **SnarfNAS `14.10.10.72`**) via NFSv4.
   - Set Longhorn's `backup-target` setting on both clusters to `nfs://14.10.10.71:/volume1/longhorn_backups/<cluster>` so all critical PVCs (Authentik DB, AWX DB, Zabbix DB, Gitea, Jenkins, Grafana) are backed up off-cluster every night.
3. **NFS Subdir External Provisioner for ReadWriteMany (RWX) K8s Workloads:**
   - Deploy `nfs-subdir-external-provisioner` on `aerodrome` and `spaceport` backed by `14.10.10.71:/volume1/k8s_shared` as StorageClass `synology-nfs`.
   - Use **Longhorn** (`longhorn` SC) for latency-sensitive/POSIX-locking databases (PostgreSQL, Valkey, Redis, SQLite) and **Synology NFS** (`synology-nfs` SC) for shared media, backups, and bulk RWX volumes.
4. **Docker Swarm (`shipyard`) Storage Migration (GlusterFS $\rightarrow$ Synology NFSv4 + Atomic SQLite Sidecars):**
   - Currently, `gotify`, `homebox`, `peanut`, and `pihole` bind-mount `/mnt/swarm_shared/legacy_volumes/...` (a 3-node GlusterFS volume across `.71`, `.72`, `.73`). With `.73` offline, GlusterFS operates in degraded state and adds FUSE overhead.
   - Migrate Swarm persistent volumes to structured NFSv4 shares on `14.10.10.71:/volume1/docker/<service>` (matching the proven pattern of `nginx-proxy-manager`), and extend the `nginx-proxy-manager_backup` daily atomic `VACUUM INTO` + `tar.gz` backup daemon to also snapshot Gotify (`gotify.db`) and Homebox (`homebox.db`).

#### Predicted Roadblocks & Mitigations
- **Roadblock 2.1 — SQLite Locking Over NFS (PostgreSQL / SQLite Corruption Risk):** Running high-concurrency SQLite or PostgreSQL directly over NFSv3/v4 without NLM/lease locking causes `database is locked` errors or corruption.
  - *Mitigation:* Never move K8s PostgreSQL PVCs (`authentik-postgresql`, `awx-postgres`, `zabbix-postgresql`, `gitea-postgresql`) onto raw NFS; keep them on **Longhorn block storage (iSCSI-backed ext4)** and use NFS *strictly* as the Longhorn `backup-target` and for RWX file shares. For Swarm SQLite apps (`npm`, `gotify`, `homebox`), enforce `replicas: 1`, `update_config.order: stop-first`, and NFSv4 (`nfsvers=4.1,hard,timeo=600,retrans=2`).
- **Roadblock 2.2 — Synology NFS Squash Permissions (`root_squash`):** By default, Synology DSM maps container `root` (UID 0) to `nobody` (UID 65534), breaking container startup `chown`.
  - *Mitigation:* Configure Synology NFS rules for `140.44.4.0/25` with `Squash: Map all users to admin` (or `No mapping` for Longhorn backup target) and enable `Allow connections from non-privileged ports (>1024)`.

---

### Phase 3: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7)

#### Architecture & Implementation Steps

```mermaid
flowchart TD
    subgraph Sources ["Homelab Event & Alert Sources"]
        ZBX["Zabbix 7.0.16 Server\n(140.44.4.61 / spaceport)"]
        HA["Home Assistant\n(14.4.4.20 / VLAN 50)"]
        NUT["NUT UPS Master\n(14.4.4.44 / nut-notify.sh)"]
        AWX_N["AWX / ArgoCD / Backup Jobs"]
    end

    subgraph GotifyHub ["Central Local Notification Hub"]
        GTF["Gotify Server v2.9.1\nInternal VIP: http://140.44.4.70:8090\nExternal: https://gotify.snarfnet.net"]
        BRIDGE["gotify-discord-bridge\n(WebSocket Listener Container)"]
    end

    subgraphRecipients ["Important Devices & External Channels"]
        Mobile["Mobile Devices (Android/iOS)\n& Desktop Browser Push"]
        Homepage["Homepage Dashboard Widget"]
        Discord["Discord Server Channels\n(#infra-alerts, #smart-home, #ups-power)"]
    end

    ZBX -- "1. Native Gotify MediaType" --> GTF
    ZBX -- "2. Native Discord Webhook (Direct Fallback)" --> Discord
    HA -- "1. REST notify.gotify" --> GTF
    HA -- "2. Native Discord Integration" --> Discord
    NUT -- "Direct HTTP (Bypasses SSO)" --> GTF
    AWX_N -- "App Token Webhook" --> GTF

    GTF -- "WebSocket Stream (Client Tokens)" --> Mobile
    GTF -- "Client Token API" --> Homepage
    GTF -- "ws://140.44.4.70:8090/stream" --> BRIDGE
    BRIDGE -- "Formatted Discord Embeds" --> Discord
```

1. **Gotify Broadcast Channels & Device Enrollment (Item 1):**
   - Automated Python script (`scripts/setup_gotify_channels.py`) provisions 5 structured Gotify Applications via `http://140.44.4.70:8090/application`:
     - `UPS & Power Resilience` (Default Priority: `9`)
     - `Zabbix Infrastructure Alerts` (Priority mapped `1-10` by severity)
     - `Home Assistant Smart Home` (Default Priority: `5`)
     - `AWX & GitOps Automation` (Default Priority: `4`)
     - `Backups & Storage Health` (Default Priority: `6`)
   - Provision dedicated **Client Tokens** (`http://140.44.4.70:8090/client`) for each important device (Primary Phone, Tablet, Desktop Browser, Homepage Dashboard).
2. **Discord Forwarding Architecture (Item 2):**
   - **Gotify $\rightarrow$ Discord Bridge:** Deploy a lightweight, self-healing container service (`gotify-discord-bridge`) on Shipyard Swarm alongside Gotify. It connects internally to `ws://gotify:80/stream?token=<CLIENT_TOKEN>`, transforms every incoming Gotify message into a rich, color-coded Discord Embed (Red for priority $\ge 8$, Orange for $5-7$, Blue for $\le 4$), and POSTs it to your Discord Webhook URL.
   - **Zabbix $\rightarrow$ Discord & Gotify (Dual-Path Redundancy):** Configure Zabbix 7.0.16 via JSON-RPC API to enable both the native **Discord** webhook media type and a **Gotify** webhook media type, linked to the `Admin` user, and **enable** Trigger Action `'Report problems to Zabbix administrators'` (`status=0`, currently disabled!). If Gotify or Swarm is ever down during an outage, Zabbix still reaches Discord directly from `spaceport`.
   - **Home Assistant $\rightarrow$ Discord & Gotify:** Add a unified `notify.snarfnet_broadcast` group in Home Assistant (`14.4.4.20`) combining `notify.gotify` and the native Discord webhook integration.
3. **Complete Zabbix Sensors, Thresholds, & Templates (Item 7):**
   - **Fix Missing Hosts (100% Coverage):**
     - Deploy `zabbix-agent2` with shared PSK (`PSK_autoregister`) via Ansible across all missing Linux nodes: Spaceport Masters (`140.44.4.41-44`), Aerodrome Workers (`140.44.4.25`, `.27`), AI Pis (`140.44.4.7`, `.8`), and Harvester nodes (`140.44.4.81-83`).
     - Enable SNMPv2c/v3 on **SN-NAS (`14.10.10.71`)**, **SnarfNAS (`14.10.10.72`)**, and **UDM Pro (`14.44.4.1`)**, registering them in Zabbix with templates `Synology DSM by SNMP` and `Ubiquiti AirOS/UniFi by SNMP`.
   - **Custom Sensors & Threshold Alarms:**
     - **NUT UPS Template (`nutserver` `14.4.4.44`):** Custom `UserParameter=ups.metric[*],upsc snarf-ups@127.0.0.1 $1` polling `ups.status` (Trigger: `!= OL` $\rightarrow$ High), `battery.charge` (`< 80%` Warning, `< 50%` High, `< 25%` Disaster), `ups.load` (`> 85%` High), and `battery.runtime` (`< 600s` High).
     - **Kubernetes & Longhorn Health:** Link `Kubernetes nodes by HTTP` and `Kubernetes cluster state by HTTP` to `aerodrome` and `spaceport`; alert on Pod `CrashLoopBackOff`, Node `NotReady` $> 3\text{m}$, and Longhorn volume robustness `Degraded`/`Faulted`.
     - **Hardware & Thermal Thresholds:** CPU temperature $> 75^\circ\text{C}$ (Warning) / $> 82^\circ\text{C}$ (High) for Raspberry Pi 5 nodes in `spaceport`; NVMe/HDD SMART health and Btrfs pool utilization $> 80\%$ (Warning) / $> 90\%$ (High).
     - **TLS Certificate & Web Scenarios:** Zabbix Web Scenarios checking HTTP 200 and SSL certificate expiry ($< 14$ days Warning, $< 7$ days High) across all 35 `*.snarfnet.net` proxy hosts.

#### Predicted Roadblocks & Mitigations
- **Roadblock 3.1 — Authentik Forward Auth Blocking Gotify Mobile App:** The Android/iOS Gotify app uses its own `X-Gotify-Key` header and cannot complete an interactive Authentik OAuth2 browser redirect flow when connecting externally to `https://gotify.snarfnet.net`.
  - *Mitigation:* In NPM Proxy Host #32 (`gotify.snarfnet.net`), bypass Authentik `auth_request` specifically when a valid `X-Gotify-Key` header or `/message` / `/stream` API path is requested, while enforcing Authentik Forward Auth on the web UI root (`/`, `/static/*`)!
- **Roadblock 3.2 — Discord Webhook Rate Limiting During Alert Storms:** During a power outage or switch flap, 25+ hosts firing simultaneously can hit Discord's `30 requests / 60 seconds` webhook rate limit (`HTTP 429`).
  - *Mitigation:* Implement a 2-second batching queue with exponential backoff on `HTTP 429` (`Retry-After`) in the `gotify-discord-bridge` and configure Zabbix host dependency rules (all VLAN 20 hosts depend on UDM Pro `14.44.4.1` ICMP ping) so downstream alerts are suppressed if a core switch or gateway drops.

---

### Phase 4: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6)

#### Architecture & Implementation Steps
1. **AWX Provisioning (`https://awx.snarfnet.net` on `aerodrome`):**
   - Execute an automated bootstrap script via `kubectl exec` into `awx-task` (or AWX REST API `/api/v2/`) to create:
     - **Organization:** `Snarfnet Homelab`
     - **Credentials:**
       1. `Snarfnet Master SSH Key` (`Machine` credential with `id_ed25519_snarfnet` + passwordless sudo)
       2. `Gitea Automation Token` (`Source Control` credential for pulling playbooks from `http://140.44.4.62:90/snarf/SnarfCode.git`)
       3. `Authentik & Zabbix API Vault` (`Vault` credential)
     - **Project:** `Snarfnet Infrastructure Playbooks` synced from local **Gitea** (`git.snarfnet.net`).
     - **Inventory (`Snarfnet Master Inventory`):** Structured into groups:
       - `k3s_aerodrome_masters` (`140.44.4.11-13`) & `k3s_aerodrome_workers` (`140.44.4.21-27`)
       - `k3s_spaceport_masters` (`140.44.4.41-44`) & `k3s_spaceport_workers` (`140.44.4.51-56`)
       - `harvester_spacedock` (`140.44.4.81-83`)
       - `swarm_shipyard` (`140.44.4.71-75`)
       - `ai_pi_cluster` (`140.44.4.7-8`)
       - `smart_home_nodes` (`14.4.4.20`, `14.4.4.44`, `14.44.4.14`)
       - `synology_nas` (`14.10.10.71-72`)
2. **Remediation of `aerodrome` K3s Control Plane Version Skew (Priority Upgrade):**
   - `aerodrome` masters (`140.44.4.11-13`) are at `v1.26.10+k3s1` while workers are at `v1.30.6+k3s1`. Kubernetes strictly forbids skipping minor versions on control-plane upgrades.
   - Create `ansible/playbooks/upgrade_k3s_aerodrome.yml` with `serial: 1`:
     - Step 1: Take an `etcd-snapshot save pre-upgrade-1.26` on `.11`.
     - Step 2: Sequentially step all 3 masters through each minor release channel: `v1.27.16+k3s1` $\rightarrow$ `v1.28.15+k3s1` $\rightarrow$ `v1.29.10+k3s1` $\rightarrow$ `v1.30.6+k3s1` (and optionally align with `spaceport` at `v1.31.4+k3s1`), waiting for `kubectl get nodes` `Ready` and etcd quorum health (`etcdctl endpoint health`) between every single node and minor step.
3. **AWX Job Templates & Scheduled Maintenance Windows:**
   - **Template 1 — `Weekly OS & Security Patching` (Cron: Sundays 03:00 AM):**
     - Runs `apt-get update && apt-get dist-upgrade -y` with `serial: 1`.
     - Checks `/var/run/reboot-required`. If a kernel reboot is needed on a K8s or Swarm node, it runs `kubectl drain <node> --ignore-daemonsets --delete-emptydir-data` (or `docker node update --availability drain`), reboots, waits for SSH + kubelet `Ready` + Longhorn volume health (`Healthy`), and runs `kubectl uncordon <node>`.
   - **Template 2 — `Monthly K3s & Container Stack Update` (Cron: 1st Sunday 04:00 AM):**
     - Updates K3s binaries and pulls latest pinned container digests for Shipyard Swarm stacks, followed by HTTP 200 health verification across all 35 NPM endpoints.
   - **Template 3 — `Daily Drift & Agent Health Audit` (Cron: Daily 06:00 AM):**
     - Verifies `zabbix-agent2`, `upsmon` slave daemons, NFS mounts, and SSH `authorized_keys` across all nodes, sending a summary report to Gotify/Discord.

#### Predicted Roadblocks & Mitigations
- **Roadblock 4.1 — Longhorn Replica Rebuilding During Rolling Node Reboots:** If Ansible reboots node B immediately after node A finishes rebooting, Longhorn volumes may still be in `Degraded` (rebuilding replicas) on node A; rebooting node B can take the last healthy replica offline!
  - *Mitigation:* Add an explicit Ansible `uri` / `command` pre-task gate in the rolling reboot playbook that queries the Longhorn API (`volumes` list) and pauses the next node's drain until **zero attached volumes are in `degraded` state**.
- **Roadblock 4.2 — AWX Running Inside the Cluster It Is Upgrading (`aerodrome`):** When AWX upgrades or reboots the `aerodrome` node currently hosting the `awx-task` pod, the active AWX job itself will be killed mid-execution!
  - *Mitigation:* Pin `awx-task` and `awx-web` to `sninfr2lnxwrk06` during `aerodrome` master upgrades, and execute `aerodrome` worker reboots either via a Workflow Job Template that upgrades `wrk06` last, or trigger `aerodrome` node reboots from **Semaphore** (`140.44.4.58` on `spaceport`) so the orchestrator lives on a separate physical cluster!

---

### Phase 5: Terraform Plans & ArgoCD GitOps Integrity (Item 4)

#### Architecture & Implementation Steps
1. **Local GitOps Repository Structure on Gitea (`https://git.snarfnet.net/snarf/homelab-gitops`):**
   ```text
   homelab-gitops/
   ├── terraform/
   │   ├── authentik/       # Providers, Applications, Outposts, Conditional MFA Policies
   │   ├── harvester/       # Spacedock VM definitions, VLAN networks, SSH keys
   │   ├── shipyard-swarm/  # Docker Swarm stacks (NPM, Gotify, Homebox, Peanut, Pihole)
   │   └── synology/        # Shared folders, NFS rules
   └── argocd/
       ├── bootstrap/       # Root App-of-Apps for spaceport & aerodrome
       ├── spaceport/       # Gitea, Zabbix, Jenkins, Semaphore, Coder, Homepage, Traefik
       └── aerodrome/       # Authentik, AWX, Grafana, Uptime Kuma, Longhorn settings
   ```
2. **Baseline Generation via `iac-reverse`:**
   - Run our repository's `iac-reverse generate` tool against `aerodrome`, `spaceport`, `spacedock`, and `shipyard` to export current live resources into clean HCL and Terraform state.
   - Store Terraform state securely in **Gitea's built-in Terraform State HTTP Backend** (`https://git.snarfnet.net/api/packages/snarf/terraform/state/...`) or on Synology NAS (`14.10.10.71`), backed up daily.
3. **ArgoCD Multi-Cluster Registration (`https://argo.snarfnet.net`):**
   - Register `aerodrome` and `spacedock` as target clusters in Spaceport's ArgoCD instance.
   - Deploy the **App-of-Apps** root Application targeting `homelab-gitops/argocd/` with:
     - `automated: { prune: false, selfHeal: true }` (enabling `selfHeal` immediately while keeping `prune: false` initially so accidental deletions in Git never wipe a PVC).
     - ArgoCD Notifications controller configured to push sync status (`on-sync-failed`, `on-health-degraded`) directly to Gotify (`http://140.44.4.70:8090`).

#### Predicted Roadblocks & Mitigations
- **Roadblock 5.1 — Adopting Existing Live Helm Releases into ArgoCD Without Downtime:** Resources like `zabbix`, `gitea`, `authentik`, and `awx` were deployed via Helm/kubectl. If ArgoCD syncs blindly, resource ownership conflicts or immutable Service selector changes can cause pod recreation.
  - *Mitigation:* Use `ServerSideApply=true` and `Replace=false` sync options in ArgoCD, and test every Application in **Diff / Manual Sync mode** (`kubectl diff`) until ArgoCD reports `Synced / Healthy` with zero destructive diffs before flipping `selfHeal: true`.
- **Roadblock 5.2 — Circular Dependency if Gitea or ArgoCD Goes Down:**
  - *Mitigation:* Mirror the `homelab-gitops` repository automatically to a bare Git repository on Synology NAS (`/volume1/git_mirror/homelab-gitops.git`) and your workstation (`SnarfCode`), ensuring Terraform and `kubectl apply` can always run directly from local disk if `spaceport` is offline.

---

### Phase 6: Power Resilience & UPS Automation Plan (Updated Core Pillar)

#### 6.1 Core Power Architecture & IP Assignments

The power resilience architecture uses a centralized **NUT Master Server** (`14.4.4.44`) that polls the physical UPS and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.

| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol |
|---|---|---|---|---|
| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master` and holds `/root/.ssh/id_ed25519_snarfnet`. |
| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; stopped in Tier 2 prior to NAS Safe Mode. |
| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth & Conditional MFA. Runs on Longhorn storage in Aerodrome. |
| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. |
| **Emergency Push & Discord** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server + `gotify-discord-bridge`. **Internal outage alerts use `http://140.44.4.70:8090`** (+ direct Discord webhook fallback) to bypass Authentik during shutdown. |
| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and non-essential load shedding. |
| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix 7.0.16 polling `14.4.4.44` via Zabbix Agent 2 `upsc` UserParameters. |
| **Primary Storage 1 & 2** | `sn-nas` / `snarfnas` | `14.10.10.71` / `.72` | VLAN 40 (Storage) | Synology DSM UPS clients polling `14.4.4.44`. Host NFSv4 shares and Longhorn BackupTargets. |

#### 6.2 Tiered Graceful Shutdown Hierarchy (6 Stages)

```text
+--------------------------------------------------------------------------------------------------+
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT                                |
| - Broadcasts emergency alert to Gotify (http://140.44.4.70:8090), Discord Webhook, & HA Push     |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads                                 |
| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped      |
| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted via SSH key            |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM, Gotify, Homebox, Peanut)    |
| - Send final pre-shutdown notification to Gotify & Discord                                       |
| - Scale down Swarm NFS services (`nginx-proxy-manager_app`, `backup`, `peanut`, `homebox`)       |
| - Flushes open SQLite databases and cleanly releases all NFS file locks on 14.10.10.71           |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach              |
| - Harvester guest VMs (140.44.4.81-83) gracefully shut down via `virsh shutdown` / API           |
| - Stop K3s agent on Aerodrome (140.44.4.21-27) & Spaceport (140.44.4.51-56) worker nodes         |
| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes     |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode)                            |
| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode (`synopoweroff` / NUT)       |
| - Unmounts Btrfs volume pools, terminates NFS/iSCSI daemons, parks drive heads                   |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors                             |
| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Masters save etcd snapshot & halt      |
| - Remaining Swarm Managers (140.44.4.71-75) & Harvester Hypervisors (140.44.4.81-83) halt        |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut                                |
| - NUT Master (14.4.4.44) sends direct Discord webhook ("Final FSD Cut") & issues `upsdrvctl`     |
| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops  |
+--------------------------------------------------------------------------------------------------+
```

#### 6.3 Automated Power Restoration & Boot Verification Gates
1. **Stage 1 (Grid AC Restored, Battery $\ge 30\%$):** UPS re-energizes load outlets.
2. **Stage 2 (Network Core):** UDM Pro (`14.44.4.1`) & Core switches boot, establishing VLANs 1–60 and inter-VLAN routing.
3. **Stage 3 (NUT Master):** `14.4.4.44` boots (`BIOS: Restore on AC Power Loss = Always On`), starts `upsd`, and sends `ONLINE` alert to Discord/Gotify.
4. **Stage 4 (Storage Gate — Must Complete Before Compute):** Synology NAS arrays (`14.10.10.71` & `14.10.10.72`) boot, mount Btrfs pools, and expose NFS (`:2049`).
5. **Stage 5 (Control Planes & Hypervisors):** Harvester (`140.44.4.81-83`), K3s Masters (`aerodrome` & `spaceport`), and Swarm Managers (`140.44.4.71-75`) boot and establish etcd/Raft quorum.
6. **Stage 6 (Compute Workers & Ingress):** K3s workers boot, Longhorn reattaches PVCs (`data-authentik-postgresql-0`, `awx-postgres`, `zabbix-postgresql`), and NPM on `snshpd4lnxswm04` (`140.44.4.74`) binds host ports `80/443/81`.

---

## 4. Access & Inputs Required to Begin Automation

We already have verified administrative API access to:
- **Authentik API** (`https://authentik.snarfnet.net`)
- **Rancher & Kubernetes APIs** (`aerodrome`, `spaceport`, and `spacedock` Harvester)
- **Portainer & Docker Swarm API** (`https://shipyard.snarfnet.net`)
- **Zabbix 7.0.16 API** (`https://zabbix.snarfnet.net/api_jsonrpc.php`)

To automate the remaining items end-to-end, we will only need the following when we reach each corresponding task:

1. **Discord Webhook URL(s) (For Phase 3 — Item 2):**
   - One Discord Webhook URL (or separate URLs if you prefer separate channels like `#infra-alerts`, `#smart-home`, and `#ups-power`).
2. **Gotify Admin Credentials (For Phase 3 — Item 1):**
   - Current Gotify admin username/password (or we can inspect/bootstrap a token directly via the Gotify SQLite database on Shipyard Swarm).
3. **Home Assistant Long-Lived Access Token or SSH (For Phase 3 — Item 2):**
   - To inject the Gotify/Discord notification configuration into `14.4.4.20` (`configuration.yaml` / `automations.yaml`).
4. **Host SSH / Synology DSM Credentials (For Phase 1 — Item 8 & Phase 2 — Item 3):**
   - Confirmation of whether the existing `snarf` / `Katzir476!` credential (from `setup_storage_cluster.sh`) is valid across the Debian/RPi nodes, plus SSH/DSM credentials for Synology NAS (`14.10.10.71` / `14.10.10.72`) and Harvester nodes (`140.44.4.81-83`) so we can enroll the unified `id_ed25519_snarfnet` key.