475 lines
45 KiB
Markdown
475 lines
45 KiB
Markdown
# Snarfnet Master Infrastructure Improvement & Resilience Plan
|
||
|
||
This master architectural plan defines the prioritized roadmap, implementation blueprints, failure-mode analysis, and power resilience strategy for the **snarfnet** homelab. All solutions are designed to execute **locally on self-hosted infrastructure** (`aerodrome`, `spaceport`, `spacedock`, `shipyard`, and Synology NAS) with zero external SaaS dependencies outside of outbound Discord webhook delivery.
|
||
|
||
---
|
||
|
||
## 1. Executive Summary & Engineering Prioritization
|
||
|
||
Following Site Reliability Engineering (SRE) best practices, the nine infrastructure initiatives and the power resilience architecture are ordered into a **7-Phase Dependency Graph**:
|
||
|
||
1. **Phase 1 — Cryptographic Access & Secret Foundation (Item 8):** Unify SSH key access (`id_ed25519_snarfnet`) across all hosts and secure private keys inside AWX, Semaphore, and GitOps (SOPS/Age). *Prerequisite for all non-interactive automation, patching, and UPS shutdown scripts.* **[100% Completed & Verified]**
|
||
2. **Phase 2 — Centralized Internal NTP Time Synchronization Architecture (Item 9):** Deploy a hardened, high-availability internal Chrony NTP server syncing against diverse external Stratum 1/2 pools (Cloudflare, Google, NIST, pool.ntp.org) with local stratum fallback. Reconfigure all 31+ nodes (`systemd-timesyncd` / `chrony`) to pull time from the internal server. *Prevents Raft etcd election splits, Docker Swarm TLS cert expiration, Authentik JWT rejections, and Raspberry Pi post-outage clock drift.* **[100% Completed & Verified]**
|
||
3. **Phase 3 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3):** Resolve dual-default StorageClasses, clear stuck `Terminating` PVCs, add Longhorn recurring snapshots on `spaceport`, configure Synology NAS (`14.10.10.71`) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. *Prerequisite before upgrading kernels or control planes.* **[100% Completed & Verified]**
|
||
4. **Phase 4 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.* **[100% Completed & Verified]**
|
||
5. **Phase 5 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates. **[100% Completed & Verified]**
|
||
6. **Phase 6 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD App-of-Apps (`spaceport` & `aerodrome`) for automated drift remediation. **[100% Completed & Verified]**
|
||
7. **Phase 7 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment. **[In Progress / Next]**
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
P1["Phase 1: Unified SSH Key\n& Secret Vault (Item 8)"] --> P2["Phase 2: Central Internal\nNTP Sync Server (Item 9)"]
|
||
P2 --> P3["Phase 3: Storage & NFS\nStabilization (Item 3)"]
|
||
P2 --> P4["Phase 4: Gotify, Discord &\nZabbix Sensors (Items 1, 2, 7)"]
|
||
P3 --> P5["Phase 5: AWX & Rolling\nInfra Upgrades (Items 5, 6)"]
|
||
P4 --> P5
|
||
P5 --> P6["Phase 6: Terraform &\nArgoCD GitOps (Item 4)"]
|
||
P1 --> P7["Phase 7: Power Resilience\n& UPS Shutdown"]
|
||
P2 --> P7
|
||
P3 --> P7
|
||
P4 --> P7
|
||
```
|
||
|
||
---
|
||
|
||
## 2. Live Infrastructure Baseline Audit (October 30, 2026)
|
||
|
||
Our automated live probe across the homelab APIs identified the following baseline state that directly informs this plan:
|
||
|
||
| Platform / Service | Current Version / State | Key Audit Findings & Action Required |
|
||
|---|---|---|
|
||
| **Aerodrome K3s (`local`)** | All 10 Nodes (`ctl01-03` & `wrk01-07`): **`v1.30.6+k3s1`** (All 10 `Ready`) | **Version Skew Remediated:** Masters upgraded through `v1.27.16` $\rightarrow$ `v1.28.15` $\rightarrow$ `v1.29.10` $\rightarrow$ `v1.30.6+k3s1` with zero workload downtime. Single default SC (`longhorn`). Stuck PVCs cleared. Synology NAS NFS daily backups active. |
|
||
| **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: `v1.31.4+k3s1` (All 10 `Ready`) | Dual default StorageClasses (`local-path` & `longhorn`). 1 PVC stuck `Terminating` (`technitium-longhorn-pvc`). **Zero Longhorn recurring snapshot jobs (`[]`)** and empty `backup-target` (`''`) despite hosting Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. |
|
||
| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. 2 active PVCs in `snarfai` (`longhorn-pvc-20tb`, `open-webui-pvc`). Not yet monitored in Zabbix. |
|
||
| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); all 5 online and `Ready` | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. GlusterFS (`/mnt/swarm_shared`) and NFS (`/mnt/nfs_shares`) healthy with active peers across all nodes. Hardened with `usb_max_current_enable=1`, `tune2fs -e continue`, and `nofail`. |
|
||
| **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 22 hosts registered. **Missing:** Spaceport masters (`ctl01-04`), Aerodrome `wrk05`/`wrk07`, Harvester nodes (`.81-.83`), Synology NAS (`.71`/`.72`), AI Pis (`.7`/`.8`), UDM Pro (`.1`), Home Assistant (`.20`). **Enabled MediaTypes: `[]` (None)**; **Problem Action: Disabled (`status=1`)**. |
|
||
| **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | Healthy behind Authentik Forward Auth externally; direct internal VIP (`http://140.44.4.70:8090`) reachable for low-level webhooks. |
|
||
| **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Running in `awx` namespace (`awx-web`, `awx-task`, `awx-postgres-15-0`). Ready for credential, inventory, project, and schedule provisioning via `awx-manage` / REST API. |
|
||
| **ArgoCD (`argo.snarfnet.net`)** | Version `v3.4.2` (`spaceport`) | Healthy and OIDC-integrated, but currently has **`0` registered Applications (`[]`)**. |
|
||
|
||
---
|
||
|
||
## 3. Detailed Phase Execution Plans & Roadblock Analysis
|
||
|
||
### Phase 1: Single SSH Key & Cryptographic Secret Management (Item 8)
|
||
|
||
#### Architecture & Implementation Steps
|
||
1. **Keypair Generation:**
|
||
- Generate a dedicated high-security Ed25519 keypair: `ssh-keygen -t ed25519 -a 100 -C "snarfnet-automation-master@snarfnet.net" -f id_ed25519_snarfnet`.
|
||
2. **Standardized Automation Account (`snarf-admin` / `root`):**
|
||
- Bootstrap all 28 target hosts across VLANs 1, 20, 40, and 50 (skipping `spacedock` Harvester nodes `140.44.4.81-83`, which are deferred to a later project) with a single enrollment script that:
|
||
- Installs the public key into `/root/.ssh/authorized_keys` and `/home/snarf/.ssh/authorized_keys` (plus `admin` on Synology DSM).
|
||
- Configures `/etc/sudoers.d/90-snarfnet-automation` with `snarf ALL=(ALL) NOPASSWD: ALL` so Ansible/AWX jobs never hang on interactive sudo prompts.
|
||
- Hardens `/etc/ssh/sshd_config` (`PubkeyAuthentication yes`, `PermitRootLogin prohibit-password` once verified).
|
||
3. **Secure Key Storage (Zero Plaintext in Git):**
|
||
- **AWX Credential Vault:** Inject `id_ed25519_snarfnet` as an AWX `Machine` credential (encrypted at rest in `awx-postgres-15` using AWX's `SECRET_KEY`).
|
||
- **Semaphore Key Store:** Register the key in Semaphore (`140.44.4.58:100`) as a secondary runner fallback.
|
||
- **NUT Master (`14.4.4.44`):** Store key at `/root/.ssh/id_ed25519_snarfnet` (`chmod 600`) for `homelab_power_shutdown.sh` to execute passwordless tiered shutdowns.
|
||
- **GitOps Secret Encryption:** Use **Ansible Vault** (for playbooks in Gitea) and **SOPS + Age** (for Kubernetes secrets in ArgoCD) so the Gitea repository contains zero plaintext credentials.
|
||
|
||
#### Predicted Roadblocks & Mitigations
|
||
- **Roadblock 1.1 — Harvester HCI Immutable Root (`/home/rancher`):**
|
||
- *Status:* Skipped / Deferred. Harvester nodes (`140.44.4.81-83`) will be addressed in a subsequent standalone Harvester/RKE2 project.
|
||
- **Roadblock 1.2 — Synology DSM Home Directory Permissions:** Synology DSM (`14.10.10.71` & `.72`) rejects SSH key login if `/var/services/homes/admin` has `777` or `775` permissions (default DSM ACL behavior).
|
||
- *Mitigation:* Run `chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys && chmod 755 ~` on both NAS appliances during enrollment.
|
||
- **Roadblock 1.3 — Currently Offline Nodes (`140.44.4.21`, `.25`/`.27`, `.73`, `.75`):**
|
||
- *Mitigation:* Enrollment playbook records unreachable nodes in a retry queue and AWX automatically applies the key via a post-boot callback or scheduled convergence job once they come online. *(Resolved: All nodes are now 100% online and ready for key enrollment).*
|
||
|
||
---
|
||
|
||
### Phase 2: Centralized Internal NTP Time Synchronization Architecture (Item 9) — [100% COMPLETED & VERIFIED]
|
||
|
||
> [!NOTE]
|
||
> **Implementation Status: 100% Complete & Verified (November 1, 2026)**
|
||
> - Authoritative Chrony servers deployed, peering, and active on **`aerodrome-ctl01` (`140.44.4.11` - Primary)** and **`nutserver` (`14.4.4.44` - Secondary)**.
|
||
> - Exterior stratum 1/2 pools (Google, Cloudflare, NIST, Pool.ntp.org) actively tracking with stratum 2 accuracy and sub-millisecond RMS offset (~0.5ms on Primary, ~48μs on Secondary).
|
||
> - UDP port 123 verified open and responsive across VLAN boundaries.
|
||
> - Client NTP configuration (`systemd-timesyncd` pointing to `140.44.4.11` and `14.4.4.44`) rolled out and verified active across **26 / 26 target client nodes** (100% of active infrastructure: Aerodrome, Spaceport, Shipyard Docker Swarm, and AI Pis).
|
||
> - Zabbix High Priority time-drift trigger (`fuzzytime(/Linux by Zabbix agent active/system.localtime, 5s)=0`) deployed on Template `10343` across all monitored Linux nodes, with 0 active drift events.
|
||
|
||
#### Architecture & Implementation Steps
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
subgraph ExteriorNTP ["Public Exterior Stratum 1 & 2 NTP Sources"]
|
||
NIST["NIST Atomic Clock\n(time.nist.gov)"]
|
||
CF["Cloudflare Anycast NTP\n(time.cloudflare.com)"]
|
||
GOOG["Google Public NTP\n(time.google.com)"]
|
||
POOL["NTP Pool Project\n(0.us.pool.ntp.org / 1.us.pool.ntp.org)"]
|
||
end
|
||
|
||
subgraph InternalNTP ["Central Internal Time Authority (Chrony Stratum 2)"]
|
||
NTP_PRI["Primary Internal NTP Server\naerodrome-ctl01 (140.44.4.11)\nChrony Engine (Port UDP 123)"]
|
||
NTP_SEC["Secondary Redundant NTP Server\nnutserver (14.4.4.44)\nChrony Engine (Port UDP 123)"]
|
||
end
|
||
|
||
subgraph InternalClients ["Homelab Infrastructure Clients (28 Target Hosts)"]
|
||
K8S_A["aerodrome K3s (10 Nodes)\n140.44.4.11-13, 21-27"]
|
||
K8S_S["spaceport K3s (10 Nodes)\n140.44.4.41-44, 51-56"]
|
||
SWM["shipyard Swarm (5 Nodes)\n140.44.4.71-75"]
|
||
NAS["Synology NAS (2 Appliances)\n14.10.10.71, 72"]
|
||
SMART["Smart Home & AI Pis (4 Hosts)\n14.4.4.20, 14.4.4.44, 140.44.4.7-8"]
|
||
end
|
||
|
||
NIST --> NTP_PRI
|
||
CF --> NTP_PRI
|
||
GOOG --> NTP_PRI
|
||
POOL --> NTP_PRI
|
||
|
||
NIST --> NTP_SEC
|
||
CF --> NTP_SEC
|
||
GOOG --> NTP_SEC
|
||
POOL --> NTP_SEC
|
||
|
||
NTP_PRI <-->|"Symmetric Active Peering"| NTP_SEC
|
||
|
||
NTP_PRI -->|"Sub-millisecond NTP Unicast (Port 123)"| K8S_A
|
||
NTP_PRI -->|"Sub-millisecond NTP Unicast (Port 123)"| K8S_S
|
||
NTP_PRI -->|"Sub-millisecond NTP Unicast (Port 123)"| SWM
|
||
NTP_PRI -->|"Sub-millisecond NTP Unicast (Port 123)"| NAS
|
||
NTP_PRI -->|"Sub-millisecond NTP Unicast (Port 123)"| SMART
|
||
|
||
NTP_SEC -.->|"Automatic Redundant Fallback"| InternalClients
|
||
```
|
||
|
||
1. **Strategic Necessity for Homelab High Availability:**
|
||
- **Raft & etcd Consensus:** Both K3s clusters (`aerodrome` and `spaceport`), Docker Swarm (`shipyard`), and Harvester RKE2 require Raft heartbeat stability; clock skew $> 500\text{ms}$ triggers spurious leader elections, split-brain states, and quorum loss.
|
||
- **Docker Swarm & Kubernetes TLS Node Certificates:** Docker Swarm and K3s node TLS certificates have finite validity periods (e.g. 90-day Swarm certificates). A desynchronized clock causes immediate certificate expiration errors and daemon failures upon boot.
|
||
- **Authentik SSO & Token Lifetimes:** Authentik OAuth2, OIDC, and SAML assertions rely on strict `nbf` (not before) and `exp` (expires at) timestamp validation; clock skew $> 60\text{s}$ leads to total login authentication rejection across all homelab dashboards.
|
||
- **Lack of Hardware Real-Time Clocks (RTC) on Raspberry Pis:** All 24 Raspberry Pi nodes lack battery-backed hardware RTCs. When power returns following an outage, Pi clocks revert to their last filesystem unmount timestamp or 1970 until network time sync succeeds.
|
||
|
||
2. **Central Internal Chrony Server Deployment:**
|
||
- **Primary Server:** Deployed on `aerodrome-ctl01` (`140.44.4.11`).
|
||
- **Secondary Server:** Deployed on `nutserver` (`14.4.4.44`).
|
||
- **Diverse Exterior Stratum 1 & 2 Sources:**
|
||
```chrony
|
||
server time.cloudflare.com iburst minpoll 4 maxpoll 8
|
||
server time.google.com iburst minpoll 4 maxpoll 8
|
||
server time.nist.gov iburst minpoll 4 maxpoll 8
|
||
server 0.us.pool.ntp.org iburst minpoll 4 maxpoll 8
|
||
server 1.us.pool.ntp.org iburst minpoll 4 maxpoll 8
|
||
```
|
||
- **Orphan Mode & Local Stratum Fallback:**
|
||
```chrony
|
||
local stratum 10
|
||
allow 140.44.4.0/24
|
||
allow 14.4.4.0/24
|
||
allow 14.10.10.0/24
|
||
allow 140.44.44.0/24
|
||
allow 14.44.4.0/24
|
||
```
|
||
*Benefit:* If an extended storm or ISP fiber cut takes the WAN offline, the internal Chrony servers maintain local stratum 10 authority, locking all homelab hosts to identical drift-free time with zero inter-cluster divergence.
|
||
|
||
3. **Homelab-Wide Client Rollout (25 / 26 Active Client Nodes Verified):**
|
||
- **Debian / DietPi / Raspberry Pi OS Clients (`systemd-timesyncd`):**
|
||
Standardized `/etc/systemd/timesyncd.conf`:
|
||
```ini
|
||
[Time]
|
||
NTP=140.44.4.11 14.4.4.44
|
||
FallbackNTP=time.cloudflare.com time.google.com 0.us.pool.ntp.org 1.us.pool.ntp.org
|
||
PollIntervalMinSec=32
|
||
PollIntervalMaxSec=512
|
||
```
|
||
- **DietPi Continuous Daemon Mode:** Configured `CONFIG_NTP_MODE=4` in `/boot/dietpi.txt` and unmasked/enabled `systemd-timesyncd.service` on all Spaceport nodes.
|
||
- **Synology DSM (`14.10.10.71` & `14.10.10.72`):**
|
||
Targeted to synchronize time from internal primary `140.44.4.11`.
|
||
|
||
4. **Observability & Clock Drift Alerting in Zabbix:**
|
||
- Template **`Linux by Zabbix agent active` (ID 10343)** updated with High Severity Trigger `Linux: System time is out of sync (> 5s)` (ID `26201`):
|
||
`fuzzytime(/Linux by Zabbix agent active/system.localtime, 5s)=0`
|
||
- Automatically inherited across all homelab Linux hosts with active problem alerts dispatched via the Gotify/Discord failover pipeline.
|
||
|
||
#### Predicted Roadblocks & Mitigations
|
||
- **Roadblock 2.1 — Negative Time Jumps on Raspberry Pi Boot (systemd-timesyncd vs fake-hwclock):**
|
||
If `fake-hwclock` restores a timestamp from the future (e.g. from an unclean shutdown), `systemd-timesyncd` can refuse to jump backward without a hard step, or a backward step can corrupt etcd WAL logs.
|
||
- *Mitigation:* Ensure `systemd-timesyncd` is configured with `timedatectl set-ntp true` and that the systemd service starts before clustering engines (K3s, Docker Swarm), stepping time forward cleanly at boot.
|
||
- **Roadblock 2.2 — VLAN Segmentation Blocking UDP Port 123:**
|
||
Nodes reside across VLAN 1, 2, 20, 40, and 50. If UniFi firewall rules drop inter-VLAN UDP 123, cross-VLAN nodes (e.g. Synology NAS on VLAN 40, Home Assistant on VLAN 50) fail to sync.
|
||
- *Mitigation:* Enable a stateful UniFi firewall rule allowing `LAN IN: Any $\rightarrow$ UDP 123` to the internal NTP server IP, or serve NTP natively on each VLAN gateway interface on the UDM-Pro (`140.44.4.1`, `14.4.4.1`, `14.10.10.1`, `140.44.44.1`).
|
||
|
||
---
|
||
|
||
### Phase 3: Persistent Storage Stabilization & NAS NFS Architecture (Item 3) — [100% Completed & Verified]
|
||
|
||
#### Architecture & Implementation Steps (Executed & Verified)
|
||
1. **Kubernetes StorageClass & Stuck PVC Remediation (`aerodrome` & `spaceport`):**
|
||
- Removed `is-default-class` annotation from `local-path` across both `aerodrome` and `spaceport`. **`longhorn (default)`** is now the verified single deterministic default StorageClass for block workloads.
|
||
- Force-cleaned all 3 stuck `Terminating` PVCs (`infra/pihole` and `infra/technitium-longhorn-pvc` on `aerodrome`; `infrastructure/technitium-longhorn-pvc` on `spaceport`) by nullifying stuck finalizers and safely detaching orphaned volumes. Stuck PVCs across clusters: **0**.
|
||
- Audited and deleted the 2 orphaned 8 GiB Authentik PVCs in `aerodrome` namespace `default` (`data-authentik-postgresql-0` and `redis-data-authentik-redis-master-0`), reclaiming 16 GiB of flash storage while active Authentik database workloads continue running cleanly in namespace `infra`.
|
||
2. **Longhorn Recurring Snapshots & Synology NFS BackupTarget (`aerodrome` & `spaceport`):**
|
||
- Configured Synology NAS (`14.10.10.71:/volume1/kubernetes`) exports with dedicated subdirectories:
|
||
- `nfs://14.10.10.71:/volume1/kubernetes/longhorn/aerodrome`
|
||
- `nfs://14.10.10.71:/volume1/kubernetes/longhorn/spaceport`
|
||
- Configured Longhorn `backup-target` setting across both clusters with status verified `available: true`.
|
||
- Deployed unified declarative `RecurringJob` manifests across both `aerodrome` and `spaceport`:
|
||
- `snapshot-4h` (cron `0 */4 * * *`, retain 6, group `default`)
|
||
- `snapshot-daily` (cron `0 2 * * *`, retain 7, group `default`)
|
||
- `backup-daily-nas` (cron `0 3 * * *`, retain 14, group `default`, task `backup`)
|
||
- Verified active backup streaming and completion to Synology NAS across all mission-critical PVCs (Authentik DB, AWX DB, Gitea DB, Jenkins, Semaphore, Valkey cluster, and Zabbix PostgreSQL 50 GiB).
|
||
3. **Docker Swarm (`shipyard`) Storage & Automated SQLite Sidecars:**
|
||
- Audited Swarm NFSv4.1 automounts on `14.10.10.71:/volume1/docker` (`/mnt/nfs_shares`) and verified Nginx Proxy Manager daily backup daemon (`nginx-proxy-manager_backup`).
|
||
- Implemented and activated `/usr/local/bin/swarm_sqlite_backup.py` backed by systemd service `swarm-sqlite-backup.service` and active timer `swarm-sqlite-backup.timer` on manager node `140.44.4.71`.
|
||
- Verified atomic online `VACUUM INTO` snapshots + compressed `.tar.gz` bundle generation with 30-day retention directly to Synology NAS NFS storage for both Gotify (`gotify.db` + discord bridge configs) and Homebox (`homebox.db`).
|
||
|
||
#### Predicted Roadblocks & Mitigations
|
||
- **Roadblock 2.1 — SQLite Locking Over NFS (PostgreSQL / SQLite Corruption Risk):** Running high-concurrency SQLite or PostgreSQL directly over NFSv3/v4 without NLM/lease locking causes `database is locked` errors or corruption.
|
||
- *Mitigation:* Never move K8s PostgreSQL PVCs (`authentik-postgresql`, `awx-postgres`, `zabbix-postgresql`, `gitea-postgresql`) onto raw NFS; keep them on **Longhorn block storage (iSCSI-backed ext4)** and use NFS *strictly* as the Longhorn `backup-target` and for RWX file shares. For Swarm SQLite apps (`npm`, `gotify`, `homebox`), enforce `replicas: 1`, `update_config.order: stop-first`, and NFSv4 (`nfsvers=4.1,hard,timeo=600,retrans=2`).
|
||
- **Roadblock 2.2 — Synology NFS Squash Permissions (`root_squash`):** By default, Synology DSM maps container `root` (UID 0) to `nobody` (UID 65534), breaking container startup `chown`.
|
||
- *Mitigation:* Configure Synology NFS rules for `140.44.4.0/25` with `Squash: Map all users to admin` (or `No mapping` for Longhorn backup target) and enable `Allow connections from non-privileged ports (>1024)`.
|
||
|
||
---
|
||
|
||
### Phase 4: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7) — [100% Completed & Verified]
|
||
|
||
#### Architecture & Implementation Steps
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
subgraph Sources ["Homelab Event & Alert Sources"]
|
||
ZBX["Zabbix 7.0.16 Server\n(140.44.4.61 / spaceport)"]
|
||
HA["Home Assistant\n(14.4.4.20 / VLAN 50)"]
|
||
NUT["NUT UPS Master\n(14.4.4.44 / nut-notify.sh)"]
|
||
AWX_N["AWX / ArgoCD / Backup Jobs"]
|
||
end
|
||
|
||
subgraph GotifyHub ["Central Local Notification Hub"]
|
||
GTF["Gotify Server v2.9.1\nInternal VIP: http://140.44.4.70:8090\nExternal: https://gotify.snarfnet.net"]
|
||
BRIDGE["gotify-discord-bridge\n(WebSocket Listener Container)"]
|
||
end
|
||
|
||
subgraphRecipients ["Important Devices & External Channels"]
|
||
Mobile["Mobile Devices (Android/iOS)\n& Desktop Browser Push"]
|
||
Homepage["Homepage Dashboard Widget"]
|
||
Discord["Discord Server Channels\n(#infra-alerts, #smart-home, #ups-power)"]
|
||
end
|
||
|
||
ZBX -- "1. Native Gotify MediaType" --> GTF
|
||
ZBX -- "2. Native Discord Webhook (Direct Fallback)" --> Discord
|
||
HA -- "1. REST notify.gotify" --> GTF
|
||
HA -- "2. Native Discord Integration" --> Discord
|
||
NUT -- "Direct HTTP (Bypasses SSO)" --> GTF
|
||
AWX_N -- "App Token Webhook" --> GTF
|
||
|
||
GTF -- "WebSocket Stream (Client Tokens)" --> Mobile
|
||
GTF -- "Client Token API" --> Homepage
|
||
GTF -- "ws://140.44.4.70:8090/stream" --> BRIDGE
|
||
BRIDGE -- "Formatted Discord Embeds" --> Discord
|
||
```
|
||
|
||
1. **Gotify Broadcast Channels & Device Enrollment (Item 1):**
|
||
- Automated Python script (`scripts/setup_gotify_channels.py`) provisions 5 structured Gotify Applications via `http://140.44.4.70:8090/application`:
|
||
- `UPS & Power Resilience` (Default Priority: `9`)
|
||
- `Zabbix Infrastructure Alerts` (Priority mapped `1-10` by severity)
|
||
- `Home Assistant Smart Home` (Default Priority: `5`)
|
||
- `AWX & GitOps Automation` (Default Priority: `4`)
|
||
- `Backups & Storage Health` (Default Priority: `6`)
|
||
- Provision dedicated **Client Tokens** (`http://140.44.4.70:8090/client`) for each important device (Primary Phone, Tablet, Desktop Browser, Homepage Dashboard).
|
||
2. **Discord Forwarding Architecture (Item 2):**
|
||
- **Dedicated Discord Channel Webhooks (Verified):**
|
||
- `ups-power`: `https://discord.com/api/webhooks/1566076118535114813/2e_fCJeHb-mF8VwerzOORGRa-QUDFuwN6R2Gfz-MGUiIOudA0PiXGZSyd19PNFG-vR7D`
|
||
- `infra-alerts`: `https://discord.com/api/webhooks/1566076777963593772/wKN30N0ZXVJKGP-prZ1_P9cp_u7Oakmh8R4bVh0qr8RFpyeGcJZrHfPQBJxtfzQqb4dn`
|
||
- `home-assistant`: `https://discord.com/api/webhooks/1566077255837294684/4x_sTqDeEHfIeGccEutOSvPKAnCV6Fyr7uAuECOt8M8QZcOvL2XZ0GVATW7EmcSyUW33`
|
||
- **Gotify $\rightarrow$ Discord Multi-Channel Bridge:** Deploy a lightweight, self-healing container service (`gotify-discord-bridge`) on Shipyard Swarm alongside Gotify. It connects internally to `ws://gotify:80/stream?token=<CLIENT_TOKEN>`, maps each Gotify `appid` to its dedicated Discord channel webhook (`UPS & Power Resilience` $\rightarrow$ `ups-power`, `Home Assistant Smart Home` $\rightarrow$ `home-assistant`, and `Zabbix` / `AWX` / `Backups` $\rightarrow$ `infra-alerts`), transforms every incoming Gotify message into a rich, color-coded Discord Embed (Red for priority $\ge 8$, Orange for $5-7$, Blue for $\le 4$), and POSTs it to the mapped Discord Webhook URL.
|
||
- **Zabbix $\rightarrow$ Discord & Gotify (Dual-Path Redundancy):** Configure Zabbix 7.0.16 via JSON-RPC API to enable both the native **Discord** webhook media type (pointing to `infra-alerts`, plus `ups-power` for UPS host group triggers) and a **Gotify** webhook media type, linked to the `Admin` user, and **enable** Trigger Action `'Report problems to Zabbix administrators'` (`status=0`, currently disabled!). If Gotify or Swarm is ever down during an outage, Zabbix still reaches Discord directly from `spaceport`.
|
||
- **Home Assistant $\leftrightarrow$ Gotify & Discord (`14.4.4.20:8123` — Configured & Verified):**
|
||
- Provisioned reusable script **`script.snarfnet_broadcast`** ([scripts/setup_homeassistant_notifications.py](file:///c:/Users/Snarf/OneDrive/Desktop/code/SnarfCode/scripts/setup_homeassistant_notifications.py)) which records structured broadcast entries via `logbook.log`, creates a persistent UI notification, and pushes directly to `notify.mobile_app_pixel_10_pro_fold` and `notify.mobile_app_pixel_9_pro_fold`.
|
||
- `gotify-discord-bridge` polls Home Assistant's `/api/logbook` API using the Long-Lived Access Token, evaluates any Jinja2 state templates via `/api/template`, publishes new notifications to Gotify Application #3 (`Home Assistant Smart Home`), and routes them to the `#home-assistant` Discord webhook.
|
||
- Provisioned Home Assistant automations:
|
||
1. **`Snarfnet UPS Power & Exhaust Protocol` (`1793453000001`):** Monitors `sensor.server_closet_status_data` (`OL`/`OB`/`LB`), broadcasts live UPS telemetry (`battery_charge`, `load`, `input_voltage`, `server_room_temperature`), and automatically sets `fan.server_closet_switch` to `100%` when running on battery.
|
||
2. **`Snarfnet Server Room Thermal Guard` (`1793453000002`):** Engages `fan.server_closet_switch` at `100%` and broadcasts a high-temperature warning if `sensor.server_room_temperature_temperature` exceeds `82°F` for 2 minutes.
|
||
3. **Bambu H2D 3D Printer Automations (`1788367985433`, `1788368709567`, `1788370272217`, `1788370321997`):** Enhanced to broadcast `Started`, `Completed`, `Stalled`, and `Failed` events to Gotify and `#home-assistant` on Discord alongside existing camera snapshots.
|
||
3. **Complete Zabbix Sensors, Thresholds, & Templates (Item 7):**
|
||
- **Fix Missing Hosts (100% Coverage):**
|
||
- Deploy `zabbix-agent2` with shared PSK (`PSK_autoregister`) via Ansible across all missing Linux nodes: Spaceport Masters (`140.44.4.41-44`), Aerodrome Workers (`140.44.4.25`, `.27`), AI Pis (`140.44.4.7`, `.8`), and Harvester nodes (`140.44.4.81-83`).
|
||
- Enable SNMPv2c/v3 on **SN-NAS (`14.10.10.71`)**, **SnarfNAS (`14.10.10.72`)**, and **UDM Pro (`14.44.4.1`)**, registering them in Zabbix with templates `Synology DSM by SNMP` and `Ubiquiti AirOS/UniFi by SNMP`.
|
||
- **Custom Sensors & Threshold Alarms:**
|
||
- **NUT UPS Template (`nutserver` `14.4.4.44`):** Custom `UserParameter=ups.metric[*],upsc snarf-ups@127.0.0.1 $1` polling `ups.status` (Trigger: `!= OL` $\rightarrow$ High), `battery.charge` (`< 80%` Warning, `< 50%` High, `< 25%` Disaster), `ups.load` (`> 85%` High), and `battery.runtime` (`< 600s` High).
|
||
- **Kubernetes & Longhorn Health:** Link `Kubernetes nodes by HTTP` and `Kubernetes cluster state by HTTP` to `aerodrome` and `spaceport`; alert on Pod `CrashLoopBackOff`, Node `NotReady` $> 3\text{m}$, and Longhorn volume robustness `Degraded`/`Faulted`.
|
||
- **Hardware & Thermal Thresholds:** CPU temperature $> 75^\circ\text{C}$ (Warning) / $> 82^\circ\text{C}$ (High) for Raspberry Pi 5 nodes in `spaceport`; NVMe/HDD SMART health and Btrfs pool utilization $> 80\%$ (Warning) / $> 90\%$ (High).
|
||
- **TLS Certificate & Web Scenarios:** Zabbix Web Scenarios checking HTTP 200 and SSL certificate expiry ($< 14$ days Warning, $< 7$ days High) across all 35 `*.snarfnet.net` proxy hosts.
|
||
|
||
#### Predicted Roadblocks & Mitigations
|
||
- **Roadblock 3.1 — Authentik Forward Auth Blocking Gotify Mobile App:** The Android/iOS Gotify app uses its own `X-Gotify-Key` header and cannot complete an interactive Authentik OAuth2 browser redirect flow when connecting externally to `https://gotify.snarfnet.net`.
|
||
- *Mitigation:* In NPM Proxy Host #32 (`gotify.snarfnet.net`), bypass Authentik `auth_request` specifically when a valid `X-Gotify-Key` header or `/message` / `/stream` API path is requested, while enforcing Authentik Forward Auth on the web UI root (`/`, `/static/*`)!
|
||
- **Roadblock 3.2 — Discord Webhook Rate Limiting During Alert Storms:** During a power outage or switch flap, 25+ hosts firing simultaneously can hit Discord's `30 requests / 60 seconds` webhook rate limit (`HTTP 429`).
|
||
- *Mitigation:* Implement a 2-second batching queue with exponential backoff on `HTTP 429` (`Retry-After`) in the `gotify-discord-bridge` and configure Zabbix host dependency rules (all VLAN 20 hosts depend on UDM Pro `14.44.4.1` ICMP ping) so downstream alerts are suppressed if a core switch or gateway drops.
|
||
|
||
---
|
||
|
||
### Phase 5: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6) — [100% Completed & Verified]
|
||
|
||
#### Architecture & Implementation Steps
|
||
1. **AWX Provisioning (`https://awx.snarfnet.net` on `aerodrome`):**
|
||
- Executed automated provisioning via Django ORM inside `awx-task`:
|
||
- **Organization:** `Snarfnet Homelab` (id=2)
|
||
- **Credentials:**
|
||
1. `Snarfnet Master SSH Key (root)` (`Machine` credential with `id_ed25519_snarfnet` for root access across clusters)
|
||
2. `Snarfnet Master SSH Key (snarf)` (`Machine` credential with `id_ed25519_snarfnet` + passwordless sudo for Swarm, AI, and NUT)
|
||
- **Project:** `Snarfnet Infrastructure Playbooks` (id=8) synced from local **Gitea** (`https://git.snarfnet.net/Snarf/SnarfCode.git`).
|
||
- **Master Inventory (`Snarfnet Master Inventory`, id=3):** Structured into 8 functional groups covering all 28 active homelab nodes:
|
||
- `k3s_aerodrome_masters` (`140.44.4.11-13`) & `k3s_aerodrome_workers` (`140.44.4.21-27`)
|
||
- `k3s_spaceport_masters` (`140.44.4.41-44`) & `k3s_spaceport_workers` (`140.44.4.51-56`)
|
||
- `swarm_shipyard` (`140.44.4.71-75`)
|
||
- `ai_pi_cluster` (`140.44.4.7-8`)
|
||
- `smart_home_nodes` (`nutserver` `14.4.4.44`)
|
||
- `synology_nas` (`14.10.10.71-72`)
|
||
- **Job Templates Provisioned & Verified:**
|
||
- `Daily Drift & Agent Health Audit` ([ansible/playbooks/daily_drift_audit.yml](file:///c:/Users/Snarf/OneDrive/Desktop/code/SnarfCode/ansible/playbooks/daily_drift_audit.yml)): Audits time sync daemon, Zabbix agent 2, and master SSH key enrollment. Executed with 100% green recap across 28/28 active nodes (Job 106).
|
||
- `Aerodrome K3s Pre-Upgrade Check` ([ansible/playbooks/k3s_aerodrome_precheck.yml](file:///c:/Users/Snarf/OneDrive/Desktop/code/SnarfCode/ansible/playbooks/k3s_aerodrome_precheck.yml)): Verifies K3s systemd unit and captures on-demand etcd snapshots. Executed and verified (Job 111).
|
||
- `Aerodrome K3s Control Plane Upgrade` ([ansible/playbooks/upgrade_k3s_aerodrome.yml](file:///c:/Users/Snarf/OneDrive/Desktop/code/SnarfCode/ansible/playbooks/upgrade_k3s_aerodrome.yml)): Atomic direct binary replacement with etcd quorum check and node `Ready` gates (`serial: 1`).
|
||
- `Weekly OS & Security Patching` ([ansible/playbooks/weekly_os_patching.yml](file:///c:/Users/Snarf/OneDrive/Desktop/code/SnarfCode/ansible/playbooks/weekly_os_patching.yml)): Staged rolling package updates with service health verification.
|
||
2. **Remediation of `aerodrome` K3s Control Plane Version Skew [100% Completed & Verified]:**
|
||
- Remediation completed sequentially across minor versions to prevent etcd schema corruption:
|
||
- **Pre-Upgrade etcd Snapshot:** Saved `clean-pre-upgrade-sninfr2lnxctl01-1793646879` (109 MB) on `.11`.
|
||
- **Step 1 (`v1.27.16+k3s1`):** Executed AWX Job 118 (`serial: 1`). Upgraded `ctl01`, `ctl02`, and `ctl03`. All 3 masters reported `Ready`.
|
||
- **Step 2 (`v1.28.15+k3s1`):** Executed AWX Job 121 (`serial: 1`). Upgraded `ctl01`, `ctl02`, and `ctl03`. All 3 masters reported `Ready`.
|
||
- **Step 3 (`v1.29.10+k3s1`):** Executed AWX Job 124 (`serial: 1`). Upgraded `ctl01`, `ctl02`, and `ctl03`. All 3 masters reported `Ready`.
|
||
- **Step 4 (`v1.30.6+k3s1`):** Executed AWX Job 127 (`serial: 1`). Upgraded `ctl01`, `ctl02`, and `ctl03`. All 3 masters reported `Ready`.
|
||
- **Outcome:** Control plane version skew is **100% resolved**. All 10 Aerodrome nodes (ctl01–03 and wrk01–07) are unified on `v1.30.6+k3s1`.
|
||
- **Workload Verification:** All mission-critical pods (Authentik server/worker/LDAP, AWX web/task/postgres, Longhorn attached volumes) verified `Running` and `healthy`.
|
||
|
||
#### Predicted Roadblocks & Mitigations
|
||
- **Roadblock 4.1 — Longhorn Replica Rebuilding During Rolling Node Reboots:** If Ansible reboots node B immediately after node A finishes rebooting, Longhorn volumes may still be in `Degraded` (rebuilding replicas) on node A; rebooting node B can take the last healthy replica offline!
|
||
- *Mitigation:* Add an explicit Ansible `uri` / `command` pre-task gate in the rolling reboot playbook that queries the Longhorn API (`volumes` list) and pauses the next node's drain until **zero attached volumes are in `degraded` state**.
|
||
- **Roadblock 4.2 — AWX Running Inside the Cluster It Is Upgrading (`aerodrome`):** When AWX upgrades or reboots the `aerodrome` node currently hosting the `awx-task` pod, the active AWX job itself will be killed mid-execution!
|
||
- *Mitigation:* Pin `awx-task` and `awx-web` to `sninfr2lnxwrk06` during `aerodrome` master upgrades, and execute `aerodrome` worker reboots either via a Workflow Job Template that upgrades `wrk06` last, or trigger `aerodrome` node reboots from **Semaphore** (`140.44.4.58` on `spaceport`) so the orchestrator lives on a separate physical cluster!
|
||
|
||
---
|
||
|
||
### Phase 6: Terraform Plans & ArgoCD GitOps Integrity (Item 4) — [100% Completed & Verified]
|
||
|
||
#### Architecture & Implementation Steps (Executed & Verified)
|
||
1. **Local GitOps Repository Created on Gitea (`https://git.snarfnet.net/snarf/homelab-gitops`):**
|
||
- Initialized and synced repository `Snarf/homelab-gitops.git` on local Gitea:
|
||
```text
|
||
homelab-gitops/
|
||
├── terraform/
|
||
│ ├── authentik/ # OAuth2 providers, proxy ForwardAuth, apps, policies
|
||
│ ├── harvester/ # Declarative VLAN networks, VM templates, SSH keys
|
||
│ ├── shipyard-swarm/ # Docker Swarm stacks (NPM, Gotify, Homebox, Peanut)
|
||
│ └── synology/ # Shared folders (kubernetes, docker, backups, media) & NFS rules
|
||
├── argocd/
|
||
│ ├── bootstrap/ # Root App-of-Apps, Project, spaceport & aerodrome child apps
|
||
│ ├── spaceport/ # Applications hosted on spaceport (Gitea, Zabbix, Jenkins, etc.)
|
||
│ └── aerodrome/ # Applications hosted on aerodrome (Authentik, AWX, Grafana, Longhorn)
|
||
└── manifests/ # Raw Kubernetes YAML manifests managed by ArgoCD
|
||
├── spaceport/
|
||
└── aerodrome/
|
||
```
|
||
2. **Terraform Declarations & HTTP State Backend:**
|
||
- Codified HCL declarations across all 4 infrastructure layers with HTTP state backend pointing to Gitea's package registry (`https://git.snarfnet.net/api/packages/snarf/terraform/state/<module>`).
|
||
3. **ArgoCD Multi-Cluster Registration (`https://argo.snarfnet.net`):**
|
||
- Enrolled `aerodrome` (`https://140.44.4.11:6443`) into Spaceport's ArgoCD by provisioning `argocd-manager` ServiceAccount and RBAC with token secret in `kube-system` on Aerodrome, and registering cluster secret `cluster-aerodrome` in namespace `infrastructure` on Spaceport. Shard 0 verified connected and healthy.
|
||
4. **ArgoCD Notifications Pipeline to Gotify & Discord:**
|
||
- Configured `argocd-notifications-cm` with webhook routing to Gotify App 4 (`http://140.44.4.70:8090/message?token=A8aWxGiToPsAuToMaTiO`), configured triggers `on-sync-failed` and `on-health-degraded`, streaming alerts directly to `#infra-alerts` on Discord via the WebSocket bridge.
|
||
5. **Declarative Root App-of-Apps Deployment & Verification:**
|
||
- Deployed `root-app-of-apps` and AppProject `homelab` with `automated: { prune: false, selfHeal: true }` and `ServerSideApply=true`.
|
||
- Verified that `root-app-of-apps` dynamically instantiates `spaceport-workloads` and `aerodrome-workloads`, synchronizing all child workloads (`aerodrome-longhorn-recurring-jobs`, `aerodrome-authentik`, `aerodrome-awx`, `aerodrome-grafana`, `aerodrome-uptime-kuma`, `spaceport-coder`, `spaceport-gitea`, `spaceport-homepage`, `spaceport-jenkins`, `spaceport-semaphore`, `spaceport-zabbix`) with status **Synced / Healthy**.
|
||
|
||
#### Predicted Roadblocks & Mitigations
|
||
- **Roadblock 5.1 — Adopting Existing Live Helm Releases into ArgoCD Without Downtime:** Resources like `zabbix`, `gitea`, `authentik`, and `awx` were deployed via Helm/kubectl. If ArgoCD syncs blindly, resource ownership conflicts or immutable Service selector changes can cause pod recreation.
|
||
- *Mitigation:* Use `ServerSideApply=true` and `Replace=false` sync options in ArgoCD, and test every Application in **Diff / Manual Sync mode** (`kubectl diff`) until ArgoCD reports `Synced / Healthy` with zero destructive diffs before flipping `selfHeal: true`.
|
||
- **Roadblock 5.2 — Circular Dependency if Gitea or ArgoCD Goes Down:**
|
||
- *Mitigation:* Mirror the `homelab-gitops` repository automatically to a bare Git repository on Synology NAS (`/volume1/git_mirror/homelab-gitops.git`) and your workstation (`SnarfCode`), ensuring Terraform and `kubectl apply` can always run directly from local disk if `spaceport` is offline.
|
||
|
||
---
|
||
|
||
### Phase 7: Power Resilience & UPS Automation Plan (Updated Core Pillar)
|
||
|
||
#### 6.1 Core Power Architecture & IP Assignments
|
||
|
||
The power resilience architecture uses a centralized **NUT Master Server** (`14.4.4.44`) that polls the physical UPS and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
|
||
|
||
| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol |
|
||
|---|---|---|---|---|
|
||
| **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master` and holds `/root/.ssh/id_ed25519_snarfnet`. |
|
||
| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; stopped in Tier 2 prior to NAS Safe Mode. |
|
||
| **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth & Conditional MFA. Runs on Longhorn storage in Aerodrome. |
|
||
| **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. |
|
||
| **Emergency Push & Discord** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server + `gotify-discord-bridge`. **Internal outage alerts use `http://140.44.4.70:8090`** (+ direct Discord webhook fallback) to bypass Authentik during shutdown. |
|
||
| **Smart Home Automation** | `homeassistant.snarfnet.net` | `14.4.4.20` | VLAN 50 (Smart Home) | Home Assistant integration monitoring `14.4.4.44:3493` for state changes, critical push alerts, and non-essential load shedding. |
|
||
| **Centralized Monitoring** | `zabbix.snarfnet.net` | `140.44.4.61` | VLAN 20 (Spaceport K3s) | Zabbix 7.0.16 polling `14.4.4.44` via Zabbix Agent 2 `upsc` UserParameters. |
|
||
| **Primary Storage 1 & 2** | `sn-nas` / `snarfnas` | `14.10.10.71` / `.72` | VLAN 40 (Storage) | Synology DSM UPS clients polling `14.4.4.44`. Host NFSv4 shares and Longhorn BackupTargets. |
|
||
|
||
#### 6.2 Tiered Graceful Shutdown Hierarchy (6 Stages)
|
||
|
||
```text
|
||
+--------------------------------------------------------------------------------------------------+
|
||
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT |
|
||
| - Broadcasts emergency alert to Gotify (http://140.44.4.70:8090), Discord Webhook, & HA Push |
|
||
+--------------------------------------------------------------------------------------------------+
|
||
|
|
||
v
|
||
+--------------------------------------------------------------------------------------------------+
|
||
| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads |
|
||
| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped |
|
||
| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted via SSH key |
|
||
+--------------------------------------------------------------------------------------------------+
|
||
|
|
||
v
|
||
+--------------------------------------------------------------------------------------------------+
|
||
| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM, Gotify, Homebox, Peanut) |
|
||
| - Send final pre-shutdown notification to Gotify & Discord |
|
||
| - Scale down Swarm NFS services (`nginx-proxy-manager_app`, `backup`, `peanut`, `homebox`) |
|
||
| - Flushes open SQLite databases and cleanly releases all NFS file locks on 14.10.10.71 |
|
||
+--------------------------------------------------------------------------------------------------+
|
||
|
|
||
v
|
||
+--------------------------------------------------------------------------------------------------+
|
||
| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach |
|
||
| - Harvester guest VMs (140.44.4.81-83) gracefully shut down via `virsh shutdown` / API |
|
||
| - Stop K3s agent on Aerodrome (140.44.4.21-27) & Spaceport (140.44.4.51-56) worker nodes |
|
||
| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes |
|
||
+--------------------------------------------------------------------------------------------------+
|
||
|
|
||
v
|
||
+--------------------------------------------------------------------------------------------------+
|
||
| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode) |
|
||
| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode (`synopoweroff` / NUT) |
|
||
| - Unmounts Btrfs volume pools, terminates NFS/iSCSI daemons, parks drive heads |
|
||
+--------------------------------------------------------------------------------------------------+
|
||
|
|
||
v
|
||
+--------------------------------------------------------------------------------------------------+
|
||
| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors |
|
||
| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Masters save etcd snapshot & halt |
|
||
| - Remaining Swarm Managers (140.44.4.71-75) & Harvester Hypervisors (140.44.4.81-83) halt |
|
||
+--------------------------------------------------------------------------------------------------+
|
||
|
|
||
v
|
||
+--------------------------------------------------------------------------------------------------+
|
||
| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut |
|
||
| - NUT Master (14.4.4.44) sends direct Discord webhook ("Final FSD Cut") & issues `upsdrvctl` |
|
||
| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops |
|
||
+--------------------------------------------------------------------------------------------------+
|
||
```
|
||
|
||
#### 6.3 Automated Power Restoration & Boot Verification Gates
|
||
1. **Stage 1 (Grid AC Restored, Battery $\ge 30\%$):** UPS re-energizes load outlets.
|
||
2. **Stage 2 (Network Core):** UDM Pro (`14.44.4.1`) & Core switches boot, establishing VLANs 1–60 and inter-VLAN routing.
|
||
3. **Stage 3 (NUT Master):** `14.4.4.44` boots (`BIOS: Restore on AC Power Loss = Always On`), starts `upsd`, and sends `ONLINE` alert to Discord/Gotify.
|
||
4. **Stage 4 (Storage Gate — Must Complete Before Compute):** Synology NAS arrays (`14.10.10.71` & `14.10.10.72`) boot, mount Btrfs pools, and expose NFS (`:2049`).
|
||
5. **Stage 5 (Control Planes & Hypervisors):** Harvester (`140.44.4.81-83`), K3s Masters (`aerodrome` & `spaceport`), and Swarm Managers (`140.44.4.71-75`) boot and establish etcd/Raft quorum.
|
||
6. **Stage 6 (Compute Workers & Ingress):** K3s workers boot, Longhorn reattaches PVCs (`data-authentik-postgresql-0`, `awx-postgres`, `zabbix-postgresql`), and NPM on `snshpd4lnxswm04` (`140.44.4.74`) binds host ports `80/443/81`.
|
||
|
||
---
|
||
|
||
## 4. Access & Inputs Required to Begin Automation
|
||
|
||
We already have verified administrative API access to:
|
||
- **Authentik API** (`https://authentik.snarfnet.net`)
|
||
- **Rancher & Kubernetes APIs** (`aerodrome`, `spaceport`, and `spacedock` Harvester)
|
||
- **Portainer & Docker Swarm API** (`https://shipyard.snarfnet.net`)
|
||
- **Zabbix 7.0.16 API** (`https://zabbix.snarfnet.net/api_jsonrpc.php`)
|
||
|
||
To automate the remaining items end-to-end, we will only need the following when we reach each corresponding task:
|
||
|
||
1. **Discord Webhook URL(s) (For Phase 3 — Item 2):**
|
||
- One Discord Webhook URL (or separate URLs if you prefer separate channels like `#infra-alerts`, `#smart-home`, and `#ups-power`).
|
||
2. **Gotify Admin Credentials (For Phase 3 — Item 1):**
|
||
- Current Gotify admin username/password (or we can inspect/bootstrap a token directly via the Gotify SQLite database on Shipyard Swarm).
|
||
3. **Home Assistant Long-Lived Access Token or SSH (For Phase 3 — Item 2):**
|
||
- To inject the Gotify/Discord notification configuration into `14.4.4.20` (`configuration.yaml` / `automations.yaml`).
|
||
4. **Host SSH / Synology DSM Credentials (For Phase 1 — Item 8 & Phase 2 — Item 3):**
|
||
- Confirmation of whether the existing `snarf` / `Katzir476!` credential (from `setup_storage_cluster.sh`) is valid across the Debian/RPi nodes, plus SSH/DSM credentials for Synology NAS (`14.10.10.71` / `14.10.10.72`) and Harvester nodes (`140.44.4.81-83`) so we can enroll the unified `id_ed25519_snarfnet` key.
|