@@ -107,7 +107,9 @@ All Kubernetes resources are hosted on the **Compute VLAN (VLAN 20 — `140.44.4
External WAN traffic hitting `174.106.252.96` is forwarded to internal hosts as follows:
1.**Nginx Reverse Proxy:** Ports `80` & `443` forward to **`140.44.4.74`** (Legacy NPM/Pi-hole VM in Swarm).
1.**Nginx Reverse Proxy:** Ports `80` & `443` forward to **`140.44.4.74`** (`snshpd4lnxswm04` in Docker Swarm).
**Real Client IP Preservation:* NPM runs with `mode: host` (`PublishMode: host`) pinned to `snshpd4lnxswm04`. This eliminates Docker Swarm overlay ingress SNAT (which previously masked all clients as `10.0.0.6`), ensuring `X-Forwarded-For` and `X-Real-IP` preserve the true WAN or LAN client IP.
**Conditional Authentik MFA:* Authentik's `default-authentication-flow-authenticator-validate-stage` evaluates the real client IP. Connections from internal subnets (`14.4.4.0/24`, `14.44.4.0/24`, `140.44.4.0/24`, `14.10.10.0/24`) bypass MFA for frictionless homelab usage, while external WAN connections strictly enforce MFA.
3.**Zabbix Trapper:** Ports `10051` & `10052` forward to **`140.44.4.51`** (Klipper ServiceLB on worker `sninfr0lnxwrk01` $\rightarrow$ routes to primary Zabbix server pod in `spaceport`).
@@ -129,7 +131,6 @@ This table records how subdomains on `*.snarfnet.net` route to internal IP addre
# Snarfnet Master Infrastructure Improvement & Resilience Plan
This master architectural plan defines the prioritized roadmap, implementation blueprints, failure-mode analysis, and power resilience strategy for the **snarfnet** homelab. All solutions are designed to execute **locally on self-hosted infrastructure** (`aerodrome`, `spaceport`, `spacedock`, `shipyard`, and Synology NAS) with zero external SaaS dependencies outside of outbound Discord webhook delivery.
Following Site Reliability Engineering (SRE) best practices, the eight infrastructure initiatives and the power resilience architecture are ordered into a **6-Phase Dependency Graph**:
1.**Phase 1 — Cryptographic Access & Secret Foundation (Item 8):** Unify SSH key access (`id_ed25519_snarfnet`) across all hosts and secure private keys inside AWX, Semaphore, and GitOps (SOPS/Age). *Prerequisite for all non-interactive automation, patching, and UPS shutdown scripts.*
2.**Phase 2 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3):** Resolve dual-default StorageClasses, clear stuck `Terminating` PVCs, add Longhorn recurring snapshots on `spaceport`, configure Synology NAS (`14.10.10.71`) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. *Prerequisite before upgrading kernels or control planes.*
3.**Phase 3 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.*
4.**Phase 4 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates.
5.**Phase 5 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using `iac-reverse` Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD ApplicationSets (`spaceport` & `aerodrome`) for automated drift remediation.
6.**Phase 6 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment.
| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. 2 active PVCs in `snarfai` (`longhorn-pvc-20tb`, `open-webui-pvc`). Not yet monitored in Zabbix. |
| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); `.73` & `.75` offline | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. Legacy stacks (`gotify`, `homebox`, `peanut`, `pihole`) use `/mnt/swarm_shared` (GlusterFS across `.71-.73`), which is degraded when `.73` is offline. |
| **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 22 hosts registered. **Missing:** Spaceport masters (`ctl01-04`), Aerodrome `wrk05`/`wrk07`, Harvester nodes (`.81-.83`), Synology NAS (`.71`/`.72`), AI Pis (`.7`/`.8`), UDM Pro (`.1`), Home Assistant (`.20`). **Enabled MediaTypes: `[]` (None)**; **Problem Action: Disabled (`status=1`)**. |
| **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | Healthy behind Authentik Forward Auth externally; direct internal VIP (`http://140.44.4.70:8090`) reachable for low-level webhooks. |
| **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Running in `awx` namespace (`awx-web`, `awx-task`, `awx-postgres-15-0`). Ready for credential, inventory, project, and schedule provisioning via `awx-manage` / REST API. |
| **ArgoCD (`argo.snarfnet.net`)** | Version `v3.4.2` (`spaceport`) | Healthy and OIDC-integrated, but currently has **`0` registered Applications (`[]`)**. |
- Bootstrap all 30+ hosts across VLANs 1, 2, 20, 40, and 50 with a single enrollment script that:
- Installs the public key into `/root/.ssh/authorized_keys` and `/home/snarf/.ssh/authorized_keys` (plus `rancher` on Harvester nodes and `admin` on Synology DSM).
- Configures `/etc/sudoers.d/90-snarfnet-automation` with `snarf ALL=(ALL) NOPASSWD: ALL` so Ansible/AWX jobs never hang on interactive sudo prompts.
- Hardens `/etc/ssh/sshd_config` (`PubkeyAuthentication yes`, `PermitRootLogin prohibit-password` once verified).
3.**Secure Key Storage (Zero Plaintext in Git):**
- **AWX Credential Vault:** Inject `id_ed25519_snarfnet` as an AWX `Machine` credential (encrypted at rest in `awx-postgres-15` using AWX's `SECRET_KEY`).
- **Semaphore Key Store:** Register the key in Semaphore (`140.44.4.58:100`) as a secondary runner fallback.
- **NUT Master (`14.4.4.44`):** Store key at `/root/.ssh/id_ed25519_snarfnet` (`chmod 600`) for `homelab_power_shutdown.sh` to execute passwordless tiered shutdowns.
- **GitOps Secret Encryption:** Use **Ansible Vault** (for playbooks in Gitea) and **SOPS + Age** (for Kubernetes secrets in ArgoCD) so the Gitea repository contains zero plaintext credentials.
#### Predicted Roadblocks & Mitigations
- **Roadblock 1.1 — Harvester HCI Immutable Root (`/home/rancher`):** Harvester v1.8.0 uses an elemental immutable filesystem; writing to `/etc/ssh` directly can be wiped on upgrade.
- *Mitigation:* Persist the SSH public key via Harvester's `/oem/99_custom.yaml` cloud-init configuration and the `Harvester Setting -> ssh-keys` CRD in Kubernetes (`spacedock`).
- **Roadblock 1.2 — Synology DSM Home Directory Permissions:** Synology DSM (`14.10.10.71` & `.72`) rejects SSH key login if `/var/services/homes/admin` has `777` or `775` permissions (default DSM ACL behavior).
- *Mitigation:* Run `chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys && chmod 755 ~` on both NAS appliances during enrollment.
- *Mitigation:* Enrollment playbook records unreachable nodes in a retry queue and AWX automatically applies the key via a post-boot callback or scheduled convergence job once they come online.
- Unset `storageclass.kubernetes.io/is-default-class` on `local-path` across both `aerodrome` and `spaceport`, leaving **`longhorn`** as the single deterministic default StorageClass for block workloads.
- Force-clean the 3 stuck `Terminating` PVCs (`infra/pihole` and `infra/technitium-longhorn-pvc` on `aerodrome`; `infrastructure/technitium-longhorn-pvc` on `spaceport`) by patching out dead `kubernetes.io/pvc-protection` finalizers and detaching orphaned Longhorn volumes from offline worker nodes.
- Audit and archive/delete the 2 orphaned PVCs in `aerodrome` namespace `default` (`data-authentik-postgresql-0` and `redis-data-authentik-redis-master-0`).
- Configure `spaceport` Longhorn with the same `snapshot-4h` (retains 6) and `snapshot-daily` (retains 7) recurring jobs we deployed on `aerodrome`, plus a new `backup-daily-nas` job (retains 14).
- Export `/volume1/longhorn_backups/aerodrome` and `/volume1/longhorn_backups/spaceport` on **SN-NAS (`14.10.10.71`)** (or replicate to **SnarfNAS `14.10.10.72`**) via NFSv4.
- Set Longhorn's `backup-target` setting on both clusters to `nfs://14.10.10.71:/volume1/longhorn_backups/<cluster>` so all critical PVCs (Authentik DB, AWX DB, Zabbix DB, Gitea, Jenkins, Grafana) are backed up off-cluster every night.
3.**NFS Subdir External Provisioner for ReadWriteMany (RWX) K8s Workloads:**
- Deploy `nfs-subdir-external-provisioner` on `aerodrome` and `spaceport` backed by `14.10.10.71:/volume1/k8s_shared` as StorageClass `synology-nfs`.
- Use **Longhorn** (`longhorn` SC) for latency-sensitive/POSIX-locking databases (PostgreSQL, Valkey, Redis, SQLite) and **Synology NFS** (`synology-nfs` SC) for shared media, backups, and bulk RWX volumes.
- Currently, `gotify`, `homebox`, `peanut`, and `pihole` bind-mount `/mnt/swarm_shared/legacy_volumes/...` (a 3-node GlusterFS volume across `.71`, `.72`, `.73`). With `.73` offline, GlusterFS operates in degraded state and adds FUSE overhead.
- Migrate Swarm persistent volumes to structured NFSv4 shares on `14.10.10.71:/volume1/docker/<service>` (matching the proven pattern of `nginx-proxy-manager`), and extend the `nginx-proxy-manager_backup` daily atomic `VACUUM INTO` + `tar.gz` backup daemon to also snapshot Gotify (`gotify.db`) and Homebox (`homebox.db`).
#### Predicted Roadblocks & Mitigations
- **Roadblock 2.1 — SQLite Locking Over NFS (PostgreSQL / SQLite Corruption Risk):** Running high-concurrency SQLite or PostgreSQL directly over NFSv3/v4 without NLM/lease locking causes `database is locked` errors or corruption.
- *Mitigation:* Never move K8s PostgreSQL PVCs (`authentik-postgresql`, `awx-postgres`, `zabbix-postgresql`, `gitea-postgresql`) onto raw NFS; keep them on **Longhorn block storage (iSCSI-backed ext4)** and use NFS *strictly* as the Longhorn `backup-target` and for RWX file shares. For Swarm SQLite apps (`npm`, `gotify`, `homebox`), enforce `replicas: 1`, `update_config.order: stop-first`, and NFSv4 (`nfsvers=4.1,hard,timeo=600,retrans=2`).
- *Mitigation:* Configure Synology NFS rules for `140.44.4.0/25` with `Squash: Map all users to admin` (or `No mapping` for Longhorn backup target) and enable `Allow connections from non-privileged ports (>1024)`.
- Provision dedicated **Client Tokens** (`http://140.44.4.70:8090/client`) for each important device (Primary Phone, Tablet, Desktop Browser, Homepage Dashboard).
2.**Discord Forwarding Architecture (Item 2):**
- **Gotify $\rightarrow$ Discord Bridge:** Deploy a lightweight, self-healing container service (`gotify-discord-bridge`) on Shipyard Swarm alongside Gotify. It connects internally to `ws://gotify:80/stream?token=<CLIENT_TOKEN>`, transforms every incoming Gotify message into a rich, color-coded Discord Embed (Red for priority $\ge 8$, Orange for $5-7$, Blue for $\le 4$), and POSTs it to your Discord Webhook URL.
- **Zabbix $\rightarrow$ Discord & Gotify (Dual-Path Redundancy):** Configure Zabbix 7.0.16 via JSON-RPC API to enable both the native **Discord** webhook media type and a **Gotify** webhook media type, linked to the `Admin` user, and **enable** Trigger Action `'Report problems to Zabbix administrators'` (`status=0`, currently disabled!). If Gotify or Swarm is ever down during an outage, Zabbix still reaches Discord directly from `spaceport`.
- **Home Assistant $\rightarrow$ Discord & Gotify:** Add a unified `notify.snarfnet_broadcast` group in Home Assistant (`14.4.4.20`) combining `notify.gotify` and the native Discord webhook integration.
- Deploy `zabbix-agent2` with shared PSK (`PSK_autoregister`) via Ansible across all missing Linux nodes: Spaceport Masters (`140.44.4.41-44`), Aerodrome Workers (`140.44.4.25`, `.27`), AI Pis (`140.44.4.7`, `.8`), and Harvester nodes (`140.44.4.81-83`).
- Enable SNMPv2c/v3 on **SN-NAS (`14.10.10.71`)**, **SnarfNAS (`14.10.10.72`)**, and **UDM Pro (`14.44.4.1`)**, registering them in Zabbix with templates `Synology DSM by SNMP` and `Ubiquiti AirOS/UniFi by SNMP`.
- **Kubernetes & Longhorn Health:** Link `Kubernetes nodes by HTTP` and `Kubernetes cluster state by HTTP` to `aerodrome` and `spaceport`; alert on Pod `CrashLoopBackOff`, Node `NotReady` $> 3\text{m}$, and Longhorn volume robustness `Degraded`/`Faulted`.
- **Hardware & Thermal Thresholds:** CPU temperature $> 75^\circ\text{C}$ (Warning) / $> 82^\circ\text{C}$ (High) for Raspberry Pi 5 nodes in `spaceport`; NVMe/HDD SMART health and Btrfs pool utilization $> 80\%$ (Warning) / $> 90\%$ (High).
- **TLS Certificate & Web Scenarios:** Zabbix Web Scenarios checking HTTP 200 and SSL certificate expiry ($< 14$ days Warning, $< 7$ days High) across all 35 `*.snarfnet.net` proxy hosts.
#### Predicted Roadblocks & Mitigations
- **Roadblock 3.1 — Authentik Forward Auth Blocking Gotify Mobile App:** The Android/iOS Gotify app uses its own `X-Gotify-Key` header and cannot complete an interactive Authentik OAuth2 browser redirect flow when connecting externally to `https://gotify.snarfnet.net`.
- *Mitigation:* In NPM Proxy Host #32 (`gotify.snarfnet.net`), bypass Authentik `auth_request` specifically when a valid `X-Gotify-Key` header or `/message` / `/stream` API path is requested, while enforcing Authentik Forward Auth on the web UI root (`/`, `/static/*`)!
- **Roadblock 3.2 — Discord Webhook Rate Limiting During Alert Storms:** During a power outage or switch flap, 25+ hosts firing simultaneously can hit Discord's `30 requests / 60 seconds` webhook rate limit (`HTTP 429`).
- *Mitigation:* Implement a 2-second batching queue with exponential backoff on `HTTP 429` (`Retry-After`) in the `gotify-discord-bridge` and configure Zabbix host dependency rules (all VLAN 20 hosts depend on UDM Pro `14.44.4.1` ICMP ping) so downstream alerts are suppressed if a core switch or gateway drops.
2.**Remediation of `aerodrome` K3s Control Plane Version Skew (Priority Upgrade):**
-`aerodrome` masters (`140.44.4.11-13`) are at `v1.26.10+k3s1` while workers are at `v1.30.6+k3s1`. Kubernetes strictly forbids skipping minor versions on control-plane upgrades.
- Create `ansible/playbooks/upgrade_k3s_aerodrome.yml` with `serial: 1`:
- Step 1: Take an `etcd-snapshot save pre-upgrade-1.26` on `.11`.
- Step 2: Sequentially step all 3 masters through each minor release channel: `v1.27.16+k3s1` $\rightarrow$ `v1.28.15+k3s1` $\rightarrow$ `v1.29.10+k3s1` $\rightarrow$ `v1.30.6+k3s1` (and optionally align with `spaceport` at `v1.31.4+k3s1`), waiting for `kubectl get nodes``Ready` and etcd quorum health (`etcdctl endpoint health`) between every single node and minor step.
- Checks `/var/run/reboot-required`. If a kernel reboot is needed on a K8s or Swarm node, it runs `kubectl drain <node> --ignore-daemonsets --delete-emptydir-data` (or `docker node update --availability drain`), reboots, waits for SSH + kubelet `Ready` + Longhorn volume health (`Healthy`), and runs `kubectl uncordon <node>`.
- Updates K3s binaries and pulls latest pinned container digests for Shipyard Swarm stacks, followed by HTTP 200 health verification across all 35 NPM endpoints.
- Verifies `zabbix-agent2`, `upsmon` slave daemons, NFS mounts, and SSH `authorized_keys` across all nodes, sending a summary report to Gotify/Discord.
#### Predicted Roadblocks & Mitigations
- **Roadblock 4.1 — Longhorn Replica Rebuilding During Rolling Node Reboots:** If Ansible reboots node B immediately after node A finishes rebooting, Longhorn volumes may still be in `Degraded` (rebuilding replicas) on node A; rebooting node B can take the last healthy replica offline!
- *Mitigation:* Add an explicit Ansible `uri` / `command` pre-task gate in the rolling reboot playbook that queries the Longhorn API (`volumes` list) and pauses the next node's drain until **zero attached volumes are in `degraded` state**.
- **Roadblock 4.2 — AWX Running Inside the Cluster It Is Upgrading (`aerodrome`):** When AWX upgrades or reboots the `aerodrome` node currently hosting the `awx-task` pod, the active AWX job itself will be killed mid-execution!
- *Mitigation:* Pin `awx-task` and `awx-web` to `sninfr2lnxwrk06` during `aerodrome` master upgrades, and execute `aerodrome` worker reboots either via a Workflow Job Template that upgrades `wrk06` last, or trigger `aerodrome` node reboots from **Semaphore** (`140.44.4.58` on `spaceport`) so the orchestrator lives on a separate physical cluster!
- Run our repository's `iac-reverse generate` tool against `aerodrome`, `spaceport`, `spacedock`, and `shipyard` to export current live resources into clean HCL and Terraform state.
- Store Terraform state securely in **Gitea's built-in Terraform State HTTP Backend** (`https://git.snarfnet.net/api/packages/snarf/terraform/state/...`) or on Synology NAS (`14.10.10.71`), backed up daily.
- Register `aerodrome` and `spacedock` as target clusters in Spaceport's ArgoCD instance.
- Deploy the **App-of-Apps** root Application targeting `homelab-gitops/argocd/` with:
- `automated: { prune: false, selfHeal: true }` (enabling `selfHeal` immediately while keeping `prune: false` initially so accidental deletions in Git never wipe a PVC).
- ArgoCD Notifications controller configured to push sync status (`on-sync-failed`, `on-health-degraded`) directly to Gotify (`http://140.44.4.70:8090`).
#### Predicted Roadblocks & Mitigations
- **Roadblock 5.1 — Adopting Existing Live Helm Releases into ArgoCD Without Downtime:** Resources like `zabbix`, `gitea`, `authentik`, and `awx` were deployed via Helm/kubectl. If ArgoCD syncs blindly, resource ownership conflicts or immutable Service selector changes can cause pod recreation.
- *Mitigation:* Use `ServerSideApply=true` and `Replace=false` sync options in ArgoCD, and test every Application in **Diff / Manual Sync mode** (`kubectl diff`) until ArgoCD reports `Synced / Healthy` with zero destructive diffs before flipping `selfHeal: true`.
- **Roadblock 5.2 — Circular Dependency if Gitea or ArgoCD Goes Down:**
- *Mitigation:* Mirror the `homelab-gitops` repository automatically to a bare Git repository on Synology NAS (`/volume1/git_mirror/homelab-gitops.git`) and your workstation (`SnarfCode`), ensuring Terraform and `kubectl apply` can always run directly from local disk if `spaceport` is offline.
---
### Phase 6: Power Resilience & UPS Automation Plan (Updated Core Pillar)
#### 6.1 Core Power Architecture & IP Assignments
The power resilience architecture uses a centralized **NUT Master Server** (`14.4.4.44`) that polls the physical UPS and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol |
2. **Stage 2 (Network Core):** UDM Pro (`14.44.4.1`) & Core switches boot, establishing VLANs 1–60 and inter-VLAN routing.
3. **Stage 3 (NUT Master):** `14.4.4.44` boots (`BIOS: Restore on AC Power Loss = Always On`), starts `upsd`, and sends `ONLINE` alert to Discord/Gotify.
4. **Stage 4 (Storage Gate — Must Complete Before Compute):** Synology NAS arrays (`14.10.10.71` & `14.10.10.72`) boot, mount Btrfs pools, and expose NFS (`:2049`).
- Confirmation of whether the existing `snarf` / `Katzir476!` credential (from `setup_storage_cluster.sh`) is valid across the Debian/RPi nodes, plus SSH/DSM credentials for Synology NAS (`14.10.10.71` / `14.10.10.72`) and Harvester nodes (`140.44.4.81-83`) so we can enroll the unified `id_ed25519_snarfnet` key.
# Snarfnet Homelab Power Resilience & UPS Automation Plan
This document details the power resilience strategy, tiered graceful shutdown sequence, and integration architecture for the **snarfnet** homelab centered around the primary **NUT (Network UPS Tools) Master Server**.
---
## 1. Core Architecture & IP Assignments
The power resilience architecture uses a centralized **NUT Master Server** host that polls connected UPS hardware and broadcasts status alerts (`ONBATT`, `LOWBATT`, `ONLINE`, `FSD`) across all homelab VLANs.
| Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description |
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
