Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a793a04ffbc651b057a46b31c35877e72519b7e7
SnarfCode/docs/infrastructure_improvement_plan.md

34 KiB
Raw Blame History

Snarfnet Master Infrastructure Improvement & Resilience Plan

This master architectural plan defines the prioritized roadmap, implementation blueprints, failure-mode analysis, and power resilience strategy for the snarfnet homelab. All solutions are designed to execute locally on self-hosted infrastructure (aerodrome, spaceport, spacedock, shipyard, and Synology NAS) with zero external SaaS dependencies outside of outbound Discord webhook delivery.

1. Executive Summary & Engineering Prioritization

Following Site Reliability Engineering (SRE) best practices, the eight infrastructure initiatives and the power resilience architecture are ordered into a 6-Phase Dependency Graph:

  1. Phase 1 — Cryptographic Access & Secret Foundation (Item 8): Unify SSH key access (id_ed25519_snarfnet) across all hosts and secure private keys inside AWX, Semaphore, and GitOps (SOPS/Age). Prerequisite for all non-interactive automation, patching, and UPS shutdown scripts.
  2. Phase 2 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3): Resolve dual-default StorageClasses, clear stuck Terminating PVCs, add Longhorn recurring snapshots on spaceport, configure Synology NAS (14.10.10.71) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. Prerequisite before upgrading kernels or control planes.
  3. Phase 3 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7): Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant \rightarrow Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. Ensures full visibility before running automated upgrades.
  4. Phase 4 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6): Remediate the critical Kubernetes control-plane version skew on aerodrome (v1.26.10 Masters vs. v1.30.6 Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates.
  5. Phase 5 — Declarative IaC & GitOps Continuity (Item 4): Codify live infrastructure into local Gitea (git.snarfnet.net) using iac-reverse Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD ApplicationSets (spaceport & aerodrome) for automated drift remediation.
  6. Phase 6 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar): Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (14.4.4.44), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment.
flowchart LR
    P1["Phase 1: Unified SSH Key\n& Secret Vault (Item 8)"] --> P2["Phase 2: Storage & NFS\nStabilization (Item 3)"]
    P1 --> P3["Phase 3: Gotify, Discord &\nZabbix Sensors (Items 1, 2, 7)"]
    P2 --> P4["Phase 4: AWX & Rolling\nInfra Upgrades (Items 5, 6)"]
    P3 --> P4
    P4 --> P5["Phase 5: Terraform &\nArgoCD GitOps (Item 4)"]
    P1 --> P6["Phase 6: Power Resilience\n& UPS Shutdown"]
    P2 --> P6
    P3 --> P6

2. Live Infrastructure Baseline Audit (October 30, 2026)

Our automated live probe across the homelab APIs identified the following baseline state that directly informs this plan:

Platform / Service Current Version / State Key Audit Findings & Action Required
Aerodrome K3s (local) Masters (ctl01-03): v1.26.10+k3s1
Workers (wrk01-07): v1.30.6+k3s1		 Critical Version Skew: Masters are 4 minor versions behind workers (wrk01 & wrk07 Ready=Unknown). Dual default StorageClasses (local-path & longhorn). 2 PVCs stuck Terminating (pihole, technitium). Longhorn backup-target is empty ('').
Spaceport K3s (c-m-4fgkl75b) 4 Masters + 6 Workers: v1.31.4+k3s1 (All 10 Ready) Dual default StorageClasses (local-path & longhorn). 1 PVC stuck Terminating (technitium-longhorn-pvc). Zero Longhorn recurring snapshot jobs ([]) and empty backup-target ('') despite hosting Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs.
Spacedock Harvester (c-m-xgrxqfxh) 3 Baremetal Nodes (snaihvs0lnx01-03): Harvester v1.8.0 / v1.35.2+rke2r1 Healthy harvester-longhorn default StorageClass. 2 active PVCs in snarfai (longhorn-pvc-20tb, open-webui-pvc). Not yet monitored in Zabbix.
Shipyard Docker Swarm 5 Nodes (140.44.4.71-75); .73 & .75 offline NPM pinned in PublishMode: host on .74 with daily NAS backup. Legacy stacks (gotify, homebox, peanut, pihole) use /mnt/swarm_shared (GlusterFS across .71-.73), which is degraded when .73 is offline.
Zabbix (zabbix.snarfnet.net) Version 7.0.16 (140.44.4.57:8888, Trapper 140.44.4.61:10051) 22 hosts registered. Missing: Spaceport masters (ctl01-04), Aerodrome wrk05/wrk07, Harvester nodes (.81-.83), Synology NAS (.71/.72), AI Pis (.7/.8), UDM Pro (.1), Home Assistant (.20). Enabled MediaTypes: [] (None); Problem Action: Disabled (status=1).
Gotify (gotify.snarfnet.net) Version 2.9.1 (140.44.4.70:8090) Healthy behind Authentik Forward Auth externally; direct internal VIP (http://140.44.4.70:8090) reachable for low-level webhooks.
AWX (awx.snarfnet.net) Operator 2.19.1 / AWX 24.6.1 (aerodrome) Running in awx namespace (awx-web, awx-task, awx-postgres-15-0). Ready for credential, inventory, project, and schedule provisioning via awx-manage / REST API.
ArgoCD (argo.snarfnet.net) Version v3.4.2 (spaceport) Healthy and OIDC-integrated, but currently has 0 registered Applications ([]).

3. Detailed Phase Execution Plans & Roadblock Analysis

Phase 1: Single SSH Key & Cryptographic Secret Management (Item 8)

Architecture & Implementation Steps

  1. Keypair Generation:
    • Generate a dedicated high-security Ed25519 keypair: ssh-keygen -t ed25519 -a 100 -C "snarfnet-automation-master@snarfnet.net" -f id_ed25519_snarfnet.
  2. Standardized Automation Account (snarf-admin / root):
    • Bootstrap all 30+ hosts across VLANs 1, 2, 20, 40, and 50 with a single enrollment script that:
      • Installs the public key into /root/.ssh/authorized_keys and /home/snarf/.ssh/authorized_keys (plus rancher on Harvester nodes and admin on Synology DSM).
      • Configures /etc/sudoers.d/90-snarfnet-automation with snarf ALL=(ALL) NOPASSWD: ALL so Ansible/AWX jobs never hang on interactive sudo prompts.
      • Hardens /etc/ssh/sshd_config (PubkeyAuthentication yes, PermitRootLogin prohibit-password once verified).
  3. Secure Key Storage (Zero Plaintext in Git):
    • AWX Credential Vault: Inject id_ed25519_snarfnet as an AWX Machine credential (encrypted at rest in awx-postgres-15 using AWX's SECRET_KEY).
    • Semaphore Key Store: Register the key in Semaphore (140.44.4.58:100) as a secondary runner fallback.
    • NUT Master (14.4.4.44): Store key at /root/.ssh/id_ed25519_snarfnet (chmod 600) for homelab_power_shutdown.sh to execute passwordless tiered shutdowns.
    • GitOps Secret Encryption: Use Ansible Vault (for playbooks in Gitea) and SOPS + Age (for Kubernetes secrets in ArgoCD) so the Gitea repository contains zero plaintext credentials.

Predicted Roadblocks & Mitigations

  • Roadblock 1.1 — Harvester HCI Immutable Root (/home/rancher): Harvester v1.8.0 uses an elemental immutable filesystem; writing to /etc/ssh directly can be wiped on upgrade.
    • Mitigation: Persist the SSH public key via Harvester's /oem/99_custom.yaml cloud-init configuration and the Harvester Setting -> ssh-keys CRD in Kubernetes (spacedock).
  • Roadblock 1.2 — Synology DSM Home Directory Permissions: Synology DSM (14.10.10.71 & .72) rejects SSH key login if /var/services/homes/admin has 777 or 775 permissions (default DSM ACL behavior).
    • Mitigation: Run chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys && chmod 755 ~ on both NAS appliances during enrollment.
  • Roadblock 1.3 — Currently Offline Nodes (140.44.4.21, .25/.27, .73, .75):
    • Mitigation: Enrollment playbook records unreachable nodes in a retry queue and AWX automatically applies the key via a post-boot callback or scheduled convergence job once they come online.

Phase 2: Persistent Storage Stabilization & NAS NFS Architecture (Item 3)

Architecture & Implementation Steps

  1. Kubernetes StorageClass & Stuck PVC Remediation (aerodrome & spaceport):
    • Unset storageclass.kubernetes.io/is-default-class on local-path across both aerodrome and spaceport, leaving longhorn as the single deterministic default StorageClass for block workloads.
    • Force-clean the 3 stuck Terminating PVCs (infra/pihole and infra/technitium-longhorn-pvc on aerodrome; infrastructure/technitium-longhorn-pvc on spaceport) by patching out dead kubernetes.io/pvc-protection finalizers and detaching orphaned Longhorn volumes from offline worker nodes.
    • Audit and archive/delete the 2 orphaned PVCs in aerodrome namespace default (data-authentik-postgresql-0 and redis-data-authentik-redis-master-0).
  2. Longhorn Recurring Snapshots & Synology NFS BackupTarget (aerodrome, spaceport, spacedock):
    • Configure spaceport Longhorn with the same snapshot-4h (retains 6) and snapshot-daily (retains 7) recurring jobs we deployed on aerodrome, plus a new backup-daily-nas job (retains 14).
    • Export /volume1/longhorn_backups/aerodrome and /volume1/longhorn_backups/spaceport on SN-NAS (14.10.10.71) (or replicate to SnarfNAS 14.10.10.72) via NFSv4.
    • Set Longhorn's backup-target setting on both clusters to nfs://14.10.10.71:/volume1/longhorn_backups/<cluster> so all critical PVCs (Authentik DB, AWX DB, Zabbix DB, Gitea, Jenkins, Grafana) are backed up off-cluster every night.
  3. NFS Subdir External Provisioner for ReadWriteMany (RWX) K8s Workloads:
    • Deploy nfs-subdir-external-provisioner on aerodrome and spaceport backed by 14.10.10.71:/volume1/k8s_shared as StorageClass synology-nfs.
    • Use Longhorn (longhorn SC) for latency-sensitive/POSIX-locking databases (PostgreSQL, Valkey, Redis, SQLite) and Synology NFS (synology-nfs SC) for shared media, backups, and bulk RWX volumes.
  4. Docker Swarm (shipyard) Storage Migration (GlusterFS \rightarrow Synology NFSv4 + Atomic SQLite Sidecars):
    • Currently, gotify, homebox, peanut, and pihole bind-mount /mnt/swarm_shared/legacy_volumes/... (a 3-node GlusterFS volume across .71, .72, .73). With .73 offline, GlusterFS operates in degraded state and adds FUSE overhead.
    • Migrate Swarm persistent volumes to structured NFSv4 shares on 14.10.10.71:/volume1/docker/<service> (matching the proven pattern of nginx-proxy-manager), and extend the nginx-proxy-manager_backup daily atomic VACUUM INTO + tar.gz backup daemon to also snapshot Gotify (gotify.db) and Homebox (homebox.db).

Predicted Roadblocks & Mitigations

  • Roadblock 2.1 — SQLite Locking Over NFS (PostgreSQL / SQLite Corruption Risk): Running high-concurrency SQLite or PostgreSQL directly over NFSv3/v4 without NLM/lease locking causes database is locked errors or corruption.
    • Mitigation: Never move K8s PostgreSQL PVCs (authentik-postgresql, awx-postgres, zabbix-postgresql, gitea-postgresql) onto raw NFS; keep them on Longhorn block storage (iSCSI-backed ext4) and use NFS strictly as the Longhorn backup-target and for RWX file shares. For Swarm SQLite apps (npm, gotify, homebox), enforce replicas: 1, update_config.order: stop-first, and NFSv4 (nfsvers=4.1,hard,timeo=600,retrans=2).
  • Roadblock 2.2 — Synology NFS Squash Permissions (root_squash): By default, Synology DSM maps container root (UID 0) to nobody (UID 65534), breaking container startup chown.
    • Mitigation: Configure Synology NFS rules for 140.44.4.0/25 with Squash: Map all users to admin (or No mapping for Longhorn backup target) and enable Allow connections from non-privileged ports (>1024).

Phase 3: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7)

Architecture & Implementation Steps

flowchart TD
    subgraph Sources ["Homelab Event & Alert Sources"]
        ZBX["Zabbix 7.0.16 Server\n(140.44.4.61 / spaceport)"]
        HA["Home Assistant\n(14.4.4.20 / VLAN 50)"]
        NUT["NUT UPS Master\n(14.4.4.44 / nut-notify.sh)"]
        AWX_N["AWX / ArgoCD / Backup Jobs"]
    end

    subgraph GotifyHub ["Central Local Notification Hub"]
        GTF["Gotify Server v2.9.1\nInternal VIP: http://140.44.4.70:8090\nExternal: https://gotify.snarfnet.net"]
        BRIDGE["gotify-discord-bridge\n(WebSocket Listener Container)"]
    end

    subgraphRecipients ["Important Devices & External Channels"]
        Mobile["Mobile Devices (Android/iOS)\n& Desktop Browser Push"]
        Homepage["Homepage Dashboard Widget"]
        Discord["Discord Server Channels\n(#infra-alerts, #smart-home, #ups-power)"]
    end

    ZBX -- "1. Native Gotify MediaType" --> GTF
    ZBX -- "2. Native Discord Webhook (Direct Fallback)" --> Discord
    HA -- "1. REST notify.gotify" --> GTF
    HA -- "2. Native Discord Integration" --> Discord
    NUT -- "Direct HTTP (Bypasses SSO)" --> GTF
    AWX_N -- "App Token Webhook" --> GTF

    GTF -- "WebSocket Stream (Client Tokens)" --> Mobile
    GTF -- "Client Token API" --> Homepage
    GTF -- "ws://140.44.4.70:8090/stream" --> BRIDGE
    BRIDGE -- "Formatted Discord Embeds" --> Discord
  1. Gotify Broadcast Channels & Device Enrollment (Item 1):
    • Automated Python script (scripts/setup_gotify_channels.py) provisions 5 structured Gotify Applications via http://140.44.4.70:8090/application:
      • UPS & Power Resilience (Default Priority: 9)
      • Zabbix Infrastructure Alerts (Priority mapped 1-10 by severity)
      • Home Assistant Smart Home (Default Priority: 5)
      • AWX & GitOps Automation (Default Priority: 4)
      • Backups & Storage Health (Default Priority: 6)
    • Provision dedicated Client Tokens (http://140.44.4.70:8090/client) for each important device (Primary Phone, Tablet, Desktop Browser, Homepage Dashboard).
  2. Discord Forwarding Architecture (Item 2):
    • Gotify \rightarrow Discord Bridge: Deploy a lightweight, self-healing container service (gotify-discord-bridge) on Shipyard Swarm alongside Gotify. It connects internally to ws://gotify:80/stream?token=<CLIENT_TOKEN>, transforms every incoming Gotify message into a rich, color-coded Discord Embed (Red for priority \ge 8, Orange for 5-7, Blue for \le 4), and POSTs it to your Discord Webhook URL.
    • Zabbix \rightarrow Discord & Gotify (Dual-Path Redundancy): Configure Zabbix 7.0.16 via JSON-RPC API to enable both the native Discord webhook media type and a Gotify webhook media type, linked to the Admin user, and enable Trigger Action 'Report problems to Zabbix administrators' (status=0, currently disabled!). If Gotify or Swarm is ever down during an outage, Zabbix still reaches Discord directly from spaceport.
    • Home Assistant \rightarrow Discord & Gotify: Add a unified notify.snarfnet_broadcast group in Home Assistant (14.4.4.20) combining notify.gotify and the native Discord webhook integration.
  3. Complete Zabbix Sensors, Thresholds, & Templates (Item 7):
    • Fix Missing Hosts (100% Coverage):
      • Deploy zabbix-agent2 with shared PSK (PSK_autoregister) via Ansible across all missing Linux nodes: Spaceport Masters (140.44.4.41-44), Aerodrome Workers (140.44.4.25, .27), AI Pis (140.44.4.7, .8), and Harvester nodes (140.44.4.81-83).
      • Enable SNMPv2c/v3 on SN-NAS (14.10.10.71), SnarfNAS (14.10.10.72), and UDM Pro (14.44.4.1), registering them in Zabbix with templates Synology DSM by SNMP and Ubiquiti AirOS/UniFi by SNMP.
    • Custom Sensors & Threshold Alarms:
      • NUT UPS Template (nutserver 14.4.4.44): Custom UserParameter=ups.metric[*],upsc snarf-ups@127.0.0.1 $1 polling ups.status (Trigger: != OL \rightarrow High), battery.charge (< 80% Warning, < 50% High, < 25% Disaster), ups.load (> 85% High), and battery.runtime (< 600s High).
      • Kubernetes & Longhorn Health: Link Kubernetes nodes by HTTP and Kubernetes cluster state by HTTP to aerodrome and spaceport; alert on Pod CrashLoopBackOff, Node NotReady > 3\text{m}, and Longhorn volume robustness Degraded/Faulted.
      • Hardware & Thermal Thresholds: CPU temperature > 75^\circ\text{C} (Warning) / > 82^\circ\text{C} (High) for Raspberry Pi 5 nodes in spaceport; NVMe/HDD SMART health and Btrfs pool utilization > 80\% (Warning) / > 90\% (High).
      • TLS Certificate & Web Scenarios: Zabbix Web Scenarios checking HTTP 200 and SSL certificate expiry (< 14 days Warning, < 7 days High) across all 35 *.snarfnet.net proxy hosts.

Predicted Roadblocks & Mitigations

  • Roadblock 3.1 — Authentik Forward Auth Blocking Gotify Mobile App: The Android/iOS Gotify app uses its own X-Gotify-Key header and cannot complete an interactive Authentik OAuth2 browser redirect flow when connecting externally to https://gotify.snarfnet.net.
    • Mitigation: In NPM Proxy Host #32 (gotify.snarfnet.net), bypass Authentik auth_request specifically when a valid X-Gotify-Key header or /message / /stream API path is requested, while enforcing Authentik Forward Auth on the web UI root (/, /static/*)!
  • Roadblock 3.2 — Discord Webhook Rate Limiting During Alert Storms: During a power outage or switch flap, 25+ hosts firing simultaneously can hit Discord's 30 requests / 60 seconds webhook rate limit (HTTP 429).
    • Mitigation: Implement a 2-second batching queue with exponential backoff on HTTP 429 (Retry-After) in the gotify-discord-bridge and configure Zabbix host dependency rules (all VLAN 20 hosts depend on UDM Pro 14.44.4.1 ICMP ping) so downstream alerts are suppressed if a core switch or gateway drops.

Phase 4: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6)

Architecture & Implementation Steps

  1. AWX Provisioning (https://awx.snarfnet.net on aerodrome):
    • Execute an automated bootstrap script via kubectl exec into awx-task (or AWX REST API /api/v2/) to create:
      • Organization: Snarfnet Homelab
      • Credentials:
        1. Snarfnet Master SSH Key (Machine credential with id_ed25519_snarfnet + passwordless sudo)
        2. Gitea Automation Token (Source Control credential for pulling playbooks from http://140.44.4.62:90/snarf/SnarfCode.git)
        3. Authentik & Zabbix API Vault (Vault credential)
      • Project: Snarfnet Infrastructure Playbooks synced from local Gitea (git.snarfnet.net).
      • Inventory (Snarfnet Master Inventory): Structured into groups:
        • k3s_aerodrome_masters (140.44.4.11-13) & k3s_aerodrome_workers (140.44.4.21-27)
        • k3s_spaceport_masters (140.44.4.41-44) & k3s_spaceport_workers (140.44.4.51-56)
        • harvester_spacedock (140.44.4.81-83)
        • swarm_shipyard (140.44.4.71-75)
        • ai_pi_cluster (140.44.4.7-8)
        • smart_home_nodes (14.4.4.20, 14.4.4.44, 14.44.4.14)
        • synology_nas (14.10.10.71-72)
  2. Remediation of aerodrome K3s Control Plane Version Skew (Priority Upgrade):
    • aerodrome masters (140.44.4.11-13) are at v1.26.10+k3s1 while workers are at v1.30.6+k3s1. Kubernetes strictly forbids skipping minor versions on control-plane upgrades.
    • Create ansible/playbooks/upgrade_k3s_aerodrome.yml with serial: 1:
      • Step 1: Take an etcd-snapshot save pre-upgrade-1.26 on .11.
      • Step 2: Sequentially step all 3 masters through each minor release channel: v1.27.16+k3s1 \rightarrow v1.28.15+k3s1 \rightarrow v1.29.10+k3s1 \rightarrow v1.30.6+k3s1 (and optionally align with spaceport at v1.31.4+k3s1), waiting for kubectl get nodes Ready and etcd quorum health (etcdctl endpoint health) between every single node and minor step.
  3. AWX Job Templates & Scheduled Maintenance Windows:
    • Template 1 — Weekly OS & Security Patching (Cron: Sundays 03:00 AM):
      • Runs apt-get update && apt-get dist-upgrade -y with serial: 1.
      • Checks /var/run/reboot-required. If a kernel reboot is needed on a K8s or Swarm node, it runs kubectl drain <node> --ignore-daemonsets --delete-emptydir-data (or docker node update --availability drain), reboots, waits for SSH + kubelet Ready + Longhorn volume health (Healthy), and runs kubectl uncordon <node>.
    • Template 2 — Monthly K3s & Container Stack Update (Cron: 1st Sunday 04:00 AM):
      • Updates K3s binaries and pulls latest pinned container digests for Shipyard Swarm stacks, followed by HTTP 200 health verification across all 35 NPM endpoints.
    • Template 3 — Daily Drift & Agent Health Audit (Cron: Daily 06:00 AM):
      • Verifies zabbix-agent2, upsmon slave daemons, NFS mounts, and SSH authorized_keys across all nodes, sending a summary report to Gotify/Discord.

Predicted Roadblocks & Mitigations

  • Roadblock 4.1 — Longhorn Replica Rebuilding During Rolling Node Reboots: If Ansible reboots node B immediately after node A finishes rebooting, Longhorn volumes may still be in Degraded (rebuilding replicas) on node A; rebooting node B can take the last healthy replica offline!
    • Mitigation: Add an explicit Ansible uri / command pre-task gate in the rolling reboot playbook that queries the Longhorn API (volumes list) and pauses the next node's drain until zero attached volumes are in degraded state.
  • Roadblock 4.2 — AWX Running Inside the Cluster It Is Upgrading (aerodrome): When AWX upgrades or reboots the aerodrome node currently hosting the awx-task pod, the active AWX job itself will be killed mid-execution!
    • Mitigation: Pin awx-task and awx-web to sninfr2lnxwrk06 during aerodrome master upgrades, and execute aerodrome worker reboots either via a Workflow Job Template that upgrades wrk06 last, or trigger aerodrome node reboots from Semaphore (140.44.4.58 on spaceport) so the orchestrator lives on a separate physical cluster!

Phase 5: Terraform Plans & ArgoCD GitOps Integrity (Item 4)

Architecture & Implementation Steps

  1. Local GitOps Repository Structure on Gitea (https://git.snarfnet.net/snarf/homelab-gitops): 
    homelab-gitops/
├── terraform/
│   ├── authentik/       # Providers, Applications, Outposts, Conditional MFA Policies
│   ├── harvester/       # Spacedock VM definitions, VLAN networks, SSH keys
│   ├── shipyard-swarm/  # Docker Swarm stacks (NPM, Gotify, Homebox, Peanut, Pihole)
│   └── synology/        # Shared folders, NFS rules
└── argocd/
    ├── bootstrap/       # Root App-of-Apps for spaceport & aerodrome
    ├── spaceport/       # Gitea, Zabbix, Jenkins, Semaphore, Coder, Homepage, Traefik
    └── aerodrome/       # Authentik, AWX, Grafana, Uptime Kuma, Longhorn settings
  2. Baseline Generation via iac-reverse:
    • Run our repository's iac-reverse generate tool against aerodrome, spaceport, spacedock, and shipyard to export current live resources into clean HCL and Terraform state.
    • Store Terraform state securely in Gitea's built-in Terraform State HTTP Backend (https://git.snarfnet.net/api/packages/snarf/terraform/state/...) or on Synology NAS (14.10.10.71), backed up daily.
  3. ArgoCD Multi-Cluster Registration (https://argo.snarfnet.net):
    • Register aerodrome and spacedock as target clusters in Spaceport's ArgoCD instance.
    • Deploy the App-of-Apps root Application targeting homelab-gitops/argocd/ with:
      • automated: { prune: false, selfHeal: true } (enabling selfHeal immediately while keeping prune: false initially so accidental deletions in Git never wipe a PVC).
      • ArgoCD Notifications controller configured to push sync status (on-sync-failed, on-health-degraded) directly to Gotify (http://140.44.4.70:8090).

Predicted Roadblocks & Mitigations

  • Roadblock 5.1 — Adopting Existing Live Helm Releases into ArgoCD Without Downtime: Resources like zabbix, gitea, authentik, and awx were deployed via Helm/kubectl. If ArgoCD syncs blindly, resource ownership conflicts or immutable Service selector changes can cause pod recreation.
    • Mitigation: Use ServerSideApply=true and Replace=false sync options in ArgoCD, and test every Application in Diff / Manual Sync mode (kubectl diff) until ArgoCD reports Synced / Healthy with zero destructive diffs before flipping selfHeal: true.
  • Roadblock 5.2 — Circular Dependency if Gitea or ArgoCD Goes Down:
    • Mitigation: Mirror the homelab-gitops repository automatically to a bare Git repository on Synology NAS (/volume1/git_mirror/homelab-gitops.git) and your workstation (SnarfCode), ensuring Terraform and kubectl apply can always run directly from local disk if spaceport is offline.

Phase 6: Power Resilience & UPS Automation Plan (Updated Core Pillar)

6.1 Core Power Architecture & IP Assignments

The power resilience architecture uses a centralized NUT Master Server (14.4.4.44) that polls the physical UPS and broadcasts status alerts (ONBATT, LOWBATT, ONLINE, FSD) across all homelab VLANs.

Role Hostname / Identifier Target IP Subnet / VLAN Description & Failure Protocol
NUT Master Server nutserver.snarfnet.net 14.4.4.44 VLAN 50 (Smart Home) Primary NUT upsd daemon polling physical UPS via USB/Serial/SNMP. Runs upsmon master and holds /root/.ssh/id_ed25519_snarfnet.
Edge Ingress / Reverse Proxy nginx.snarfnet.net 140.44.4.74 VLAN 20 (Swarm 04) NPM running in PublishMode: host on ports 80/443/81. Dependent on Synology NFS; stopped in Tier 2 prior to NAS Safe Mode.
Authentication & SSO authentik.snarfnet.net 140.44.4.29:444 VLAN 20 (Aerodrome K3s) Handles Forward Auth & Conditional MFA. Runs on Longhorn storage in Aerodrome.
UPS Web Dashboard peanut.snarfnet.net 140.44.4.70:8050 VLAN 20 (Swarm VIP) Peanut container UI consuming 14.4.4.44:3493. Protected by Authentik Forward Auth externally.
Emergency Push & Discord gotify.snarfnet.net 140.44.4.70:8090 VLAN 20 (Swarm VIP) Gotify server + gotify-discord-bridge. Internal outage alerts use http://140.44.4.70:8090 (+ direct Discord webhook fallback) to bypass Authentik during shutdown.
Smart Home Automation homeassistant.snarfnet.net 14.4.4.20 VLAN 50 (Smart Home) Home Assistant integration monitoring 14.4.4.44:3493 for state changes, critical push alerts, and non-essential load shedding.
Centralized Monitoring zabbix.snarfnet.net 140.44.4.61 VLAN 20 (Spaceport K3s) Zabbix 7.0.16 polling 14.4.4.44 via Zabbix Agent 2 upsc UserParameters.
Primary Storage 1 & 2 sn-nas / snarfnas 14.10.10.71 / .72 VLAN 40 (Storage) Synology DSM UPS clients polling 14.4.4.44. Host NFSv4 shares and Longhorn BackupTargets.

6.2 Tiered Graceful Shutdown Hierarchy (6 Stages)

+--------------------------------------------------------------------------------------------------+
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT                                |
| - Broadcasts emergency alert to Gotify (http://140.44.4.70:8090), Discord Webhook, & HA Push     |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads                                 |
| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped      |
| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted via SSH key            |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM, Gotify, Homebox, Peanut)    |
| - Send final pre-shutdown notification to Gotify & Discord                                       |
| - Scale down Swarm NFS services (`nginx-proxy-manager_app`, `backup`, `peanut`, `homebox`)       |
| - Flushes open SQLite databases and cleanly releases all NFS file locks on 14.10.10.71           |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach              |
| - Harvester guest VMs (140.44.4.81-83) gracefully shut down via `virsh shutdown` / API           |
| - Stop K3s agent on Aerodrome (140.44.4.21-27) & Spaceport (140.44.4.51-56) worker nodes         |
| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes     |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode)                            |
| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode (`synopoweroff` / NUT)       |
| - Unmounts Btrfs volume pools, terminates NFS/iSCSI daemons, parks drive heads                   |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors                             |
| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Masters save etcd snapshot & halt      |
| - Remaining Swarm Managers (140.44.4.71-75) & Harvester Hypervisors (140.44.4.81-83) halt        |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut                                |
| - NUT Master (14.4.4.44) sends direct Discord webhook ("Final FSD Cut") & issues `upsdrvctl`     |
| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops  |
+--------------------------------------------------------------------------------------------------+

6.3 Automated Power Restoration & Boot Verification Gates

  1. Stage 1 (Grid AC Restored, Battery \ge 30\%): UPS re-energizes load outlets.
  2. Stage 2 (Network Core): UDM Pro (14.44.4.1) & Core switches boot, establishing VLANs 1–60 and inter-VLAN routing.
  3. Stage 3 (NUT Master): 14.4.4.44 boots (BIOS: Restore on AC Power Loss = Always On), starts upsd, and sends ONLINE alert to Discord/Gotify.
  4. Stage 4 (Storage Gate — Must Complete Before Compute): Synology NAS arrays (14.10.10.71 & 14.10.10.72) boot, mount Btrfs pools, and expose NFS (:2049).
  5. Stage 5 (Control Planes & Hypervisors): Harvester (140.44.4.81-83), K3s Masters (aerodrome & spaceport), and Swarm Managers (140.44.4.71-75) boot and establish etcd/Raft quorum.
  6. Stage 6 (Compute Workers & Ingress): K3s workers boot, Longhorn reattaches PVCs (data-authentik-postgresql-0, awx-postgres, zabbix-postgresql), and NPM on snshpd4lnxswm04 (140.44.4.74) binds host ports 80/443/81.

4. Access & Inputs Required to Begin Automation

We already have verified administrative API access to:

  • Authentik API (https://authentik.snarfnet.net)
  • Rancher & Kubernetes APIs (aerodrome, spaceport, and spacedock Harvester)
  • Portainer & Docker Swarm API (https://shipyard.snarfnet.net)
  • Zabbix 7.0.16 API (https://zabbix.snarfnet.net/api_jsonrpc.php)

To automate the remaining items end-to-end, we will only need the following when we reach each corresponding task:

  1. Discord Webhook URL(s) (For Phase 3 — Item 2):
    • One Discord Webhook URL (or separate URLs if you prefer separate channels like #infra-alerts, #smart-home, and #ups-power).
  2. Gotify Admin Credentials (For Phase 3 — Item 1):
    • Current Gotify admin username/password (or we can inspect/bootstrap a token directly via the Gotify SQLite database on Shipyard Swarm).
  3. Home Assistant Long-Lived Access Token or SSH (For Phase 3 — Item 2):
    • To inject the Gotify/Discord notification configuration into 14.4.4.20 (configuration.yaml / automations.yaml).
  4. Host SSH / Synology DSM Credentials (For Phase 1 — Item 8 & Phase 2 — Item 3):
    • Confirmation of whether the existing snarf / Katzir476! credential (from setup_storage_cluster.sh) is valid across the Debian/RPi nodes, plus SSH/DSM credentials for Synology NAS (14.10.10.71 / 14.10.10.72) and Harvester nodes (140.44.4.81-83) so we can enroll the unified id_ed25519_snarfnet key.
Reference in New Issue View Git Blame Copy Permalink