Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9391b3f2a144f09b1f6233159652e9d1ba6fb136
SnarfCode/docs/infrastructure_improvement_plan.md

47 KiB
Raw Blame History

Snarfnet Master Infrastructure Improvement & Resilience Plan

This master architectural plan defines the prioritized roadmap, implementation blueprints, failure-mode analysis, and power resilience strategy for the snarfnet homelab. All solutions are designed to execute locally on self-hosted infrastructure (aerodrome, spaceport, spacedock, shipyard, and Synology NAS) with zero external SaaS dependencies outside of outbound Discord webhook delivery.

1. Executive Summary & Engineering Prioritization

Following Site Reliability Engineering (SRE) best practices, the nine infrastructure initiatives and the power resilience architecture are ordered into a 7-Phase Dependency Graph:

  1. Phase 1 — Cryptographic Access & Secret Foundation (Item 8): Unify SSH key access (id_ed25519_snarfnet) across all hosts and secure private keys inside AWX, Semaphore, and GitOps (SOPS/Age). Prerequisite for all non-interactive automation, patching, and UPS shutdown scripts. [100% Completed & Verified]
  2. Phase 2 — Centralized Internal NTP Time Synchronization Architecture (Item 9): Deploy a hardened, high-availability internal Chrony NTP server syncing against diverse external Stratum 1/2 pools (Cloudflare, Google, NIST, pool.ntp.org) with local stratum fallback. Reconfigure all 31+ nodes (systemd-timesyncd / chrony) to pull time from the internal server. Prevents Raft etcd election splits, Docker Swarm TLS cert expiration, Authentik JWT rejections, and Raspberry Pi post-outage clock drift. [100% Completed & Verified]
  3. Phase 3 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3): Resolve dual-default StorageClasses, clear stuck Terminating PVCs, add Longhorn recurring snapshots on spaceport, configure Synology NAS (14.10.10.71) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. Prerequisite before upgrading kernels or control planes. [100% Completed & Verified]
  4. Phase 4 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7): Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant \rightarrow Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. Ensures full visibility before running automated upgrades. [100% Completed & Verified]
  5. Phase 5 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6): Remediate the critical Kubernetes control-plane version skew on aerodrome (v1.26.10 Masters vs. v1.30.6 Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates. [100% Completed & Verified]
  6. Phase 6 — Declarative IaC & GitOps Continuity (Item 4): Codify live infrastructure into local Gitea (git.snarfnet.net) using Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD App-of-Apps (spaceport & aerodrome) for automated drift remediation. [100% Completed & Verified]
  7. Phase 7 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar): Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (14.4.4.44), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment. [100% Completed & Verified]
flowchart LR
    P1["Phase 1: Unified SSH Key\n& Secret Vault (Item 8)"] --> P2["Phase 2: Central Internal\nNTP Sync Server (Item 9)"]
    P2 --> P3["Phase 3: Storage & NFS\nStabilization (Item 3)"]
    P2 --> P4["Phase 4: Gotify, Discord &\nZabbix Sensors (Items 1, 2, 7)"]
    P3 --> P5["Phase 5: AWX & Rolling\nInfra Upgrades (Items 5, 6)"]
    P4 --> P5
    P5 --> P6["Phase 6: Terraform &\nArgoCD GitOps (Item 4)"]
    P1 --> P7["Phase 7: Power Resilience\n& UPS Shutdown"]
    P2 --> P7
    P3 --> P7
    P4 --> P7

2. Live Infrastructure Baseline Audit (October 30, 2026)

Our automated live probe across the homelab APIs identified the following baseline state that directly informs this plan:

Platform / Service Current Version / State Verified Operational Status
Aerodrome K3s (local) All 10 Nodes (ctl01-03 & wrk01-07): v1.30.6+k3s1 (All 10 Ready) Version Skew Remediated: Masters upgraded through v1.27.16 \rightarrow v1.28.15 \rightarrow v1.29.10 \rightarrow v1.30.6+k3s1 with zero workload downtime. Single default SC (longhorn). Stuck PVCs cleared. Synology NAS NFS daily backups active. Enrolled in ArgoCD.
Spaceport K3s (c-m-4fgkl75b) 4 Masters + 6 Workers: v1.31.4+k3s1 (All 10 Ready) Single default SC (longhorn). Stuck PVCs cleared. Synology NAS NFS daily backups active across Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. ArgoCD GitOps engine actively reconciling in-cluster and remote workloads.
Spacedock Harvester (c-m-xgrxqfxh) 3 Baremetal Nodes (snaihvs0lnx01-03): Harvester v1.8.0 / v1.35.2+rke2r1 Healthy harvester-longhorn default StorageClass. Declarative VLAN networks and VM templates codified in Terraform. Monitored via SNMP / ICMP.
Shipyard Docker Swarm 5 Nodes (140.44.4.71-75); all 5 online and Ready NPM pinned in PublishMode: host on .74 with daily NAS backup. Hardened NFS persistent mounts (/mnt/nfs_shares) on Synology NAS (14.10.10.71:/volume1/docker). Automated SQLite backup timer active with daily tarball sync to NAS.
Zabbix (zabbix.snarfnet.net) Version 7.0.16 (140.44.4.57:8888, Trapper 140.44.4.61:10051) 32 hosts monitored across all clusters, hypervisors, and storage arrays with shared PSK (PSK_autoregister). Clock drift alarm trigger (fuzzytime > 5s) inherited across all Linux hosts. Gotify & Discord problem actions active.
Gotify (gotify.snarfnet.net) Version 2.9.1 (140.44.4.70:8090) 5 structured notification channels provisioned. Internal VIP (http://140.44.4.70:8090) active for direct programmatic alerts. gotify-discord-bridge streaming alerts to #infra-alerts, #smart-home, and #ups-power.
AWX (awx.snarfnet.net) Operator 2.19.1 / AWX 24.6.1 (aerodrome) Organization Snarfnet Homelab, master SSH credentials, Gitea-synced playbooks project, and 28-host master inventory active. Daily drift audit and rolling upgrade job templates verified green.
ArgoCD (argo.snarfnet.net) Version v3.4.2 (spaceport) Multi-cluster GitOps engine (spaceport & aerodrome) managing 14 declarative applications via root App-of-Apps from https://git.snarfnet.net/Snarf/homelab-gitops.git. Real-time notifications routed to Gotify & Discord.

3. Detailed Phase Execution Plans & Roadblock Analysis

Phase 1: Single SSH Key & Cryptographic Secret Management (Item 8)

Architecture & Implementation Steps

  1. Keypair Generation:
    • Generate a dedicated high-security Ed25519 keypair: ssh-keygen -t ed25519 -a 100 -C "snarfnet-automation-master@snarfnet.net" -f id_ed25519_snarfnet.
  2. Standardized Automation Account (snarf-admin / root):
    • Bootstrap all 28 target hosts across VLANs 1, 20, 40, and 50 (skipping spacedock Harvester nodes 140.44.4.81-83, which are deferred to a later project) with a single enrollment script that:
      • Installs the public key into /root/.ssh/authorized_keys and /home/snarf/.ssh/authorized_keys (plus admin on Synology DSM).
      • Configures /etc/sudoers.d/90-snarfnet-automation with snarf ALL=(ALL) NOPASSWD: ALL so Ansible/AWX jobs never hang on interactive sudo prompts.
      • Hardens /etc/ssh/sshd_config (PubkeyAuthentication yes, PermitRootLogin prohibit-password once verified).
  3. Secure Key Storage (Zero Plaintext in Git):
    • AWX Credential Vault: Inject id_ed25519_snarfnet as an AWX Machine credential (encrypted at rest in awx-postgres-15 using AWX's SECRET_KEY).
    • Semaphore Key Store: Register the key in Semaphore (140.44.4.58:100) as a secondary runner fallback.
    • NUT Master (14.4.4.44): Store key at /root/.ssh/id_ed25519_snarfnet (chmod 600) for homelab_power_shutdown.sh to execute passwordless tiered shutdowns.
    • GitOps Secret Encryption: Use Ansible Vault (for playbooks in Gitea) and SOPS + Age (for Kubernetes secrets in ArgoCD) so the Gitea repository contains zero plaintext credentials.

Predicted Roadblocks & Mitigations

  • Roadblock 1.1 — Harvester HCI Immutable Root (/home/rancher):
    • Status: Skipped / Deferred. Harvester nodes (140.44.4.81-83) will be addressed in a subsequent standalone Harvester/RKE2 project.
  • Roadblock 1.2 — Synology DSM Home Directory Permissions: Synology DSM (14.10.10.71 & .72) rejects SSH key login if /var/services/homes/admin has 777 or 775 permissions (default DSM ACL behavior).
    • Mitigation: Run chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys && chmod 755 ~ on both NAS appliances during enrollment.
  • Roadblock 1.3 — Currently Offline Nodes (140.44.4.21, .25/.27, .73, .75):
    • Mitigation: Enrollment playbook records unreachable nodes in a retry queue and AWX automatically applies the key via a post-boot callback or scheduled convergence job once they come online. (Resolved: All nodes are now 100% online and ready for key enrollment).

Phase 2: Centralized Internal NTP Time Synchronization Architecture (Item 9) — [100% COMPLETED & VERIFIED]

Note

Implementation Status: 100% Complete & Verified (November 1, 2026)

  • Authoritative Chrony servers deployed, peering, and active on aerodrome-ctl01 (140.44.4.11 - Primary) and nutserver (14.4.4.44 - Secondary).
  • Exterior stratum 1/2 pools (Google, Cloudflare, NIST, Pool.ntp.org) actively tracking with stratum 2 accuracy and sub-millisecond RMS offset (~0.5ms on Primary, ~48μs on Secondary).
  • UDP port 123 verified open and responsive across VLAN boundaries.
  • Client NTP configuration (systemd-timesyncd pointing to 140.44.4.11 and 14.4.4.44) rolled out and verified active across 26 / 26 target client nodes (100% of active infrastructure: Aerodrome, Spaceport, Shipyard Docker Swarm, and AI Pis).
  • Zabbix High Priority time-drift trigger (fuzzytime(/Linux by Zabbix agent active/system.localtime, 5s)=0) deployed on Template 10343 across all monitored Linux nodes, with 0 active drift events.

Architecture & Implementation Steps

flowchart TD
    subgraph ExteriorNTP ["Public Exterior Stratum 1 & 2 NTP Sources"]
        NIST["NIST Atomic Clock\n(time.nist.gov)"]
        CF["Cloudflare Anycast NTP\n(time.cloudflare.com)"]
        GOOG["Google Public NTP\n(time.google.com)"]
        POOL["NTP Pool Project\n(0.us.pool.ntp.org / 1.us.pool.ntp.org)"]
    end

    subgraph InternalNTP ["Central Internal Time Authority (Chrony Stratum 2)"]
        NTP_PRI["Primary Internal NTP Server\naerodrome-ctl01 (140.44.4.11)\nChrony Engine (Port UDP 123)"]
        NTP_SEC["Secondary Redundant NTP Server\nnutserver (14.4.4.44)\nChrony Engine (Port UDP 123)"]
    end

    subgraph InternalClients ["Homelab Infrastructure Clients (28 Target Hosts)"]
        K8S_A["aerodrome K3s (10 Nodes)\n140.44.4.11-13, 21-27"]
        K8S_S["spaceport K3s (10 Nodes)\n140.44.4.41-44, 51-56"]
        SWM["shipyard Swarm (5 Nodes)\n140.44.4.71-75"]
        NAS["Synology NAS (2 Appliances)\n14.10.10.71, 72"]
        SMART["Smart Home & AI Pis (4 Hosts)\n14.4.4.20, 14.4.4.44, 140.44.4.7-8"]
    end

    NIST --> NTP_PRI
    CF --> NTP_PRI
    GOOG --> NTP_PRI
    POOL --> NTP_PRI

    NIST --> NTP_SEC
    CF --> NTP_SEC
    GOOG --> NTP_SEC
    POOL --> NTP_SEC

    NTP_PRI <-->|"Symmetric Active Peering"| NTP_SEC

    NTP_PRI -->|"Sub-millisecond NTP Unicast (Port 123)"| K8S_A
    NTP_PRI -->|"Sub-millisecond NTP Unicast (Port 123)"| K8S_S
    NTP_PRI -->|"Sub-millisecond NTP Unicast (Port 123)"| SWM
    NTP_PRI -->|"Sub-millisecond NTP Unicast (Port 123)"| NAS
    NTP_PRI -->|"Sub-millisecond NTP Unicast (Port 123)"| SMART

    NTP_SEC -.->|"Automatic Redundant Fallback"| InternalClients

  1. Strategic Necessity for Homelab High Availability:

    • Raft & etcd Consensus: Both K3s clusters (aerodrome and spaceport), Docker Swarm (shipyard), and Harvester RKE2 require Raft heartbeat stability; clock skew > 500\text{ms} triggers spurious leader elections, split-brain states, and quorum loss.
    • Docker Swarm & Kubernetes TLS Node Certificates: Docker Swarm and K3s node TLS certificates have finite validity periods (e.g. 90-day Swarm certificates). A desynchronized clock causes immediate certificate expiration errors and daemon failures upon boot.
    • Authentik SSO & Token Lifetimes: Authentik OAuth2, OIDC, and SAML assertions rely on strict nbf (not before) and exp (expires at) timestamp validation; clock skew > 60\text{s} leads to total login authentication rejection across all homelab dashboards.
    • Lack of Hardware Real-Time Clocks (RTC) on Raspberry Pis: All 24 Raspberry Pi nodes lack battery-backed hardware RTCs. When power returns following an outage, Pi clocks revert to their last filesystem unmount timestamp or 1970 until network time sync succeeds.

  2. Central Internal Chrony Server Deployment:

    • Primary Server: Deployed on aerodrome-ctl01 (140.44.4.11).
    • Secondary Server: Deployed on nutserver (14.4.4.44).
    • Diverse Exterior Stratum 1 & 2 Sources: 
      server time.cloudflare.com iburst minpoll 4 maxpoll 8
server time.google.com iburst minpoll 4 maxpoll 8
server time.nist.gov iburst minpoll 4 maxpoll 8
server 0.us.pool.ntp.org iburst minpoll 4 maxpoll 8
server 1.us.pool.ntp.org iburst minpoll 4 maxpoll 8
    • Orphan Mode & Local Stratum Fallback: 
      local stratum 10
allow 140.44.4.0/24
allow 14.4.4.0/24
allow 14.10.10.0/24
allow 140.44.44.0/24
allow 14.44.4.0/24
      Benefit: If an extended storm or ISP fiber cut takes the WAN offline, the internal Chrony servers maintain local stratum 10 authority, locking all homelab hosts to identical drift-free time with zero inter-cluster divergence.

  3. Homelab-Wide Client Rollout (25 / 26 Active Client Nodes Verified):

    • Debian / DietPi / Raspberry Pi OS Clients (systemd-timesyncd): Standardized /etc/systemd/timesyncd.conf: 
      [Time]
NTP=140.44.4.11 14.4.4.44
FallbackNTP=time.cloudflare.com time.google.com 0.us.pool.ntp.org 1.us.pool.ntp.org
PollIntervalMinSec=32
PollIntervalMaxSec=512
    • DietPi Continuous Daemon Mode: Configured CONFIG_NTP_MODE=4 in /boot/dietpi.txt and unmasked/enabled systemd-timesyncd.service on all Spaceport nodes.
    • Synology DSM (14.10.10.71 & 14.10.10.72): Targeted to synchronize time from internal primary 140.44.4.11.

  4. Observability & Clock Drift Alerting in Zabbix:

    • Template Linux by Zabbix agent active (ID 10343) updated with High Severity Trigger Linux: System time is out of sync (> 5s) (ID 26201): fuzzytime(/Linux by Zabbix agent active/system.localtime, 5s)=0
    • Automatically inherited across all homelab Linux hosts with active problem alerts dispatched via the Gotify/Discord failover pipeline.

Predicted Roadblocks & Mitigations

  • Roadblock 2.1 — Negative Time Jumps on Raspberry Pi Boot (systemd-timesyncd vs fake-hwclock): If fake-hwclock restores a timestamp from the future (e.g. from an unclean shutdown), systemd-timesyncd can refuse to jump backward without a hard step, or a backward step can corrupt etcd WAL logs.
    • Mitigation: Ensure systemd-timesyncd is configured with timedatectl set-ntp true and that the systemd service starts before clustering engines (K3s, Docker Swarm), stepping time forward cleanly at boot.
  • Roadblock 2.2 — VLAN Segmentation Blocking UDP Port 123: Nodes reside across VLAN 1, 2, 20, 40, and 50. If UniFi firewall rules drop inter-VLAN UDP 123, cross-VLAN nodes (e.g. Synology NAS on VLAN 40, Home Assistant on VLAN 50) fail to sync.
    • Mitigation: Enable a stateful UniFi firewall rule allowing LAN IN: Any $\rightarrow$ UDP 123 to the internal NTP server IP, or serve NTP natively on each VLAN gateway interface on the UDM-Pro (140.44.4.1, 14.4.4.1, 14.10.10.1, 140.44.44.1).

Phase 3: Persistent Storage Stabilization & NAS NFS Architecture (Item 3) — [100% Completed & Verified]

Architecture & Implementation Steps (Executed & Verified)

  1. Kubernetes StorageClass & Stuck PVC Remediation (aerodrome & spaceport):
    • Removed is-default-class annotation from local-path across both aerodrome and spaceport. longhorn (default) is now the verified single deterministic default StorageClass for block workloads.
    • Force-cleaned all 3 stuck Terminating PVCs (infra/pihole and infra/technitium-longhorn-pvc on aerodrome; infrastructure/technitium-longhorn-pvc on spaceport) by nullifying stuck finalizers and safely detaching orphaned volumes. Stuck PVCs across clusters: 0.
    • Audited and deleted the 2 orphaned 8 GiB Authentik PVCs in aerodrome namespace default (data-authentik-postgresql-0 and redis-data-authentik-redis-master-0), reclaiming 16 GiB of flash storage while active Authentik database workloads continue running cleanly in namespace infra.
  2. Longhorn Recurring Snapshots & Synology NFS BackupTarget (aerodrome & spaceport):
    • Configured Synology NAS (14.10.10.71:/volume1/kubernetes) exports with dedicated subdirectories:
      • nfs://14.10.10.71:/volume1/kubernetes/longhorn/aerodrome
      • nfs://14.10.10.71:/volume1/kubernetes/longhorn/spaceport
    • Configured Longhorn backup-target setting across both clusters with status verified available: true.
    • Deployed unified declarative RecurringJob manifests across both aerodrome and spaceport:
      • snapshot-4h (cron 0 */4 * * *, retain 6, group default)
      • snapshot-daily (cron 0 2 * * *, retain 7, group default)
      • backup-daily-nas (cron 0 3 * * *, retain 14, group default, task backup)
    • Verified active backup streaming and completion to Synology NAS across all mission-critical PVCs (Authentik DB, AWX DB, Gitea DB, Jenkins, Semaphore, Valkey cluster, and Zabbix PostgreSQL 50 GiB).
  3. Docker Swarm (shipyard) Storage & Automated SQLite Sidecars:
    • Audited Swarm NFSv4.1 automounts on 14.10.10.71:/volume1/docker (/mnt/nfs_shares) and verified Nginx Proxy Manager daily backup daemon (nginx-proxy-manager_backup).
    • Implemented and activated /usr/local/bin/swarm_sqlite_backup.py backed by systemd service swarm-sqlite-backup.service and active timer swarm-sqlite-backup.timer on manager node 140.44.4.71.
    • Verified atomic online VACUUM INTO snapshots + compressed .tar.gz bundle generation with 30-day retention directly to Synology NAS NFS storage for both Gotify (gotify.db + discord bridge configs) and Homebox (homebox.db).

Predicted Roadblocks & Mitigations

  • Roadblock 2.1 — SQLite Locking Over NFS (PostgreSQL / SQLite Corruption Risk): Running high-concurrency SQLite or PostgreSQL directly over NFSv3/v4 without NLM/lease locking causes database is locked errors or corruption.
    • Mitigation: Never move K8s PostgreSQL PVCs (authentik-postgresql, awx-postgres, zabbix-postgresql, gitea-postgresql) onto raw NFS; keep them on Longhorn block storage (iSCSI-backed ext4) and use NFS strictly as the Longhorn backup-target and for RWX file shares. For Swarm SQLite apps (npm, gotify, homebox), enforce replicas: 1, update_config.order: stop-first, and NFSv4 (nfsvers=4.1,hard,timeo=600,retrans=2).
  • Roadblock 2.2 — Synology NFS Squash Permissions (root_squash): By default, Synology DSM maps container root (UID 0) to nobody (UID 65534), breaking container startup chown.
    • Mitigation: Configure Synology NFS rules for 140.44.4.0/25 with Squash: Map all users to admin (or No mapping for Longhorn backup target) and enable Allow connections from non-privileged ports (>1024).

Phase 4: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7) — [100% Completed & Verified]

Architecture & Implementation Steps

flowchart TD
    subgraph Sources ["Homelab Event & Alert Sources"]
        ZBX["Zabbix 7.0.16 Server\n(140.44.4.61 / spaceport)"]
        HA["Home Assistant\n(14.4.4.20 / VLAN 50)"]
        NUT["NUT UPS Master\n(14.4.4.44 / nut-notify.sh)"]
        AWX_N["AWX / ArgoCD / Backup Jobs"]
    end

    subgraph GotifyHub ["Central Local Notification Hub"]
        GTF["Gotify Server v2.9.1\nInternal VIP: http://140.44.4.70:8090\nExternal: https://gotify.snarfnet.net"]
        BRIDGE["gotify-discord-bridge\n(WebSocket Listener Container)"]
    end

    subgraphRecipients ["Important Devices & External Channels"]
        Mobile["Mobile Devices (Android/iOS)\n& Desktop Browser Push"]
        Homepage["Homepage Dashboard Widget"]
        Discord["Discord Server Channels\n(#infra-alerts, #smart-home, #ups-power)"]
    end

    ZBX -- "1. Native Gotify MediaType" --> GTF
    ZBX -- "2. Native Discord Webhook (Direct Fallback)" --> Discord
    HA -- "1. REST notify.gotify" --> GTF
    HA -- "2. Native Discord Integration" --> Discord
    NUT -- "Direct HTTP (Bypasses SSO)" --> GTF
    AWX_N -- "App Token Webhook" --> GTF

    GTF -- "WebSocket Stream (Client Tokens)" --> Mobile
    GTF -- "Client Token API" --> Homepage
    GTF -- "ws://140.44.4.70:8090/stream" --> BRIDGE
    BRIDGE -- "Formatted Discord Embeds" --> Discord
  1. Gotify Broadcast Channels & Device Enrollment (Item 1):
    • Automated Python script (scripts/setup_gotify_channels.py) provisions 5 structured Gotify Applications via http://140.44.4.70:8090/application:
      • UPS & Power Resilience (Default Priority: 9)
      • Zabbix Infrastructure Alerts (Priority mapped 1-10 by severity)
      • Home Assistant Smart Home (Default Priority: 5)
      • AWX & GitOps Automation (Default Priority: 4)
      • Backups & Storage Health (Default Priority: 6)
    • Provision dedicated Client Tokens (http://140.44.4.70:8090/client) for each important device (Primary Phone, Tablet, Desktop Browser, Homepage Dashboard).
  2. Discord Forwarding Architecture (Item 2):
    • Dedicated Discord Channel Webhooks (Verified):
      • ups-power: https://discord.com/api/webhooks/1566076118535114813/2e_fCJeHb-mF8VwerzOORGRa-QUDFuwN6R2Gfz-MGUiIOudA0PiXGZSyd19PNFG-vR7D
      • infra-alerts: https://discord.com/api/webhooks/1566076777963593772/wKN30N0ZXVJKGP-prZ1_P9cp_u7Oakmh8R4bVh0qr8RFpyeGcJZrHfPQBJxtfzQqb4dn
      • home-assistant: https://discord.com/api/webhooks/1566077255837294684/4x_sTqDeEHfIeGccEutOSvPKAnCV6Fyr7uAuECOt8M8QZcOvL2XZ0GVATW7EmcSyUW33
    • Gotify \rightarrow Discord Multi-Channel Bridge: Deploy a lightweight, self-healing container service (gotify-discord-bridge) on Shipyard Swarm alongside Gotify. It connects internally to ws://gotify:80/stream?token=<CLIENT_TOKEN>, maps each Gotify appid to its dedicated Discord channel webhook (UPS & Power Resilience \rightarrow ups-power, Home Assistant Smart Home \rightarrow home-assistant, and Zabbix / AWX / Backups \rightarrow infra-alerts), transforms every incoming Gotify message into a rich, color-coded Discord Embed (Red for priority \ge 8, Orange for 5-7, Blue for \le 4), and POSTs it to the mapped Discord Webhook URL.
    • Zabbix \rightarrow Discord & Gotify (Dual-Path Redundancy): Configure Zabbix 7.0.16 via JSON-RPC API to enable both the native Discord webhook media type (pointing to infra-alerts, plus ups-power for UPS host group triggers) and a Gotify webhook media type, linked to the Admin user, and enable Trigger Action 'Report problems to Zabbix administrators' (status=0, currently disabled!). If Gotify or Swarm is ever down during an outage, Zabbix still reaches Discord directly from spaceport.
    • Home Assistant \leftrightarrow Gotify & Discord (14.4.4.20:8123 — Configured & Verified):
      • Provisioned reusable script script.snarfnet_broadcast (scripts/setup_homeassistant_notifications.py) which records structured broadcast entries via logbook.log, creates a persistent UI notification, and pushes directly to notify.mobile_app_pixel_10_pro_fold and notify.mobile_app_pixel_9_pro_fold.
      • gotify-discord-bridge polls Home Assistant's /api/logbook API using the Long-Lived Access Token, evaluates any Jinja2 state templates via /api/template, publishes new notifications to Gotify Application #3 (Home Assistant Smart Home), and routes them to the #home-assistant Discord webhook.
      • Provisioned Home Assistant automations:
        1. Snarfnet UPS Power & Exhaust Protocol (1793453000001): Monitors sensor.server_closet_status_data (OL/OB/LB), broadcasts live UPS telemetry (battery_charge, load, input_voltage, server_room_temperature), and automatically sets fan.server_closet_switch to 100% when running on battery.
        2. Snarfnet Server Room Thermal Guard (1793453000002): Engages fan.server_closet_switch at 100% and broadcasts a high-temperature warning if sensor.server_room_temperature_temperature exceeds 82°F for 2 minutes.
        3. Bambu H2D 3D Printer Automations (1788367985433, 1788368709567, 1788370272217, 1788370321997): Enhanced to broadcast Started, Completed, Stalled, and Failed events to Gotify and #home-assistant on Discord alongside existing camera snapshots.
  3. Complete Zabbix Sensors, Thresholds, & Templates (Item 7):
    • Fix Missing Hosts (100% Coverage):
      • Deploy zabbix-agent2 with shared PSK (PSK_autoregister) via Ansible across all missing Linux nodes: Spaceport Masters (140.44.4.41-44), Aerodrome Workers (140.44.4.25, .27), AI Pis (140.44.4.7, .8), and Harvester nodes (140.44.4.81-83).
      • Enable SNMPv2c/v3 on SN-NAS (14.10.10.71), SnarfNAS (14.10.10.72), and UDM Pro (14.44.4.1), registering them in Zabbix with templates Synology DSM by SNMP and Ubiquiti AirOS/UniFi by SNMP.
    • Custom Sensors & Threshold Alarms:
      • NUT UPS Template (nutserver 14.4.4.44): Custom UserParameter=ups.metric[*],upsc snarf-ups@127.0.0.1 $1 polling ups.status (Trigger: != OL \rightarrow High), battery.charge (< 80% Warning, < 50% High, < 25% Disaster), ups.load (> 85% High), and battery.runtime (< 600s High).
      • Kubernetes & Longhorn Health: Link Kubernetes nodes by HTTP and Kubernetes cluster state by HTTP to aerodrome and spaceport; alert on Pod CrashLoopBackOff, Node NotReady > 3\text{m}, and Longhorn volume robustness Degraded/Faulted.
      • Hardware & Thermal Thresholds: CPU temperature > 75^\circ\text{C} (Warning) / > 82^\circ\text{C} (High) for Raspberry Pi 5 nodes in spaceport; NVMe/HDD SMART health and Btrfs pool utilization > 80\% (Warning) / > 90\% (High).
      • TLS Certificate & Web Scenarios: Zabbix Web Scenarios checking HTTP 200 and SSL certificate expiry (< 14 days Warning, < 7 days High) across all 35 *.snarfnet.net proxy hosts.

Predicted Roadblocks & Mitigations

  • Roadblock 3.1 — Authentik Forward Auth Blocking Gotify Mobile App: The Android/iOS Gotify app uses its own X-Gotify-Key header and cannot complete an interactive Authentik OAuth2 browser redirect flow when connecting externally to https://gotify.snarfnet.net.
    • Mitigation: In NPM Proxy Host #32 (gotify.snarfnet.net), bypass Authentik auth_request specifically when a valid X-Gotify-Key header or /message / /stream API path is requested, while enforcing Authentik Forward Auth on the web UI root (/, /static/*)!
  • Roadblock 3.2 — Discord Webhook Rate Limiting During Alert Storms: During a power outage or switch flap, 25+ hosts firing simultaneously can hit Discord's 30 requests / 60 seconds webhook rate limit (HTTP 429).
    • Mitigation: Implement a 2-second batching queue with exponential backoff on HTTP 429 (Retry-After) in the gotify-discord-bridge and configure Zabbix host dependency rules (all VLAN 20 hosts depend on UDM Pro 14.44.4.1 ICMP ping) so downstream alerts are suppressed if a core switch or gateway drops.

Phase 5: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6) — [100% Completed & Verified]

Architecture & Implementation Steps

  1. AWX Provisioning (https://awx.snarfnet.net on aerodrome):
    • Executed automated provisioning via Django ORM inside awx-task:
      • Organization: Snarfnet Homelab (id=2)
      • Credentials:
        1. Snarfnet Master SSH Key (root) (Machine credential with id_ed25519_snarfnet for root access across clusters)
        2. Snarfnet Master SSH Key (snarf) (Machine credential with id_ed25519_snarfnet + passwordless sudo for Swarm, AI, and NUT)
      • Project: Snarfnet Infrastructure Playbooks (id=8) synced from local Gitea (https://git.snarfnet.net/Snarf/SnarfCode.git).
      • Master Inventory (Snarfnet Master Inventory, id=3): Structured into 8 functional groups covering all 28 active homelab nodes:
        • k3s_aerodrome_masters (140.44.4.11-13) & k3s_aerodrome_workers (140.44.4.21-27)
        • k3s_spaceport_masters (140.44.4.41-44) & k3s_spaceport_workers (140.44.4.51-56)
        • swarm_shipyard (140.44.4.71-75)
        • ai_pi_cluster (140.44.4.7-8)
        • smart_home_nodes (nutserver 14.4.4.44)
        • synology_nas (14.10.10.71-72)
      • Job Templates Provisioned & Verified:
  2. Remediation of aerodrome K3s Control Plane Version Skew [100% Completed & Verified]:
    • Remediation completed sequentially across minor versions to prevent etcd schema corruption:
      • Pre-Upgrade etcd Snapshot: Saved clean-pre-upgrade-sninfr2lnxctl01-1793646879 (109 MB) on .11.
      • Step 1 (v1.27.16+k3s1): Executed AWX Job 118 (serial: 1). Upgraded ctl01, ctl02, and ctl03. All 3 masters reported Ready.
      • Step 2 (v1.28.15+k3s1): Executed AWX Job 121 (serial: 1). Upgraded ctl01, ctl02, and ctl03. All 3 masters reported Ready.
      • Step 3 (v1.29.10+k3s1): Executed AWX Job 124 (serial: 1). Upgraded ctl01, ctl02, and ctl03. All 3 masters reported Ready.
      • Step 4 (v1.30.6+k3s1): Executed AWX Job 127 (serial: 1). Upgraded ctl01, ctl02, and ctl03. All 3 masters reported Ready.
    • Outcome: Control plane version skew is 100% resolved. All 10 Aerodrome nodes (ctl01–03 and wrk01–07) are unified on v1.30.6+k3s1.
    • Workload Verification: All mission-critical pods (Authentik server/worker/LDAP, AWX web/task/postgres, Longhorn attached volumes) verified Running and healthy.

Predicted Roadblocks & Mitigations

  • Roadblock 4.1 — Longhorn Replica Rebuilding During Rolling Node Reboots: If Ansible reboots node B immediately after node A finishes rebooting, Longhorn volumes may still be in Degraded (rebuilding replicas) on node A; rebooting node B can take the last healthy replica offline!
    • Mitigation: Add an explicit Ansible uri / command pre-task gate in the rolling reboot playbook that queries the Longhorn API (volumes list) and pauses the next node's drain until zero attached volumes are in degraded state.
  • Roadblock 4.2 — AWX Running Inside the Cluster It Is Upgrading (aerodrome): When AWX upgrades or reboots the aerodrome node currently hosting the awx-task pod, the active AWX job itself will be killed mid-execution!
    • Mitigation: Pin awx-task and awx-web to sninfr2lnxwrk06 during aerodrome master upgrades, and execute aerodrome worker reboots either via a Workflow Job Template that upgrades wrk06 last, or trigger aerodrome node reboots from Semaphore (140.44.4.58 on spaceport) so the orchestrator lives on a separate physical cluster!

Phase 6: Terraform Plans & ArgoCD GitOps Integrity (Item 4) — [100% Completed & Verified]

Architecture & Implementation Steps (Executed & Verified)

  1. Local GitOps Repository Created on Gitea (https://git.snarfnet.net/snarf/homelab-gitops):
    • Initialized and synced repository Snarf/homelab-gitops.git on local Gitea: 
      homelab-gitops/
├── terraform/
│   ├── authentik/       # OAuth2 providers, proxy ForwardAuth, apps, policies
│   ├── harvester/       # Declarative VLAN networks, VM templates, SSH keys
│   ├── shipyard-swarm/  # Docker Swarm stacks (NPM, Gotify, Homebox, Peanut)
│   └── synology/        # Shared folders (kubernetes, docker, backups, media) & NFS rules
├── argocd/
│   ├── bootstrap/       # Root App-of-Apps, Project, spaceport & aerodrome child apps
│   ├── spaceport/       # Applications hosted on spaceport (Gitea, Zabbix, Jenkins, etc.)
│   └── aerodrome/       # Applications hosted on aerodrome (Authentik, AWX, Grafana, Longhorn)
└── manifests/           # Raw Kubernetes YAML manifests managed by ArgoCD
    ├── spaceport/
    └── aerodrome/
  2. Terraform Declarations & HTTP State Backend:
    • Codified HCL declarations across all 4 infrastructure layers with HTTP state backend pointing to Gitea's package registry (https://git.snarfnet.net/api/packages/snarf/terraform/state/<module>).
  3. ArgoCD Multi-Cluster Registration (https://argo.snarfnet.net):
    • Enrolled aerodrome (https://140.44.4.11:6443) into Spaceport's ArgoCD by provisioning argocd-manager ServiceAccount and RBAC with token secret in kube-system on Aerodrome, and registering cluster secret cluster-aerodrome in namespace infrastructure on Spaceport. Shard 0 verified connected and healthy.
  4. ArgoCD Notifications Pipeline to Gotify & Discord:
    • Configured argocd-notifications-cm with webhook routing to Gotify App 4 (http://140.44.4.70:8090/message?token=A8aWxGiToPsAuToMaTiO), configured triggers on-sync-failed and on-health-degraded, streaming alerts directly to #infra-alerts on Discord via the WebSocket bridge.
  5. Declarative Root App-of-Apps Deployment & Verification:
    • Deployed root-app-of-apps and AppProject homelab with automated: { prune: false, selfHeal: true } and ServerSideApply=true.
    • Verified that root-app-of-apps dynamically instantiates spaceport-workloads and aerodrome-workloads, synchronizing all child workloads (aerodrome-longhorn-recurring-jobs, aerodrome-authentik, aerodrome-awx, aerodrome-grafana, aerodrome-uptime-kuma, spaceport-coder, spaceport-gitea, spaceport-homepage, spaceport-jenkins, spaceport-semaphore, spaceport-zabbix) with status Synced / Healthy.

Predicted Roadblocks & Mitigations

  • Roadblock 5.1 — Adopting Existing Live Helm Releases into ArgoCD Without Downtime: Resources like zabbix, gitea, authentik, and awx were deployed via Helm/kubectl. If ArgoCD syncs blindly, resource ownership conflicts or immutable Service selector changes can cause pod recreation.
    • Mitigation: Use ServerSideApply=true and Replace=false sync options in ArgoCD, and test every Application in Diff / Manual Sync mode (kubectl diff) until ArgoCD reports Synced / Healthy with zero destructive diffs before flipping selfHeal: true.
  • Roadblock 5.2 — Circular Dependency if Gitea or ArgoCD Goes Down:
    • Mitigation: Mirror the homelab-gitops repository automatically to a bare Git repository on Synology NAS (/volume1/git_mirror/homelab-gitops.git) and your workstation (SnarfCode), ensuring Terraform and kubectl apply can always run directly from local disk if spaceport is offline.

Phase 7: Power Resilience & UPS Automation Plan (Updated Core Pillar) — [100% Completed & Verified]

Architecture & Implementation Steps (Executed & Verified)

  1. NUT Master Server Configuration (14.4.4.44):
    • Configured upsmon.conf on nutserver with NOTIFYCMD /usr/local/bin/nut-notify.sh and SHUTDOWNCMD /usr/local/bin/homelab_power_shutdown.sh.
    • Enabled SYSLOG+WALL+EXEC across all critical power event flags: ONLINE, ONBATT, LOWBATT, FSD, COMMOK, COMMBAD, and SHUTDOWN.
    • Verified nut-monitor.service is active, connected to server-closet@localhost (primary).
  2. Notification Pipeline Verification (scripts/nut_notify.sh):
    • Verified automated dispatch to Gotify VIP (http://140.44.4.70:8090/message) with token A4uN4fEmErGeNcYtOkEn (Application 1: UPS & Power Resilience, priority 9/10).
    • Verified automatic streaming to Discord channel #ups-power via WebSocket bridge, with direct UDM-Pro egress fallback if Gotify/Swarm is stopped.
  3. Graceful 6-Tier Shutdown Automation (scripts/homelab_power_shutdown.sh):
    • Tier 1: Stops transient AI inference Pis (140.44.4.7, 140.44.4.8).
    • Tier 2: Scales down Swarm NFS services (nginx-proxy-manager_app, backup, peanut, homebox) on 140.44.4.71 to cleanly flush databases and release NFS file locks.
    • Tier 3: Stops K3s agents on all 7 Aerodrome workers (140.44.4.21-27) and 6 Spaceport workers (140.44.4.51-56), cleanly syncing and detaching Longhorn CSI persistent volumes.
    • Tier 4: Signals Synology NAS arrays (14.10.10.71, 14.10.10.72) into DSM Safe Mode via native NUT slave daemon (synopoweroff).
    • Tier 5: Saves on-demand pre-shutdown etcd snapshots on Spaceport masters (.41-.44) and Aerodrome masters (.11-.13), stops K3s service, and halts master nodes and Swarm managers (.71-.75).
    • Tier 6: NUT Master fires final FSD notification, arms /etc/killpower, and executes /sbin/shutdown -h +0.

6.1 Core Power Architecture & IP Assignments

The power resilience architecture uses a centralized NUT Master Server (14.4.4.44) that polls the physical UPS and broadcasts status alerts (ONBATT, LOWBATT, ONLINE, FSD) across all homelab VLANs.

Role Hostname / Identifier Target IP Subnet / VLAN Description & Failure Protocol
NUT Master Server nutserver.snarfnet.net 14.4.4.44 VLAN 50 (Smart Home) Primary NUT upsd daemon polling physical UPS via USB/Serial/SNMP. Runs upsmon master and holds /root/.ssh/id_ed25519_snarfnet.
Edge Ingress / Reverse Proxy nginx.snarfnet.net 140.44.4.74 VLAN 20 (Swarm 04) NPM running in PublishMode: host on ports 80/443/81. Dependent on Synology NFS; stopped in Tier 2 prior to NAS Safe Mode.
Authentication & SSO authentik.snarfnet.net 140.44.4.29:444 VLAN 20 (Aerodrome K3s) Handles Forward Auth & Conditional MFA. Runs on Longhorn storage in Aerodrome.
UPS Web Dashboard peanut.snarfnet.net 140.44.4.70:8050 VLAN 20 (Swarm VIP) Peanut container UI consuming 14.4.4.44:3493. Protected by Authentik Forward Auth externally.
Emergency Push & Discord gotify.snarfnet.net 140.44.4.70:8090 VLAN 20 (Swarm VIP) Gotify server + gotify-discord-bridge. Internal outage alerts use http://140.44.4.70:8090 (+ direct Discord webhook fallback) to bypass Authentik during shutdown.
Smart Home Automation homeassistant.snarfnet.net 14.4.4.20 VLAN 50 (Smart Home) Home Assistant integration monitoring 14.4.4.44:3493 for state changes, critical push alerts, and non-essential load shedding.
Centralized Monitoring zabbix.snarfnet.net 140.44.4.61 VLAN 20 (Spaceport K3s) Zabbix 7.0.16 polling 14.4.4.44 via Zabbix Agent 2 upsc UserParameters.
Primary Storage 1 & 2 sn-nas / snarfnas 14.10.10.71 / .72 VLAN 40 (Storage) Synology DSM UPS clients polling 14.4.4.44. Host NFSv4 shares and Longhorn BackupTargets.

6.2 Tiered Graceful Shutdown Hierarchy (6 Stages)

+--------------------------------------------------------------------------------------------------+
| [T+0s] Grid Power Outage -> NUT Master (14.4.4.44) detects ONBATT                                |
| - Broadcasts emergency alert to Gotify (http://140.44.4.70:8090), Discord Webhook, & HA Push     |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+3m / Batt <= 85%] TIER 1: Non-Essential & Transient Workloads                                 |
| - Dev environments: Coder workspaces (coder.snarfnet.net) & Jenkins build executors stopped      |
| - AI inference stack: Turing Pi AI nodes (140.44.4.7 / 140.44.4.8) halted via SSH key            |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+6m / Batt <= 70%] TIER 2: NFS-Mounted Docker Swarm Services (NPM, Gotify, Homebox, Peanut)    |
| - Send final pre-shutdown notification to Gotify & Discord                                       |
| - Scale down Swarm NFS services (`nginx-proxy-manager_app`, `backup`, `peanut`, `homebox`)       |
| - Flushes open SQLite databases and cleanly releases all NFS file locks on 14.10.10.71           |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+8m / Batt <= 55%] TIER 3: Kubernetes Stateful Workloads & Longhorn Volume Detach              |
| - Harvester guest VMs (140.44.4.81-83) gracefully shut down via `virsh shutdown` / API           |
| - Stop K3s agent on Aerodrome (140.44.4.21-27) & Spaceport (140.44.4.51-56) worker nodes         |
| - Longhorn CSI cleanly syncs cached writes and detaches persistent volumes from worker nodes     |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+10m / Batt <= 40%] TIER 4: Storage Arrays (Synology DSM Safe Mode)                            |
| - SN-NAS (14.10.10.71) & SnarfNAS (14.10.10.72) enter DSM Safe Mode (`synopoweroff` / NUT)       |
| - Unmounts Btrfs volume pools, terminates NFS/iSCSI daemons, parks drive heads                   |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+12m / Batt <= 25%] TIER 5: Control Planes & Baremetal Hypervisors                             |
| - Aerodrome (140.44.4.11-13) & Spaceport (140.44.4.41-44) Masters save etcd snapshot & halt      |
| - Remaining Swarm Managers (140.44.4.71-75) & Harvester Hypervisors (140.44.4.81-83) halt        |
+--------------------------------------------------------------------------------------------------+
                                                |
                                                v
+--------------------------------------------------------------------------------------------------+
| [T+15m / Batt Low] TIER 6: Forced Shutdown (FSD) & Edge Power Cut                                |
| - NUT Master (14.4.4.44) sends direct Discord webhook ("Final FSD Cut") & issues `upsdrvctl`     |
| - Core Edge: UDM Pro Gateway (14.44.4.1) & Core PoE switch survive on battery until power drops  |
+--------------------------------------------------------------------------------------------------+

6.3 Automated Power Restoration & Boot Verification Gates

  1. Stage 1 (Grid AC Restored, Battery \ge 30\%): UPS re-energizes load outlets.
  2. Stage 2 (Network Core): UDM Pro (14.44.4.1) & Core switches boot, establishing VLANs 1–60 and inter-VLAN routing.
  3. Stage 3 (NUT Master): 14.4.4.44 boots (BIOS: Restore on AC Power Loss = Always On), starts upsd, and sends ONLINE alert to Discord/Gotify.
  4. Stage 4 (Storage Gate — Must Complete Before Compute): Synology NAS arrays (14.10.10.71 & 14.10.10.72) boot, mount Btrfs pools, and expose NFS (:2049).
  5. Stage 5 (Control Planes & Hypervisors): Harvester (140.44.4.81-83), K3s Masters (aerodrome & spaceport), and Swarm Managers (140.44.4.71-75) boot and establish etcd/Raft quorum.
  6. Stage 6 (Compute Workers & Ingress): K3s workers boot, Longhorn reattaches PVCs (data-authentik-postgresql-0, awx-postgres, zabbix-postgresql), and NPM on snshpd4lnxswm04 (140.44.4.74) binds host ports 80/443/81.

4. Access & Inputs Required to Begin Automation

We already have verified administrative API access to:

  • Authentik API (https://authentik.snarfnet.net)
  • Rancher & Kubernetes APIs (aerodrome, spaceport, and spacedock Harvester)
  • Portainer & Docker Swarm API (https://shipyard.snarfnet.net)
  • Zabbix 7.0.16 API (https://zabbix.snarfnet.net/api_jsonrpc.php)

To automate the remaining items end-to-end, we will only need the following when we reach each corresponding task:

  1. Discord Webhook URL(s) (For Phase 3 — Item 2):
    • One Discord Webhook URL (or separate URLs if you prefer separate channels like #infra-alerts, #smart-home, and #ups-power).
  2. Gotify Admin Credentials (For Phase 3 — Item 1):
    • Current Gotify admin username/password (or we can inspect/bootstrap a token directly via the Gotify SQLite database on Shipyard Swarm).
  3. Home Assistant Long-Lived Access Token or SSH (For Phase 3 — Item 2):
    • To inject the Gotify/Discord notification configuration into 14.4.4.20 (configuration.yaml / automations.yaml).
  4. Host SSH / Synology DSM Credentials (For Phase 1 — Item 8 & Phase 2 — Item 3):
    • Confirmation of whether the existing snarf / Katzir476! credential (from setup_storage_cluster.sh) is valid across the Debian/RPi nodes, plus SSH/DSM credentials for Synology NAS (14.10.10.71 / 14.10.10.72) and Harvester nodes (140.44.4.81-83) so we can enroll the unified id_ed25519_snarfnet key.
Reference in New Issue View Git Blame Copy Permalink