Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(infra): Phase 3 & 4 completion, deploy AWX playbooks & Zabbix monitoring

Browse Source
This commit is contained in:
Snarf
2026-11-02 04:17:38 -05:00
parent 9c3de8bb4d
commit 003ae80c9c
15 changed files with 711 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/infrastructure_improvement_plan.md
View File

@@ -11,7 +11,7 @@ Following Site Reliability Engineering (SRE) best practices, the nine infrastruc
1. **Phase 1 — Cryptographic Access & Secret Foundation (Item 8):** Unify SSH key access (`id_ed25519_snarfnet`) across all hosts and secure private keys inside AWX, Semaphore, and GitOps (SOPS/Age). *Prerequisite for all non-interactive automation, patching, and UPS shutdown scripts.* **[100% Completed & Verified]**
2. **Phase 2 — Centralized Internal NTP Time Synchronization Architecture (Item 9):** Deploy a hardened, high-availability internal Chrony NTP server syncing against diverse external Stratum 1/2 pools (Cloudflare, Google, NIST, pool.ntp.org) with local stratum fallback. Reconfigure all 31+ nodes (`systemd-timesyncd` / `chrony`) to pull time from the internal server. *Prevents Raft etcd election splits, Docker Swarm TLS cert expiration, Authentik JWT rejections, and Raspberry Pi post-outage clock drift.* **[100% Completed & Verified]**
3. **Phase 3 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3):** Resolve dual-default StorageClasses, clear stuck `Terminating` PVCs, add Longhorn recurring snapshots on `spaceport`, configure Synology NAS (`14.10.10.71`) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. *Prerequisite before upgrading kernels or control planes.* **[100% Completed & Verified]**
4. **Phase 4 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.*
4. **Phase 4 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.* **[100% Completed & Verified]**
5. **Phase 5 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates.
6. **Phase 6 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using `iac-reverse` Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD ApplicationSets (`spaceport` & `aerodrome`) for automated drift remediation.
7. **Phase 7 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment.
@@ -219,7 +219,7 @@ flowchart TD
---
### Phase 4: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7)
### Phase 4: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7) — [100% Completed & Verified]
#### Architecture & Implementation Steps