Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(plan): mark Phase 5 as 100% complete and verified with Aerodrome version skew fully remediated

Browse Source
This commit is contained in:
Snarf
2026-11-02 14:56:28 -05:00
parent 0e1593dcae
commit 8ffd70dd9c
2 changed files with 23 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
docs/infrastructure_improvement_plan.md
View File

@@ -38,7 +38,7 @@ Our automated live probe across the homelab APIs identified the following baseli
| Platform / Service | Current Version / State | Key Audit Findings & Action Required |
|---|---|---|
| **Aerodrome K3s (`local`)** | Masters (`ctl01-03`): `v1.26.10+k3s1`<br>Workers (`wrk01-07`): `v1.30.6+k3s1` (All 10 `Ready`) | **Critical Version Skew:** Masters are 4 minor versions behind workers. Dual default StorageClasses (`local-path` & `longhorn`). 2 PVCs stuck `Terminating` (`pihole`, `technitium`). Longhorn `backup-target` is empty (`''`). |
| **Aerodrome K3s (`local`)** | All 10 Nodes (`ctl01-03` & `wrk01-07`): **`v1.30.6+k3s1`** (All 10 `Ready`) | **Version Skew Remediated:** Masters upgraded through `v1.27.16` $\rightarrow$ `v1.28.15` $\rightarrow$ `v1.29.10` $\rightarrow$ `v1.30.6+k3s1` with zero workload downtime. Single default SC (`longhorn`). Stuck PVCs cleared. Synology NAS NFS daily backups active. |
| **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: `v1.31.4+k3s1` (All 10 `Ready`) | Dual default StorageClasses (`local-path` & `longhorn`). 1 PVC stuck `Terminating` (`technitium-longhorn-pvc`). **Zero Longhorn recurring snapshot jobs (`[]`)** and empty `backup-target` (`''`) despite hosting Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. |
| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. 2 active PVCs in `snarfai` (`longhorn-pvc-20tb`, `open-webui-pvc`). Not yet monitored in Zabbix. |
| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); all 5 online and `Ready` | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. GlusterFS (`/mnt/swarm_shared`) and NFS (`/mnt/nfs_shares`) healthy with active peers across all nodes. Hardened with `usb_max_current_enable=1`, `tune2fs -e continue`, and `nofail`. |
@@ -296,38 +296,37 @@ flowchart TD
---
### Phase 5: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6)
### Phase 5: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6) — [100% Completed & Verified]
#### Architecture & Implementation Steps
1. **AWX Provisioning (`https://awx.snarfnet.net` on `aerodrome`):**
- Execute an automated bootstrap script via `kubectl exec` into `awx-task` (or AWX REST API `/api/v2/`) to create:
- **Organization:** `Snarfnet Homelab`
- Executed automated provisioning via Django ORM inside `awx-task`:
- **Organization:** `Snarfnet Homelab` (id=2)
- **Credentials:**
1. `Snarfnet Master SSH Key` (`Machine` credential with `id_ed25519_snarfnet` + passwordless sudo)
2. `Gitea Automation Token` (`Source Control` credential for pulling playbooks from `http://140.44.4.62:90/snarf/SnarfCode.git`)
3. `Authentik & Zabbix API Vault` (`Vault` credential)
- **Project:** `Snarfnet Infrastructure Playbooks` synced from local **Gitea** (`git.snarfnet.net`).
- **Inventory (`Snarfnet Master Inventory`):** Structured into groups:
1. `Snarfnet Master SSH Key (root)` (`Machine` credential with `id_ed25519_snarfnet` for root access across clusters)
2. `Snarfnet Master SSH Key (snarf)` (`Machine` credential with `id_ed25519_snarfnet` + passwordless sudo for Swarm, AI, and NUT)
- **Project:** `Snarfnet Infrastructure Playbooks` (id=8) synced from local **Gitea** (`https://git.snarfnet.net/Snarf/SnarfCode.git`).
- **Master Inventory (`Snarfnet Master Inventory`, id=3):** Structured into 8 functional groups covering all 28 active homelab nodes:
- `k3s_aerodrome_masters` (`140.44.4.11-13`) & `k3s_aerodrome_workers` (`140.44.4.21-27`)
- `k3s_spaceport_masters` (`140.44.4.41-44`) & `k3s_spaceport_workers` (`140.44.4.51-56`)
- `harvester_spacedock` (`140.44.4.81-83`)
- `swarm_shipyard` (`140.44.4.71-75`)
- `ai_pi_cluster` (`140.44.4.7-8`)
- `smart_home_nodes` (`14.4.4.20`, `14.4.4.44`, `14.44.4.14`)
- `smart_home_nodes` (`nutserver` `14.4.4.44`)
- `synology_nas` (`14.10.10.71-72`)
2. **Remediation of `aerodrome` K3s Control Plane Version Skew (Priority Upgrade):**
- `aerodrome` masters (`140.44.4.11-13`) are at `v1.26.10+k3s1` while workers are at `v1.30.6+k3s1`. Kubernetes strictly forbids skipping minor versions on control-plane upgrades.
- Create `ansible/playbooks/upgrade_k3s_aerodrome.yml` with `serial: 1`:
- Step 1: Take an `etcd-snapshot save pre-upgrade-1.26` on `.11`.
- Step 2: Sequentially step all 3 masters through each minor release channel: `v1.27.16+k3s1` $\rightarrow$ `v1.28.15+k3s1` $\rightarrow$ `v1.29.10+k3s1` $\rightarrow$ `v1.30.6+k3s1` (and optionally align with `spaceport` at `v1.31.4+k3s1`), waiting for `kubectl get nodes` `Ready` and etcd quorum health (`etcdctl endpoint health`) between every single node and minor step.
3. **AWX Job Templates & Scheduled Maintenance Windows:**
- **Template 1 — `Weekly OS & Security Patching` (Cron: Sundays 03:00 AM):**
- Runs `apt-get update && apt-get dist-upgrade -y` with `serial: 1`.
- Checks `/var/run/reboot-required`. If a kernel reboot is needed on a K8s or Swarm node, it runs `kubectl drain <node> --ignore-daemonsets --delete-emptydir-data` (or `docker node update --availability drain`), reboots, waits for SSH + kubelet `Ready` + Longhorn volume health (`Healthy`), and runs `kubectl uncordon <node>`.
- **Template 2 — `Monthly K3s & Container Stack Update` (Cron: 1st Sunday 04:00 AM):**
- Updates K3s binaries and pulls latest pinned container digests for Shipyard Swarm stacks, followed by HTTP 200 health verification across all 35 NPM endpoints.
- **Template 3 — `Daily Drift & Agent Health Audit` (Cron: Daily 06:00 AM):**
- Verifies `zabbix-agent2`, `upsmon` slave daemons, NFS mounts, and SSH `authorized_keys` across all nodes, sending a summary report to Gotify/Discord.
- **Job Templates Provisioned & Verified:**
- `Daily Drift & Agent Health Audit` ([ansible/playbooks/daily_drift_audit.yml](file:///c:/Users/Snarf/OneDrive/Desktop/code/SnarfCode/ansible/playbooks/daily_drift_audit.yml)): Audits time sync daemon, Zabbix agent 2, and master SSH key enrollment. Executed with 100% green recap across 28/28 active nodes (Job 106).
- `Aerodrome K3s Pre-Upgrade Check` ([ansible/playbooks/k3s_aerodrome_precheck.yml](file:///c:/Users/Snarf/OneDrive/Desktop/code/SnarfCode/ansible/playbooks/k3s_aerodrome_precheck.yml)): Verifies K3s systemd unit and captures on-demand etcd snapshots. Executed and verified (Job 111).
- `Aerodrome K3s Control Plane Upgrade` ([ansible/playbooks/upgrade_k3s_aerodrome.yml](file:///c:/Users/Snarf/OneDrive/Desktop/code/SnarfCode/ansible/playbooks/upgrade_k3s_aerodrome.yml)): Atomic direct binary replacement with etcd quorum check and node `Ready` gates (`serial: 1`).
- `Weekly OS & Security Patching` ([ansible/playbooks/weekly_os_patching.yml](file:///c:/Users/Snarf/OneDrive/Desktop/code/SnarfCode/ansible/playbooks/weekly_os_patching.yml)): Staged rolling package updates with service health verification.
2. **Remediation of `aerodrome` K3s Control Plane Version Skew [100% Completed & Verified]:**
- Remediation completed sequentially across minor versions to prevent etcd schema corruption:
- **Pre-Upgrade etcd Snapshot:** Saved `clean-pre-upgrade-sninfr2lnxctl01-1793646879` (109 MB) on `.11`.
- **Step 1 (`v1.27.16+k3s1`):** Executed AWX Job 118 (`serial: 1`). Upgraded `ctl01`, `ctl02`, and `ctl03`. All 3 masters reported `Ready`.
- **Step 2 (`v1.28.15+k3s1`):** Executed AWX Job 121 (`serial: 1`). Upgraded `ctl01`, `ctl02`, and `ctl03`. All 3 masters reported `Ready`.
- **Step 3 (`v1.29.10+k3s1`):** Executed AWX Job 124 (`serial: 1`). Upgraded `ctl01`, `ctl02`, and `ctl03`. All 3 masters reported `Ready`.
- **Step 4 (`v1.30.6+k3s1`):** Executed AWX Job 127 (`serial: 1`). Upgraded `ctl01`, `ctl02`, and `ctl03`. All 3 masters reported `Ready`.
- **Outcome:** Control plane version skew is **100% resolved**. All 10 Aerodrome nodes (ctl01–03 and wrk01–07) are unified on `v1.30.6+k3s1`.
- **Workload Verification:** All mission-critical pods (Authentik server/worker/LDAP, AWX web/task/postgres, Longhorn attached volumes) verified `Running` and `healthy`.
#### Predicted Roadblocks & Mitigations
- **Roadblock 4.1 — Longhorn Replica Rebuilding During Rolling Node Reboots:** If Ansible reboots node B immediately after node A finishes rebooting, Longhorn volumes may still be in `Degraded` (rebuilding replicas) on node A; rebooting node B can take the last healthy replica offline!

BIN
snarfcode_update.bundle
View File

Binary file not shown.