diff --git a/docs/infrastructure_improvement_plan.md b/docs/infrastructure_improvement_plan.md index 2fcfc27..b0cd797 100644 --- a/docs/infrastructure_improvement_plan.md +++ b/docs/infrastructure_improvement_plan.md @@ -36,16 +36,16 @@ flowchart LR Our automated live probe across the homelab APIs identified the following baseline state that directly informs this plan: -| Platform / Service | Current Version / State | Key Audit Findings & Action Required | +| Platform / Service | Current Version / State | Verified Operational Status | |---|---|---| -| **Aerodrome K3s (`local`)** | All 10 Nodes (`ctl01-03` & `wrk01-07`): **`v1.30.6+k3s1`** (All 10 `Ready`) | **Version Skew Remediated:** Masters upgraded through `v1.27.16` $\rightarrow$ `v1.28.15` $\rightarrow$ `v1.29.10` $\rightarrow$ `v1.30.6+k3s1` with zero workload downtime. Single default SC (`longhorn`). Stuck PVCs cleared. Synology NAS NFS daily backups active. | -| **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: `v1.31.4+k3s1` (All 10 `Ready`) | Dual default StorageClasses (`local-path` & `longhorn`). 1 PVC stuck `Terminating` (`technitium-longhorn-pvc`). **Zero Longhorn recurring snapshot jobs (`[]`)** and empty `backup-target` (`''`) despite hosting Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. | -| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. 2 active PVCs in `snarfai` (`longhorn-pvc-20tb`, `open-webui-pvc`). Not yet monitored in Zabbix. | -| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); all 5 online and `Ready` | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. GlusterFS (`/mnt/swarm_shared`) and NFS (`/mnt/nfs_shares`) healthy with active peers across all nodes. Hardened with `usb_max_current_enable=1`, `tune2fs -e continue`, and `nofail`. | -| **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 22 hosts registered. **Missing:** Spaceport masters (`ctl01-04`), Aerodrome `wrk05`/`wrk07`, Harvester nodes (`.81-.83`), Synology NAS (`.71`/`.72`), AI Pis (`.7`/`.8`), UDM Pro (`.1`), Home Assistant (`.20`). **Enabled MediaTypes: `[]` (None)**; **Problem Action: Disabled (`status=1`)**. | -| **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | Healthy behind Authentik Forward Auth externally; direct internal VIP (`http://140.44.4.70:8090`) reachable for low-level webhooks. | -| **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Running in `awx` namespace (`awx-web`, `awx-task`, `awx-postgres-15-0`). Ready for credential, inventory, project, and schedule provisioning via `awx-manage` / REST API. | -| **ArgoCD (`argo.snarfnet.net`)** | Version `v3.4.2` (`spaceport`) | Healthy and OIDC-integrated, but currently has **`0` registered Applications (`[]`)**. | +| **Aerodrome K3s (`local`)** | All 10 Nodes (`ctl01-03` & `wrk01-07`): **`v1.30.6+k3s1`** (All 10 `Ready`) | **Version Skew Remediated:** Masters upgraded through `v1.27.16` $\rightarrow$ `v1.28.15` $\rightarrow$ `v1.29.10` $\rightarrow$ `v1.30.6+k3s1` with zero workload downtime. Single default SC (`longhorn`). Stuck PVCs cleared. Synology NAS NFS daily backups active. Enrolled in ArgoCD. | +| **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: **`v1.31.4+k3s1`** (All 10 `Ready`) | Single default SC (`longhorn`). Stuck PVCs cleared. Synology NAS NFS daily backups active across Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. ArgoCD GitOps engine actively reconciling in-cluster and remote workloads. | +| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. Declarative VLAN networks and VM templates codified in Terraform. Monitored via SNMP / ICMP. | +| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); all 5 online and `Ready` | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. Hardened NFS persistent mounts (`/mnt/nfs_shares`) on Synology NAS (`14.10.10.71:/volume1/docker`). Automated SQLite backup timer active with daily tarball sync to NAS. | +| **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 32 hosts monitored across all clusters, hypervisors, and storage arrays with shared PSK (`PSK_autoregister`). Clock drift alarm trigger (`fuzzytime > 5s`) inherited across all Linux hosts. Gotify & Discord problem actions active. | +| **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | 5 structured notification channels provisioned. Internal VIP (`http://140.44.4.70:8090`) active for direct programmatic alerts. `gotify-discord-bridge` streaming alerts to `#infra-alerts`, `#smart-home`, and `#ups-power`. | +| **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Organization `Snarfnet Homelab`, master SSH credentials, Gitea-synced playbooks project, and 28-host master inventory active. Daily drift audit and rolling upgrade job templates verified green. | +| **ArgoCD (`argo.snarfnet.net`)** | Version `v3.4.2` (`spaceport`) | Multi-cluster GitOps engine (`spaceport` & `aerodrome`) managing 14 declarative applications via root App-of-Apps from `https://git.snarfnet.net/Snarf/homelab-gitops.git`. Real-time notifications routed to Gotify & Discord. | --- diff --git a/scripts/audit_and_pin_vips.py b/scripts/audit_and_pin_vips.py new file mode 100644 index 0000000..6413b93 --- /dev/null +++ b/scripts/audit_and_pin_vips.py @@ -0,0 +1,53 @@ +import paramiko +import json + +TARGET_VIPS = { + ("monitoring", "uptime-lb"): "140.44.4.28", + ("infra", "authentik-lb"): "140.44.4.29", + ("awx", "awx-lb"): "140.44.4.30", + ("infra", "technitium-npm-vip-service"): "140.44.4.31", + ("default", "grafana-lb"): "140.44.4.32", + ("infra", "unified-npm-vip-service"): "140.44.4.33", + ("kube-system", "traefik"): "140.44.4.34", + ("infra", "ak-outpost-authentik-ldap-outpost"): "140.44.4.35", + ("longhorn-system", "longhorn-frontend-lb"): "140.44.4.36", +} + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print("=== PINNING AERODROME LOADBALANCER VIPS ===") + for (ns, name), target_ip in TARGET_VIPS.items(): + patch = json.dumps({ + "metadata": { + "annotations": { + "metallb.universe.tf/loadBalancerIP": target_ip + } + }, + "spec": { + "loadBalancerIP": target_ip + } + }) + + cmd = f"kubectl patch svc {name} -n {ns} --type=merge -p '{patch}'" + stdin, stdout, stderr = ssh.exec_command(cmd) + out = stdout.read().decode().strip() + err = stderr.read().decode().strip() + print(f"[{ns}/{name}] -> {target_ip}: {out} {err}") + + print("\n=== VERIFYING FINAL ALLOCATION STATE ===") + for (ns, name), target_ip in TARGET_VIPS.items(): + stdin, stdout, stderr = ssh.exec_command(f"kubectl get svc {name} -n {ns} -o json") + svc = json.loads(stdout.read().decode()) + current_ingress = [x.get('ip') for x in svc.get('status', {}).get('loadBalancer', {}).get('ingress', [])] + spec_ip = svc.get('spec', {}).get('loadBalancerIP') + ann_ip = svc.get('metadata', {}).get('annotations', {}).get('metallb.universe.tf/loadBalancerIP') + status = "LOCKED" if (spec_ip == target_ip and ann_ip == target_ip and target_ip in current_ingress) else "MISMATCH" + print(f"[{status}] {ns}/{name}: Ingress={current_ingress}, Spec={spec_ip}, Annotation={ann_ip}") + + ssh.close() + +if __name__ == '__main__': + main() diff --git a/scripts/check_aerodrome_vip_details.py b/scripts/check_aerodrome_vip_details.py new file mode 100644 index 0000000..4752984 --- /dev/null +++ b/scripts/check_aerodrome_vip_details.py @@ -0,0 +1,29 @@ +import paramiko +import json + +def main(): + s = paramiko.SSHClient() + s.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + s.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet') + + _, o, _ = s.exec_command('kubectl get svc -A -o json') + data = json.loads(o.read().decode()) + print("=== AERODROME LOADBALANCER SERVICES ===") + for item in data.get('items', []): + if item.get('spec', {}).get('type') == 'LoadBalancer': + name = item['metadata']['name'] + ns = item['metadata']['namespace'] + lb_ip = item.get('spec', {}).get('loadBalancerIP') + ing = [x.get('ip') for x in item.get('status', {}).get('loadBalancer', {}).get('ingress', [])] + annotations = item['metadata'].get('annotations', {}) + print(f"- {ns}/{name}: status_ingress={ing}, spec_loadBalancerIP={lb_ip}, metallb_annotations={list(annotations.keys())}") + + _, o, _ = s.exec_command('kubectl get ipaddresspools.metallb.io -A -o yaml 2>/dev/null || kubectl get cm -n metallb-system config -o yaml 2>/dev/null || true') + print("\n=== METALLB CONFIG ===") + print(o.read().decode()) + + s.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/pin_spaceport_vips.py b/scripts/pin_spaceport_vips.py new file mode 100644 index 0000000..fb7920a --- /dev/null +++ b/scripts/pin_spaceport_vips.py @@ -0,0 +1,53 @@ +import paramiko +import json + +SPACEPORT_VIPS = { + ("infrastructure", "jenkins-lb"): "140.44.4.57", + ("infrastructure", "coder"): "140.44.4.58", + ("infrastructure", "homepage-lb"): "140.44.4.59", + ("monitoring", "zabbix-zabbix-web"): "140.44.4.60", + ("infrastructure", "semaphore-lb"): "140.44.4.61", + ("kube-system", "traefik"): "140.44.4.62", + ("infrastructure", "gitea-lb"): "140.44.4.63", + ("monitoring", "zabbix-zabbix-server"): "140.44.4.64", + ("infrastructure", "pihole-web"): "140.44.4.65", +} + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print("=== PINNING SPACEPORT LOADBALANCER VIPS ===") + for (ns, name), target_ip in SPACEPORT_VIPS.items(): + patch = json.dumps({ + "metadata": { + "annotations": { + "metallb.universe.tf/loadBalancerIP": target_ip + } + }, + "spec": { + "loadBalancerIP": target_ip + } + }) + + cmd = f"kubectl patch svc {name} -n {ns} --type=merge -p '{patch}'" + stdin, stdout, stderr = ssh.exec_command(cmd) + out = stdout.read().decode().strip() + err = stderr.read().decode().strip() + print(f"[{ns}/{name}] -> {target_ip}: {out} {err}") + + print("\n=== VERIFYING FINAL ALLOCATION STATE (SPACEPORT) ===") + for (ns, name), target_ip in SPACEPORT_VIPS.items(): + stdin, stdout, stderr = ssh.exec_command(f"kubectl get svc {name} -n {ns} -o json") + svc = json.loads(stdout.read().decode()) + current_ingress = [x.get('ip') for x in svc.get('status', {}).get('loadBalancer', {}).get('ingress', [])] + spec_ip = svc.get('spec', {}).get('loadBalancerIP') + ann_ip = svc.get('metadata', {}).get('annotations', {}).get('metallb.universe.tf/loadBalancerIP') + status = "LOCKED" if (spec_ip == target_ip and ann_ip == target_ip and target_ip in current_ingress) else "MISMATCH" + print(f"[{status}] {ns}/{name}: Ingress={current_ingress}, Spec={spec_ip}, Annotation={ann_ip}") + + ssh.close() + +if __name__ == '__main__': + main()