feat(vip): lock MetalLB LoadBalancer VIPs across Aerodrome and Spaceport (Pillar 2 completed)
This commit is contained in:
@@ -36,16 +36,16 @@ flowchart LR
|
||||
|
||||
Our automated live probe across the homelab APIs identified the following baseline state that directly informs this plan:
|
||||
|
||||
| Platform / Service | Current Version / State | Key Audit Findings & Action Required |
|
||||
| Platform / Service | Current Version / State | Verified Operational Status |
|
||||
|---|---|---|
|
||||
| **Aerodrome K3s (`local`)** | All 10 Nodes (`ctl01-03` & `wrk01-07`): **`v1.30.6+k3s1`** (All 10 `Ready`) | **Version Skew Remediated:** Masters upgraded through `v1.27.16` $\rightarrow$ `v1.28.15` $\rightarrow$ `v1.29.10` $\rightarrow$ `v1.30.6+k3s1` with zero workload downtime. Single default SC (`longhorn`). Stuck PVCs cleared. Synology NAS NFS daily backups active. |
|
||||
| **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: `v1.31.4+k3s1` (All 10 `Ready`) | Dual default StorageClasses (`local-path` & `longhorn`). 1 PVC stuck `Terminating` (`technitium-longhorn-pvc`). **Zero Longhorn recurring snapshot jobs (`[]`)** and empty `backup-target` (`''`) despite hosting Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. |
|
||||
| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. 2 active PVCs in `snarfai` (`longhorn-pvc-20tb`, `open-webui-pvc`). Not yet monitored in Zabbix. |
|
||||
| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); all 5 online and `Ready` | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. GlusterFS (`/mnt/swarm_shared`) and NFS (`/mnt/nfs_shares`) healthy with active peers across all nodes. Hardened with `usb_max_current_enable=1`, `tune2fs -e continue`, and `nofail`. |
|
||||
| **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 22 hosts registered. **Missing:** Spaceport masters (`ctl01-04`), Aerodrome `wrk05`/`wrk07`, Harvester nodes (`.81-.83`), Synology NAS (`.71`/`.72`), AI Pis (`.7`/`.8`), UDM Pro (`.1`), Home Assistant (`.20`). **Enabled MediaTypes: `[]` (None)**; **Problem Action: Disabled (`status=1`)**. |
|
||||
| **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | Healthy behind Authentik Forward Auth externally; direct internal VIP (`http://140.44.4.70:8090`) reachable for low-level webhooks. |
|
||||
| **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Running in `awx` namespace (`awx-web`, `awx-task`, `awx-postgres-15-0`). Ready for credential, inventory, project, and schedule provisioning via `awx-manage` / REST API. |
|
||||
| **ArgoCD (`argo.snarfnet.net`)** | Version `v3.4.2` (`spaceport`) | Healthy and OIDC-integrated, but currently has **`0` registered Applications (`[]`)**. |
|
||||
| **Aerodrome K3s (`local`)** | All 10 Nodes (`ctl01-03` & `wrk01-07`): **`v1.30.6+k3s1`** (All 10 `Ready`) | **Version Skew Remediated:** Masters upgraded through `v1.27.16` $\rightarrow$ `v1.28.15` $\rightarrow$ `v1.29.10` $\rightarrow$ `v1.30.6+k3s1` with zero workload downtime. Single default SC (`longhorn`). Stuck PVCs cleared. Synology NAS NFS daily backups active. Enrolled in ArgoCD. |
|
||||
| **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: **`v1.31.4+k3s1`** (All 10 `Ready`) | Single default SC (`longhorn`). Stuck PVCs cleared. Synology NAS NFS daily backups active across Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. ArgoCD GitOps engine actively reconciling in-cluster and remote workloads. |
|
||||
| **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. Declarative VLAN networks and VM templates codified in Terraform. Monitored via SNMP / ICMP. |
|
||||
| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); all 5 online and `Ready` | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. Hardened NFS persistent mounts (`/mnt/nfs_shares`) on Synology NAS (`14.10.10.71:/volume1/docker`). Automated SQLite backup timer active with daily tarball sync to NAS. |
|
||||
| **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 32 hosts monitored across all clusters, hypervisors, and storage arrays with shared PSK (`PSK_autoregister`). Clock drift alarm trigger (`fuzzytime > 5s`) inherited across all Linux hosts. Gotify & Discord problem actions active. |
|
||||
| **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | 5 structured notification channels provisioned. Internal VIP (`http://140.44.4.70:8090`) active for direct programmatic alerts. `gotify-discord-bridge` streaming alerts to `#infra-alerts`, `#smart-home`, and `#ups-power`. |
|
||||
| **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Organization `Snarfnet Homelab`, master SSH credentials, Gitea-synced playbooks project, and 28-host master inventory active. Daily drift audit and rolling upgrade job templates verified green. |
|
||||
| **ArgoCD (`argo.snarfnet.net`)** | Version `v3.4.2` (`spaceport`) | Multi-cluster GitOps engine (`spaceport` & `aerodrome`) managing 14 declarative applications via root App-of-Apps from `https://git.snarfnet.net/Snarf/homelab-gitops.git`. Real-time notifications routed to Gotify & Discord. |
|
||||
|
||||
---
|
||||
|
||||
|
||||
53
scripts/audit_and_pin_vips.py
Normal file
53
scripts/audit_and_pin_vips.py
Normal file
@@ -0,0 +1,53 @@
|
||||
import paramiko
|
||||
import json
|
||||
|
||||
TARGET_VIPS = {
|
||||
("monitoring", "uptime-lb"): "140.44.4.28",
|
||||
("infra", "authentik-lb"): "140.44.4.29",
|
||||
("awx", "awx-lb"): "140.44.4.30",
|
||||
("infra", "technitium-npm-vip-service"): "140.44.4.31",
|
||||
("default", "grafana-lb"): "140.44.4.32",
|
||||
("infra", "unified-npm-vip-service"): "140.44.4.33",
|
||||
("kube-system", "traefik"): "140.44.4.34",
|
||||
("infra", "ak-outpost-authentik-ldap-outpost"): "140.44.4.35",
|
||||
("longhorn-system", "longhorn-frontend-lb"): "140.44.4.36",
|
||||
}
|
||||
|
||||
def main():
|
||||
ssh = paramiko.SSHClient()
|
||||
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
|
||||
|
||||
print("=== PINNING AERODROME LOADBALANCER VIPS ===")
|
||||
for (ns, name), target_ip in TARGET_VIPS.items():
|
||||
patch = json.dumps({
|
||||
"metadata": {
|
||||
"annotations": {
|
||||
"metallb.universe.tf/loadBalancerIP": target_ip
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"loadBalancerIP": target_ip
|
||||
}
|
||||
})
|
||||
|
||||
cmd = f"kubectl patch svc {name} -n {ns} --type=merge -p '{patch}'"
|
||||
stdin, stdout, stderr = ssh.exec_command(cmd)
|
||||
out = stdout.read().decode().strip()
|
||||
err = stderr.read().decode().strip()
|
||||
print(f"[{ns}/{name}] -> {target_ip}: {out} {err}")
|
||||
|
||||
print("\n=== VERIFYING FINAL ALLOCATION STATE ===")
|
||||
for (ns, name), target_ip in TARGET_VIPS.items():
|
||||
stdin, stdout, stderr = ssh.exec_command(f"kubectl get svc {name} -n {ns} -o json")
|
||||
svc = json.loads(stdout.read().decode())
|
||||
current_ingress = [x.get('ip') for x in svc.get('status', {}).get('loadBalancer', {}).get('ingress', [])]
|
||||
spec_ip = svc.get('spec', {}).get('loadBalancerIP')
|
||||
ann_ip = svc.get('metadata', {}).get('annotations', {}).get('metallb.universe.tf/loadBalancerIP')
|
||||
status = "LOCKED" if (spec_ip == target_ip and ann_ip == target_ip and target_ip in current_ingress) else "MISMATCH"
|
||||
print(f"[{status}] {ns}/{name}: Ingress={current_ingress}, Spec={spec_ip}, Annotation={ann_ip}")
|
||||
|
||||
ssh.close()
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
29
scripts/check_aerodrome_vip_details.py
Normal file
29
scripts/check_aerodrome_vip_details.py
Normal file
@@ -0,0 +1,29 @@
|
||||
import paramiko
|
||||
import json
|
||||
|
||||
def main():
|
||||
s = paramiko.SSHClient()
|
||||
s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
s.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet')
|
||||
|
||||
_, o, _ = s.exec_command('kubectl get svc -A -o json')
|
||||
data = json.loads(o.read().decode())
|
||||
print("=== AERODROME LOADBALANCER SERVICES ===")
|
||||
for item in data.get('items', []):
|
||||
if item.get('spec', {}).get('type') == 'LoadBalancer':
|
||||
name = item['metadata']['name']
|
||||
ns = item['metadata']['namespace']
|
||||
lb_ip = item.get('spec', {}).get('loadBalancerIP')
|
||||
ing = [x.get('ip') for x in item.get('status', {}).get('loadBalancer', {}).get('ingress', [])]
|
||||
annotations = item['metadata'].get('annotations', {})
|
||||
print(f"- {ns}/{name}: status_ingress={ing}, spec_loadBalancerIP={lb_ip}, metallb_annotations={list(annotations.keys())}")
|
||||
|
||||
_, o, _ = s.exec_command('kubectl get ipaddresspools.metallb.io -A -o yaml 2>/dev/null || kubectl get cm -n metallb-system config -o yaml 2>/dev/null || true')
|
||||
print("\n=== METALLB CONFIG ===")
|
||||
print(o.read().decode())
|
||||
|
||||
s.close()
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
||||
53
scripts/pin_spaceport_vips.py
Normal file
53
scripts/pin_spaceport_vips.py
Normal file
@@ -0,0 +1,53 @@
|
||||
import paramiko
|
||||
import json
|
||||
|
||||
SPACEPORT_VIPS = {
|
||||
("infrastructure", "jenkins-lb"): "140.44.4.57",
|
||||
("infrastructure", "coder"): "140.44.4.58",
|
||||
("infrastructure", "homepage-lb"): "140.44.4.59",
|
||||
("monitoring", "zabbix-zabbix-web"): "140.44.4.60",
|
||||
("infrastructure", "semaphore-lb"): "140.44.4.61",
|
||||
("kube-system", "traefik"): "140.44.4.62",
|
||||
("infrastructure", "gitea-lb"): "140.44.4.63",
|
||||
("monitoring", "zabbix-zabbix-server"): "140.44.4.64",
|
||||
("infrastructure", "pihole-web"): "140.44.4.65",
|
||||
}
|
||||
|
||||
def main():
|
||||
ssh = paramiko.SSHClient()
|
||||
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
|
||||
ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
|
||||
|
||||
print("=== PINNING SPACEPORT LOADBALANCER VIPS ===")
|
||||
for (ns, name), target_ip in SPACEPORT_VIPS.items():
|
||||
patch = json.dumps({
|
||||
"metadata": {
|
||||
"annotations": {
|
||||
"metallb.universe.tf/loadBalancerIP": target_ip
|
||||
}
|
||||
},
|
||||
"spec": {
|
||||
"loadBalancerIP": target_ip
|
||||
}
|
||||
})
|
||||
|
||||
cmd = f"kubectl patch svc {name} -n {ns} --type=merge -p '{patch}'"
|
||||
stdin, stdout, stderr = ssh.exec_command(cmd)
|
||||
out = stdout.read().decode().strip()
|
||||
err = stderr.read().decode().strip()
|
||||
print(f"[{ns}/{name}] -> {target_ip}: {out} {err}")
|
||||
|
||||
print("\n=== VERIFYING FINAL ALLOCATION STATE (SPACEPORT) ===")
|
||||
for (ns, name), target_ip in SPACEPORT_VIPS.items():
|
||||
stdin, stdout, stderr = ssh.exec_command(f"kubectl get svc {name} -n {ns} -o json")
|
||||
svc = json.loads(stdout.read().decode())
|
||||
current_ingress = [x.get('ip') for x in svc.get('status', {}).get('loadBalancer', {}).get('ingress', [])]
|
||||
spec_ip = svc.get('spec', {}).get('loadBalancerIP')
|
||||
ann_ip = svc.get('metadata', {}).get('annotations', {}).get('metallb.universe.tf/loadBalancerIP')
|
||||
status = "LOCKED" if (spec_ip == target_ip and ann_ip == target_ip and target_ip in current_ingress) else "MISMATCH"
|
||||
print(f"[{status}] {ns}/{name}: Ingress={current_ingress}, Spec={spec_ip}, Annotation={ann_ip}")
|
||||
|
||||
ssh.close()
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user