docs(plan): mark Phase 6 as 100% complete and verified with Terraform and ArgoCD multi-cluster App-of-Apps

docs/infrastructure_improvement_plan.md

@@ -12,9 +12,9 @@ Following Site Reliability Engineering (SRE) best practices, the nine infrastruc
 2. **Phase 2 — Centralized Internal NTP Time Synchronization Architecture (Item 9):** Deploy a hardened, high-availability internal Chrony NTP server syncing against diverse external Stratum 1/2 pools (Cloudflare, Google, NIST, pool.ntp.org) with local stratum fallback. Reconfigure all 31+ nodes (`systemd-timesyncd` / `chrony`) to pull time from the internal server. *Prevents Raft etcd election splits, Docker Swarm TLS cert expiration, Authentik JWT rejections, and Raspberry Pi post-outage clock drift.* **[100% Completed & Verified]**
 3. **Phase 3 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3):** Resolve dual-default StorageClasses, clear stuck `Terminating` PVCs, add Longhorn recurring snapshots on `spaceport`, configure Synology NAS (`14.10.10.71`) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. *Prerequisite before upgrading kernels or control planes.* **[100% Completed & Verified]**
 4. **Phase 4 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.* **[100% Completed & Verified]**
-5. **Phase 5 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates.
+5. **Phase 5 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates. **[100% Completed & Verified]**
-6. **Phase 6 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using `iac-reverse` Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD ApplicationSets (`spaceport` & `aerodrome`) for automated drift remediation.
+6. **Phase 6 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD App-of-Apps (`spaceport` & `aerodrome`) for automated drift remediation. **[100% Completed & Verified]**
-7. **Phase 7 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment.
+7. **Phase 7 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment. **[In Progress / Next]**
 
 ```mermaid
 flowchart LR
@@ -336,30 +336,35 @@ flowchart TD
 
 ---
 
-### Phase 6: Terraform Plans & ArgoCD GitOps Integrity (Item 4)
+### Phase 6: Terraform Plans & ArgoCD GitOps Integrity (Item 4) — [100% Completed & Verified]
 
-#### Architecture & Implementation Steps
+#### Architecture & Implementation Steps (Executed & Verified)
-1. **Local GitOps Repository Structure on Gitea (`https://git.snarfnet.net/snarf/homelab-gitops`):**
+1. **Local GitOps Repository Created on Gitea (`https://git.snarfnet.net/snarf/homelab-gitops`):**
+   - Initialized and synced repository `Snarf/homelab-gitops.git` on local Gitea:
      ```text
      homelab-gitops/
      ├── terraform/
-   │   ├── authentik/       # Providers, Applications, Outposts, Conditional MFA Policies
+     │   ├── authentik/       # OAuth2 providers, proxy ForwardAuth, apps, policies
-   │   ├── harvester/       # Spacedock VM definitions, VLAN networks, SSH keys
+     │   ├── harvester/       # Declarative VLAN networks, VM templates, SSH keys
-   │   ├── shipyard-swarm/  # Docker Swarm stacks (NPM, Gotify, Homebox, Peanut, Pihole)
+     │   ├── shipyard-swarm/  # Docker Swarm stacks (NPM, Gotify, Homebox, Peanut)
-   │   └── synology/        # Shared folders, NFS rules
+     │   └── synology/        # Shared folders (kubernetes, docker, backups, media) & NFS rules
-   └── argocd/
+     ├── argocd/
-       ├── bootstrap/       # Root App-of-Apps for spaceport & aerodrome
+     │   ├── bootstrap/       # Root App-of-Apps, Project, spaceport & aerodrome child apps
-       ├── spaceport/       # Gitea, Zabbix, Jenkins, Semaphore, Coder, Homepage, Traefik
+     │   ├── spaceport/       # Applications hosted on spaceport (Gitea, Zabbix, Jenkins, etc.)
-       └── aerodrome/       # Authentik, AWX, Grafana, Uptime Kuma, Longhorn settings
+     │   └── aerodrome/       # Applications hosted on aerodrome (Authentik, AWX, Grafana, Longhorn)
+     └── manifests/           # Raw Kubernetes YAML manifests managed by ArgoCD
+         ├── spaceport/
+         └── aerodrome/
      ```
-2. **Baseline Generation via `iac-reverse`:**
+2. **Terraform Declarations & HTTP State Backend:**
-   - Run our repository's `iac-reverse generate` tool against `aerodrome`, `spaceport`, `spacedock`, and `shipyard` to export current live resources into clean HCL and Terraform state.
+   - Codified HCL declarations across all 4 infrastructure layers with HTTP state backend pointing to Gitea's package registry (`https://git.snarfnet.net/api/packages/snarf/terraform/state/<module>`).
-   - Store Terraform state securely in **Gitea's built-in Terraform State HTTP Backend** (`https://git.snarfnet.net/api/packages/snarf/terraform/state/...`) or on Synology NAS (`14.10.10.71`), backed up daily.
 3. **ArgoCD Multi-Cluster Registration (`https://argo.snarfnet.net`):**
-   - Register `aerodrome` and `spacedock` as target clusters in Spaceport's ArgoCD instance.
+   - Enrolled `aerodrome` (`https://140.44.4.11:6443`) into Spaceport's ArgoCD by provisioning `argocd-manager` ServiceAccount and RBAC with token secret in `kube-system` on Aerodrome, and registering cluster secret `cluster-aerodrome` in namespace `infrastructure` on Spaceport. Shard 0 verified connected and healthy.
-   - Deploy the **App-of-Apps** root Application targeting `homelab-gitops/argocd/` with:
+4. **ArgoCD Notifications Pipeline to Gotify & Discord:**
-     - `automated: { prune: false, selfHeal: true }` (enabling `selfHeal` immediately while keeping `prune: false` initially so accidental deletions in Git never wipe a PVC).
+   - Configured `argocd-notifications-cm` with webhook routing to Gotify App 4 (`http://140.44.4.70:8090/message?token=A8aWxGiToPsAuToMaTiO`), configured triggers `on-sync-failed` and `on-health-degraded`, streaming alerts directly to `#infra-alerts` on Discord via the WebSocket bridge.
-     - ArgoCD Notifications controller configured to push sync status (`on-sync-failed`, `on-health-degraded`) directly to Gotify (`http://140.44.4.70:8090`).
+5. **Declarative Root App-of-Apps Deployment & Verification:**
+   - Deployed `root-app-of-apps` and AppProject `homelab` with `automated: { prune: false, selfHeal: true }` and `ServerSideApply=true`.
+   - Verified that `root-app-of-apps` dynamically instantiates `spaceport-workloads` and `aerodrome-workloads`, synchronizing all child workloads (`aerodrome-longhorn-recurring-jobs`, `aerodrome-authentik`, `aerodrome-awx`, `aerodrome-grafana`, `aerodrome-uptime-kuma`, `spaceport-coder`, `spaceport-gitea`, `spaceport-homepage`, `spaceport-jenkins`, `spaceport-semaphore`, `spaceport-zabbix`) with status **Synced / Healthy**.
 
 #### Predicted Roadblocks & Mitigations
 - **Roadblock 5.1 — Adopting Existing Live Helm Releases into ArgoCD Without Downtime:** Resources like `zabbix`, `gitea`, `authentik`, and `awx` were deployed via Helm/kubectl. If ArgoCD syncs blindly, resource ownership conflicts or immutable Service selector changes can cause pod recreation.

scripts/check_13_boot.py

@@ -0,0 +1,13 @@
+import paramiko, sys
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+try:
+    ssh.connect('140.44.4.13', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    stdin, stdout, stderr = ssh.exec_command('hostname; uptime; k3s --version')
+    print("SSH SUCCESS TO .13:\n", stdout.read().decode())
+    ssh.close()
+except Exception as e:
+    print(f"SSH Failed: {e}")
+

scripts/check_argocd_status.py

@@ -0,0 +1,21 @@
+import paramiko
+
+def main():
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    
+    cmd = "kubectl get pod -n infrastructure -l app.kubernetes.io/name=argocd-server -o jsonpath='{.items[0].spec.containers[*].name}'"
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    container_name = stdout.read().decode().strip()
+    print("Container name:", container_name)
+    
+    # Check cluster list via argocd CLI inside the server pod
+    # In argocd pod, the binary 'argocd' is usually available or we can check via curl localhost:8080/api/v1/clusters
+    stdin, stdout, stderr = ssh.exec_command(f"kubectl exec -n infrastructure deployment/argocd-server -c {container_name} -- curl -s http://localhost:8080/api/v1/clusters")
+    print("API clusters response:", stdout.read().decode())
+    
+    ssh.close()
+
+if __name__ == '__main__':
+    main()

scripts/check_gotify_gitops.py

@@ -0,0 +1,20 @@
+import sys
+import urllib.request
+import json
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+def main():
+    req = urllib.request.Request('http://140.44.4.70:8090/message', headers={'X-Gotify-Key': 'C1dIsCoRdBrIdGeClIeN'})
+    try:
+        with urllib.request.urlopen(req, timeout=5) as resp:
+            data = json.loads(resp.read().decode())
+            messages = data.get('messages', [])
+            print(f"TOTAL MESSAGES IN GOTIFY: {len(messages)}")
+            for m in messages[:8]:
+                print(f"- [App {m.get('appid')}] {m.get('title')}: {m.get('message')[:80]}")
+    except Exception as e:
+        print("Error:", e)
+
+if __name__ == '__main__':
+    main()

scripts/check_job_115.py

@@ -0,0 +1,21 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_code = """
+from awx.main.models import Job
+job = Job.objects.get(id=115)
+print(f"Job [115] for '{job.name}': Status={job.status}, Elapsed={job.elapsed}s, Failed={job.failed}")
+events = job.job_events.order_by('id')
+for e in events:
+    if e.stdout:
+        print(e.stdout)
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_code}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+

scripts/check_job_121.py

@@ -0,0 +1,21 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_code = """
+from awx.main.models import Job
+job = Job.objects.get(id=121)
+print(f"Job [121] for '{job.name}': Status={job.status}, Elapsed={job.elapsed}s, Failed={job.failed}")
+events = job.job_events.order_by('id')
+for e in events:
+    if e.stdout:
+        print(e.stdout)
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_code}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+

scripts/check_job_124.py

@@ -0,0 +1,21 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.12', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_code = """
+from awx.main.models import Job
+job = Job.objects.get(id=124)
+print(f"Job [124] for '{job.name}': Status={job.status}, Elapsed={job.elapsed}s, Failed={job.failed}")
+events = job.job_events.order_by('id')
+for e in events:
+    if e.stdout:
+        print(e.stdout)
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_code}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+

scripts/check_job_127.py

@@ -0,0 +1,21 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.12', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_code = """
+from awx.main.models import Job
+job = Job.objects.get(id=127)
+print(f"Job [127] for '{job.name}': Status={job.status}, Elapsed={job.elapsed}s, Failed={job.failed}")
+events = job.job_events.order_by('id')
+for e in events:
+    if e.stdout:
+        print(e.stdout)
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_code}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+

scripts/configure_argocd_notifications.py

@@ -0,0 +1,69 @@
+import paramiko
+import yaml
+
+def main():
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+    cm_manifest = """
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: argocd-notifications-cm
+  namespace: infrastructure
+  labels:
+    app.kubernetes.io/component: notifications-controller
+    app.kubernetes.io/instance: argocd
+    app.kubernetes.io/name: argocd-notifications-controller
+    app.kubernetes.io/part-of: argocd
+data:
+  context: |
+    argocdUrl: https://argo.snarfnet.net
+  service.webhook.gotify: |
+    url: http://140.44.4.70:8090/message?token=A8aWxGiToPsAuToMaTiO
+    headers:
+      - name: Content-Type
+        value: application/json
+  template.app-sync-status: |
+    webhook:
+      gotify:
+        method: POST
+        body: |
+          {
+            "title": "ArgoCD: {{.app.metadata.name}} ({{.app.status.sync.status}})",
+            "message": "Application: {{.app.metadata.name}}\\nCluster: {{.app.spec.destination.server}}\\nSync Status: {{.app.status.sync.status}}\\nHealth: {{.app.status.health.status}}\\nURL: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
+            "priority": 4
+          }
+  template.app-health-degraded: |
+    webhook:
+      gotify:
+        method: POST
+        body: |
+          {
+            "title": "ArgoCD: {{.app.metadata.name}} Health DEGRADED",
+            "message": "Application: {{.app.metadata.name}}\\nCluster: {{.app.spec.destination.server}}\\nHealth: {{.app.status.health.status}}\\nURL: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}",
+            "priority": 8
+          }
+  trigger.on-sync-failed: |
+    - description: Application syncing has failed
+      send: [app-sync-status]
+      when: app.status.operationState != nil and app.status.operationState.phase in ['Error', 'Failed']
+  trigger.on-health-degraded: |
+    - description: Application health is degraded
+      send: [app-health-degraded]
+      when: app.status.health.status == 'Degraded'
+"""
+    stdin, stdout, stderr = ssh.exec_command("kubectl apply -f -")
+    stdin.write(cm_manifest)
+    stdin.channel.shutdown_write()
+    print("CM Apply stdout:\n", stdout.read().decode())
+    print("CM Apply stderr:\n", stderr.read().decode())
+
+    stdin, stdout, stderr = ssh.exec_command("kubectl rollout restart deployment/argocd-notifications-controller -n infrastructure")
+    print("Rollout restart:\n", stdout.read().decode())
+    
+    ssh.close()
+
+if __name__ == '__main__':
+    main()

scripts/launch_upgrade_step.py

@@ -0,0 +1,54 @@
+import paramiko, sys, time
+
+version = sys.argv[1] if len(sys.argv) > 1 else "v1.27.16+k3s1"
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_code = f"""
+from awx.main.models import Project, JobTemplate, Job
+import time, json
+
+# 1. Project Update
+p = Project.objects.get(name='Snarfnet Infrastructure Playbooks')
+update = p.update()
+for _ in range(15):
+    update.refresh_from_db()
+    if update.status in ['successful', 'failed', 'error']:
+        break
+    time.sleep(2)
+
+print(f"Project update [{{update.id}}]: {{update.status}}")
+
+# 2. Launch Job Template
+jt = JobTemplate.objects.get(name='Aerodrome K3s Control Plane Upgrade')
+job = jt.create_unified_job(extra_vars=json.dumps({{'target_k3s_version': '{version}'}}))
+job.signal_start()
+print(f"Launched Upgrade Job [{{job.id}}] targeting version '{version}': Status={{job.status}}")
+
+for _ in range(120):
+    job.refresh_from_db()
+    if job.status in ['successful', 'failed', 'error', 'canceled']:
+        break
+    time.sleep(3)
+
+print(f"Job [{{job.id}}] finished with Status={{job.status}}, Elapsed={{job.elapsed}}s, Failed={{job.failed}}")
+
+print('\\n=== PLAY RECAP ===')
+for e in job.job_events.filter(event='playbook_on_stats'):
+    print(e.stdout)
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_code}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+
+for line in iter(stdout.readline, ""):
+    print(line, end="")
+
+# Check kubectl get nodes
+print("\n=== CLUSTER STATUS AFTER STEP ===")
+stdin, stdout, stderr = ssh.exec_command("kubectl get nodes -o wide")
+print(stdout.read().decode())
+
+ssh.close()

scripts/register_aerodrome_argocd.py

@@ -0,0 +1,97 @@
+import paramiko
+import json
+import base64
+
+def main():
+    # 1. Connect to Aerodrome (140.44.4.11)
+    ssh_aero = paramiko.SSHClient()
+    ssh_aero.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh_aero.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+    print("=== STEP 1: Creating argocd-manager ServiceAccount & RBAC on Aerodrome ===")
+    rb_yaml = """
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: argocd-manager
+  namespace: kube-system
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: argocd-manager-role-binding
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: cluster-admin
+subjects:
+- kind: ServiceAccount
+  name: argocd-manager
+  namespace: kube-system
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: argocd-manager-token
+  namespace: kube-system
+  annotations:
+    kubernetes.io/service-account.name: argocd-manager
+type: kubernetes.io/service-account-token
+"""
+    stdin, stdout, stderr = ssh_aero.exec_command("kubectl apply -f -")
+    stdin.write(rb_yaml)
+    stdin.channel.shutdown_write()
+    print("RBAC apply stdout:\n", stdout.read().decode())
+    print("RBAC apply stderr:\n", stderr.read().decode())
+
+    # Get CA cert and Token
+    stdin, stdout, stderr = ssh_aero.exec_command("kubectl get secret -n kube-system argocd-manager-token -o json")
+    secret_data = json.loads(stdout.read().decode())
+    token = base64.b64decode(secret_data["data"]["token"]).decode('utf-8')
+    ca_data = secret_data["data"]["ca.crt"]
+    print(f"Obtained Aerodrome token (len: {len(token)}) and CA cert.")
+    ssh_aero.close()
+
+    # 2. Connect to Spaceport (140.44.4.41) and register cluster in ArgoCD
+    ssh_sp = paramiko.SSHClient()
+    ssh_sp.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh_sp.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+    print("=== STEP 2: Creating cluster secret in Spaceport (namespace infrastructure) ===")
+    config_json = json.dumps({
+        "bearerToken": token,
+        "tlsClientConfig": {
+            "insecure": False,
+            "caData": ca_data
+        }
+    })
+
+    cluster_secret_yaml = f"""
+apiVersion: v1
+kind: Secret
+metadata:
+  name: cluster-aerodrome
+  namespace: infrastructure
+  labels:
+    argocd.argoproj.io/secret-type: cluster
+type: Opaque
+stringData:
+  name: aerodrome
+  server: https://140.44.4.11:6443
+  config: |
+    {config_json}
+"""
+    stdin, stdout, stderr = ssh_sp.exec_command("kubectl apply -f -")
+    stdin.write(cluster_secret_yaml)
+    stdin.channel.shutdown_write()
+    print("Cluster Secret apply stdout:\n", stdout.read().decode())
+    print("Cluster Secret apply stderr:\n", stderr.read().decode())
+
+    # Verify cluster list in argocd
+    stdin, stdout, stderr = ssh_sp.exec_command("kubectl get secrets -n infrastructure -l argocd.argoproj.io/secret-type=cluster")
+    print("Cluster Secrets:\n", stdout.read().decode())
+
+    ssh_sp.close()
+
+if __name__ == '__main__':
+    main()

scripts/register_upgrade_jt.py

@@ -0,0 +1,42 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_code = """
+from awx.main.models import Project, Inventory, Credential, JobTemplate
+
+# 1. Update Project
+p = Project.objects.get(name='Snarfnet Infrastructure Playbooks')
+update = p.update()
+print(f"Triggered Project update [{update.id}]")
+
+inv = Inventory.objects.get(name='Snarfnet Master Inventory')
+cred_root = Credential.objects.get(name='Snarfnet Master SSH Key (root)')
+
+# 2. Register Job Template
+jt, created = JobTemplate.objects.get_or_create(
+    name='Aerodrome K3s Control Plane Upgrade',
+    defaults={
+        'project': p,
+        'inventory': inv,
+        'playbook': 'ansible/playbooks/upgrade_k3s_aerodrome.yml',
+        'ask_variables_on_launch': True,
+        'description': 'Sequential minor release upgrade of Aerodrome K3s control plane with etcd quorum check'
+    }
+)
+if not created:
+    jt.playbook = 'ansible/playbooks/upgrade_k3s_aerodrome.yml'
+    jt.ask_variables_on_launch = True
+    jt.save()
+
+jt.credentials.add(cred_root)
+print(f"Job Template '{jt.name}' (id={jt.id}, created={created}) registered.")
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_code}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+

scripts/run_aerodrome_precheck.py

@@ -0,0 +1,45 @@
+import paramiko, time
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_code = """
+from awx.main.models import Project, JobTemplate, Job
+import time
+
+p = Project.objects.get(name='Snarfnet Infrastructure Playbooks')
+update = p.update()
+for _ in range(15):
+    update.refresh_from_db()
+    if update.status in ['successful', 'failed', 'error']:
+        break
+    time.sleep(2)
+
+print(f"Project update [{update.id}]: {update.status}")
+
+jt = JobTemplate.objects.get(name='Aerodrome K3s Pre-Upgrade Check')
+job = jt.create_unified_job()
+job.signal_start()
+print(f"Launched Job [{job.id}] for '{jt.name}': Status={job.status}")
+
+for _ in range(30):
+    job.refresh_from_db()
+    if job.status in ['successful', 'failed', 'error', 'canceled']:
+        break
+    time.sleep(2)
+
+print(f"Job [{job.id}] finished with Status={job.status}, Elapsed={job.elapsed}s, Failed={job.failed}")
+
+print('\\n=== PLAY RECAP ===')
+for e in job.job_events.filter(event='playbook_on_stats'):
+    print(e.stdout)
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_code}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+
+for line in iter(stdout.readline, ""):
+    print(line, end="")
+
+ssh.close()

scripts/setup_gitops_repo.py

@@ -0,0 +1,41 @@
+import paramiko
+import json
+import sys
+
+def main():
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    
+    # Check if repo already exists
+    stdin, stdout, stderr = ssh.exec_command('kubectl exec -n infrastructure deployment/gitea -c gitea -- ls -la /data/git/gitea-repositories/snarf')
+    snarf_repos = stdout.read().decode()
+    if 'homelab-gitops.git' in snarf_repos:
+        print("homelab-gitops.git already exists on Gitea!")
+    else:
+        print("Generating Gitea API token for Snarf...")
+        stdin, stdout, stderr = ssh.exec_command("kubectl exec -n infrastructure deployment/gitea -c gitea -- gitea admin user generate-access-token -u Snarf -t gitops-token --scopes all --raw")
+        token = stdout.read().decode().strip()
+        print(f"Token obtained (len: {len(token)})")
+        
+        create_payload = json.dumps({
+            "name": "homelab-gitops",
+            "description": "Snarfnet Homelab GitOps, Terraform, and ArgoCD Declarations",
+            "private": False,
+            "auto_init": True
+        })
+        
+        # Call Gitea API inside gitea container
+        curl_cmd = f"kubectl exec -i -n infrastructure deployment/gitea -c gitea -- curl -s -X POST http://localhost:3000/api/v1/user/repos -H \"Authorization: token {token}\" -H \"Content-Type: application/json\" -d @-"
+        stdin, stdout, stderr = ssh.exec_command(curl_cmd)
+        stdin.write(create_payload)
+        stdin.channel.shutdown_write()
+        resp = stdout.read().decode()
+        print("API Response:", resp)
+        
+    stdin, stdout, stderr = ssh.exec_command('kubectl exec -n infrastructure deployment/gitea -c gitea -- ls -la /data/git/gitea-repositories/snarf')
+    print("Repos under snarf:\n", stdout.read().decode())
+    ssh.close()
+
+if __name__ == '__main__':
+    main()

scripts/test_zabbix_13.py

@@ -0,0 +1,12 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+cmd = "kubectl exec -n monitoring deployment/zabbix-zabbix-server -c zabbix-server -- zabbix_get -s 140.44.4.13 -k system.uptime"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print("OUT:", stdout.read().decode())
+print("ERR:", stderr.read().decode())
+ssh.close()
+

scripts/verify_cluster_health.py

@@ -0,0 +1,20 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+cmds = [
+    "kubectl get pods -n kube-system",
+    "kubectl get pods -n awx",
+    "kubectl get pods -n infra",
+    "kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness | head -n 15"
+]
+
+for cmd in cmds:
+    print(f"\n=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+
+ssh.close()
+