docs: add network blueprint for high-availability DNS, honeypot, and UDM firewall hardening

docs/DNS_Setup_Plan.md

@@ -0,0 +1,182 @@
+# Project Blueprint: High-Availability DNS Stack, Network Honeypot & UDM Pro Network Security
+
+## 1. Executive Summary & Live UDM Pro Topology Assessment
+
+Based on the live UDM Pro controller diagnostics, we have mapped the exact network topography, identified critical security exposures, and tailored the implementation plan.
+
+---
+
+### 1.1 Live Network & VLAN Topography
+
+| VLAN ID | Network Name | Subnet / CIDR | Gateway IP | Active Leases | Security Classification |
+|:---:|---|---|---|:---:|---|
+| **1** | **LAN** | `14.44.4.0/25` | `14.44.4.1` | 14 / 117 | Trusted Internal Clients |
+| **2** | **Management** | `140.44.44.0/24` | `140.44.44.1` | 9 / 51 | Out-of-band (iDRAC, Switches, APs) |
+| **3** | **Spacedock** | `140.44.1.0/24` | `140.44.1.1` | 0 / 249 | Harvester VM / Hypervisor Tenant |
+| **20** | **Compute** | `140.44.4.0/25` | `140.44.4.1` | 1 / 15 (Static Nodes) | Mission-Critical K3s Clusters, Swarm |
+| **30** | **Media** | `192.168.1.0/25` | `192.168.1.1` | 1 / 91 | Streaming & Entertainment Devices |
+| **40** | **Storage** | `14.10.10.0/25` | `14.10.10.1` | 2 / 125 | Synology NAS (`.71`, `.72`) NFS / iSCSI |
+| **50** | **Smart Home** | `14.4.4.0/25` | `14.4.4.1` | 53 / 85 | IoT Devices, Gaming Host (`14.4.4.15`) |
+| **60** | **HA** | `140.4.4.0/25` | `140.4.4.1` | 0 / 7 | High-Availability Interconnects |
+| **VPN** | **WireGuard / Teleport** | `192.168.3.0/24` / `.2.0/24` | UDM Pro | Remote | Ingress Remote Access |
+
+---
+
+### 1.2 Critical Security Findings from UDM Pro Diagnostics
+
+1. **Default Security Posture is `Allow All`:**
+   * Currently, the UDM Pro routes traffic freely between all subnets without restriction.
+   * **Risk:** The 53 IoT devices in Smart Home (`14.4.4.0/25`) and Media devices can directly initiate connections to Kubernetes nodes, etcd ports, SSH, and Synology NAS shares in `Compute` (`140.44.4.0/25`) and `Storage` (`14.10.10.0/25`).
+2. **WAN Port Forwarding to Smart Home Host (`14.4.4.15`):**
+   * Port forwards for game servers (`FS25` ports 8443/10823, `Stationeers` ports 6443/27015) point directly from WAN (`174.106.252.96`) to host `14.4.4.15` on the Smart Home VLAN.
+   * If an exploit compromises this host, an attacker currently has unhindered lateral access to the entire homelab infrastructure.
+3. **DNS Leaks & Lack of Local Filtering:**
+   * UDM Pro WAN is set to `Auto DNS Server` (Spectrum ISP DNS `0.0.0.0`).
+   * No DNS ad/malware filtering or split-horizon DNS caching is currently active across the subnets.
+
+---
+
+## 2. High-Availability DNS Stack (Pi-hole + Technitium)
+
+```mermaid
+flowchart TD
+    subgraph Clients["LAN Clients across all VLANs (VLAN 1, 20, 30, 50, VPN)"]
+        Devices["IoT Devices (53), Laptops, Servers, Hypervisors"]
+    end
+
+    subgraph UDM["UniFi Dream Machine Pro (140.44.4.1)"]
+        DHCP["DHCP Option 6 per VLAN:<br>Primary DNS: 140.44.4.35 (Aerodrome VIP)<br>Secondary DNS: 140.44.4.65 (Spaceport VIP)"]
+        NAT["DNS Redirection Rule:<br>Capture port 53 to hardcoded DNS (8.8.8.8) -> Redirect to VIPs"]
+    end
+
+    subgraph AeroCluster["Aerodrome K3s Cluster (VLAN 20)"]
+        AeroVIP["MetalLB VIP: 140.44.4.35:53 (UDP/TCP)"]
+        AeroPihole["Pi-hole Instance 1<br>(Ad-blocking, Caching, FTL rate-limit: 0)"]
+        AeroTech["Technitium Instance 1 (Primary Authoritative)<br>(snarfnet.net Zone, Reverse PTR, DNSSEC)"]
+        AeroStorage["Longhorn Storage Volume"]
+    end
+
+    subgraph SpaceCluster["Spaceport K3s Cluster (VLAN 20)"]
+        SpaceVIP["MetalLB VIP: 140.44.4.65:53 (UDP/TCP)"]
+        SpacePihole["Pi-hole Instance 2<br>(Ad-blocking, Caching, FTL rate-limit: 0)"]
+        SpaceTech["Technitium Instance 2 (Secondary Authoritative)<br>(Zone Transfer AXFR/IXFR from Primary)"]
+        SpaceStorage["Longhorn Storage Volume"]
+    end
+
+    subgraph Upstream["Encrypted Upstream Resolvers"]
+        DoH["DNS-over-HTTPS (Quad9 / Cloudflare / Mullvad)"]
+    end
+
+    Devices -->|DNS Queries (UDP/TCP 53)| DHCP
+    DHCP -->|Primary| AeroVIP
+    DHCP -.->|Secondary / Failover| SpaceVIP
+    AeroVIP --> AeroPihole
+    SpaceVIP --> SpacePihole
+    AeroPihole -->|Recursive / Local Lookups| AeroTech
+    SpacePihole -->|Recursive / Local Lookups| SpaceTech
+    AeroTech <==>|Automatic Zone Sync (AXFR)| SpaceTech
+    AeroTech -->|Encrypted DoH| DoH
+    SpaceTech -->|Encrypted DoH| DoH
+    AeroPihole --- AeroStorage
+    AeroTech --- AeroStorage
+    SpacePihole --- SpaceStorage
+    SpaceTech --- SpaceStorage
+```
+
+### 2.1 Addressing the Rate-Limiting & Sluggishness Issue
+1. **FTL Rate Limit Bypass:** Set `FTLCONF_RATE_LIMIT=0` so local bursts (such as 53 Smart Home devices polling simultaneously) are never throttled.
+2. **Technitium Engine:** Replaces single-threaded Unbound with a high-performance .NET 9 asynchronous resolver capable of servicing tens of thousands of queries per second with sub-millisecond RAM caching.
+3. **High Availability:** MetalLB VIPs on Aerodrome (`140.44.4.35`) and Spaceport (`140.44.4.65`) allow continuous service even during node reboots or maintenance.
+4. **Authoritative Split-Horizon:** Technitium will authoritatively host the internal `snarfnet.net` forward zone and reverse PTR zones (`4.44.140.in-addr.arpa`, `4.4.14.in-addr.arpa`, `10.10.14.in-addr.arpa`), eliminating slow mDNS or external lookup delays.
+
+---
+
+## 3. Network Honeypot & UDM Pro Firewall Hardening
+
+```mermaid
+flowchart LR
+    subgraph WAN_Threats["External & Lateral Threats"]
+        Internet["Public Internet"]
+        CompromisedIoT["Compromised IoT / Host (e.g. 14.4.4.15)"]
+    end
+
+    subgraph UDM_FW["UDM Pro Firewall Rules (LAN IN)"]
+        Rule1["Rule 1: Allow Established & Related"]
+        Rule2["Rule 2: Allow DNS (Port 53 to .35 & .65)"]
+        Rule3["Rule 3: Allow NTP (Port 123 to .11 & .44)"]
+        Rule4["Rule 4: Drop IoT -> Compute (VLAN 50 -> 20)"]
+        Rule5["Rule 5: Drop IoT -> Storage (VLAN 50 -> 40)"]
+        Rule6["Rule 6: Drop Media -> Compute (VLAN 30 -> 20)"]
+    end
+
+    subgraph HoneypotNet["Honeypot Decoy (VLAN 50 / Decoy IP 14.4.4.99)"]
+        OpenCanary["OpenCanary Honeypot<br>(Simulating SSH:22, HTTP:80, SMB:445, RDP:3389)"]
+    end
+
+    subgraph AlertSystem["Alerting & Audit Pipeline"]
+        Gotify["Gotify Push Notification<br>(Instant alert to your phone)"]
+        Zabbix["Zabbix Security Audit Log"]
+    end
+
+    CompromisedIoT -.->|Probes decoy IP| OpenCanary
+    OpenCanary -->|High Priority Webhook| Gotify
+    OpenCanary -->|Syslog Event| Zabbix
+    CompromisedIoT -->|Blocked by Rule 4| UDM_FW
+```
+
+### 3.1 Network Honeypot Deployment
+* **Tool:** **OpenCanary** (lightweight, zero false-positives).
+* **Placement:**
+  * Assigned a decoy IP on the Smart Home subnet (e.g., `14.4.4.99`) where the gaming port forwards exist.
+  * Emulates vulnerable services: SSH (`22`), Telnet (`23`), Web admin (`80`/`8080`), SMB (`445`), and RDP (`3389`).
+* **Alerting:**
+  * When any internal device attempts a port scan or lateral connection to `14.4.4.99`, OpenCanary immediately sends a critical push notification to **Gotify** (`gotify.snarfnet.net`) and generates an alert in **Zabbix**.
+
+---
+
+### 3.2 UDM Pro Firewall Rules Matrix (LAN IN)
+
+We will define firewall groups on the UDM Pro:
+* **RFC1918_Subnets:** All internal private ranges (`140.44.0.0/16`, `14.0.0.0/8`, `192.168.0.0/16`).
+* **K8s_DNS_VIPs:** `140.44.4.35`, `140.44.4.65`.
+* **NTP_Servers:** `140.44.4.11`, `14.4.4.44`.
+
+| Order | Rule Name | Type | Source | Destination | Action | Purpose |
+|:---:|---|:---:|---|---|:---:|---|
+| **2001** | `Allow Established/Related` | LAN IN | Any | Any (State: Established, Related) | **ACCEPT** | Essential return traffic |
+| **2002** | `Allow Cluster DNS` | LAN IN | All VLANs | `K8s_DNS_VIPs` (Port: 53 UDP/TCP) | **ACCEPT** | Clients resolve DNS |
+| **2003** | `Allow Cluster NTP` | LAN IN | All VLANs | `NTP_Servers` (Port: 123 UDP) | **ACCEPT** | Chrony time sync |
+| **2004** | `Allow Trusted to All` | LAN IN | `VLAN 1 (LAN)`, `VLAN 20 (Compute)` | RFC1918_Subnets | **ACCEPT** | Admins & servers access all VLANs |
+| **2005** | `Isolate SmartHome to Compute` | LAN IN | `VLAN 50 (Smart Home)` | `VLAN 20 (Compute)` | **DROP** | Block IoT pivoting into K8s |
+| **2006** | `Isolate SmartHome to Storage` | LAN IN | `VLAN 50 (Smart Home)` | `VLAN 40 (Storage)` | **DROP** | Protect Synology NAS from IoT |
+| **2007** | `Isolate Media to Compute` | LAN IN | `VLAN 30 (Media)` | `VLAN 20 (Compute)` | **DROP** | Block media devices from cluster |
+| **2008** | `Isolate Guest Network` | GUEST IN | Guest Network | RFC1918_Subnets | **DROP** | Restrict guest traffic to Internet only |
+
+---
+
+## 4. Phased Implementation Roadmap
+
+### Phase 1: Teardown Lingering Stubs
+* Cleanly delete orphaned, broken deployments and services in `infra` (Aerodrome) and `infrastructure` (Spaceport):
+  * Remove `deployment/pihole`, `deployment/technitium-dns`, and non-functional NodePort services.
+
+### Phase 2: Deploy Dual-Cluster Pi-hole + Technitium
+* Create Longhorn storage volumes on both clusters for persistent configs and blocklists.
+* Deploy **Aerodrome Primary Stack**:
+  * Technitium on internal port `5353`.
+  * Pi-hole with MetalLB LoadBalancer VIP **`140.44.4.35:53`** (UDP & TCP).
+* Deploy **Spaceport Secondary Stack**:
+  * Technitium on internal port `5353` configured for AXFR zone transfers from Aerodrome.
+  * Pi-hole with MetalLB LoadBalancer VIP **`140.44.4.65:53`** (UDP & TCP).
+* Populate `snarfnet.net` local records and reverse PTR zones.
+
+### Phase 3: Network Honeypot Deployment
+* Deploy containerized OpenCanary with virtual listener on `14.4.4.99`.
+* Configure webhook bridge to Gotify server (`140.44.4.70`) for immediate push notification upon any probe.
+
+### Phase 4: UDM Pro Network Configuration & Policy Hardening
+* Configure DHCP Scope DNS servers on UDM Pro for all VLANs:
+  * Primary: `140.44.4.35`
+  * Secondary: `140.44.4.65`
+* Set UDM Pro WAN DNS to `140.44.4.35` & `140.44.4.65` (or Quad9 fallback).
+* Apply LAN IN firewall rule matrix on UDM Pro to enforce strict VLAN isolation.

scripts/apply_coredns_spaceport.py

@@ -0,0 +1,48 @@
+import paramiko
+import sys
+import time
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+
+cm_yaml = """cat << 'EOF' | k3s kubectl apply -f -
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: coredns-custom
+  namespace: kube-system
+data:
+  snarfnet.override: |
+    hosts {
+      140.44.4.34 aerodrome.snarfnet.net
+      140.44.4.90 harvester.spacedock.snarfnet.net
+      fallthrough
+    }
+EOF
+"""
+
+print("Applying coredns-custom on Spaceport...")
+stdin, stdout, stderr = client.exec_command(cm_yaml)
+print(stdout.read().decode('utf-8'))
+print(stderr.read().decode('utf-8'))
+
+print("\nRestarting coredns deployment on Spaceport...")
+client.exec_command("k3s kubectl -n kube-system rollout restart deployment/coredns")
+client.exec_command("k3s kubectl -n kube-system rollout status deployment/coredns --timeout=60s")
+
+print("\nRestarting cattle-cluster-agent on Spaceport...")
+client.exec_command("k3s kubectl -n cattle-system rollout restart deployment/cattle-cluster-agent")
+client.exec_command("k3s kubectl -n cattle-system rollout status deployment/cattle-cluster-agent --timeout=60s")
+
+time.sleep(5)
+print("\nChecking cattle-cluster-agent logs:")
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system logs -l app=cattle-cluster-agent --tail=25")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+

scripts/audit_cluster_workloads.py

@@ -0,0 +1,21 @@
+import paramiko
+
+def inspect_cluster(ip, name):
+    print(f"\n=======================================================")
+    print(f"=== {name} WORKLOADS ({ip}) ===")
+    print(f"=======================================================")
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    
+    cmd = "kubectl get deploy,sts,ds -A -o custom-columns=NAMESPACE:.metadata.namespace,KIND:.kind,NAME:.metadata.name,IMAGES:.spec.template.spec.containers[*].image"
+    _, out, _ = ssh.exec_command(cmd)
+    lines = out.read().decode().splitlines()
+    for l in lines:
+        if not any(k in l for k in ['kube-system', 'cattle-', 'metallb-system', 'longhorn-system']):
+            print(l)
+    ssh.close()
+
+inspect_cluster('140.44.4.11', 'AERODROME')
+inspect_cluster('140.44.4.41', 'SPACEPORT')
+

scripts/audit_gitops_manifests.py

@@ -0,0 +1,40 @@
+import os
+import re
+import paramiko
+
+manifest_root = r"c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests"
+
+print("=== GITOPS MANIFEST IMAGE AUDIT ===")
+
+image_re = re.compile(r'image:\s*([^\s"\']+)', re.IGNORECASE)
+chart_re = re.compile(r'chart:\s*([^\s"\']+)', re.IGNORECASE)
+version_re = re.compile(r'version:\s*([^\s"\']+)', re.IGNORECASE)
+
+manifest_data = {}
+
+for root, dirs, files in os.walk(manifest_root):
+    for f in files:
+        if f.endswith('.yaml') or f.endswith('.yml'):
+            path = os.path.join(root, f)
+            rel_path = os.path.relpath(path, manifest_root)
+            with open(path, 'r', encoding='utf-8', errors='ignore') as fp:
+                content = fp.read()
+                images = image_re.findall(content)
+                charts = chart_re.findall(content)
+                versions = version_re.findall(content)
+                if images or charts:
+                    manifest_data[rel_path] = {
+                        'images': list(set(images)),
+                        'charts': list(set(charts)),
+                        'versions': list(set(versions))
+                    }
+
+for rel_path, data in sorted(manifest_data.items()):
+    print(f"\nFile: {rel_path}")
+    if data['images']:
+        print("  Images:", ", ".join(data['images']))
+    if data['charts']:
+        print("  Charts:", ", ".join(data['charts']))
+    if data['versions']:
+        print("  Versions:", ", ".join(data['versions']))
+

scripts/audit_spaceport_kernels.py

@@ -0,0 +1,39 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+spaceport_nodes = [
+    ('sninfr0lnxctl01', '140.44.4.41'),
+    ('sninfr0lnxctl02', '140.44.4.42'),
+    ('sninfr0lnxctl03', '140.44.4.43'),
+    ('sninfr0lnxctl04', '140.44.4.44'),
+    ('sninfr0lnxwrk01', '140.44.4.51'),
+    ('sninfr0lnxwrk02', '140.44.4.52'),
+    ('sninfr0lnxwrk03', '140.44.4.53'),
+    ('sninfr0lnxwrk04', '140.44.4.54'),
+    ('sninfr0lnxwrk05', '140.44.4.55'),
+    ('sninfr0lnxwrk06', '140.44.4.56'),
+]
+
+print(f"{'NODE':<18} {'IP':<15} {'RUNNING KERNEL':<22} {'LATEST MODULES':<25} {'VETH LOADED':<12} {'UPTIME'}")
+print("="*105)
+
+for name, ip in spaceport_nodes:
+    try:
+        client.connect(ip, username="root", key_filename=key_path, timeout=5)
+        stdin, stdout, stderr = client.exec_command("uname -r; ls -1 /lib/modules | sort -V | tail -n 1; lsmod | grep -q veth && echo YES || echo NO; uptime -p")
+        lines = [l.strip() for l in stdout.read().decode('utf-8').strip().split('\n') if l.strip()]
+        running_k = lines[0] if len(lines) > 0 else "?"
+        latest_mod = lines[1] if len(lines) > 1 else "?"
+        veth = lines[2] if len(lines) > 2 else "?"
+        uptime_str = lines[3] if len(lines) > 3 else "?"
+        print(f"{name:<18} {ip:<15} {running_k:<22} {latest_mod:<25} {veth:<12} {uptime_str}")
+        client.close()
+    except Exception as e:
+        print(f"{name:<18} {ip:<15} ERROR: {e}")
+

scripts/authorize_and_test.py

@@ -0,0 +1,35 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+pub_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6ycMke4AaV2qzWfULZCTZOKhSKf4pY2qRGYjYzs1MGKutJxt2jkYNK2aS0XffxLRR49v23zHbBwLYraNl2Mkb9rsKsyvKn8CAa1CEl9NxOci4LDRT8CeWHRnMZn8Mdu1pfeRwM26/Bjnd4RJrSKC7fyM0OlOpEp0DgnuPBqBtPMFGw+TCUT8eZ0pBrAI94wEan4l35kmuBcqm6RH3KuD1dDJKnPLzJ3/T21CViYL0GgFIDDor/hGI45VfOI/jKmkgM4YoudTCde6uPV5e2yaJpORZlR4nDxg9ntXp8HxbHGZkXYr55Ibsjem0lMU6nBh33jGbJVvat1Ay4oNCbygJsHfC/8ppe0bnLhnasxVAXhTvklZZL/sf7K/VS0ZifpfqB6DKtr1lziOm2wQeahv+Y1uoHkVNa3XY8Fx7xGPeT1BGd9lfcbUoweoTdMPMIPaSZlNGY32AHYkM4PShG9yI1coAqXzMby0+Z/Qwtr4l6cLKVvcGgqh5sd+z1+YvZbs= root@SNINFR2LNXCTL01"
+
+# 1. Authorize on 140.44.4.71
+c71 = paramiko.SSHClient()
+c71.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c71.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+
+cmd = f"""
+mkdir -p ~/.ssh && chmod 700 ~/.ssh
+grep -qF '{pub_key}' ~/.ssh/authorized_keys 2>/dev/null || echo '{pub_key}' >> ~/.ssh/authorized_keys
+chmod 600 ~/.ssh/authorized_keys
+"""
+c71.exec_command(cmd)
+c71.close()
+print("Authorized 140.44.4.11 key on 140.44.4.71.")
+
+# 2. Test SSH from 140.44.4.11 to 140.44.4.71
+c11 = paramiko.SSHClient()
+c11.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c11.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = c11.exec_command("ssh -o BatchMode=yes -o StrictHostKeyChecking=no root@140.44.4.71 hostname")
+out = stdout.read().decode('utf-8').strip()
+err = stderr.read().decode('utf-8').strip()
+print("Result of ssh from 140.44.4.11 to 140.44.4.71:")
+print("STDOUT:", out)
+if err: print("STDERR:", err)
+c11.close()
+

scripts/check_13_conf.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command('docker exec $(docker ps -q -f name=nginx-proxy-manager_app) cat /data/nginx/proxy_host/13.conf | grep server_name')
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_214_pods.py

@@ -0,0 +1,25 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system get pods -o wide")
+print("=== PODS IN CATTLE-SYSTEM ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system rollout status deployment/rancher --timeout=30s")
+print("=== ROLLOUT STATUS ===")
+print(stdout.read().decode('utf-8'))
+print(stderr.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("curl -k -I -s https://aerodrome.snarfnet.net | head -n 10")
+print("=== ENDPOINT ===")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+

scripts/check_215_pods.py

@@ -0,0 +1,29 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system get pods -o wide")
+print("=== PODS IN CATTLE-SYSTEM ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system rollout status deployment/rancher --timeout=30s")
+print("=== ROLLOUT STATUS ===")
+print(stdout.read().decode('utf-8'))
+print(stderr.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("curl -k -I -s https://aerodrome.snarfnet.net | head -n 10")
+print("=== ENDPOINT ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm list -n cattle-system")
+print("=== HELM LIST ===")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+

scripts/check_51.py

@@ -0,0 +1,19 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.51', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+run("ss -tulpn | grep ':80 '")
+run("ip a")
+run("kubectl get pod -A -o wide | grep sninfr4lnxwrk01 | grep pihole || true")
+ssh.close()
+

scripts/check_51_nat.py

@@ -0,0 +1,18 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.51', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+run("iptables -t nat -S | grep -E '80\b|31905|31937'")
+run("crictl ps")
+ssh.close()
+

scripts/check_71_containers.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_aero_health.py

@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = client.exec_command('k3s kubectl get pods -n cattle-system')
+print("=== AERODROME CATTLE-SYSTEM PODS ===")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_aerodrome_pods.py

@@ -0,0 +1,15 @@
+import paramiko
+
+s = paramiko.SSHClient()
+s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+s.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    _, out, _ = s.exec_command(cmd)
+    print(out.read().decode())
+
+run("kubectl get pods -A | grep -v -E 'Running|Completed' || true")
+run("kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,ROBUSTNESS:.status.robustness,STATE:.status.state")
+s.close()
+

scripts/check_aerodrome_preflight.py

@@ -0,0 +1,19 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+run("kubectl get nodes")
+run("kubectl get pods -A | grep -v -E 'Running|Completed' || true")
+run("kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,ROBUSTNESS:.status.robustness | grep -v healthy || true")
+ssh.close()
+

scripts/check_aerodrome_quick.py

@@ -0,0 +1,8 @@
+import paramiko
+ssh=paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+_,o,_=ssh.exec_command('kubectl get nodes')
+print(o.read().decode())
+ssh.close()
+

scripts/check_after_reboot.py

@@ -0,0 +1,29 @@
+import paramiko
+import sys
+import time
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+print("=== CHECKING VETH ON SNINFR0LNXCTL03 ===")
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("modprobe veth; lsmod | grep veth; systemctl is-active k3s-server || systemctl is-active k3s")
+print(stdout.read().decode('utf-8'))
+client.close()
+
+time.sleep(5)
+
+print("=== CHECKING PODS IN CATTLE-SYSTEM ON SPACEPORT ===")
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system get pods -o wide")
+print(stdout.read().decode('utf-8'))
+
+print("=== CHECKING POD EVENTS ===")
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system describe pod -l app=cattle-cluster-agent | grep -A 10 Events:")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+

scripts/check_agent_deploy.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system get deployment cattle-cluster-agent -o yaml")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_agent_logs.py

@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system logs -l app=cattle-cluster-agent --tail=100")
+print("=== SPACEPORT CATTLE-CLUSTER-AGENT LOGS ===")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_agent_logs_tail.py

@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system logs -l app=cattle-cluster-agent -c cluster-register --tail=20")
+print("=== AGENT LOGS ===")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_agent_now.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system logs -l app=cattle-cluster-agent -c cluster-register --tail=25")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_all_coder_snapshots.py

@@ -0,0 +1,24 @@
+import paramiko
+
+nodes = [
+    ("wrk01", "140.44.4.51"),
+    ("wrk02", "140.44.4.52"),
+    ("wrk03", "140.44.4.53"),
+    ("wrk04", "140.44.4.54"),
+    ("wrk05", "140.44.4.55"),
+    ("wrk06", "140.44.4.56"),
+]
+
+for name, ip in nodes:
+    try:
+        ssh = paramiko.SSHClient()
+        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+        ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=4)
+        _, out, _ = ssh.exec_command("find /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/ -path '*/home/coder/.config/coderv2' 2>/dev/null")
+        res = out.read().decode().strip()
+        if res:
+            print(f"[{name} - {ip}] Found:\n{res}")
+        ssh.close()
+    except Exception as e:
+        print(f"[{name}] err: {e}")
+

scripts/check_all_endpoints.py

@@ -0,0 +1,19 @@
+import subprocess
+
+endpoints = [
+    ("Argo CD", "https://argo.snarfnet.net"),
+    ("Rancher Manager", "https://aerodrome.snarfnet.net"),
+    ("AWX Console", "https://awx.snarfnet.net"),
+    ("Authentik SSO", "https://authentik.snarfnet.net"),
+    ("Uptime Kuma", "https://uptime.snarfnet.net/dashboard"),
+    ("Gitea GitOps", "http://140.44.4.51:90"),
+    ("Zabbix Server", "http://140.44.4.51:8888"),
+    ("Portainer Server", "https://140.44.4.71:9443")
+]
+
+print("=== ENDPOINT HEALTH CHECK ===")
+for name, url in endpoints:
+    res = subprocess.run(f"curl.exe -k -s -o nul -w '%{{http_code}}' {url}", capture_output=True, text=True, shell=True)
+    code = res.stdout.strip()
+    print(f"[{code}] {name} ({url})")
+

scripts/check_all_tls.py

@@ -0,0 +1,27 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+# Check certificates in Aerodrome across all namespaces
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("""
+KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get secrets -A --field-selector=type=kubernetes.io/tls
+""")
+print("=== TLS SECRETS ON AERODROME ===")
+print(stdout.read().decode('utf-8'))
+client.close()
+
+# Check certificates on Spaceport across all namespaces
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("""
+KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get secrets -A --field-selector=type=kubernetes.io/tls
+""")
+print("=== TLS SECRETS ON SPACEPORT ===")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_apis.py

@@ -0,0 +1,25 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+
+test_cmd = """
+cd /tmp/rancher-test
+KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm template rancher ./rancher --namespace cattle-system --set hostname=aerodrome.snarfnet.net --set ingress.tls.source=secret | grep "apiVersion:" | sort | uniq -c
+"""
+
+stdin, stdout, stderr = client.exec_command(test_cmd)
+out = stdout.read().decode('utf-8')
+err = stderr.read().decode('utf-8')
+print("APIs used:")
+print(out)
+if err:
+    print("STDERR:", err)
+
+client.close()

scripts/check_argo_apps.py

@@ -0,0 +1,14 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+stdin, stdout, stderr = ssh.exec_command("kubectl get applications -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status")
+print(stdout.read().decode())
+err = stderr.read().decode()
+if err:
+    print("ERR:", err)
+
+ssh.close()
+

scripts/check_argo_ing.py

@@ -0,0 +1,22 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+run("kubectl get ingress argo-ing -n infrastructure -o yaml")
+run("kubectl get svc pihole-web -n infrastructure -o yaml")
+run("kubectl get svc -n infrastructure")
+run("kubectl get svc -A | grep -E 'LoadBalancer|140.44.4.62'")
+run("curl -k -I -H 'Host: argo.snarfnet.net' http://140.44.4.62")
+run("curl -k -I -H 'Host: argo.snarfnet.net' https://140.44.4.62")
+ssh.close()
+

scripts/check_argocd_now.py

@@ -0,0 +1,11 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+_, out, _ = ssh.exec_command("kubectl get applications -n infrastructure -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status")
+print(out.read().decode())
+
+ssh.close()
+

scripts/check_blkid.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("blkid")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_boot_ctl03.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("ls -la /boot; ls -la /boot/firmware 2>/dev/null | head -n 20")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_cattle_all.py

@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system get pods -o wide")
+print("=== SPACEPORT CATTLE-SYSTEM PODS ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system get deployment cattle-cluster-agent")
+print("=== DEPLOYMENT STATUS ===")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+

scripts/check_cattle_pods.py

@@ -0,0 +1,14 @@
+import paramiko
+
+def check_cattle(ip, cluster):
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    print(f"\n[{cluster}] Checking cattle-system pods...")
+    _, out, _ = ssh.exec_command("kubectl get pods -n cattle-system -o wide")
+    print(out.read().decode().strip())
+    ssh.close()
+
+check_cattle('140.44.4.11', 'AERODROME')
+check_cattle('140.44.4.41', 'SPACEPORT')
+

scripts/check_cattle_spaceport.py

@@ -0,0 +1,29 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+
+cmds = [
+    "k3s kubectl -n cattle-system get pods -o wide",
+    "k3s kubectl -n cattle-system describe pod -l app=cattle-cluster-agent",
+    "k3s kubectl get events -A --sort-by='.lastTimestamp' | tail -n 20"
+]
+
+for cmd in cmds:
+    print(f"\n==========================================")
+    print(f">>> Running on Spaceport: {cmd}")
+    print(f"==========================================")
+    stdin, stdout, stderr = client.exec_command(cmd)
+    print(stdout.read().decode('utf-8'))
+    err = stderr.read().decode('utf-8')
+    if err:
+        print("STDERR:", err)
+
+client.close()
+

scripts/check_cattle_spaceport_clean.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system get pods -o wide")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_certbot.py

@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = client.exec_command("docker exec $(docker ps -q -f name=nginx-proxy-manager_app) certbot certificates")
+print("OUT:", stdout.read().decode('utf-8'))
+print("ERR:", stderr.read().decode('utf-8'))
+client.close()
+

scripts/check_chart_kubeversions.py

@@ -0,0 +1,22 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+versions = [
+    "2.7.10", "2.8.5", "2.9.3", "2.10.3", "2.11.3", "2.12.3", "2.13.3", "2.14.3", "2.15.3"
+]
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+
+for ver in versions:
+    stdin, stdout, stderr = client.exec_command(f"KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm show chart rancher-stable/rancher --version {ver} | grep kubeVersion")
+    out = stdout.read().decode('utf-8').strip()
+    print(f"Rancher {ver}: {out}")
+
+client.close()
+

scripts/check_chart_versions.py

@@ -0,0 +1,13 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm search repo rancher-stable/rancher --versions > /tmp/rancher_versions.txt && head -n 30 /tmp/rancher_versions.txt")
+print(stdout.read().decode('utf-8'))
+client.close()

scripts/check_cluster_cond.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = client.exec_command('k3s kubectl get clusters.provisioning.cattle.io -A')
+print("=== PROVISIONING CLUSTERS ===")
+print(stdout.read().decode('utf-8'))
+client.close()

scripts/check_cluster_details.py

@@ -0,0 +1,13 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command('KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get clusters.management.cattle.io -o wide')
+print(stdout.read().decode('utf-8'))
+client.close()

scripts/check_cluster_ready.py

@@ -0,0 +1,16 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("""
+KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get clusters.management.cattle.io -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.displayName}{"\tConnected: "}{.status.agentImage}{"\tReady: "}{range .status.conditions[?(@.type=="Ready")]}{.status}{" - "}{.message}{end}{"\n"}{end}'
+""")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_cluster_tokens.py

@@ -0,0 +1,25 @@
+import paramiko
+import sys
+import yaml
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get clusterregistrationtokens.management.cattle.io -A -o yaml")
+data = yaml.safe_load(stdout.read().decode('utf-8'))
+client.close()
+
+for item in data.get('items', []):
+    cluster = item.get('spec', {}).get('clusterName')
+    token = item.get('status', {}).get('token')
+    cmd = item.get('status', {}).get('command')
+    manifest_url = item.get('status', {}).get('manifestUrl')
+    print(f"\nCluster: {cluster}")
+    print(f"Token: {token}")
+    print(f"Command: {cmd}")
+    print(f"Manifest URL: {manifest_url}")
+

scripts/check_cluster_yaml.py

@@ -0,0 +1,26 @@
+import paramiko
+import sys
+import yaml
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get clusters.management.cattle.io -o yaml")
+data = yaml.safe_load(stdout.read().decode('utf-8'))
+client.close()
+
+for item in data.get('items', []):
+    name = item.get('metadata', {}).get('name')
+    disp = item.get('spec', {}).get('displayName')
+    print(f"\n==========================================")
+    print(f"Cluster: {disp} ({name})")
+    print(f"==========================================")
+    conditions = item.get('status', {}).get('conditions', [])
+    for c in conditions:
+        if c.get('status') != 'True' or c.get('type') in ['Ready', 'Connected']:
+            print(f"  [{c.get('type')}]: status={c.get('status')} reason={c.get('reason')} msg={c.get('message')}")
+

scripts/check_cmdline.py

@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("cat /proc/cmdline; dmesg | grep -i 'Kernel command line'")
+print("CMDLINE:")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("ls -la /dev/disk/by-id/; ls -la /dev/disk/by-uuid/")
+print("DISKS:")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+

scripts/check_coder_logs.py

@@ -0,0 +1,11 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+_, out, _ = ssh.exec_command('kubectl logs -n infrastructure -l app.kubernetes.io/name=coder --tail=30')
+print(out.read().decode())
+
+ssh.close()
+

scripts/check_coder_page.py

@@ -0,0 +1,14 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+_, out, _ = ssh.exec_command("curl -s https://coder.snarfnet.net/api/v2/users/first")
+print("First user status:", out.read().decode())
+
+_, out2, _ = ssh.exec_command("curl -s https://coder.snarfnet.net/api/v2/experiments")
+print("Experiments:", out2.read().decode())
+
+ssh.close()
+

scripts/check_coredns.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n kube-system get configmap coredns -o yaml")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_coredns_logs.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n kube-system logs -l k8s-app=kube-dns --tail=30")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_ctl02_uptime.py

@@ -0,0 +1,18 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+try:
+    client.connect("140.44.4.42", username="root", key_filename=key_path, timeout=5)
+    stdin, stdout, stderr = client.exec_command("uptime; uname -r; modprobe veth; lsmod | grep veth")
+    print("OUTPUT from ctl02:")
+    print(stdout.read().decode('utf-8'))
+    client.close()
+except Exception as e:
+    print("Connect error:", e)
+

scripts/check_ctl03.py

@@ -0,0 +1,33 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+node_ip = "140.44.4.43"
+print(f"Connecting to sninfr0lnxctl03 ({node_ip})...")
+client.connect(node_ip, username="root", key_filename=key_path, timeout=10)
+
+cmds = [
+    "uname -r",
+    "ls -d /lib/modules/*",
+    "lsmod | grep -E 'veth|vxlan|br_netfilter'",
+    "modprobe veth",
+    "dmesg | tail -n 25"
+]
+
+for cmd in cmds:
+    print(f"\n>>> Running on {node_ip}: {cmd}")
+    stdin, stdout, stderr = client.exec_command(cmd)
+    out = stdout.read().decode('utf-8').strip()
+    err = stderr.read().decode('utf-8').strip()
+    if out:
+        print(out)
+    if err:
+        print("STDERR:", err)
+
+client.close()
+

scripts/check_current_status.py

@@ -0,0 +1,25 @@
+import paramiko
+import sys
+import yaml
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get clusters.management.cattle.io -o yaml")
+data = yaml.safe_load(stdout.read().decode('utf-8'))
+client.close()
+
+for item in data.get('items', []):
+    name = item.get('metadata', {}).get('name')
+    disp = item.get('spec', {}).get('displayName')
+    print(f"\n==========================================")
+    print(f"Cluster: {disp} ({name})")
+    print(f"==========================================")
+    for c in item.get('status', {}).get('conditions', []):
+        if c.get('type') in ['Ready', 'Connected']:
+            print(f"  [{c.get('type')}]: status={c.get('status')} reason={c.get('reason')} msg={c.get('message')}")
+

scripts/check_custom_coredns.py

@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n kube-system get configmap coredns-custom")
+print(stdout.read().decode('utf-8'))
+print(stderr.read().decode('utf-8'))
+client.close()
+

scripts/check_diff_kernel.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("cmp /boot/vmlinuz-6.12.109+rpt-rpi-2712 /boot/firmware/kernel_2712.img && echo IDENTICAL || echo DIFFERENT")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_dmesg_head.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("dmesg | head -n 10")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_dpkg_kernel.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("dpkg -l | grep -E 'linux-|raspberrypi-'")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_firmware_version.py

@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("zcat /boot/firmware/kernel_2712.img 2>/dev/null | strings | grep 'Linux version' | head -n 5")
+print("kernel_2712.img version:")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_fstab_ctl03.py

@@ -0,0 +1,24 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+
+cmds = [
+    "lsblk -f",
+    "cat /etc/fstab",
+    "mount | grep -i boot"
+]
+
+for cmd in cmds:
+    print(f"\n>>> Running on 140.44.4.43: {cmd}")
+    stdin, stdout, stderr = client.exec_command(cmd)
+    print(stdout.read().decode('utf-8'))
+
+client.close()
+

scripts/check_gateway.py

@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = c.exec_command("curl -k -s -I http://140.44.4.1/ || curl -k -s -I https://140.44.4.1/")
+print("Gateway 140.44.4.1 HTTP headers:")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = c.exec_command("arp -an | grep 140.44.4.1")
+print("ARP for 140.44.4.1:")
+print(stdout.read().decode('utf-8'))
+
+c.close()
+

scripts/check_gateway_type.py

@@ -0,0 +1,16 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = c.exec_command("curl -k -s -L https://140.44.4.1/ | grep -iE 'title|unifi|opnsense|pfsense|mikrotik|netgate|firewall|router'")
+print("Gateway 140.44.4.1 HTML match:")
+print(stdout.read().decode('utf-8'))
+
+c.close()
+

scripts/check_getent.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("getent hosts aerodrome.snarfnet.net")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_ghcr_tags.py

@@ -0,0 +1,24 @@
+import urllib.request
+import json
+
+def get_ghcr_tags(package):
+    # Public unauthenticated ghcr tags endpoint
+    url = f"https://ghcr.io/v2/{package}/tags/list"
+    try:
+        # Get anonymous token
+        token_url = f"https://ghcr.io/token?service=ghcr.io&scope=repository:{package}:pull"
+        with urllib.request.urlopen(token_url, timeout=5) as t_resp:
+            token = json.loads(t_resp.read().decode())['token']
+        
+        req = urllib.request.Request(url, headers={'Authorization': f'Bearer {token}'})
+        with urllib.request.urlopen(req, timeout=5) as resp:
+            data = json.loads(resp.read().decode())
+            tags = [t for t in data.get('tags', []) if not t.endswith('-dev') and not 'beta' in t]
+            return tags[-10:]
+    except Exception as e:
+        return [f"Err: {e}"]
+
+print("Homepage tags:", get_ghcr_tags("gethomepage/homepage"))
+print("Coder tags:", get_ghcr_tags("coder/coder"))
+print("Authentik tags:", get_ghcr_tags("goauthentik/server"))
+

scripts/check_grafana_spec.py

@@ -0,0 +1,15 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    _, out, _ = ssh.exec_command(cmd)
+    print(out.read().decode())
+
+run("kubectl get deployment grafana -n default -o jsonpath='{.spec.template.spec.containers[*].name}'")
+run("kubectl get deployment grafana -n default -o jsonpath='{.spec.template.spec.containers[*].image}'")
+ssh.close()
+

scripts/check_harvester_ip.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("ping -c 2 harvester.spacedock.snarfnet.net; getent hosts harvester.spacedock.snarfnet.net")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_helm_cattle.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm list -n cattle-system")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_helm_releases.py

@@ -0,0 +1,14 @@
+import paramiko
+
+def check_helm(ip, name):
+    print(f"\n=== {name} HELM RELEASES ({ip}) ===")
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    _, out, _ = ssh.exec_command("helm list -A")
+    print(out.read().decode())
+    ssh.close()
+
+check_helm('140.44.4.11', 'AERODROME')
+check_helm('140.44.4.41', 'SPACEPORT')
+

scripts/check_historical_data.py

@@ -0,0 +1,60 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+# 1. Inspect Gotify database on Swarm
+print("=================== GOTIFY (Swarm 140.44.4.71) ===================")
+ssh_swm = paramiko.SSHClient()
+ssh_swm.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh_swm.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run_swm(cmd):
+    _, out, err = ssh_swm.exec_command(cmd)
+    res = out.read().decode('utf-8', errors='replace').strip()
+    if res:
+        print(res)
+
+run_swm("cat c:\\Users\\Snarf\\OneDrive\\Desktop\\code\\SnarfCode\\docker-deployments\\gotify.yml 2>/dev/null || true")
+run_swm("sudo docker service inspect gotify_gotify --format '{{json .Spec.TaskTemplate.ContainerSpec.Mounts}}'")
+run_swm("sudo docker run --rm -v gotify_gotify_data:/data alpine ls -la /data 2>/dev/null || find /var/lib/docker/volumes/ -name 'gotify.db' 2>/dev/null")
+
+# Query gotify.db if found
+run_swm("""python3 -c '
+import sqlite3, glob
+db_files = glob.glob("/var/lib/docker/volumes/**/gotify.db", recursive=True)
+print("Gotify DB files:", db_files)
+for db in db_files:
+    try:
+        conn = sqlite3.connect(db)
+        cur = conn.cursor()
+        print("Tables in", db)
+        for row in cur.execute("SELECT name FROM sqlite_master WHERE type=\"table\""):
+            print(" ", row[0])
+        print("Applications count:", cur.execute("SELECT count(*) FROM applications").fetchone()[0])
+        print("Clients count:", cur.execute("SELECT count(*) FROM clients").fetchone()[0])
+        print("Messages count:", cur.execute("SELECT count(*) FROM messages").fetchone()[0])
+    except Exception as e:
+        print("Err reading", db, e)
+'""")
+
+ssh_swm.close()
+
+# 2. Inspect Semaphore BoltDB on Spaceport
+print("\n=================== SEMAPHORE (Spaceport 140.44.4.41) ===================")
+ssh_sp = paramiko.SSHClient()
+ssh_sp.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh_sp.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run_sp(cmd):
+    _, out, err = ssh_sp.exec_command(cmd)
+    res = out.read().decode('utf-8', errors='replace').strip()
+    if res:
+        print(res)
+
+run_sp("kubectl exec -n infrastructure deployment/semaphore -- ls -lh /var/lib/semaphore/database.boltdb")
+# Let's inspect projects and templates using semaphore CLI or strings
+run_sp("kubectl exec -n infrastructure deployment/semaphore -- strings /var/lib/semaphore/database.boltdb | grep -E 'project|playbook|inventory|repository' | head -n 30")
+
+ssh_sp.close()
+

scripts/check_homepage_spec.py

@@ -0,0 +1,15 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    _, out, _ = ssh.exec_command(cmd)
+    print(out.read().decode())
+
+run("kubectl get deployment homepage-2-1744068158 -n infrastructure -o jsonpath='{.spec.template.spec.containers[*].name}'")
+run("kubectl get deployment homepage-2-1744068158 -n infrastructure -o jsonpath='{.spec.template.spec.containers[*].image}'")
+ssh.close()
+

scripts/check_housekeeping.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system get pods --field-selector=status.phase=Succeeded")
+print("Completed pods:", stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_idrac_ips.py

@@ -0,0 +1,18 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("""
+for h in spacedock1.snarfnet.net spacedock2.snarfnet.net spacedock3.snarfnet.net spacedock4.snarfnet.net; do
+    getent hosts $h
+done
+""")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_jenkins.py

@@ -0,0 +1,23 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    _, out, err = ssh.exec_command(cmd)
+    res = out.read().decode().strip()
+    if res:
+        print(res)
+    e = err.read().decode().strip()
+    if e:
+        print("ERR:", e)
+
+run("kubectl get pod -n infrastructure jenkins-0 -o wide")
+run("kubectl describe pod -n infrastructure jenkins-0 | tail -n 25")
+run("curl -k -s -I https://jenkins.snarfnet.net | head -n 5")
+run("curl -s -I http://140.44.4.52:101 | head -n 5")
+
+ssh.close()
+

scripts/check_job134.py

@@ -0,0 +1,28 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=134)
+print(f"Job [134]: Status={job.status}, Elapsed={job.elapsed}s, Failed={job.failed}")
+events = job.job_events.filter(event__in=['runner_on_ok', 'runner_on_failed', 'playbook_on_stats']).order_by('-id')[:10]
+for e in events:
+    print(f"[{e.event}] Host: {e.event_data.get('host')} - Task: {e.event_data.get('task')}")
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+
+_, out, _ = ssh.exec_command('kubectl get nodes')
+print("=== KUBERNETES NODES ===")
+print(out.read().decode())
+
+ssh.close()
+

scripts/check_job134_end.py

@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=134)
+print(job.result_stdout[-1500:])
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+

scripts/check_job134_explain.py

@@ -0,0 +1,22 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=134)
+print(f"Status: {job.status}")
+print(f"Job explanation: {job.job_explanation}")
+print(f"Result Traceback: {job.result_traceback}")
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+

scripts/check_job134_fail.py

@@ -0,0 +1,25 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=134)
+failed_events = job.job_events.filter(failed=True)
+for e in failed_events:
+    res = e.event_data.get('res', {})
+    msg = res.get('msg') or res.get('stderr') or res.get('stdout') or str(res)
+    print(f"Host: {e.event_data.get('host')}, Task: {e.event_data.get('task')}")
+    print(f"Details: {msg[:300]}")
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+

scripts/check_job134_stdout.py

@@ -0,0 +1,21 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=134)
+lines = [l for l in job.result_stdout.split('\\n') if l.strip()]
+print('\\n'.join(lines[-35:]))
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+

scripts/check_job_status.py

@@ -0,0 +1,21 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=131)
+print(f"Job [131]: Status={job.status}, Elapsed={job.elapsed}s, Failed={job.failed}")
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+
+ssh.close()
+

scripts/check_k3s_channels.py

@@ -0,0 +1,12 @@
+import urllib.request
+import json
+
+url = "https://update.k3s.io/v1-release/channels"
+try:
+    req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0', 'Accept': 'application/json'})
+    with urllib.request.urlopen(req, timeout=10) as resp:
+        data = json.loads(resp.read().decode())
+        for item in data.get('data', []):
+            print(f"{item.get('id')}: {item.get('latest')}")
+except Exception as e:
+    print("Error:", e)

scripts/check_kernel_file.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("file /boot/firmware/kernel_2712.img; ls -la /boot/vmlinuz*")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_kernel_str.py

@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("strings /boot/firmware/kernel_2712.img | grep -E '^Linux version' | head -n 5")
+print("Kernel version inside kernel_2712.img:")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_logs_utf8.py

@@ -0,0 +1,32 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh_sp = paramiko.SSHClient()
+ssh_sp.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh_sp.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+ssh_swm = paramiko.SSHClient()
+ssh_swm.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh_swm.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(ssh, cmd):
+    print(f"\n--- {cmd} ---")
+    _, out, err = ssh.exec_command(cmd)
+    res = out.read().decode('utf-8', errors='replace').strip()
+    if res:
+        print(res)
+
+print("=== PEANUT LOGS (Swarm) ===")
+run(ssh_swm, "sudo docker service logs peanut_peanut --tail=50")
+
+print("\n=== CODER LOGS (Spaceport) ===")
+run(ssh_sp, "kubectl logs -n infrastructure -l app.kubernetes.io/name=coder --tail=50")
+
+print("\n=== SEMAPHORE LOGS (Spaceport) ===")
+run(ssh_sp, "kubectl logs -n infrastructure -l app.kubernetes.io/name=semaphore --tail=50")
+
+ssh_sp.close()
+ssh_swm.close()
+

scripts/check_longhorn_version.py

@@ -0,0 +1,15 @@
+import paramiko
+
+def check_lh(ip, cluster):
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    print(f"\n[{cluster}] Checking Longhorn images...")
+    _, out, _ = ssh.exec_command("kubectl get daemonset -n longhorn-system longhorn-manager -o jsonpath='{.spec.template.spec.containers[0].image}{\"\\n\"}'")
+    print("Manager image:", out.read().decode().strip())
+    _, out, _ = ssh.exec_command("kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,ROBUSTNESS:.status.robustness,STATE:.status.state | head -n 10")
+    print(out.read().decode().strip())
+    ssh.close()
+
+check_lh('140.44.4.11', 'AERODROME')
+check_lh('140.44.4.41', 'SPACEPORT')

scripts/check_npm_all_hosts.py

@@ -0,0 +1,11 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+stdin, stdout, stderr = ssh.exec_command("""sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') grep -E "server_name|set \\$server|set \\$port" /data/nginx/proxy_host/*.conf""")
+print(stdout.read().decode())
+
+ssh.close()
+

scripts/check_npm_argo.py

@@ -0,0 +1,21 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+run("sudo docker ps | grep nginx")
+run("sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') find /data -maxdepth 3")
+run("sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') cat /etc/hosts")
+run("""sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') grep -rn "argo" /etc/nginx/ /data/nginx/""")
+run("""sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') grep -rn "pihole" /etc/nginx/ /data/nginx/""")
+
+ssh.close()

scripts/check_npm_certs.py

@@ -0,0 +1,23 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = client.exec_command(cmd)
+    out = stdout.read().decode('utf-8')
+    err = stderr.read().decode('utf-8')
+    if out: print(out)
+    if err: print(f"ERR: {err}")
+
+run("docker ps")
+run("find / -name '*.crt' -o -name '*.pem' 2>/dev/null | grep -E 'live|letsencrypt|snarfnet' | head -n 30")
+
+client.close()
+

scripts/check_npm_db.py

@@ -0,0 +1,15 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+cmd = "sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') sh -c \"python3 -c \\\"import sqlite3; con = sqlite3.connect('/data/database.sqlite'); cur = con.cursor(); print('\\n'.join(str(r) for r in cur.execute('SELECT id, domain_names, forward_host, forward_port, forward_scheme, enabled FROM proxy_host WHERE is_deleted=0')))\\\"\""
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+err = stderr.read().decode()
+if err:
+    print("ERR:", err)
+
+ssh.close()
+

scripts/check_npm_harvester.py

@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("docker exec $(docker ps -q -f name=npm) grep -rnw '/etc/nginx/conf.d' -e 'harvester' 2>/dev/null; docker exec $(docker ps -q -f name=npm) grep -rn '/data/nginx/proxy_host' -e 'harvester' 2>/dev/null")
+print("NPM config:")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_npm_hosts.py

@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command('docker exec $(docker ps -q -f name=npm) sqlite3 /data/database.sqlite "SELECT domain_names, forward_host, forward_port FROM proxy_host;"')
+print("NPM hosts:")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_npm_live_certs.py

@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+
+cmd = """
+for cert in /data/glusterfs/swarm/brick/legacy_volumes/nginx/letsencrypt/live/npm-*/cert.pem; do
+  echo "=== $cert ==="
+  openssl x509 -in "$cert" -noout -subject -dates -ext subjectAltName 2>/dev/null
+done
+"""
+stdin, stdout, stderr = client.exec_command(cmd)
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_npm_mounts.py

@@ -0,0 +1,15 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+
+run("sudo docker service inspect nginx-proxy-manager_app --format '{{json .Spec.TaskTemplate.ContainerSpec.Mounts}}'")
+run("sudo docker inspect $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') --format '{{json .Mounts}}'")
+ssh.close()
+

scripts/check_npm_nodes.py

@@ -0,0 +1,14 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+
+run("sudo docker service ps nginx-proxy-manager_app --filter 'desired-state=running'")
+ssh.close()
+

scripts/check_npm_proxy_conf.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command('docker exec $(docker ps -q -f name=nginx-proxy-manager_app) grep -r "proxy_pass" /data/nginx/proxy_host')
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_npm_real.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command('docker exec $(docker ps -q -f name=nginx-proxy-manager_app) sqlite3 /data/database.sqlite "SELECT domain_names, forward_host, forward_port FROM proxy_host;"')
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_npm_user.py

@@ -0,0 +1,19 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+# Check user table in NPM db
+run("""sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') sh -c "python3 -c \\\"import sqlite3; con = sqlite3.connect('/data/database.sqlite'); cur = con.cursor(); print(list(cur.execute('SELECT id, name, email FROM user')))\\\"\"""")
+
+ssh.close()
+

scripts/check_old_coder_layer.py

@@ -0,0 +1,26 @@
+import paramiko
+
+nodes = [
+    ("ctl01", "140.44.4.41"),
+    ("wrk01", "140.44.4.51"),
+    ("wrk02", "140.44.4.52"),
+    ("wrk03", "140.44.4.53"),
+    ("wrk04", "140.44.4.54"),
+    ("wrk05", "140.44.4.55"),
+    ("wrk06", "140.44.4.56"),
+]
+
+for name, ip in nodes:
+    try:
+        ssh = paramiko.SSHClient()
+        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+        ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=5)
+        _, out, _ = ssh.exec_command("find /run/k3s/containerd/ /var/lib/rancher/k3s/agent/containerd/ -type d -name 'coderv2' 2>/dev/null")
+        res = out.read().decode().strip()
+        if res:
+            print(f"[{name} - {ip}] FOUND coderv2:\n{res}")
+        else:
+            print(f"[{name} - {ip}] none")
+        ssh.close()
+    except Exception as e:
+        print(f"[{name} - {ip}] err: {e}")

scripts/check_other_apps.py

@@ -0,0 +1,22 @@
+import paramiko
+
+def query_cluster(ip, name):
+    print(f"\n=================== CLUSTER {name} ({ip}) ===================")
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    
+    def run(cmd):
+        print(f"=== {cmd} ===")
+        _, out, err = ssh.exec_command(cmd)
+        res = out.read().decode().strip()
+        if res:
+            print(res)
+    
+    run("kubectl get ns")
+    run("kubectl get deploy,sts -A | grep -E 'awx|authentik|zabbix|coder'")
+    ssh.close()
+
+query_cluster('140.44.4.11', 'AERODROME')
+query_cluster('140.44.4.41', 'SPACEPORT')
+

scripts/check_peanut_now.py

@@ -0,0 +1,22 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    _, out, err = ssh.exec_command(cmd)
+    res = out.read().decode('utf-8', errors='replace').strip()
+    if res:
+        print(res)
+
+run("sudo docker service logs peanut_peanut --tail=25")
+run("curl -s -I http://140.44.4.70:8050/")
+run("curl -s http://140.44.4.70:8050/ | grep -i setup || echo 'NO SETUP REDIRECT'")
+
+ssh.close()
+

scripts/check_peanut_storage.py

@@ -0,0 +1,19 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"\n--- {cmd} ---")
+    _, out, err = ssh.exec_command(cmd)
+    res = out.read().decode('utf-8', errors='replace').strip()
+    if res:
+        print(res)
+
+run("ls -la /mnt/swarm_shared/legacy_volumes/peanut/")
+run("ssh snshpd4lnxswm05 'ls -la /mnt/swarm_shared/legacy_volumes/peanut/'")
+run("sudo docker service inspect peanut_peanut")
+
+ssh.close()
+

scripts/check_pending_reasons.py

@@ -0,0 +1,25 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl -n infra get events --field-selector reason=FailedScheduling")
+print("Aerodrome Scheduling Events:")
+print(stdout.read().decode('utf-8'))
+c.close()
+
+c2 = paramiko.SSHClient()
+c2.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c2.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = c2.exec_command("k3s kubectl -n infrastructure get events --field-selector reason=FailedScheduling")
+print("Spaceport Scheduling Events:")
+print(stdout.read().decode('utf-8'))
+c2.close()
+

scripts/check_pkg_files.py

@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("dpkg -L linux-image-6.12.109+rpt-rpi-2712 | grep -E '/boot|/lib/modules'")
+print(stdout.read().decode('utf-8'))
+client.close()
+

scripts/check_pod_events.py

@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system describe pod rancher-574bb974f4-877rf | grep -A 10 -B 2 Events:")
+print("=== POD EVENTS ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system describe pod rancher-574bb974f4-877rf | grep -A 15 'Last State:'")
+print("=== LAST STATE ===")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+