fix: remove obsolete file

feat: add cluster management and audit utility scripts
docs: add network blueprint for high-availability DNS, honeypot, and UDM firewall hardening
2026-11-05 21:40:37 -05:00 · 2026-11-05 21:40:22 -05:00 · 2026-11-05 11:44:43 -05:00 · 2026-11-05 11:20:57 -05:00 · 2026-11-05 11:17:12 -05:00 · 2026-11-05 11:10:07 -05:00
335 changed files with 9955 additions and 10 deletions
--- a/docs/DNS_Setup_Plan.md
+++ b/docs/DNS_Setup_Plan.md
@@ -0,0 +1,182 @@
+# Project Blueprint: High-Availability DNS Stack, Network Honeypot & UDM Pro Network Security
+
+## 1. Executive Summary & Live UDM Pro Topology Assessment
+
+Based on the live UDM Pro controller diagnostics, we have mapped the exact network topography, identified critical security exposures, and tailored the implementation plan.
+
+---
+
+### 1.1 Live Network & VLAN Topography
+
+| VLAN ID | Network Name | Subnet / CIDR | Gateway IP | Active Leases | Security Classification |
+|:---:|---|---|---|:---:|---|
+| **1** | **LAN** | `14.44.4.0/25` | `14.44.4.1` | 14 / 117 | Trusted Internal Clients |
+| **2** | **Management** | `140.44.44.0/24` | `140.44.44.1` | 9 / 51 | Out-of-band (iDRAC, Switches, APs) |
+| **3** | **Spacedock** | `140.44.1.0/24` | `140.44.1.1` | 0 / 249 | Harvester VM / Hypervisor Tenant |
+| **20** | **Compute** | `140.44.4.0/25` | `140.44.4.1` | 1 / 15 (Static Nodes) | Mission-Critical K3s Clusters, Swarm |
+| **30** | **Media** | `192.168.1.0/25` | `192.168.1.1` | 1 / 91 | Streaming & Entertainment Devices |
+| **40** | **Storage** | `14.10.10.0/25` | `14.10.10.1` | 2 / 125 | Synology NAS (`.71`, `.72`) NFS / iSCSI |
+| **50** | **Smart Home** | `14.4.4.0/25` | `14.4.4.1` | 53 / 85 | IoT Devices, Gaming Host (`14.4.4.15`) |
+| **60** | **HA** | `140.4.4.0/25` | `140.4.4.1` | 0 / 7 | High-Availability Interconnects |
+| **VPN** | **WireGuard / Teleport** | `192.168.3.0/24` / `.2.0/24` | UDM Pro | Remote | Ingress Remote Access |
+
+---
+
+### 1.2 Critical Security Findings from UDM Pro Diagnostics
+
+1. **Default Security Posture is `Allow All`:**
+   * Currently, the UDM Pro routes traffic freely between all subnets without restriction.
+   * **Risk:** The 53 IoT devices in Smart Home (`14.4.4.0/25`) and Media devices can directly initiate connections to Kubernetes nodes, etcd ports, SSH, and Synology NAS shares in `Compute` (`140.44.4.0/25`) and `Storage` (`14.10.10.0/25`).
+2. **WAN Port Forwarding to Smart Home Host (`14.4.4.15`):**
+   * Port forwards for game servers (`FS25` ports 8443/10823, `Stationeers` ports 6443/27015) point directly from WAN (`174.106.252.96`) to host `14.4.4.15` on the Smart Home VLAN.
+   * If an exploit compromises this host, an attacker currently has unhindered lateral access to the entire homelab infrastructure.
+3. **DNS Leaks & Lack of Local Filtering:**
+   * UDM Pro WAN is set to `Auto DNS Server` (Spectrum ISP DNS `0.0.0.0`).
+   * No DNS ad/malware filtering or split-horizon DNS caching is currently active across the subnets.
+
+---
+
+## 2. High-Availability DNS Stack (Pi-hole + Technitium)
+
+```mermaid
+flowchart TD
+    subgraph Clients["LAN Clients across all VLANs (VLAN 1, 20, 30, 50, VPN)"]
+        Devices["IoT Devices (53), Laptops, Servers, Hypervisors"]
+    end
+
+    subgraph UDM["UniFi Dream Machine Pro (140.44.4.1)"]
+        DHCP["DHCP Option 6 per VLAN:<br>Primary DNS: 140.44.4.35 (Aerodrome VIP)<br>Secondary DNS: 140.44.4.65 (Spaceport VIP)"]
+        NAT["DNS Redirection Rule:<br>Capture port 53 to hardcoded DNS (8.8.8.8) -> Redirect to VIPs"]
+    end
+
+    subgraph AeroCluster["Aerodrome K3s Cluster (VLAN 20)"]
+        AeroVIP["MetalLB VIP: 140.44.4.35:53 (UDP/TCP)"]
+        AeroPihole["Pi-hole Instance 1<br>(Ad-blocking, Caching, FTL rate-limit: 0)"]
+        AeroTech["Technitium Instance 1 (Primary Authoritative)<br>(snarfnet.net Zone, Reverse PTR, DNSSEC)"]
+        AeroStorage["Longhorn Storage Volume"]
+    end
+
+    subgraph SpaceCluster["Spaceport K3s Cluster (VLAN 20)"]
+        SpaceVIP["MetalLB VIP: 140.44.4.65:53 (UDP/TCP)"]
+        SpacePihole["Pi-hole Instance 2<br>(Ad-blocking, Caching, FTL rate-limit: 0)"]
+        SpaceTech["Technitium Instance 2 (Secondary Authoritative)<br>(Zone Transfer AXFR/IXFR from Primary)"]
+        SpaceStorage["Longhorn Storage Volume"]
+    end
+
+    subgraph Upstream["Encrypted Upstream Resolvers"]
+        DoH["DNS-over-HTTPS (Quad9 / Cloudflare / Mullvad)"]
+    end
+
+    Devices -->|DNS Queries (UDP/TCP 53)| DHCP
+    DHCP -->|Primary| AeroVIP
+    DHCP -.->|Secondary / Failover| SpaceVIP
+    AeroVIP --> AeroPihole
+    SpaceVIP --> SpacePihole
+    AeroPihole -->|Recursive / Local Lookups| AeroTech
+    SpacePihole -->|Recursive / Local Lookups| SpaceTech
+    AeroTech <==>|Automatic Zone Sync (AXFR)| SpaceTech
+    AeroTech -->|Encrypted DoH| DoH
+    SpaceTech -->|Encrypted DoH| DoH
+    AeroPihole --- AeroStorage
+    AeroTech --- AeroStorage
+    SpacePihole --- SpaceStorage
+    SpaceTech --- SpaceStorage
+```
+
+### 2.1 Addressing the Rate-Limiting & Sluggishness Issue
+1. **FTL Rate Limit Bypass:** Set `FTLCONF_RATE_LIMIT=0` so local bursts (such as 53 Smart Home devices polling simultaneously) are never throttled.
+2. **Technitium Engine:** Replaces single-threaded Unbound with a high-performance .NET 9 asynchronous resolver capable of servicing tens of thousands of queries per second with sub-millisecond RAM caching.
+3. **High Availability:** MetalLB VIPs on Aerodrome (`140.44.4.35`) and Spaceport (`140.44.4.65`) allow continuous service even during node reboots or maintenance.
+4. **Authoritative Split-Horizon:** Technitium will authoritatively host the internal `snarfnet.net` forward zone and reverse PTR zones (`4.44.140.in-addr.arpa`, `4.4.14.in-addr.arpa`, `10.10.14.in-addr.arpa`), eliminating slow mDNS or external lookup delays.
+
+---
+
+## 3. Network Honeypot & UDM Pro Firewall Hardening
+
+```mermaid
+flowchart LR
+    subgraph WAN_Threats["External & Lateral Threats"]
+        Internet["Public Internet"]
+        CompromisedIoT["Compromised IoT / Host (e.g. 14.4.4.15)"]
+    end
+
+    subgraph UDM_FW["UDM Pro Firewall Rules (LAN IN)"]
+        Rule1["Rule 1: Allow Established & Related"]
+        Rule2["Rule 2: Allow DNS (Port 53 to .35 & .65)"]
+        Rule3["Rule 3: Allow NTP (Port 123 to .11 & .44)"]
+        Rule4["Rule 4: Drop IoT -> Compute (VLAN 50 -> 20)"]
+        Rule5["Rule 5: Drop IoT -> Storage (VLAN 50 -> 40)"]
+        Rule6["Rule 6: Drop Media -> Compute (VLAN 30 -> 20)"]
+    end
+
+    subgraph HoneypotNet["Honeypot Decoy (VLAN 50 / Decoy IP 14.4.4.99)"]
+        OpenCanary["OpenCanary Honeypot<br>(Simulating SSH:22, HTTP:80, SMB:445, RDP:3389)"]
+    end
+
+    subgraph AlertSystem["Alerting & Audit Pipeline"]
+        Gotify["Gotify Push Notification<br>(Instant alert to your phone)"]
+        Zabbix["Zabbix Security Audit Log"]
+    end
+
+    CompromisedIoT -.->|Probes decoy IP| OpenCanary
+    OpenCanary -->|High Priority Webhook| Gotify
+    OpenCanary -->|Syslog Event| Zabbix
+    CompromisedIoT -->|Blocked by Rule 4| UDM_FW
+```
+
+### 3.1 Network Honeypot Deployment
+* **Tool:** **OpenCanary** (lightweight, zero false-positives).
+* **Placement:**
+  * Assigned a decoy IP on the Smart Home subnet (e.g., `14.4.4.99`) where the gaming port forwards exist.
+  * Emulates vulnerable services: SSH (`22`), Telnet (`23`), Web admin (`80`/`8080`), SMB (`445`), and RDP (`3389`).
+* **Alerting:**
+  * When any internal device attempts a port scan or lateral connection to `14.4.4.99`, OpenCanary immediately sends a critical push notification to **Gotify** (`gotify.snarfnet.net`) and generates an alert in **Zabbix**.
+
+---
+
+### 3.2 UDM Pro Firewall Rules Matrix (LAN IN)
+
+We will define firewall groups on the UDM Pro:
+* **RFC1918_Subnets:** All internal private ranges (`140.44.0.0/16`, `14.0.0.0/8`, `192.168.0.0/16`).
+* **K8s_DNS_VIPs:** `140.44.4.35`, `140.44.4.65`.
+* **NTP_Servers:** `140.44.4.11`, `14.4.4.44`.
+
+| Order | Rule Name | Type | Source | Destination | Action | Purpose |
+|:---:|---|:---:|---|---|:---:|---|
+| **2001** | `Allow Established/Related` | LAN IN | Any | Any (State: Established, Related) | **ACCEPT** | Essential return traffic |
+| **2002** | `Allow Cluster DNS` | LAN IN | All VLANs | `K8s_DNS_VIPs` (Port: 53 UDP/TCP) | **ACCEPT** | Clients resolve DNS |
+| **2003** | `Allow Cluster NTP` | LAN IN | All VLANs | `NTP_Servers` (Port: 123 UDP) | **ACCEPT** | Chrony time sync |
+| **2004** | `Allow Trusted to All` | LAN IN | `VLAN 1 (LAN)`, `VLAN 20 (Compute)` | RFC1918_Subnets | **ACCEPT** | Admins & servers access all VLANs |
+| **2005** | `Isolate SmartHome to Compute` | LAN IN | `VLAN 50 (Smart Home)` | `VLAN 20 (Compute)` | **DROP** | Block IoT pivoting into K8s |
+| **2006** | `Isolate SmartHome to Storage` | LAN IN | `VLAN 50 (Smart Home)` | `VLAN 40 (Storage)` | **DROP** | Protect Synology NAS from IoT |
+| **2007** | `Isolate Media to Compute` | LAN IN | `VLAN 30 (Media)` | `VLAN 20 (Compute)` | **DROP** | Block media devices from cluster |
+| **2008** | `Isolate Guest Network` | GUEST IN | Guest Network | RFC1918_Subnets | **DROP** | Restrict guest traffic to Internet only |
+
+---
+
+## 4. Phased Implementation Roadmap
+
+### Phase 1: Teardown Lingering Stubs
+* Cleanly delete orphaned, broken deployments and services in `infra` (Aerodrome) and `infrastructure` (Spaceport):
+  * Remove `deployment/pihole`, `deployment/technitium-dns`, and non-functional NodePort services.
+
+### Phase 2: Deploy Dual-Cluster Pi-hole + Technitium
+* Create Longhorn storage volumes on both clusters for persistent configs and blocklists.
+* Deploy **Aerodrome Primary Stack**:
+  * Technitium on internal port `5353`.
+  * Pi-hole with MetalLB LoadBalancer VIP **`140.44.4.35:53`** (UDP & TCP).
+* Deploy **Spaceport Secondary Stack**:
+  * Technitium on internal port `5353` configured for AXFR zone transfers from Aerodrome.
+  * Pi-hole with MetalLB LoadBalancer VIP **`140.44.4.65:53`** (UDP & TCP).
+* Populate `snarfnet.net` local records and reverse PTR zones.
+
+### Phase 3: Network Honeypot Deployment
+* Deploy containerized OpenCanary with virtual listener on `14.4.4.99`.
+* Configure webhook bridge to Gotify server (`140.44.4.70`) for immediate push notification upon any probe.
+
+### Phase 4: UDM Pro Network Configuration & Policy Hardening
+* Configure DHCP Scope DNS servers on UDM Pro for all VLANs:
+  * Primary: `140.44.4.35`
+  * Secondary: `140.44.4.65`
+* Set UDM Pro WAN DNS to `140.44.4.35` & `140.44.4.65` (or Quad9 fallback).
+* Apply LAN IN firewall rule matrix on UDM Pro to enforce strict VLAN isolation.
--- a/2
+++ b/2
--- a/scripts/apply_36.py
+++ b/scripts/apply_36.py
@@ -0,0 +1,65 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+cmd = """
+k3s kubectl -n dns delete svc pihole-dns-vip pihole-web-vip --wait=false
+sleep 2
+cat << 'EOF' | k3s kubectl apply -f -
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-dns-vip
+  namespace: dns
+  annotations:
+    metallb.universe.tf/loadBalancerIPs: 140.44.4.36
+    metallb.universe.tf/allow-shared-ip: "aerodrome-dns-key"
+  labels:
+    app: pihole
+spec:
+  type: LoadBalancer
+  selector:
+    app: pihole
+  ports:
+  - name: dns-udp
+    port: 53
+    targetPort: 53
+    protocol: UDP
+  - name: dns-tcp
+    port: 53
+    targetPort: 53
+    protocol: TCP
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: pihole-web-vip
+  namespace: dns
+  annotations:
+    metallb.universe.tf/loadBalancerIPs: 140.44.4.36
+    metallb.universe.tf/allow-shared-ip: "aerodrome-dns-key"
+  labels:
+    app: pihole
+spec:
+  type: LoadBalancer
+  selector:
+    app: pihole
+  ports:
+  - name: web-http
+    port: 80
+    targetPort: 80
+    protocol: TCP
+EOF
+sleep 3
+k3s kubectl -n dns get svc
+"""
+stdin, stdout, stderr = c.exec_command(cmd)
+print(stdout.read().decode('utf-8'))
+c.close()
+
--- a/scripts/apply_coredns_spaceport.py
+++ b/scripts/apply_coredns_spaceport.py
@@ -0,0 +1,48 @@
+import paramiko
+import sys
+import time
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+
+cm_yaml = """cat << 'EOF' | k3s kubectl apply -f -
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: coredns-custom
+  namespace: kube-system
+data:
+  snarfnet.override: |
+    hosts {
+      140.44.4.34 aerodrome.snarfnet.net
+      140.44.4.90 harvester.spacedock.snarfnet.net
+      fallthrough
+    }
+EOF
+"""
+
+print("Applying coredns-custom on Spaceport...")
+stdin, stdout, stderr = client.exec_command(cm_yaml)
+print(stdout.read().decode('utf-8'))
+print(stderr.read().decode('utf-8'))
+
+print("\nRestarting coredns deployment on Spaceport...")
+client.exec_command("k3s kubectl -n kube-system rollout restart deployment/coredns")
+client.exec_command("k3s kubectl -n kube-system rollout status deployment/coredns --timeout=60s")
+
+print("\nRestarting cattle-cluster-agent on Spaceport...")
+client.exec_command("k3s kubectl -n cattle-system rollout restart deployment/cattle-cluster-agent")
+client.exec_command("k3s kubectl -n cattle-system rollout status deployment/cattle-cluster-agent --timeout=60s")
+
+time.sleep(5)
+print("\nChecking cattle-cluster-agent logs:")
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system logs -l app=cattle-cluster-agent --tail=25")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+
--- a/scripts/apply_dns_updates.py
+++ b/scripts/apply_dns_updates.py
@@ -0,0 +1,38 @@
+import paramiko
+import sys
+import time
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+def apply_to_cluster(ip, name, manifest_path):
+    print(f"Connecting to {name} ({ip})...")
+    with open(manifest_path, "r", encoding="utf-8") as f:
+        content = f.read()
+
+    c = paramiko.SSHClient()
+    c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    c.connect(ip, username="root", key_filename=key_path, timeout=10)
+    
+    stdin, stdout, stderr = c.exec_command("k3s kubectl apply -f -")
+    stdin.write(content)
+    stdin.channel.shutdown_write()
+    
+    out = stdout.read().decode('utf-8')
+    err = stderr.read().decode('utf-8')
+    print(f"[{name}] Apply Output:\n{out}")
+    if err.strip():
+        print(f"[{name}] Apply Error:\n{err}")
+    c.close()
+
+aero_path = r"c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\aerodrome\dns\dns-stack.yaml"
+space_path = r"c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\spaceport\dns\dns-stack.yaml"
+
+apply_to_cluster("140.44.4.11", "Aerodrome", aero_path)
+time.sleep(2)
+apply_to_cluster("140.44.4.41", "Spaceport", space_path)
+print("Finished applying manifests to both clusters!")
+
+
+
--- a/scripts/apply_web_vips.py
+++ b/scripts/apply_web_vips.py
@@ -0,0 +1,22 @@
+import paramiko
+import sys
+import base64
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+def apply_manifest(host, file_path):
+    with open(file_path, "rb") as f:
+        b64 = base64.b64encode(f.read()).decode('ascii')
+    c = paramiko.SSHClient()
+    c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    c.connect(host, username="root", key_filename=key_path, timeout=5)
+    stdin, stdout, stderr = c.exec_command(f"echo '{b64}' | base64 -d | k3s kubectl apply -f -")
+    print(f"Apply on {host}:")
+    print(stdout.read().decode('utf-8'))
+    c.close()
+
+apply_manifest("140.44.4.11", r"C:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\aerodrome\dns\dns-stack.yaml")
+apply_manifest("140.44.4.41", r"C:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\spaceport\dns\dns-stack.yaml")
+
--- a/scripts/audit_authentik.py
+++ b/scripts/audit_authentik.py
@@ -0,0 +1,30 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+def run_query(title, q):
+    cmd = f"""k3s kubectl -n infra exec authentik-postgresql-0 -- bash -c "PGPASSWORD=\\$(cat /opt/bitnami/postgresql/secrets/password) psql -U authentik -d authentik -c \\"{q}\\"" """
+    stdin, stdout, stderr = c.exec_command(cmd)
+    print(f"=== {title} ===")
+    out = stdout.read().decode('utf-8')
+    err = stderr.read().decode('utf-8')
+    if out.strip(): print(out.strip())
+    if err.strip(): print("ERR:", err.strip())
+    print()
+
+run_query("OUTPOST PROVIDERS", "SELECT outpost_id, provider_id FROM authentik_outposts_outpost_providers;")
+
+
+
+
+
+
+
+
+c.close()
--- a/scripts/audit_cluster_workloads.py
+++ b/scripts/audit_cluster_workloads.py
@@ -0,0 +1,21 @@
+import paramiko
+
+def inspect_cluster(ip, name):
+    print(f"\n=======================================================")
+    print(f"=== {name} WORKLOADS ({ip}) ===")
+    print(f"=======================================================")
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    
+    cmd = "kubectl get deploy,sts,ds -A -o custom-columns=NAMESPACE:.metadata.namespace,KIND:.kind,NAME:.metadata.name,IMAGES:.spec.template.spec.containers[*].image"
+    _, out, _ = ssh.exec_command(cmd)
+    lines = out.read().decode().splitlines()
+    for l in lines:
+        if not any(k in l for k in ['kube-system', 'cattle-', 'metallb-system', 'longhorn-system']):
+            print(l)
+    ssh.close()
+
+inspect_cluster('140.44.4.11', 'AERODROME')
+inspect_cluster('140.44.4.41', 'SPACEPORT')
+
--- a/scripts/audit_gitops_manifests.py
+++ b/scripts/audit_gitops_manifests.py
@@ -0,0 +1,40 @@
+import os
+import re
+import paramiko
+
+manifest_root = r"c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests"
+
+print("=== GITOPS MANIFEST IMAGE AUDIT ===")
+
+image_re = re.compile(r'image:\s*([^\s"\']+)', re.IGNORECASE)
+chart_re = re.compile(r'chart:\s*([^\s"\']+)', re.IGNORECASE)
+version_re = re.compile(r'version:\s*([^\s"\']+)', re.IGNORECASE)
+
+manifest_data = {}
+
+for root, dirs, files in os.walk(manifest_root):
+    for f in files:
+        if f.endswith('.yaml') or f.endswith('.yml'):
+            path = os.path.join(root, f)
+            rel_path = os.path.relpath(path, manifest_root)
+            with open(path, 'r', encoding='utf-8', errors='ignore') as fp:
+                content = fp.read()
+                images = image_re.findall(content)
+                charts = chart_re.findall(content)
+                versions = version_re.findall(content)
+                if images or charts:
+                    manifest_data[rel_path] = {
+                        'images': list(set(images)),
+                        'charts': list(set(charts)),
+                        'versions': list(set(versions))
+                    }
+
+for rel_path, data in sorted(manifest_data.items()):
+    print(f"\nFile: {rel_path}")
+    if data['images']:
+        print("  Images:", ", ".join(data['images']))
+    if data['charts']:
+        print("  Charts:", ", ".join(data['charts']))
+    if data['versions']:
+        print("  Versions:", ", ".join(data['versions']))
+
--- a/scripts/audit_spaceport_kernels.py
+++ b/scripts/audit_spaceport_kernels.py
@@ -0,0 +1,39 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+spaceport_nodes = [
+    ('sninfr0lnxctl01', '140.44.4.41'),
+    ('sninfr0lnxctl02', '140.44.4.42'),
+    ('sninfr0lnxctl03', '140.44.4.43'),
+    ('sninfr0lnxctl04', '140.44.4.44'),
+    ('sninfr0lnxwrk01', '140.44.4.51'),
+    ('sninfr0lnxwrk02', '140.44.4.52'),
+    ('sninfr0lnxwrk03', '140.44.4.53'),
+    ('sninfr0lnxwrk04', '140.44.4.54'),
+    ('sninfr0lnxwrk05', '140.44.4.55'),
+    ('sninfr0lnxwrk06', '140.44.4.56'),
+]
+
+print(f"{'NODE':<18} {'IP':<15} {'RUNNING KERNEL':<22} {'LATEST MODULES':<25} {'VETH LOADED':<12} {'UPTIME'}")
+print("="*105)
+
+for name, ip in spaceport_nodes:
+    try:
+        client.connect(ip, username="root", key_filename=key_path, timeout=5)
+        stdin, stdout, stderr = client.exec_command("uname -r; ls -1 /lib/modules | sort -V | tail -n 1; lsmod | grep -q veth && echo YES || echo NO; uptime -p")
+        lines = [l.strip() for l in stdout.read().decode('utf-8').strip().split('\n') if l.strip()]
+        running_k = lines[0] if len(lines) > 0 else "?"
+        latest_mod = lines[1] if len(lines) > 1 else "?"
+        veth = lines[2] if len(lines) > 2 else "?"
+        uptime_str = lines[3] if len(lines) > 3 else "?"
+        print(f"{name:<18} {ip:<15} {running_k:<22} {latest_mod:<25} {veth:<12} {uptime_str}")
+        client.close()
+    except Exception as e:
+        print(f"{name:<18} {ip:<15} ERROR: {e}")
+
--- a/scripts/authorize_and_test.py
+++ b/scripts/authorize_and_test.py
@@ -0,0 +1,35 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+pub_key = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6ycMke4AaV2qzWfULZCTZOKhSKf4pY2qRGYjYzs1MGKutJxt2jkYNK2aS0XffxLRR49v23zHbBwLYraNl2Mkb9rsKsyvKn8CAa1CEl9NxOci4LDRT8CeWHRnMZn8Mdu1pfeRwM26/Bjnd4RJrSKC7fyM0OlOpEp0DgnuPBqBtPMFGw+TCUT8eZ0pBrAI94wEan4l35kmuBcqm6RH3KuD1dDJKnPLzJ3/T21CViYL0GgFIDDor/hGI45VfOI/jKmkgM4YoudTCde6uPV5e2yaJpORZlR4nDxg9ntXp8HxbHGZkXYr55Ibsjem0lMU6nBh33jGbJVvat1Ay4oNCbygJsHfC/8ppe0bnLhnasxVAXhTvklZZL/sf7K/VS0ZifpfqB6DKtr1lziOm2wQeahv+Y1uoHkVNa3XY8Fx7xGPeT1BGd9lfcbUoweoTdMPMIPaSZlNGY32AHYkM4PShG9yI1coAqXzMby0+Z/Qwtr4l6cLKVvcGgqh5sd+z1+YvZbs= root@SNINFR2LNXCTL01"
+
+# 1. Authorize on 140.44.4.71
+c71 = paramiko.SSHClient()
+c71.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c71.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+
+cmd = f"""
+mkdir -p ~/.ssh && chmod 700 ~/.ssh
+grep -qF '{pub_key}' ~/.ssh/authorized_keys 2>/dev/null || echo '{pub_key}' >> ~/.ssh/authorized_keys
+chmod 600 ~/.ssh/authorized_keys
+"""
+c71.exec_command(cmd)
+c71.close()
+print("Authorized 140.44.4.11 key on 140.44.4.71.")
+
+# 2. Test SSH from 140.44.4.11 to 140.44.4.71
+c11 = paramiko.SSHClient()
+c11.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c11.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = c11.exec_command("ssh -o BatchMode=yes -o StrictHostKeyChecking=no root@140.44.4.71 hostname")
+out = stdout.read().decode('utf-8').strip()
+err = stderr.read().decode('utf-8').strip()
+print("Result of ssh from 140.44.4.11 to 140.44.4.71:")
+print("STDOUT:", out)
+if err: print("STDERR:", err)
+c11.close()
+
--- a/scripts/check_13_conf.py
+++ b/scripts/check_13_conf.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command('docker exec $(docker ps -q -f name=nginx-proxy-manager_app) cat /data/nginx/proxy_host/13.conf | grep server_name')
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_214_pods.py
+++ b/scripts/check_214_pods.py
@@ -0,0 +1,25 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system get pods -o wide")
+print("=== PODS IN CATTLE-SYSTEM ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system rollout status deployment/rancher --timeout=30s")
+print("=== ROLLOUT STATUS ===")
+print(stdout.read().decode('utf-8'))
+print(stderr.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("curl -k -I -s https://aerodrome.snarfnet.net | head -n 10")
+print("=== ENDPOINT ===")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+
--- a/scripts/check_215_pods.py
+++ b/scripts/check_215_pods.py
@@ -0,0 +1,29 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system get pods -o wide")
+print("=== PODS IN CATTLE-SYSTEM ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system rollout status deployment/rancher --timeout=30s")
+print("=== ROLLOUT STATUS ===")
+print(stdout.read().decode('utf-8'))
+print(stderr.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("curl -k -I -s https://aerodrome.snarfnet.net | head -n 10")
+print("=== ENDPOINT ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm list -n cattle-system")
+print("=== HELM LIST ===")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+
--- a/scripts/check_51.py
+++ b/scripts/check_51.py
@@ -0,0 +1,19 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.51', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+run("ss -tulpn | grep ':80 '")
+run("ip a")
+run("kubectl get pod -A -o wide | grep sninfr4lnxwrk01 | grep pihole || true")
+ssh.close()
+
--- a/scripts/check_51_nat.py
+++ b/scripts/check_51_nat.py
@@ -0,0 +1,18 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.51', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+run("iptables -t nat -S | grep -E '80\b|31905|31937'")
+run("crictl ps")
+ssh.close()
+
--- a/scripts/check_71_containers.py
+++ b/scripts/check_71_containers.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_aero_health.py
+++ b/scripts/check_aero_health.py
@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = client.exec_command('k3s kubectl get pods -n cattle-system')
+print("=== AERODROME CATTLE-SYSTEM PODS ===")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_aero_infra.py
+++ b/scripts/check_aero_infra.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl -n infra get deploy,svc,pod")
+print(stdout.read().decode('utf-8'))
+c.close()
+
--- a/scripts/check_aero_svc.py
+++ b/scripts/check_aero_svc.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl -n infra get svc")
+print(stdout.read().decode('utf-8'))
+c.close()
+
--- a/scripts/check_aerodrome_pods.py
+++ b/scripts/check_aerodrome_pods.py
@@ -0,0 +1,15 @@
+import paramiko
+
+s = paramiko.SSHClient()
+s.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+s.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    _, out, _ = s.exec_command(cmd)
+    print(out.read().decode())
+
+run("kubectl get pods -A | grep -v -E 'Running|Completed' || true")
+run("kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,ROBUSTNESS:.status.robustness,STATE:.status.state")
+s.close()
+
--- a/scripts/check_aerodrome_preflight.py
+++ b/scripts/check_aerodrome_preflight.py
@@ -0,0 +1,19 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+run("kubectl get nodes")
+run("kubectl get pods -A | grep -v -E 'Running|Completed' || true")
+run("kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,ROBUSTNESS:.status.robustness | grep -v healthy || true")
+ssh.close()
+
--- a/scripts/check_aerodrome_quick.py
+++ b/scripts/check_aerodrome_quick.py
@@ -0,0 +1,8 @@
+import paramiko
+ssh=paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+_,o,_=ssh.exec_command('kubectl get nodes')
+print(o.read().decode())
+ssh.close()
+
--- a/scripts/check_after_reboot.py
+++ b/scripts/check_after_reboot.py
@@ -0,0 +1,29 @@
+import paramiko
+import sys
+import time
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+print("=== CHECKING VETH ON SNINFR0LNXCTL03 ===")
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("modprobe veth; lsmod | grep veth; systemctl is-active k3s-server || systemctl is-active k3s")
+print(stdout.read().decode('utf-8'))
+client.close()
+
+time.sleep(5)
+
+print("=== CHECKING PODS IN CATTLE-SYSTEM ON SPACEPORT ===")
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system get pods -o wide")
+print(stdout.read().decode('utf-8'))
+
+print("=== CHECKING POD EVENTS ===")
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system describe pod -l app=cattle-cluster-agent | grep -A 10 Events:")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+
--- a/scripts/check_agent_deploy.py
+++ b/scripts/check_agent_deploy.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system get deployment cattle-cluster-agent -o yaml")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_agent_logs.py
+++ b/scripts/check_agent_logs.py
@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system logs -l app=cattle-cluster-agent --tail=100")
+print("=== SPACEPORT CATTLE-CLUSTER-AGENT LOGS ===")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_agent_logs_tail.py
+++ b/scripts/check_agent_logs_tail.py
@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system logs -l app=cattle-cluster-agent -c cluster-register --tail=20")
+print("=== AGENT LOGS ===")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_agent_now.py
+++ b/scripts/check_agent_now.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system logs -l app=cattle-cluster-agent -c cluster-register --tail=25")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_all_coder_snapshots.py
+++ b/scripts/check_all_coder_snapshots.py
@@ -0,0 +1,24 @@
+import paramiko
+
+nodes = [
+    ("wrk01", "140.44.4.51"),
+    ("wrk02", "140.44.4.52"),
+    ("wrk03", "140.44.4.53"),
+    ("wrk04", "140.44.4.54"),
+    ("wrk05", "140.44.4.55"),
+    ("wrk06", "140.44.4.56"),
+]
+
+for name, ip in nodes:
+    try:
+        ssh = paramiko.SSHClient()
+        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+        ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=4)
+        _, out, _ = ssh.exec_command("find /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/ -path '*/home/coder/.config/coderv2' 2>/dev/null")
+        res = out.read().decode().strip()
+        if res:
+            print(f"[{name} - {ip}] Found:\n{res}")
+        ssh.close()
+    except Exception as e:
+        print(f"[{name}] err: {e}")
+
--- a/scripts/check_all_endpoints.py
+++ b/scripts/check_all_endpoints.py
@@ -0,0 +1,19 @@
+import subprocess
+
+endpoints = [
+    ("Argo CD", "https://argo.snarfnet.net"),
+    ("Rancher Manager", "https://aerodrome.snarfnet.net"),
+    ("AWX Console", "https://awx.snarfnet.net"),
+    ("Authentik SSO", "https://authentik.snarfnet.net"),
+    ("Uptime Kuma", "https://uptime.snarfnet.net/dashboard"),
+    ("Gitea GitOps", "http://140.44.4.51:90"),
+    ("Zabbix Server", "http://140.44.4.51:8888"),
+    ("Portainer Server", "https://140.44.4.71:9443")
+]
+
+print("=== ENDPOINT HEALTH CHECK ===")
+for name, url in endpoints:
+    res = subprocess.run(f"curl.exe -k -s -o nul -w '%{{http_code}}' {url}", capture_output=True, text=True, shell=True)
+    code = res.stdout.strip()
+    print(f"[{code}] {name} ({url})")
+
--- a/scripts/check_all_tls.py
+++ b/scripts/check_all_tls.py
@@ -0,0 +1,27 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+# Check certificates in Aerodrome across all namespaces
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("""
+KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get secrets -A --field-selector=type=kubernetes.io/tls
+""")
+print("=== TLS SECRETS ON AERODROME ===")
+print(stdout.read().decode('utf-8'))
+client.close()
+
+# Check certificates on Spaceport across all namespaces
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("""
+KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get secrets -A --field-selector=type=kubernetes.io/tls
+""")
+print("=== TLS SECRETS ON SPACEPORT ===")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_apis.py
+++ b/scripts/check_apis.py
@@ -0,0 +1,25 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+
+test_cmd = """
+cd /tmp/rancher-test
+KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm template rancher ./rancher --namespace cattle-system --set hostname=aerodrome.snarfnet.net --set ingress.tls.source=secret | grep "apiVersion:" | sort | uniq -c
+"""
+
+stdin, stdout, stderr = client.exec_command(test_cmd)
+out = stdout.read().decode('utf-8')
+err = stderr.read().decode('utf-8')
+print("APIs used:")
+print(out)
+if err:
+    print("STDERR:", err)
+
+client.close()
--- a/scripts/check_argo_apps.py
+++ b/scripts/check_argo_apps.py
@@ -0,0 +1,14 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+stdin, stdout, stderr = ssh.exec_command("kubectl get applications -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status")
+print(stdout.read().decode())
+err = stderr.read().decode()
+if err:
+    print("ERR:", err)
+
+ssh.close()
+
--- a/scripts/check_argo_ing.py
+++ b/scripts/check_argo_ing.py
@@ -0,0 +1,22 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+run("kubectl get ingress argo-ing -n infrastructure -o yaml")
+run("kubectl get svc pihole-web -n infrastructure -o yaml")
+run("kubectl get svc -n infrastructure")
+run("kubectl get svc -A | grep -E 'LoadBalancer|140.44.4.62'")
+run("curl -k -I -H 'Host: argo.snarfnet.net' http://140.44.4.62")
+run("curl -k -I -H 'Host: argo.snarfnet.net' https://140.44.4.62")
+ssh.close()
+
--- a/scripts/check_argocd_now.py
+++ b/scripts/check_argocd_now.py
@@ -0,0 +1,11 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+_, out, _ = ssh.exec_command("kubectl get applications -n infrastructure -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status")
+print(out.read().decode())
+
+ssh.close()
+
--- a/scripts/check_auth_links.py
+++ b/scripts/check_auth_links.py
@@ -0,0 +1,25 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+cmd = """
+k3s kubectl -n infra exec authentik-postgresql-0 -- bash -c "PGPASSWORD=\\$(cat /opt/bitnami/postgresql/secrets/password) psql -U authentik -d authentik -c \\"SELECT provider_id, application_id FROM authentik_core_application_backchannel_providers;\\""
+"""
+stdin, stdout, stderr = c.exec_command(cmd)
+print("Backchannel Providers:")
+print(stdout.read().decode('utf-8'))
+
+cmd2 = """
+k3s kubectl -n infra exec authentik-postgresql-0 -- bash -c "PGPASSWORD=\\$(cat /opt/bitnami/postgresql/secrets/password) psql -U authentik -d authentik -c \\"SELECT * FROM authentik_providers_proxy_proxyprovider;\\""
+"""
+stdin, stdout, stderr = c.exec_command(cmd2)
+print("Proxy Providers:")
+print(stdout.read().decode('utf-8'))
+
+c.close()
--- a/scripts/check_auth_tables.py
+++ b/scripts/check_auth_tables.py
@@ -0,0 +1,26 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+cmd = "k3s kubectl -n infra exec authentik-postgresql-0 -- bash -c 'PGPASSWORD=$(cat /opt/bitnami/postgresql/secrets/password) psql -U authentik -d authentik -c \"SELECT name, slug FROM authentik_core_application;\"'"
+stdin, stdout, stderr = c.exec_command(cmd)
+print("Apps:")
+print(stdout.read().decode('utf-8'))
+
+cmd2 = "k3s kubectl -n infra exec authentik-postgresql-0 -- bash -c 'PGPASSWORD=$(cat /opt/bitnami/postgresql/secrets/password) psql -U authentik -d authentik -c \"SELECT name, external_host FROM authentik_providers_proxy_proxyprovider;\"'"
+stdin, stdout, stderr = c.exec_command(cmd2)
+print("Proxy Providers:")
+print(stdout.read().decode('utf-8'))
+
+cmd3 = "k3s kubectl -n infra exec authentik-postgresql-0 -- bash -c 'PGPASSWORD=$(cat /opt/bitnami/postgresql/secrets/password) psql -U authentik -d authentik -c \"SELECT name, client_id FROM authentik_providers_oauth2_oauth2provider;\"'"
+stdin, stdout, stderr = c.exec_command(cmd3)
+print("OAuth2/OIDC Providers:")
+print(stdout.read().decode('utf-8'))
+
+c.close()
--- a/scripts/check_authentik_apps.py
+++ b/scripts/check_authentik_apps.py
@@ -0,0 +1,26 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+cmd = "k3s kubectl -n infra exec authentik-postgresql-0 -- psql -U authentik -d authentik -c 'SELECT id, name, authorization_flow_id FROM authentik_core_application;'"
+stdin, stdout, stderr = c.exec_command(cmd)
+print("Authentik Applications:")
+print(stdout.read().decode('utf-8'))
+
+cmd2 = "k3s kubectl -n infra exec authentik-postgresql-0 -- psql -U authentik -d authentik -c 'SELECT id, name, external_host FROM authentik_providers_proxy_proxyprovider;'"
+stdin, stdout, stderr = c.exec_command(cmd2)
+print("Authentik Proxy Providers:")
+print(stdout.read().decode('utf-8'))
+
+cmd3 = "k3s kubectl -n infra exec authentik-postgresql-0 -- psql -U authentik -d authentik -c 'SELECT id, name, client_id FROM authentik_providers_oauth2_oauth2provider;'"
+stdin, stdout, stderr = c.exec_command(cmd3)
+print("Authentik OAuth2/OIDC Providers:")
+print(stdout.read().decode('utf-8'))
+
+c.close()
--- a/scripts/check_authentik_ingress.py
+++ b/scripts/check_authentik_ingress.py
@@ -0,0 +1,19 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl get middleware.traefik.io,middleware.traefik.containo.us -A 2>/dev/null")
+print("=== TRAEFIK MIDDLEWARES ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl get ingress -A")
+print("=== ALL INGRESSES ===")
+print(stdout.read().decode('utf-8'))
+
+c.close()
--- a/scripts/check_blkid.py
+++ b/scripts/check_blkid.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("blkid")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_boot_ctl03.py
+++ b/scripts/check_boot_ctl03.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("ls -la /boot; ls -la /boot/firmware 2>/dev/null | head -n 20")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_cattle_all.py
+++ b/scripts/check_cattle_all.py
@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system get pods -o wide")
+print("=== SPACEPORT CATTLE-SYSTEM PODS ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system get deployment cattle-cluster-agent")
+print("=== DEPLOYMENT STATUS ===")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+
--- a/scripts/check_cattle_pods.py
+++ b/scripts/check_cattle_pods.py
@@ -0,0 +1,14 @@
+import paramiko
+
+def check_cattle(ip, cluster):
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    print(f"\n[{cluster}] Checking cattle-system pods...")
+    _, out, _ = ssh.exec_command("kubectl get pods -n cattle-system -o wide")
+    print(out.read().decode().strip())
+    ssh.close()
+
+check_cattle('140.44.4.11', 'AERODROME')
+check_cattle('140.44.4.41', 'SPACEPORT')
+
--- a/scripts/check_cattle_spaceport.py
+++ b/scripts/check_cattle_spaceport.py
@@ -0,0 +1,29 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+
+cmds = [
+    "k3s kubectl -n cattle-system get pods -o wide",
+    "k3s kubectl -n cattle-system describe pod -l app=cattle-cluster-agent",
+    "k3s kubectl get events -A --sort-by='.lastTimestamp' | tail -n 20"
+]
+
+for cmd in cmds:
+    print(f"\n==========================================")
+    print(f">>> Running on Spaceport: {cmd}")
+    print(f"==========================================")
+    stdin, stdout, stderr = client.exec_command(cmd)
+    print(stdout.read().decode('utf-8'))
+    err = stderr.read().decode('utf-8')
+    if err:
+        print("STDERR:", err)
+
+client.close()
+
--- a/scripts/check_cattle_spaceport_clean.py
+++ b/scripts/check_cattle_spaceport_clean.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n cattle-system get pods -o wide")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_certbot.py
+++ b/scripts/check_certbot.py
@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+client.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = client.exec_command("docker exec $(docker ps -q -f name=nginx-proxy-manager_app) certbot certificates")
+print("OUT:", stdout.read().decode('utf-8'))
+print("ERR:", stderr.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_chart_kubeversions.py
+++ b/scripts/check_chart_kubeversions.py
@@ -0,0 +1,22 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+versions = [
+    "2.7.10", "2.8.5", "2.9.3", "2.10.3", "2.11.3", "2.12.3", "2.13.3", "2.14.3", "2.15.3"
+]
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+
+for ver in versions:
+    stdin, stdout, stderr = client.exec_command(f"KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm show chart rancher-stable/rancher --version {ver} | grep kubeVersion")
+    out = stdout.read().decode('utf-8').strip()
+    print(f"Rancher {ver}: {out}")
+
+client.close()
+
--- a/scripts/check_chart_versions.py
+++ b/scripts/check_chart_versions.py
@@ -0,0 +1,13 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm search repo rancher-stable/rancher --versions > /tmp/rancher_versions.txt && head -n 30 /tmp/rancher_versions.txt")
+print(stdout.read().decode('utf-8'))
+client.close()
--- a/scripts/check_cluster_cond.py
+++ b/scripts/check_cluster_cond.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = client.exec_command('k3s kubectl get clusters.provisioning.cattle.io -A')
+print("=== PROVISIONING CLUSTERS ===")
+print(stdout.read().decode('utf-8'))
+client.close()
--- a/scripts/check_cluster_details.py
+++ b/scripts/check_cluster_details.py
@@ -0,0 +1,13 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command('KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get clusters.management.cattle.io -o wide')
+print(stdout.read().decode('utf-8'))
+client.close()
--- a/scripts/check_cluster_ready.py
+++ b/scripts/check_cluster_ready.py
@@ -0,0 +1,16 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("""
+KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get clusters.management.cattle.io -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.displayName}{"\tConnected: "}{.status.agentImage}{"\tReady: "}{range .status.conditions[?(@.type=="Ready")]}{.status}{" - "}{.message}{end}{"\n"}{end}'
+""")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_cluster_tokens.py
+++ b/scripts/check_cluster_tokens.py
@@ -0,0 +1,25 @@
+import paramiko
+import sys
+import yaml
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get clusterregistrationtokens.management.cattle.io -A -o yaml")
+data = yaml.safe_load(stdout.read().decode('utf-8'))
+client.close()
+
+for item in data.get('items', []):
+    cluster = item.get('spec', {}).get('clusterName')
+    token = item.get('status', {}).get('token')
+    cmd = item.get('status', {}).get('command')
+    manifest_url = item.get('status', {}).get('manifestUrl')
+    print(f"\nCluster: {cluster}")
+    print(f"Token: {token}")
+    print(f"Command: {cmd}")
+    print(f"Manifest URL: {manifest_url}")
+
--- a/scripts/check_cluster_yaml.py
+++ b/scripts/check_cluster_yaml.py
@@ -0,0 +1,26 @@
+import paramiko
+import sys
+import yaml
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get clusters.management.cattle.io -o yaml")
+data = yaml.safe_load(stdout.read().decode('utf-8'))
+client.close()
+
+for item in data.get('items', []):
+    name = item.get('metadata', {}).get('name')
+    disp = item.get('spec', {}).get('displayName')
+    print(f"\n==========================================")
+    print(f"Cluster: {disp} ({name})")
+    print(f"==========================================")
+    conditions = item.get('status', {}).get('conditions', [])
+    for c in conditions:
+        if c.get('status') != 'True' or c.get('type') in ['Ready', 'Connected']:
+            print(f"  [{c.get('type')}]: status={c.get('status')} reason={c.get('reason')} msg={c.get('message')}")
+
--- a/scripts/check_cmdline.py
+++ b/scripts/check_cmdline.py
@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("cat /proc/cmdline; dmesg | grep -i 'Kernel command line'")
+print("CMDLINE:")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = client.exec_command("ls -la /dev/disk/by-id/; ls -la /dev/disk/by-uuid/")
+print("DISKS:")
+print(stdout.read().decode('utf-8'))
+
+client.close()
+
--- a/scripts/check_coder_logs.py
+++ b/scripts/check_coder_logs.py
@@ -0,0 +1,11 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+_, out, _ = ssh.exec_command('kubectl logs -n infrastructure -l app.kubernetes.io/name=coder --tail=30')
+print(out.read().decode())
+
+ssh.close()
+
--- a/scripts/check_coder_page.py
+++ b/scripts/check_coder_page.py
@@ -0,0 +1,14 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+_, out, _ = ssh.exec_command("curl -s https://coder.snarfnet.net/api/v2/users/first")
+print("First user status:", out.read().decode())
+
+_, out2, _ = ssh.exec_command("curl -s https://coder.snarfnet.net/api/v2/experiments")
+print("Experiments:", out2.read().decode())
+
+ssh.close()
+
--- a/scripts/check_coredns.py
+++ b/scripts/check_coredns.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n kube-system get configmap coredns -o yaml")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_coredns_logs.py
+++ b/scripts/check_coredns_logs.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n kube-system logs -l k8s-app=kube-dns --tail=30")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_ctl02_uptime.py
+++ b/scripts/check_ctl02_uptime.py
@@ -0,0 +1,18 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+try:
+    client.connect("140.44.4.42", username="root", key_filename=key_path, timeout=5)
+    stdin, stdout, stderr = client.exec_command("uptime; uname -r; modprobe veth; lsmod | grep veth")
+    print("OUTPUT from ctl02:")
+    print(stdout.read().decode('utf-8'))
+    client.close()
+except Exception as e:
+    print("Connect error:", e)
+
--- a/scripts/check_ctl03.py
+++ b/scripts/check_ctl03.py
@@ -0,0 +1,33 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+node_ip = "140.44.4.43"
+print(f"Connecting to sninfr0lnxctl03 ({node_ip})...")
+client.connect(node_ip, username="root", key_filename=key_path, timeout=10)
+
+cmds = [
+    "uname -r",
+    "ls -d /lib/modules/*",
+    "lsmod | grep -E 'veth|vxlan|br_netfilter'",
+    "modprobe veth",
+    "dmesg | tail -n 25"
+]
+
+for cmd in cmds:
+    print(f"\n>>> Running on {node_ip}: {cmd}")
+    stdin, stdout, stderr = client.exec_command(cmd)
+    out = stdout.read().decode('utf-8').strip()
+    err = stderr.read().decode('utf-8').strip()
+    if out:
+        print(out)
+    if err:
+        print("STDERR:", err)
+
+client.close()
+
--- a/scripts/check_current_status.py
+++ b/scripts/check_current_status.py
@@ -0,0 +1,25 @@
+import paramiko
+import sys
+import yaml
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl get clusters.management.cattle.io -o yaml")
+data = yaml.safe_load(stdout.read().decode('utf-8'))
+client.close()
+
+for item in data.get('items', []):
+    name = item.get('metadata', {}).get('name')
+    disp = item.get('spec', {}).get('displayName')
+    print(f"\n==========================================")
+    print(f"Cluster: {disp} ({name})")
+    print(f"==========================================")
+    for c in item.get('status', {}).get('conditions', []):
+        if c.get('type') in ['Ready', 'Connected']:
+            print(f"  [{c.get('type')}]: status={c.get('status')} reason={c.get('reason')} msg={c.get('message')}")
+
--- a/scripts/check_custom_coredns.py
+++ b/scripts/check_custom_coredns.py
@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("k3s kubectl -n kube-system get configmap coredns-custom")
+print(stdout.read().decode('utf-8'))
+print(stderr.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_diff_kernel.py
+++ b/scripts/check_diff_kernel.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("cmp /boot/vmlinuz-6.12.109+rpt-rpi-2712 /boot/firmware/kernel_2712.img && echo IDENTICAL || echo DIFFERENT")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_dmesg_head.py
+++ b/scripts/check_dmesg_head.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("dmesg | head -n 10")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_dns_query.py
+++ b/scripts/check_dns_query.py
@@ -0,0 +1,27 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+cmd = "dig @140.44.4.36 google.com"
+stdin, stdout, stderr = c.exec_command(cmd)
+print("=== DIG FULL OUTPUT ===")
+print(stdout.read().decode('utf-8'))
+
+cmd2 = "k3s kubectl -n dns logs deploy/pihole --tail=25"
+stdin, stdout, stderr = c.exec_command(cmd2)
+print("=== PIHOLE LOGS ===")
+print(stdout.read().decode('utf-8'))
+
+cmd3 = "k3s kubectl -n dns logs deploy/technitium-dns --tail=25"
+stdin, stdout, stderr = c.exec_command(cmd3)
+print("=== TECHNITIUM LOGS ===")
+print(stdout.read().decode('utf-8'))
+
+c.close()
+
--- a/scripts/check_dns_status.py
+++ b/scripts/check_dns_status.py
@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl -n dns get pods,svc")
+print("=== PODS & SVCS ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl -n dns describe svc pihole-dns-vip | grep -E 'Events:|Message|Failed'")
+print("=== SERVICE EVENTS ===")
+print(stdout.read().decode('utf-8'))
+
+c.close()
+
--- a/scripts/check_dns_svcs.py
+++ b/scripts/check_dns_svcs.py
@@ -0,0 +1,21 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+cmd = "k3s kubectl -n dns get pods,svc -o wide"
+stdin, stdout, stderr = c.exec_command(cmd)
+print("=== AERODROME DNS WORKLOADS ===")
+print(stdout.read().decode('utf-8'))
+
+c.connect("140.44.4.41", username="root", key_filename=key_path, timeout=5)
+stdin, stdout, stderr = c.exec_command(cmd)
+print("=== SPACEPORT DNS WORKLOADS ===")
+print(stdout.read().decode('utf-8'))
+
+c.close()
--- a/scripts/check_dpkg_kernel.py
+++ b/scripts/check_dpkg_kernel.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("dpkg -l | grep -E 'linux-|raspberrypi-'")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_env_in_pod.py
+++ b/scripts/check_env_in_pod.py
@@ -0,0 +1,21 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl -n dns exec deployment/technitium-dns -- env")
+print("ENV IN POD:")
+for line in stdout.read().decode('utf-8').splitlines():
+    if "DNS_SERVER" in line:
+        print(" ", line)
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl -n dns exec deployment/technitium-dns -- ls -la /etc/dns/logs /etc/dns")
+print("FILES:")
+print(stdout.read().decode('utf-8'))
+
+c.close()
--- a/scripts/check_firmware_version.py
+++ b/scripts/check_firmware_version.py
@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("zcat /boot/firmware/kernel_2712.img 2>/dev/null | strings | grep 'Linux version' | head -n 5")
+print("kernel_2712.img version:")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_flows.py
+++ b/scripts/check_flows.py
@@ -0,0 +1,29 @@
+import urllib.request
+import ssl
+import json
+
+ctx = ssl.create_default_context()
+ctx.check_hostname = False
+ctx.verify_mode = ssl.CERT_NONE
+
+req = urllib.request.Request(
+    "https://authentik.snarfnet.net/api/v3/flows/instances/?designation=authorization",
+    headers={"Authorization": "Bearer SnarfNetPermanentAutomationToken2026"}
+)
+
+with urllib.request.urlopen(req, context=ctx, timeout=5) as resp:
+    data = json.loads(resp.read().decode('utf-8'))
+    print("Authorization flows:")
+    for f in data.get('results', []):
+        print(f"  {f.get('name')} (slug: {f.get('slug')}, pk: {f.get('pk')})")
+
+req3 = urllib.request.Request(
+    "https://authentik.snarfnet.net/api/v3/propertymappings/provider/scope/",
+    headers={"Authorization": "Bearer SnarfNetPermanentAutomationToken2026"}
+)
+with urllib.request.urlopen(req3, context=ctx, timeout=5) as resp:
+    data3 = json.loads(resp.read().decode('utf-8'))
+    print("Scope Property Mappings:")
+    for m in data3.get('results', []):
+        print(f"  {m.get('scope_name')}: {m.get('pk')}")
+
--- a/scripts/check_forwardauth.py
+++ b/scripts/check_forwardauth.py
@@ -0,0 +1,13 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl -n longhorn-system get middleware authentik-forwardauth -o yaml")
+print(stdout.read().decode('utf-8'))
+c.close()
--- a/scripts/check_fstab_ctl03.py
+++ b/scripts/check_fstab_ctl03.py
@@ -0,0 +1,24 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+
+cmds = [
+    "lsblk -f",
+    "cat /etc/fstab",
+    "mount | grep -i boot"
+]
+
+for cmd in cmds:
+    print(f"\n>>> Running on 140.44.4.43: {cmd}")
+    stdin, stdout, stderr = client.exec_command(cmd)
+    print(stdout.read().decode('utf-8'))
+
+client.close()
+
--- a/scripts/check_gateway.py
+++ b/scripts/check_gateway.py
@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = c.exec_command("curl -k -s -I http://140.44.4.1/ || curl -k -s -I https://140.44.4.1/")
+print("Gateway 140.44.4.1 HTTP headers:")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = c.exec_command("arp -an | grep 140.44.4.1")
+print("ARP for 140.44.4.1:")
+print(stdout.read().decode('utf-8'))
+
+c.close()
+
--- a/scripts/check_gateway_type.py
+++ b/scripts/check_gateway_type.py
@@ -0,0 +1,16 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+
+stdin, stdout, stderr = c.exec_command("curl -k -s -L https://140.44.4.1/ | grep -iE 'title|unifi|opnsense|pfsense|mikrotik|netgate|firewall|router'")
+print("Gateway 140.44.4.1 HTML match:")
+print(stdout.read().decode('utf-8'))
+
+c.close()
+
--- a/scripts/check_getent.py
+++ b/scripts/check_getent.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.41", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("getent hosts aerodrome.snarfnet.net")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_ghcr_tags.py
+++ b/scripts/check_ghcr_tags.py
@@ -0,0 +1,24 @@
+import urllib.request
+import json
+
+def get_ghcr_tags(package):
+    # Public unauthenticated ghcr tags endpoint
+    url = f"https://ghcr.io/v2/{package}/tags/list"
+    try:
+        # Get anonymous token
+        token_url = f"https://ghcr.io/token?service=ghcr.io&scope=repository:{package}:pull"
+        with urllib.request.urlopen(token_url, timeout=5) as t_resp:
+            token = json.loads(t_resp.read().decode())['token']
+        
+        req = urllib.request.Request(url, headers={'Authorization': f'Bearer {token}'})
+        with urllib.request.urlopen(req, timeout=5) as resp:
+            data = json.loads(resp.read().decode())
+            tags = [t for t in data.get('tags', []) if not t.endswith('-dev') and not 'beta' in t]
+            return tags[-10:]
+    except Exception as e:
+        return [f"Err: {e}"]
+
+print("Homepage tags:", get_ghcr_tags("gethomepage/homepage"))
+print("Coder tags:", get_ghcr_tags("coder/coder"))
+print("Authentik tags:", get_ghcr_tags("goauthentik/server"))
+
--- a/scripts/check_grafana_spec.py
+++ b/scripts/check_grafana_spec.py
@@ -0,0 +1,15 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    _, out, _ = ssh.exec_command(cmd)
+    print(out.read().decode())
+
+run("kubectl get deployment grafana -n default -o jsonpath='{.spec.template.spec.containers[*].name}'")
+run("kubectl get deployment grafana -n default -o jsonpath='{.spec.template.spec.containers[*].image}'")
+ssh.close()
+
--- a/scripts/check_harvester_ip.py
+++ b/scripts/check_harvester_ip.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("ping -c 2 harvester.spacedock.snarfnet.net; getent hosts harvester.spacedock.snarfnet.net")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_helm_cattle.py
+++ b/scripts/check_helm_cattle.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml helm list -n cattle-system")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_helm_releases.py
+++ b/scripts/check_helm_releases.py
@@ -0,0 +1,14 @@
+import paramiko
+
+def check_helm(ip, name):
+    print(f"\n=== {name} HELM RELEASES ({ip}) ===")
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    _, out, _ = ssh.exec_command("helm list -A")
+    print(out.read().decode())
+    ssh.close()
+
+check_helm('140.44.4.11', 'AERODROME')
+check_helm('140.44.4.41', 'SPACEPORT')
+
--- a/scripts/check_historical_data.py
+++ b/scripts/check_historical_data.py
@@ -0,0 +1,60 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+# 1. Inspect Gotify database on Swarm
+print("=================== GOTIFY (Swarm 140.44.4.71) ===================")
+ssh_swm = paramiko.SSHClient()
+ssh_swm.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh_swm.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run_swm(cmd):
+    _, out, err = ssh_swm.exec_command(cmd)
+    res = out.read().decode('utf-8', errors='replace').strip()
+    if res:
+        print(res)
+
+run_swm("cat c:\\Users\\Snarf\\OneDrive\\Desktop\\code\\SnarfCode\\docker-deployments\\gotify.yml 2>/dev/null || true")
+run_swm("sudo docker service inspect gotify_gotify --format '{{json .Spec.TaskTemplate.ContainerSpec.Mounts}}'")
+run_swm("sudo docker run --rm -v gotify_gotify_data:/data alpine ls -la /data 2>/dev/null || find /var/lib/docker/volumes/ -name 'gotify.db' 2>/dev/null")
+
+# Query gotify.db if found
+run_swm("""python3 -c '
+import sqlite3, glob
+db_files = glob.glob("/var/lib/docker/volumes/**/gotify.db", recursive=True)
+print("Gotify DB files:", db_files)
+for db in db_files:
+    try:
+        conn = sqlite3.connect(db)
+        cur = conn.cursor()
+        print("Tables in", db)
+        for row in cur.execute("SELECT name FROM sqlite_master WHERE type=\"table\""):
+            print(" ", row[0])
+        print("Applications count:", cur.execute("SELECT count(*) FROM applications").fetchone()[0])
+        print("Clients count:", cur.execute("SELECT count(*) FROM clients").fetchone()[0])
+        print("Messages count:", cur.execute("SELECT count(*) FROM messages").fetchone()[0])
+    except Exception as e:
+        print("Err reading", db, e)
+'""")
+
+ssh_swm.close()
+
+# 2. Inspect Semaphore BoltDB on Spaceport
+print("\n=================== SEMAPHORE (Spaceport 140.44.4.41) ===================")
+ssh_sp = paramiko.SSHClient()
+ssh_sp.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh_sp.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run_sp(cmd):
+    _, out, err = ssh_sp.exec_command(cmd)
+    res = out.read().decode('utf-8', errors='replace').strip()
+    if res:
+        print(res)
+
+run_sp("kubectl exec -n infrastructure deployment/semaphore -- ls -lh /var/lib/semaphore/database.boltdb")
+# Let's inspect projects and templates using semaphore CLI or strings
+run_sp("kubectl exec -n infrastructure deployment/semaphore -- strings /var/lib/semaphore/database.boltdb | grep -E 'project|playbook|inventory|repository' | head -n 30")
+
+ssh_sp.close()
+
--- a/scripts/check_homepage_spec.py
+++ b/scripts/check_homepage_spec.py
@@ -0,0 +1,15 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    _, out, _ = ssh.exec_command(cmd)
+    print(out.read().decode())
+
+run("kubectl get deployment homepage-2-1744068158 -n infrastructure -o jsonpath='{.spec.template.spec.containers[*].name}'")
+run("kubectl get deployment homepage-2-1744068158 -n infrastructure -o jsonpath='{.spec.template.spec.containers[*].image}'")
+ssh.close()
+
--- a/scripts/check_housekeeping.py
+++ b/scripts/check_housekeeping.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=15)
+stdin, stdout, stderr = client.exec_command("KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubectl -n cattle-system get pods --field-selector=status.phase=Succeeded")
+print("Completed pods:", stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_idrac_ips.py
+++ b/scripts/check_idrac_ips.py
@@ -0,0 +1,18 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.11", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("""
+for h in spacedock1.snarfnet.net spacedock2.snarfnet.net spacedock3.snarfnet.net spacedock4.snarfnet.net; do
+    getent hosts $h
+done
+""")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_ingress_yaml.py
+++ b/scripts/check_ingress_yaml.py
@@ -0,0 +1,23 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+c = paramiko.SSHClient()
+c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl -n monitoring get ingress uptime-ing -o yaml")
+print("=== UPTIME INGRESS ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl -n longhorn-system get ingress aerodrome-longhorn -o yaml")
+print("=== LONGHORN INGRESS ===")
+print(stdout.read().decode('utf-8'))
+
+stdin, stdout, stderr = c.exec_command("k3s kubectl get middleware -A")
+print("=== MIDDLEWARES ===")
+print(stdout.read().decode('utf-8'))
+
+c.close()
--- a/scripts/check_jenkins.py
+++ b/scripts/check_jenkins.py
@@ -0,0 +1,23 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    _, out, err = ssh.exec_command(cmd)
+    res = out.read().decode().strip()
+    if res:
+        print(res)
+    e = err.read().decode().strip()
+    if e:
+        print("ERR:", e)
+
+run("kubectl get pod -n infrastructure jenkins-0 -o wide")
+run("kubectl describe pod -n infrastructure jenkins-0 | tail -n 25")
+run("curl -k -s -I https://jenkins.snarfnet.net | head -n 5")
+run("curl -s -I http://140.44.4.52:101 | head -n 5")
+
+ssh.close()
+
--- a/scripts/check_job134.py
+++ b/scripts/check_job134.py
@@ -0,0 +1,28 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=134)
+print(f"Job [134]: Status={job.status}, Elapsed={job.elapsed}s, Failed={job.failed}")
+events = job.job_events.filter(event__in=['runner_on_ok', 'runner_on_failed', 'playbook_on_stats']).order_by('-id')[:10]
+for e in events:
+    print(f"[{e.event}] Host: {e.event_data.get('host')} - Task: {e.event_data.get('task')}")
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+
+_, out, _ = ssh.exec_command('kubectl get nodes')
+print("=== KUBERNETES NODES ===")
+print(out.read().decode())
+
+ssh.close()
+
--- a/scripts/check_job134_end.py
+++ b/scripts/check_job134_end.py
@@ -0,0 +1,20 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=134)
+print(job.result_stdout[-1500:])
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+
--- a/scripts/check_job134_explain.py
+++ b/scripts/check_job134_explain.py
@@ -0,0 +1,22 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=134)
+print(f"Status: {job.status}")
+print(f"Job explanation: {job.job_explanation}")
+print(f"Result Traceback: {job.result_traceback}")
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+
--- a/scripts/check_job134_fail.py
+++ b/scripts/check_job134_fail.py
@@ -0,0 +1,25 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=134)
+failed_events = job.job_events.filter(failed=True)
+for e in failed_events:
+    res = e.event_data.get('res', {})
+    msg = res.get('msg') or res.get('stderr') or res.get('stdout') or str(res)
+    print(f"Host: {e.event_data.get('host')}, Task: {e.event_data.get('task')}")
+    print(f"Details: {msg[:300]}")
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+
--- a/scripts/check_job134_stdout.py
+++ b/scripts/check_job134_stdout.py
@@ -0,0 +1,21 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=134)
+lines = [l for l in job.result_stdout.split('\\n') if l.strip()]
+print('\\n'.join(lines[-35:]))
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+ssh.close()
+
--- a/scripts/check_job_status.py
+++ b/scripts/check_job_status.py
@@ -0,0 +1,21 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+py_check = """
+from awx.main.models import Job
+job = Job.objects.get(id=131)
+print(f"Job [131]: Status={job.status}, Elapsed={job.elapsed}s, Failed={job.failed}")
+"""
+
+cmd = f"kubectl exec -i -n awx -c awx-task deployment/awx-task -- awx-manage shell_plus --quiet << 'EOF'\n{py_check}\nEOF"
+stdin, stdout, stderr = ssh.exec_command(cmd)
+print(stdout.read().decode())
+
+ssh.close()
+
--- a/scripts/check_k3s_channels.py
+++ b/scripts/check_k3s_channels.py
@@ -0,0 +1,12 @@
+import urllib.request
+import json
+
+url = "https://update.k3s.io/v1-release/channels"
+try:
+    req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0', 'Accept': 'application/json'})
+    with urllib.request.urlopen(req, timeout=10) as resp:
+        data = json.loads(resp.read().decode())
+        for item in data.get('data', []):
+            print(f"{item.get('id')}: {item.get('latest')}")
+except Exception as e:
+    print("Error:", e)
--- a/scripts/check_kernel_file.py
+++ b/scripts/check_kernel_file.py
@@ -0,0 +1,14 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("file /boot/firmware/kernel_2712.img; ls -la /boot/vmlinuz*")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_kernel_str.py
+++ b/scripts/check_kernel_str.py
@@ -0,0 +1,15 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
+client = paramiko.SSHClient()
+client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+
+client.connect("140.44.4.43", username="root", key_filename=key_path, timeout=10)
+stdin, stdout, stderr = client.exec_command("strings /boot/firmware/kernel_2712.img | grep -E '^Linux version' | head -n 5")
+print("Kernel version inside kernel_2712.img:")
+print(stdout.read().decode('utf-8'))
+client.close()
+
--- a/scripts/check_logs_utf8.py
+++ b/scripts/check_logs_utf8.py
@@ -0,0 +1,32 @@
+import paramiko
+import sys
+
+sys.stdout.reconfigure(encoding='utf-8')
+
+ssh_sp = paramiko.SSHClient()
+ssh_sp.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh_sp.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+ssh_swm = paramiko.SSHClient()
+ssh_swm.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh_swm.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(ssh, cmd):
+    print(f"\n--- {cmd} ---")
+    _, out, err = ssh.exec_command(cmd)
+    res = out.read().decode('utf-8', errors='replace').strip()
+    if res:
+        print(res)
+
+print("=== PEANUT LOGS (Swarm) ===")
+run(ssh_swm, "sudo docker service logs peanut_peanut --tail=50")
+
+print("\n=== CODER LOGS (Spaceport) ===")
+run(ssh_sp, "kubectl logs -n infrastructure -l app.kubernetes.io/name=coder --tail=50")
+
+print("\n=== SEMAPHORE LOGS (Spaceport) ===")
+run(ssh_sp, "kubectl logs -n infrastructure -l app.kubernetes.io/name=semaphore --tail=50")
+
+ssh_sp.close()
+ssh_swm.close()
+
--- a/scripts/check_longhorn_version.py
+++ b/scripts/check_longhorn_version.py
@@ -0,0 +1,15 @@
+import paramiko
+
+def check_lh(ip, cluster):
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    ssh.connect(ip, username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+    print(f"\n[{cluster}] Checking Longhorn images...")
+    _, out, _ = ssh.exec_command("kubectl get daemonset -n longhorn-system longhorn-manager -o jsonpath='{.spec.template.spec.containers[0].image}{\"\\n\"}'")
+    print("Manager image:", out.read().decode().strip())
+    _, out, _ = ssh.exec_command("kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,ROBUSTNESS:.status.robustness,STATE:.status.state | head -n 10")
+    print(out.read().decode().strip())
+    ssh.close()
+
+check_lh('140.44.4.11', 'AERODROME')
+check_lh('140.44.4.41', 'SPACEPORT')
--- a/scripts/check_npm_all_hosts.py
+++ b/scripts/check_npm_all_hosts.py
@@ -0,0 +1,11 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+stdin, stdout, stderr = ssh.exec_command("""sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') grep -E "server_name|set \\$server|set \\$port" /data/nginx/proxy_host/*.conf""")
+print(stdout.read().decode())
+
+ssh.close()
+
--- a/scripts/check_npm_argo.py
+++ b/scripts/check_npm_argo.py
@@ -0,0 +1,21 @@
+import paramiko
+
+ssh = paramiko.SSHClient()
+ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
+
+def run(cmd):
+    print(f"=== {cmd} ===")
+    stdin, stdout, stderr = ssh.exec_command(cmd)
+    print(stdout.read().decode())
+    err = stderr.read().decode()
+    if err:
+        print("ERR:", err)
+
+run("sudo docker ps | grep nginx")
+run("sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') find /data -maxdepth 3")
+run("sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') cat /etc/hosts")
+run("""sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') grep -rn "argo" /etc/nginx/ /data/nginx/""")
+run("""sudo docker exec $(sudo docker ps -q --filter 'name=nginx-proxy-manager_app') grep -rn "pihole" /etc/nginx/ /data/nginx/""")
+
+ssh.close()
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Snarf	47acedb455	fix: remove obsolete file	2026-11-05 21:40:37 -05:00
Snarf	fffb446206	feat: add cluster management and audit utility scripts	2026-11-05 21:40:22 -05:00
Snarf	fa969fc1d3	docs: add network blueprint for high-availability DNS, honeypot, and UDM firewall hardening	2026-11-05 11:44:43 -05:00
Snarf	f1c197422d	docs: add DNS setup plan documentation	2026-11-05 11:20:57 -05:00
Snarf	17ca433d0c	feat: add diagnostic scripts for gateway and DNS investigation	2026-11-05 11:17:12 -05:00
Snarf	88191155d2	feat: add administrative and deployment scripts for infrastructure management	2026-11-05 11:10:07 -05:00
Snarf	cbdb37a453	feat: add script to read and parse transcript jsonl steps	2026-11-05 10:17:02 -05:00
Snarf	1fce3f2488	feat: add infrastructure health check and management automation scripts	2026-11-05 10:13:07 -05:00
Snarf	14fac7151f	feat: add scripts to audit and troubleshoot spaceport kernels and cluster nodes	2026-11-05 07:28:22 -05:00
Snarf	b974980d08	feat: add script to check cattle-system pods and events on spaceport server	2026-11-05 06:50:56 -05:00
Snarf	5b955fcbc3	feat: add housekeeping and cleanup scripts for cluster maintenance	2026-11-05 06:50:35 -05:00
Snarf	80738fd8c1	feat: add script to check Rancher cluster and cattle-system status via SSH	2026-11-04 23:32:54 -05:00
Snarf	11749df634	feat: add diagnostic and management scripts for Rancher k3s deployment	2026-11-04 23:27:34 -05:00
Snarf	1e7935e789	feat: add scripts for automating and checking Rancher upgrades via SSH	2026-11-04 23:10:28 -05:00
Snarf	706fccacb1	test: add dry-run script for rancher 2.14.3 helm upgrade	2026-11-04 22:34:52 -05:00
Snarf	114b619c1f	test: add SSH dry-run scripts for Rancher chart versions 2.11.3, 2.12.3, and 2.13.3	2026-11-04 22:25:05 -05:00
Snarf	45f9b557b9	test: add dry-run script for rancher 2.10.3 helm chart deployment	2026-11-04 21:58:01 -05:00
Snarf	07dae0c73e	feat: add scripts for checking rancher clusters and testing helm dry-run	2026-11-04 21:48:30 -05:00
Snarf	93e13c7482	feat: add scripts for testing and automating Rancher upgrades and chart checks	2026-11-04 21:35:52 -05:00
Snarf	8b21087c89	feat: add scripts for testing and automating Rancher upgrades and chart checks	2026-11-04 21:34:42 -05:00
Snarf	ba47f6a384	feat: add scripts to inspect and check Rancher deployments via SSH	2026-11-04 21:26:45 -05:00
Snarf	1b378b29f3	feat: add cluster auditing, management, and upgrade scripts	2026-11-04 17:58:07 -05:00