diff --git a/docs/infrastructure_improvement_plan.md b/docs/infrastructure_improvement_plan.md index b0cd797..6a377c4 100644 --- a/docs/infrastructure_improvement_plan.md +++ b/docs/infrastructure_improvement_plan.md @@ -41,7 +41,7 @@ Our automated live probe across the homelab APIs identified the following baseli | **Aerodrome K3s (`local`)** | All 10 Nodes (`ctl01-03` & `wrk01-07`): **`v1.30.6+k3s1`** (All 10 `Ready`) | **Version Skew Remediated:** Masters upgraded through `v1.27.16` $\rightarrow$ `v1.28.15` $\rightarrow$ `v1.29.10` $\rightarrow$ `v1.30.6+k3s1` with zero workload downtime. Single default SC (`longhorn`). Stuck PVCs cleared. Synology NAS NFS daily backups active. Enrolled in ArgoCD. | | **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: **`v1.31.4+k3s1`** (All 10 `Ready`) | Single default SC (`longhorn`). Stuck PVCs cleared. Synology NAS NFS daily backups active across Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. ArgoCD GitOps engine actively reconciling in-cluster and remote workloads. | | **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. Declarative VLAN networks and VM templates codified in Terraform. Monitored via SNMP / ICMP. | -| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); all 5 online and `Ready` | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. Hardened NFS persistent mounts (`/mnt/nfs_shares`) on Synology NAS (`14.10.10.71:/volume1/docker`). Automated SQLite backup timer active with daily tarball sync to NAS. | +| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); all 5 online and `Ready` | Dual-NPM HA Keepalived cluster active on VIP `140.44.4.70` across `snshpd4lnxswm01` (.71) and `snshpd4lnxswm02` (.72) in `PublishMode: host`. Hardened NFS persistent mounts (`/mnt/nfs_shares`) on Synology NAS (`14.10.10.71:/volume1/docker`). Automated SQLite backup timer active with daily tarball sync to NAS. | | **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 32 hosts monitored across all clusters, hypervisors, and storage arrays with shared PSK (`PSK_autoregister`). Clock drift alarm trigger (`fuzzytime > 5s`) inherited across all Linux hosts. Gotify & Discord problem actions active. | | **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | 5 structured notification channels provisioned. Internal VIP (`http://140.44.4.70:8090`) active for direct programmatic alerts. `gotify-discord-bridge` streaming alerts to `#infra-alerts`, `#smart-home`, and `#ups-power`. | | **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Organization `Snarfnet Homelab`, master SSH credentials, Gitea-synced playbooks project, and 28-host master inventory active. Daily drift audit and rolling upgrade job templates verified green. | @@ -399,7 +399,7 @@ The power resilience architecture uses a centralized **NUT Master Server** (`14. | Role | Hostname / Identifier | Target IP | Subnet / VLAN | Description & Failure Protocol | |---|---|---|---|---| | **NUT Master Server** | `nutserver.snarfnet.net` | `14.4.4.44` | VLAN 50 (Smart Home) | Primary NUT `upsd` daemon polling physical UPS via USB/Serial/SNMP. Runs `upsmon master` and holds `/root/.ssh/id_ed25519_snarfnet`. | -| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.74` | VLAN 20 (Swarm 04) | NPM running in `PublishMode: host` on ports 80/443/81. Dependent on Synology NFS; stopped in Tier 2 prior to NAS Safe Mode. | +| **Edge Ingress / Reverse Proxy** | `nginx.snarfnet.net` | `140.44.4.70` | VLAN 20 (Keepalived VIP) | Dual-NPM active/standby running in `PublishMode: host` on ports 80/443/81 across Swarm 01 & 02. Dependent on Synology NFS; stopped in Tier 2 prior to NAS Safe Mode. | | **Authentication & SSO** | `authentik.snarfnet.net` | `140.44.4.29:444` | VLAN 20 (Aerodrome K3s) | Handles Forward Auth & Conditional MFA. Runs on Longhorn storage in Aerodrome. | | **UPS Web Dashboard** | `peanut.snarfnet.net` | `140.44.4.70:8050` | VLAN 20 (Swarm VIP) | Peanut container UI consuming `14.4.4.44:3493`. Protected by Authentik Forward Auth externally. | | **Emergency Push & Discord** | `gotify.snarfnet.net` | `140.44.4.70:8090` | VLAN 20 (Swarm VIP) | Gotify server + `gotify-discord-bridge`. **Internal outage alerts use `http://140.44.4.70:8090`** (+ direct Discord webhook fallback) to bypass Authentik during shutdown. | @@ -466,7 +466,7 @@ The power resilience architecture uses a centralized **NUT Master Server** (`14. 3. **Stage 3 (NUT Master):** `14.4.4.44` boots (`BIOS: Restore on AC Power Loss = Always On`), starts `upsd`, and sends `ONLINE` alert to Discord/Gotify. 4. **Stage 4 (Storage Gate — Must Complete Before Compute):** Synology NAS arrays (`14.10.10.71` & `14.10.10.72`) boot, mount Btrfs pools, and expose NFS (`:2049`). 5. **Stage 5 (Control Planes & Hypervisors):** Harvester (`140.44.4.81-83`), K3s Masters (`aerodrome` & `spaceport`), and Swarm Managers (`140.44.4.71-75`) boot and establish etcd/Raft quorum. -6. **Stage 6 (Compute Workers & Ingress):** K3s workers boot, Longhorn reattaches PVCs (`data-authentik-postgresql-0`, `awx-postgres`, `zabbix-postgresql`), and NPM on `snshpd4lnxswm04` (`140.44.4.74`) binds host ports `80/443/81`. +6. **Stage 6 (Compute Workers & Ingress):** K3s workers boot, Longhorn reattaches PVCs (`data-authentik-postgresql-0`, `awx-postgres`, `zabbix-postgresql`), and Dual-NPM on Swarm 01 & 02 (`140.44.4.71-72`) binds host ports `80/443/81` behind Keepalived VIP `140.44.4.70`. --- diff --git a/scripts/check_host_ports.py b/scripts/check_host_ports.py new file mode 100644 index 0000000..86f48b4 --- /dev/null +++ b/scripts/check_host_ports.py @@ -0,0 +1,21 @@ +import sys +import paramiko + +sys.stdout.reconfigure(encoding='utf-8') + +def main(): + for ip, node in [('140.44.4.71', 'snshpd4lnxswm01'), ('140.44.4.72', 'snshpd4lnxswm02')]: + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(ip, username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print(f"=== {node} ({ip}) ===") + stdin, stdout, stderr = ssh.exec_command('sudo ss -tulpn | grep -E ":80 |:81 |:443 " || true') + print("SS:\n", stdout.read().decode()) + stdin, stdout, stderr = ssh.exec_command('sudo docker ps -a --filter "name=nginx" --format "{{.ID}} {{.Status}} {{.Names}}"') + print("Containers:\n", stdout.read().decode()) + ssh.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/check_keepalived_all.py b/scripts/check_keepalived_all.py new file mode 100644 index 0000000..e5a6390 --- /dev/null +++ b/scripts/check_keepalived_all.py @@ -0,0 +1,35 @@ +import paramiko + +NODES = [ + ("swm01", "140.44.4.71"), + ("swm02", "140.44.4.72"), + ("swm03", "140.44.4.73"), + ("swm04", "140.44.4.74"), + ("swm05", "140.44.4.75"), +] + +def main(): + for name, ip in NODES: + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + ssh.connect(ip, username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=5) + stdin, stdout, stderr = ssh.exec_command("systemctl is-active keepalived 2>/dev/null || echo inactive") + status = stdout.read().decode().strip() + + stdin, stdout, stderr = ssh.exec_command("ip -4 addr show | grep '140.44.4.70' || true") + vip = stdout.read().decode().strip() + + stdin, stdout, stderr = ssh.exec_command("grep -E 'state|priority' /etc/keepalived/keepalived.conf 2>/dev/null || true") + conf = stdout.read().decode().strip() + + print(f"[{name} - {ip}] Keepalived: {status} | VIP: {'YES' if '140.44.4.70' in vip else 'NO'}") + if conf: + print(f" Config: {conf.splitlines()}") + ssh.close() + except Exception as e: + print(f"[{name} - {ip}] Error:", e) + +if __name__ == '__main__': + main() + diff --git a/scripts/check_npm_status.py b/scripts/check_npm_status.py new file mode 100644 index 0000000..c7881bd --- /dev/null +++ b/scripts/check_npm_status.py @@ -0,0 +1,27 @@ +import sys +import paramiko + +sys.stdout.reconfigure(encoding='utf-8') + +def main(): + for ip in ['140.44.4.71', '140.44.4.72']: + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(ip, username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + stdin, stdout, stderr = ssh.exec_command('sudo ss -tulpn | grep -E ":80 |:443 |:81 " || true') + print(f"=== LISTENING PORTS ON {ip} ===") + print(stdout.read().decode()) + + stdin, stdout, stderr = ssh.exec_command('sudo docker ps --filter "name=nginx-proxy-manager_app" --format "{{.ID}} {{.Status}}"') + cid = stdout.read().decode().strip() + print(f"Container on {ip}: {cid}") + if cid: + cid_clean = cid.split()[0] + stdin, stdout, stderr = ssh.exec_command(f'sudo docker logs {cid_clean} --tail 15') + print(f"Logs on {ip}:\n{stdout.read().decode()}") + + ssh.close() + +if __name__ == '__main__': + main() diff --git a/scripts/check_npm_volumes.py b/scripts/check_npm_volumes.py new file mode 100644 index 0000000..e72b1cb --- /dev/null +++ b/scripts/check_npm_volumes.py @@ -0,0 +1,16 @@ +import paramiko + +def main(): + for ip in ['140.44.4.71', '140.44.4.72']: + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(ip, username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + stdin, stdout, stderr = ssh.exec_command("sudo docker volume inspect nginx-proxy-manager_nginx-data --format '{{json .Options}}'") + print(f"Volume data on {ip}: {stdout.read().decode().strip()}") + stdin, stdout, stderr = ssh.exec_command("sudo docker volume inspect nginx-proxy-manager_nginx-letsencrypt --format '{{json .Options}}'") + print(f"Volume letsencrypt on {ip}: {stdout.read().decode().strip()}") + ssh.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/check_port_forwards.py b/scripts/check_port_forwards.py new file mode 100644 index 0000000..6cf21e1 --- /dev/null +++ b/scripts/check_port_forwards.py @@ -0,0 +1,16 @@ +import paramiko + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + ssh.connect('14.44.4.1', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=5) + stdin, stdout, stderr = ssh.exec_command('iptables -t nat -S | grep -E "80|443"') + print("IPTABLES NAT:\n", stdout.read().decode()) + ssh.close() + except Exception as e: + print("SSH to UDM Pro error:", e) + +if __name__ == '__main__': + main() + diff --git a/scripts/check_swarm_ports.py b/scripts/check_swarm_ports.py new file mode 100644 index 0000000..ddfbf2c --- /dev/null +++ b/scripts/check_swarm_ports.py @@ -0,0 +1,15 @@ +import paramiko + +def main(): + for ip in ['140.44.4.71', '140.44.4.72']: + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(ip, username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + stdin, stdout, stderr = ssh.exec_command('sudo ss -tulpn | grep -E ":80 |:443 |:81 " || true') + print(f"=== PORTS ON {ip} ===") + print(stdout.read().decode()) + ssh.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/check_swarm_service_status.py b/scripts/check_swarm_service_status.py new file mode 100644 index 0000000..e734b7c --- /dev/null +++ b/scripts/check_swarm_service_status.py @@ -0,0 +1,23 @@ +import sys +import paramiko + +sys.stdout.reconfigure(encoding='utf-8') + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print("=== DOCKER SERVICE LS ===") + stdin, stdout, stderr = ssh.exec_command('sudo docker service ls') + print(stdout.read().decode()) + + print("=== DOCKER SERVICE PS ===") + stdin, stdout, stderr = ssh.exec_command('sudo docker service ps nginx-proxy-manager_app') + print(stdout.read().decode()) + + ssh.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/deploy_dual_npm.py b/scripts/deploy_dual_npm.py new file mode 100644 index 0000000..7e7ea6b --- /dev/null +++ b/scripts/deploy_dual_npm.py @@ -0,0 +1,60 @@ +import paramiko +import time +import urllib.request +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print("=== STEP 1: Updating nginx-proxy-manager_app to Dual-Node Swarm Placement ===") + cmd = ( + "sudo docker service update " + "--constraint-rm 'node.hostname == snshpd4lnxswm04' " + "--constraint-add 'node.labels.npm == true' " + "--replicas 2 " + "--update-order start-first " + "nginx-proxy-manager_app" + ) + stdin, stdout, stderr = ssh.exec_command(cmd) + print("Service update output:\n", stdout.read().decode()) + print("Service update stderr:\n", stderr.read().decode()) + + print("=== STEP 2: Waiting for tasks to stabilize ===") + time.sleep(10) + + stdin, stdout, stderr = ssh.exec_command("sudo docker service ps nginx-proxy-manager_app --no-trunc") + print("Tasks:\n", stdout.read().decode()) + + ssh.close() + + print("=== STEP 3: Verifying HTTP / HTTPS endpoints ===") + for ip in ['140.44.4.70', '140.44.4.71', '140.44.4.72']: + # Port 80 + try: + req = urllib.request.Request(f"http://{ip}", headers={"Host": "test.snarfnet.net"}) + with urllib.request.urlopen(req, timeout=4) as r: + print(f"[{ip}:80] HTTP {r.status}") + except urllib.error.HTTPError as e: + print(f"[{ip}:80] HTTP {e.code} (NPM responding)") + except Exception as e: + print(f"[{ip}:80] Error: {e}") + + # Port 81 (Admin UI) + try: + req = urllib.request.Request(f"http://{ip}:81") + with urllib.request.urlopen(req, timeout=4) as r: + print(f"[{ip}:81] HTTP {r.status} (Admin UI responding)") + except urllib.error.HTTPError as e: + print(f"[{ip}:81] HTTP {e.code}") + except Exception as e: + print(f"[{ip}:81] Error: {e}") + +if __name__ == '__main__': + main() + diff --git a/scripts/inspect_container_nginx.py b/scripts/inspect_container_nginx.py new file mode 100644 index 0000000..a34d4b6 --- /dev/null +++ b/scripts/inspect_container_nginx.py @@ -0,0 +1,25 @@ +import paramiko + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + stdin, stdout, stderr = ssh.exec_command('sudo docker ps --filter "name=nginx-proxy-manager_app" -q') + cid = stdout.read().decode().strip() + print("Container ID:", cid) + + if cid: + stdin, stdout, stderr = ssh.exec_command(f'sudo docker exec {cid} nginx -t') + print("NGINX -t STDOUT:\n", stdout.read().decode()) + print("NGINX -t STDERR:\n", stderr.read().decode()) + + stdin, stdout, stderr = ssh.exec_command(f'sudo docker exec {cid} curl -I http://127.0.0.1:80') + print("CURL 80 INSIDE:\n", stdout.read().decode()) + print("CURL 80 STDERR:\n", stderr.read().decode()) + + ssh.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/inspect_keepalived_conf.py b/scripts/inspect_keepalived_conf.py new file mode 100644 index 0000000..31fa4de --- /dev/null +++ b/scripts/inspect_keepalived_conf.py @@ -0,0 +1,25 @@ +import sys +import paramiko + +sys.stdout.reconfigure(encoding='utf-8') + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print("=== /etc/keepalived/keepalived.conf ===") + stdin, stdout, stderr = ssh.exec_command('sudo cat /etc/keepalived/keepalived.conf') + print(stdout.read().decode()) + + print("=== /etc/keepalived/scripts/ ===") + stdin, stdout, stderr = ssh.exec_command('sudo ls -la /etc/keepalived/scripts/ || true') + print(stdout.read().decode()) + + stdin, stdout, stderr = ssh.exec_command('sudo cat /etc/keepalived/scripts/* || true') + print(stdout.read().decode()) + + ssh.close() + +if __name__ == '__main__': + main() diff --git a/scripts/inspect_npm_72.py b/scripts/inspect_npm_72.py new file mode 100644 index 0000000..7b6c316 --- /dev/null +++ b/scripts/inspect_npm_72.py @@ -0,0 +1,34 @@ +import sys +import paramiko + +sys.stdout.reconfigure(encoding='utf-8') + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect('140.44.4.72', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print("=== DOCKER EXEC PS INSIDE f026d1c24ef6 ===") + stdin, stdout, stderr = ssh.exec_command('sudo docker exec f026d1c24ef6 ps aux') + print(stdout.read().decode()) + print(stderr.read().decode()) + + print("=== REDIRECTION_HOST LS ===") + stdin, stdout, stderr = ssh.exec_command('sudo ls -la /var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/nginx/redirection_host || true') + print(stdout.read().decode()) + print(stderr.read().decode()) + + print("=== RECENT DOCKER LOGS WITH TIMESTAMPS ===") + stdin, stdout, stderr = ssh.exec_command('sudo docker logs f026d1c24ef6 --tail 30') + print(stdout.read().decode()) + + print("=== LOCAL CURL TEST ON 140.44.4.72 ===") + stdin, stdout, stderr = ssh.exec_command('curl -I http://127.0.0.1:80 ; curl -I http://127.0.0.1:81') + print(stdout.read().decode()) + print(stderr.read().decode()) + + ssh.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/inspect_swarm_nodes_labels.py b/scripts/inspect_swarm_nodes_labels.py new file mode 100644 index 0000000..0473295 --- /dev/null +++ b/scripts/inspect_swarm_nodes_labels.py @@ -0,0 +1,25 @@ +import sys +import paramiko + +sys.stdout.reconfigure(encoding='utf-8') + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + stdin, stdout, stderr = ssh.exec_command('sudo docker node ls --format "{{.Hostname}}"') + nodes = stdout.read().decode().split() + for n in nodes: + stdin, stdout, stderr = ssh.exec_command(f'sudo docker node inspect {n} --format "{{{{.Description.Hostname}}}}: {{{{.Spec.Labels}}}}"') + print(stdout.read().decode().strip()) + + print("\n=== DOCKER SERVICE PS ALL ===") + stdin, stdout, stderr = ssh.exec_command('sudo docker service ps nginx-proxy-manager_app --no-trunc') + print(stdout.read().decode()) + + ssh.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/pin_spaceport_vips.py b/scripts/pin_spaceport_vips.py index fb7920a..29e1a8a 100644 --- a/scripts/pin_spaceport_vips.py +++ b/scripts/pin_spaceport_vips.py @@ -51,3 +51,4 @@ def main(): if __name__ == '__main__': main() + diff --git a/scripts/rescale_npm.py b/scripts/rescale_npm.py new file mode 100644 index 0000000..09e2ab9 --- /dev/null +++ b/scripts/rescale_npm.py @@ -0,0 +1,30 @@ +import sys +import time +import paramiko + +sys.stdout.reconfigure(encoding='utf-8') + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print("=== SCALING TO 1 ===") + stdin, stdout, stderr = ssh.exec_command('sudo docker service scale nginx-proxy-manager_app=1') + print(stdout.read().decode()) + time.sleep(3) + + print("=== SCALING TO 2 ===") + stdin, stdout, stderr = ssh.exec_command('sudo docker service scale nginx-proxy-manager_app=2') + print(stdout.read().decode()) + time.sleep(5) + + print("=== SERVICE PS ===") + stdin, stdout, stderr = ssh.exec_command('sudo docker service ps nginx-proxy-manager_app --filter desired-state=running') + print(stdout.read().decode()) + + ssh.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/setup_dual_npm_cluster.py b/scripts/setup_dual_npm_cluster.py new file mode 100644 index 0000000..7ffe057 --- /dev/null +++ b/scripts/setup_dual_npm_cluster.py @@ -0,0 +1,24 @@ +import paramiko +import json +import time + +def main(): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print("=== STEP 1: Add node labels 'npm=true' to swm01 (.71) and swm02 (.72) ===") + stdin, stdout, stderr = ssh.exec_command('sudo docker node update --label-add npm=true snshpd4lnxswm01 && sudo docker node update --label-add npm=true snshpd4lnxswm02') + print("Labels update:", stdout.read().decode().strip(), stderr.read().decode().strip()) + + # Check node labels + stdin, stdout, stderr = ssh.exec_command("sudo docker node inspect snshpd4lnxswm01 --format '{{json .Spec.Labels}}'") + print("Labels swm01:", stdout.read().decode().strip()) + stdin, stdout, stderr = ssh.exec_command("sudo docker node inspect snshpd4lnxswm02 --format '{{json .Spec.Labels}}'") + print("Labels swm02:", stdout.read().decode().strip()) + + ssh.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/test_keepalived_failover.py b/scripts/test_keepalived_failover.py new file mode 100644 index 0000000..7ecd9e8 --- /dev/null +++ b/scripts/test_keepalived_failover.py @@ -0,0 +1,66 @@ +import sys +import time +import paramiko + +sys.stdout.reconfigure(encoding='utf-8') + +def check_vip(ssh): + stdin, stdout, stderr = ssh.exec_command("ip -4 addr show dev eth0 | grep '140.44.4.70' || true") + return bool(stdout.read().decode().strip()) + +def curl_vip(ssh): + stdin, stdout, stderr = ssh.exec_command("curl -s -o /dev/null -w '%{http_code}' -m 2 http://140.44.4.70:80 || true") + return stdout.read().decode().strip() + +def main(): + ssh1 = paramiko.SSHClient() + ssh1.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh1.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + ssh2 = paramiko.SSHClient() + ssh2.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh2.connect('140.44.4.72', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print("=== INITIAL STATE ===") + v1 = check_vip(ssh1) + v2 = check_vip(ssh2) + code = curl_vip(ssh1) + print(f"swm01 has VIP: {v1}, swm02 has VIP: {v2}, HTTP code on VIP: {code}") + + print("\n=== TRIGGERING FAILOVER: STOPPING KEEPALIVED ON swm01 ===") + ssh1.exec_command("sudo systemctl stop keepalived") + + # Wait up to 5 seconds for swm02 to claim VIP + claimed = False + for i in range(5): + time.sleep(1) + if check_vip(ssh2): + claimed = True + break + + v1 = check_vip(ssh1) + v2 = check_vip(ssh2) + code = curl_vip(ssh2) + print(f"Failover result -> swm01 has VIP: {v1}, swm02 has VIP: {v2}, HTTP code on VIP: {code}") + + print("\n=== RESTORING swm01: STARTING KEEPALIVED ON swm01 ===") + ssh1.exec_command("sudo systemctl start keepalived") + + # Wait up to 5 seconds for swm01 to reclaim VIP + reclaimed = False + for i in range(5): + time.sleep(1) + if check_vip(ssh1): + reclaimed = True + break + + v1 = check_vip(ssh1) + v2 = check_vip(ssh2) + code = curl_vip(ssh1) + print(f"Fallback result -> swm01 has VIP: {v1}, swm02 has VIP: {v2}, HTTP code on VIP: {code}") + + ssh1.close() + ssh2.close() + +if __name__ == '__main__': + main() diff --git a/scripts/test_npm_both_nodes.py b/scripts/test_npm_both_nodes.py new file mode 100644 index 0000000..5ba0f44 --- /dev/null +++ b/scripts/test_npm_both_nodes.py @@ -0,0 +1,26 @@ +import sys +import paramiko + +sys.stdout.reconfigure(encoding='utf-8') + +def main(): + for ip, node in [('140.44.4.71', 'snshpd4lnxswm01'), ('140.44.4.72', 'snshpd4lnxswm02')]: + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(ip, username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + print(f"=== TESTING {node} ({ip}) ===") + stdin, stdout, stderr = ssh.exec_command('sudo docker ps --filter "name=nginx-proxy-manager_app" --format "{{.ID}} {{.Status}} {{.Names}}"') + print("Containers:", stdout.read().decode().strip()) + + stdin, stdout, stderr = ssh.exec_command('curl -I http://127.0.0.1:80 ; curl -I http://127.0.0.1:81') + print("Curls:\n", stdout.read().decode()) + err = stderr.read().decode() + if err: + print("Errors:\n", err) + + ssh.close() + +if __name__ == '__main__': + main() + diff --git a/scripts/update_keepalived_healthcheck.py b/scripts/update_keepalived_healthcheck.py new file mode 100644 index 0000000..3e645fd --- /dev/null +++ b/scripts/update_keepalived_healthcheck.py @@ -0,0 +1,49 @@ +import sys +import paramiko + +sys.stdout.reconfigure(encoding='utf-8') + +NODES = [ + ("swm01", "140.44.4.71"), + ("swm02", "140.44.4.72"), + ("swm03", "140.44.4.73"), + ("swm04", "140.44.4.74"), + ("swm05", "140.44.4.75"), +] + +SCRIPT_CONTENT = """#!/bin/bash +# Check docker daemon health +systemctl is-active --quiet docker || exit 1 + +# If NPM container is running on this node, verify HTTP responsiveness +if docker ps --format '{{.Names}}' 2>/dev/null | grep -q "nginx-proxy-manager_app"; then + curl -sf -m 2 http://127.0.0.1:80 >/dev/null 2>&1 || curl -sf -m 2 http://127.0.0.1:81 >/dev/null 2>&1 || exit 1 +fi + +exit 0 +""" + +def main(): + for name, ip in NODES: + print(f"=== Updating {name} ({ip}) ===") + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(ip, username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + + # Write to temp file then sudo mv + sftp = ssh.open_sftp() + with sftp.open('/tmp/check_docker.sh', 'w') as f: + f.write(SCRIPT_CONTENT.replace('\r\n', '\n')) + sftp.close() + + cmd = "sudo mv /tmp/check_docker.sh /etc/keepalived/scripts/check_docker.sh && sudo chmod +x /etc/keepalived/scripts/check_docker.sh && sudo /etc/keepalived/scripts/check_docker.sh && echo 'TEST: OK'" + stdin, stdout, stderr = ssh.exec_command(cmd) + print(stdout.read().decode().strip()) + err = stderr.read().decode().strip() + if err: + print("ERR:", err) + + ssh.close() + +if __name__ == '__main__': + main()