diff --git a/README.md b/README.md deleted file mode 100644 index 424de2c..0000000 --- a/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# SnarfCode -# I added this line to test the syncing - diff --git a/Untitled-1.md b/Untitled-1.md deleted file mode 100644 index 4dd28ef..0000000 --- a/Untitled-1.md +++ /dev/null @@ -1,27 +0,0 @@ -Here is a concise, technical summary of your system architecture, the steps completed, and the current roadblock. You can copy and paste this text directly into another AI session to resume troubleshooting without losing context. ------------------------------- -## System Architecture & Context - -* Backend Server (Pi 5, 4GB RAM): Hosts the official Raspberry Pi AI HAT+ 2 (Hailo-10H NPU, 40 TOPS, 8GB onboard LPDDR4X VRAM). It is running a bare-metal, native C++ compiled framework (oatpp/1.4.0) provided by the Hailo GenAI Model Zoo. It currently has 5 hardware-quantized models loaded into its local store (including llama3.2:3b and deepseek_r1_distill_qwen:1.5b) listening on local IP 140.44.4.7:8000. -* Frontend Interface (Pi 4): Running Debian 13 (Trixie). To maximize performance and keep layers of abstraction at absolute zero, it hosts a bare-metal Python 3.11 environment running Open WebUI natively via pip (no Docker/containers). -* The Bridge: Because the Pi 5's oatpp C++ server doesn't match standard OpenAI/Ollama API schemas out-of-the-box, Open WebUI is connected to a local instance of Open WebUI Pipelines (uvicorn running on port 9099 with a security passkey string hailo-bridge). - ------------------------------- -## Work Completed So Far - - 1. Pi 4 Python Environment Fixes: Successfully compiled Python 3.11 from source on Debian Trixie. Handled and patched multiple modern Python deprecation roadblocks (injecting runtime compatibility hooks for collections.Mapping, collections.MutableMapping, and collections.Callable directly into Open WebUI and the Pipelines initialization scripts). - 2. Database Cleaning: Cleaned out stuck background network strings in Open WebUI’s native SQLite database file (/root/open-webui/venv/lib/python3.11/site-packages/open_webui/data/webui.db) to force it to drop hardcoded configurations and route connections entirely through the Pipeline bridge. - 3. Pipeline Live Status: Open WebUI is successfully communicating with the Pipelines bridge. Running a local curl call (curl -H "Authorization: Bearer hailo-bridge" http://localhost:9099/v1/models) returns a clean 200 OK JSON block, showing that Hailo NPU Core is fully mapped and available in the browser dropdown interface. - 4. Network Continuity: The Pi 4 successfully hits the Pi 5 hardware engine over the network. A GET call to http://140.44.4 safely returns a 200 OK, verifying that physical PCIe communication paths are clear. - ------------------------------- -## Remaining Issues & Current Roadblock -The exact API endpoint paths and JSON payload structures for Hailo's native C++ oatpp server are highly proprietary and poorly documented. Because of this, our custom python pipeline bridge (hailo_pipeline.py) is running into API request mismatches when passing chats over the network. - -* When the script targets http://140.44.4 or /chat/completions, the oatpp server returns a 404 Not Found. -* When the script targets http://140.44.4, the connection goes through, but the script throws a python KeyError: 'choices' or KeyError: 'message'. This indicates that the server processed the tensor inference successfully but returned its JSON text wrapper under an unknown key name (e.g., "response", "text", or "generated_text"). -* Attempts to manually probe the JSON output schema using a direct POST terminal command (curl -X POST http://140.44.4.7:8000 -d '{"model":"llama3.2:3b","messages":[...],"stream":false}') returned a 404 No mapping for HTTP-method: 'POST', URL: '/'. - -## What the Next LLM Needs to Solve: -We need the exact, structurally correct POST endpoint path and the corresponding JSON response key architecture for the Hailo GenAI Model Zoo C++ (oatpp) server so we can update the pipe() function inside hailo_pipeline.py to extract and stream the generated response strings back to Open WebUI. - diff --git a/argocd-traefik-fix.md b/argocd-traefik-fix.md deleted file mode 100644 index 0837de1..0000000 --- a/argocd-traefik-fix.md +++ /dev/null @@ -1,107 +0,0 @@ -# ArgoCD Ingress Fix - Traefik Bad Gateway - -## Environment - -- **Cluster**: RKE2 managed by Rancher -- **Ingress Controller**: Traefik (kube-system namespace) -- **ArgoCD Version**: v3.4.2 (Helm chart argo-cd-9.5.14) -- **Namespace**: infrastructure -- **Hostname**: argo.snarfnet.net - -## Problem - -After deploying ArgoCD, accessing `https://argo.snarfnet.net` returned a **502 Bad Gateway** from Traefik. - -## Root Cause - -Two issues were identified: - -### 1. Service TargetPort Mismatch - -The ArgoCD server was listening on port **8080**, but the Kubernetes service had `targetPort: 8081`. This was corrected by patching the service to point both ports (80 and 443) to targetPort 8080. - -### 2. Traefik Protocol Mismatch (Primary Issue) - -The ArgoCD service defined two ports: - -```yaml -ports: - - name: http - port: 80 - targetPort: 8080 - - name: https - port: 443 - targetPort: 8080 -``` - -The Ingress resource routed traffic to port 80, but Traefik's Kubernetes provider saw the port named `https` (443) on the service and automatically selected it, connecting to the backend using **HTTPS**: - -``` -"servers":[{"url":"https://10.42.1.76:8080"}] -``` - -However, ArgoCD was configured to run in insecure mode (`server.insecure: true`), meaning it only served plain **HTTP** on port 8080. Traefik's HTTPS connection to an HTTP backend resulted in the Bad Gateway. - -Working services (Gitea, Jenkins, etc.) did not have this problem because they only exposed a single HTTP port with no `https` named port to confuse Traefik. - -## Fix - -Removed the `https` (port 443) entry from the `argocd-server` service, leaving only the HTTP port: - -```yaml -spec: - ports: - - name: http - port: 80 - targetPort: 8080 -``` - -This forced Traefik to use `http://` when connecting to the backend, which matched ArgoCD's insecure mode. - -After the change, Traefik's internal service config showed: - -``` -"servers":[{"url":"http://10.42.1.76:8080"}] -``` - -## Permanent Fix for Helm Upgrades - -To prevent the Helm chart from recreating the 443 port on future upgrades, use one of these approaches: - -### Option A: Annotate the Ingress - -Add this annotation to the `argo-ing` Ingress resource so Traefik always uses HTTP regardless of service port names: - -```yaml -metadata: - annotations: - traefik.ingress.kubernetes.io/service.serversscheme: http -``` - -### Option B: Helm Values - -Configure the chart to not expose the HTTPS service port (check chart documentation for exact key, as it varies by version): - -```yaml -configs: - params: - server.insecure: true - -server: - service: - type: ClusterIP -``` - -## Debugging Steps That Led to the Fix - -1. Verified the pod was running and healthy (`1/1 Ready`) -2. Confirmed the pod was listening on port 8080 via `/proc/net/tcp6` -3. Tested direct pod connectivity from another pod in the cluster — returned HTTP 200 -4. Queried Traefik's internal API at `http://127.0.0.1:9000/api/http/services` -5. Discovered Traefik was using `https://` to connect to the backend -6. Compared with working services (Gitea, Jenkins) which all used `http://` -7. Identified the `https` named port on the service as the cause - -## Key Takeaway - -Traefik's Kubernetes Ingress provider infers the backend protocol from the service port name. A port named `https` causes Traefik to connect using HTTPS, regardless of what port number the Ingress backend specifies. When running ArgoCD in insecure mode behind a TLS-terminating reverse proxy, ensure the service does not expose an `https` named port, or use the `traefik.ingress.kubernetes.io/service.serversscheme` annotation to override the behavior. diff --git a/homelab-gitops-repo b/homelab-gitops-repo index 5169a02..50e49a9 160000 --- a/homelab-gitops-repo +++ b/homelab-gitops-repo @@ -1 +1 @@ -Subproject commit 5169a02440e7672b42c7b8efea4aeca92e4d56bf +Subproject commit 50e49a926416fa11bafa8e676408686c6a03fbcf diff --git a/scripts/add_kuma_spacedock.py b/scripts/add_kuma_spacedock.py new file mode 100644 index 0000000..9a0e28d --- /dev/null +++ b/scripts/add_kuma_spacedock.py @@ -0,0 +1,62 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run_sql(pod, sql): + cmd = f"kubectl exec -i -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root" + stdin, stdout, stderr = c.exec_command(cmd) + stdin.write(sql) + stdin.close() + out = stdout.read().decode().strip() + err = stderr.read().decode().strip() + return out, err + +_, o, _ = c.exec_command("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'") +pod = o.read().decode().strip() +print("Kuma Pod:", pod) + +monitors = [ + ("Spacedock Harvester Node 01 (140.44.4.81)", "140.44.4.81"), + ("Spacedock Harvester Node 02 (140.44.4.82)", "140.44.4.82"), + ("Spacedock Harvester Node 03 (140.44.4.83)", "140.44.4.83"), + ("Spacedock Harvester Node 04 (140.44.4.84)", "140.44.4.84"), + ("Dell PowerEdge R730 Node 1 iDRAC (140.44.44.121)", "140.44.44.121"), + ("Dell PowerEdge R730 Node 2 iDRAC (140.44.44.122)", "140.44.44.122"), + ("Dell PowerEdge R730 Node 3 iDRAC (140.44.44.123)", "140.44.44.123"), + ("Dell PowerEdge R730 Node 4 iDRAC (140.44.44.124)", "140.44.44.124") +] + +for name, host in monitors: + out, err = run_sql(pod, f"USE kuma; SELECT id FROM monitor WHERE hostname='{host}' AND type='ping';") + if "id" in out and len(out.splitlines()) > 1: + mon_id = out.splitlines()[1].strip() + print(f"Monitor already exists: {name} (ID: {mon_id})") + else: + insert_sql = f"""USE kuma; +INSERT INTO monitor (name, active, user_id, `interval`, type, weight, hostname, maxretries, retry_interval) +VALUES ('{name}', 1, 1, 60, 'ping', 2000, '{host}', 1, 60); +SELECT LAST_INSERT_ID() AS id; +""" + out, err = run_sql(pod, insert_sql) + if err: + print(f"Error inserting {name}: {err}") + continue + lines = out.splitlines() + mon_id = lines[-1].strip() + print(f"Created monitor: {name} (ID: {mon_id})") + + # Link notifications (1 = Discord, 2 = Gotify) + for notif_id in [1, 2]: + link_check = f"USE kuma; SELECT id FROM monitor_notification WHERE monitor_id={mon_id} AND notification_id={notif_id};" + out_link, _ = run_sql(pod, link_check) + if "id" not in out_link or len(out_link.splitlines()) <= 1: + run_sql(pod, f"USE kuma; INSERT INTO monitor_notification (monitor_id, notification_id) VALUES ({mon_id}, {notif_id});") + print(f" Linked monitor {mon_id} to notification {notif_id}") + else: + print(f" Monitor {mon_id} already linked to notification {notif_id}") + +c.close() diff --git a/scripts/add_npm_games.py b/scripts/add_npm_games.py new file mode 100644 index 0000000..c048e3b --- /dev/null +++ b/scripts/add_npm_games.py @@ -0,0 +1,173 @@ +import paramiko +import sys +import datetime +import json + +sys.stdout.reconfigure(encoding='utf-8') +key_path = r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet' + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10) + +def main(): + remote_script = """import sqlite3, datetime, json + +db_path = '/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite' +conn = sqlite3.connect(db_path) +cur = conn.cursor() + +# Check existing games host +cur.execute("SELECT id, domain_names, forward_host, forward_port FROM proxy_host WHERE domain_names LIKE '%games.snarfnet.net%'") +existing = cur.fetchone() + +now = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S") + +if existing: + pid = existing[0] + print(f"Updating existing proxy_host ID {pid}...") + cur.execute('''UPDATE proxy_host + SET forward_host = '140.44.44.80', forward_port = 8080, forward_scheme = 'http', + certificate_id = 1, ssl_forced = 1, hsts_enabled = 1, http2_support = 1, + block_exploits = 1, allow_websocket_upgrade = 1, enabled = 1, modified_on = ? + WHERE id = ?''', (now, pid)) +else: + cur.execute("SELECT MAX(id) FROM proxy_host") + max_id = cur.fetchone()[0] or 0 + pid = max_id + 1 + print(f"Creating new proxy_host with ID {pid}...") + cur.execute('''INSERT INTO proxy_host ( + id, created_on, modified_on, owner_user_id, domain_names, + forward_host, forward_port, forward_scheme, certificate_id, + ssl_forced, hsts_enabled, hsts_subdomains, http2_support, + block_exploits, caching_enabled, allow_websocket_upgrade, + access_list_id, advanced_config, enabled, meta + ) VALUES ( + ?, ?, ?, 1, '["games.snarfnet.net"]', + '140.44.44.80', 8080, 'http', 1, + 1, 1, 0, 1, + 1, 0, 1, + 0, '', 1, '{}' + )''', (pid, now, now)) + +conn.commit() +conn.close() +print(f"SUCCESS: pid={pid}") +""" + sftp = c.open_sftp() + with sftp.open("/tmp/add_games_proxy.py", "w") as f: + f.write(remote_script) + sftp.close() + + stdin, stdout, stderr = c.exec_command("python3 /tmp/add_games_proxy.py") + out = stdout.read().decode().strip() + err = stderr.read().decode().strip() + print("DB Result:\n", out) + if err: + print("DB Err:\n", err) + + # Extract pid + pid = None + for line in out.splitlines(): + if "pid=" in line: + pid = line.split("pid=")[1].strip() + + if not pid: + print("Failed to get pid!") + return + + # Write nginx config + nginx_conf = f"""# ------------------------------------------------------------ +# games.snarfnet.net +# ------------------------------------------------------------ + +map $scheme $hsts_header {{ + https "max-age=63072000; preload"; +}} + +server {{ + set $forward_scheme http; + set $server "140.44.44.80"; + set $port 8080; + + listen 80; + listen [::]:80; + + listen 443 ssl; + listen [::]:443 ssl; + + server_name games.snarfnet.net; + http2 on; + + # Let's Encrypt SSL (*.snarfnet.net is cert 1) + include conf.d/include/letsencrypt-acme-challenge.conf; + include conf.d/include/ssl-cache.conf; + include conf.d/include/ssl-ciphers.conf; + ssl_certificate /etc/letsencrypt/live/npm-1/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/npm-1/privkey.pem; + + # Asset Caching + include conf.d/include/assets.conf; + + # Block Exploits + include conf.d/include/block-exploits.conf; + + # Force SSL + set $trust_forwarded_proto "F"; + include conf.d/include/force-ssl.conf; + + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $http_connection; + proxy_http_version 1.1; + + access_log /data/logs/proxy-host-{pid}_access.log proxy; + error_log /data/logs/proxy-host-{pid}_error.log warn; + + location / {{ + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $http_connection; + proxy_http_version 1.1; + + # Proxy! + include conf.d/include/proxy.conf; + }} + + # Custom + include /data/nginx/custom/server_proxy[.]conf; +}} +""" + conf_path = f"/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/nginx/proxy_host/{pid}.conf" + sftp = c.open_sftp() + with sftp.open(conf_path, "w") as f: + f.write(nginx_conf) + sftp.close() + print(f"Wrote {conf_path}") + + # Test Nginx + stdin, stdout, stderr = c.exec_command("docker exec $(docker ps -q -f name=nginx-proxy-manager_app | head -n1) nginx -t") + t_out = stdout.read().decode() + t_err = stderr.read().decode() + print("nginx -t stdout:\n", t_out) + print("nginx -t stderr:\n", t_err) + + if "successful" in t_out or "successful" in t_err: + print("Reloading Nginx in container...") + c.exec_command("docker exec $(docker ps -q -f name=nginx-proxy-manager_app | head -n1) nginx -s reload") + # Also check other swarm node if running dual + c2 = paramiko.SSHClient() + c2.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + c2.connect("140.44.4.72", username="root", key_filename=key_path, timeout=5) + c2.exec_command("docker exec $(docker ps -q -f name=nginx-proxy-manager_app | head -n1) nginx -s reload") + c2.close() + print("Reloaded Swarm 02 NPM as well!") + except Exception as e: + print("Swarm 02 reload skipped/note:", e) + print("Proxy setup complete!") + else: + print("ERROR: nginx -t failed!") + + c.close() + +if __name__ == "__main__": + main() diff --git a/scripts/adjust_zabbix_thresholds.py b/scripts/adjust_zabbix_thresholds.py new file mode 100644 index 0000000..cc96fcc --- /dev/null +++ b/scripts/adjust_zabbix_thresholds.py @@ -0,0 +1,67 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1} + if auth: payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + +auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result") +print("Zabbix Auth OK:", bool(auth)) + +# 1. Update Template Macros for 10001 (Agent passive), 10343 (Agent active), 10285 (Prom) +# Find hostmacroids for these templates +template_ids = ["10001", "10343", "10285"] +for tid in template_ids: + macros = zabbix_api("usermacro.get", { + "output": ["hostmacroid", "macro", "value"], + "hostids": [tid] + }, auth=auth).get("result", []) + for m in macros: + mid = m["hostmacroid"] + macro_name = m["macro"] + if macro_name == "{$CPU.UTIL.CRIT}": + res = zabbix_api("usermacro.update", {"hostmacroid": mid, "value": "95"}, auth=auth) + print(f"Template {tid} macro {macro_name} updated to 95: {res.get('result')}") + elif macro_name == "{$MEMORY.UTIL.MAX}": + res = zabbix_api("usermacro.update", {"hostmacroid": mid, "value": "95"}, auth=auth) + print(f"Template {tid} macro {macro_name} updated to 95: {res.get('result')}") + elif macro_name == "{$LOAD_AVG_PER_CPU.MAX.WARN}": + res = zabbix_api("usermacro.update", {"hostmacroid": mid, "value": "2.5"}, auth=auth) + print(f"Template {tid} macro {macro_name} updated to 2.5: {res.get('result')}") + +# 2. Update trigger expressions from 5m to 15m +# Triggers: 22370 (Passive CPU), 22376 (Passive Mem), 22415 (Active CPU), 22421 (Active Mem) +# In Zabbix 6.0/7.0, expression format is min(//,15m)>{$MACRO} +# Let's inspect the exact expression syntax for trigger update +for tr_id in ["22370", "22376", "22415", "22421"]: + tr = zabbix_api("trigger.get", { + "output": ["triggerid", "description", "expression"], + "triggerids": [tr_id] + }, auth=auth).get("result", [{}])[0] + expr = tr.get("expression", "") + print(f"\nTrigger {tr_id} ({tr.get('description')}): current expr = {expr}") + # Replace 5m with 15m in the expression + if "5m" in expr: + new_expr = expr.replace("5m", "15m") + # In trigger.update, Zabbix expects expression with host/item syntax if updated + # Or description update: "Linux: High CPU utilization (>95% for 15m)" + desc = tr.get("description", "") + # Update trigger + up_res = zabbix_api("trigger.update", { + "triggerid": tr_id, + "expression": new_expr + }, auth=auth) + print(f" Updated expr to 15m: {up_res}") + +print("\nFinished updating Zabbix noise reduction thresholds!") + diff --git a/scripts/apply_clean_cm.py b/scripts/apply_clean_cm.py new file mode 100644 index 0000000..93164d6 --- /dev/null +++ b/scripts/apply_clean_cm.py @@ -0,0 +1,37 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +with open(r'c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\aerodrome\prometheus\prometheus.yaml', 'r', encoding='utf-8') as f: + text = f.read() + +# Extract just the ConfigMap section +# It starts with "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: prometheus-config" and ends with "---" +parts = text.split("---") +cm_part = None +for p in parts: + if "name: prometheus-config" in p: + cm_part = p + break + +if cm_part: + print("Found prometheus-config ConfigMap! Size:", len(cm_part)) + stdin, stdout, stderr = c.exec_command("cat > /tmp/prom_cm_clean.yaml") + stdin.write(cm_part) + stdin.close() + stdout.channel.recv_exit_status() + + _, o, e = c.exec_command("kubectl apply -f /tmp/prom_cm_clean.yaml") + print("Apply result:\n", o.read().decode().strip()) + err = e.read().decode().strip() + if err: + print("Apply err:\n", err) + + # Reload prometheus via lifecycle endpoint + _, o, _ = c.exec_command("kubectl exec deploy/prometheus -- wget -qO- --post-action='' http://localhost:9090/-/reload") + print("Reload endpoint output:", o.read().decode().strip()) + +c.close() + diff --git a/scripts/apply_kuma_tags.py b/scripts/apply_kuma_tags.py new file mode 100644 index 0000000..f79e496 --- /dev/null +++ b/scripts/apply_kuma_tags.py @@ -0,0 +1,79 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run_sql(sql): + pod_cmd = "kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'" + _, o, _ = c.exec_command(pod_cmd) + pod = o.read().decode().strip() + cmd = f"kubectl exec -i -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root" + stdin, stdout, stderr = c.exec_command(cmd) + stdin.write(sql) + stdin.close() + return stdout.read().decode().strip(), stderr.read().decode().strip() + +# Target monitors: +# 98: Spacedock Harvester Node 01 +# 99: Spacedock Harvester Node 02 +# 100: Spacedock Harvester Node 03 +# 101: Spacedock Harvester Node 04 +# 102: Dell PowerEdge R730 Node 1 iDRAC +# 103: Dell PowerEdge R730 Node 2 iDRAC +# 104: Dell PowerEdge R730 Node 3 iDRAC +# 105: Dell PowerEdge R730 Node 4 iDRAC + +# Tags to attach: +# 5: ai +# 9: IP +# 11: Tier-2-Operational + +monitors = [98, 99, 100, 101, 102, 103, 104, 105] +tag_ids = [5, 9, 11] + +sql_commands = ["USE kuma;"] + +for m_id in monitors: + for t_id in tag_ids: + # Check if already exists + check = f"SELECT id FROM monitor_tag WHERE monitor_id={m_id} AND tag_id={t_id};" + sql_commands.append(f""" + INSERT INTO monitor_tag (monitor_id, tag_id, value) + SELECT {m_id}, {t_id}, NULL + WHERE NOT EXISTS (SELECT 1 FROM monitor_tag WHERE monitor_id={m_id} AND tag_id={t_id}); + """) + +# Also ensure Harvester Console (57) and Harvester HCI Console VIP (94) have ai (5) +for m_id in [57, 94]: + sql_commands.append(f""" + INSERT INTO monitor_tag (monitor_id, tag_id, value) + SELECT {m_id}, 5, NULL + WHERE NOT EXISTS (SELECT 1 FROM monitor_tag WHERE monitor_id={m_id} AND tag_id=5); + """) + +full_sql = "\n".join(sql_commands) +out, err = run_sql(full_sql) +if err: + print("SQL Error:", err) +else: + print("Tags applied successfully!") + +# Verification query +verify_sql = """ +USE kuma; +SELECT m.id, m.name, GROUP_CONCAT(t.name ORDER BY t.id SEPARATOR ', ') AS tags +FROM monitor m +JOIN monitor_tag mt ON m.id = mt.monitor_id +JOIN tag t ON mt.tag_id = t.id +WHERE m.id IN (57, 94, 98, 99, 100, 101, 102, 103, 104, 105) +GROUP BY m.id, m.name +ORDER BY m.id; +""" +res, _ = run_sql(verify_sql) +print("\nUpdated Monitors and Tags:\n", res) + +c.close() + diff --git a/scripts/apply_prom_cm_only.py b/scripts/apply_prom_cm_only.py new file mode 100644 index 0000000..f1aeff5 --- /dev/null +++ b/scripts/apply_prom_cm_only.py @@ -0,0 +1,38 @@ +import paramiko +import yaml + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +with open(r'c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\aerodrome\prometheus\prometheus.yaml', 'r') as f: + docs = list(yaml.safe_load_all(f)) + +# Find ConfigMap prometheus-config +cm_doc = None +for d in docs: + if d and d.get('kind') == 'ConfigMap' and d.get('metadata', {}).get('name') == 'prometheus-config': + cm_doc = d + break + +if cm_doc: + cm_yaml = yaml.dump(cm_doc) + stdin, stdout, stderr = c.exec_command("cat > /tmp/prometheus_cm.yaml") + stdin.write(cm_yaml) + stdin.close() + stdout.channel.recv_exit_status() + + _, o, e = c.exec_command("kubectl apply -f /tmp/prometheus_cm.yaml") + print("CM apply:\n", o.read().decode().strip()) + err = e.read().decode().strip() + if err: + print("CM err:", err) + + # Restart prometheus pod + _, o, _ = c.exec_command("kubectl rollout restart deployment/prometheus") + print("Prometheus restart:", o.read().decode().strip()) + _, o, _ = c.exec_command("kubectl rollout status deployment/prometheus --timeout=60s") + print("Rollout status:", o.read().decode().strip()) + +c.close() + diff --git a/scripts/apply_prom_shipyard.py b/scripts/apply_prom_shipyard.py new file mode 100644 index 0000000..771105d --- /dev/null +++ b/scripts/apply_prom_shipyard.py @@ -0,0 +1,36 @@ +import paramiko +import time + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +with open(r'c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\aerodrome\prometheus\prometheus.yaml', 'r') as f: + prom_yaml = f.read() + +stdin, stdout, stderr = c.exec_command("cat > /tmp/prometheus.yaml") +stdin.write(prom_yaml) +stdin.close() +stdout.channel.recv_exit_status() + +_, o, e = c.exec_command("kubectl apply -f /tmp/prometheus.yaml") +print("Kubectl apply:\n", o.read().decode().strip()) +err = e.read().decode().strip() +if err: + print("Err:", err) + +# Find prometheus pod +_, o, _ = c.exec_command("kubectl get pod -l app.kubernetes.io/name=prometheus -o jsonpath='{.items[0].metadata.name}'") +pod = o.read().decode().strip() +print("Prometheus pod:", pod) + +# Reload config +_, o, _ = c.exec_command(f"kubectl exec {pod} -- wget -qO- --post-action='' http://localhost:9090/-/reload || kubectl exec {pod} -- curl -X POST http://localhost:9090/-/reload") +print("Reload result:", o.read().decode().strip()) + +time.sleep(5) +# Check targets for shipyard +_, o, _ = c.exec_command(f"""kubectl exec {pod} -- wget -qO- 'http://localhost:9090/api/v1/targets' | grep -o '"cluster":"shipyard"[^}}]*' | head -10""") +print("Shipyard targets found in Prometheus:\n", o.read().decode().strip()) + +c.close() diff --git a/scripts/audit_spacedock_failure.py b/scripts/audit_spacedock_failure.py new file mode 100644 index 0000000..0bd2b1d --- /dev/null +++ b/scripts/audit_spacedock_failure.py @@ -0,0 +1,84 @@ +import paramiko +import json +import time + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +HARVESTER_VIP = "140.44.4.90" + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + +try: + print(f"Connecting to Harvester VIP {HARVESTER_VIP}...") + c.connect(HARVESTER_VIP, username="rancher", key_filename=SSH_KEY, timeout=10) + print("Connected to Harvester VIP!") + + def run_k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + stdin, stdout, stderr = c.exec_command(full_cmd) + out = stdout.read().decode().strip() + err = stderr.read().decode().strip() + return out + + print("\n=== 1. HARVESTER CLUSTER NODES ===") + print(run_k8s("kubectl get nodes -o wide")) + + print("\n=== 2. NODE UPTIME ===") + print(run_k8s("uptime")) + + print("\n=== 3. VIRTUAL MACHINES (KubeVirt) ===") + print(run_k8s("kubectl get vms,vmis -A -o wide")) + + print("\n=== 4. HARVESTER SYSTEM POD STATUS (Not Running / Degraded) ===") + print(run_k8s("kubectl get pods -A | grep -v 'Running\\|Completed'")) + + print("\n=== 5. LONGHORN STORAGE VOLUME STATUS ===") + print(run_k8s("kubectl get volumes.longhorn.io -n longhorn-system")) + + c.close() +except Exception as e: + print(f"Error connecting to Harvester: {e}") + +# Check iDRACs +idracs = { + "Spacedock Node 1": "140.44.44.121", + "Spacedock Node 2": "140.44.44.122", + "Spacedock Node 3": "140.44.44.123", + "Spacedock Node 4": "140.44.44.124", +} + +print("\n=== 6. IDRAC SYSTEM EVENT LOGS & POWER STATUS ===") +for name, ip in idracs.items(): + print(f"\n--- {name} ({ip}) ---") + try: + ic = paramiko.SSHClient() + ic.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ic.connect(ip, username="root", password="Katzir476!", timeout=8) + chan = ic.invoke_shell() + time.sleep(1.5) + if chan.recv_ready(): + chan.recv(4096) + + def racadm(cmd): + chan.send(cmd + "\r\n") + time.sleep(2) + out = "" + while chan.recv_ready(): + out += chan.recv(8192).decode('utf-8', errors='ignore') + time.sleep(0.5) + return out + + pwr = racadm("racadm serveraction powerstatus") + for line in pwr.splitlines(): + if "power" in line.lower() or "server" in line.lower(): + print(line.strip()) + + sel = racadm("racadm getsel -s 10") + print("Last 10 SEL Events:") + for line in sel.splitlines()[-12:]: + print(" ", line.strip()) + + ic.close() + except Exception as e: + print(f"Error querying {name}: {e}") + diff --git a/scripts/build_and_upload_dashboards.py b/scripts/build_and_upload_dashboards.py new file mode 100644 index 0000000..437e766 --- /dev/null +++ b/scripts/build_and_upload_dashboards.py @@ -0,0 +1,467 @@ +import json +import requests +import copy +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +base_url = "https://grafana.snarfnet.net" + +# Load templates +with open("homelab-gitops-repo/dashboards/grafana/systems-dashboard.json") as f: + sys_template = json.load(f) + +with open("homelab-gitops-repo/dashboards/grafana/container-metrics-dashboard.json") as f: + cnt_template = json.load(f) + +def build_cluster_dashboard(cluster_key, cluster_title, folder_id, folder_uid, cluster_emoji): + dash = { + "annotations": {"list": []}, + "editable": True, + "fiscalYearStartMonth": 0, + "graphTooltip": 1, + "id": None, + "links": [], + "liveNow": False, + "refresh": "15s", + "schemaVersion": 39, + "tags": [cluster_key, "resource-monitor", "infrastructure"], + "time": {"from": "now-1h", "to": "now"}, + "timepicker": {"refresh_intervals": ["5s", "10s", "15s", "30s", "1m", "5m"]}, + "timezone": "browser", + "title": f"{cluster_emoji} {cluster_title} Resource Monitor", + "uid": f"{cluster_key}-resource-monitor", + "version": 1, + "panels": [] + } + + panels = [] + current_y = 0 + + # ========================================== + # GROUP 1: Systems & Cluster Host Performance + # ========================================== + # Elevate Systems Row Header + panels.append({ + "collapsed": False, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": current_y}, + "id": 100, + "title": f"🖥️ Systems & Host Performance — {cluster_title} Cluster Fleet", + "type": "row" + }) + current_y += 1 + + # Systems Stat Panels (height 4) + stat_panels = [ + { + "id": 1, + "title": f"Online Nodes ({cluster_title})", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'count(node_load1{{cluster="{cluster_key}"}})', "legendFormat": "Nodes", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": None}]}, "unit": "none"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + }, + { + "id": 2, + "title": "Avg Cluster CPU Load (1m)", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 6, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'avg(node_load1{{cluster="{cluster_key}"}})', "legendFormat": "Load", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "blue", "value": None}, {"color": "orange", "value": 2.0}, {"color": "red", "value": 4.0}]}, "unit": "none"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + }, + { + "id": 3, + "title": "Total Monitored RAM", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 12, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'sum(node_memory_MemTotal_bytes{{cluster="{cluster_key}"}})', "legendFormat": "RAM", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "purple", "value": None}]}, "unit": "bytes"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + }, + { + "id": 4, + "title": "Cluster RAM In-Use %", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 18, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'(1 - (sum(node_memory_MemAvailable_bytes{{cluster="{cluster_key}"}}) / sum(node_memory_MemTotal_bytes{{cluster="{cluster_key}"}}))) * 100', "legendFormat": "In-Use", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": None}, {"color": "orange", "value": 75}, {"color": "red", "value": 90}]}, "unit": "percent"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + } + ] + panels.extend(stat_panels) + current_y += 4 + + # Systems CPU Timeseries + panels.append({ + "id": 5, + "title": "CPU Utilization by Host (%)", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'100 - (avg by (instance) (rate(node_cpu_seconds_total{{mode="idle", cluster="{cluster_key}"}}[2m])) * 100)', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "percent", "min": 0, "max": 100}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}} + }) + panels.append({ + "id": 6, + "title": "1-Minute Load Average by Host", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'node_load1{{cluster="{cluster_key}"}}', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 10}, "unit": "none", "min": 0}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}} + }) + current_y += 8 + + # Systems Memory Timeseries + panels.append({ + "id": 7, + "title": "Memory Used % by Host", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'(1 - (node_memory_MemAvailable_bytes{{cluster="{cluster_key}"}} / node_memory_MemTotal_bytes{{cluster="{cluster_key}"}})) * 100', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "percent", "min": 0, "max": 100}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}} + }) + panels.append({ + "id": 8, + "title": "Available RAM by Host (GiB)", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'node_memory_MemAvailable_bytes{{cluster="{cluster_key}"}}', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 10}, "unit": "bytes"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "min"]}} + }) + current_y += 8 + + # Filesystem & Disk I/O + panels.append({ + "id": 9, + "title": "Root (/) Filesystem Usage %", + "type": "bargauge", + "gridPos": {"h": 8, "w": 8, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'(1 - (node_filesystem_avail_bytes{{mountpoint="/", cluster="{cluster_key}"}} / node_filesystem_size_bytes{{mountpoint="/", cluster="{cluster_key}"}})) * 100', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": None}, {"color": "orange", "value": 75}, {"color": "red", "value": 90}]}, "unit": "percent", "min": 0, "max": 100}}, + "options": {"displayMode": "gradient", "orientation": "horizontal", "showUnfilled": True} + }) + panels.append({ + "id": 10, + "title": "Disk Read Throughput", + "type": "timeseries", + "gridPos": {"h": 8, "w": 8, "x": 8, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'sum by (instance) (rate(node_disk_read_bytes_total{{cluster="{cluster_key}"}}[2m]))', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 10}, "unit": "Bps"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}} + }) + panels.append({ + "id": 11, + "title": "Disk Write Throughput", + "type": "timeseries", + "gridPos": {"h": 8, "w": 8, "x": 16, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'sum by (instance) (rate(node_disk_written_bytes_total{{cluster="{cluster_key}"}}[2m]))', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 10}, "unit": "Bps"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}} + }) + current_y += 8 + + # Network RX/TX + panels.append({ + "id": 12, + "title": "Host Network Receive (RX)", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'sum by (instance) (rate(node_network_receive_bytes_total{{device!~"lo|veth.*|flannel.*|cali.*", cluster="{cluster_key}"}}[2m]))', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}} + }) + panels.append({ + "id": 13, + "title": "Host Network Transmit (TX)", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'sum by (instance) (rate(node_network_transmit_bytes_total{{device!~"lo|veth.*|flannel.*|cali.*", cluster="{cluster_key}"}}[2m]))', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}} + }) + current_y += 8 + + # ========================================== + # GROUP 2: Containers & Workloads + # ========================================== + if cluster_key in ["aerodrome", "spaceport"]: + cadvisor_filter = f'cluster="{cluster_key}"' if cluster_key == "spaceport" else 'instance=~"sninfr2.*"' + + panels.append({ + "collapsed": False, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": current_y}, + "id": 200, + "title": f"📦 Kubernetes Containers & Workloads — {cluster_title} Cluster", + "type": "row" + }) + current_y += 1 + + # Container Stats + panels.extend([ + { + "id": 14, + "title": "Monitored Containers", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'count(container_last_seen{{container!="", {cadvisor_filter}}})', "legendFormat": "Containers", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": None}]}, "unit": "none"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + }, + { + "id": 15, + "title": "Active Kubernetes Pods", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 6, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'count(count by (pod, namespace) (container_last_seen{{container!="", {cadvisor_filter}}}))', "legendFormat": "Pods", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "blue", "value": None}]}, "unit": "none"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + }, + { + "id": 16, + "title": "Total Container CPU Load", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 12, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'sum(rate(container_cpu_usage_seconds_total{{container!="", {cadvisor_filter}}}[2m]))', "legendFormat": "Cores", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "orange", "value": None}]}, "unit": "none"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + }, + { + "id": 17, + "title": "Total Container Working RAM", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 18, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'sum(container_memory_working_set_bytes{{container!="", {cadvisor_filter}}})', "legendFormat": "RAM", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "purple", "value": None}]}, "unit": "bytes"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + } + ]) + current_y += 4 + + # Container CPU + panels.append({ + "id": 18, + "title": "Top 10 CPU Consuming Pods (Cores)", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'topk(10, sum by (pod, namespace) (rate(container_cpu_usage_seconds_total{{container!="", {cadvisor_filter}}}[2m])))', "legendFormat": "{{namespace}}/{{pod}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "none"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}} + }) + panels.append({ + "id": 19, + "title": "CPU Throttling by Namespace", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'sum by (namespace) (rate(container_cpu_cfs_throttled_periods_total{{{cadvisor_filter}}}[2m]))', "legendFormat": "{{namespace}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 10}, "unit": "none"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}} + }) + current_y += 8 + + # Container Memory + panels.append({ + "id": 20, + "title": "Top 10 Memory Consuming Pods", + "type": "timeseries", + "gridPos": {"h": 8, "w": 14, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'topk(10, sum by (pod, namespace) (container_memory_working_set_bytes{{container!="", {cadvisor_filter}}}))', "legendFormat": "{{namespace}}/{{pod}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "bytes"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}} + }) + panels.append({ + "id": 21, + "title": "Memory Working Set by Namespace", + "type": "bargauge", + "gridPos": {"h": 8, "w": 10, "x": 14, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'sum by (namespace) (container_memory_working_set_bytes{{container!="", {cadvisor_filter}}})', "legendFormat": "{{namespace}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "unit": "bytes"}}, + "options": {"displayMode": "gradient", "orientation": "horizontal", "showUnfilled": True} + }) + current_y += 8 + + # Container Network + panels.append({ + "id": 22, + "title": "Top Pod Network RX (Receive)", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'topk(10, sum by (pod, namespace) (rate(container_network_receive_bytes_total{{{cadvisor_filter}}}[2m])))', "legendFormat": "{{namespace}}/{{pod}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}} + }) + panels.append({ + "id": 23, + "title": "Top Pod Network TX (Transmit)", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": f'topk(10, sum by (pod, namespace) (rate(container_network_transmit_bytes_total{{{cadvisor_filter}}}[2m])))', "legendFormat": "{{namespace}}/{{pod}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}} + }) + current_y += 8 + + else: + # Shipyard (Docker Swarm) + panels.append({ + "collapsed": False, + "gridPos": {"h": 1, "w": 24, "x": 0, "y": current_y}, + "id": 200, + "title": "📦 Docker Swarm Containers & Service Stacks — Shipyard Fleet", + "type": "row" + }) + current_y += 1 + + panels.extend([ + { + "id": 14, + "title": "Monitored Swarm Containers", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": 'count(container_last_seen{cluster="shipyard", name!=""}) or count(container_last_seen{cluster="shipyard"})', "legendFormat": "Containers", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": None}]}, "unit": "none"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + }, + { + "id": 15, + "title": "Active Swarm Nodes", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 6, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": 'count(up{cluster="shipyard", job="node-exporter"})', "legendFormat": "Swarm Nodes", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "blue", "value": None}]}, "unit": "none"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + }, + { + "id": 16, + "title": "Total Container CPU Load", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 12, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": 'sum(rate(container_cpu_usage_seconds_total{cluster="shipyard"}[2m]))', "legendFormat": "Cores", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "orange", "value": None}]}, "unit": "none"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + }, + { + "id": 17, + "title": "Total Container Working RAM", + "type": "stat", + "gridPos": {"h": 4, "w": 6, "x": 18, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": 'sum(container_memory_working_set_bytes{cluster="shipyard"})', "legendFormat": "RAM", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "purple", "value": None}]}, "unit": "bytes"}}, + "options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"} + } + ]) + current_y += 4 + + # Shipyard CPU & Memory + panels.append({ + "id": 18, + "title": "Container CPU Usage by Service/Task", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": 'topk(10, sum by (instance, id) (rate(container_cpu_usage_seconds_total{cluster="shipyard"}[2m])))', "legendFormat": "{{instance}} ({{id}})", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "none"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}} + }) + panels.append({ + "id": 19, + "title": "Container Memory Working Set by Service/Task", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": 'topk(10, sum by (instance, id) (container_memory_working_set_bytes{cluster="shipyard"}))', "legendFormat": "{{instance}} ({{id}})", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "bytes"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}} + }) + current_y += 8 + + # Shipyard Network RX/TX + panels.append({ + "id": 20, + "title": "Container Network Receive (RX)", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": 'topk(10, sum by (instance) (rate(container_network_receive_bytes_total{cluster="shipyard"}[2m])))', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}} + }) + panels.append({ + "id": 21, + "title": "Container Network Transmit (TX)", + "type": "timeseries", + "gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y}, + "datasource": "Prometheus", + "targets": [{"expr": 'topk(10, sum by (instance) (rate(container_network_transmit_bytes_total{cluster="shipyard"}[2m])))', "legendFormat": "{{instance}}", "refId": "A"}], + "fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}}, + "options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}} + }) + current_y += 8 + + dash["panels"] = panels + return dash + +clusters = [ + {"key": "aerodrome", "title": "Aerodrome", "folder_id": 5, "folder_uid": "folder-aerodrome", "emoji": "✈️"}, + {"key": "spaceport", "title": "Spaceport", "folder_id": 6, "folder_uid": "folder-spaceport", "emoji": "🚀"}, + {"key": "shipyard", "title": "Shipyard", "folder_id": 7, "folder_uid": "folder-shipyard", "emoji": "⚓"} +] + +# 1. Generate and Upload Dashboards +for c in clusters: + dash_json = build_cluster_dashboard(c["key"], c["title"], c["folder_id"], c["folder_uid"], c["emoji"]) + + # Save to local gitops repo + filename = f"homelab-gitops-repo/dashboards/grafana/{c['key']}-resource-monitor.json" + with open(filename, "w", encoding="utf-8") as f: + json.dump(dash_json, f, indent=2) + print(f"Saved {filename}") + + # Upload to Grafana API + payload = { + "dashboard": dash_json, + "folderId": c["folder_id"], + "folderUid": c["folder_uid"], + "overwrite": True + } + resp = requests.post(f"{base_url}/api/dashboards/db", json=payload, auth=auth, verify=False, timeout=10) + print(f"Uploaded {c['title']} dashboard to Grafana: {resp.status_code} - {resp.json().get('status')}") + +# 2. Delete old dashboards from AI Stack folder +for old_uid in ["snarfnet-systems", "snarfnet-containers"]: + del_resp = requests.delete(f"{base_url}/api/dashboards/uid/{old_uid}", auth=auth, verify=False, timeout=10) + print(f"Deleted old dashboard {old_uid}: {del_resp.status_code}") + +print("\nDashboard generation and deployment complete!") + diff --git a/scripts/check_30.py b/scripts/check_30.py new file mode 100644 index 0000000..86c39a8 --- /dev/null +++ b/scripts/check_30.py @@ -0,0 +1,15 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +try: + c.connect('140.44.4.30', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=5) + print("Connected to 140.44.4.30 with snarf") + stdin, stdout, _ = c.exec_command("hostname && ip a") + print(stdout.read().decode()) + c.close() +except Exception as e: + print("Failed 140.44.4.30:", e) + diff --git a/scripts/check_all_pods.py b/scripts/check_all_pods.py new file mode 100644 index 0000000..5cc8e81 --- /dev/null +++ b/scripts/check_all_pods.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl get pods -A") +print(o.read().decode().strip()) +c.close() + diff --git a/scripts/check_api_routes.py b/scripts/check_api_routes.py index ae20f89..654ae83 100644 --- a/scripts/check_api_routes.py +++ b/scripts/check_api_routes.py @@ -1,17 +1,34 @@ -import os -import sys import paramiko +import sys -sys.stdout.reconfigure(encoding="utf-8") -pkey = paramiko.Ed25519Key.from_private_key_file(os.path.expanduser('~/.ssh/id_ed25519_snarfnet')) +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.75', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) -client = paramiko.SSHClient() -client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) -client.connect("140.44.4.11", username="root", pkey=pkey, timeout=10) +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() -cmd = "curl -s http://140.44.4.21:80/main.105b4163.async.js | grep -o -E '\"/api/[a-zA-Z0-9_/]+\"' | sort -u | head -n 30" -stdin, stdout, stderr = client.exec_command(cmd) -print("API ROUTES ON 140.44.4.21:") -print(stdout.read().decode()) -client.close() +cid, _ = run("echo 'Katzir476!' | sudo -S docker ps -q -f name=peanut_peanut") +# Check readdir in src/app or src +script = """ +const fs = require('fs'); +function walk(dir) { + let res = []; + for (let f of fs.readdirSync(dir)) { + let p = dir + '/' + f; + if (fs.statSync(p).isDirectory()) res.push(...walk(p)); + else if (f.endsWith('.ts') || f.endsWith('.js') || f.endsWith('.tsx')) res.push(p); + } + return res; +} +console.log(walk('src/app/api')); +""" +run(f"cat << 'EOF' > /tmp/walk.js\n{script}\nEOF") +out, err = run(f"echo 'Katzir476!' | sudo -S docker exec -i {cid} node < /tmp/walk.js") +print(out) +if err: print(err) + +c.close() diff --git a/scripts/check_argo_rev.py b/scripts/check_argo_rev.py new file mode 100644 index 0000000..5271f7f --- /dev/null +++ b/scripts/check_argo_rev.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.41', username='root', password='dietpi', timeout=10) +_, o, e = c.exec_command("kubectl get app aerodrome-prometheus -n infrastructure") +print("out:", o.read().decode().strip()) +print("err:", e.read().decode().strip()) +c.close() diff --git a/scripts/check_argocd_apps.py b/scripts/check_argocd_apps.py index dfb4386..12347cf 100644 --- a/scripts/check_argocd_apps.py +++ b/scripts/check_argocd_apps.py @@ -1,73 +1,8 @@ -import requests -import json -import urllib3 -urllib3.disable_warnings() - -token = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJhcmdvY2QiLCJzdWIiOiJob21lcGFnZTphcGlLZXkiLCJuYmYiOjE3OTQ1MDIwNzgsImlhdCI6MTc5NDUwMjA3OCwianRpIjoiZjQ2NzU0NjMtODRhOS00MjI0LWI3ODItNmY5ODQxYWEwY2UxIn0.aHhrny2RBG-vAJAyXMvan0jw65jkP3bguqEduaG78O0" -headers = { - "Authorization": f"Bearer {token}", - "Content-Type": "application/json" -} - -try: - resp = requests.get("https://argo.snarfnet.net/api/v1/applications", headers=headers, verify=False, timeout=10) - data = resp.json() - items = data.get("items", []) - print(f"Total Applications: {len(items)}\n") - - out_of_sync = [] - synced = [] - - for app in items: - name = app.get("metadata", {}).get("name") - sync_status = app.get("status", {}).get("sync", {}).get("status") - health_status = app.get("status", {}).get("health", {}).get("status") - dest_server = app.get("spec", {}).get("destination", {}).get("server") - dest_ns = app.get("spec", {}).get("destination", {}).get("namespace") - repo = app.get("spec", {}).get("source", {}).get("repoURL") - path = app.get("spec", {}).get("source", {}).get("path") - - # Check operation or sync conditions/messages - operation_state = app.get("status", {}).get("operationState", {}) - op_phase = operation_state.get("phase") - op_msg = operation_state.get("message") - conditions = app.get("status", {}).get("conditions", []) - - info = { - "name": name, - "sync": sync_status, - "health": health_status, - "dest_ns": dest_ns, - "dest_server": dest_server, - "repo": repo, - "path": path, - "op_phase": op_phase, - "op_msg": op_msg, - "conditions": conditions - } - - if sync_status != "Synced" or health_status not in ["Healthy", None]: - out_of_sync.append(info) - else: - synced.append(info) - - print(f"=== OUT OF SYNC / DEGRADED APPS ({len(out_of_sync)}) ===") - for a in out_of_sync: - print(f"\nApp: {a['name']}") - print(f" Sync: {a['sync']} | Health: {a['health']}") - print(f" Dest: {a['dest_ns']} on {a['dest_server']}") - print(f" Path: {a['path']}") - if a['op_phase']: - print(f" Op Phase: {a['op_phase']}") - if a['op_msg']: - print(f" Op Msg: {a['op_msg']}") - if a['conditions']: - print(f" Conditions: {a['conditions']}") - - print(f"\n=== SYNCED APPS ({len(synced)}) ===") - for a in synced: - print(f" {a['name']}: Sync={a['sync']}, Health={a['health']}, NS={a['dest_ns']}") - -except Exception as e: - print("Error querying Argo CD API:", e) +import paramiko +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.41', username='root', password='dietpi', timeout=10) +_, o, _ = c.exec_command("kubectl get applications.argoproj.io -n infrastructure") +print(o.read().decode().strip()) +c.close() diff --git a/scripts/check_authentik_apps.py b/scripts/check_authentik_apps.py index 81d65b2..4f2bfd9 100644 --- a/scripts/check_authentik_apps.py +++ b/scripts/check_authentik_apps.py @@ -1,27 +1,35 @@ import paramiko -import sys +import json -sys.stdout.reconfigure(encoding='utf-8') - -key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" c = paramiko.SSHClient() c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) -c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5) +c.connect("140.44.4.41", username="root", key_filename=SSH_KEY, timeout=10) -cmd = "k3s kubectl -n infra exec authentik-postgresql-0 -- psql -U authentik -d authentik -c 'SELECT id, name, authorization_flow_id FROM authentik_core_application;'" -stdin, stdout, stderr = c.exec_command(cmd) -print("Authentik Applications:") -print(stdout.read().decode('utf-8')) +def k8s(cmd): + full_cmd = f"kubectl {cmd}" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() -cmd2 = "k3s kubectl -n infra exec authentik-postgresql-0 -- psql -U authentik -d authentik -c 'SELECT id, name, external_host FROM authentik_providers_proxy_proxyprovider;'" -stdin, stdout, stderr = c.exec_command(cmd2) -print("Authentik Proxy Providers:") -print(stdout.read().decode('utf-8')) +print("Authentik pods:") +out, _ = k8s("get pods -A | grep -i authentik") +print(out) -cmd3 = "k3s kubectl -n infra exec authentik-postgresql-0 -- psql -U authentik -d authentik -c 'SELECT id, name, client_id FROM authentik_providers_oauth2_oauth2provider;'" -stdin, stdout, stderr = c.exec_command(cmd3) -print("Authentik OAuth2/OIDC Providers:") -print(stdout.read().decode('utf-8')) +# Check Authentik pod name +pod = "" +for line in out.splitlines(): + if "server" in line: + pod = line.split()[1] + break + +print("Authentik server pod:", pod) +if pod: + # Query Authentik applications and proxy providers via manage.py or python + out, err = k8s(f"exec -n authentik {pod} -- ak list_apps 2>/dev/null || kubectl exec -n authentik {pod} -- python -m authentik.manage shell -c 'from authentik.core.models import Application; print([(a.name, a.slug) for a in Application.objects.all()])'") + print("Applications:", out, err) + + # Query Authentik proxy providers + out, err = k8s(f"exec -n authentik {pod} -- python -m authentik.manage shell -c 'from authentik.providers.proxy.models import ProxyProvider; print([(p.name, p.external_host) for p in ProxyProvider.objects.all()])'") + print("Proxy Providers:", out, err) c.close() - diff --git a/scripts/check_authentik_db.py b/scripts/check_authentik_db.py new file mode 100644 index 0000000..9b322d2 --- /dev/null +++ b/scripts/check_authentik_db.py @@ -0,0 +1,35 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10) + +_, o, _ = c.exec_command("kubectl exec -n infra authentik-postgresql-0 -- cat /opt/bitnami/postgresql/secrets/password") +pw = o.read().decode().strip() + +def psql(query): + cmd = f'kubectl exec -n infra authentik-postgresql-0 -- sh -c "PGPASSWORD={pw} psql -U authentik -d authentik -c \\"{query}\\""' + _, o, e = c.exec_command(cmd) + res = o.read().decode().strip() + err = e.read().decode().strip() + if err and not res: + return f"[ERR: {err}]" + return res + +print("--- APPLICATIONS ---") +print(psql("SELECT name, slug FROM authentik_core_application;")) + +print("\n--- PROXY PROVIDERS ---") +print(psql("SELECT name, external_host, mode FROM authentik_providers_proxy_proxyprovider;")) + +print("\n--- EXPRESSION POLICIES ---") +print(psql("SELECT name, expression FROM authentik_policies_expression_expressionpolicy;")) + +print("\n--- POLICY BINDINGS ---") +print(psql("SELECT target_id, enabled FROM authentik_policies_policybinding;")) + +print("\n--- AUTHENTIK SETTINGS / SESSIONS ---") +print(psql("SELECT key, value FROM authentik_tenants_tenant;")) + +c.close() diff --git a/scripts/check_authentik_file.py b/scripts/check_authentik_file.py new file mode 100644 index 0000000..36ef58f --- /dev/null +++ b/scripts/check_authentik_file.py @@ -0,0 +1,32 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10) + +_, o, _ = c.exec_command("kubectl exec -n infra authentik-postgresql-0 -- cat /opt/bitnami/postgresql/secrets/password") +pw = o.read().decode().strip() + +sql = """ +\\echo '=== APPS ===' +SELECT name, slug FROM authentik_core_application; +\\echo '=== PROVIDERS ===' +SELECT name, external_host, mode FROM authentik_providers_proxy_proxyprovider; +\\echo '=== POLICIES ===' +SELECT name, expression FROM authentik_policies_expression_expressionpolicy; +""" + +stdin, stdout, stderr = c.exec_command("cat > /tmp/query.sql") +stdin.write(sql) +stdin.close() +stdout.channel.recv_exit_status() + +c.exec_command("kubectl cp /tmp/query.sql infra/authentik-postgresql-0:/tmp/query.sql") +cmd = f"kubectl exec -n infra authentik-postgresql-0 -- env PGPASSWORD='{pw}' psql -U authentik -d authentik -f /tmp/query.sql" +_, o, e = c.exec_command(cmd) +print(o.read().decode()) +print("ERR:", e.read().decode()) + +c.close() + diff --git a/scripts/check_authentik_fixed.py b/scripts/check_authentik_fixed.py new file mode 100644 index 0000000..18fcf1f --- /dev/null +++ b/scripts/check_authentik_fixed.py @@ -0,0 +1,46 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"kubectl {cmd}" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +pod = "authentik-server-cd7d9b949-vfl2m" + +script = ''' +from authentik.core.models import Application +from authentik.providers.proxy.models import ProxyProvider +from authentik.policies.expression.models import ExpressionPolicy + +print("=== APPLICATIONS ===") +for app in Application.objects.all(): + print(f"{app.name} (slug: {app.slug}) | Provider: {app.provider}") + +print("=== PROXY PROVIDERS ===") +for p in ProxyProvider.objects.all(): + print(f"{p.name} | Host: {p.external_host} | Mode: {p.mode}") + +print("=== EXPRESSION POLICIES ===") +for ep in ExpressionPolicy.objects.all(): + print(f"Policy: {ep.name}") + print(ep.expression) +''' + +# Write script to /tmp/ak_query.py on 140.44.4.11 +stdin, stdout, stderr = c.exec_command("cat > /tmp/ak_query.py") +stdin.write(script) +stdin.close() +stdout.channel.recv_exit_status() + +# Copy into pod and execute +k8s(f"cp /tmp/ak_query.py infra/{pod}:/tmp/ak_query.py -c server") +out, err = k8s(f"exec -n infra {pod} -c server -- /lifecycle/ak shell < /tmp/ak_query.py") +print("Authentik query output:\n", out) +if err: print("ERR:\n", err) + +c.close() diff --git a/scripts/check_authentik_policies.py b/scripts/check_authentik_policies.py new file mode 100644 index 0000000..b526e3d --- /dev/null +++ b/scripts/check_authentik_policies.py @@ -0,0 +1,49 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"kubectl {cmd}" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +pod = "authentik-server-cd7d9b949-vfl2m" + +# Inspect applications in Authentik +py_query = """ +python3 - << 'PYEOF' +from authentik.core.models import Application +from authentik.providers.proxy.models import ProxyProvider +from authentik.policies.models import PolicyBindingModel, Policy + +print("=== APPLICATIONS ===") +for app in Application.objects.all(): + print(f"App: {app.name} (slug: {app.slug})") + print(f" Provider: {app.provider}") + +print("\n=== PROXY PROVIDERS ===") +for p in ProxyProvider.objects.all(): + print(f"Provider: {p.name}") + print(f" External host: {p.external_host}") + print(f" Internal host: {p.internal_host}") + print(f" Mode: {p.mode}") + +print("\n=== POLICIES ===") +for pol in Policy.objects.all(): + print(f"Policy: {pol.name} ({pol.__class__.__name__})") + +print("\n=== POLICY BINDINGS ===") +for pb in PolicyBindingModel.objects.all(): + print(f"Binding: {pb.target} -> {pb.policy} (enabled={pb.enabled}, order={pb.order})") +PYEOF +""" + +out, err = k8s(f"exec -n infra {pod} -c authentik-server -- {py_query}") +print(out) +if err: print("ERR:", err) + +c.close() + diff --git a/scripts/check_authentik_policies_sql.py b/scripts/check_authentik_policies_sql.py new file mode 100644 index 0000000..2911ea2 --- /dev/null +++ b/scripts/check_authentik_policies_sql.py @@ -0,0 +1,28 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10) + +_, o, _ = c.exec_command("kubectl exec -n infra authentik-postgresql-0 -- cat /opt/bitnami/postgresql/secrets/password") +pw = o.read().decode().strip() + +sql = """ +SELECT p.name, ep.expression +FROM authentik_policies_expression_expressionpolicy ep +JOIN authentik_policies_policy p ON ep.policy_ptr_id = p.policy_uuid; +""" + +stdin, stdout, stderr = c.exec_command("cat > /tmp/query_policies.sql") +stdin.write(sql) +stdin.close() +stdout.channel.recv_exit_status() + +c.exec_command("kubectl cp /tmp/query_policies.sql infra/authentik-postgresql-0:/tmp/query_policies.sql") +cmd = f"kubectl exec -n infra authentik-postgresql-0 -- env PGPASSWORD='{pw}' psql -U authentik -d authentik -f /tmp/query_policies.sql" +_, o, e = c.exec_command(cmd) +print(o.read().decode()) + +c.close() + diff --git a/scripts/check_authentik_stdin.py b/scripts/check_authentik_stdin.py new file mode 100644 index 0000000..9bba611 --- /dev/null +++ b/scripts/check_authentik_stdin.py @@ -0,0 +1,28 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10) + +_, o, _ = c.exec_command("kubectl exec -n infra authentik-postgresql-0 -- cat /opt/bitnami/postgresql/secrets/password") +pw = o.read().decode().strip() + +def run_sql(sql): + cmd = f"kubectl exec -i -n infra authentik-postgresql-0 -- /bin/bash -c 'export PGPASSWORD={pw}; psql -U authentik -d authentik'" + stdin, stdout, stderr = c.exec_command(cmd) + stdin.write(sql + "\n") + stdin.close() + return stdout.read().decode().strip() + +print("--- APPLICATIONS ---") +print(run_sql("SELECT name, slug FROM authentik_core_application;")) + +print("\n--- PROXY PROVIDERS ---") +print(run_sql("SELECT name, external_host, mode FROM authentik_providers_proxy_proxyprovider;")) + +print("\n--- EXPRESSION POLICIES ---") +print(run_sql("SELECT name, expression FROM authentik_policies_expression_expressionpolicy;")) + +c.close() + diff --git a/scripts/check_blockdevices.py b/scripts/check_blockdevices.py new file mode 100644 index 0000000..5ea9db9 --- /dev/null +++ b/scripts/check_blockdevices.py @@ -0,0 +1,22 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Harvester Block Devices:") +out, _ = k8s("kubectl get blockdevices.harvesterhci.io -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName,DEV:.status.devicePath,FS:.status.fileSystem.type,MOUNT:.status.fileSystem.mountPoint,PROVISIONED:.status.provisioned") +print(out) + +print("\nHarvester Node Disk status:") +out, _ = k8s("kubectl get nodes.harvesterhci.io -A -o yaml | grep -E 'nodeName|diskStatus|mountPath|provisioned'") +print(out[:1000]) + +c.close() + diff --git a/scripts/check_cadvisor_ps.py b/scripts/check_cadvisor_ps.py new file mode 100644 index 0000000..d4ed7d4 --- /dev/null +++ b/scripts/check_cadvisor_ps.py @@ -0,0 +1,10 @@ +import paramiko +import time + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("echo 'Katzir476!' | sudo -S docker stack ps monitoring") +print(o.read().decode().strip()) +c.close() + diff --git a/scripts/check_cm_live.py b/scripts/check_cm_live.py new file mode 100644 index 0000000..e9e3775 --- /dev/null +++ b/scripts/check_cm_live.py @@ -0,0 +1,23 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +_, o, _ = c.exec_command("kubectl get configmap prometheus-config -o json") +import json +data = json.loads(o.read().decode().strip()) +prom_yml = data.get("data", {}).get("prometheus.yml", "") +print("Length of prometheus.yml in CM:", len(prom_yml)) +print("Contains shipyard:", "shipyard" in prom_yml) +if "shipyard" in prom_yml: + for line in prom_yml.splitlines(): + if "shipyard" in line: + print(" ", line) +else: + print("Last 20 lines of prometheus.yml:") + for line in prom_yml.splitlines()[-20:]: + print(" ", line) + +c.close() + diff --git a/scripts/check_cm_meta.py b/scripts/check_cm_meta.py new file mode 100644 index 0000000..8b36892 --- /dev/null +++ b/scripts/check_cm_meta.py @@ -0,0 +1,11 @@ +import paramiko +import json + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl get configmap prometheus-config -o json") +data = json.loads(o.read().decode().strip()) +print(json.dumps(data.get("metadata"), indent=2)) +c.close() + diff --git a/scripts/check_cm_now.py b/scripts/check_cm_now.py new file mode 100644 index 0000000..7a0882b --- /dev/null +++ b/scripts/check_cm_now.py @@ -0,0 +1,11 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl get configmap prometheus-config -o yaml") +yaml_str = o.read().decode().strip() +print("Length of CM:", len(yaml_str)) +print("Contains shipyard:", "shipyard" in yaml_str) +c.close() + diff --git a/scripts/check_cm_tmp.py b/scripts/check_cm_tmp.py new file mode 100644 index 0000000..3f74ba9 --- /dev/null +++ b/scripts/check_cm_tmp.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("cat /tmp/prometheus_cm.yaml | grep -A 25 'job_name: node-exporter'") +print(o.read().decode().strip()) +c.close() + diff --git a/scripts/check_compute_fw.py b/scripts/check_compute_fw.py new file mode 100644 index 0000000..bac3af4 --- /dev/null +++ b/scripts/check_compute_fw.py @@ -0,0 +1,25 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +s = requests.Session() +s.verify = False + +r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'}) +csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token') +headers = {'X-CSRF-Token': csrf} if csrf else {} + +res = s.get('https://14.44.4.1/proxy/network/v2/api/site/default/firewall-policies', headers=headers) +policies = res.json() +compute_net_id = "1748bb1d-f674-4dcd-85d0-f58109a02e77" + +for p in policies: + src_nets = p.get('source', {}).get('network_ids', []) or [] + dst_nets = p.get('destination', {}).get('network_ids', []) or [] + if compute_net_id in src_nets or compute_net_id in dst_nets: + print(f"Policy: {p.get('name')} (Index {p.get('index')}) | Action: {p.get('action')} | Type: {p.get('connection_state_type')}") + print(f" Src: {p.get('source')}") + print(f" Dst: {p.get('destination')}") + diff --git a/scripts/check_crd_salvage.py b/scripts/check_crd_salvage.py new file mode 100644 index 0000000..4e985a8 --- /dev/null +++ b/scripts/check_crd_salvage.py @@ -0,0 +1,31 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Replica spec fields:") +out, _ = k8s("kubectl explain replica.longhorn.io.spec") +print(out) + +print("\nVolume spec fields:") +out, _ = k8s("kubectl explain volume.longhorn.io.spec") +print(out) + +print("\nEngine spec fields:") +out, _ = k8s("kubectl explain engine.longhorn.io.spec") +print(out) + +# Check engine CR directly +out, _ = k8s("kubectl get engines.longhorn.io -n longhorn-system pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-e-0 -o yaml") +print("\nEngine YAML for pvc-4da99c48:") +print(out) + +c.close() + diff --git a/scripts/check_cross_vlan.py b/scripts/check_cross_vlan.py new file mode 100644 index 0000000..53c6069 --- /dev/null +++ b/scripts/check_cross_vlan.py @@ -0,0 +1,30 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("Ping 14.4.4.20 (HomeAssistant):") +out, _ = run("ping -c 2 -W 2 14.4.4.20") +print(out) + +print("Ping 14.4.4.15:") +out, _ = run("ping -c 2 -W 2 14.4.4.15") +print(out) + +print("Ping 14.4.4.44 (nutserver):") +out, _ = run("ping -c 2 -W 2 14.4.4.44") +print(out) + +print("Traceroute to 14.4.4.44:") +out, _ = run("traceroute 14.4.4.44 || tracepath 14.4.4.44") +print(out) + +c.close() + diff --git a/scripts/check_current_harvester_tags.py b/scripts/check_current_harvester_tags.py new file mode 100644 index 0000000..ef94f34 --- /dev/null +++ b/scripts/check_current_harvester_tags.py @@ -0,0 +1,33 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run_sql(sql): + pod_cmd = "kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'" + _, o, _ = c.exec_command(pod_cmd) + pod = o.read().decode().strip() + cmd = f"kubectl exec -i -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root" + stdin, stdout, stderr = c.exec_command(cmd) + stdin.write(sql) + stdin.close() + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- Monitor tags for 57, 94, 98-105 ---") +query = """ +USE kuma; +SELECT m.id, m.name, t.name as tag_name +FROM monitor m +LEFT JOIN monitor_tag mt ON m.id = mt.monitor_id +LEFT JOIN tag t ON mt.tag_id = t.id +WHERE m.id IN (57, 94, 98, 99, 100, 101, 102, 103, 104, 105) +ORDER BY m.id, t.id; +""" +out, _ = run_sql(query) +print(out) + +c.close() + diff --git a/scripts/check_deploy_vols.py b/scripts/check_deploy_vols.py new file mode 100644 index 0000000..2801046 --- /dev/null +++ b/scripts/check_deploy_vols.py @@ -0,0 +1,13 @@ +import paramiko +import json + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl get deploy prometheus -o json") +data = json.loads(o.read().decode().strip()) +volumes = data.get("spec", {}).get("template", {}).get("spec", {}).get("volumes", []) +for v in volumes: + print(v) +c.close() + diff --git a/scripts/check_disk_contents.py b/scripts/check_disk_contents.py new file mode 100644 index 0000000..e72d9eb --- /dev/null +++ b/scripts/check_disk_contents.py @@ -0,0 +1,23 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Mount /dev/sdf to /mnt and check contents:") +out, err = run_cmd("mkdir -p /mnt/test && mount /dev/sdf /mnt/test && ls -la /mnt/test") +print("sdf:", out, err) + +# Check all devices sdb through sdh +for d in ["sdb", "sdc", "sdd", "sde", "sdf", "sdg", "sdh"]: + out, _ = run_cmd(f"mkdir -p /mnt/{d} && mount /dev/{d} /mnt/{d} 2>/dev/null && ls -la /mnt/{d} | grep -E 'longhorn|replicas'") + print(f"{d}: {out}") + +c.close() + diff --git a/scripts/check_disk_contents2.py b/scripts/check_disk_contents2.py new file mode 100644 index 0000000..46f413e --- /dev/null +++ b/scripts/check_disk_contents2.py @@ -0,0 +1,23 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Mount /dev/sdf to /tmp/test:") +out, err = run_cmd("mkdir -p /tmp/test && mount /dev/sdf /tmp/test && ls -la /tmp/test; umount /tmp/test") +print("sdf:", out, err) + +# Check all devices on Node 1: +for d in ["sdb", "sdc", "sdd", "sde", "sdf", "sdg", "sdh"]: + out, _ = run_cmd(f"mkdir -p /tmp/{d} && mount /dev/{d} /tmp/{d} 2>/dev/null && ls -la /tmp/{d}; umount /tmp/{d} 2>/dev/null") + print(f"--- {d} ---:\n{out}") + +c.close() + diff --git a/scripts/check_disks_mount.py b/scripts/check_disks_mount.py new file mode 100644 index 0000000..64484db --- /dev/null +++ b/scripts/check_disks_mount.py @@ -0,0 +1,35 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("=== NODES.LONGHORN.IO DISK STATUS ===") +out, _ = k8s("kubectl get nodes.longhorn.io -n longhorn-system -o yaml | grep -E 'diskStatus|diskUUID|ready|schedulable|storageAvailable|storageMaximum|path:'") +print(out[:1500]) + +print("\n=== LSBLK & MOUNTS ON NODE 01 ===") +out, _ = k8s("lsblk -f; echo '--- DF ---'; df -h | grep -E 'Filesystem|extra-disks|harvester'") +print(out) + +# Check all 4 nodes for extra-disks mount +for ip in ["140.44.4.81", "140.44.4.82", "140.44.4.83", "140.44.4.84"]: + print(f"\n--- Node {ip} ---") + try: + nc = paramiko.SSHClient() + nc.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + nc.connect(ip, username="rancher", key_filename=SSH_KEY, timeout=5) + _, o, _ = nc.exec_command("echo Katzir476! | sudo -S lsblk | grep -E 'sd|nvme'; df -h | grep extra") + print(o.read().decode().strip()) + nc.close() + except Exception as e: + print("Error connecting:", e) + +c.close() + diff --git a/scripts/check_dns_heimdall.py b/scripts/check_dns_heimdall.py new file mode 100644 index 0000000..32ebefa --- /dev/null +++ b/scripts/check_dns_heimdall.py @@ -0,0 +1,27 @@ +import urllib.request +import urllib.parse +import ssl +import json +import http.client + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def do_req(ip, host_header, path): + conn = http.client.HTTPSConnection(ip, 443, context=ctx, timeout=8) + conn.request("GET", path, headers={"Host": host_header}) + resp = conn.getresponse() + data = resp.read().decode('utf-8') + conn.close() + return json.loads(data) + +try: + token = do_req("140.44.4.62", "dns2.snarfnet.net", "/api/user/login?user=admin&pass=snarfnetadmin&includeInfo=true")['token'] + rec_heimdall = do_req("140.44.4.62", "dns2.snarfnet.net", f"/api/zones/records/get?token={token}&zone=snarfnet.net&domain=heimdall.snarfnet.net") + print("DNS2 heimdall:", rec_heimdall) + rec_portal = do_req("140.44.4.62", "dns2.snarfnet.net", f"/api/zones/records/get?token={token}&zone=snarfnet.net&domain=portal.snarfnet.net") + print("DNS2 portal:", rec_portal) +except Exception as e: + print("DNS check err:", e) + diff --git a/scripts/check_docker_sudo.py b/scripts/check_docker_sudo.py new file mode 100644 index 0000000..8e51d50 --- /dev/null +++ b/scripts/check_docker_sudo.py @@ -0,0 +1,19 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- echo 'Katzir476!' | sudo -S docker service ls ---") +out, err = run("echo 'Katzir476!' | sudo -S docker service ls") +print("OUT:\n", out) +print("ERR:\n", err) + +c.close() + diff --git a/scripts/check_etc_prom.py b/scripts/check_etc_prom.py new file mode 100644 index 0000000..e4b42eb --- /dev/null +++ b/scripts/check_etc_prom.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl exec deploy/prometheus -- ls -la /etc/prometheus") +print("ls -la /etc/prometheus:\n", o.read().decode().strip()) +c.close() + diff --git a/scripts/check_fstab_mounts.py b/scripts/check_fstab_mounts.py new file mode 100644 index 0000000..5ba4ab6 --- /dev/null +++ b/scripts/check_fstab_mounts.py @@ -0,0 +1,30 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("--- /etc/fstab on Node 1 ---") +out, _ = run_cmd("cat /etc/fstab") +print(out) + +print("\n--- Systemd mount units on Node 1 ---") +out, _ = run_cmd("systemctl list-unit-files | grep -i var-lib-harvester") +print(out) + +print("\n--- Blkid on Node 1 ---") +out, _ = run_cmd("blkid") +print(out) + +print("\n--- Any harvester disk mount script or cloud-init ---") +out, _ = run_cmd("ls -la /oem; cat /oem/99_custom.yaml 2>/dev/null || cat /oem/harvester.config 2>/dev/null") +print(out[:1000]) + +c.close() + diff --git a/scripts/check_gotify_discord.py b/scripts/check_gotify_discord.py new file mode 100644 index 0000000..1a40c48 --- /dev/null +++ b/scripts/check_gotify_discord.py @@ -0,0 +1,28 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', password='Katzir476!', timeout=10) + +def run(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("=== GOTIFY DISCORD BRIDGE LOGS ===") +cid, _ = run("docker ps -q -f name=gotify-discord-bridge") +if cid: + logs, _ = run(f"docker logs {cid} --tail=100") + print(logs) +else: + print("Bridge container not found!") + +print("\n=== GOTIFY SERVER RECENT MESSAGES ===") +# Check messages inside Gotify DB or logs +gid, _ = run("docker ps -q -f name=gotify_gotify") +if gid: + logs, _ = run(f"docker logs {gid} --tail=50") + print(logs) + +c.close() + diff --git a/scripts/check_grafana_ds.py b/scripts/check_grafana_ds.py new file mode 100644 index 0000000..af1d39c --- /dev/null +++ b/scripts/check_grafana_ds.py @@ -0,0 +1,14 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +# Let's query Prometheus datasource from Grafana or directly from 140.44.4.11/140.44.4.34 +url = "https://grafana.snarfnet.net/api/datasources" +resp = requests.get(url, auth=auth, verify=False, timeout=10) +datasources = resp.json() +print("Datasources:") +for ds in datasources: + print(f"ID: {ds['id']} | Name: {ds['name']} | Type: {ds['type']} | URL: {ds['url']}") + diff --git a/scripts/check_ha.py b/scripts/check_ha.py new file mode 100644 index 0000000..23b8291 --- /dev/null +++ b/scripts/check_ha.py @@ -0,0 +1,17 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +try: + c.connect('14.4.4.20', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=5) + print("Connected to 14.4.4.20") +except Exception as e: + print("Failed key:", e) + try: + c.connect('14.4.4.20', username='root', password='dietpi', timeout=5) + print("Connected with root/dietpi") + except Exception as e2: + print("Failed root/dietpi:", e2) + diff --git a/scripts/check_ha_ups.py b/scripts/check_ha_ups.py new file mode 100644 index 0000000..a5cace4 --- /dev/null +++ b/scripts/check_ha_ups.py @@ -0,0 +1,17 @@ +import urllib.request +import json + +HA_URL = "http://14.4.4.20:8123" +HA_TOKEN = "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJkZWNlMWI5ZjJhZDQ0MTRmODU4NGYzN2I1YjgwZDQxMiIsImlhdCI6MTc5MzQ1MjQ2MywiZXhwIjoyMTA4ODEyNDYzfQ.JznpOqsCSnHz6aTVa8K9pQCz1816DC8SrmS1xn1Sm2k" + +req = urllib.request.Request(f"{HA_URL}/api/states", headers={"Authorization": f"Bearer {HA_TOKEN}", "Content-Type": "application/json"}) +try: + with urllib.request.urlopen(req, timeout=5) as r: + states = json.loads(r.read().decode()) + ups_states = [s for s in states if "ups" in s["entity_id"] or "server_closet" in s["entity_id"] or "nut" in s["entity_id"]] + print(f"Found {len(ups_states)} UPS entities in Home Assistant:") + for s in ups_states: + print(f" {s['entity_id']}: {s['state']} {s.get('attributes', {}).get('unit_of_measurement', '')}") +except Exception as e: + print("HA err:", e) + diff --git a/scripts/check_heimdall_db_deep.py b/scripts/check_heimdall_db_deep.py new file mode 100644 index 0000000..ac72606 --- /dev/null +++ b/scripts/check_heimdall_db_deep.py @@ -0,0 +1,49 @@ +import paramiko +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +py_script = """python3 - << 'PYEOF' +import sqlite3 +import shutil +import time + +db_path = '/var/lib/docker/volumes/heimdall_heimdall_config/_data/www/app.sqlite' + +# Make a fresh backup +shutil.copyfile(db_path, f"{db_path}.verified_backup_{int(time.time())}") +print("Backup created at verified_backup!") + +con = sqlite3.connect(db_path) +cur = con.cursor() + +print("=== USERS ===") +for r in cur.execute("SELECT id, username, email, public_front FROM users").fetchall(): + print("User:", r) + +print("\n=== ITEMS ===") +rows = cur.execute("SELECT id, title, url, pinned, user_id, apporder FROM items").fetchall() +print(f"Total items in DB: {len(rows)}") +for r in rows: + print("Item:", r) + +print("\n=== SETTINGS ===") +for r in cur.execute("SELECT * FROM settings").fetchall(): + print("Setting:", r) + +PYEOF +""" + +_, o, e = c.exec_command(f"echo Katzir476! | sudo -S {py_script}") +print(o.read().decode()) +print(e.read().decode()) + +# Check volume mount persistence +_, o, _ = c.exec_command("echo Katzir476! | sudo -S ls -la /var/lib/docker/volumes/heimdall_heimdall_config/_data/www/") +print("Files in Heimdall www volume:\n", o.read().decode()) + +c.close() + diff --git a/scripts/check_heimdall_db_fixed.py b/scripts/check_heimdall_db_fixed.py new file mode 100644 index 0000000..d3759fa --- /dev/null +++ b/scripts/check_heimdall_db_fixed.py @@ -0,0 +1,47 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +py_code = ''' +import sqlite3 +import shutil +import time + +db_path = '/var/lib/docker/volumes/heimdall_heimdall_config/_data/www/app.sqlite' +backup_name = db_path + '.backup_' + str(int(time.time())) +shutil.copyfile(db_path, backup_name) +print('Backup created:', backup_name) + +con = sqlite3.connect(db_path) +cur = con.cursor() + +print('=== USERS ===') +for r in cur.execute('SELECT id, username, email, public_front FROM users').fetchall(): + print('User:', r) + +print('\\n=== ITEMS ===') +rows = cur.execute('SELECT id, title, url, pinned, user_id, apporder FROM items').fetchall() +print('Total items in DB:', len(rows)) +for r in rows: + print('Item:', r) + +print('\\n=== SETTINGS ===') +for r in cur.execute('SELECT * FROM settings').fetchall(): + print('Setting:', r) +''' + +# Write script to /tmp/inspect_heimdall.py on remote host +stdin, stdout, stderr = c.exec_command("cat > /tmp/inspect_heimdall.py") +stdin.write(py_code) +stdin.close() +stdout.channel.recv_exit_status() + +# Execute script with sudo +_, o, e = c.exec_command("echo Katzir476! | sudo -S python3 /tmp/inspect_heimdall.py") +print(o.read().decode()) +print(e.read().decode()) + +c.close() + diff --git a/scripts/check_heimdall_final.py b/scripts/check_heimdall_final.py new file mode 100644 index 0000000..604212e --- /dev/null +++ b/scripts/check_heimdall_final.py @@ -0,0 +1,19 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +cmd = """echo 'Katzir476!' | sudo -S python3 -c " +import sqlite3 +con = sqlite3.connect('/var/lib/docker-data/heimdall/config/www/app.sqlite') +cur = con.cursor() +print('Users:', cur.execute('SELECT id, username, public_front FROM users').fetchall()) +print('Pinned Items:', cur.execute('SELECT count(*) FROM items WHERE pinned=1').fetchone()[0]) +con.close() +" """ +_, o, _ = c.exec_command(cmd) +print("Output:\n", o.read().decode().strip()) +c.close() diff --git a/scripts/check_heimdall_mount_real.py b/scripts/check_heimdall_mount_real.py new file mode 100644 index 0000000..755895a --- /dev/null +++ b/scripts/check_heimdall_mount_real.py @@ -0,0 +1,19 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +def run(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +out, _ = run("docker inspect $(docker ps -q -f name=heimdall) | grep -A 10 -i mounts") +print("Container Mounts:\n", out) + +out, _ = run("docker service inspect heimdall_heimdall --pretty") +print("\nService inspect pretty:\n", out) + +c.close() + diff --git a/scripts/check_heimdall_volume.py b/scripts/check_heimdall_volume.py new file mode 100644 index 0000000..70fe65a --- /dev/null +++ b/scripts/check_heimdall_volume.py @@ -0,0 +1,21 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +def run(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Docker Volume Inspect:") +out, _ = run("docker volume inspect heimdall_heimdall_config") +print(out) + +print("\nHeimdall service constraints:") +out, _ = run("docker service inspect heimdall_heimdall --format '{{json .Spec.TaskTemplate.Placement}}'") +print(out) + +c.close() + diff --git a/scripts/check_idrac4_details.py b/scripts/check_idrac4_details.py new file mode 100644 index 0000000..f3d4232 --- /dev/null +++ b/scripts/check_idrac4_details.py @@ -0,0 +1,27 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1} + if auth: payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + +auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result") + +h = zabbix_api("host.get", { + "filter": {"host": ["idrac-spacedock4"]}, + "selectInterfaces": "extend", + "selectGroups": "extend", + "selectParentTemplates": "extend" +}, auth=auth).get("result", []) +print(json.dumps(h, indent=2)) + diff --git a/scripts/check_instance_manager.py b/scripts/check_instance_manager.py new file mode 100644 index 0000000..8425acb --- /dev/null +++ b/scripts/check_instance_manager.py @@ -0,0 +1,23 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Instance Manager pods:") +out, _ = k8s("kubectl get pods -n longhorn-system -l longhorn.io/component=instance-manager -o wide") +print(out) + +# Check Longhorn volume CR +out, _ = k8s("kubectl get volume.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495 -n longhorn-system -o yaml") +print("\nVolume YAML for pvc-4da99c48:") +print(out) + +c.close() + diff --git a/scripts/check_items_schema.py b/scripts/check_items_schema.py new file mode 100644 index 0000000..eec03cc --- /dev/null +++ b/scripts/check_items_schema.py @@ -0,0 +1,23 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +cmd = """echo 'Katzir476!' | sudo -S python3 -c " +import sqlite3 +con = sqlite3.connect('/var/lib/docker-data/heimdall/config/www/app.sqlite') +cur = con.cursor() +print('Item columns:', [d[0] for d in cur.execute('SELECT * FROM items LIMIT 1').description]) +rows = cur.execute('SELECT id, title, pinned, user_id FROM items LIMIT 5').fetchall() +print('Sample items:', rows) +print('Total items count:', cur.execute('SELECT count(*) FROM items').fetchone()[0]) +print('Distinct pinned values:', cur.execute('SELECT DISTINCT pinned FROM items').fetchall()) +con.close() +" """ +_, o, _ = c.exec_command(cmd) +print("Output:\n", o.read().decode().strip()) +c.close() + diff --git a/scripts/check_kuma_monitor_schema.py b/scripts/check_kuma_monitor_schema.py new file mode 100644 index 0000000..b45812c --- /dev/null +++ b/scripts/check_kuma_monitor_schema.py @@ -0,0 +1,21 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + _, o, e = c.exec_command(cmd) + return o.read().decode().strip() + +pod = run("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'") +res = run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "USE kuma; DESCRIBE monitor;" """) +print("Monitor Columns:\n", res) + +res2 = run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "USE kuma; SELECT * FROM monitor WHERE id=94;" """) +print("\nMonitor 94:\n", res2) + +c.close() + diff --git a/scripts/check_kuma_notifs.py b/scripts/check_kuma_notifs.py new file mode 100644 index 0000000..0cdfe74 --- /dev/null +++ b/scripts/check_kuma_notifs.py @@ -0,0 +1,21 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + _, o, e = c.exec_command(cmd) + return o.read().decode().strip() + +pod = run("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'") +res = run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "USE kuma; DESCRIBE notification; SELECT * FROM notification;" """) +print(res) + +notif_mon = run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "USE kuma; SELECT * FROM monitor_notification WHERE monitor_id IN (57, 94);" """) +print("\nMonitor Notifications for 57, 94:") +print(notif_mon) + +c.close() diff --git a/scripts/check_kuma_ping_format.py b/scripts/check_kuma_ping_format.py new file mode 100644 index 0000000..02fa7ce --- /dev/null +++ b/scripts/check_kuma_ping_format.py @@ -0,0 +1,18 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + _, o, e = c.exec_command(cmd) + return o.read().decode().strip() + +pod = run("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'") +res = run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "USE kuma; SELECT * FROM monitor WHERE type='ping' LIMIT 2 \G" """) +print(res) + +c.close() + diff --git a/scripts/check_kuma_spacedock.py b/scripts/check_kuma_spacedock.py new file mode 100644 index 0000000..637a63f --- /dev/null +++ b/scripts/check_kuma_spacedock.py @@ -0,0 +1,26 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + _, o, e = c.exec_command(cmd) + return o.read().decode().strip() + +pod = run("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'") +print("Kuma Pod:", pod) + +# Check monitors related to harvester or spacedock +sql = "USE kuma; SELECT id, name, type, hostname, port, url, active FROM monitor WHERE name LIKE '%harvester%' OR name LIKE '%spacedock%' OR hostname LIKE '%140.44.4.8%' OR hostname LIKE '%140.44.44.12%';" +res = run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "{sql}" """) +print(res) + +# Also check notifications +notifs = run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "USE kuma; SELECT id, name, type, is_default, active FROM notification;" """) +print("\nNotifications:") +print(notifs) + +c.close() diff --git a/scripts/check_kuma_tags.py b/scripts/check_kuma_tags.py new file mode 100644 index 0000000..13cfc2e --- /dev/null +++ b/scripts/check_kuma_tags.py @@ -0,0 +1,32 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run_sql(sql): + pod_cmd = "kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'" + _, o, _ = c.exec_command(pod_cmd) + pod = o.read().decode().strip() + cmd = f"kubectl exec -i -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root" + stdin, stdout, stderr = c.exec_command(cmd) + stdin.write(sql) + stdin.close() + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- Tables ---") +out, _ = run_sql("USE kuma; SHOW TABLES LIKE '%tag%';") +print(out) + +print("--- Tags ---") +out, _ = run_sql("USE kuma; SELECT * FROM tag;") +print(out) + +print("--- Monitor Tags sample ---") +out, _ = run_sql("USE kuma; SELECT * FROM monitor_tag LIMIT 15;") +print(out) + +c.close() + diff --git a/scripts/check_kuma_types.py b/scripts/check_kuma_types.py new file mode 100644 index 0000000..b105cd5 --- /dev/null +++ b/scripts/check_kuma_types.py @@ -0,0 +1,21 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + _, o, e = c.exec_command(cmd) + return o.read().decode().strip() + +pod = run("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'") +res = run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "USE kuma; SELECT DISTINCT type FROM monitor;" """) +print("Types:", res) + +res2 = run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "USE kuma; SELECT id, name, type, hostname, port, interval FROM monitor LIMIT 5;" """) +print("Sample monitors:\n", res2) + +c.close() + diff --git a/scripts/check_lh_disks.py b/scripts/check_lh_disks.py new file mode 100644 index 0000000..a77d6f0 --- /dev/null +++ b/scripts/check_lh_disks.py @@ -0,0 +1,30 @@ +import paramiko +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Nodes in Longhorn (spec.disks vs status.diskStatus):") +out, _ = k8s("kubectl get nodes.longhorn.io -n longhorn-system -o json") +data = json.loads(out) +for node in data.get("items", []): + name = node["metadata"]["name"] + print(f"\n================ Node: {name} ================") + disks = node.get("spec", {}).get("disks", {}) + status_disks = node.get("status", {}).get("diskStatus", {}) + for dname, dspec in disks.items(): + dstat = status_disks.get(dname, {}) + print(f"Disk {dname}:") + print(f" Path: {dspec.get('path')}, AllowScheduling: {dspec.get('allowScheduling')}") + print(f" Conditions: {dstat.get('conditions')}") + print(f" Schedulable: {dstat.get('schedulable')}, StorageAvailable: {dstat.get('storageAvailable')}, StorageMax: {dstat.get('storageMaximum')}") + +c.close() + diff --git a/scripts/check_lh_now.py b/scripts/check_lh_now.py new file mode 100644 index 0000000..c229949 --- /dev/null +++ b/scripts/check_lh_now.py @@ -0,0 +1,27 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Longhorn Volumes:") +out, _ = k8s("kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,REPLICAS:.status.currentReplicas,NODE:.status.currentNodeID") +print(out) + +print("\nAI-Stack pods:") +out, _ = k8s("kubectl get pods -n ai-stack") +print(out) + +# Check why curl to longhorn failed +out, _ = k8s("kubectl get ep -n longhorn-system longhorn-backend") +print("\nLonghorn-backend endpoint:") +print(out) + +c.close() + diff --git a/scripts/check_lh_recovery.py b/scripts/check_lh_recovery.py new file mode 100644 index 0000000..15e9719 --- /dev/null +++ b/scripts/check_lh_recovery.py @@ -0,0 +1,34 @@ +import paramiko +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Checking Longhorn nodes disk conditions:") +out, _ = k8s("kubectl get nodes.longhorn.io -n longhorn-system -o json") +data = json.loads(out) +for item in data.get("items", []): + name = item["metadata"]["name"] + print(f"\nNode: {name}") + for dname, dstat in item.get("status", {}).get("diskStatus", {}).items(): + conds = {c.get("type"): c.get("status") for c in dstat.get("conditions", [])} + storage_avail = dstat.get("storageAvailable", 0) / (1024**3) + print(f" {dname}: Ready={conds.get('Ready')}, Schedulable={conds.get('Schedulable')}, Avail={storage_avail:.1f} GB") + +print("\nLonghorn Volumes Status:") +out, _ = k8s("kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness") +print(out) + +print("\nAI-stack pods:") +out, _ = k8s("kubectl get pods -n ai-stack") +print(out) + +c.close() + diff --git a/scripts/check_linux_template.py b/scripts/check_linux_template.py new file mode 100644 index 0000000..f9e2a39 --- /dev/null +++ b/scripts/check_linux_template.py @@ -0,0 +1,40 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1} + if auth: payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + +auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result") + +# Get templates +templates = zabbix_api("template.get", { + "output": ["templateid", "name"], + "search": {"name": "Linux"}, + "selectMacros": "extend" +}, auth=auth).get("result", []) + +for t in templates: + print(f"\nTemplate: {t['name']} (ID: {t['templateid']})") + for m in t.get("macros", []): + if "CPU" in m['macro'] or "MEM" in m['macro'] or "LOAD" in m['macro']: + print(f" Macro: {m['macro']} = {m['value']}") + # Get triggers in this template + triggers = zabbix_api("trigger.get", { + "output": ["triggerid", "description", "expression", "priority"], + "templateids": [t['templateid']], + "search": {"description": "utilization"} + }, auth=auth).get("result", []) + for tr in triggers: + print(f" Trigger: {tr['description']} | Expr: {tr['expression']} | Priority: {tr['priority']}") + diff --git a/scripts/check_local_raw_yml.py b/scripts/check_local_raw_yml.py new file mode 100644 index 0000000..96ded45 --- /dev/null +++ b/scripts/check_local_raw_yml.py @@ -0,0 +1,13 @@ +with open(r'c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\aerodrome\prometheus\prometheus.yaml', 'r', encoding='utf-8') as f: + text = f.read() + +start_marker = "prometheus.yml: |" +start_idx = text.find(start_marker) +end_idx = text.find("\n---", start_idx) +raw_yml = text[start_idx + len(start_marker):end_idx] + +print("Contains shipyard:", "shipyard" in raw_yml) +for line in raw_yml.splitlines(): + if "shipyard" in line or "spaceport" in line or "node-exporter" in line: + print(line) + diff --git a/scripts/check_mount_history.py b/scripts/check_mount_history.py new file mode 100644 index 0000000..63a678a --- /dev/null +++ b/scripts/check_mount_history.py @@ -0,0 +1,22 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Search for extra-disks on Node 1:") +out, _ = run_cmd("grep -rn 'extra-disks' /oem/ /etc/ /usr/local/ 2>/dev/null") +print(out) + +print("\nHistory on Node 1:") +out, _ = run_cmd("cat /home/rancher/.bash_history 2>/dev/null | tail -n 40") +print(out) + +c.close() + diff --git a/scripts/check_mount_unit.py b/scripts/check_mount_unit.py new file mode 100644 index 0000000..864c3bf --- /dev/null +++ b/scripts/check_mount_unit.py @@ -0,0 +1,22 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Content of var-lib-harvester-extrax2ddisks-sdb.mount:") +out, _ = run_cmd("cat $(readlink -f /etc/systemd/system/multi-user.target.wants/var-lib-harvester-extrax2ddisks-sdb.mount 2>/dev/null || echo /etc/systemd/system/multi-user.target.wants/var-lib-harvester-extrax2ddisks-sdb.mount)") +print(out) + +print("\nStatus of var-lib-harvester-extrax2ddisks-sdb.mount:") +out, err = run_cmd("systemctl status var-lib-harvester-extrax2ddisks-sdb.mount") +print(out, err) + +c.close() + diff --git a/scripts/check_ndm.py b/scripts/check_ndm.py new file mode 100644 index 0000000..ef87ce7 --- /dev/null +++ b/scripts/check_ndm.py @@ -0,0 +1,22 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Disk manager pods:") +out, _ = k8s("kubectl get pods -A | grep -i disk") +print(out) + +print("\nSample blockdevice YAML:") +out, _ = k8s("kubectl get blockdevices.harvesterhci.io -n longhorn-system 18c78622-16ec-4be2-a833-21ad879cb7b9 -o yaml") +print(out) + +c.close() + diff --git a/scripts/check_ndm_logs.py b/scripts/check_ndm_logs.py new file mode 100644 index 0000000..c786055 --- /dev/null +++ b/scripts/check_ndm_logs.py @@ -0,0 +1,18 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Harvester node-disk-manager logs:") +out, _ = k8s("kubectl logs -n harvester-system -l app.kubernetes.io/name=harvester-node-disk-manager --tail=50") +print(out[:2000]) + +c.close() + diff --git a/scripts/check_nfs_peanut.py b/scripts/check_nfs_peanut.py new file mode 100644 index 0000000..8e0e3da --- /dev/null +++ b/scripts/check_nfs_peanut.py @@ -0,0 +1,22 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- ls -la /mnt/nfs_shares/peanut ---") +out, _ = run("echo 'Katzir476!' | sudo -S ls -la /mnt/nfs_shares/peanut") +print(out) + +print("--- contents of files in /mnt/nfs_shares/peanut ---") +out, _ = run("echo 'Katzir476!' | sudo -S cat /mnt/nfs_shares/peanut/* 2>/dev/null") +print(out) + +c.close() + diff --git a/scripts/check_node5.py b/scripts/check_node5.py new file mode 100644 index 0000000..151675d --- /dev/null +++ b/scripts/check_node5.py @@ -0,0 +1,26 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +# Note task is now on snshpd4lnxswm05 (140.44.4.75) +c.connect('140.44.4.75', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +cid, _ = run("echo 'Katzir476!' | sudo -S docker ps -q -f name=peanut_peanut") +print("CID on 140.44.4.75:", cid) + +# Check what files are in the web app to see the API routes +out, _ = run(f"echo 'Katzir476!' | sudo -S docker exec {cid} node -e 'console.log(require(\"fs\").readdirSync(\"./\"))'") +print("App root files:\n", out) + +# Test connecting to nutserver from 140.44.4.75 as well to be 100% sure node 05 can also reach it +out, err = run("ping -c 2 -W 2 14.4.4.44") +print("Ping from 140.44.4.75 to 14.4.4.44:\n", out) + +c.close() + diff --git a/scripts/check_npm_conf4.py b/scripts/check_npm_conf4.py new file mode 100644 index 0000000..0dc5d89 --- /dev/null +++ b/scripts/check_npm_conf4.py @@ -0,0 +1,21 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +def run_npm(cmd): + full_cmd = f"echo Katzir476! | sudo -S docker exec $(docker ps -q -f name=nginx-proxy-manager) {cmd}" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("=== PORTAL (4.conf) ===") +out, _ = run_npm("cat /data/nginx/proxy_host/4.conf") +print(out) + +print("\n=== WILDCARD (*.snarfnet.net) (1.conf) ===") +out, _ = run_npm("cat /data/nginx/proxy_host/1.conf") +print(out) + +c.close() + diff --git a/scripts/check_npm_db_location.py b/scripts/check_npm_db_location.py new file mode 100644 index 0000000..f74dbc4 --- /dev/null +++ b/scripts/check_npm_db_location.py @@ -0,0 +1,32 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +py_code = """ +import sqlite3 +import json + +db = '/var/lib/docker/volumes/nginx-proxy-manager_data/_data/database.sqlite' +# Check if file exists there or where npm data is +import glob +files = glob.glob('/var/lib/docker/**/database.sqlite', recursive=True) +print('NPM DB files found:', files) +if files: + con = sqlite3.connect(files[0]) + cur = con.cursor() + for row in cur.execute('SELECT id, domain_names, forward_host, forward_port FROM proxy_host WHERE id=4').fetchall(): + print('Host 4:', row) +""" + +stdin, stdout, stderr = c.exec_command("cat > /tmp/find_npm_db.py") +stdin.write(py_code) +stdin.close() +stdout.channel.recv_exit_status() + +_, o, _ = c.exec_command("echo Katzir476! | sudo -S python3 /tmp/find_npm_db.py") +print(o.read().decode()) + +c.close() + diff --git a/scripts/check_npm_hosts.py b/scripts/check_npm_hosts.py index 880e7b2..92bb09e 100644 --- a/scripts/check_npm_hosts.py +++ b/scripts/check_npm_hosts.py @@ -1,18 +1,25 @@ import paramiko import sys +import json sys.stdout.reconfigure(encoding='utf-8') - -key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" c = paramiko.SSHClient() c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) -c.connect("140.44.4.71", username="root", key_filename=key_path, timeout=5) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) -cmd = """docker service ls""" -stdin, stdout, stderr = c.exec_command(cmd) -print("SERVICES:") -print(stdout.read().decode('utf-8')) +def run_npm_sql(sql): + cmd = f"""echo 'Katzir476!' | sudo -S docker exec -i $(echo 'Katzir476!' | sudo -S docker ps -qf name=nginx-proxy-manager_app) sqlite3 /data/database.sqlite "{sql}" """ + _, o, e = c.exec_command(cmd) + return o.read().decode().strip(), e.read().decode().strip() +# Check Proxy Host 4 +out, err = run_npm_sql("SELECT id, domain_names, forward_host, forward_port FROM proxy_host WHERE id=4;") +print("Proxy Host 4 in NPM:") +print(out) +# Check all proxy hosts with portal or heimdall +out2, _ = run_npm_sql("SELECT id, domain_names, forward_host, forward_port FROM proxy_host WHERE domain_names LIKE '%portal%' OR domain_names LIKE '%heimdall%';") +print("\nMatching Proxy Hosts:") +print(out2) c.close() diff --git a/scripts/check_npm_streams.py b/scripts/check_npm_streams.py index e06fb1b..9c8911a 100644 --- a/scripts/check_npm_streams.py +++ b/scripts/check_npm_streams.py @@ -1,11 +1,29 @@ import paramiko +import sys -ssh = paramiko.SSHClient() -ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) -ssh.connect('140.44.4.71', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) -stdin, stdout, stderr = ssh.exec_command("python3 -c \"import sqlite3; con = sqlite3.connect('/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite'); print(list(con.execute('SELECT id, incoming_port, forwarding_host, forwarding_port, is_deleted FROM stream')))\"") -print("STREAMS:", stdout.read().decode()) -print("ERR:", stderr.read().decode()) +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() -ssh.close() +check_streams = """ +import sqlite3 +conn = sqlite3.connect('/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite') +print("NPM STREAMS:") +for row in conn.execute("SELECT * FROM stream"): + print(row) +print("ALL TABLES:") +for row in conn.execute("SELECT name FROM sqlite_master WHERE type='table'"): + print(row) +conn.close() +""" +run(f"cat << 'EOF' > /tmp/check_streams.py\n{check_streams}\nEOF") +out, err = run("echo 'Katzir476!' | sudo -S python3 /tmp/check_streams.py") +print(out) +if err: print(err) + +c.close() diff --git a/scripts/check_nut_arp.py b/scripts/check_nut_arp.py new file mode 100644 index 0000000..0aa07be --- /dev/null +++ b/scripts/check_nut_arp.py @@ -0,0 +1,26 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('14.4.4.44', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("ip route get 140.44.4.74:") +out, _ = run("ip route get 140.44.4.74") +print(out) + +print("ip route get 140.44.4.70:") +out, _ = run("ip route get 140.44.4.70") +print(out) + +print("arp -n:") +out, _ = run("arp -n") +print(out) + +c.close() + diff --git a/scripts/check_nut_configs.py b/scripts/check_nut_configs.py new file mode 100644 index 0000000..6510b47 --- /dev/null +++ b/scripts/check_nut_configs.py @@ -0,0 +1,28 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +KEY_FILE = r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet' +IP = '14.4.4.44' + +ssh = paramiko.SSHClient() +ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +ssh.connect(IP, username='snarf', key_filename=KEY_FILE, timeout=10) + +def run(cmd): + print(f"\n>>> {cmd}") + _, o, e = ssh.exec_command(cmd) + out = o.read().decode().strip() + if out: + print(out) + return out + +run("uptime") +run("cat /etc/nut/nut.conf") +run("cat /etc/nut/ups.conf") +run("cat /etc/nut/upsmon.conf | grep -v '^#' | grep -v '^$'") +run("ps aux | grep -i nut") +run("ls -la /run/nut") + +ssh.close() + diff --git a/scripts/check_nut_firewall.py b/scripts/check_nut_firewall.py new file mode 100644 index 0000000..2dca049 --- /dev/null +++ b/scripts/check_nut_firewall.py @@ -0,0 +1,33 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('14.4.4.44', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- ping 14.4.4.20 from nutserver ---") +out, _ = run("ping -c 2 14.4.4.20") +print(out) + +print("--- ping 14.44.4.108 from nutserver ---") +out, _ = run("ping -c 2 14.44.4.108") +print(out) + +print("--- traceroute 140.44.4.74 from nutserver ---") +out, _ = run("traceroute -n -m 5 140.44.4.74 || tracepath -n 140.44.4.74") +print(out) + +print("--- ufw / iptables on nutserver ---") +out, _ = run("echo 'Katzir476!' | sudo -S ufw status verbose") +print("UFW:", out) + +out, _ = run("echo 'Katzir476!' | sudo -S nft list ruleset") +print("NFT:", out) + +c.close() + diff --git a/scripts/check_nut_neigh.py b/scripts/check_nut_neigh.py new file mode 100644 index 0000000..deeeb63 --- /dev/null +++ b/scripts/check_nut_neigh.py @@ -0,0 +1,26 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('14.4.4.44', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("ip neighbor:") +out, _ = run("ip neigh") +print(out) + +print("ping gateway 14.4.4.1:") +out, _ = run("ping -c 2 14.4.4.1") +print(out) + +print("ip neigh after ping:") +out, _ = run("ip neigh") +print(out) + +c.close() + diff --git a/scripts/check_nut_routes.py b/scripts/check_nut_routes.py new file mode 100644 index 0000000..1f3ded4 --- /dev/null +++ b/scripts/check_nut_routes.py @@ -0,0 +1,26 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('14.4.4.44', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- ping 140.44.4.1 (cluster gateway) ---") +out, _ = run("ping -c 2 140.44.4.1") +print(out) + +print("--- ping 1.1.1.1 (internet) ---") +out, _ = run("ping -c 2 1.1.1.1") +print(out) + +print("--- default route & ip route show table all ---") +out, _ = run("ip route show table all") +print(out) + +c.close() + diff --git a/scripts/check_nut_wifi.py b/scripts/check_nut_wifi.py new file mode 100644 index 0000000..f14e6e7 --- /dev/null +++ b/scripts/check_nut_wifi.py @@ -0,0 +1,22 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('14.4.4.44', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- iwconfig / nmcli on nutserver ---") +out, _ = run("iwconfig 2>&1 || nmcli 2>&1") +print(out) + +print("--- SSID / Network connection ---") +out, _ = run("echo 'Katzir476!' | sudo -S grep -h ssid /etc/wpa_supplicant/wpa_supplicant.conf /etc/NetworkManager/system-connections/* 2>/dev/null") +print(out) + +c.close() + diff --git a/scripts/check_nutserver.py b/scripts/check_nutserver.py new file mode 100644 index 0000000..8ae36d4 --- /dev/null +++ b/scripts/check_nutserver.py @@ -0,0 +1,43 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +try: + c.connect('14.4.4.44', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + print("Connected to 14.4.4.44 via key") +except Exception as e: + print("Key auth failed, trying password:", e) + c.connect('14.4.4.44', username='snarf', password='Katzir476!', timeout=10) + print("Connected to 14.4.4.44 via password") + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- Listening ports on nutserver ---") +out, _ = run("ss -tulpn | grep 3493 || netstat -tulpn | grep 3493") +print(out) + +print("\n--- /etc/nut/upsd.conf ---") +out, _ = run("echo 'Katzir476!' | sudo -S cat /etc/nut/upsd.conf | grep -v '^#' | grep -v '^$'") +print(out) + +print("\n--- /etc/nut/nut.conf ---") +out, _ = run("echo 'Katzir476!' | sudo -S cat /etc/nut/nut.conf | grep -v '^#' | grep -v '^$'") +print(out) + +print("\n--- /etc/nut/upsd.users ---") +out, _ = run("echo 'Katzir476!' | sudo -S cat /etc/nut/upsd.users | grep -v '^#' | grep -v '^$'") +print(out) + +print("\n--- /etc/nut/ups.conf ---") +out, _ = run("echo 'Katzir476!' | sudo -S cat /etc/nut/ups.conf | grep -v '^#' | grep -v '^$'") +print(out) + +print("\n--- upsc server-closet@localhost ---") +out, _ = run("upsc server-closet@localhost") +print(out[:300]) + +c.close() diff --git a/scripts/check_nutserver_net.py b/scripts/check_nutserver_net.py index 62368d1..7a45759 100644 --- a/scripts/check_nutserver_net.py +++ b/scripts/check_nutserver_net.py @@ -2,19 +2,32 @@ import paramiko import sys sys.stdout.reconfigure(encoding='utf-8') -key_path = r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet' - c = paramiko.SSHClient() c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) -c.connect("14.4.4.44", username="root", key_filename=key_path, timeout=10) +c.connect('14.4.4.44', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) def run(cmd): stdin, stdout, stderr = c.exec_command(cmd) - return stdout.read().decode('utf-8', errors='ignore').strip() + return stdout.read().decode().strip(), stderr.read().decode().strip() -print("iptables check:", run("iptables --version; which iptables nft ipset")) -print("ip forwarding:", run("cat /proc/sys/net/ipv4/ip_forward")) -print("canary test ping to gateway:", run("ping -c 2 14.4.4.1")) +print("--- nutserver ip a ---") +out, _ = run("ip a") +print(out) + +print("--- nutserver ip route ---") +out, _ = run("ip route") +print(out) + +print("--- nutserver ping gateway ---") +out, _ = run("ping -c 2 14.4.4.1 || ping -c 2 14.4.4.254") +print(out) + +print("--- nutserver iptables / nftables ---") +out, _ = run("echo 'Katzir476!' | sudo -S iptables -L -n -v") +print(out) + +print("--- nutserver ping 140.44.4.74 ---") +out, _ = run("ping -c 2 140.44.4.74") +print(out) c.close() - diff --git a/scripts/check_one_mount.py b/scripts/check_one_mount.py new file mode 100644 index 0000000..d83690a --- /dev/null +++ b/scripts/check_one_mount.py @@ -0,0 +1,26 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Files in /etc/systemd/system matching var-lib*:") +out, _ = run_cmd("ls -la /etc/systemd/system/var-lib*") +print(out) + +print("\nTry running systemctl start var-lib-harvester-extrax2ddisks-sdb.mount:") +out, err = run_cmd("systemctl start var-lib-harvester-extrax2ddisks-sdb.mount") +print("out:", out, "err:", err) + +print("\nStatus:") +out, _ = run_cmd("systemctl status var-lib-harvester-extrax2ddisks-sdb.mount") +print(out) + +c.close() + diff --git a/scripts/check_panel_targets.py b/scripts/check_panel_targets.py new file mode 100644 index 0000000..d9fe860 --- /dev/null +++ b/scripts/check_panel_targets.py @@ -0,0 +1,24 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +resp = requests.get("https://grafana.snarfnet.net/api/dashboards/uid/snarfnet-systems", auth=auth, verify=False) +dash = resp.json()["dashboard"] +print("--- Systems Dashboard Targets ---") +for p in dash.get("panels", []): + targets = p.get("targets", []) + if targets: + exprs = [t.get("expr") for t in targets] + print(f"Panel: {p.get('title')} -> Exprs: {exprs}") + +resp2 = requests.get("https://grafana.snarfnet.net/api/dashboards/uid/snarfnet-containers", auth=auth, verify=False) +dash2 = resp2.json()["dashboard"] +print("\n--- Containers Dashboard Targets ---") +for p in dash2.get("panels", []): + targets = p.get("targets", []) + if targets: + exprs = [t.get("expr") for t in targets] + print(f"Panel: {p.get('title')} -> Exprs: {exprs}") + diff --git a/scripts/check_peanut_access.py b/scripts/check_peanut_access.py new file mode 100644 index 0000000..d5cbac0 --- /dev/null +++ b/scripts/check_peanut_access.py @@ -0,0 +1,45 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + _, o, e = c.exec_command(cmd) + return o.read().decode().strip() + +# Check direct curl to port 8050 on Swarm VIP and nodes +for ip in ["140.44.4.70", "140.44.4.71", "140.44.4.74"]: + res = run(f"curl -s -o /dev/null -w '%{{http_code}}' http://{ip}:8050") + print(f"Curl http://{ip}:8050 -> HTTP {res}") + +# Check NPM database +script = ''' +import sqlite3 +conn = sqlite3.connect('/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite') +cur = conn.cursor() +rows = cur.execute("SELECT id, domain_names, forward_host, forward_port, enabled FROM proxy_host").fetchall() +for r in rows: + if any(k in str(r).lower() for k in ["peanut", "ups", "8050"]): + print(r) +conn.close() +''' +out = run(f"echo 'Katzir476!' | sudo -S python3 -c {repr(script)}") +print("\nNPM matching records:\n", out) + +# Check Heimdall tile for PeaNUT +heimdall_script = ''' +import sqlite3 +con = sqlite3.connect('/var/lib/docker-data/heimdall/config/www/app.sqlite') +cur = con.cursor() +row = cur.execute("SELECT id, title, url FROM items WHERE title LIKE '%peanut%' OR title LIKE '%nut%'").fetchall() +print("Heimdall tile for PeaNUT:", row) +con.close() +''' +out_h = run(f"echo 'Katzir476!' | sudo -S python3 -c {repr(heimdall_script)}") +print(out_h) + +c.close() + diff --git a/scripts/check_peanut_device.py b/scripts/check_peanut_device.py new file mode 100644 index 0000000..5bbf99e --- /dev/null +++ b/scripts/check_peanut_device.py @@ -0,0 +1,22 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- PeaNUT Container Logs ---") +out, _ = run("echo 'Katzir476!' | sudo -S docker service logs peanut_peanut --tail 30") +print(out) + +print("\n--- Curl http://140.44.4.70:8050/device/server-closet ---") +out, _ = run("curl -s http://140.44.4.70:8050/device/server-closet | grep -i -E 'battery|charge|runtime|cyberpower|or2200|server-closet' | head -n 30") +print("Matches in HTML:\n", out[:500]) + +c.close() + diff --git a/scripts/check_peanut_history.py b/scripts/check_peanut_history.py new file mode 100644 index 0000000..bbd884e --- /dev/null +++ b/scripts/check_peanut_history.py @@ -0,0 +1,22 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- History of /mnt/swarm_shared/legacy_volumes/peanut ---") +out, _ = run("ls -la /mnt/swarm_shared/legacy_volumes/peanut") +print(out) + +print("--- Any other peanut directories on NFS / storage ---") +out, _ = run("echo 'Katzir476!' | sudo -S find /mnt -iname '*peanut*' 2>/dev/null") +print(out) + +c.close() + diff --git a/scripts/check_peanut_logs.py b/scripts/check_peanut_logs.py new file mode 100644 index 0000000..01df27b --- /dev/null +++ b/scripts/check_peanut_logs.py @@ -0,0 +1,32 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd, timeout=10) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- Container logs on 140.44.4.74 ---") +cid, _ = run("echo 'Katzir476!' | sudo -S docker ps -q -f name=peanut_peanut") +print("CID:", cid) +if cid: + logs, _ = run(f"echo 'Katzir476!' | sudo -S docker logs --tail 30 {cid}") + print("LOGS:\n", logs) + +print("\n--- Files in /mnt/swarm_shared/legacy_volumes/peanut ---") +out, _ = run("echo 'Katzir476!' | sudo -S ls -la /mnt/swarm_shared/legacy_volumes/peanut") +print(out) + +# Check what SQLite or config files are inside /mnt/swarm_shared/legacy_volumes/peanut +out, _ = run("echo 'Katzir476!' | sudo -S find /mnt/swarm_shared/legacy_volumes/peanut -type f") +print("Files found:\n", out) + +# If db exists, check tables / contents +out, _ = run("python3 -c 'import urllib.request; print(urllib.request.urlopen(\"http://140.44.4.70:8050/api/devices\").read().decode())' 2>&1") +print("API /api/devices:", out) + +c.close() diff --git a/scripts/check_peanut_status.py b/scripts/check_peanut_status.py new file mode 100644 index 0000000..ad6068e --- /dev/null +++ b/scripts/check_peanut_status.py @@ -0,0 +1,33 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + print(f"\n>>> {cmd}") + _, o, e = c.exec_command(cmd) + out = o.read().decode().strip() + if out: + print(out) + return out + +print("--- PeaNUT Service Status ---") +run("echo 'Katzir476!' | sudo -S docker service ps peanut_peanut") +run("echo 'Katzir476!' | sudo -S docker service logs peanut_peanut --tail 30") + +print("\n--- Check PeaNUT NPM Proxy Host ---") +npm_query = """import sqlite3 +conn = sqlite3.connect('/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite') +cur = conn.cursor() +rows = cur.execute("SELECT id, domain_names, forward_host, forward_port, enabled FROM proxy_host WHERE domain_names LIKE '%peanut%' OR domain_names LIKE '%ups%' OR forward_port=8050").fetchall() +for r in rows: + print(r) +conn.close() +""" +run(f"""echo 'Katzir476!' | sudo -S python3 -c "{npm_query}" """) + +c.close() + diff --git a/scripts/check_pings_from_clusters.py b/scripts/check_pings_from_clusters.py new file mode 100644 index 0000000..f327f87 --- /dev/null +++ b/scripts/check_pings_from_clusters.py @@ -0,0 +1,28 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') + +hosts = [ + ('140.44.4.34', 'Aerodrome 04'), + ('140.44.4.41', 'Spaceport 01'), + ('140.44.4.71', 'Shipyard 01'), + ('140.44.4.81', 'Spacedock 01'), +] + +for ip, name in hosts: + try: + c = paramiko.SSHClient() + c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + if ip == '140.44.4.41': + c.connect(ip, username='root', password='dietpi', timeout=5) + else: + c.connect(ip, username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=5) + _, stdout, _ = c.exec_command("ping -c 2 -W 2 14.4.4.44") + res = stdout.read().decode().strip() + has_reply = "64 bytes from 14.4.4.44" in res + print(f"{name} ({ip}) -> 14.4.4.44: {'REPLIED' if has_reply else 'NO REPLY'}") + c.close() + except Exception as e: + print(f"{name} ({ip}) error: {e}") + diff --git a/scripts/check_pod_shipyard.py b/scripts/check_pod_shipyard.py new file mode 100644 index 0000000..16807d3 --- /dev/null +++ b/scripts/check_pod_shipyard.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl exec deploy/prometheus -- cat /etc/prometheus/prometheus.yml | grep -i shipyard") +print("Shipyard in pod /etc/prometheus/prometheus.yml:\n", o.read().decode().strip()) +c.close() + diff --git a/scripts/check_ports.py b/scripts/check_ports.py index 7ceaa21..eb85d59 100644 --- a/scripts/check_ports.py +++ b/scripts/check_ports.py @@ -1,24 +1,8 @@ -import socket - -hosts = [ - ('140.44.4.11', 9100, 'Aerodrome Master 01'), - ('140.44.4.41', 9100, 'Spaceport Master 01'), - ('140.44.4.71', 9100, 'Shipyard Swarm'), - ('140.44.4.81', 9100, 'Harvester Node 01'), - ('140.44.4.81', 9400, 'Harvester Node 01 DCGM'), - ('140.44.4.11', 10250, 'Aerodrome Kubelet'), - ('140.44.4.41', 10250, 'Spaceport Kubelet'), - ('140.44.4.60', 8888, 'Zabbix Web'), - ('140.44.4.64', 10051, 'Zabbix Server'), -] - -for ip, port, desc in hosts: - s = socket.socket() - s.settimeout(1.5) - try: - s.connect((ip, port)) - print(f"[OPEN] {ip}:{port} ({desc})") - s.close() - except Exception as e: - print(f"[CLOSED/ERR] {ip}:{port} ({desc}) - {e}") +import paramiko +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("echo 'Katzir476!' | sudo -S ss -tulpn | grep -E ':9100|:9101|:8080'") +print(o.read().decode().strip()) +c.close() diff --git a/scripts/check_ports_all.py b/scripts/check_ports_all.py new file mode 100644 index 0000000..fedc01b --- /dev/null +++ b/scripts/check_ports_all.py @@ -0,0 +1,15 @@ +import paramiko + +nodes = ['140.44.4.71', '140.44.4.72', '140.44.4.73', '140.44.4.74', '140.44.4.75'] +for ip in nodes: + try: + c = paramiko.SSHClient() + c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + c.connect(ip, username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=5) + _, o, _ = c.exec_command("echo 'Katzir476!' | sudo -S ss -tulpn | grep -E ':9100|:9101|:8080'") + out = o.read().decode().strip() + print(f"Node {ip}: {out if out else 'All clear'}") + c.close() + except Exception as e: + print(f"Node {ip} err: {e}") + diff --git a/scripts/check_prom_all_targets.py b/scripts/check_prom_all_targets.py new file mode 100644 index 0000000..98e9f95 --- /dev/null +++ b/scripts/check_prom_all_targets.py @@ -0,0 +1,15 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +def prom_query(q): + url = "https://grafana.snarfnet.net/api/datasources/proxy/1/api/v1/query" + resp = requests.get(url, params={"query": q}, auth=auth, verify=False, timeout=10) + return resp.json().get("data", {}).get("result", []) + +res = prom_query("count by (instance, job, cluster) (up)") +for r in res: + print(r.get("metric")) + diff --git a/scripts/check_prom_cadvisor_nodes.py b/scripts/check_prom_cadvisor_nodes.py new file mode 100644 index 0000000..39093e9 --- /dev/null +++ b/scripts/check_prom_cadvisor_nodes.py @@ -0,0 +1,16 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +def prom_query(q): + url = "https://grafana.snarfnet.net/api/datasources/proxy/1/api/v1/query" + resp = requests.get(url, params={"query": q}, auth=auth, verify=False, timeout=10) + return resp.json().get("data", {}).get("result", []) + +res = prom_query("count by (instance, node) (container_last_seen{container!=''})") +print("Container nodes in Prometheus:") +for r in res: + print(r.get("metric")) + diff --git a/scripts/check_prom_cm.py b/scripts/check_prom_cm.py new file mode 100644 index 0000000..5e4aeb1 --- /dev/null +++ b/scripts/check_prom_cm.py @@ -0,0 +1,8 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl get configmap prometheus-config -o yaml | grep -A 40 'job_name: node-exporter'") +print(o.read().decode().strip()) +c.close() diff --git a/scripts/check_prom_content.py b/scripts/check_prom_content.py new file mode 100644 index 0000000..a4ccab1 --- /dev/null +++ b/scripts/check_prom_content.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl exec deploy/prometheus -- grep -C 5 spaceport /etc/prometheus/prometheus.yml") +print(o.read().decode().strip()) +c.close() + diff --git a/scripts/check_prom_in_pod.py b/scripts/check_prom_in_pod.py new file mode 100644 index 0000000..c7bc429 --- /dev/null +++ b/scripts/check_prom_in_pod.py @@ -0,0 +1,11 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl exec deploy/prometheus -- cat /etc/prometheus/prometheus.yml") +content = o.read().decode().strip() +lines = content.splitlines() +for l in lines[76:125]: + print(l) +c.close() diff --git a/scripts/check_prom_labels.py b/scripts/check_prom_labels.py new file mode 100644 index 0000000..5889efc --- /dev/null +++ b/scripts/check_prom_labels.py @@ -0,0 +1,28 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +def prom_query(q): + url = "https://grafana.snarfnet.net/api/datasources/proxy/1/api/v1/query" + resp = requests.get(url, params={"query": q}, auth=auth, verify=False, timeout=10) + return resp.json().get("data", {}).get("result", []) + +res_nodes = prom_query("node_load1") +print(f"Node Exporter targets ({len(res_nodes)}):") +for r in res_nodes: + m = r.get("metric", {}) + print(f" instance: {m.get('instance')}, job: {m.get('job')}, nodename: {m.get('nodename')}, cluster: {m.get('cluster')}") + +res_containers = prom_query("count by (cluster, job, node) (container_last_seen{container!=''})") +print("\nContainer metric groupings:") +for r in res_containers: + print(" ", r.get("metric")) + +# Check all label values for 'cluster' or 'job' +res_jobs = prom_query("count by (job) (up)") +print("\nPrometheus jobs:") +for r in res_jobs: + print(" ", r.get("metric")) + diff --git a/scripts/check_prom_mounts.py b/scripts/check_prom_mounts.py new file mode 100644 index 0000000..2eb5743 --- /dev/null +++ b/scripts/check_prom_mounts.py @@ -0,0 +1,13 @@ +import paramiko +import json + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl get deploy prometheus -o json") +data = json.loads(o.read().decode().strip()) +c_spec = data.get("spec", {}).get("template", {}).get("spec", {}).get("containers", [])[0] +print("Args:", c_spec.get("args")) +print("VolumeMounts:", c_spec.get("volumeMounts")) +c.close() + diff --git a/scripts/check_prom_ns.py b/scripts/check_prom_ns.py new file mode 100644 index 0000000..5cf7851 --- /dev/null +++ b/scripts/check_prom_ns.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("kubectl get configmap -A | grep prometheus") +print(o.read().decode().strip()) +c.close() + diff --git a/scripts/check_real_heimdall_db.py b/scripts/check_real_heimdall_db.py new file mode 100644 index 0000000..a8a3a1b --- /dev/null +++ b/scripts/check_real_heimdall_db.py @@ -0,0 +1,46 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +py_code = ''' +import sqlite3 +import shutil +import time + +db_path = '/var/lib/docker-data/heimdall/config/www/app.sqlite' +backup_name = db_path + '.backup_live_' + str(int(time.time())) +shutil.copyfile(db_path, backup_name) +print('Backup created:', backup_name) + +con = sqlite3.connect(db_path) +cur = con.cursor() + +print('=== USERS ===') +for r in cur.execute('SELECT * FROM users').fetchall(): + print('User:', r) + +print('\\n=== ITEMS ===') +rows = cur.execute('SELECT id, title, url, pinned, user_id, apporder FROM items').fetchall() +print('Total items in DB:', len(rows)) +for r in rows: + print('Item:', r) + +print('\\n=== SETTINGS ===') +for r in cur.execute('SELECT * FROM settings').fetchall(): + print('Setting:', r) +''' + +stdin, stdout, stderr = c.exec_command("cat > /tmp/inspect_heimdall_real.py") +stdin.write(py_code) +stdin.close() +stdout.channel.recv_exit_status() + +_, o, e = c.exec_command("echo Katzir476! | sudo -S python3 /tmp/inspect_heimdall_real.py") +print(o.read().decode()) +if e.read().decode(): + print("ERR:", e.read().decode()) + +c.close() + diff --git a/scripts/check_real_heimdall_db2.py b/scripts/check_real_heimdall_db2.py new file mode 100644 index 0000000..cc4114a --- /dev/null +++ b/scripts/check_real_heimdall_db2.py @@ -0,0 +1,40 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +py_code = """ +import sqlite3 + +db_path = '/var/lib/docker-data/heimdall/config/www/app.sqlite' +con = sqlite3.connect(db_path) +cur = con.cursor() + +print('=== PRAGMA table_info(items) ===') +for col in cur.execute('PRAGMA table_info(items)').fetchall(): + print(col) + +print('=== ITEMS COUNT ===') +print('Count:', cur.execute('SELECT count(*) FROM items').fetchone()[0]) + +print('=== SAMPLE ITEMS ===') +for row in cur.execute('SELECT * FROM items').fetchall()[:10]: + print(row) + +print('=== SETTINGS ===') +for row in cur.execute('SELECT * FROM settings').fetchall(): + print(row) +""" + +stdin, stdout, stderr = c.exec_command("cat > /tmp/inspect_heimdall_real.py") +stdin.write(py_code) +stdin.close() +stdout.channel.recv_exit_status() + +_, o, e = c.exec_command("echo Katzir476! | sudo -S python3 /tmp/inspect_heimdall_real.py") +print(o.read().decode()) +print(e.read().decode()) + +c.close() + diff --git a/scripts/check_route.py b/scripts/check_route.py new file mode 100644 index 0000000..3fb9c91 --- /dev/null +++ b/scripts/check_route.py @@ -0,0 +1,23 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- Ping 14.4.4.44 from 140.44.4.74 ---") +out, err = run("ping -c 3 -W 2 14.4.4.44") +print("OUT:\n", out) +print("ERR:\n", err) + +print("--- ip route get 14.4.4.44 ---") +out, _ = run("ip route get 14.4.4.44") +print(out) + +c.close() + diff --git a/scripts/check_salvage_progress.py b/scripts/check_salvage_progress.py new file mode 100644 index 0000000..f6e21b4 --- /dev/null +++ b/scripts/check_salvage_progress.py @@ -0,0 +1,27 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Longhorn manager recent logs for salvage:") +out, _ = k8s("kubectl logs -n longhorn-system -l app=longhorn-manager --tail=50 | grep -i salvage") +print(out) + +print("\nReplica salvage status:") +out, _ = k8s("kubectl get replicas.longhorn.io -n longhorn-system -l longhornvolume=pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495 -o custom-columns=NAME:.metadata.name,STATE:.status.currentState,SALVAGE_REQ:.spec.salvageRequested,FAILED:.spec.failedAt") +print(out) + +# Check engine +out, _ = k8s("kubectl get engines.longhorn.io -n longhorn-system pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-e-0 -o yaml | grep -E 'salvage|state|replicas|status'") +print("\nEngine status:") +print(out) + +c.close() + diff --git a/scripts/check_services.py b/scripts/check_services.py new file mode 100644 index 0000000..9b21e9c --- /dev/null +++ b/scripts/check_services.py @@ -0,0 +1,28 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + out = stdout.read().decode('utf-8', errors='replace') + err = stderr.read().decode('utf-8', errors='replace') + return out.strip(), err.strip() + +print("Docker services:") +out, _ = run("docker service ls") +print(out) + +print("\nDocker stack ls:") +out, _ = run("docker stack ls") +print(out) + +print("\nDocker ps on manager (140.44.4.71):") +out, _ = run("docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'") +print(out) + +c.close() + diff --git a/scripts/check_settings_yml.py b/scripts/check_settings_yml.py new file mode 100644 index 0000000..d968584 --- /dev/null +++ b/scripts/check_settings_yml.py @@ -0,0 +1,36 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- settings.yml ---") +out, _ = run("cat /mnt/swarm_shared/legacy_volumes/peanut/settings.yml") +print(out) + +print("\n--- Test network from 140.44.4.74 to 14.4.4.44:3493 using python socket ---") +test_sock = """ +import socket +s = socket.socket() +s.settimeout(3) +try: + s.connect(('14.4.4.44', 3493)) + print('SUCCESS connecting to 14.4.4.44:3493') + s.sendall(b'VER\\n') + print('Response:', s.recv(1024)) +except Exception as e: + print('FAIL:', e) +finally: + s.close() +""" +out, _ = run(f"python3 -c {repr(test_sock)}") +print(out) + +c.close() + diff --git a/scripts/check_ship_neigh.py b/scripts/check_ship_neigh.py new file mode 100644 index 0000000..4636121 --- /dev/null +++ b/scripts/check_ship_neigh.py @@ -0,0 +1,18 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("ip neigh on 140.44.4.74:") +out, _ = run("ip neigh | grep 140.44.4.1") +print(out) + +c.close() + diff --git a/scripts/check_shipyard_containers.py b/scripts/check_shipyard_containers.py new file mode 100644 index 0000000..f1368ce --- /dev/null +++ b/scripts/check_shipyard_containers.py @@ -0,0 +1,16 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +def prom_query(q): + url = "https://grafana.snarfnet.net/api/datasources/proxy/1/api/v1/query" + resp = requests.get(url, params={"query": q}, auth=auth, verify=False, timeout=10) + return resp.json().get("data", {}).get("result", []) + +res = prom_query("count by (instance, image, name) (container_last_seen{cluster='shipyard'})") +print(f"Shipyard containers scraped ({len(res)}):") +for r in res[:10]: + print(" ", r.get("metric")) + diff --git a/scripts/check_shipyard_exporter.py b/scripts/check_shipyard_exporter.py new file mode 100644 index 0000000..73990b9 --- /dev/null +++ b/scripts/check_shipyard_exporter.py @@ -0,0 +1,9 @@ +import socket + +for ip in ["140.44.4.71", "140.44.4.72", "140.44.4.73"]: + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.settimeout(2) + res = s.connect_ex((ip, 9100)) + s.close() + print(f"Node {ip}:9100 -> {'OPEN' if res == 0 else 'CLOSED'}") + diff --git a/scripts/check_shipyard_net.py b/scripts/check_shipyard_net.py new file mode 100644 index 0000000..da2ca8f --- /dev/null +++ b/scripts/check_shipyard_net.py @@ -0,0 +1,22 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- ip a on 140.44.4.74 ---") +out, _ = run("ip a") +print(out) + +print("--- ip route on 140.44.4.74 ---") +out, _ = run("ip route") +print(out) + +c.close() + diff --git a/scripts/check_shipyard_nodes.py b/scripts/check_shipyard_nodes.py new file mode 100644 index 0000000..6c8d772 --- /dev/null +++ b/scripts/check_shipyard_nodes.py @@ -0,0 +1,24 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + _, o, e = c.exec_command(cmd) + out = o.read().decode().strip() + err = e.read().decode().strip() + if err: + print("ERR:", err) + return out + +print("--- Docker Swarm Nodes ---") +print(run("echo 'Katzir476!' | sudo -S docker node ls")) + +print("\n--- Existing Docker Swarm Stacks/Services ---") +print(run("echo 'Katzir476!' | sudo -S docker service ls")) + +c.close() + diff --git a/scripts/check_shipyard_prom_targets.py b/scripts/check_shipyard_prom_targets.py new file mode 100644 index 0000000..d36af08 --- /dev/null +++ b/scripts/check_shipyard_prom_targets.py @@ -0,0 +1,15 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +url = "https://grafana.snarfnet.net/api/datasources/proxy/1/api/v1/targets" +resp = requests.get(url, auth=auth, verify=False, timeout=10) +targets = resp.json().get("data", {}).get("activeTargets", []) +print(f"Total Active Targets: {len(targets)}") +jobs = set(t.get("labels", {}).get("job") for t in targets) +print("Active Jobs:", jobs) +for t in targets: + if "node" in t.get("labels", {}).get("job", "") or "shipyard" in t.get("labels", {}).get("job", ""): + print(f" Job: {t.get('labels', {}).get('job')} | Labels: {t.get('labels')} | Health: {t.get('health')}") diff --git a/scripts/check_shipyard_services.py b/scripts/check_shipyard_services.py new file mode 100644 index 0000000..3d7f255 --- /dev/null +++ b/scripts/check_shipyard_services.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("ps aux | grep -iE 'exporter|zabbix|telegraf|cadvisor'") +print(o.read().decode().strip()) +c.close() + diff --git a/scripts/check_spaceport_pods.py b/scripts/check_spaceport_pods.py new file mode 100644 index 0000000..e948bc6 --- /dev/null +++ b/scripts/check_spaceport_pods.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.41', username='root', password='dietpi', timeout=10) +_, o, _ = c.exec_command("kubectl get pods -A | grep -iE 'metrics|prom'") +print(o.read().decode().strip()) +c.close() + diff --git a/scripts/check_spaceport_ports.py b/scripts/check_spaceport_ports.py new file mode 100644 index 0000000..3be0e1a --- /dev/null +++ b/scripts/check_spaceport_ports.py @@ -0,0 +1,9 @@ +import socket + +for p in [10250, 10255, 9100]: + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.settimeout(2) + res = s.connect_ex(('140.44.4.41', p)) + s.close() + print(f"Spaceport 140.44.4.41:{p} -> {'OPEN' if res == 0 else 'CLOSED'}") + diff --git a/scripts/check_symlinks.py b/scripts/check_symlinks.py new file mode 100644 index 0000000..9e890f9 --- /dev/null +++ b/scripts/check_symlinks.py @@ -0,0 +1,18 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Listing multi-user.target.wants:") +out, _ = run_cmd("ls -la /etc/systemd/system/multi-user.target.wants/") +print(out) + +c.close() + diff --git a/scripts/check_systemd_units.py b/scripts/check_systemd_units.py new file mode 100644 index 0000000..4809b63 --- /dev/null +++ b/scripts/check_systemd_units.py @@ -0,0 +1,27 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Services in /etc/systemd/system:") +out, _ = run_cmd("ls -la /etc/systemd/system/") +print(out) + +print("\nFiles in /usr/local/bin or /opt:") +out, _ = run_cmd("ls -la /usr/local/bin /opt") +print(out) + +# Check if there is a disk-mount service or script +out, _ = run_cmd("find /etc/systemd/system -name '*mount*' -o -name '*disk*'") +print("\nSystemd mount/disk units:") +print(out) + +c.close() + diff --git a/scripts/check_tmp_prom.py b/scripts/check_tmp_prom.py new file mode 100644 index 0000000..68e1fb0 --- /dev/null +++ b/scripts/check_tmp_prom.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("cat /tmp/prometheus.yaml | grep -A 20 'job_name: shipyard-cadvisor'") +print(o.read().decode().strip()) +c.close() + diff --git a/scripts/check_trigger_functions.py b/scripts/check_trigger_functions.py new file mode 100644 index 0000000..6e2c65c --- /dev/null +++ b/scripts/check_trigger_functions.py @@ -0,0 +1,34 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1} + if auth: payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + +auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result") + +# Get full trigger detail for template 10343 and 10001 +triggers = zabbix_api("trigger.get", { + "output": "extend", + "templateids": ["10343", "10001"], + "search": {"description": "utilization"}, + "selectFunctions": "extend" +}, auth=auth).get("result", []) + +for t in triggers: + print(f"\nTrigger: {t['description']} (ID: {t['triggerid']}, Template: {t['templateid']})") + print(f" Expr: {t['expression']}") + print(f" Priority: {t['priority']}") + print(f" Comments: {t['comments']}") + print(f" Functions: {t.get('functions')}") + diff --git a/scripts/check_unifi_access.py b/scripts/check_unifi_access.py new file mode 100644 index 0000000..f0a19a8 --- /dev/null +++ b/scripts/check_unifi_access.py @@ -0,0 +1,22 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- curl -k https://140.44.4.1 from Shipyard ---") +out, _ = run("curl -k -I https://140.44.4.1") +print(out) + +print("--- curl -k https://14.44.4.1 from Shipyard ---") +out, _ = run("curl -k -I https://14.44.4.1") +print(out) + +c.close() + diff --git a/scripts/check_unifi_creds.py b/scripts/check_unifi_creds.py new file mode 100644 index 0000000..81b19d2 --- /dev/null +++ b/scripts/check_unifi_creds.py @@ -0,0 +1,18 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- Check UniFi in environment / secrets / git ---") +out, _ = run("grep -ri --exclude-dir='.git' 'unifi' /home/snarf/ /mnt/swarm_shared/ 2>/dev/null | head -n 30") +print(out) + +c.close() + diff --git a/scripts/check_unifi_login.py b/scripts/check_unifi_login.py new file mode 100644 index 0000000..c86142a --- /dev/null +++ b/scripts/check_unifi_login.py @@ -0,0 +1,21 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +session = requests.Session() +session.verify = False + +# Try logging into UniFi OS +for user in ['snarf', 'root', 'admin']: + for pw in ['Katzir476!', 'Katzir476!!!']: + try: + r = session.post('https://14.44.4.1/api/auth/login', json={'username': user, 'password': pw}, timeout=5) + print(f"Login {user} -> HTTP {r.status_code}") + if r.status_code == 200: + print("SUCCESS!", r.cookies) + break + except Exception as e: + print(f"Error {user}: {e}") + diff --git a/scripts/check_zabbix_noise.py b/scripts/check_zabbix_noise.py new file mode 100644 index 0000000..a47df41 --- /dev/null +++ b/scripts/check_zabbix_noise.py @@ -0,0 +1,70 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1} + if auth: payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + +auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result") + +# 1. Get recent problems in Zabbix +problems = zabbix_api("problem.get", { + "output": "extend", + "sortfield": ["eventid"], + "sortorder": "DESC", + "limit": 30 +}, auth=auth).get("result", []) + +print(f"Total active/recent problems: {len(problems)}") +for p in problems[:15]: + print(f"Problem: {p['name']} | Severity: {p['severity']} | Clock: {p['clock']}") + +# 2. Get triggers with High or Warning for CPU or Memory +triggers = zabbix_api("trigger.get", { + "output": ["triggerid", "description", "expression", "priority", "templateid"], + "filter": {"value": 1}, + "search": {"description": "CPU"}, + "limit": 20 +}, auth=auth).get("result", []) +print("\nActive CPU Triggers:") +for t in triggers: + print(f"ID {t['triggerid']}: {t['description']} -> {t['expression']}") + +triggers_mem = zabbix_api("trigger.get", { + "output": ["triggerid", "description", "expression", "priority", "templateid"], + "filter": {"value": 1}, + "search": {"description": "Memory"}, + "limit": 20 +}, auth=auth).get("result", []) +print("\nActive Memory Triggers:") +for t in triggers_mem: + print(f"ID {t['triggerid']}: {t['description']} -> {t['expression']}") + +# 3. Check global macros for CPU and Memory +macros = zabbix_api("usermacro.get", { + "output": "extend", + "globalmacro": True +}, auth=auth).get("result", []) +print("\nGlobal Macros:") +for m in macros: + if "CPU" in m['macro'] or "MEM" in m['macro']: + print(f" {m['macro']} = {m['value']}") + +# 4. Check host groups for Aerodrome, Shipyard, Spaceport +groups = zabbix_api("hostgroup.get", { + "output": ["groupid", "name"] +}, auth=auth).get("result", []) +print("\nHost Groups:") +for g in groups: + print(f" {g['groupid']}: {g['name']}") + diff --git a/scripts/check_zabbix_spacedock.py b/scripts/check_zabbix_spacedock.py new file mode 100644 index 0000000..390eed0 --- /dev/null +++ b/scripts/check_zabbix_spacedock.py @@ -0,0 +1,40 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1} + if auth: payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + +auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result") + +# Search hosts with idrac +hosts = zabbix_api("host.get", { + "output": ["hostid", "host", "name", "status"], + "selectInterfaces": ["ip", "port", "type"], + "selectParentTemplates": ["templateid", "name"], + "search": {"host": "idrac"} +}, auth=auth).get("result", []) + +for h in hosts: + ips = [i["ip"] for i in h.get("interfaces", [])] + tpls = [t["name"] for t in h.get("parentTemplates", [])] + print(f"Host: {h['host']} | Name: {h['name']} | IPs: {ips} | Templates: {tpls}") + +# Find ICMP template +tpls = zabbix_api("template.get", { + "output": ["templateid", "name"] +}, auth=auth).get("result", []) + +for t in tpls: + if "icmp" in t["name"].lower() or "ping" in t["name"].lower() or "idrac" in t["name"].lower() or "harvester" in t["name"].lower(): + print(f"Template {t['templateid']}: {t['name']}") diff --git a/scripts/check_zabbix_spacedock_items.py b/scripts/check_zabbix_spacedock_items.py new file mode 100644 index 0000000..c6bf779 --- /dev/null +++ b/scripts/check_zabbix_spacedock_items.py @@ -0,0 +1,31 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1} + if auth: payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + +auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result") + +hostids = ["10715", "10716", "10717", "10718", "10719", "10720", "10721", "10722"] +hosts = zabbix_api("host.get", { + "output": ["hostid", "host", "name", "status"], + "hostids": hostids, + "selectItems": ["itemid", "name", "lastvalue", "prevvalue", "status"] +}, auth=auth).get("result", []) + +for h in hosts: + items = h.get("items", []) + ping_items = [f"{i['name']}={i.get('lastvalue')}" for i in items] + print(f"Host {h['host']} ({h['name']}): Items count: {len(items)}, Values: {ping_items}") + diff --git a/scripts/check_zones.py b/scripts/check_zones.py new file mode 100644 index 0000000..26b659a --- /dev/null +++ b/scripts/check_zones.py @@ -0,0 +1,26 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +s = requests.Session() +s.verify = False + +r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'}) +csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token') +headers = {'X-CSRF-Token': csrf} if csrf else {} + +# Try various firewall zone endpoints +for ep in [ + '/proxy/network/v2/api/site/default/firewall/zones', + '/proxy/network/api/s/default/rest/firewallzone', + '/proxy/network/v2/api/site/default/firewall-policies', + '/proxy/network/v2/api/site/default/firewall/rulesets', + '/proxy/network/api/s/default/stat/routing' +]: + res = s.get('https://14.44.4.1' + ep, headers=headers) + print(ep, res.status_code) + if res.status_code == 200: + print(res.text[:500]) + diff --git a/scripts/complete_game_server_setup.py b/scripts/complete_game_server_setup.py new file mode 100644 index 0000000..a806651 --- /dev/null +++ b/scripts/complete_game_server_setup.py @@ -0,0 +1,139 @@ +import sys +import time +import paramiko + +HOST = "140.44.44.80" +USER = "root" +KEY_FILE = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +def main(): + print(f"Connecting to {USER}@{HOST}...") + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(HOST, username=USER, key_filename=KEY_FILE, timeout=10) + print("Connected!") + + def run(cmd, check=True): + print(f"\n>>> [RUN] {cmd}") + stdin, stdout, stderr = ssh.exec_command(cmd) + while not stdout.channel.exit_status_ready(): + time.sleep(0.5) + out = stdout.read().decode() + err = stderr.read().decode() + rc = stdout.channel.recv_exit_status() + if out: + print(out.strip()) + if err and "warning" not in err.lower() and "already exists" not in err.lower(): + print(f"[STDERR] {err.strip()}", file=sys.stderr) + if check and rc != 0: + print(f"[FAIL] Exit code: {rc}", file=sys.stderr) + return rc, out, err + + # ========================================== + # Step 1: Format and Mount Empty Disks + # ========================================== + print("\n--- Step 1: Formatting and Mounting Secondary Disks ---") + run("mkfs.ext4 -F -m 1 -L nvme-games /dev/nvme0n1") + run("mkfs.ext4 -F -m 1 -L storage-games /dev/sdb") + + # Get UUIDs + _, out_nvme, _ = run("blkid -s UUID -o value /dev/nvme0n1") + nvme_uuid = out_nvme.strip() + _, out_sdb, _ = run("blkid -s UUID -o value /dev/sdb") + sdb_uuid = out_sdb.strip() + print(f"NVMe UUID: {nvme_uuid}") + print(f"SDB UUID: {sdb_uuid}") + + run("mkdir -p /mnt/nvme-games /mnt/storage-games") + + # Update /etc/fstab + fstab_add = f""" +UUID={nvme_uuid} /mnt/nvme-games ext4 defaults,noatime,discard 0 2 +UUID={sdb_uuid} /mnt/storage-games ext4 defaults,noatime,discard 0 2 +""" + run(f"grep -qs '{nvme_uuid}' /etc/fstab || cat << 'EOF' >> /etc/fstab{fstab_add}EOF") + run("mount -a") + run("df -h /mnt/nvme-games /mnt/storage-games") + + # Move /opt/game-servers to NVMe and link / create directory structure + run("mkdir -p /mnt/nvme-games/game-servers") + run("cp -a /opt/game-servers/* /mnt/nvme-games/game-servers/ 2>/dev/null || true") + run("rm -rf /opt/game-servers") + run("ln -s /mnt/nvme-games/game-servers /opt/game-servers") + run("ln -s /mnt/storage-games /opt/game-storage 2>/dev/null || true") + run("chown -R snarf:snarf /mnt/nvme-games /mnt/storage-games") + run("chmod -R 775 /mnt/nvme-games/game-servers") + + # Ensure all game server subdirs exist on NVMe + run("mkdir -p /opt/game-servers/{enshrouded,fs25,stationeers,valheim,vrising}/{savegame,saves,worlds,data}") + run("chown -R snarf:snarf /opt/game-servers") + + # ========================================== + # Step 2: Install and Configure Zabbix Agent 2 + # ========================================== + print("\n--- Step 2: Installing and Configuring Zabbix Agent 2 ---") + run("DEBIAN_FRONTEND=noninteractive apt-get install -y zabbix-agent2 || true") + + # Configure PSK + run("mkdir -p /etc/zabbix") + run("echo 'a1142332ccfa9c53040e36171bb551ceb2865fb4051d203404df39e88ebe94f0' > /etc/zabbix/zabbix_agent2.psk") + run("chmod 600 /etc/zabbix/zabbix_agent2.psk") + run("chown zabbix:zabbix /etc/zabbix/zabbix_agent2.psk 2>/dev/null || true") + + zbx_conf = """Server=140.44.4.70,zabbix.snarfnet.net +ServerActive=140.44.4.70 +HostnameItem=system.hostname +HostMetadata=Linux +TLSConnect=psk +TLSAccept=psk +TLSPSKIdentity=PSK_autoregister +TLSPSKFile=/etc/zabbix/zabbix_agent2.psk +LogFile=/var/log/zabbix/zabbix_agent2.log +LogFileSize=10 +BufferSend=5 +BufferSize=100 +""" + run(f"cat << 'EOF' > /etc/zabbix/zabbix_agent2.conf\n{zbx_conf}\nEOF") + run("systemctl enable --now zabbix-agent2.service") + run("systemctl restart zabbix-agent2.service") + time.sleep(2) + run("systemctl is-active zabbix-agent2.service") + + # ========================================== + # Step 3: Game Seeds Environment & Compose + # ========================================== + print("\n--- Step 3: Configuring Game Server Passwords (.env) ---") + sftp = ssh.open_sftp() + sftp.put(r"C:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\snarfnet-game-seeds\core\manager.py", "/opt/snarfnet-game-seeds/core/manager.py") + sftp.close() + run("chmod +x /opt/snarfnet-game-seeds/core/manager.py") + + env_content = """# SnarfNet Game Server Passwords & Configuration +ENSHROUDED_SERVER_PASSWORD=SnarfNet2026! +SERVER_PASSWORD=SnarfNet2026! +FS25_SERVER_PASSWORD=SnarfNet2026! +FS25_ADMIN_PASSWORD=SnarfNetAdmin2026! +STATIONEERS_SERVER_PASSWORD=SnarfNet2026! +VALHEIM_SERVER_PASSWORD=SnarfNet2026! +SERVER_PASS=SnarfNet2026! +VRISING_SERVER_PASSWORD=SnarfNet2026! +""" + run(f"cat << 'EOF' > /opt/snarfnet-game-seeds/.env\n{env_content}\nEOF") + run("chown snarf:snarf /opt/snarfnet-game-seeds/.env && chmod 600 /opt/snarfnet-game-seeds/.env") + + # Regenerate Compose + run("cd /opt/snarfnet-game-seeds && python3 core/manager.py compose") + + # ========================================== + # Step 4: Pull Images and Launch Game Servers + # ========================================== + print("\n--- Step 4: Launching Docker Containers ---") + run("cd /opt/snarfnet-game-seeds && docker compose up -d") + time.sleep(5) + run("docker ps -a") + + ssh.close() + print("\n🎉 All steps completed successfully!") + +if __name__ == "__main__": + main() diff --git a/scripts/create_grafana_folders.py b/scripts/create_grafana_folders.py new file mode 100644 index 0000000..27952af --- /dev/null +++ b/scripts/create_grafana_folders.py @@ -0,0 +1,24 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +url = "https://grafana.snarfnet.net/api/folders" + +folders = [ + {"uid": "folder-aerodrome", "title": "Aerodrome"}, + {"uid": "folder-spaceport", "title": "Spaceport"}, + {"uid": "folder-shipyard", "title": "Shipyard"} +] + +for f in folders: + resp = requests.post(url, json=f, auth=auth, verify=False, timeout=10) + print(f"Folder {f['title']}: {resp.status_code} - {resp.text}") + +# List all folders +resp2 = requests.get(url, auth=auth, verify=False, timeout=10) +print("\nCurrent Folders:") +for item in resp2.json(): + print(f" ID: {item.get('id')} | UID: {item.get('uid')} | Title: {item.get('title')}") + diff --git a/scripts/deep_investigate.py b/scripts/deep_investigate.py new file mode 100644 index 0000000..0efe077 --- /dev/null +++ b/scripts/deep_investigate.py @@ -0,0 +1,155 @@ +import paramiko +import json +import urllib.request +import urllib.parse +import ssl + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +def run_ssh(host, user, password, cmd, sudo=False): + client = paramiko.SSHClient() + client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + client.connect(host, username=user, key_filename=SSH_KEY, password=password, timeout=10) + full_cmd = f"echo {password} | sudo -S sh -c '{cmd}'" if sudo else cmd + stdin, stdout, stderr = client.exec_command(full_cmd) + out = stdout.read().decode('utf-8', errors='ignore').strip() + err = stderr.read().decode('utf-8', errors='ignore').strip() + client.close() + return out, err + except Exception as e: + return "", str(e) + +print("=== 1. HARVESTER / LONGHORN VOLUME FAULT INSPECTION ===") +k8s_env = "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; " +# Check pods in ai-stack +out, _ = run_ssh("140.44.4.90", "rancher", "Katzir476!", k8s_env + "kubectl get pods -n ai-stack -o wide", sudo=True) +print("Pods in ai-stack:") +print(out) + +# Check Longhorn volume details for one of the faulted volumes +out, _ = run_ssh("140.44.4.90", "rancher", "Katzir476!", k8s_env + "kubectl get volume.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495 -n longhorn-system -o yaml | grep -E 'robustness|state|salvage|currentReplicas|conditions'", sudo=True) +print("Volume details (sample):") +print(out) + +# Check replicas for the volumes +out, _ = run_ssh("140.44.4.90", "rancher", "Katzir476!", k8s_env + "kubectl get replicas.longhorn.io -n longhorn-system", sudo=True) +print("Replicas summary:") +for line in out.splitlines()[:15]: + print(line) + +print("\n=== 2. HEIMDALL DATABASE & TILES INSPECTION ===") +# Find container ID on 140.44.4.71 +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker ps --filter name=heimdall -q", sudo=True) +cid = out.splitlines()[0] if out else "" +print(f"Heimdall Container ID: {cid}") +if cid: + # Inspect mounts + out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", f"docker inspect {cid} --format '{{{{json .Mounts}}}}'", sudo=True) + print("Mounts:", out) + # Check sqlite file location inside container + out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", f"docker exec {cid} ls -la /config/www", sudo=True) + print("Files in /config/www:") + print(out) + out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", f"docker exec {cid} sqlite3 /config/www/app.sqlite 'SELECT id, title, pinned, user_id FROM items;' 2>/dev/null", sudo=True) + print("Items in app.sqlite:") + print(out) + out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", f"docker exec {cid} sqlite3 /config/www/app.sqlite 'SELECT id, username, email FROM users;' 2>/dev/null", sudo=True) + print("Users in app.sqlite:") + print(out) + # Check host mount path files + out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "ls -la /var/lib/docker/volumes/heimdall_heimdall_config/_data", sudo=True) + print("Host volume data dir:") + print(out) + +print("\n=== 3. AUTHENTIK & NPM REVERSE PROXY FOR HEIMDALL ===") +# Check NPM config for heimdall +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker exec $(docker ps -q -f name=npm) grep -rnw '/data/nginx' -e 'heimdall' 2>/dev/null || docker exec $(docker ps -q -f name=proxy) grep -rn 'heimdall' /etc/nginx/conf.d/ 2>/dev/null", sudo=True) +print("NPM config search:") +print(out) + +# Check NPM proxy hosts +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker exec $(docker ps -q -f name=npm) find /data/nginx/proxy_host/ -type f -exec grep -l 'heimdall' {} + 2>/dev/null", sudo=True) +print("NPM proxy host file:") +print(out) +if out.strip(): + conf_file = out.strip().splitlines()[0] + out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", f"docker exec $(docker ps -q -f name=npm) cat {conf_file}", sudo=True) + print(f"NPM proxy host content ({conf_file}):") + print(out) + +print("\n=== 4. ZABBIX API - SPOCKDOCK MONITORING & ALERT THRESHOLDS ===") +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = { + "jsonrpc": "2.0", + "method": method, + "params": params, + "id": 1 + } + if auth: + payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + try: + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + except Exception as e: + return {"error": str(e)} + +auth_res = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}) +token = auth_res.get("result") +print("Zabbix Auth token obtained:", bool(token)) + +if token: + # 4a. Check Spacedock hosts + hosts = zabbix_api("host.get", { + "output": ["hostid", "host", "name", "status"], + "search": {"name": "spacedock"}, + "searchWildcardsEnabled": True + }, auth=token) + print("\nZabbix Spacedock hosts:") + print(json.dumps(hosts.get("result", []), indent=2)) + + # Also search by IP + hosts_ip = zabbix_api("host.get", { + "output": ["hostid", "host", "name", "status"], + "filter": {"ip": ["140.44.4.90", "140.44.4.81", "140.44.4.82", "140.44.4.83", "140.44.4.84"]}, + "selectInterfaces": ["ip", "type", "available"] + }, auth=token) + print("\nZabbix hosts by Harvester IPs:") + print(json.dumps(hosts_ip.get("result", []), indent=2)) + + # Check iDRAC hosts + hosts_idrac = zabbix_api("host.get", { + "output": ["hostid", "host", "name", "status"], + "search": {"name": "idrac"}, + "searchWildcardsEnabled": True, + "selectInterfaces": ["ip", "type", "available"] + }, auth=token) + print("\nZabbix iDRAC hosts:") + print(json.dumps(hosts_idrac.get("result", []), indent=2)) + + # 4b. Check CPU and Memory triggers firing or high noise + triggers = zabbix_api("trigger.get", { + "output": ["triggerid", "description", "expression", "priority", "status", "value"], + "search": {"description": "CPU"}, + "limit": 10 + }, auth=token) + print("\nSample CPU Triggers in Zabbix:") + for t in triggers.get("result", []): + print(f"ID: {t['triggerid']}, Desc: {t['description']}, Expression: {t['expression']}, Priority: {t['priority']}") + + triggers_mem = zabbix_api("trigger.get", { + "output": ["triggerid", "description", "expression", "priority", "status", "value"], + "search": {"description": "Memory"}, + "limit": 10 + }, auth=token) + print("\nSample Memory Triggers in Zabbix:") + for t in triggers_mem.get("result", []): + print(f"ID: {t['triggerid']}, Desc: {t['description']}, Expression: {t['expression']}, Priority: {t['priority']}") + diff --git a/scripts/deep_investigate_p1.py b/scripts/deep_investigate_p1.py new file mode 100644 index 0000000..34ac690 --- /dev/null +++ b/scripts/deep_investigate_p1.py @@ -0,0 +1,87 @@ +import paramiko +import json +import urllib.request +import ssl + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +def run_ssh(host, user, password, cmd, sudo=False): + client = paramiko.SSHClient() + client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + client.connect(host, username=user, key_filename=SSH_KEY, password=password, timeout=10) + full_cmd = f"echo {password} | sudo -S sh -c '{cmd}'" if sudo else cmd + stdin, stdout, stderr = client.exec_command(full_cmd) + out = stdout.read().decode('utf-8', errors='ignore').strip() + err = stderr.read().decode('utf-8', errors='ignore').strip() + client.close() + return out, err + except Exception as e: + return "", str(e) + +print("=== 1. CHECK LONGHORN VOLUME FAULTS & SALVAGE ===") +k8s_env = "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; " +cmd = k8s_env + "kubectl get volume.longhorn.io -n longhorn-system -o json" +out, _ = run_ssh("140.44.4.90", "rancher", "Katzir476!", cmd, sudo=True) +try: + vols = json.loads(out) + for v in vols.get("items", []): + name = v["metadata"]["name"] + status = v.get("status", {}) + spec = v.get("spec", {}) + print(f"Volume: {name}") + print(f" State: {status.get('state')}, Robustness: {status.get('robustness')}") + print(f" SalvageRequested: {status.get('salvageRequested')}, Conditions: {status.get('conditions')}") +except Exception as e: + print("Error parsing volumes json:", e) + +# Check Longhorn manager pod logs +out, _ = run_ssh("140.44.4.90", "rancher", "Katzir476!", k8s_env + "kubectl logs -n longhorn-system -l app=longhorn-manager --tail=30", sudo=True) +print("\nLonghorn manager recent logs:") +print(out[:1000]) + +print("\n=== 2. HEIMDALL MOUNT & SQLITE BACKUP ===") +# Inspect Heimdall service spec on 140.44.4.71 +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker service inspect heimdall_heimdall --format '{{json .Spec.TaskTemplate.ContainerSpec.Mounts}}'", sudo=True) +print("Heimdall Service Mounts:", out) + +# Check running container details +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker inspect $(docker ps -q -f name=heimdall) --format '{{json .Mounts}}'", sudo=True) +print("Heimdall Container Mounts:", out) + +# Check SQLite DB using python on the host +py_check_db = """ +python3 -c " +import sqlite3 +import shutil + +src = '/var/lib/docker/volumes/heimdall_heimdall_config/_data/www/app.sqlite' +# Backup first! +shutil.copyfile(src, src + '.bak_' + str(int(__import__('time').time()))) +print('Backup created!') + +conn = sqlite3.connect(src) +c = conn.cursor() +print('Users:') +for row in c.execute('SELECT id, username, email, public_front FROM users').fetchall(): + print(' ', row) +print('Items count:', c.execute('SELECT count(*) FROM items').fetchone()[0]) +print('Pinned items count:', c.execute('SELECT count(*) FROM items WHERE pinned=1').fetchone()[0]) +print('Items list:') +for row in c.execute('SELECT id, title, pinned, user_id, apporder FROM items').fetchall(): + print(' ', row) +" +""" +out, err = run_ssh("140.44.4.71", "snarf", "Katzir476!", py_check_db, sudo=True) +print("Heimdall DB check output:") +print(out if out else err) + +# Check heimdall container logs from the restart 5 hours ago +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker logs $(docker ps -q -f name=heimdall) --tail=60", sudo=True) +print("\nHeimdall Container logs:") +print(out) + +print("\n=== 3. DOCKER CONTAINERS ON 140.44.4.71 (Proxy, Kuma, etc.) ===") +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'", sudo=True) +print(out) + diff --git a/scripts/deep_investigate_p2.py b/scripts/deep_investigate_p2.py new file mode 100644 index 0000000..fc72045 --- /dev/null +++ b/scripts/deep_investigate_p2.py @@ -0,0 +1,74 @@ +import paramiko +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +def run_ssh(host, user, password, cmd, sudo=False): + client = paramiko.SSHClient() + client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + client.connect(host, username=user, key_filename=SSH_KEY, password=password, timeout=10) + full_cmd = f"echo {password} | sudo -S sh -c '{cmd}'" if sudo else cmd + stdin, stdout, stderr = client.exec_command(full_cmd) + out = stdout.read().decode('utf-8', errors='ignore').strip() + err = stderr.read().decode('utf-8', errors='ignore').strip() + client.close() + return out, err + except Exception as e: + return "", str(e) + +print("--- A. CHECK LONGHORN REPLICA DETAILS ---") +k8s_env = "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; " +# Look at one replica of pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495 +out, _ = run_ssh("140.44.4.90", "rancher", "Katzir476!", k8s_env + "kubectl get replica.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-r-3a9069af -n longhorn-system -o yaml", sudo=True) +print("Replica YAML sample:") +print(out) + +# Check all volume specs +out, _ = run_ssh("140.44.4.90", "rancher", "Katzir476!", k8s_env + "kubectl get engines.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,CURRENT_STATE:.status.currentState,SALVAGE:.status.salvageRequested", sudo=True) +print("\nEngines status:") +print(out) + +print("\n--- B. HEIMDALL SQLITE INSPECTION ---") +# On 140.44.4.71, let's create a small python script and run it +py_script = """ +import sqlite3 +import shutil +import glob + +db_path = '/var/lib/docker/volumes/heimdall_heimdall_config/_data/www/app.sqlite' +shutil.copyfile(db_path, db_path + '.backup_today') +print('Backed up successfully to app.sqlite.backup_today') + +conn = sqlite3.connect(db_path) +cur = conn.cursor() +print('Tables:') +for r in cur.execute("SELECT name FROM sqlite_master WHERE type='table'").fetchall(): + print(' ', r[0]) + +print('\\nUsers table:') +for r in cur.execute("SELECT id, username, email, public_front FROM users").fetchall(): + print(' ', r) + +print('\\nItems table count:', cur.execute("SELECT count(*) FROM items").fetchone()[0]) +print('\\nItems:') +for r in cur.execute("SELECT id, title, pinned, user_id, order_column FROM (SELECT id, title, pinned, user_id, apporder as order_column FROM items)").fetchall(): + print(' ', r) + +print('\\nSettings table:') +for r in cur.execute("SELECT key, value FROM settings").fetchall(): + print(' ', r) +""" +run_ssh("140.44.4.71", "snarf", "Katzir476!", "cat << 'EOF' > /tmp/check_heimdall.py\n" + py_script + "\nEOF", sudo=True) +out, err = run_ssh("140.44.4.71", "snarf", "Katzir476!", "python3 /tmp/check_heimdall.py", sudo=True) +print(out if out else err) + +print("\n--- C. NPM PROXY CONFIG FOR HEIMDALL ---") +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker exec $(docker ps -q -f name=nginx-proxy-manager) grep -rnw '/data/nginx' -e 'heimdall' 2>/dev/null", sudo=True) +print("NPM search result:") +print(out) + +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker exec $(docker ps -q -f name=nginx-proxy-manager) cat /data/nginx/proxy_host/1.conf 2>/dev/null || docker exec $(docker ps -q -f name=nginx-proxy-manager) ls -la /data/nginx/proxy_host", sudo=True) +print("NPM proxy host list:") +print(out) + diff --git a/scripts/deploy_game_server.py b/scripts/deploy_game_server.py new file mode 100644 index 0000000..9615898 --- /dev/null +++ b/scripts/deploy_game_server.py @@ -0,0 +1,173 @@ +import os +import sys +import time +from pathlib import Path +import paramiko + +HOST = "140.44.44.80" +USER = "root" +KEY_FILE = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +LOCAL_SEEDS_DIR = Path(r"C:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\snarfnet-game-seeds") + +def main(): + print(f"Connecting to {USER}@{HOST} with SSH key...") + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(HOST, username=USER, key_filename=KEY_FILE, timeout=10) + print("Connected successfully!") + + def run(cmd, check=True): + print(f"\n>>> [RUN] {cmd}") + stdin, stdout, stderr = ssh.exec_command(cmd) + while not stdout.channel.exit_status_ready(): + time.sleep(0.5) + out = stdout.read().decode() + err = stderr.read().decode() + rc = stdout.channel.recv_exit_status() + if out: + print(out.strip()) + if err and "warning" not in err.lower(): + print(f"[STDERR] {err.strip()}", file=sys.stderr) + if check and rc != 0: + print(f"[FAIL] Exit code: {rc}", file=sys.stderr) + return rc, out, err + + # Step 1: Configure Persistent DNS in systemd-resolved + print("\n--- Step 1: Persist DNS in systemd-resolved ---") + dns_conf = """[Resolve] +DNS=1.1.1.1 8.8.8.8 +FallbackDNS=1.0.0.1 8.8.4.4 +Domains=snarfnet.net +""" + run(f"cat << 'EOF' > /etc/systemd/resolved.conf.d/snarfnet-dns.conf\n{dns_conf}\nEOF") + run("systemctl restart systemd-resolved") + run("resolvectl dns enp0s31f6 1.1.1.1 8.8.8.8 || true") + + # Step 2: Expand LVM to use all 1.8TB on sda + print("\n--- Step 2: Expand LVM to full 1.8TB disk ---") + run("lvextend -l +100%FREE /dev/ubuntu-vg/ubuntu-lv -r || true") + run("df -h /") + + # Step 3: Install Essential Packages + print("\n--- Step 3: Apt Update & Install Tools ---") + run("DEBIAN_FRONTEND=noninteractive apt-get update") + run("DEBIAN_FRONTEND=noninteractive apt-get install -y curl wget git jq ufw nfs-common ethtool net-tools python3 python3-yaml python3-pip apt-transport-https ca-certificates gnupg lsb-release") + + # Step 4: Configure Wake-on-LAN persistently + print("\n--- Step 4: Configure Wake-on-LAN ---") + run("ethtool -s enp0s31f6 wol g || true") + wol_svc = """[Unit] +Description=Enable Wake-on-LAN on enp0s31f6 +After=network.target + +[Service] +Type=oneshot +ExecStart=/sbin/ethtool -s enp0s31f6 wol g + +[Install] +WantedBy=basic.target +""" + run(f"cat << 'EOF' > /etc/systemd/system/wol.service\n{wol_svc}\nEOF") + run("systemctl daemon-reload && systemctl enable wol.service") + + # Step 5: Mount Synology NAS Game Backup Share + print("\n--- Step 5: Configure Synology NAS NFS Mount ---") + run("mkdir -p /mnt/nas/game-backups") + run("grep -qs '/mnt/nas/game-backups' /etc/fstab || echo '14.10.10.71:/volume1/docker/game-backups /mnt/nas/game-backups nfs rw,hard,intr,nolock,timeo=14 0 0' >> /etc/fstab") + run("mount -a || true") + run("df -h /mnt/nas/game-backups") + + # Step 6: Install Docker & Compose + print("\n--- Step 6: Install Docker Engine & Compose ---") + rc, out, _ = run("docker --version", check=False) + if rc != 0: + run("curl -fsSL https://get.docker.com | sh") + run("systemctl enable docker && systemctl start docker") + run("usermod -aG docker snarf") + run("docker --version && docker compose version") + + # Step 7: Prepare Directory Structure + print("\n--- Step 7: Game Server Storage Directories ---") + run("mkdir -p /opt/game-servers/{enshrouded,fs25,stationeers,valheim,vrising}/{savegame,saves,worlds,data}") + run("mkdir -p /opt/snarfnet-game-seeds") + run("chown -R snarf:snarf /opt/game-servers") + run("chmod -R 775 /opt/game-servers") + + # Step 8: Upload snarfnet-game-seeds + print("\n--- Step 8: Upload snarfnet-game-seeds repo ---") + sftp = ssh.open_sftp() + + def put_dir(local_path, remote_path): + for item in os.listdir(local_path): + s = os.path.join(local_path, item) + d = f"{remote_path}/{item}" + if os.path.isdir(s): + try: + sftp.mkdir(d) + except Exception: + pass + put_dir(s, d) + else: + sftp.put(s, d) + print(f"Uploaded: {d}") + + try: + sftp.mkdir("/opt/snarfnet-game-seeds") + except Exception: + pass + put_dir(str(LOCAL_SEEDS_DIR), "/opt/snarfnet-game-seeds") + sftp.close() + + run("chown -R snarf:snarf /opt/snarfnet-game-seeds") + run("chmod +x /opt/snarfnet-game-seeds/core/manager.py /opt/snarfnet-game-seeds/scripts/*.sh || true") + + # Step 9: Configure Backup Systemd Timer + print("\n--- Step 9: Configure Automated Save Backup Timer ---") + backup_svc = """[Unit] +Description=SnarfNet Declarative Game Server Save Backup +After=network.target docker.service + +[Service] +Type=oneshot +WorkingDirectory=/opt/snarfnet-game-seeds +ExecStart=/usr/bin/python3 /opt/snarfnet-game-seeds/core/manager.py backup + +[Install] +WantedBy=multi-user.target +""" + backup_timer = """[Unit] +Description=Nightly Game Save Backup Timer (04:00 AM) + +[Timer] +OnCalendar=*-*-* 04:00:00 +Persistent=true + +[Install] +WantedBy=timers.target +""" + run(f"cat << 'EOF' > /etc/systemd/system/gameseed-backup.service\n{backup_svc}\nEOF") + run(f"cat << 'EOF' > /etc/systemd/system/gameseed-backup.timer\n{backup_timer}\nEOF") + run("systemctl daemon-reload && systemctl enable --now gameseed-backup.timer") + + # Step 10: Generate Docker Compose + print("\n--- Step 10: Generate docker-compose.yml from Seeds ---") + run("cd /opt/snarfnet-game-seeds && python3 core/manager.py plan") + run("cd /opt/snarfnet-game-seeds && python3 core/manager.py compose") + + # Step 11: SSH Security Hardening (Disable password login, enforce SSH keys) + print("\n--- Step 11: Enforce SSH Key Authentication ---") + ssh_dropin = """# SnarfNet Hardening: Enforce SSH Key authentication +PasswordAuthentication no +KbdInteractiveAuthentication no +PubkeyAuthentication yes +AuthorizedKeysFile .ssh/authorized_keys +""" + run(f"cat << 'EOF' > /etc/ssh/sshd_config.d/99-snarfnet-keys-only.conf\n{ssh_dropin}\nEOF") + run("systemctl restart ssh || systemctl restart sshd") + print("SSH PasswordAuthentication disabled; SSH Key authentication enforced!") + + ssh.close() + print("\n🎉 Deployment completed successfully!") + +if __name__ == "__main__": + main() diff --git a/scripts/deploy_mount_units_all_nodes.py b/scripts/deploy_mount_units_all_nodes.py new file mode 100644 index 0000000..d1f2359 --- /dev/null +++ b/scripts/deploy_mount_units_all_nodes.py @@ -0,0 +1,146 @@ +import paramiko +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +nodes = { + "snaisvr44hrvstr01": "140.44.4.81", + "snaisvr44hrvstr02": "140.44.4.82", + "snaisvr44hrvstr03": "140.44.4.83", + "snaisvr44hrvstr04": "140.44.4.84" +} + +# Python script that runs on each node +node_script = """python3 - << 'PYEOF' +import os +import subprocess +import json +import glob + +# Clean up any bad units first +for f in glob.glob("/etc/systemd/system/var-lib-harvester-extra*"): + if os.path.isfile(f) or os.path.islink(f): + os.remove(f) + +# 1. Map physical disks to their Longhorn diskUUID +out = subprocess.check_output("blkid", shell=True).decode() +disks = {} +for line in out.splitlines(): + if not line.startswith("/dev/sd"): continue + dev = line.split(":")[0] + if "sda" in dev: continue + uuid = "" + for part in line.split(): + if part.startswith("UUID="): + uuid = part.split('"')[1] + if uuid: + disks[dev] = uuid + +disk_info = {} # diskUUID -> fs_uuid +for dev, uuid in disks.items(): + tmp = f"/tmp/m_{os.path.basename(dev)}" + os.makedirs(tmp, exist_ok=True) + try: + subprocess.check_call(f"mount {dev} {tmp}", shell=True) + cfg_file = os.path.join(tmp, "longhorn-disk.cfg") + if os.path.exists(cfg_file): + with open(cfg_file) as f: + data = json.load(f) + lh_uuid = data.get("diskUUID") + disk_info[lh_uuid] = uuid + subprocess.call(f"umount {tmp}", shell=True) + except Exception as e: + pass + +# 2. Read expected from /tmp/node_expected.json +with open("/tmp/node_expected.json") as f: + expected = json.load(f) + +created_units = [] +for disk_name, info in expected.items(): + lh_uuid = info["diskUUID"] + path = info["path"] + if not path or path == "/var/lib/harvester/defaultdisk": + continue + + if lh_uuid in disk_info: + fs_uuid = disk_info[lh_uuid] + os.makedirs(path, exist_ok=True) + # Use systemd-escape binary to get the exact canonical unit filename + unit_name = subprocess.check_output(["systemd-escape", "-p", "--suffix=mount", path]).decode().strip() + unit_path = os.path.join("/etc/systemd/system", unit_name) + + unit_content = f\"\"\"[Unit] +Description=Harvester Extra Disk {disk_name} +Before=rke2-server.service rke2-agent.service + +[Mount] +What=/dev/disk/by-uuid/{fs_uuid} +Where={path} +Type=ext4 +Options=defaults + +[Install] +WantedBy=multi-user.target +\"\"\" + with open(unit_path, "w") as uf: + uf.write(unit_content) + + created_units.append(unit_name) + # Ensure it is mounted right now + mounts = subprocess.check_output("mount", shell=True).decode() + if path not in mounts: + subprocess.call(f"mount /dev/disk/by-uuid/{fs_uuid} {path}", shell=True) + +subprocess.call(["systemctl", "daemon-reload"]) +for u in created_units: + subprocess.call(["systemctl", "enable", u]) + subprocess.call(["systemctl", "start", u]) + +print("Configured & mounted units:", created_units) +PYEOF +""" + +# Fetch latest expected disks from Harvester VIP +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) +_, o, _ = c.exec_command("echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; kubectl get nodes.longhorn.io -n longhorn-system -o json'") +data = json.loads(o.read().decode()) +c.close() + +all_expected = {} +for node in data.get("items", []): + nname = node["metadata"]["name"] + all_expected[nname] = {} + for dname, dstat in node.get("status", {}).get("diskStatus", {}).items(): + all_expected[nname][dname] = { + "path": node.get("spec", {}).get("disks", {}).get(dname, {}).get("path"), + "diskUUID": dstat.get("diskUUID") + } + +for hname, ip in nodes.items(): + print(f"\n=================== Deploying to {hname} ({ip}) ===================") + expected_for_node = all_expected.get(hname, {}) + nc = paramiko.SSHClient() + nc.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + nc.connect(ip, username="rancher", key_filename=SSH_KEY, timeout=10) + # Write /tmp/node_expected.json via stdin + stdin, stdout, stderr = nc.exec_command("cat > /tmp/node_expected.json") + stdin.write(json.dumps(expected_for_node)) + stdin.close() + stdout.channel.recv_exit_status() + + _, o, e = nc.exec_command(f"echo Katzir476! | sudo -S {node_script}") + print(o.read().decode()) + if e.read().decode(): + pass + + _, o, _ = nc.exec_command("df -h | grep extra-disks") + print(f"Active mounts on {hname}:\n" + o.read().decode()) + nc.close() + except Exception as e: + print(f"Error on {hname}: {e}") + +print("\nAll nodes deployed successfully!") diff --git a/scripts/deploy_shipyard_monitoring.py b/scripts/deploy_shipyard_monitoring.py new file mode 100644 index 0000000..9de8841 --- /dev/null +++ b/scripts/deploy_shipyard_monitoring.py @@ -0,0 +1,71 @@ +import paramiko +import sys +import time + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +compose = """version: '3.8' + +services: + node-exporter: + image: prom/node-exporter:v1.8.2 + command: + - '--path.procfs=/host/proc' + - '--path.sysfs=/host/sys' + - '--path.rootfs=/host/root' + - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' + volumes: + - /proc:/host/proc:ro + - /sys:/host/sys:ro + - /:/host/root:ro + ports: + - target: 9100 + published: 9100 + protocol: tcp + mode: host + deploy: + mode: global + restart_policy: + condition: on-failure + + cadvisor: + image: gcr.io/cadvisor/cadvisor:v0.49.1 + command: + - '--housekeeping_interval=10s' + - '--docker_only=true' + - '--disable_metrics=disk,tcp,udp,percpu,sched,process' + volumes: + - /:/rootfs:ro + - /var/run:/var/run:ro + - /sys:/sys:ro + - /var/lib/docker/:/var/lib/docker:ro + - /dev/disk/:/dev/disk:ro + ports: + - target: 8080 + published: 8080 + protocol: tcp + mode: host + deploy: + mode: global + restart_policy: + condition: on-failure +""" + +sftp = c.open_sftp() +with sftp.file('/tmp/monitoring-stack.yaml', 'w') as f: + f.write(compose) +sftp.close() + +stdin, stdout, stderr = c.exec_command("echo 'Katzir476!' | sudo -S docker stack deploy -c /tmp/monitoring-stack.yaml monitoring") +print("Deploy output:\n", stdout.read().decode().strip()) +print("Deploy err:\n", stderr.read().decode().strip()) + +time.sleep(5) +stdin, stdout, _ = c.exec_command("echo 'Katzir476!' | sudo -S docker stack services monitoring") +print("Stack services:\n", stdout.read().decode().strip()) + +c.close() + diff --git a/scripts/diagnose_peanut.py b/scripts/diagnose_peanut.py new file mode 100644 index 0000000..2413d40 --- /dev/null +++ b/scripts/diagnose_peanut.py @@ -0,0 +1,50 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + out = stdout.read().decode('utf-8', errors='replace') + err = stderr.read().decode('utf-8', errors='replace') + return out.strip(), err.strip() + +print("docker service ls:") +out, _ = run("docker service ls") +print(out) + +print("\nPython3 test:") +out, err = run("python3 --version") +print(out, err) + +# Run python script inside a temporary script file on the server so quoting doesn't get messed up +check_py = """ +import sqlite3 +try: + conn = sqlite3.connect('/var/lib/docker-data/heimdall/config/www/app.sqlite') + print("HEIMDALL TILES:") + for row in conn.execute("SELECT id, title, url, pinned FROM items"): + print(row) + conn.close() +except Exception as e: + print("Heimdall error:", e) + +try: + conn = sqlite3.connect('/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite') + print("NPM PROXY HOSTS:") + for row in conn.execute("SELECT id, domain_names, forward_host, forward_port, enabled FROM proxy_host"): + print(row) + conn.close() +except Exception as e: + print("NPM error:", e) +""" + +run(f"cat << 'EOF' > /tmp/check_dbs.py\n{check_py}\nEOF") +out, err = run("echo 'Katzir476!' | sudo -S python3 /tmp/check_dbs.py") +print("\nDB check results:\n", out) +if err: print("\nDB check err:\n", err) + +c.close() diff --git a/scripts/dump_firewall_policies.py b/scripts/dump_firewall_policies.py new file mode 100644 index 0000000..3640e66 --- /dev/null +++ b/scripts/dump_firewall_policies.py @@ -0,0 +1,24 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +s = requests.Session() +s.verify = False + +r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'}) +csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token') +headers = {'X-CSRF-Token': csrf} if csrf else {} + +res = s.get('https://14.44.4.1/proxy/network/v2/api/site/default/firewall-policies', headers=headers) +policies = res.json() +print(f"Found {len(policies)} policies:") +for p in policies: + print(f"Index {p.get('index')}: {p.get('name')} | action={p.get('action')} | enabled={p.get('enabled')}") + src = p.get('source', {}) + dst = p.get('destination', {}) + print(f" Src: zone={src.get('zone_id')} net={src.get('network_id')} ip={src.get('ip_addresses')}") + print(f" Dst: zone={dst.get('zone_id')} net={dst.get('network_id')} ip={dst.get('ip_addresses')} port={dst.get('port_ranges')}") + print(f" States: {p.get('connection_states')}, state_type: {p.get('connection_state_type')}") + diff --git a/scripts/dump_lh_disk_uuids.py b/scripts/dump_lh_disk_uuids.py new file mode 100644 index 0000000..72ed0de --- /dev/null +++ b/scripts/dump_lh_disk_uuids.py @@ -0,0 +1,29 @@ +import paramiko +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +out, _ = k8s("kubectl get nodes.longhorn.io -n longhorn-system -o json") +data = json.loads(out) + +node_disk_uuids = {} +for node in data.get("items", []): + nname = node["metadata"]["name"] + node_disk_uuids[nname] = {} + for dname, dstat in node.get("status", {}).get("diskStatus", {}).items(): + node_disk_uuids[nname][dname] = { + "path": node.get("spec", {}).get("disks", {}).get(dname, {}).get("path"), + "diskUUID": dstat.get("diskUUID") + } + +print(json.dumps(node_disk_uuids, indent=2)) +c.close() + diff --git a/scripts/dump_lh_expected_disks.py b/scripts/dump_lh_expected_disks.py new file mode 100644 index 0000000..5c7e3bf --- /dev/null +++ b/scripts/dump_lh_expected_disks.py @@ -0,0 +1,26 @@ +import paramiko +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +out, _ = k8s("kubectl get nodes.longhorn.io -n longhorn-system -o json") +data = json.loads(out) + +expected_disks = {} +for node in data.get("items", []): + nname = node["metadata"]["name"] + expected_disks[nname] = {} + for dname, dspec in node.get("spec", {}).get("disks", {}).items(): + expected_disks[nname][dname] = dspec.get("path") + +print(json.dumps(expected_disks, indent=2)) +c.close() + diff --git a/scripts/dump_network_conf.py b/scripts/dump_network_conf.py new file mode 100644 index 0000000..8da1848 --- /dev/null +++ b/scripts/dump_network_conf.py @@ -0,0 +1,19 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +s = requests.Session() +s.verify = False + +r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'}) +csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token') +headers = {'X-CSRF-Token': csrf} if csrf else {} + +r_net = s.get('https://14.44.4.1/proxy/network/api/s/default/rest/networkconf', headers=headers) +for n in r_net.json().get('data', []): + if n.get('name') in ['Smart Home', 'Compute']: + print(f"=== {n.get('name')} ===") + print(json.dumps(n, indent=2)) + diff --git a/scripts/dump_target_policies.py b/scripts/dump_target_policies.py new file mode 100644 index 0000000..3c55ce8 --- /dev/null +++ b/scripts/dump_target_policies.py @@ -0,0 +1,20 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +s = requests.Session() +s.verify = False + +r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'}) +csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token') +headers = {'X-CSRF-Token': csrf} if csrf else {} + +res = s.get('https://14.44.4.1/proxy/network/v2/api/site/default/firewall-policies', headers=headers) +policies = res.json() +for p in policies: + if any(k in p.get('name', '').lower() for k in ['smarthome', 'snarfhome', 'compute', 'drop']): + print(f"=== {p.get('name')} (Index {p.get('index')}) ===") + print(json.dumps(p, indent=2)) + diff --git a/scripts/dump_unifi_fw.py b/scripts/dump_unifi_fw.py new file mode 100644 index 0000000..735a0f8 --- /dev/null +++ b/scripts/dump_unifi_fw.py @@ -0,0 +1,27 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +s = requests.Session() +s.verify = False + +r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'}) +csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token') +headers = {'X-CSRF-Token': csrf} if csrf else {} + +r_fw = s.get('https://14.44.4.1/proxy/network/api/s/default/rest/firewallrule', headers=headers) +print("Firewall rule status:", r_fw.status_code) +print("Firewall rule json:", r_fw.json()) + +# Check firewall groups +r_fg = s.get('https://14.44.4.1/proxy/network/api/s/default/rest/firewallgroup', headers=headers) +print("Firewall groups json:", r_fg.json()) + +# Check port forwarding +r_pf = s.get('https://14.44.4.1/proxy/network/api/s/default/rest/portforward', headers=headers) +print("Port forwards:", r_pf.json()) + +c.close() if 'c' in locals() else None + diff --git a/scripts/final_health_check.py b/scripts/final_health_check.py new file mode 100644 index 0000000..3b19854 --- /dev/null +++ b/scripts/final_health_check.py @@ -0,0 +1,25 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') + +# 1. Spacedock +c_sd = paramiko.SSHClient() +c_sd.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c_sd.connect('140.44.4.90', username='rancher', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c_sd.exec_command("kubectl get pods -A | grep -E 'glm|vllm|qdrant|triton'") +print("Spacedock Pods:\n", o.read().decode().strip()) +_, o, _ = c_sd.exec_command("kubectl get pvc -A | grep -E 'glm|vllm|qdrant|triton'") +print("\nSpacedock PVCs:\n", o.read().decode().strip()) +c_sd.close() + +# 2. Heimdall +c_h = paramiko.SSHClient() +c_h.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c_h.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c_h.exec_command("echo 'Katzir476!' | sudo -S sqlite3 /var/lib/docker-data/heimdall/config/www/app.sqlite 'SELECT id, username, public_front FROM users;'") +print("\nHeimdall Users:\n", o.read().decode().strip()) +_, o, _ = c_h.exec_command("echo 'Katzir476!' | sudo -S sqlite3 /var/lib/docker-data/heimdall/config/www/app.sqlite 'SELECT count(*) FROM items WHERE pinned=1;'") +print("Heimdall Pinned Items Count:\n", o.read().decode().strip()) +c_h.close() + diff --git a/scripts/find_peanut.py b/scripts/find_peanut.py new file mode 100644 index 0000000..51374cd --- /dev/null +++ b/scripts/find_peanut.py @@ -0,0 +1,26 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- ss -tulpn | grep 8050 on 140.44.4.71 ---") +out, _ = run("ss -tulpn | grep 8050") +print(out) + +print("--- docker ps on 140.44.4.71 ---") +out, _ = run("docker ps --format 'table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.Ports}}'") +print(out) + +print("--- docker service ls ---") +out, _ = run("docker service ls") +print(out) + +c.close() + diff --git a/scripts/find_peanut_files.py b/scripts/find_peanut_files.py new file mode 100644 index 0000000..078f1fb --- /dev/null +++ b/scripts/find_peanut_files.py @@ -0,0 +1,18 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- Find compose / stack files for peanut ---") +out, _ = run("echo 'Katzir476!' | sudo -S find /mnt/swarm_shared /home/snarf -name '*peanut*' 2>/dev/null") +print(out) + +c.close() + diff --git a/scripts/fix_node1_sdc.py b/scripts/fix_node1_sdc.py new file mode 100644 index 0000000..96dbf85 --- /dev/null +++ b/scripts/fix_node1_sdc.py @@ -0,0 +1,26 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Check /dev/sdc with file and blkid:") +out, _ = run_cmd("file -s /dev/sdc; blkid /dev/sdc; dmesg | tail -n 20") +print(out) + +print("\nRunning fsck on /dev/sdc:") +out, err = run_cmd("fsck.ext4 -fy /dev/sdc") +print(out, err) + +print("\nTry mounting /dev/sdc now:") +out, err = run_cmd("mkdir -p /tmp/sdc_test && mount /dev/sdc /tmp/sdc_test && ls -la /tmp/sdc_test; cat /tmp/sdc_test/longhorn-disk.cfg 2>/dev/null; umount /tmp/sdc_test 2>/dev/null") +print(out, err) + +c.close() + diff --git a/scripts/fs25_screen.png b/scripts/fs25_screen.png new file mode 100644 index 0000000..6369e9a Binary files /dev/null and b/scripts/fs25_screen.png differ diff --git a/scripts/get_spaceport_token.py b/scripts/get_spaceport_token.py new file mode 100644 index 0000000..6b52728 --- /dev/null +++ b/scripts/get_spaceport_token.py @@ -0,0 +1,10 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.41', username='root', password='dietpi', timeout=10) +_, o, _ = c.exec_command("kubectl create token prometheus-scraper -n kube-system --duration=87600h") +token = o.read().decode().strip() +print("TOKEN=" + token) +c.close() + diff --git a/scripts/inspect_current_grafana_dashboards.py b/scripts/inspect_current_grafana_dashboards.py new file mode 100644 index 0000000..f980635 --- /dev/null +++ b/scripts/inspect_current_grafana_dashboards.py @@ -0,0 +1,28 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +for uid in ["snarfnet-containers", "snarfnet-systems", "snarfnet-ai-stack"]: + url = f"https://grafana.snarfnet.net/api/dashboards/uid/{uid}" + resp = requests.get(url, auth=auth, verify=False, timeout=10) + data = resp.json() + dash = data.get("dashboard", {}) + title = dash.get("title") + folder_id = data.get("meta", {}).get("folderId") + folder_title = data.get("meta", {}).get("folderTitle") + panels = dash.get("panels", []) + templating = [v.get("name") for v in dash.get("templating", {}).get("list", [])] + print(f"\n==========================================") + print(f"UID: {uid} | Title: {title} | Folder: {folder_title} (ID: {folder_id})") + print(f"Template variables: {templating}") + print(f"Panels ({len(panels)}):") + for p in panels: + p_type = p.get("type") + p_title = p.get("title") + if p_type == "row": + print(f" [ROW] {p_title}") + else: + print(f" - [{p_type}] {p_title}") + diff --git a/scripts/inspect_npm_4.py b/scripts/inspect_npm_4.py new file mode 100644 index 0000000..6ea8dab --- /dev/null +++ b/scripts/inspect_npm_4.py @@ -0,0 +1,25 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +cmd = """echo 'Katzir476!' | sudo -S python3 -c " +import sqlite3 +conn = sqlite3.connect('/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite') +cur = conn.cursor() +row = cur.execute('SELECT id, domain_names, forward_host, forward_port FROM proxy_host WHERE id=4').fetchone() +print('Proxy Host 4 in DB:', row) +conn.close() +" """ +_, o, e = c.exec_command(cmd) +print(o.read().decode().strip()) + +cmd2 = "echo 'Katzir476!' | sudo -S cat /var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/nginx/proxy_host/4.conf" +_, o2, _ = c.exec_command(cmd2) +print("\n4.conf contents:\n", o2.read().decode().strip()) + +c.close() + diff --git a/scripts/inspect_peanut_container.py b/scripts/inspect_peanut_container.py new file mode 100644 index 0000000..d23376d --- /dev/null +++ b/scripts/inspect_peanut_container.py @@ -0,0 +1,28 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') + +for node in ["140.44.4.71", "140.44.4.72", "140.44.4.73", "140.44.4.74", "140.44.4.75"]: + try: + c = paramiko.SSHClient() + c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + c.connect(node, username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=5) + stdin, stdout, stderr = c.exec_command("docker ps -a --filter name=peanut --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'") + out = stdout.read().decode().strip() + if out and len(out.splitlines()) > 1: + print(f"--- Node {node} ---") + print(out) + + # Inspect the container + _, stdout, _ = c.exec_command("docker inspect $(docker ps -q --filter name=peanut) --format '{{json .Config.Env}}'") + env = stdout.read().decode().strip() + print("Env:", env) + + _, stdout, _ = c.exec_command("docker logs --tail 30 $(docker ps -q --filter name=peanut)") + logs = stdout.read().decode().strip() + print("Logs:\n", logs) + c.close() + except Exception as e: + print(f"Error on {node}: {e}") + diff --git a/scripts/inspect_peanut_service.py b/scripts/inspect_peanut_service.py new file mode 100644 index 0000000..7907169 --- /dev/null +++ b/scripts/inspect_peanut_service.py @@ -0,0 +1,31 @@ +import paramiko +import sys +import json + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- Service Inspect ---") +out, _ = run("echo 'Katzir476!' | sudo -S docker service inspect peanut_peanut") +try: + data = json.loads(out) + spec = data[0]['Spec']['TaskTemplate']['ContainerSpec'] + print("Image:", spec.get('Image')) + print("Env:", spec.get('Env')) + print("Mounts:", spec.get('Mounts')) +except Exception as e: + print("Parse error:", e) + print(out[:1000]) + +print("\n--- Service Tasks / Nodes ---") +out, _ = run("echo 'Katzir476!' | sudo -S docker service ps peanut_peanut --no-trunc") +print(out) + +c.close() + diff --git a/scripts/inspect_unifi.py b/scripts/inspect_unifi.py new file mode 100644 index 0000000..bad3d72 --- /dev/null +++ b/scripts/inspect_unifi.py @@ -0,0 +1,39 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +s = requests.Session() +s.verify = False + +r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'}) +csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token') +headers = {'X-CSRF-Token': csrf} if csrf else {} + +# Get sites +r_sites = s.get('https://14.44.4.1/proxy/network/api/self/sites', headers=headers) +print("Sites status:", r_sites.status_code) +sites = r_sites.json().get('data', []) +site_name = sites[0]['name'] if sites else 'default' +print("Site name:", site_name) + +# Get networks +r_net = s.get(f'https://14.44.4.1/proxy/network/api/s/{site_name}/rest/networkconf', headers=headers) +print("Networks:") +for n in r_net.json().get('data', []): + print(f" {n.get('name')}: subnet={n.get('ip_subnet')}, vlan={n.get('vlan')}, purpose={n.get('purpose')}, isolation={n.get('isolation')}") + +# Get firewall rules +r_fw = s.get(f'https://14.44.4.1/proxy/network/api/s/{site_name}/rest/firewallrule', headers=headers) +print("\nFirewall Rules:") +for rule in r_fw.json().get('data', []): + print(f" Rule #{rule.get('rule_index')}: {rule.get('name')} | ruleset={rule.get('ruleset')} | action={rule.get('action')} | enabled={rule.get('enabled')}") + print(f" src={rule.get('src_networkconf_id') or rule.get('src_address') or rule.get('src_firewallgroup_ids')} dst={rule.get('dst_networkconf_id') or rule.get('dst_address') or rule.get('dst_firewallgroup_ids')}") + +# Get client nutserver +r_client = s.get(f'https://14.44.4.1/proxy/network/api/s/{site_name}/stat/sta', headers=headers) +for c in r_client.json().get('data', []): + if c.get('ip') == '14.4.4.44' or 'nut' in c.get('hostname', '').lower() or 'nut' in c.get('name', '').lower(): + print("\nNut Client:", c.get('name'), c.get('hostname'), c.get('ip'), c.get('mac'), c.get('network_id'), c.get('essid'), c.get('is_guest')) + diff --git a/scripts/inspect_unifi_fw.py b/scripts/inspect_unifi_fw.py new file mode 100644 index 0000000..542cd77 --- /dev/null +++ b/scripts/inspect_unifi_fw.py @@ -0,0 +1,34 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +s = requests.Session() +s.verify = False + +r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'}) +csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token') +headers = {'X-CSRF-Token': csrf} if csrf else {} + +site_name = 'default' + +# Try v2 firewall policies / rules +for endpoint in [ + f'/proxy/network/v2/api/site/{site_name}/firewall/rules', + f'/proxy/network/v2/api/site/{site_name}/firewall/policies', + f'/proxy/network/v2/api/site/{site_name}/trafficrules', + f'/proxy/network/v2/api/site/{site_name}/trafficroutes', + f'/proxy/network/api/s/{site_name}/rest/firewallgroup' +]: + r = s.get('https://14.44.4.1' + endpoint, headers=headers) + print(f"{endpoint}: {r.status_code}") + if r.status_code == 200: + data = r.json() + print(" Data count / sample:", len(data) if isinstance(data, list) else data.keys()) + if isinstance(data, list) and data: + for item in data: + print(" ", item.get('description') or item.get('name'), "| action:", item.get('action'), item.get('ruleset'), item.get('traffic_action')) + +c.close() if 'c' in locals() else None + diff --git a/scripts/inspect_unifi_wlan.py b/scripts/inspect_unifi_wlan.py new file mode 100644 index 0000000..fe79941 --- /dev/null +++ b/scripts/inspect_unifi_wlan.py @@ -0,0 +1,28 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +s = requests.Session() +s.verify = False + +r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'}) +csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token') +headers = {'X-CSRF-Token': csrf} if csrf else {} + +# Get wlanconf +r_wlan = s.get('https://14.44.4.1/proxy/network/api/s/default/rest/wlanconf', headers=headers) +print("WLANs:") +for w in r_wlan.json().get('data', []): + print(f" SSID: {w.get('name')}, enabled: {w.get('enabled')}, networkconf_id: {w.get('networkconf_id')}, is_guest: {w.get('is_guest')}, l2_isolation: {w.get('l2_isolation')}, client_isolation: {w.get('client_isolation')}") + +# Get nutserver client details in depth +r_sta = s.get('https://14.44.4.1/proxy/network/api/s/default/stat/sta', headers=headers) +for c in r_sta.json().get('data', []): + if c.get('ip') == '14.4.4.44': + print("\nNutserver client full details:") + for k, v in c.items(): + if any(term in k for term in ['ip', 'mac', 'network', 'vlan', 'ap', 'bssid', 'guest', 'blocked', 'isolate']): + print(f" {k}: {v}") + diff --git a/scripts/investigate_issues.py b/scripts/investigate_issues.py new file mode 100644 index 0000000..2e15ee7 --- /dev/null +++ b/scripts/investigate_issues.py @@ -0,0 +1,57 @@ +import paramiko +import json +import sqlite3 +import os + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +def run_ssh(host, user, password, cmd, sudo=False): + client = paramiko.SSHClient() + client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + if password and not SSH_KEY: + client.connect(host, username=user, password=password, timeout=10) + else: + client.connect(host, username=user, key_filename=SSH_KEY, password=password, timeout=10) + if sudo: + full_cmd = f"echo {password} | sudo -S sh -c '{cmd}'" + else: + full_cmd = cmd + stdin, stdout, stderr = client.exec_command(full_cmd) + out = stdout.read().decode('utf-8', errors='ignore').strip() + err = stderr.read().decode('utf-8', errors='ignore').strip() + client.close() + return out, err + except Exception as e: + return "", str(e) + +print("=== 1. HARVESTER LONGHORN VOLUMES & VMS ===") +k8s_env = "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; " +out, err = run_ssh("140.44.4.90", "rancher", "Katzir476!", k8s_env + "kubectl get virtualmachines.kubevirt.io -A -o wide", sudo=True) +print("VMs:") +print(out) +out, err = run_ssh("140.44.4.90", "rancher", "Katzir476!", k8s_env + "kubectl get pvc -A", sudo=True) +print("\nPVCs:") +print(out) +out, err = run_ssh("140.44.4.90", "rancher", "Katzir476!", k8s_env + "kubectl get nodes.longhorn.io -n longhorn-system", sudo=True) +print("\nLonghorn Nodes:") +print(out) +out, err = run_ssh("140.44.4.90", "rancher", "Katzir476!", k8s_env + "kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,FRONTEND:.spec.frontend,REPLICAS:.status.currentReplicas", sudo=True) +print("\nLonghorn Volumes Detail:") +print(out) + +print("\n=== 2. HEIMDALL CONTAINER & SQLITE DATABASE ===") +# Check Shipyard swarm leader (140.44.4.71) +out, err = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker service ls | grep -i heimdall || docker ps | grep -i heimdall", sudo=True) +print("Heimdall Swarm / Docker process on 140.44.4.71:") +print(out) +# If not on 71, check docker service inspect or swarm tasks +out, err = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker service ps $(docker service ls -q -f name=heimdall) --no-trunc 2>/dev/null", sudo=True) +print("Heimdall Service Task:") +print(out) + +# Also check Spaceport K3s (140.44.4.41) just in case heimdall is on k8s +out, err = run_ssh("140.44.4.41", "snarf", "Katzir476!", "kubectl get pods -A | grep -i heimdall", sudo=True) +print("Heimdall on Spaceport K3s (if any):") +print(out) + diff --git a/scripts/investigate_kuma_npm_db.py b/scripts/investigate_kuma_npm_db.py new file mode 100644 index 0000000..8ee30ce --- /dev/null +++ b/scripts/investigate_kuma_npm_db.py @@ -0,0 +1,84 @@ +import paramiko +import json +import urllib.request +import ssl + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +def run_ssh(host, user, password, cmd, sudo=False): + client = paramiko.SSHClient() + client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + client.connect(host, username=user, key_filename=SSH_KEY, password=password, timeout=10) + full_cmd = f"echo {password} | sudo -S sh -c '{cmd}'" if sudo else cmd + stdin, stdout, stderr = client.exec_command(full_cmd) + out = stdout.read().decode('utf-8', errors='ignore').strip() + err = stderr.read().decode('utf-8', errors='ignore').strip() + client.close() + return out, err + except Exception as e: + return "", str(e) + +print("=== 1. NPM PROXY HOSTS LIST ===") +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker exec $(docker ps -q -f name=nginx-proxy-manager) ls -la /data/nginx/proxy_host/", sudo=True) +print("Proxy host files:") +print(out) + +# Search for heimdall in all files under /data/nginx +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker exec $(docker ps -q -f name=nginx-proxy-manager) grep -r 'heimdall' /data/nginx/", sudo=True) +print("Heimdall occurrences in /data/nginx:") +print(out) + +print("\n=== 2. SWARM SERVICES & UPTIME KUMA LOCATION ===") +out, _ = run_ssh("140.44.4.71", "snarf", "Katzir476!", "docker service ls", sudo=True) +print(out) + +print("\n=== 3. HEIMDALL DATABASE CONTENT ===") +py_check = '''python3 - << 'PYEOF' +import sqlite3 + +db = '/var/lib/docker/volumes/heimdall_heimdall_config/_data/www/app.sqlite' +con = sqlite3.connect(db) +cur = con.cursor() +print("USERS:") +for r in cur.execute("SELECT id, username, email, public_front FROM users").fetchall(): + print(r) + +print("\nITEMS:") +for r in cur.execute("SELECT id, title, url, pinned, user_id FROM items").fetchall(): + print(r) + +print("\nSETTINGS:") +for r in cur.execute("SELECT * FROM settings").fetchall(): + print(r) +PYEOF +''' +out, err = run_ssh("140.44.4.71", "snarf", "Katzir476!", py_check, sudo=True) +print(out if out else err) + +print("\n=== 4. ZABBIX ALL HOSTS ===") +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1} + if auth: payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + try: + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + except Exception as e: + return {"error": str(e)} + +auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result") +if auth: + hosts = zabbix_api("host.get", {"output": ["hostid", "host", "name", "status"], "selectInterfaces": ["ip"]}, auth=auth).get("result", []) + print(f"Total Zabbix Hosts: {len(hosts)}") + print("Sample hosts:") + for h in hosts[:20]: + ips = [i['ip'] for i in h.get('interfaces', [])] + print(f" {h['host']} ({h['name']}): IPs={ips}, status={h['status']}") + diff --git a/scripts/investigate_kuma_npm_longhorn.py b/scripts/investigate_kuma_npm_longhorn.py new file mode 100644 index 0000000..85fbe5b --- /dev/null +++ b/scripts/investigate_kuma_npm_longhorn.py @@ -0,0 +1,63 @@ +import paramiko +import json +import urllib.request +import ssl + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +def run_ssh(host, user, password, cmd, sudo=False): + client = paramiko.SSHClient() + client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + client.connect(host, username=user, key_filename=SSH_KEY, password=password, timeout=10) + full_cmd = f"echo {password} | sudo -S sh -c '{cmd}'" if sudo else cmd + stdin, stdout, stderr = client.exec_command(full_cmd) + out = stdout.read().decode('utf-8', errors='ignore').strip() + err = stderr.read().decode('utf-8', errors='ignore').strip() + client.close() + return out, err + except Exception as e: + return "", str(e) + +print("=== 1. WHERE IS UPTIME KUMA? ===") +# Spaceport 140.44.4.41 +out, _ = run_ssh("140.44.4.41", "snarf", "Katzir476!", "kubectl get pods -A | grep -i kuma", sudo=True) +print("Spaceport Kuma:", out) +# Aerodrome 140.44.4.11 +out, _ = run_ssh("140.44.4.11", "snarf", "Katzir476!", "kubectl get pods -A | grep -i kuma", sudo=True) +print("Aerodrome Kuma:", out) + +print("\n=== 2. HEIMDALL SQLITE CONTENT ===") +cmd = """python3 -c " +import sqlite3 +con = sqlite3.connect('/var/lib/docker/volumes/heimdall_heimdall_config/_data/www/app.sqlite') +c = con.cursor() +print('--- USERS ---') +for r in c.execute('SELECT id, username, email, public_front FROM users').fetchall(): + print('User:', r) +print('--- ITEMS ---') +for r in c.execute('SELECT id, title, url, pinned, user_id FROM items').fetchall(): + print('Item:', r) +print('--- SETTINGS ---') +for r in c.execute('SELECT key, value, type FROM settings').fetchall(): + print('Setting:', r) +" """ +out, err = run_ssh("140.44.4.71", "snarf", "Katzir476!", cmd, sudo=True) +print(out if out else err) + +print("\n=== 3. WHICH NPM FILE HANDLES HEIMDALL? ===") +# Search inside NPM database or sqlite +cmd = "docker exec $(docker ps -q -f name=nginx-proxy-manager) sqlite3 /data/database.sqlite 'SELECT id, domain_names, forward_host, forward_port, advanced_config FROM proxy_host;'" +out, err = run_ssh("140.44.4.71", "snarf", "Katzir476!", cmd, sudo=True) +print("NPM Proxy Hosts from SQLite:") +print(out if out else err) + +print("\n=== 4. LONGHORN SALVAGE RESTORATION ===") +# Check longhorn salvage command +k8s_env = "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; " +# List faulted volumes +cmd = k8s_env + "kubectl get volumes.longhorn.io -n longhorn-system -o jsonpath='{range .items[?(@.status.robustness==\"faulted\")]}{.metadata.name}{\"\\n\"}{end}'" +out, _ = run_ssh("140.44.4.90", "rancher", "Katzir476!", cmd, sudo=True) +print("Faulted Longhorn volumes:") +print(out) + diff --git a/scripts/list_items.py b/scripts/list_items.py new file mode 100644 index 0000000..36447c2 --- /dev/null +++ b/scripts/list_items.py @@ -0,0 +1,21 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +cmd = """echo 'Katzir476!' | sudo -S python3 -c " +import sqlite3 +con = sqlite3.connect('/var/lib/docker-data/heimdall/config/www/app.sqlite') +cur = con.cursor() +rows = cur.execute('SELECT id, title, pinned, user_id, \`order\` FROM items').fetchall() +for r in rows: + print(r) +con.close() +" """ +_, o, _ = c.exec_command(cmd) +print("Output:\n", o.read().decode().strip()) +c.close() + diff --git a/scripts/mount_and_persist_all_disks.py b/scripts/mount_and_persist_all_disks.py new file mode 100644 index 0000000..24a34b5 --- /dev/null +++ b/scripts/mount_and_persist_all_disks.py @@ -0,0 +1,141 @@ +import paramiko +import json +import time + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +nodes = { + "snaisvr44hrvstr01": "140.44.4.81", + "snaisvr44hrvstr02": "140.44.4.82", + "snaisvr44hrvstr03": "140.44.4.83", + "snaisvr44hrvstr04": "140.44.4.84" +} + +# Python script to run on each node +remote_mount_script = """python3 - << 'PYEOF' +import os +import subprocess +import json + +# 1. Map physical disks to their Longhorn diskUUID +out = subprocess.check_output("blkid", shell=True).decode() +disks = {} +for line in out.splitlines(): + if not line.startswith("/dev/sd"): continue + dev = line.split(":")[0] + if "sda" in dev: continue + uuid = "" + for part in line.split(): + if part.startswith("UUID="): + uuid = part.split('"')[1] + if uuid: + disks[dev] = uuid + +disk_info = {} # diskUUID -> fs_uuid +for dev, uuid in disks.items(): + tmp = f"/tmp/m_{os.path.basename(dev)}" + os.makedirs(tmp, exist_ok=True) + try: + subprocess.check_call(f"mount {dev} {tmp}", shell=True) + cfg_file = os.path.join(tmp, "longhorn-disk.cfg") + if os.path.exists(cfg_file): + with open(cfg_file) as f: + data = json.load(f) + lh_uuid = data.get("diskUUID") + disk_info[lh_uuid] = uuid + subprocess.call(f"umount {tmp}", shell=True) + except Exception as e: + pass + +print("Disks probed with valid longhorn-disk.cfg:", len(disk_info)) + +# 2. Get Longhorn expected disk mapping from env or file +# We read expected disk mapping from /tmp/node_expected.json +with open("/tmp/node_expected.json") as f: + expected = json.load(f) + +for disk_name, info in expected.items(): + lh_uuid = info["diskUUID"] + path = info["path"] + if not path or path == "/var/lib/harvester/defaultdisk": + continue + + if lh_uuid in disk_info: + fs_uuid = disk_info[lh_uuid] + print(f"Mounting {disk_name} ({lh_uuid}) with fs UUID {fs_uuid} -> {path}") + os.makedirs(path, exist_ok=True) + # Check if already mounted + mounts = subprocess.check_output("mount", shell=True).decode() + if path not in mounts: + subprocess.call(f"mount /dev/disk/by-uuid/{fs_uuid} {path}", shell=True) + + # Write systemd mount unit + unit_name = f"var-lib-harvester-extrax2ddisks-{disk_name.replace('disk-', '')}.mount" + unit_content = f\"\"\"[Unit] +Description=Harvester Extra Disk {disk_name} +Before=rke2-server.service rke2-agent.service + +[Mount] +What=/dev/disk/by-uuid/{fs_uuid} +Where={path} +Type=ext4 +Options=defaults + +[Install] +WantedBy=multi-user.target +\"\"\" + with open(f"/etc/systemd/system/{unit_name}", "w") as uf: + uf.write(unit_content) + print(f"Created /etc/systemd/system/{unit_name}") + else: + print(f"WARNING: Disk {disk_name} ({lh_uuid}) not found on this node!") + +subprocess.call("systemctl daemon-reload", shell=True) +PYEOF +""" + +# Read node expected from earlier dump +with open("scripts/dump_lh_disk_uuids.py") as f: + pass + +# We will fetch latest expected directly +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) +_, o, _ = c.exec_command("echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; kubectl get nodes.longhorn.io -n longhorn-system -o json'") +data = json.loads(o.read().decode()) +c.close() + +all_expected = {} +for node in data.get("items", []): + nname = node["metadata"]["name"] + all_expected[nname] = {} + for dname, dstat in node.get("status", {}).get("diskStatus", {}).items(): + all_expected[nname][dname] = { + "path": node.get("spec", {}).get("disks", {}).get(dname, {}).get("path"), + "diskUUID": dstat.get("diskUUID") + } + +for hname, ip in nodes.items(): + print(f"\n=================== Configuring {hname} ({ip}) ===================") + expected_for_node = all_expected.get(hname, {}) + nc = paramiko.SSHClient() + nc.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + nc.connect(ip, username="rancher", key_filename=SSH_KEY, timeout=10) + # Write expected json + exp_str = json.dumps(expected_for_node) + nc.exec_command(f"cat << 'EOF' > /tmp/node_expected.json\n{exp_str}\nEOF") + # Run remote mount script + _, o, e = nc.exec_command(f"echo Katzir476! | sudo -S {remote_mount_script}") + print(o.read().decode()) + print(e.read().decode()) + + # Check df -h on node + _, o, _ = nc.exec_command("df -h | grep extra-disks") + print("Mounted extra-disks on", hname, ":\n", o.read().decode()) + nc.close() + except Exception as e: + print("Error on", hname, ":", e) + +print("\nDone configuring all nodes!") diff --git a/scripts/pin_heimdall_items.py b/scripts/pin_heimdall_items.py new file mode 100644 index 0000000..4e168ce --- /dev/null +++ b/scripts/pin_heimdall_items.py @@ -0,0 +1,27 @@ +import paramiko +import sys +import time + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +cmd = f"""echo 'Katzir476!' | sudo -S cp /var/lib/docker-data/heimdall/config/www/app.sqlite /var/lib/docker-data/heimdall/config/www/app.sqlite.backup_{int(time.time())} && echo 'Katzir476!' | sudo -S python3 -c " +import sqlite3 +con = sqlite3.connect('/var/lib/docker-data/heimdall/config/www/app.sqlite') +cur = con.cursor() +cur.execute('UPDATE items SET pinned = 1 WHERE id > 0') +con.commit() +pinned_count = cur.execute('SELECT count(*) FROM items WHERE pinned = 1').fetchone()[0] +print('Successfully updated! Pinned items count:', pinned_count) +con.close() +" """ + +_, o, e = c.exec_command(cmd) +print(o.read().decode().strip()) +err = e.read().decode().strip() +if err: + print("Err:", err) +c.close() + diff --git a/scripts/prepare_fs25.py b/scripts/prepare_fs25.py new file mode 100644 index 0000000..53c5401 --- /dev/null +++ b/scripts/prepare_fs25.py @@ -0,0 +1,35 @@ +import paramiko + +ssh = paramiko.SSHClient() +ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +ssh.connect("140.44.44.80", username="root", key_filename=r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet", timeout=10) + +commands = """ +set -e +mkdir -p /opt/game-servers/fs25/{install,dlc,docs,wine} + +# Link files from ISO to install dir +cd /opt/game-servers/fs25/install +rm -rf * +for f in /mnt/storage-games/installers/fs25/iso/*; do + ln -sf "$f" . +done +ln -sf Setup.exe FarmingSimulator2025.exe +ln -sf Setup.exe FarmingSimulator25.exe + +# Copy DLCs to dlc dir +cd /opt/game-servers/fs25/dlc +cp -u /mnt/storage-games/installers/fs25/*.exe . 2>/dev/null || true + +# Set permissions +chmod -R 777 /opt/game-servers/fs25 + +ls -la /opt/game-servers/fs25/install +ls -la /opt/game-servers/fs25/dlc +""" + +stdin, stdout, stderr = ssh.exec_command(commands) +print(stdout.read().decode()) +print(stderr.read().decode()) +ssh.close() + diff --git a/scripts/probe_all_node_disks.py b/scripts/probe_all_node_disks.py new file mode 100644 index 0000000..540e3a4 --- /dev/null +++ b/scripts/probe_all_node_disks.py @@ -0,0 +1,63 @@ +import paramiko +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +nodes = { + "snaisvr44hrvstr01": "140.44.4.81", + "snaisvr44hrvstr02": "140.44.4.82", + "snaisvr44hrvstr03": "140.44.4.83", + "snaisvr44hrvstr04": "140.44.4.84" +} + +scan_script = """python3 - << 'PYEOF' +import os +import subprocess +import json + +# Get blkid +out = subprocess.check_output("blkid", shell=True).decode() +disks = {} +for line in out.splitlines(): + if not line.startswith("/dev/sd"): continue + dev = line.split(":")[0] + if "sda" in dev: continue # skip OS disk + uuid = "" + for part in line.split(): + if part.startswith("UUID="): + uuid = part.split('"')[1] + disks[dev] = uuid + +print(f"Disks found: {len(disks)}") +results = [] +for dev, uuid in disks.items(): + tmp_mount = f"/tmp/probe_{os.path.basename(dev)}" + os.makedirs(tmp_mount, exist_ok=True) + try: + subprocess.check_call(f"mount {dev} {tmp_mount}", shell=True) + cfg_path = os.path.join(tmp_mount, "longhorn-disk.cfg") + cfg_data = "" + if os.path.exists(cfg_path): + with open(cfg_path) as fp: + cfg_data = fp.read().strip() + results.append({"dev": dev, "uuid": uuid, "cfg": cfg_data}) + subprocess.call(f"umount {tmp_mount}", shell=True) + except Exception as e: + results.append({"dev": dev, "uuid": uuid, "error": str(e)}) + +print(json.dumps(results, indent=2)) +PYEOF +""" + +for hname, ip in nodes.items(): + print(f"\n=================== {hname} ({ip}) ===================") + c = paramiko.SSHClient() + c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + c.connect(ip, username="rancher", key_filename=SSH_KEY, timeout=10) + _, o, e = c.exec_command(f"echo Katzir476! | sudo -S {scan_script}") + print(o.read().decode()) + c.close() + except Exception as e: + print("Error:", e) + diff --git a/scripts/query_npm.py b/scripts/query_npm.py new file mode 100644 index 0000000..87bc2f8 --- /dev/null +++ b/scripts/query_npm.py @@ -0,0 +1,24 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +py_query = """ +import sqlite3 +conn = sqlite3.connect('/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite') +cur = conn.cursor() +cur.execute('SELECT id, domain_names, forward_host, forward_port, certificate_id, ssl_forced, enabled FROM proxy_host ORDER BY id') +for r in cur.fetchall(): + print(r) +print('--- CERTIFICATES ---') +cur.execute('SELECT id, nice_name, domain_names FROM certificate') +for r in cur.fetchall(): + print(r) +conn.close() +""" + +stdin, stdout, stderr = c.exec_command(f"python3 -c \"{py_query}\"") +print(stdout.read().decode()) +c.close() + diff --git a/scripts/register_zabbix_spacedock.py b/scripts/register_zabbix_spacedock.py new file mode 100644 index 0000000..4e5507e --- /dev/null +++ b/scripts/register_zabbix_spacedock.py @@ -0,0 +1,107 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1} + if auth: payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + +auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result") +print("Zabbix Auth:", bool(auth)) + +hosts_to_create = [ + # Spacedock Nodes & VIP in Hypervisors (group 7) + { + "host": "spacedock-harvester-vip", + "name": "Spacedock Harvester HCI Cluster VIP (140.44.4.90)", + "ip": "140.44.4.90", + "groupid": "7", + "templateid": "10564" + }, + { + "host": "snaisvr44hrvstr01", + "name": "Spacedock Harvester Node 01 (snaisvr44hrvstr01)", + "ip": "140.44.4.81", + "groupid": "7", + "templateid": "10564" + }, + { + "host": "snaisvr44hrvstr02", + "name": "Spacedock Harvester Node 02 (snaisvr44hrvstr02)", + "ip": "140.44.4.82", + "groupid": "7", + "templateid": "10564" + }, + { + "host": "snaisvr44hrvstr03", + "name": "Spacedock Harvester Node 03 (snaisvr44hrvstr03)", + "ip": "140.44.4.83", + "groupid": "7", + "templateid": "10564" + }, + { + "host": "snaisvr44hrvstr04", + "name": "Spacedock Harvester Node 04 (snaisvr44hrvstr04)", + "ip": "140.44.4.84", + "groupid": "7", + "templateid": "10564" + }, + # iDRACs in Management & IPMI (group 26) + { + "host": "idrac-spacedock1", + "name": "Dell PowerEdge R730 Node 1 iDRAC (idrac-spacedock1)", + "ip": "140.44.44.121", + "groupid": "26", + "templateid": "10564" + }, + { + "host": "idrac-spacedock2", + "name": "Dell PowerEdge R730 Node 2 iDRAC (idrac-spacedock2)", + "ip": "140.44.44.122", + "groupid": "26", + "templateid": "10564" + }, + { + "host": "idrac-spacedock3", + "name": "Dell PowerEdge R730 Node 3 iDRAC (idrac-spacedock3)", + "ip": "140.44.44.123", + "groupid": "26", + "templateid": "10564" + } +] + +for item in hosts_to_create: + # Check if host already exists + existing = zabbix_api("host.get", {"filter": {"host": [item["host"]]}}, auth=auth).get("result", []) + if existing: + print(f"Host {item['host']} already exists (ID: {existing[0]['hostid']})") + continue + + params = { + "host": item["host"], + "name": item["name"], + "groups": [{"groupid": item["groupid"]}], + "templates": [{"templateid": item["templateid"]}], + "interfaces": [ + { + "type": 1, + "main": 1, + "useip": 1, + "ip": item["ip"], + "dns": "", + "port": "10050" + } + ] + } + res = zabbix_api("host.create", params, auth=auth) + print(f"Created {item['host']} ({item['ip']}): {res}") + diff --git a/scripts/replace_prom.py b/scripts/replace_prom.py new file mode 100644 index 0000000..2a9fde2 --- /dev/null +++ b/scripts/replace_prom.py @@ -0,0 +1,10 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, e = c.exec_command("kubectl replace -f /tmp/prometheus.yaml") +print("Kubectl replace:\n", o.read().decode().strip()) +print("Kubectl err:\n", e.read().decode().strip()) +c.close() + diff --git a/scripts/restart_peanut.py b/scripts/restart_peanut.py new file mode 100644 index 0000000..ec78c82 --- /dev/null +++ b/scripts/restart_peanut.py @@ -0,0 +1,23 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("Current settings.yml:") +out, _ = run("cat /mnt/swarm_shared/legacy_volumes/peanut/settings.yml") +print(out) + +print("\nRestarting PeaNUT service...") +out, err = run("echo 'Katzir476!' | sudo -S docker service update --force peanut_peanut") +print(out) +if err: print(err) + +c.close() + diff --git a/scripts/revert_npm_heimdall.py b/scripts/revert_npm_heimdall.py new file mode 100644 index 0000000..8e29f1d --- /dev/null +++ b/scripts/revert_npm_heimdall.py @@ -0,0 +1,47 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +remote_code = ''' +import sqlite3 +import json + +db_path = '/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite' +conn = sqlite3.connect(db_path) +cur = conn.cursor() +domains = json.dumps(["portal.snarfnet.net"]) +cur.execute("UPDATE proxy_host SET domain_names = ? WHERE id = 4", (domains,)) +conn.commit() +print("Updated DB row:", cur.execute("SELECT id, domain_names, forward_host, forward_port FROM proxy_host WHERE id=4").fetchone()) +conn.close() + +conf_path = '/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/nginx/proxy_host/4.conf' +with open(conf_path, 'r') as f: + content = f.read() + +new_content = content.replace("server_name portal.snarfnet.net heimdall.snarfnet.net;", "server_name portal.snarfnet.net;") +with open(conf_path, 'w') as f: + f.write(new_content) +print("Updated 4.conf successfully!") +''' + +# Write script to remote /tmp/revert_npm_4.py +sftp = c.open_sftp() +with sftp.file('/tmp/revert_npm_4.py', 'w') as f: + f.write(remote_code) +sftp.close() + +# Run script with sudo +stdin, stdout, stderr = c.exec_command("echo 'Katzir476!' | sudo -S python3 /tmp/revert_npm_4.py") +print("Script output:\n", stdout.read().decode().strip()) +print("Script err:\n", stderr.read().decode().strip()) + +# Reload Nginx +_, o, _ = c.exec_command("echo 'Katzir476!' | sudo -S docker exec $(echo 'Katzir476!' | sudo -S docker ps -qf name=nginx-proxy-manager_app) nginx -s reload") +print("Reload Nginx:\n", o.read().decode().strip()) + +c.close() diff --git a/scripts/run_ak_shell.py b/scripts/run_ak_shell.py new file mode 100644 index 0000000..139c9db --- /dev/null +++ b/scripts/run_ak_shell.py @@ -0,0 +1,44 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10) + +pod = "authentik-server-cd7d9b949-vfl2m" + +py_script = """ +import json +from authentik.core.models import Application +from authentik.providers.proxy.models import ProxyProvider +from authentik.policies.models import PolicyBindingModel, Policy + +print('--- APPLICATIONS ---') +for a in Application.objects.all(): + print(f"App: {a.name} (slug: {a.slug})") + +print('\\n--- PROXY PROVIDERS ---') +for p in ProxyProvider.objects.all(): + print(f"Provider: {p.name} | External: {p.external_host} | Internal: {p.internal_host} | Mode: {p.mode}") + +print('\\n--- POLICIES ---') +for p in Policy.objects.all(): + print(f"Policy: {p.name} ({p.__class__.__name__})") + +print('\\n--- POLICY BINDINGS ---') +for pb in PolicyBindingModel.objects.all(): + print(f"Binding: target={pb.target} -> policy={pb.policy} (enabled={pb.enabled}, order={pb.order})") +""" + +stdin, stdout, stderr = c.exec_command("cat > /tmp/run_ak.py") +stdin.write(py_script) +stdin.close() +stdout.channel.recv_exit_status() + +c.exec_command(f"kubectl cp /tmp/run_ak.py infra/{pod}:/tmp/run_ak.py -c server") +_, o, e = c.exec_command(f"kubectl exec -n infra {pod} -c server -- /ak-root/.venv/bin/python3 /manage.py shell < /tmp/run_ak.py") +print(o.read().decode()) +print(e.read().decode()) + +c.close() + diff --git a/scripts/run_ak_standalone.py b/scripts/run_ak_standalone.py new file mode 100644 index 0000000..a7bbfe0 --- /dev/null +++ b/scripts/run_ak_standalone.py @@ -0,0 +1,46 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10) + +pod = "authentik-server-cd7d9b949-vfl2m" + +script_content = """import os +import django +os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'authentik.root.settings') +django.setup() + +from authentik.core.models import Application +from authentik.providers.proxy.models import ProxyProvider +from authentik.policies.models import PolicyBindingModel, Policy + +print("=== APPLICATIONS ===") +for a in Application.objects.all(): + print(f"App: {a.name} (slug: {a.slug}) | Provider: {a.provider}") + +print("\\n=== PROXY PROVIDERS ===") +for p in ProxyProvider.objects.all(): + print(f"Provider: {p.name} | External: {p.external_host} | Mode: {p.mode}") + +print("\\n=== POLICIES ===") +for p in Policy.objects.all(): + print(f"Policy: {p.name} ({p.__class__.__name__})") + +print("\\n=== POLICY BINDINGS ===") +for pb in PolicyBindingModel.objects.all(): + print(f"Target: {pb.target} -> Policy: {pb.policy}") +""" + +stdin, stdout, stderr = c.exec_command("cat > /tmp/ak_standalone.py") +stdin.write(script_content) +stdin.close() +stdout.channel.recv_exit_status() + +c.exec_command(f"kubectl cp /tmp/ak_standalone.py infra/{pod}:/tmp/ak_standalone.py -c server") +_, o, e = c.exec_command(f"kubectl exec -n infra {pod} -c server -- /ak-root/.venv/bin/python3 /tmp/ak_standalone.py") +print("OUTPUT:\n", o.read().decode()) + +c.close() + diff --git a/scripts/run_fs25_do_install.py b/scripts/run_fs25_do_install.py new file mode 100644 index 0000000..4801d06 --- /dev/null +++ b/scripts/run_fs25_do_install.py @@ -0,0 +1,34 @@ +import paramiko + +ssh = paramiko.SSHClient() +ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +ssh.connect("140.44.44.80", username="root", key_filename=r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet", timeout=10) + +script = """#!/bin/bash +export DISPLAY=:1 +export WINEPREFIX=/home/nobody/.fs_server +export WINEARCH=win64 +export WINEDLLOVERRIDES="mscoree=d" +export WINEDEBUG=-all +export USER=nobody + +echo "1. Booting wineprefix..." +wineboot -u +wineserver -w +echo "2. Wineprefix booted." + +echo "3. Starting Setup.exe /SILENT /NOCANCEL..." +wine /opt/fs25/install/Setup.exe /SILENT /NOCANCEL +wineserver -w +echo "4. Setup.exe completed with code $?!" +""" + +sftp = ssh.open_sftp() +with sftp.open("/opt/game-servers/fs25/do_install.sh", "w") as f: + f.write(script) +sftp.close() + +ssh.exec_command("chmod 777 /opt/game-servers/fs25/do_install.sh; docker cp /opt/game-servers/fs25/do_install.sh gameseed-farming-simulator-25:/opt/fs25/do_install.sh") +ssh.close() +print("Uploaded do_install.sh") + diff --git a/scripts/run_fs25_install.py b/scripts/run_fs25_install.py new file mode 100644 index 0000000..4d248bf --- /dev/null +++ b/scripts/run_fs25_install.py @@ -0,0 +1,28 @@ +import paramiko + +ssh = paramiko.SSHClient() +ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +ssh.connect("140.44.44.80", username="root", key_filename=r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet", timeout=10) + +script = """#!/bin/bash +export DISPLAY=:1 +export WINEPREFIX=/home/nobody/.fs_server +export WINEARCH=win64 +export WINEDLLOVERRIDES="mscoree=d" +export WINEDEBUG=-all +export USER=nobody + +echo "Starting Setup.exe /SILENT /NOCANCEL..." +wine /opt/fs25/install/Setup.exe /SILENT /NOCANCEL +echo "Setup.exe completed with exit code: $?" +""" + +sftp = ssh.open_sftp() +with sftp.open("/opt/game-servers/fs25/install_fs25.sh", "w") as f: + f.write(script) +sftp.close() + +ssh.exec_command("chmod +x /opt/game-servers/fs25/install_fs25.sh") +ssh.close() +print("Saved install_fs25.sh successfully!") + diff --git a/scripts/salvage_all_volumes.py b/scripts/salvage_all_volumes.py new file mode 100644 index 0000000..7abad4c --- /dev/null +++ b/scripts/salvage_all_volumes.py @@ -0,0 +1,59 @@ +import paramiko +import time +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +# 1. Write the salvage patch file safely using base64 +k8s("echo eyJzcGVjIjp7InNhbHZhZ2VSZXF1ZXN0ZWQiOnRydWV9fQ== | base64 -d > /tmp/salvage_patch.json") +out, _ = k8s("cat /tmp/salvage_patch.json") +print("Patch file content:", out) + +# 2. Get all faulted volumes +cmd = "kubectl get volumes.longhorn.io -n longhorn-system -o json" +out, _ = k8s(cmd) +vol_data = json.loads(out) + +faulted_vols = [] +for item in vol_data.get("items", []): + if item.get("status", {}).get("robustness") == "faulted": + faulted_vols.append(item["metadata"]["name"]) + +print(f"Faulted volumes count: {len(faulted_vols)} -> {faulted_vols}") + +# 3. For each faulted volume, find its replicas and apply salvageRequested=true +for vol in faulted_vols: + print(f"\nProcessing volume: {vol}") + # Get replicas for this volume + r_cmd = f"kubectl get replicas.longhorn.io -n longhorn-system -l longhornvolume={vol} -o jsonpath='{{.items[*].metadata.name}}'" + reps_out, _ = k8s(r_cmd) + replicas = reps_out.split() + print(f" Replicas ({len(replicas)}): {replicas}") + for rep in replicas: + p_cmd = f"kubectl patch replica.longhorn.io {rep} -n longhorn-system --type=merge --patch-file=/tmp/salvage_patch.json" + p_out, p_err = k8s(p_cmd) + print(f" Patched {rep}: {p_out}") + +print("\nWaiting 10 seconds for Longhorn volume reconciler...") +time.sleep(10) + +# Check volume statuses +out, _ = k8s("kubectl get volumes.longhorn.io -n longhorn-system -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,REPLICAS:.status.currentReplicas") +print("\nLonghorn Volumes Status:") +print(out) + +# Check pods in ai-stack +out, _ = k8s("kubectl get pods -n ai-stack -o wide") +print("\nPods in ai-stack:") +print(out) + +c.close() + diff --git a/scripts/salvage_api.py b/scripts/salvage_api.py new file mode 100644 index 0000000..1369330 --- /dev/null +++ b/scripts/salvage_api.py @@ -0,0 +1,40 @@ +import paramiko +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Longhorn services:") +out, _ = k8s("kubectl get svc -n longhorn-system") +print(out) + +# Call Longhorn API directly to salvage volumes! +# Longhorn manager backend exposes port 9500 (longhorn-backend service) +out, _ = k8s("kubectl exec -n longhorn-system $(kubectl get pods -n longhorn-system -l app=longhorn-manager -o jsonpath='{.items[0].metadata.name}') -c longhorn-manager -- curl -s http://127.0.0.1:9500/v1/volumes") +try: + vols = json.loads(out) + for v in vols.get("data", []): + vname = v["id"] + robustness = v.get("robustness") + actions = list(v.get("actions", {}).keys()) + print(f"Volume: {vname}, Robustness: {robustness}, Actions: {actions}") + if "salvage" in actions: + print(f" --> Executing salvage on {vname}...") + # Get replica names + reps = [r["name"] for r in v.get("replicas", [])] + salvage_payload = json.dumps({"names": reps}) + salv_cmd = f"kubectl exec -n longhorn-system $(kubectl get pods -n longhorn-system -l app=longhorn-manager -o jsonpath='{{.items[0].metadata.name}}') -c longhorn-manager -- curl -s -X POST -H 'Content-Type: application/json' -d '{salvage_payload}' http://127.0.0.1:9500/v1/volumes/{vname}?action=salvage" + s_out, _ = k8s(salv_cmd) + print(" Salvage result:", s_out[:200]) +except Exception as e: + print("Error querying Longhorn API:", e, out[:300]) + +c.close() + diff --git a/scripts/salvage_api2.py b/scripts/salvage_api2.py new file mode 100644 index 0000000..203c13b --- /dev/null +++ b/scripts/salvage_api2.py @@ -0,0 +1,36 @@ +import paramiko +import json + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +out, err = run_cmd("curl -s http://10.53.49.56:9500/v1/volumes") +print("Curl output length:", len(out)) +try: + vols = json.loads(out) + print(f"Total volumes from Longhorn backend: {len(vols.get('data', []))}") + for v in vols.get("data", []): + vname = v["id"] + robustness = v.get("robustness") + actions = list(v.get("actions", {}).keys()) + print(f"\nVolume: {vname}, Robustness: {robustness}") + print(f" Actions: {actions}") + if "salvage" in actions: + reps = [r["name"] for r in v.get("replicas", [])] + print(f" Replicas to salvage: {reps}") + salvage_url = v["actions"]["salvage"] + payload = json.dumps({"names": reps}) + salv_out, _ = run_cmd(f"curl -s -X POST -H 'Content-Type: application/json' -d '{payload}' '{salvage_url}'") + print(" Salvage response:", salv_out[:300]) +except Exception as e: + print("Error:", e, out[:300], err) + +c.close() + diff --git a/scripts/screen2.png b/scripts/screen2.png new file mode 100644 index 0000000..9ee3079 Binary files /dev/null and b/scripts/screen2.png differ diff --git a/scripts/screen3.png b/scripts/screen3.png new file mode 100644 index 0000000..4ddf577 Binary files /dev/null and b/scripts/screen3.png differ diff --git a/scripts/screen4.png b/scripts/screen4.png new file mode 100644 index 0000000..b8874ab Binary files /dev/null and b/scripts/screen4.png differ diff --git a/scripts/screen_nh.png b/scripts/screen_nh.png new file mode 100644 index 0000000..d318893 Binary files /dev/null and b/scripts/screen_nh.png differ diff --git a/scripts/screen_nh2.png b/scripts/screen_nh2.png new file mode 100644 index 0000000..30d4188 Binary files /dev/null and b/scripts/screen_nh2.png differ diff --git a/scripts/set_heimdall_public.py b/scripts/set_heimdall_public.py new file mode 100644 index 0000000..c91a56a --- /dev/null +++ b/scripts/set_heimdall_public.py @@ -0,0 +1,34 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +py_code = """ +import sqlite3 + +db_path = '/var/lib/docker-data/heimdall/config/www/app.sqlite' +con = sqlite3.connect(db_path) +cur = con.cursor() + +# Set public_front to 1 for user 1 so dashboard tiles are always visible even when logged out! +cur.execute('UPDATE users SET public_front = 1 WHERE id = 1') +con.commit() + +print('Updated user 1:') +print(cur.execute('SELECT id, username, email, public_front FROM users WHERE id = 1').fetchone()) +con.close() +""" + +stdin, stdout, stderr = c.exec_command("cat > /tmp/set_public_front.py") +stdin.write(py_code) +stdin.close() +stdout.channel.recv_exit_status() + +_, o, e = c.exec_command("echo Katzir476! | sudo -S python3 /tmp/set_public_front.py") +print(o.read().decode()) +if e.read().decode(): + print("ERR:", e.read().decode()) + +c.close() + diff --git a/scripts/setup_game_dashboard.py b/scripts/setup_game_dashboard.py new file mode 100644 index 0000000..d5885ae --- /dev/null +++ b/scripts/setup_game_dashboard.py @@ -0,0 +1,566 @@ +import os +import sys +import time +import paramiko + +HOST = "140.44.44.80" +USER = "root" +KEY_FILE = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +def main(): + print(f"Connecting to {USER}@{HOST}...") + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(HOST, username=USER, key_filename=KEY_FILE, timeout=10) + print("Connected!") + + def run(cmd, check=True): + print(f"\n>>> [RUN] {cmd}") + stdin, stdout, stderr = ssh.exec_command(cmd) + while not stdout.channel.exit_status_ready(): + time.sleep(0.5) + out = stdout.read().decode() + err = stderr.read().decode() + rc = stdout.channel.recv_exit_status() + if out: + print(out.strip()) + if err and "warning" not in err.lower(): + print(f"[STDERR] {err.strip()}", file=sys.stderr) + if check and rc != 0: + print(f"[FAIL] Exit code: {rc}", file=sys.stderr) + return rc, out, err + + # 1. Create directories + run("mkdir -p /opt/game-web/html") + + # 2. Write the python generator script + py_script = '''#!/usr/bin/env python3 +import json +import subprocess +import datetime +import os +import yaml + +GAMES = [ + { + "id": "valheim", + "name": "Valheim", + "tagline": "Viking Survival Craft (Crossplay)", + "icon": "⚔️", + "container": "gameseed-valheim", + "ports": "2456 - 2457 UDP", + "public_endpoint": "174.106.252.96:2456", + "local_endpoint": "140.44.44.80:2456", + "server_name": "SnarfNet Valheim Valhalla", + "world_name": "Snarfheim", + "password": "SnarfNet2026!", + "save_path": "/opt/game-servers/valheim/worlds (NVMe)", + "steam_app_id": "896660", + }, + { + "id": "enshrouded", + "name": "Enshrouded", + "tagline": "Action RPG Survival", + "icon": "🌲", + "container": "gameseed-enshrouded", + "ports": "15636 - 15637 UDP", + "public_endpoint": "174.106.252.96:15636", + "local_endpoint": "140.44.44.80:15636", + "server_name": "SnarfNet Enshrouded Sanctuary", + "world_name": "Sanctuary", + "password": "SnarfNet2026!", + "save_path": "/opt/game-servers/enshrouded/savegame (NVMe)", + "steam_app_id": "2278520", + }, + { + "id": "vrising", + "name": "V Rising", + "tagline": "Gothic Action RPG", + "icon": "🧛", + "container": "gameseed-vrising", + "ports": "9876 - 9877 UDP", + "public_endpoint": "174.106.252.96:9876", + "local_endpoint": "140.44.44.80:9876", + "server_name": "SnarfNet V Rising Vampire Lair", + "world_name": "SnarfNet_Vampires", + "password": "SnarfNet2026!", + "save_path": "/opt/game-servers/vrising/Saves (NVMe)", + "steam_app_id": "1829350", + }, + { + "id": "stationeers", + "name": "Stationeers", + "tagline": "Hardcore Space Colonization", + "icon": "🚀", + "container": "gameseed-stationeers", + "ports": "27015 - 27016 UDP", + "public_endpoint": "174.106.252.96:27016", + "local_endpoint": "140.44.44.80:27016", + "server_name": "SnarfNet Stationeers Moonbase", + "world_name": "Moon", + "password": "SnarfNet2026!", + "save_path": "/opt/game-servers/stationeers/saves (NVMe)", + "steam_app_id": "600760", + }, + { + "id": "fs25", + "name": "Farming Simulator 25", + "tagline": "Agriculture & Machinery Simulator", + "icon": "🚜", + "container": "gameseed-farming-simulator-25", + "ports": "10823 TCP/UDP", + "public_endpoint": "174.106.252.96:10823", + "local_endpoint": "140.44.44.80:10823", + "server_name": "SnarfNet FS25 Countryside", + "world_name": "Countryside", + "password": "SnarfNetAdmin2026!", + "save_path": "/opt/game-servers/fs25/savegame (NVMe)", + "steam_app_id": "2300320", + } +] + +def get_container_status(container_name): + cmd = ["docker", "inspect", container_name] + res = subprocess.run(cmd, capture_output=True, text=True) + if res.returncode != 0: + return "shutdown", "Not Found", "Container not created" + try: + data = json.loads(res.stdout)[0] + state = data.get("State", {}) + status = state.get("Status", "unknown").lower() + started_at = state.get("StartedAt", "") + health = state.get("Health", {}).get("Status", "") + + if status == "running": + if health == "unhealthy": + return "error", "Unhealthy", "Container running but failing health checks" + elif health == "starting": + return "starting", "Starting", "Initializing container and game world" + return "active", "Active / Online", f"Running since {started_at[:19].replace('T', ' ')}" + elif status == "restarting": + return "error", "Error / Restarting", "Container is crash-looping or awaiting assets" + elif status == "exited": + return "shutdown", "Shutdown", f"Exited (code {state.get('ExitCode')})" + else: + return status, status.capitalize(), f"State: {status}" + except Exception as e: + return "error", "Error", str(e) + +def get_host_metrics(): + try: + uptime = subprocess.check_output(["uptime", "-p"], text=True).strip() + except Exception: + uptime = "Up" + + try: + mem_raw = subprocess.check_output(["free", "-m"], text=True).splitlines()[1].split() + mem_total = round(int(mem_raw[1]) / 1024, 1) + mem_used = round(int(mem_raw[2]) / 1024, 1) + mem_str = f"{mem_used} GB / {mem_total} GB" + except Exception: + mem_str = "N/A" + + try: + df_nvme = subprocess.check_output(["df", "-h", "/mnt/nvme-games"], text=True).splitlines()[1].split() + disk_str = f"{df_nvme[2]} used / {df_nvme[1]} total ({df_nvme[3]} free)" + except Exception: + disk_str = "1 TB NVMe" + + return uptime, mem_str, disk_str + +def main(): + now_str = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S EST") + uptime, mem_str, disk_str = get_host_metrics() + + cards_html = "" + active_count = 0 + + for g in GAMES: + status_key, status_title, status_desc = get_container_status(g["container"]) + if status_key == "active": + active_count += 1 + badge_class = "badge-active" + dot_class = "dot-active" + elif status_key == "starting": + badge_class = "badge-starting" + dot_class = "dot-starting" + elif status_key == "error": + badge_class = "badge-error" + dot_class = "dot-error" + else: + badge_class = "badge-shutdown" + dot_class = "dot-shutdown" + + cards_html += f""" +
+
+
+ {g['icon']} +
+

{g['name']}

+ {g['tagline']} +
+
+
+ + {status_title} +
+
+
+
{status_desc}
+
+
+ + {g['public_endpoint']} +
+
+ + {g['local_endpoint']} +
+
+ + {g['ports']} +
+
+ + {g['password']} +
+
+
+ Server Name: {g['server_name']} + World: {g['world_name']} +
+
+ Save Location: {g['save_path']} +
+
+
+ """ + + html = f""" + + + + + SnarfNet Game Servers + + + + + + +
+
+
+
+
🎮
+
+

SnarfNet Game Servers

+
sngame4lnxp01 • 140.44.44.80 (Compute VLAN) • Public IP: 174.106.252.96
+
+
+
+
+
+ Active Games + {active_count} of {len(GAMES)} Online +
+
+ Memory Utilization + {mem_str} +
+
+ NVMe Game Storage + {disk_str} +
+
+ System Uptime + {uptime} +
+
+ Last Status Probe + {now_str} +
+
+
+ +
+ {cards_html} +
+ +
+ SnarfNet Dedicated Game Management Plane • Auto-refreshed hourly via cron • Powered by Docker on NVMe Storage +
+
+ + +""" + with open("/opt/game-web/html/index.html", "w", encoding="utf-8") as f: + f.write(html) + print("Generated /opt/game-web/html/index.html successfully!") + + # Restart web container if running + res = subprocess.run(["docker", "ps", "-q", "-f", "name=gameseed-web"], capture_output=True, text=True) + if res.stdout.strip(): + subprocess.run(["docker", "restart", "gameseed-web"]) + print("Restarted gameseed-web container.") + +if __name__ == "__main__": + main() +''' + + # Write py script to remote + sftp = ssh.open_sftp() + with sftp.open("/opt/game-web/update_status.py", "w") as f: + f.write(py_script) + sftp.close() + run("chmod +x /opt/game-web/update_status.py") + + # 3. Generate initial index.html + run("python3 /opt/game-web/update_status.py") + + # 4. Start Nginx container + run("docker run -d --name gameseed-web --restart unless-stopped -p 8080:80 -v /opt/game-web/html:/usr/share/nginx/html:ro nginx:alpine || true") + run("docker ps -f name=gameseed-web") + + # 5. Configure hourly cron job + cron_line = "0 * * * * /usr/bin/python3 /opt/game-web/update_status.py > /var/log/gameseed-web.log 2>&1" + run(f"(crontab -l 2>/dev/null | grep -v 'update_status.py'; echo '{cron_line}') | crontab -") + run("crontab -l") + + # 6. Test local curl + run("curl -I http://localhost:8080") + + ssh.close() + print("\nWeb server and hourly cron setup completed successfully!") + +if __name__ == "__main__": + main() + diff --git a/scripts/setup_game_server.py b/scripts/setup_game_server.py new file mode 100644 index 0000000..9b36d03 --- /dev/null +++ b/scripts/setup_game_server.py @@ -0,0 +1,68 @@ +import os +import sys +import time +from pathlib import Path +import paramiko + +HOST = "140.44.44.80" +USER = "snarf" +PASSWORD = "Katzir476!" + +def main(): + print(f"Connecting to {USER}@{HOST}...") + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + ssh.connect(HOST, username=USER, password=PASSWORD, timeout=10) + print("Connected successfully via password!") + + def run_cmd(cmd, sudo=False): + if sudo: + # Run with sudo using password + full_cmd = f"echo '{PASSWORD}' | sudo -S bash -c \"{cmd}\"" + else: + full_cmd = cmd + print(f"\n[RUN] {cmd}") + stdin, stdout, stderr = ssh.exec_command(full_cmd) + out = stdout.read().decode() + err = stderr.read().decode() + rc = stdout.channel.recv_exit_status() + if out: + print(out.strip()) + if err and "password" not in err.lower(): + print(f"[ERR] {err.strip()}", file=sys.stderr) + return rc, out, err + + # 1. Inspect system + run_cmd("whoami; hostname; ip addr show | grep inet; lsb_release -a || cat /etc/os-release") + + # 2. Setup SSH Keys + pub_keys = [] + ssh_dir = Path(r"C:\Users\Snarf\.ssh") + for key_file in ["id_ed25519_snarfnet.pub", "Devices.pub"]: + p = ssh_dir / key_file + if p.exists(): + pub_keys.append(p.read_text(encoding="utf-8").strip()) + print(f"Loaded public key from {p}") + + if pub_keys: + joined_keys = "\n".join(pub_keys) + # Ensure ~/.ssh exists and add keys for snarf + run_cmd("mkdir -p ~/.ssh && chmod 700 ~/.ssh") + # Write to authorized_keys + sftp = ssh.open_sftp() + with sftp.open(".ssh/authorized_keys", "a") as f: + for k in pub_keys: + f.write(f"\n{k}\n") + sftp.chmod(".ssh/authorized_keys", 0o600) + print("Updated ~/.ssh/authorized_keys for snarf") + + # Also add to /root/.ssh/authorized_keys + run_cmd(f"mkdir -p /root/.ssh && chmod 700 /root/.ssh && echo '{joined_keys}' >> /root/.ssh/authorized_keys && chmod 600 /root/.ssh/authorized_keys", sudo=True) + print("Updated /root/.ssh/authorized_keys for root") + + ssh.close() + print("Initial connection and SSH key deployment finished!") + +if __name__ == "__main__": + main() + diff --git a/scripts/setup_uptime_kuma_spacedock.py b/scripts/setup_uptime_kuma_spacedock.py new file mode 100644 index 0000000..95f218e --- /dev/null +++ b/scripts/setup_uptime_kuma_spacedock.py @@ -0,0 +1,35 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10) + +pod = "uptime-kuma-2-1756668961-5d74fcf9d-hm2xr" + +# 1. Attach Discord notification (id: 1) to monitors 57 and 94 +sql = """ +USE kuma; + +-- Attach Discord notification (1) to monitors 57 and 94 if not already attached +INSERT IGNORE INTO monitor_notification (monitor_id, notification_id) VALUES (57, 1); +INSERT IGNORE INTO monitor_notification (monitor_id, notification_id) VALUES (94, 1); + +-- Set Discord Notification (1) to active and is_default = 1 so all future monitors get Discord alerts! +UPDATE notification SET active = 1, is_default = 1 WHERE id = 1; + +-- Check notifications for 57 and 94 +SELECT mn.*, m.name, n.name as notif_name FROM monitor_notification mn +JOIN monitor m ON mn.monitor_id=m.id +JOIN notification n ON mn.notification_id=n.id +WHERE mn.monitor_id IN (57, 94); +""" + +cmd = f'kubectl exec -n monitoring {pod} -- mariadb -u root --socket=/app/data/run/mariadb.sock kuma -e "{sql}"' +_, o, e = c.exec_command(cmd) +print(o.read().decode()) +if e.read().decode(): + print("ERR:", e.read().decode()) + +c.close() + diff --git a/scripts/start_all_mounts.py b/scripts/start_all_mounts.py new file mode 100644 index 0000000..18e1c1f --- /dev/null +++ b/scripts/start_all_mounts.py @@ -0,0 +1,36 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" + +nodes = { + "snaisvr44hrvstr01": "140.44.4.81", + "snaisvr44hrvstr02": "140.44.4.82", + "snaisvr44hrvstr03": "140.44.4.83", + "snaisvr44hrvstr04": "140.44.4.84" +} + +start_cmd = """ +systemctl daemon-reload +for u in /etc/systemd/system/var-lib-harvester-extrax2ddisks-*.mount; do + [ -f "$u" ] || continue + uname=$(basename "$u") + systemctl start "$uname" + systemctl enable "$uname" +done +""" + +for hname, ip in nodes.items(): + print(f"\n=================== Starting mounts on {hname} ({ip}) ===================") + nc = paramiko.SSHClient() + nc.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + try: + nc.connect(ip, username="rancher", key_filename=SSH_KEY, timeout=10) + _, o, e = nc.exec_command(f"echo Katzir476! | sudo -S sh -c '{start_cmd}'") + o.channel.recv_exit_status() + + _, o, _ = nc.exec_command("df -h | grep extra-disks") + print(o.read().decode()) + nc.close() + except Exception as e: + print("Error:", e) + diff --git a/scripts/summarize_panels.py b/scripts/summarize_panels.py new file mode 100644 index 0000000..7208ab9 --- /dev/null +++ b/scripts/summarize_panels.py @@ -0,0 +1,16 @@ +import json + +with open("homelab-gitops-repo/dashboards/grafana/systems-dashboard.json") as f: + sys_dash = json.load(f) + +with open("homelab-gitops-repo/dashboards/grafana/container-metrics-dashboard.json") as f: + cnt_dash = json.load(f) + +print("Systems panels summary:") +for p in sys_dash.get("panels", []): + print(f"ID {p.get('id')} | Type: {p.get('type')} | Title: {p.get('title')} | Grid: {p.get('gridPos')}") + +print("\nContainers panels summary:") +for p in cnt_dash.get("panels", []): + print(f"ID {p.get('id')} | Type: {p.get('type')} | Title: {p.get('title')} | Grid: {p.get('gridPos')}") + diff --git a/scripts/sync_argo_grafana.py b/scripts/sync_argo_grafana.py new file mode 100644 index 0000000..693856a --- /dev/null +++ b/scripts/sync_argo_grafana.py @@ -0,0 +1,12 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.41', username='root', password='dietpi', timeout=10) + +# Trigger sync for aerodrome-grafana +_, o, _ = c.exec_command("kubectl patch applications.argoproj.io aerodrome-grafana -n infrastructure --type merge -p '{\"operation\":{\"sync\":{\"prune\":true}}}'") +print("Trigger sync aerodrome-grafana:", o.read().decode().strip()) + +c.close() + diff --git a/scripts/sync_argo_prom.py b/scripts/sync_argo_prom.py new file mode 100644 index 0000000..7656692 --- /dev/null +++ b/scripts/sync_argo_prom.py @@ -0,0 +1,17 @@ +import paramiko +import json + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.41', username='root', password='dietpi', timeout=10) +_, o, _ = c.exec_command("kubectl get applications.argoproj.io aerodrome-prometheus -n infrastructure -o json") +data = json.loads(o.read().decode().strip()) +print("Sync status:", data.get("status", {}).get("sync", {}).get("status")) +print("Revision:", data.get("status", {}).get("sync", {}).get("revision")) + +# Trigger sync via argocd CLI or patch +_, o, _ = c.exec_command("kubectl patch applications.argoproj.io aerodrome-prometheus -n infrastructure --type merge -p '{\"operation\":{\"sync\":{\"prune\":true}}}'") +print("Trigger sync:", o.read().decode().strip()) + +c.close() + diff --git a/scripts/sync_argocd.py b/scripts/sync_argocd.py new file mode 100644 index 0000000..0a040aa --- /dev/null +++ b/scripts/sync_argocd.py @@ -0,0 +1,10 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.41', username='root', password='dietpi', timeout=10) +# Annotate aerodrome-prometheus with refresh=hard +_, o, _ = c.exec_command("kubectl annotate app aerodrome-prometheus -n infrastructure argocd.argoproj.io/refresh=hard --overwrite") +print("Refresh annotated:", o.read().decode().strip()) +c.close() + diff --git a/scripts/test_after_fw.py b/scripts/test_after_fw.py new file mode 100644 index 0000000..57ff4d8 --- /dev/null +++ b/scripts/test_after_fw.py @@ -0,0 +1,47 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- Ping 14.4.4.44 from 140.44.4.74 ---") +out, err = run("ping -c 3 -W 2 14.4.4.44") +print("PING OUT:\n", out) +if err: print("PING ERR:\n", err) + +print("--- Test TCP 3493 from inside PeaNUT container ---") +cid, _ = run("echo 'Katzir476!' | sudo -S docker ps -q -f name=peanut_peanut") +if cid: + node_test = """ +const net = require('net'); +const sock = new net.Socket(); +sock.setTimeout(3000); +sock.on('connect', () => { + console.log('SUCCESS: CONNECTED to 14.4.4.44:3493!'); + sock.write('VER\\n'); +}); +sock.on('data', (d) => { + console.log('RECV from NUT:', d.toString()); + sock.write('LIST UPS\\n'); +}); +sock.on('timeout', () => { + console.log('TIMEOUT'); + sock.destroy(); +}); +sock.on('error', (e) => { + console.log('ERROR:', e.message); +}); +""" + run(f"cat << 'EOF' > /tmp/test_nut2.js\n{node_test}\nEOF") + out, err = run(f"echo 'Katzir476!' | sudo -S docker exec -i {cid} node < /tmp/test_nut2.js") + print("NODE TEST OUT:\n", out) + if err: print("NODE TEST ERR:\n", err) + +c.close() + diff --git a/scripts/test_clear_failed.py b/scripts/test_clear_failed.py new file mode 100644 index 0000000..18578f1 --- /dev/null +++ b/scripts/test_clear_failed.py @@ -0,0 +1,34 @@ +import paramiko +import time + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +# Try clearing failedAt +k8s("echo eyJzcGVjIjp7ImZhaWxlZEF0IjoiIn19 | base64 -d > /tmp/clear_failed.json") +out, _ = k8s("cat /tmp/clear_failed.json") +print("Patch file:", out) + +out, err = k8s("kubectl patch replica.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-r-3a9069af -n longhorn-system --type=merge --patch-file=/tmp/clear_failed.json") +print("Patch failedAt to empty:", out, err) + +# Check replica spec now +out, _ = k8s("kubectl get replica.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-r-3a9069af -n longhorn-system -o yaml | grep -E 'failedAt|healthyAt|salvageRequested'") +print("\nReplica spec after patch:") +print(out) + +# Check longhorn manager log for this volume +time.sleep(3) +out, _ = k8s("kubectl logs -n longhorn-system -l app=longhorn-manager --tail=20 | grep pvc-4da99c48") +print("\nLonghorn manager log:") +print(out) + +c.close() + diff --git a/scripts/test_cluster_queries.py b/scripts/test_cluster_queries.py new file mode 100644 index 0000000..1f7353b --- /dev/null +++ b/scripts/test_cluster_queries.py @@ -0,0 +1,23 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") +def prom_query(q): + url = "https://grafana.snarfnet.net/api/datasources/proxy/1/api/v1/query" + resp = requests.get(url, params={"query": q}, auth=auth, verify=False, timeout=10) + return resp.json().get("data", {}).get("result", []) + +print("--- Testing Aerodrome queries ---") +print("Nodes:", len(prom_query("node_load1{cluster='aerodrome'}"))) +print("Containers:", len(prom_query("container_last_seen{container!='', instance=~'sninfr2.*'}"))) + +print("\n--- Testing Spaceport queries ---") +print("Nodes:", len(prom_query("node_load1{cluster='spaceport'}"))) +print("Containers:", len(prom_query("container_last_seen{container!='', cluster='spaceport'}"))) + +print("\n--- Testing Shipyard queries ---") +print("Nodes:", len(prom_query("node_load1{cluster='shipyard'}"))) +print("Containers:", len(prom_query("container_last_seen{cluster='shipyard'}"))) + diff --git a/scripts/test_container_inside.py b/scripts/test_container_inside.py new file mode 100644 index 0000000..2ba752d --- /dev/null +++ b/scripts/test_container_inside.py @@ -0,0 +1,52 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +cid, _ = run("echo 'Katzir476!' | sudo -S docker ps -q -f name=peanut_peanut") + +# Run node script inside container to test reading settings.yml and connecting to NUT server +node_test = """ +const fs = require('fs'); +const net = require('net'); + +console.log('--- Reading /config/settings.yml ---'); +try { + console.log(fs.readFileSync('/config/settings.yml', 'utf8')); +} catch (e) { + console.log('Error reading settings:', e.message); +} + +console.log('--- Testing TCP connection to 14.4.4.44:3493 ---'); +const sock = new net.Socket(); +sock.setTimeout(3000); +sock.on('connect', () => { + console.log('CONNECTED to 14.4.4.44:3493!'); + sock.write('VER\\n'); +}); +sock.on('data', (d) => { + console.log('RECV:', d.toString()); + sock.end(); +}); +sock.on('timeout', () => { + console.log('TIMEOUT connecting to 14.4.4.44:3493'); + sock.destroy(); +}); +sock.on('error', (e) => { + console.log('ERROR:', e.message); +}); +""" + +out, err = run(f"echo 'Katzir476!' | sudo -S docker exec {cid} node -e {repr(node_test)}") +print(out) +if err: print("ERR:", err) + +c.close() + diff --git a/scripts/test_container_inside2.py b/scripts/test_container_inside2.py new file mode 100644 index 0000000..fc4574e --- /dev/null +++ b/scripts/test_container_inside2.py @@ -0,0 +1,52 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +cid, _ = run("echo 'Katzir476!' | sudo -S docker ps -q -f name=peanut_peanut") + +node_script = """ +const fs = require('fs'); +const net = require('net'); + +console.log('--- Reading /config/settings.yml ---'); +try { + console.log(fs.readFileSync('/config/settings.yml', 'utf8')); +} catch (e) { + console.log('Error reading settings:', e.message); +} + +console.log('--- Testing TCP connection to 14.4.4.44:3493 ---'); +const sock = new net.Socket(); +sock.setTimeout(3000); +sock.on('connect', () => { + console.log('CONNECTED to 14.4.4.44:3493!'); + sock.write('VER\\n'); +}); +sock.on('data', (d) => { + console.log('RECV:', d.toString()); + sock.end(); +}); +sock.on('timeout', () => { + console.log('TIMEOUT connecting to 14.4.4.44:3493'); + sock.destroy(); +}); +sock.on('error', (e) => { + console.log('ERROR:', e.message); +}); +""" + +run(f"cat << 'EOF' > /tmp/test_nut.js\n{node_script}\nEOF") +out, err = run(f"echo 'Katzir476!' | sudo -S docker exec -i {cid} node < /tmp/test_nut.js") +print("OUTPUT:\n", out) +if err: print("ERR:\n", err) + +c.close() + diff --git a/scripts/test_discord_webhook.py b/scripts/test_discord_webhook.py new file mode 100644 index 0000000..ed228c6 --- /dev/null +++ b/scripts/test_discord_webhook.py @@ -0,0 +1,20 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +webhook_url = "https://discord.com/api/webhooks/1131789867286806629/2Yjm-RF-OVDYKOIacvdEcOC1iHtoDbFBdIZWO2F3DNEHHKGWOmVns9uMFh19UsV_4cg9" +payload = { + "content": "✅ **[SnarfNet Ops] Infrastructure Alerting Audit & Verification**\n- Spacedock HCI Cluster & Nodes registered in Zabbix & Uptime Kuma.\n- Active dual-channel notifications (Discord & Gotify) verified.\n- All 5 Longhorn volumes & AI stack pods healthy." +} +req = urllib.request.Request(webhook_url, data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json", "User-Agent": "SnarfNetOps/1.0"}) +try: + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + print("Discord Webhook Status:", resp.getcode()) +except Exception as e: + print("Discord Webhook Error:", e) + diff --git a/scripts/test_discord_webhook_infra.py b/scripts/test_discord_webhook_infra.py new file mode 100644 index 0000000..235a821 --- /dev/null +++ b/scripts/test_discord_webhook_infra.py @@ -0,0 +1,20 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +webhook_url = "https://discord.com/api/webhooks/1566076777963593772/wKN30N0ZXVJKGP-prZ1_P9cp_u7Oakmh8R4bVh0qr8RFpyeGcJZrHfPQBJxtfzQqb4dn" +payload = { + "content": "✅ **[SnarfNet Ops] Alerting Audit: Discord Webhook Verification**\n- Testing `#infra-alerts` channel delivery." +} +req = urllib.request.Request(webhook_url, data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json", "User-Agent": "SnarfNetOps/1.0"}) +try: + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + print("Discord Webhook Status:", resp.getcode()) +except Exception as e: + print("Discord Webhook Error:", e) + diff --git a/scripts/test_docker_pull.py b/scripts/test_docker_pull.py new file mode 100644 index 0000000..7cc79c7 --- /dev/null +++ b/scripts/test_docker_pull.py @@ -0,0 +1,10 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, e = c.exec_command("echo 'Katzir476!' | sudo -S docker pull prom/node-exporter:v1.8.2 && echo 'Katzir476!' | sudo -S docker pull gcr.io/cadvisor/cadvisor:v0.49.1") +print(o.read().decode().strip()) +print(e.read().decode().strip()) +c.close() + diff --git a/scripts/test_final.py b/scripts/test_final.py new file mode 100644 index 0000000..0c5fc1f --- /dev/null +++ b/scripts/test_final.py @@ -0,0 +1,21 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') + +c_sd = paramiko.SSHClient() +c_sd.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c_sd.connect('140.44.4.90', username='rancher', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, e = c_sd.exec_command("sudo KUBECONFIG=/etc/rancher/rke2/rke2.yaml /var/lib/rancher/rke2/bin/kubectl get pods -A | grep -E 'glm|vllm|qdrant|triton'") +print("Spacedock Pods:\n", o.read().decode().strip()) +print("Spacedock Err:\n", e.read().decode().strip()) +c_sd.close() + +c_h = paramiko.SSHClient() +c_h.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c_h.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, e = c_h.exec_command("echo 'Katzir476!' | sudo -S sqlite3 /var/lib/docker-data/heimdall/config/www/app.sqlite 'SELECT id, username, public_front FROM users;'") +print("\nHeimdall Users:\n", o.read().decode().strip()) +print("Heimdall Err:\n", e.read().decode().strip()) +c_h.close() + diff --git a/scripts/test_heimdall_html.py b/scripts/test_heimdall_html.py new file mode 100644 index 0000000..04b365c --- /dev/null +++ b/scripts/test_heimdall_html.py @@ -0,0 +1,13 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +cmd = "curl -sk https://heimdall.snarfnet.net | grep -iE 'HomeAssistant|Rancher|Portainer|Uptime|Harvester'" +_, o, _ = c.exec_command(cmd) +print("Heimdall apps found in HTML:\n", o.read().decode().strip()) +c.close() + diff --git a/scripts/test_heimdall_http.py b/scripts/test_heimdall_http.py new file mode 100644 index 0000000..656e354 --- /dev/null +++ b/scripts/test_heimdall_http.py @@ -0,0 +1,19 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +cmd = "curl -sk https://heimdall.snarfnet.net | grep -o -E 'item-title[^<]+' | head -10" +_, o, e = c.exec_command(cmd) +print("Heimdall HTML Tiles rendered:\n", o.read().decode().strip()) + +# Also verify HTTP status code +cmd2 = "curl -sk -o /dev/null -w '%{http_code}' https://heimdall.snarfnet.net" +_, o2, _ = c.exec_command(cmd2) +print("HTTP status code:", o2.read().decode().strip()) + +c.close() + diff --git a/scripts/test_insert_kuma.py b/scripts/test_insert_kuma.py new file mode 100644 index 0000000..87d89e8 --- /dev/null +++ b/scripts/test_insert_kuma.py @@ -0,0 +1,22 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + _, o, e = c.exec_command(cmd) + out = o.read().decode().strip() + err = e.read().decode().strip() + if err: + print("ERR:", err) + return out + +pod = run("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'") +insert_sql = """INSERT INTO kuma.monitor (name, active, user_id, `interval`, type, weight, hostname, maxretries, retry_interval) VALUES ('Spacedock Harvester Node 01 (140.44.4.81)', 1, 1, 60, 'ping', 2000, '140.44.4.81', 1, 60);""" +print(run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "{insert_sql}" """)) + +c.close() + diff --git a/scripts/test_mount_sdb.py b/scripts/test_mount_sdb.py new file mode 100644 index 0000000..b1c0732 --- /dev/null +++ b/scripts/test_mount_sdb.py @@ -0,0 +1,27 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +print("Listing /var/lib/harvester/extra-disks/:") +out, _ = run_cmd("ls -la /var/lib/harvester/extra-disks/") +print(out) + +# Check mount command for /dev/sdb to /var/lib/harvester/extra-disks/sdb +print("\nTest mounting /dev/sdb on /var/lib/harvester/extra-disks/sdb:") +out, err = run_cmd("mount /dev/sdb /var/lib/harvester/extra-disks/sdb; ls -la /var/lib/harvester/extra-disks/sdb") +print(out, err) + +# Check if longhorn-disk.cfg exists on /dev/sdb +out, _ = run_cmd("ls -la /var/lib/harvester/extra-disks/sdb/longhorn-disk.cfg 2>/dev/null; cat /var/lib/harvester/extra-disks/sdb/longhorn-disk.cfg 2>/dev/null") +print("longhorn-disk.cfg:", out) + +c.close() + diff --git a/scripts/test_nut_direct.py b/scripts/test_nut_direct.py new file mode 100644 index 0000000..bc1313d --- /dev/null +++ b/scripts/test_nut_direct.py @@ -0,0 +1,35 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +print("--- nc test from 140.44.4.74 ---") +out, err = run("nc -zv -w 2 14.4.4.44 3493") +print("OUT:", out) +print("ERR:", err) + +# Check NUT communication using python +py_test = """ +import socket +s = socket.socket() +s.settimeout(3) +s.connect(('14.4.4.44', 3493)) +s.sendall(b'VER\\n') +print('VER response:', s.recv(1024)) +s.sendall(b'LIST UPS\\n') +print('LIST UPS response:', s.recv(1024)) +s.close() +""" +out, err = run(f"python3 -c {repr(py_test)}") +print("Python test out:\n", out) +if err: print("Python test err:\n", err) + +c.close() + diff --git a/scripts/test_packet_flow.py b/scripts/test_packet_flow.py new file mode 100644 index 0000000..dd891db --- /dev/null +++ b/scripts/test_packet_flow.py @@ -0,0 +1,35 @@ +import paramiko +import sys +import time + +sys.stdout.reconfigure(encoding='utf-8') + +# Connect to nutserver (14.4.4.44) +c_nut = paramiko.SSHClient() +c_nut.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c_nut.connect('14.4.4.44', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +# Connect to shipyard node 04 (140.44.4.74) +c_ship = paramiko.SSHClient() +c_ship.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c_ship.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +# Start tcpdump on nutserver +print("Starting tcpdump on nutserver...") +stdin, stdout_dump, stderr_dump = c_nut.exec_command("echo 'Katzir476!' | sudo -S tcpdump -i wlan0 -n -c 10 'host 140.44.4.74 or port 3493'") + +time.sleep(1) + +# Send curl / nc from 140.44.4.74 +print("Sending test probe from 140.44.4.74...") +c_ship.exec_command("nc -z -w 3 14.4.4.44 3493 || ping -c 2 -W 2 14.4.4.44") + +time.sleep(3) + +# Read tcpdump output +dump_out = stdout_dump.read().decode('utf-8', errors='replace') +print("TCPDUMP OUTPUT ON NUTSERVER:\n", dump_out) + +c_nut.close() +c_ship.close() + diff --git a/scripts/test_peanut_http.py b/scripts/test_peanut_http.py new file mode 100644 index 0000000..5a953ff --- /dev/null +++ b/scripts/test_peanut_http.py @@ -0,0 +1,36 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + out = stdout.read().decode('utf-8', errors='replace') + err = stderr.read().decode('utf-8', errors='replace') + return out.strip(), err.strip() + +print("1. Curl peanut.snarfnet.net directly via NPM localhost/VIP:") +out, _ = run("curl -k -I https://peanut.snarfnet.net --resolve peanut.snarfnet.net:443:140.44.4.70") +print(out) + +print("\n2. Curl peanut.snarfnet.net/device/server-closet:") +out, _ = run("curl -k -I https://peanut.snarfnet.net/device/server-closet --resolve peanut.snarfnet.net:443:140.44.4.70") +print(out) + +print("\n3. Curl http://140.44.4.70:8050/device/server-closet:") +out, _ = run("curl -I http://140.44.4.70:8050/device/server-closet") +print(out) + +print("\n4. Get body of http://140.44.4.70:8050/api (or / or inspect html):") +out, _ = run("curl -s http://140.44.4.70:8050/device/server-closet | head -n 30") +print(out[:500]) + +print("\n5. Check DNS for peanut.snarfnet.net from dns1 (140.44.4.34) or dns2 (140.44.4.62):") +out, _ = run("nslookup peanut.snarfnet.net 140.44.4.61 || dig @140.44.4.61 peanut.snarfnet.net") +print(out) + +c.close() + diff --git a/scripts/test_peanut_node.py b/scripts/test_peanut_node.py new file mode 100644 index 0000000..f1a540f --- /dev/null +++ b/scripts/test_peanut_node.py @@ -0,0 +1,24 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.74', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + stdin, stdout, stderr = c.exec_command(cmd) + return stdout.read().decode().strip(), stderr.read().decode().strip() + +cid, _ = run("echo 'Katzir476!' | sudo -S docker ps -q -f name=peanut_peanut") +print("PeaNUT CID:", cid) + +out, _ = run(f"echo 'Katzir476!' | sudo -S docker exec {cid} cat /config/settings.yml") +print("Container sees /config/settings.yml as:\n", out) + +# Can container resolve or connect? +out, err = run(f"echo 'Katzir476!' | sudo -S docker exec {cid} node -e 'console.log(process.env)'") +print("Process env in container:\n", out) + +c.close() + diff --git a/scripts/test_portal_vs_heimdall.py b/scripts/test_portal_vs_heimdall.py new file mode 100644 index 0000000..86b9e80 --- /dev/null +++ b/scripts/test_portal_vs_heimdall.py @@ -0,0 +1,26 @@ +import paramiko +import sys + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run(cmd): + _, o, _ = c.exec_command(cmd) + return o.read().decode().strip() + +print("Testing https://portal.snarfnet.net:") +status_portal = run("curl -sk -o /dev/null -w '%{http_code}' https://portal.snarfnet.net") +print(" Status code:", status_portal) +title_portal = run("curl -sk https://portal.snarfnet.net | grep -i ''") +print(" Title:", title_portal) + +print("\nTesting https://heimdall.snarfnet.net:") +status_heimdall = run("curl -sk -o /dev/null -w '%{http_code}' https://heimdall.snarfnet.net") +redirect_heimdall = run("curl -sk -I https://heimdall.snarfnet.net | grep -i 'location'") +print(" Status code:", status_heimdall) +print(" Redirect location:", redirect_heimdall) + +c.close() + diff --git a/scripts/test_proper_escape.py b/scripts/test_proper_escape.py new file mode 100644 index 0000000..f507346 --- /dev/null +++ b/scripts/test_proper_escape.py @@ -0,0 +1,41 @@ +import paramiko + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10) + +def run_cmd(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +# Remove the bad unit +run_cmd("rm -f /etc/systemd/system/var-lib-harvester-extrax2ddisks*") + +# Write the properly named unit +unit_name = r"var-lib-harvester-extra\x2ddisks-sdb.mount" +content = """[Unit] +Description=Harvester Extra Disk disk-sdb +Before=rke2-server.service rke2-agent.service + +[Mount] +What=/dev/disk/by-uuid/b5471f4d-aa33-409a-8ba4-2330b98ebe80 +Where=/var/lib/harvester/extra-disks/sdb +Type=ext4 +Options=defaults + +[Install] +WantedBy=multi-user.target +""" +run_cmd(f"cat << 'EOF' > '/etc/systemd/system/{unit_name}'\n{content}\nEOF") + +out, err = run_cmd("systemctl daemon-reload && systemctl start 'var-lib-harvester-extra\\x2ddisks-sdb.mount' && systemctl status 'var-lib-harvester-extra\\x2ddisks-sdb.mount'") +print("Status output:\n", out) +if err: print("Error:\n", err) + +out, _ = run_cmd("df -h | grep extra-disks") +print("df -h output:\n", out) + +c.close() + diff --git a/scripts/test_replace_cm.py b/scripts/test_replace_cm.py new file mode 100644 index 0000000..f50d85a --- /dev/null +++ b/scripts/test_replace_cm.py @@ -0,0 +1,18 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +cmd = "kubectl create configmap prometheus-config --from-file=prometheus.yml=/tmp/prometheus_new.yml --dry-run=client -o yaml | kubectl replace -f -" +_, o, e = c.exec_command(cmd) +print("Replace output:\n", o.read().decode().strip()) +print("Replace err:\n", e.read().decode().strip()) + +_, o, _ = c.exec_command("kubectl get configmap prometheus-config -o yaml") +yaml_str = o.read().decode().strip() +print("Length of CM:", len(yaml_str)) +print("Contains shipyard:", "shipyard" in yaml_str) + +c.close() + diff --git a/scripts/test_salvage.py b/scripts/test_salvage.py new file mode 100644 index 0000000..797cffa --- /dev/null +++ b/scripts/test_salvage.py @@ -0,0 +1,37 @@ +import paramiko +import json +import time + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +# Check replicas for pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495 +print("Replicas for pvc-4da99c48:") +out, _ = k8s("kubectl get replicas.longhorn.io -n longhorn-system -l longhornvolume=pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495 -o custom-columns=NAME:.metadata.name,FAILED:.spec.failedAt,SALVAGE:.spec.salvageRequested,NODE:.spec.nodeID") +print(out) + +# Let's test patching salvageRequested: true on replicas of pvc-4da99c48 +print("\nPatching replicas of pvc-4da99c48 to salvageRequested=true...") +patch_cmd = """ +for r in $(kubectl get replicas.longhorn.io -n longhorn-system -l longhornvolume=pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495 -o jsonpath='{.items[*].metadata.name}'); do + kubectl patch replica.longhorn.io $r -n longhorn-system --type merge -p '{"spec":{"salvageRequested":true}}' +done +""" +out, err = k8s(patch_cmd) +print("Patch result:", out, err) + +time.sleep(3) +# Check volume status now +out, _ = k8s("kubectl get volume.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495 -n longhorn-system -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness") +print("Volume status after patch:") +print(out) + +c.close() + diff --git a/scripts/test_salvage2.py b/scripts/test_salvage2.py new file mode 100644 index 0000000..3824e0b --- /dev/null +++ b/scripts/test_salvage2.py @@ -0,0 +1,38 @@ +import paramiko +import time + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +# Patch using kubectl patch with escaped json +patch_cmd = 'kubectl patch replica.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-r-3a9069af -n longhorn-system --type=merge -p \'{"spec":{"salvageRequested":true}}\'' +out, err = k8s(patch_cmd) +print("Patch 1:", out, err) + +patch_cmd = 'kubectl patch replica.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-r-593bd03f -n longhorn-system --type=merge -p \'{"spec":{"salvageRequested":true}}\'' +out, err = k8s(patch_cmd) +print("Patch 2:", out, err) + +patch_cmd = 'kubectl patch replica.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-r-b5bbbcbf -n longhorn-system --type=merge -p \'{"spec":{"salvageRequested":true}}\'' +out, err = k8s(patch_cmd) +print("Patch 3:", out, err) + +time.sleep(5) +out, _ = k8s("kubectl get volume.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495 -n longhorn-system -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness") +print("\nVolume status after patch:") +print(out) + +# Check pod using this pvc +out, _ = k8s("kubectl get pods -n ai-stack -l app=glm-superexpert") +print("\nPod status:") +print(out) + +c.close() + diff --git a/scripts/test_salvage3.py b/scripts/test_salvage3.py new file mode 100644 index 0000000..8fb022f --- /dev/null +++ b/scripts/test_salvage3.py @@ -0,0 +1,40 @@ +import paramiko +import time + +SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet" +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10) + +def k8s(cmd): + full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'" + _, o, e = c.exec_command(full_cmd) + return o.read().decode().strip(), e.read().decode().strip() + +# Create patch file +k8s('echo \'{"spec":{"salvageRequested":true}}\' > /tmp/salvage_patch.json') + +# Check patch file content +out, _ = k8s("cat /tmp/salvage_patch.json") +print("Patch file:", out) + +# Apply patch to replicas +patch_cmd = "kubectl patch replica.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-r-3a9069af -n longhorn-system --type=merge --patch-file=/tmp/salvage_patch.json" +out, err = k8s(patch_cmd) +print("Patch 1:", out, err) + +patch_cmd = "kubectl patch replica.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-r-593bd03f -n longhorn-system --type=merge --patch-file=/tmp/salvage_patch.json" +out, err = k8s(patch_cmd) +print("Patch 2:", out, err) + +patch_cmd = "kubectl patch replica.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-r-b5bbbcbf -n longhorn-system --type=merge --patch-file=/tmp/salvage_patch.json" +out, err = k8s(patch_cmd) +print("Patch 3:", out, err) + +time.sleep(4) +out, _ = k8s("kubectl get volume.longhorn.io pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495 -n longhorn-system -o custom-columns=NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness") +print("\nVolume status after patch:") +print(out) + +c.close() + diff --git a/scripts/test_shipyard_metrics.py b/scripts/test_shipyard_metrics.py new file mode 100644 index 0000000..eaa5dc8 --- /dev/null +++ b/scripts/test_shipyard_metrics.py @@ -0,0 +1,11 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("curl -s http://127.0.0.1:9100/metrics | grep -E '^node_load1 '") +print("Node exporter test:", o.read().decode().strip()) +_, o2, _ = c.exec_command("curl -s http://127.0.0.1:8080/metrics | grep -E '^container_last_seen' | head -3") +print("cAdvisor test:\n", o2.read().decode().strip()) +c.close() + diff --git a/scripts/test_spaceport_kubelet.py b/scripts/test_spaceport_kubelet.py new file mode 100644 index 0000000..ad6100c --- /dev/null +++ b/scripts/test_spaceport_kubelet.py @@ -0,0 +1,9 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) +_, o, _ = c.exec_command("curl -sk https://140.44.4.41:10250/metrics/cadvisor | head -5") +print(o.read().decode().strip()) +c.close() + diff --git a/scripts/test_spaceport_token.py b/scripts/test_spaceport_token.py new file mode 100644 index 0000000..8606cba --- /dev/null +++ b/scripts/test_spaceport_token.py @@ -0,0 +1,10 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.41', username='root', password='dietpi', timeout=10) +_, o, _ = c.exec_command("kubectl create token prometheus-scraper -n kube-system --duration=87600h") +token = o.read().decode().strip() +_, o2, e = c.exec_command(f"curl -sk -H 'Authorization: Bearer {token}' https://140.44.4.41:10250/metrics/cadvisor | head -10") +print("Head:\n", o2.read().decode().strip()) +c.close() diff --git a/scripts/update_fw_policy.py b/scripts/update_fw_policy.py new file mode 100644 index 0000000..00606a6 --- /dev/null +++ b/scripts/update_fw_policy.py @@ -0,0 +1,31 @@ +import requests +import urllib3 +import json + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +s = requests.Session() +s.verify = False + +r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'}) +csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token') +headers = {'X-CSRF-Token': csrf} if csrf else {} + +policy_id = "d92c8e87-d76e-4e69-9c19-ab0cdbe82080" +url = f'https://14.44.4.1/proxy/network/v2/api/site/default/firewall-policies/{policy_id}' + +# Fetch current policy +p = s.get(url, headers=headers).json() +print("Current policy:", p.get('name'), p.get('source', {}).get('ips')) + +# Add 14.4.4.44 +p['source']['ips'] = ['14.4.4.20', '14.4.4.44'] + +# PUT update +put_res = s.put(url, json=p, headers=headers) +print("PUT status:", put_res.status_code) +if put_res.status_code == 200: + print("PUT success! New ips:", put_res.json().get('source', {}).get('ips')) +else: + print("PUT error:", put_res.text) + diff --git a/scripts/update_grafana_gitops.py b/scripts/update_grafana_gitops.py new file mode 100644 index 0000000..6d01ad6 --- /dev/null +++ b/scripts/update_grafana_gitops.py @@ -0,0 +1,143 @@ +import json +import yaml + +# Read the generated dashboard JSONs +with open("homelab-gitops-repo/dashboards/grafana/aerodrome-resource-monitor.json", "r", encoding="utf-8") as f: + aero_json = f.read() + +with open("homelab-gitops-repo/dashboards/grafana/spaceport-resource-monitor.json", "r", encoding="utf-8") as f: + space_json = f.read() + +with open("homelab-gitops-repo/dashboards/grafana/shipyard-resource-monitor.json", "r", encoding="utf-8") as f: + ship_json = f.read() + +with open("homelab-gitops-repo/dashboards/grafana/ai-stack-dashboard.json", "r", encoding="utf-8") as f: + ai_json = f.read() + +# Load current grafana.yaml +with open("homelab-gitops-repo/manifests/aerodrome/grafana/grafana.yaml", "r", encoding="utf-8") as f: + manifest_text = f.read() + +docs = list(yaml.safe_load_all(manifest_text)) + +# Document 0: Deployment/grafana +# Document 1: Service/grafana +# Document 2: ConfigMap/grafana-dashboards-provider +# Document 3: ConfigMap/grafana-dashboard-ai-stack +# Document 4: ConfigMap/grafana-dashboard-systems +# Document 5: ConfigMap/grafana-dashboard-containers + +deploy = docs[0] +containers = deploy["spec"]["template"]["spec"]["containers"] +grafana_container = None +for c in containers: + if c["name"] == "grafana": + grafana_container = c + break + +# Update volumeMounts in grafana container +vm = [m for m in grafana_container["volumeMounts"] if m["name"] not in ["systems-dashboard", "container-metrics-dashboard", "ai-stack-dashboard"]] +vm.extend([ + {"mountPath": "/var/lib/grafana-dashboards/ai-stack/ai-stack-dashboard.json", "name": "ai-stack-dashboard", "subPath": "ai-stack-dashboard.json"}, + {"mountPath": "/var/lib/grafana-dashboards/aerodrome/aerodrome-resource-monitor.json", "name": "aerodrome-dashboard", "subPath": "aerodrome-resource-monitor.json"}, + {"mountPath": "/var/lib/grafana-dashboards/spaceport/spaceport-resource-monitor.json", "name": "spaceport-dashboard", "subPath": "spaceport-resource-monitor.json"}, + {"mountPath": "/var/lib/grafana-dashboards/shipyard/shipyard-resource-monitor.json", "name": "shipyard-dashboard", "subPath": "shipyard-resource-monitor.json"} +]) +grafana_container["volumeMounts"] = vm + +# Update volumes in pod spec +vols = [v for v in deploy["spec"]["template"]["spec"]["volumes"] if v["name"] not in ["systems-dashboard", "container-metrics-dashboard", "ai-stack-dashboard"]] +vols.extend([ + {"name": "ai-stack-dashboard", "configMap": {"defaultMode": 420, "name": "grafana-dashboard-ai-stack"}}, + {"name": "aerodrome-dashboard", "configMap": {"defaultMode": 420, "name": "grafana-dashboard-aerodrome"}}, + {"name": "spaceport-dashboard", "configMap": {"defaultMode": 420, "name": "grafana-dashboard-spaceport"}}, + {"name": "shipyard-dashboard", "configMap": {"defaultMode": 420, "name": "grafana-dashboard-shipyard"}} +]) +deploy["spec"]["template"]["spec"]["volumes"] = vols + +# Update ConfigMap/grafana-dashboards-provider (Document 2) +provider_cm = None +for d in docs: + if d and d.get("kind") == "ConfigMap" and d.get("metadata", {}).get("name") == "grafana-dashboards-provider": + provider_cm = d + break + +provider_cm["data"]["dashboards.yaml"] = """apiVersion: 1 +providers: + - name: 'ai-stack-provider' + orgId: 1 + folder: 'AI Stack & Inference' + type: file + disableDeletion: false + editable: true + options: + path: /var/lib/grafana-dashboards/ai-stack + - name: 'aerodrome-provider' + orgId: 1 + folder: 'Aerodrome' + type: file + disableDeletion: false + editable: true + options: + path: /var/lib/grafana-dashboards/aerodrome + - name: 'spaceport-provider' + orgId: 1 + folder: 'Spaceport' + type: file + disableDeletion: false + editable: true + options: + path: /var/lib/grafana-dashboards/spaceport + - name: 'shipyard-provider' + orgId: 1 + folder: 'Shipyard' + type: file + disableDeletion: false + editable: true + options: + path: /var/lib/grafana-dashboards/shipyard +""" + +# Filter out old systems and containers configmaps +new_docs = [d for d in docs if d.get("metadata", {}).get("name") not in ["grafana-dashboard-systems", "grafana-dashboard-containers"]] + +# Add new configmaps +new_docs.append({ + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": { + "name": "grafana-dashboard-aerodrome", + "namespace": "default", + "labels": {"app.kubernetes.io/name": "grafana", "app.kubernetes.io/instance": "grafana"} + }, + "data": {"aerodrome-resource-monitor.json": aero_json} +}) + +new_docs.append({ + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": { + "name": "grafana-dashboard-spaceport", + "namespace": "default", + "labels": {"app.kubernetes.io/name": "grafana", "app.kubernetes.io/instance": "grafana"} + }, + "data": {"spaceport-resource-monitor.json": space_json} +}) + +new_docs.append({ + "apiVersion": "v1", + "kind": "ConfigMap", + "metadata": { + "name": "grafana-dashboard-shipyard", + "namespace": "default", + "labels": {"app.kubernetes.io/name": "grafana", "app.kubernetes.io/instance": "grafana"} + }, + "data": {"shipyard-resource-monitor.json": ship_json} +}) + +# Write updated grafana.yaml +with open("homelab-gitops-repo/manifests/aerodrome/grafana/grafana.yaml", "w", encoding="utf-8") as f: + yaml.dump_all(new_docs, f, sort_keys=False) + +print("Successfully updated homelab-gitops-repo/manifests/aerodrome/grafana/grafana.yaml!") + diff --git a/scripts/update_kuma_discord_webhook.py b/scripts/update_kuma_discord_webhook.py new file mode 100644 index 0000000..3388107 --- /dev/null +++ b/scripts/update_kuma_discord_webhook.py @@ -0,0 +1,54 @@ +import paramiko +import sys +import json + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run_sql(pod, sql): + cmd = f"kubectl exec -i -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root" + stdin, stdout, stderr = c.exec_command(cmd) + stdin.write(sql) + stdin.close() + return stdout.read().decode().strip(), stderr.read().decode().strip() + +_, o, _ = c.exec_command("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'") +pod = o.read().decode().strip() +print("Kuma Pod:", pod) + +out, _ = run_sql(pod, "USE kuma; SELECT id, name, config FROM notification WHERE id=1;") +print("Current config for notif 1:") +print(out) + +# Update config with active webhook +cfg = { + "name": "Discord Infra Alerts (#infra-alerts)", + "type": "discord", + "isDefault": True, + "telegramServerUrl": "https://api.telegram.org", + "discordChannelType": "channel", + "disableUrl": False, + "discordSuppressNotifications": False, + "discordMessageFormat": "normal", + "discordWebhookUrl": "https://discord.com/api/webhooks/1566076777963593772/wKN30N0ZXVJKGP-prZ1_P9cp_u7Oakmh8R4bVh0qr8RFpyeGcJZrHfPQBJxtfzQqb4dn", + "applyExisting": True +} +cfg_json = json.dumps(cfg).replace("'", "''") + +update_sql = f"""USE kuma; +UPDATE notification +SET name='Discord Infra Alerts (#infra-alerts)', is_default=1, active=1, config='{cfg_json}' +WHERE id=1; +""" + +out, err = run_sql(pod, update_sql) +print("Update result err:", err) + +# Restart pod so kuma caches updated webhook +_, o, _ = c.exec_command("kubectl rollout restart deploy/uptime-kuma-2-1756668961 -n monitoring") +print("Restarting deployment:", o.read().decode().strip()) + +c.close() + diff --git a/scripts/update_npm_direct.py b/scripts/update_npm_direct.py new file mode 100644 index 0000000..369ecff --- /dev/null +++ b/scripts/update_npm_direct.py @@ -0,0 +1,53 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +py_code = """ +import sqlite3 +import json + +db_path = '/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite' +con = sqlite3.connect(db_path) +cur = con.cursor() +domains = ["portal.snarfnet.net", "heimdall.snarfnet.net"] +cur.execute("UPDATE proxy_host SET domain_names = ? WHERE id = 4", (json.dumps(domains),)) +con.commit() +print("Updated DB host 4 domain_names:", cur.execute("SELECT id, domain_names FROM proxy_host WHERE id=4").fetchone()) +con.close() + +# Also update 4.conf on disk directly +conf_path = '/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/nginx/proxy_host/4.conf' +with open(conf_path, 'r') as f: + text = f.read() + +if 'heimdall.snarfnet.net' not in text: + text = text.replace('server_name portal.snarfnet.net;', 'server_name portal.snarfnet.net heimdall.snarfnet.net;') + with open(conf_path, 'w') as f: + f.write(text) + print("Updated 4.conf successfully!") +else: + print("4.conf already has heimdall.snarfnet.net") +""" + +stdin, stdout, stderr = c.exec_command("cat > /tmp/update_npm_conf.py") +stdin.write(py_code) +stdin.close() +stdout.channel.recv_exit_status() + +_, o, e = c.exec_command("echo Katzir476! | sudo -S python3 /tmp/update_npm_conf.py") +print(o.read().decode()) +if e.read().decode(): print("ERR:", e.read().decode()) + +# Test and reload nginx inside NPM container +cid_cmd = "docker ps -q -f name=nginx-proxy-manager" +_, cid_out, _ = c.exec_command(f"echo Katzir476! | sudo -S {cid_cmd}") +cid = cid_out.read().decode().strip() +if cid: + _, o, e = c.exec_command(f"echo Katzir476! | sudo -S docker exec {cid} nginx -t") + print("Nginx -t:\n", o.read().decode(), e.read().decode()) + _, o, e = c.exec_command(f"echo Katzir476! | sudo -S docker exec {cid} nginx -s reload") + print("Nginx reload:\n", o.read().decode(), e.read().decode()) + +c.close() diff --git a/scripts/update_npm_heimdall.py b/scripts/update_npm_heimdall.py new file mode 100644 index 0000000..8ac7301 --- /dev/null +++ b/scripts/update_npm_heimdall.py @@ -0,0 +1,42 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect("140.44.4.71", username="snarf", password="Katzir476!", timeout=10) + +py_code = """ +import sqlite3 +import json + +db_path = '/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite' +con = sqlite3.connect(db_path) +cur = con.cursor() + +# Update proxy host 4 to include heimdall.snarfnet.net +domains = ["portal.snarfnet.net", "heimdall.snarfnet.net"] +cur.execute("UPDATE proxy_host SET domain_names = ? WHERE id = 4", (json.dumps(domains),)) +con.commit() +print("Updated NPM DB host 4:", cur.execute("SELECT id, domain_names FROM proxy_host WHERE id=4").fetchone()) +con.close() +""" + +stdin, stdout, stderr = c.exec_command("cat > /tmp/update_npm.py") +stdin.write(py_code) +stdin.close() +stdout.channel.recv_exit_status() + +_, o, _ = c.exec_command("echo Katzir476! | sudo -S python3 /tmp/update_npm.py") +print(o.read().decode()) + +# Update 4.conf inside NPM container and test nginx config +update_conf_cmd = """ +docker exec $(docker ps -q -f name=nginx-proxy-manager) sed -i 's/server_name portal.snarfnet.net;/server_name portal.snarfnet.net heimdall.snarfnet.net;/g' /data/nginx/proxy_host/4.conf +docker exec $(docker ps -q -f name=nginx-proxy-manager) nginx -t +docker exec $(docker ps -q -f name=nginx-proxy-manager) nginx -s reload +""" +_, o, e = c.exec_command(f"echo Katzir476! | sudo -S sh -c '{update_conf_cmd}'") +print(o.read().decode()) +print(e.read().decode()) + +c.close() + diff --git a/scripts/update_prom_cm_direct.py b/scripts/update_prom_cm_direct.py new file mode 100644 index 0000000..e9ba91f --- /dev/null +++ b/scripts/update_prom_cm_direct.py @@ -0,0 +1,54 @@ +import paramiko + +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +# 1. Get the prometheus.yml content from local file +with open(r'c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\aerodrome\prometheus\prometheus.yaml', 'r', encoding='utf-8') as f: + text = f.read() + +# Extract prometheus.yml +# It is between "prometheus.yml: |" and next "---" +start_marker = "prometheus.yml: |" +start_idx = text.find(start_marker) +end_idx = text.find("\n---", start_idx) +raw_yml = text[start_idx + len(start_marker):end_idx] + +# Unindent 4 spaces from raw_yml +unindented_lines = [] +for line in raw_yml.splitlines(): + if line.startswith(" "): + unindented_lines.append(line[4:]) + elif not line.strip(): + unindented_lines.append("") + else: + unindented_lines.append(line) +prom_clean_content = "\n".join(unindented_lines).strip() + "\n" + +# Write directly to /tmp/prometheus_new.yml +stdin, stdout, stderr = c.exec_command("cat > /tmp/prometheus_new.yml") +stdin.write(prom_clean_content) +stdin.close() +stdout.channel.recv_exit_status() + +# Update configmap using kubectl create ... --dry-run=client -o yaml | kubectl apply -f - +cmd = "kubectl create configmap prometheus-config --from-file=prometheus.yml=/tmp/prometheus_new.yml --dry-run=client -o yaml | kubectl apply -f -" +_, o, e = c.exec_command(cmd) +print("Create/Apply CM result:\n", o.read().decode().strip()) +print("Create/Apply CM err:\n", e.read().decode().strip()) + +# Verify +_, o, _ = c.exec_command("kubectl get configmap prometheus-config -o json") +import json +data = json.loads(o.read().decode().strip()) +prom_yml = data.get("data", {}).get("prometheus.yml", "") +print("\nUpdated CM contains shipyard:", "shipyard" in prom_yml) +print("Updated CM length:", len(prom_yml)) + +# Reload prometheus +_, o, _ = c.exec_command("kubectl exec deploy/prometheus -- wget -qO- --post-action='' http://localhost:9090/-/reload") +print("Reload endpoint:", o.read().decode().strip()) + +c.close() + diff --git a/scripts/update_trigger_15m.py b/scripts/update_trigger_15m.py new file mode 100644 index 0000000..fc379ba --- /dev/null +++ b/scripts/update_trigger_15m.py @@ -0,0 +1,45 @@ +import urllib.request +import json +import ssl + +ctx = ssl.create_default_context() +ctx.check_hostname = False +ctx.verify_mode = ssl.CERT_NONE + +def zabbix_api(method, params, auth=None): + payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1} + if auth: payload["auth"] = auth + req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php", + data=json.dumps(payload).encode('utf-8'), + headers={"Content-Type": "application/json-rpc"}) + resp = urllib.request.urlopen(req, context=ctx, timeout=10) + return json.loads(resp.read().decode('utf-8')) + +auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result") + +# Update Active agent template triggers (10343) +up1 = zabbix_api("trigger.update", { + "triggerid": "22415", + "expression": "min(/Linux by Zabbix agent active/system.cpu.util,15m)>{$CPU.UTIL.CRIT}" +}, auth=auth) +print("Update Active CPU trigger (15m):", up1) + +up2 = zabbix_api("trigger.update", { + "triggerid": "22421", + "expression": "min(/Linux by Zabbix agent active/vm.memory.utilization,15m)>{$MEMORY.UTIL.MAX}" +}, auth=auth) +print("Update Active Memory trigger (15m):", up2) + +# Update Passive agent template triggers (10001) +up3 = zabbix_api("trigger.update", { + "triggerid": "22370", + "expression": "min(/Linux by Zabbix agent/system.cpu.util,15m)>{$CPU.UTIL.CRIT}" +}, auth=auth) +print("Update Passive CPU trigger (15m):", up3) + +up4 = zabbix_api("trigger.update", { + "triggerid": "22376", + "expression": "min(/Linux by Zabbix agent/vm.memory.utilization,15m)>{$MEMORY.UTIL.MAX}" +}, auth=auth) +print("Update Passive Memory trigger (15m):", up4) + diff --git a/scripts/upload_nh.py b/scripts/upload_nh.py new file mode 100644 index 0000000..d3694e9 --- /dev/null +++ b/scripts/upload_nh.py @@ -0,0 +1,26 @@ +import paramiko + +ssh = paramiko.SSHClient() +ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +ssh.connect("140.44.44.80", username="root", key_filename=r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet", timeout=10) + +script = """#!/bin/bash +export DISPLAY=:1 +export WINEPREFIX=/home/nobody/.fs_server +export WINEARCH=win64 +export WINEDLLOVERRIDES="mscoree=d" +export WINEDEBUG=-all +export USER=nobody + +wine /opt/fs25/dlc/FarmingSimulator25_extraContentNewHollandCR11_1_0_0_0_ESD.exe /SILENT /NOCANCEL & +""" + +sftp = ssh.open_sftp() +with sftp.open("/opt/game-servers/fs25/run_nh.sh", "w") as f: + f.write(script) +sftp.close() + +ssh.exec_command("chmod 777 /opt/game-servers/fs25/run_nh.sh; docker cp /opt/game-servers/fs25/run_nh.sh gameseed-farming-simulator-25:/opt/fs25/run_nh.sh") +ssh.close() +print("Uploaded run_nh.sh") + diff --git a/scripts/verify_final_dashboards.py b/scripts/verify_final_dashboards.py new file mode 100644 index 0000000..3534c81 --- /dev/null +++ b/scripts/verify_final_dashboards.py @@ -0,0 +1,24 @@ +import requests +import json +import urllib3 +urllib3.disable_warnings() + +auth = ("admin", "Katzir476!!!") + +for uid in ["aerodrome-resource-monitor", "spaceport-resource-monitor", "shipyard-resource-monitor", "snarfnet-ai-stack"]: + url = f"https://grafana.snarfnet.net/api/dashboards/uid/{uid}" + resp = requests.get(url, auth=auth, verify=False, timeout=10) + data = resp.json() + dash = data.get("dashboard", {}) + title = dash.get("title") + folder_title = data.get("meta", {}).get("folderTitle") + panels = dash.get("panels", []) + rows = [p.get("title") for p in panels if p.get("type") == "row"] + non_rows = [p for p in panels if p.get("type") != "row"] + print(f"\n==========================================") + print(f"UID: {uid}") + print(f"Title: {title}") + print(f"Folder: {folder_title}") + print(f"Elevated Row Groups ({len(rows)}): {rows}") + print(f"Metric Panels Count: {len(non_rows)}") + diff --git a/scripts/verify_kuma_spacedock_status.py b/scripts/verify_kuma_spacedock_status.py new file mode 100644 index 0000000..a787385 --- /dev/null +++ b/scripts/verify_kuma_spacedock_status.py @@ -0,0 +1,31 @@ +import paramiko +import sys +import time + +sys.stdout.reconfigure(encoding='utf-8') +c = paramiko.SSHClient() +c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) +c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10) + +def run_sql(pod, sql): + cmd = f"kubectl exec -i -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root" + stdin, stdout, stderr = c.exec_command(cmd) + stdin.write(sql) + stdin.close() + return stdout.read().decode().strip(), stderr.read().decode().strip() + +_, o, _ = c.exec_command("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'") +pod = o.read().decode().strip() +print("Kuma Pod:", pod) + +out, _ = run_sql(pod, "USE kuma; SELECT id, name, type, hostname, active FROM monitor WHERE id >= 98;") +print("Monitors:") +print(out) + +# Check recent heartbeats for these monitors +out_hb, _ = run_sql(pod, "USE kuma; SELECT monitor_id, status, msg, time, ping FROM heartbeat WHERE monitor_id >= 98 ORDER BY id DESC LIMIT 10;") +print("\nRecent Heartbeats:") +print(out_hb) + +c.close() + diff --git a/snarfnet-game-seeds/core/manager.py b/snarfnet-game-seeds/core/manager.py index 2618f1f..2335f01 100644 --- a/snarfnet-game-seeds/core/manager.py +++ b/snarfnet-game-seeds/core/manager.py @@ -65,8 +65,10 @@ def cmd_plan(): print(f" Save Dir: {server.get('saveDirectory')}") print(f" Backup: {spec.get('backup', {}).get('schedule')} (Retention: {spec.get('backup', {}).get('retentionDays')}d)") -def cmd_generate_compose(output_path="docker-compose.yml"): +def cmd_generate_compose(output_path=None): """Generates standard docker-compose.yml from all declared seeds.""" + if output_path is None: + output_path = Path(__file__).resolve().parent.parent / "docker-compose.yml" seeds = load_seeds() services = {} @@ -85,31 +87,39 @@ def cmd_generate_compose(output_path="docker-compose.yml"): save_dir = server.get("saveDirectory") data_dir = f"/opt/game-servers/{name}/data" + mem_limit = str(res.get("memoryLimit", "8g")).lower().replace("gi", "g") + mem_res = str(res.get("memoryReservation", "4g")).lower().replace("gi", "g") + cpu_limit = str(res.get("cpuLimit", "4")) + + volumes = runtime.get("volumes") + if not volumes: + volumes = [ + f"{save_dir}:{save_dir}", + f"{data_dir}:/serverdata" + ] + services[name] = { "image": runtime.get("image"), "container_name": f"gameseed-{name}", "restart": runtime.get("restartPolicy", "unless-stopped"), + "env_file": [".env"], "environment": runtime.get("environment", {}), "ports": ports, - "volumes": [ - f"{save_dir}:{save_dir}", - f"{data_dir}:/serverdata" - ], + "volumes": volumes, "deploy": { "resources": { "limits": { - "memory": res.get("memoryLimit", "8Gi"), - "cpus": res.get("cpuLimit", "4") + "memory": mem_limit, + "cpus": cpu_limit }, "reservations": { - "memory": res.get("memoryReservation", "4Gi") + "memory": mem_res } } } } compose = { - "version": "3.8", "services": services } diff --git a/snarfnet-game-seeds/seeds/enshrouded.seed.yaml b/snarfnet-game-seeds/seeds/enshrouded.seed.yaml index b1efebc..08c8f45 100644 --- a/snarfnet-game-seeds/seeds/enshrouded.seed.yaml +++ b/snarfnet-game-seeds/seeds/enshrouded.seed.yaml @@ -28,7 +28,7 @@ spec: memoryLimit: "16Gi" memoryReservation: "8Gi" runtime: - image: "skarab42/enshrouded-server:latest" + image: "indifferentbroccoli/enshrouded-server-docker:latest" restartPolicy: unless-stopped environment: SERVER_NAME: "SnarfNet Enshrouded Sanctuary" diff --git a/snarfnet-game-seeds/seeds/farming-simulator-25.seed.yaml b/snarfnet-game-seeds/seeds/farming-simulator-25.seed.yaml index ab9437f..d1e9eee 100644 --- a/snarfnet-game-seeds/seeds/farming-simulator-25.seed.yaml +++ b/snarfnet-game-seeds/seeds/farming-simulator-25.seed.yaml @@ -20,20 +20,33 @@ spec: ports: - port: 10823 protocol: TCP - description: "Game Web Management & Client Connection" + description: "Game Client Connection" - port: 10823 protocol: UDP description: "Game Network Traffic" + - port: 8088 + protocol: TCP + description: "Web Administration Panel" + - port: 5900 + protocol: TCP + description: "Wine VNC Desktop" resources: cpuLimit: "6" memoryLimit: "12Gi" memoryReservation: "6Gi" runtime: - image: "thijsvanloef/farming-simulator-25:latest" + image: "ghcr.io/nwerosama/farmsim-docker:fs25" restartPolicy: unless-stopped + volumes: + - "/mnt/storage-games/installers/fs25/iso:/opt/fs25/install:ro" + - "/opt/game-servers/fs25/dlc:/opt/fs25/dlc:ro" + - "/opt/game-servers/fs25/docs:/opt/fs25/docs" + - "/opt/game-servers/fs25/wine:/home/nobody/.fs_server" environment: SERVER_NAME: "SnarfNet FS25 Countryside" GAME_PORT: 10823 + AUTOSTART_ENABLED: "true" + WEB_PORT: 8088 backup: schedule: "0 4 * * *" retentionDays: 14 diff --git a/snarfnet-game-seeds/seeds/runescape-dragonwilds.seed.yaml b/snarfnet-game-seeds/seeds/runescape-dragonwilds.seed.yaml new file mode 100644 index 0000000..1da0f43 --- /dev/null +++ b/snarfnet-game-seeds/seeds/runescape-dragonwilds.seed.yaml @@ -0,0 +1,54 @@ +apiVersion: yggdrasil.snarfnet.net/v1alpha1 +kind: GameSeed +metadata: + name: runescape-dragonwilds + displayName: "RuneScape: Dragonwilds Dedicated Server" + version: "1.0.0" +spec: + game: + appId: 4019830 + branch: public + autoUpdate: true + server: + name: "SnarfNet Dragonwilds Sanctuary" + passwordSecret: "DRAGONWILDS_SERVER_PASSWORD" + adminPasswordSecret: "DRAGONWILDS_ADMIN_PASSWORD" + ownerIdSecret: "DRAGONWILDS_OWNER_ID" + maxPlayers: 6 + saveDirectory: "/opt/game-servers/runescape-dragonwilds/server-files/RSDragonwilds/Saved/SaveGames" + network: + ip: "140.44.44.80" + ports: + - port: 7777 + protocol: UDP + description: "Game Traffic Port" + - port: 8888 + protocol: UDP + description: "Beacon & Query Port" + resources: + cpuLimit: "6" + memoryLimit: "12Gi" + memoryReservation: "6Gi" + runtime: + image: "indifferentbroccoli/runescape-dragonwilds-server-docker:latest" + restartPolicy: unless-stopped + volumes: + - "/opt/game-servers/runescape-dragonwilds/server-files:/home/steam/server-files" + environment: + PUID: 1000 + PGID: 1000 + SERVER_NAME: "SnarfNet Dragonwilds Sanctuary" + DEFAULT_WORLD_NAME: "SnarfWilds" + DEFAULT_PORT: 7777 + BEACON_PORT: 8888 + MAX_PLAYERS: 6 + PLATFORM_POLICY: "Crossplay" + UPDATE_ON_START: "true" + OWNER_ID: "${DRAGONWILDS_OWNER_ID}" + ADMIN_PASSWORD: "${DRAGONWILDS_ADMIN_PASSWORD}" + WORLD_PASSWORD: "${DRAGONWILDS_SERVER_PASSWORD}" + backup: + schedule: "0 4 * * *" # Daily 04:00 AM + retentionDays: 14 + targetPath: "/mnt/nas/game-backups/runescape-dragonwilds" + diff --git a/snarfnet-game-seeds/seeds/stationeers.seed.yaml b/snarfnet-game-seeds/seeds/stationeers.seed.yaml index b0dcda6..8933b0b 100644 --- a/snarfnet-game-seeds/seeds/stationeers.seed.yaml +++ b/snarfnet-game-seeds/seeds/stationeers.seed.yaml @@ -29,7 +29,7 @@ spec: memoryLimit: "8Gi" memoryReservation: "4Gi" runtime: - image: "sprits/stationeers-server:latest" + image: "didstopia/stationeers-server:latest" restartPolicy: unless-stopped environment: SERVER_NAME: "SnarfNet Stationeers Moonbase"