Compare commits

..

12 Commits

534 changed files with 18922 additions and 236 deletions

View File

@@ -1,3 +0,0 @@
# SnarfCode
# I added this line to test the syncing

View File

@@ -1,27 +0,0 @@
Here is a concise, technical summary of your system architecture, the steps completed, and the current roadblock. You can copy and paste this text directly into another AI session to resume troubleshooting without losing context.
------------------------------
## System Architecture & Context
* Backend Server (Pi 5, 4GB RAM): Hosts the official Raspberry Pi AI HAT+ 2 (Hailo-10H NPU, 40 TOPS, 8GB onboard LPDDR4X VRAM). It is running a bare-metal, native C++ compiled framework (oatpp/1.4.0) provided by the Hailo GenAI Model Zoo. It currently has 5 hardware-quantized models loaded into its local store (including llama3.2:3b and deepseek_r1_distill_qwen:1.5b) listening on local IP 140.44.4.7:8000.
* Frontend Interface (Pi 4): Running Debian 13 (Trixie). To maximize performance and keep layers of abstraction at absolute zero, it hosts a bare-metal Python 3.11 environment running Open WebUI natively via pip (no Docker/containers).
* The Bridge: Because the Pi 5's oatpp C++ server doesn't match standard OpenAI/Ollama API schemas out-of-the-box, Open WebUI is connected to a local instance of Open WebUI Pipelines (uvicorn running on port 9099 with a security passkey string hailo-bridge).
------------------------------
## Work Completed So Far
1. Pi 4 Python Environment Fixes: Successfully compiled Python 3.11 from source on Debian Trixie. Handled and patched multiple modern Python deprecation roadblocks (injecting runtime compatibility hooks for collections.Mapping, collections.MutableMapping, and collections.Callable directly into Open WebUI and the Pipelines initialization scripts).
2. Database Cleaning: Cleaned out stuck background network strings in Open WebUI’s native SQLite database file (/root/open-webui/venv/lib/python3.11/site-packages/open_webui/data/webui.db) to force it to drop hardcoded configurations and route connections entirely through the Pipeline bridge.
3. Pipeline Live Status: Open WebUI is successfully communicating with the Pipelines bridge. Running a local curl call (curl -H "Authorization: Bearer hailo-bridge" http://localhost:9099/v1/models) returns a clean 200 OK JSON block, showing that Hailo NPU Core is fully mapped and available in the browser dropdown interface.
4. Network Continuity: The Pi 4 successfully hits the Pi 5 hardware engine over the network. A GET call to http://140.44.4 safely returns a 200 OK, verifying that physical PCIe communication paths are clear.
------------------------------
## Remaining Issues & Current Roadblock
The exact API endpoint paths and JSON payload structures for Hailo's native C++ oatpp server are highly proprietary and poorly documented. Because of this, our custom python pipeline bridge (hailo_pipeline.py) is running into API request mismatches when passing chats over the network.
* When the script targets http://140.44.4 or /chat/completions, the oatpp server returns a 404 Not Found.
* When the script targets http://140.44.4, the connection goes through, but the script throws a python KeyError: 'choices' or KeyError: 'message'. This indicates that the server processed the tensor inference successfully but returned its JSON text wrapper under an unknown key name (e.g., "response", "text", or "generated_text").
* Attempts to manually probe the JSON output schema using a direct POST terminal command (curl -X POST http://140.44.4.7:8000 -d '{"model":"llama3.2:3b","messages":[...],"stream":false}') returned a 404 No mapping for HTTP-method: 'POST', URL: '/'.
## What the Next LLM Needs to Solve:
We need the exact, structurally correct POST endpoint path and the corresponding JSON response key architecture for the Hailo GenAI Model Zoo C++ (oatpp) server so we can update the pipe() function inside hailo_pipeline.py to extract and stream the generated response strings back to Open WebUI.

View File

@@ -1,107 +0,0 @@
# ArgoCD Ingress Fix - Traefik Bad Gateway
## Environment
- **Cluster**: RKE2 managed by Rancher
- **Ingress Controller**: Traefik (kube-system namespace)
- **ArgoCD Version**: v3.4.2 (Helm chart argo-cd-9.5.14)
- **Namespace**: infrastructure
- **Hostname**: argo.snarfnet.net
## Problem
After deploying ArgoCD, accessing `https://argo.snarfnet.net` returned a **502 Bad Gateway** from Traefik.
## Root Cause
Two issues were identified:
### 1. Service TargetPort Mismatch
The ArgoCD server was listening on port **8080**, but the Kubernetes service had `targetPort: 8081`. This was corrected by patching the service to point both ports (80 and 443) to targetPort 8080.
### 2. Traefik Protocol Mismatch (Primary Issue)
The ArgoCD service defined two ports:
```yaml
ports:
- name: http
port: 80
targetPort: 8080
- name: https
port: 443
targetPort: 8080
```
The Ingress resource routed traffic to port 80, but Traefik's Kubernetes provider saw the port named `https` (443) on the service and automatically selected it, connecting to the backend using **HTTPS**:
```
"servers":[{"url":"https://10.42.1.76:8080"}]
```
However, ArgoCD was configured to run in insecure mode (`server.insecure: true`), meaning it only served plain **HTTP** on port 8080. Traefik's HTTPS connection to an HTTP backend resulted in the Bad Gateway.
Working services (Gitea, Jenkins, etc.) did not have this problem because they only exposed a single HTTP port with no `https` named port to confuse Traefik.
## Fix
Removed the `https` (port 443) entry from the `argocd-server` service, leaving only the HTTP port:
```yaml
spec:
ports:
- name: http
port: 80
targetPort: 8080
```
This forced Traefik to use `http://` when connecting to the backend, which matched ArgoCD's insecure mode.
After the change, Traefik's internal service config showed:
```
"servers":[{"url":"http://10.42.1.76:8080"}]
```
## Permanent Fix for Helm Upgrades
To prevent the Helm chart from recreating the 443 port on future upgrades, use one of these approaches:
### Option A: Annotate the Ingress
Add this annotation to the `argo-ing` Ingress resource so Traefik always uses HTTP regardless of service port names:
```yaml
metadata:
annotations:
traefik.ingress.kubernetes.io/service.serversscheme: http
```
### Option B: Helm Values
Configure the chart to not expose the HTTPS service port (check chart documentation for exact key, as it varies by version):
```yaml
configs:
params:
server.insecure: true
server:
service:
type: ClusterIP
```
## Debugging Steps That Led to the Fix
1. Verified the pod was running and healthy (`1/1 Ready`)
2. Confirmed the pod was listening on port 8080 via `/proc/net/tcp6`
3. Tested direct pod connectivity from another pod in the cluster — returned HTTP 200
4. Queried Traefik's internal API at `http://127.0.0.1:9000/api/http/services`
5. Discovered Traefik was using `https://` to connect to the backend
6. Compared with working services (Gitea, Jenkins) which all used `http://`
7. Identified the `https` named port on the service as the cause
## Key Takeaway
Traefik's Kubernetes Ingress provider infers the backend protocol from the service port name. A port named `https` causes Traefik to connect using HTTPS, regardless of what port number the Ingress backend specifies. When running ArgoCD in insecure mode behind a TLS-terminating reverse proxy, ensure the service does not expose an `https` named port, or use the `traefik.ingress.kubernetes.io/service.serversscheme` annotation to override the behavior.

View File

@@ -1,20 +1,33 @@
version: "3.8"
services:
heimdall:
image: lscr.io/linuxserver/heimdall:latest
ports:
- "85:80"
- "403:443"
- target: 80
published: 85
mode: ingress
- target: 443
published: 403
mode: ingress
environment:
- PUID=1000
- PGID=1000
- TZ=America/New_York
volumes:
- heimdall_config:/config
- type: bind
source: /var/lib/docker-data/heimdall/config
target: /config
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.hostname == snshpd4lnxswm01
restart_policy:
condition: on-failure
volumes:
heimdall_config:
driver: local
driver_opts:
type: nfs
o: "addr=14.10.10.71,rw,nfsvers=4"
device: ":/volume1/docker/heimdall/"
networks:
default:
name: homelab_public
external: true

View File

@@ -0,0 +1,21 @@
version: "3.8"
services:
shepherd:
image: containrrr/shepherd:latest
environment:
- SLEEP_TIME=24h
- IGNORELIST_SERVICES=nginx-proxy-manager_app nginx-proxy-manager_backup
- UPDATE_OPTIONS=--with-registry-auth
- ROLLBACK_FAILURE=true
- TZ=America/New_York
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.role == manager
restart_policy:
condition: on-failure

View File

@@ -180,3 +180,5 @@ We will define firewall groups on the UDM Pro:
* Secondary: `140.44.4.65`
* Set UDM Pro WAN DNS to `140.44.4.35` & `140.44.4.65` (or Quad9 fallback).
* Apply LAN IN firewall rule matrix on UDM Pro to enforce strict VLAN isolation.
snaisvr44hrvstr0x - AppropriateBeav3r39!

File diff suppressed because it is too large Load Diff

Binary file not shown.

After

Width:  |  Height:  |  Size: 830 KiB

Submodule homelab-gitops-repo updated: 2bd66045c3...50e49a9264

View File

@@ -0,0 +1,22 @@
import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('140.44.4.71', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
print(f">>> {cmd}")
_, o, e = ssh.exec_command(cmd)
out = o.read().decode().strip()
err = e.read().decode().strip()
if out:
print(out)
if err:
print("[stderr]", err)
run("iptables -I FORWARD 1 -m state --state RELATED,ESTABLISHED -j ACCEPT")
run("iptables -I FORWARD 2 -p tcp -d 140.44.4.64 --dport 10051 -j ACCEPT")
run("iptables -L FORWARD -n -v | head -n 8")
ssh.close()

View File

@@ -0,0 +1,58 @@
import yaml
def str_presenter(dumper, data):
if '\n' in data:
return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
return dumper.represent_scalar('tag:yaml.org,2002:str', data)
yaml.add_representer(str, str_presenter)
yaml.representer.SafeRepresenter.add_representer(str, str_presenter)
with open('homelab-gitops-repo/manifests/spaceport/homepage/homepage.yaml', 'r', encoding='utf-8') as f:
text = f.read()
docs = list(yaml.safe_load_all(text))
cm = docs[2]
bookmarks_yaml = """- Quick Links:
- GitHub:
- abbr: GH
href: https://github.com/
- UniFi Site Manager:
- abbr: UI
href: https://unifi.ui.com/
- Cloudflare:
- abbr: CF
href: https://dash.cloudflare.com/
- Grafana Telemetry:
- Systems Performance:
- abbr: SYS
href: https://grafana.snarfnet.net/d/snarfnet-systems/52f6861
- Container Workloads:
- abbr: K8S
href: https://grafana.snarfnet.net/d/snarfnet-containers/f09f93a6-kubernetes-containers-and-workloads
- AI Stack Intelligence:
- abbr: AI
href: https://grafana.snarfnet.net/d/snarfnet-ai-stack/snarfnet-ai-stack-and-inference-intelligence
- Uptime Status Pages:
- Unified Overview:
- abbr: ALL
href: https://uptime.snarfnet.net/status/default
- Core & VIP Fabric:
- abbr: VIP
href: https://uptime.snarfnet.net/status/core
- GitOps & CI/CD:
- abbr: OPS
href: https://uptime.snarfnet.net/status/gitops
- Edge & Ingress:
- abbr: EDG
href: https://uptime.snarfnet.net/status/edge
"""
cm['data']['bookmarks.yaml'] = bookmarks_yaml
with open('homelab-gitops-repo/manifests/spaceport/homepage/homepage.yaml', 'w', encoding='utf-8') as f:
yaml.dump_all(docs, f, sort_keys=False, default_flow_style=False, allow_unicode=True)
print("Updated bookmarks in homepage.yaml successfully!")

View File

@@ -0,0 +1,62 @@
import paramiko
import sys
sys.stdout.reconfigure(encoding='utf-8')
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run_sql(pod, sql):
cmd = f"kubectl exec -i -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root"
stdin, stdout, stderr = c.exec_command(cmd)
stdin.write(sql)
stdin.close()
out = stdout.read().decode().strip()
err = stderr.read().decode().strip()
return out, err
_, o, _ = c.exec_command("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'")
pod = o.read().decode().strip()
print("Kuma Pod:", pod)
monitors = [
("Spacedock Harvester Node 01 (140.44.4.81)", "140.44.4.81"),
("Spacedock Harvester Node 02 (140.44.4.82)", "140.44.4.82"),
("Spacedock Harvester Node 03 (140.44.4.83)", "140.44.4.83"),
("Spacedock Harvester Node 04 (140.44.4.84)", "140.44.4.84"),
("Dell PowerEdge R730 Node 1 iDRAC (140.44.44.121)", "140.44.44.121"),
("Dell PowerEdge R730 Node 2 iDRAC (140.44.44.122)", "140.44.44.122"),
("Dell PowerEdge R730 Node 3 iDRAC (140.44.44.123)", "140.44.44.123"),
("Dell PowerEdge R730 Node 4 iDRAC (140.44.44.124)", "140.44.44.124")
]
for name, host in monitors:
out, err = run_sql(pod, f"USE kuma; SELECT id FROM monitor WHERE hostname='{host}' AND type='ping';")
if "id" in out and len(out.splitlines()) > 1:
mon_id = out.splitlines()[1].strip()
print(f"Monitor already exists: {name} (ID: {mon_id})")
else:
insert_sql = f"""USE kuma;
INSERT INTO monitor (name, active, user_id, `interval`, type, weight, hostname, maxretries, retry_interval)
VALUES ('{name}', 1, 1, 60, 'ping', 2000, '{host}', 1, 60);
SELECT LAST_INSERT_ID() AS id;
"""
out, err = run_sql(pod, insert_sql)
if err:
print(f"Error inserting {name}: {err}")
continue
lines = out.splitlines()
mon_id = lines[-1].strip()
print(f"Created monitor: {name} (ID: {mon_id})")
# Link notifications (1 = Discord, 2 = Gotify)
for notif_id in [1, 2]:
link_check = f"USE kuma; SELECT id FROM monitor_notification WHERE monitor_id={mon_id} AND notification_id={notif_id};"
out_link, _ = run_sql(pod, link_check)
if "id" not in out_link or len(out_link.splitlines()) <= 1:
run_sql(pod, f"USE kuma; INSERT INTO monitor_notification (monitor_id, notification_id) VALUES ({mon_id}, {notif_id});")
print(f" Linked monitor {mon_id} to notification {notif_id}")
else:
print(f" Monitor {mon_id} already linked to notification {notif_id}")
c.close()

View File

@@ -0,0 +1,32 @@
import json, ssl, urllib.request
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
ZABBIX_URL = 'http://140.44.4.60:8888/api_jsonrpc.php'
payload = {'jsonrpc': '2.0', 'method': 'user.login', 'params': {'username': 'Admin', 'password': 'zabbix'}, 'id': 1}
req = urllib.request.Request(ZABBIX_URL, data=json.dumps(payload).encode(), headers={'Content-Type': 'application/json-rpc'})
token = json.loads(urllib.request.urlopen(req, context=ctx).read())['result']
media_payload = {
'jsonrpc': '2.0',
'method': 'user.update',
'params': {
'userid': '3',
'medias': [
{
'mediatypeid': '70',
'sendto': 'snarf@snarfnet.net',
'active': 0,
'severity': 63, # 1+2+4+8+16+32
'period': '1-7,00:00-24:00'
}
]
},
'id': 2
}
req = urllib.request.Request(ZABBIX_URL, data=json.dumps(media_payload).encode(), headers={'Content-Type': 'application/json-rpc', 'Authorization': f'Bearer {token}'})
res = json.loads(urllib.request.urlopen(req, context=ctx).read())
print("User update result:", res)

173
scripts/add_npm_games.py Normal file
View File

@@ -0,0 +1,173 @@
import paramiko
import sys
import datetime
import json
sys.stdout.reconfigure(encoding='utf-8')
key_path = r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet'
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.71", username="root", key_filename=key_path, timeout=10)
def main():
remote_script = """import sqlite3, datetime, json
db_path = '/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite'
conn = sqlite3.connect(db_path)
cur = conn.cursor()
# Check existing games host
cur.execute("SELECT id, domain_names, forward_host, forward_port FROM proxy_host WHERE domain_names LIKE '%games.snarfnet.net%'")
existing = cur.fetchone()
now = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
if existing:
pid = existing[0]
print(f"Updating existing proxy_host ID {pid}...")
cur.execute('''UPDATE proxy_host
SET forward_host = '140.44.44.80', forward_port = 8080, forward_scheme = 'http',
certificate_id = 1, ssl_forced = 1, hsts_enabled = 1, http2_support = 1,
block_exploits = 1, allow_websocket_upgrade = 1, enabled = 1, modified_on = ?
WHERE id = ?''', (now, pid))
else:
cur.execute("SELECT MAX(id) FROM proxy_host")
max_id = cur.fetchone()[0] or 0
pid = max_id + 1
print(f"Creating new proxy_host with ID {pid}...")
cur.execute('''INSERT INTO proxy_host (
id, created_on, modified_on, owner_user_id, domain_names,
forward_host, forward_port, forward_scheme, certificate_id,
ssl_forced, hsts_enabled, hsts_subdomains, http2_support,
block_exploits, caching_enabled, allow_websocket_upgrade,
access_list_id, advanced_config, enabled, meta
) VALUES (
?, ?, ?, 1, '["games.snarfnet.net"]',
'140.44.44.80', 8080, 'http', 1,
1, 1, 0, 1,
1, 0, 1,
0, '', 1, '{}'
)''', (pid, now, now))
conn.commit()
conn.close()
print(f"SUCCESS: pid={pid}")
"""
sftp = c.open_sftp()
with sftp.open("/tmp/add_games_proxy.py", "w") as f:
f.write(remote_script)
sftp.close()
stdin, stdout, stderr = c.exec_command("python3 /tmp/add_games_proxy.py")
out = stdout.read().decode().strip()
err = stderr.read().decode().strip()
print("DB Result:\n", out)
if err:
print("DB Err:\n", err)
# Extract pid
pid = None
for line in out.splitlines():
if "pid=" in line:
pid = line.split("pid=")[1].strip()
if not pid:
print("Failed to get pid!")
return
# Write nginx config
nginx_conf = f"""# ------------------------------------------------------------
# games.snarfnet.net
# ------------------------------------------------------------
map $scheme $hsts_header {{
https "max-age=63072000; preload";
}}
server {{
set $forward_scheme http;
set $server "140.44.44.80";
set $port 8080;
listen 80;
listen [::]:80;
listen 443 ssl;
listen [::]:443 ssl;
server_name games.snarfnet.net;
http2 on;
# Let's Encrypt SSL (*.snarfnet.net is cert 1)
include conf.d/include/letsencrypt-acme-challenge.conf;
include conf.d/include/ssl-cache.conf;
include conf.d/include/ssl-ciphers.conf;
ssl_certificate /etc/letsencrypt/live/npm-1/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/npm-1/privkey.pem;
# Asset Caching
include conf.d/include/assets.conf;
# Block Exploits
include conf.d/include/block-exploits.conf;
# Force SSL
set $trust_forwarded_proto "F";
include conf.d/include/force-ssl.conf;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_http_version 1.1;
access_log /data/logs/proxy-host-{pid}_access.log proxy;
error_log /data/logs/proxy-host-{pid}_error.log warn;
location / {{
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_http_version 1.1;
# Proxy!
include conf.d/include/proxy.conf;
}}
# Custom
include /data/nginx/custom/server_proxy[.]conf;
}}
"""
conf_path = f"/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/nginx/proxy_host/{pid}.conf"
sftp = c.open_sftp()
with sftp.open(conf_path, "w") as f:
f.write(nginx_conf)
sftp.close()
print(f"Wrote {conf_path}")
# Test Nginx
stdin, stdout, stderr = c.exec_command("docker exec $(docker ps -q -f name=nginx-proxy-manager_app | head -n1) nginx -t")
t_out = stdout.read().decode()
t_err = stderr.read().decode()
print("nginx -t stdout:\n", t_out)
print("nginx -t stderr:\n", t_err)
if "successful" in t_out or "successful" in t_err:
print("Reloading Nginx in container...")
c.exec_command("docker exec $(docker ps -q -f name=nginx-proxy-manager_app | head -n1) nginx -s reload")
# Also check other swarm node if running dual
c2 = paramiko.SSHClient()
c2.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
c2.connect("140.44.4.72", username="root", key_filename=key_path, timeout=5)
c2.exec_command("docker exec $(docker ps -q -f name=nginx-proxy-manager_app | head -n1) nginx -s reload")
c2.close()
print("Reloaded Swarm 02 NPM as well!")
except Exception as e:
print("Swarm 02 reload skipped/note:", e)
print("Proxy setup complete!")
else:
print("ERROR: nginx -t failed!")
c.close()
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,26 @@
import yaml
def str_presenter(dumper, data):
if '\n' in data:
return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
return dumper.represent_scalar('tag:yaml.org,2002:str', data)
yaml.add_representer(str, str_presenter)
yaml.representer.SafeRepresenter.add_representer(str, str_presenter)
with open('homelab-gitops-repo/manifests/spaceport/homepage/homepage.yaml', 'r', encoding='utf-8') as f:
text = f.read()
docs = list(yaml.safe_load_all(text))
cm = docs[2]
services = yaml.safe_load(cm['data']['services.yaml'])
bookmarks = yaml.safe_load(cm['data']['bookmarks.yaml'])
cm['data']['services.yaml'] = yaml.dump(services, sort_keys=False, default_flow_style=False, allow_unicode=True)
cm['data']['bookmarks.yaml'] = yaml.dump(bookmarks, sort_keys=False, default_flow_style=False, allow_unicode=True)
with open('homelab-gitops-repo/manifests/spaceport/homepage/homepage.yaml', 'w', encoding='utf-8') as f:
yaml.dump_all(docs, f, sort_keys=False, default_flow_style=False, allow_unicode=True)
print("Cleaned up unicode strings in homepage.yaml!")

View File

@@ -0,0 +1,118 @@
import json
import ssl
import urllib.request
ZABBIX_URL = "http://140.44.4.60:8888/api_jsonrpc.php"
ZABBIX_USER = "Admin"
ZABBIX_PASS = "zabbix"
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
def zbx_call(method, params, auth=None):
payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1}
headers = {"Content-Type": "application/json-rpc"}
if auth:
headers["Authorization"] = f"Bearer {auth}"
req = urllib.request.Request(ZABBIX_URL, data=json.dumps(payload).encode("utf-8"), headers=headers, method="POST")
with urllib.request.urlopen(req, timeout=10, context=ctx) as r:
resp = json.loads(r.read().decode("utf-8"))
if "error" in resp:
raise RuntimeError(f"Zabbix error in {method}: {resp['error']}")
return resp["result"]
auth = zbx_call("user.login", {"username": ZABBIX_USER, "password": ZABBIX_PASS})
# Check existing host names
existing_hosts = zbx_call("host.get", {"output": ["hostid", "host", "name"]}, auth)
existing_names = {h["host"].lower(): h["hostid"] for h in existing_hosts}
# Create host groups if not exist
groups = zbx_call("hostgroup.get", {"output": ["groupid", "name"]}, auth)
group_map = {g["name"]: g["groupid"] for g in groups}
for gname in ["Network Devices", "Environmental & Sensors", "Management & IPMI"]:
if gname not in group_map:
res = zbx_call("hostgroup.create", {"name": gname}, auth)
group_map[gname] = res["groupids"][0]
print(f"Created Host Group: {gname} (ID: {group_map[gname]})")
devices_to_add = [
{
"host": "idrac-spacedock4",
"name": "Dell PowerEdge R730 Node 4 iDRAC (idrac-spacedock4)",
"ip": "140.44.44.124",
"group": "Management & IPMI"
},
{
"host": "usw-agg-core",
"name": "UniFi Switch Aggregation 10G Core",
"ip": "14.44.4.26",
"group": "Network Devices"
},
{
"host": "usw-24-poe-closet",
"name": "UniFi Switch 24 PoE Closet",
"ip": "14.44.4.61",
"group": "Network Devices"
},
{
"host": "usw-pro-24-poe-compute",
"name": "UniFi Switch Pro 24 PoE Compute",
"ip": "14.44.4.107",
"group": "Network Devices"
},
{
"host": "usw-flex-attic",
"name": "UniFi Switch Flex Attic",
"ip": "14.44.4.98",
"group": "Network Devices"
},
{
"host": "usw-flex-garage",
"name": "UniFi Switch Flex Garage",
"ip": "14.44.4.96",
"group": "Network Devices"
},
{
"host": "tempest-weather-station",
"name": "Tempest Weather Station Gateway",
"ip": "14.4.4.47",
"group": "Environmental & Sensors"
},
{
"host": "sense-energy-monitor",
"name": "Sense Home Energy Gateway",
"ip": "14.4.4.48",
"group": "Environmental & Sensors"
}
]
for dev in devices_to_add:
hname = dev["host"]
if hname.lower() in existing_names:
print(f"[EXISTS] {hname}")
continue
params = {
"host": dev["host"],
"name": dev["name"],
"interfaces": [
{
"type": 1, # Agent / ICMP
"main": 1,
"useip": 1,
"ip": dev["ip"],
"dns": "",
"port": "10050"
}
],
"groups": [{"groupid": group_map[dev["group"]]}],
"templates": [{"templateid": "10564"}] # ICMP Ping
}
try:
res = zbx_call("host.create", params, auth)
print(f"[CREATED] {dev['name']} ({dev['ip']}) -> HostID {res['hostids'][0]}")
except Exception as e:
print(f"[FAILED] {dev['name']}: {e}")

View File

@@ -0,0 +1,67 @@
import urllib.request
import json
import ssl
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
def zabbix_api(method, params, auth=None):
payload = {"jsonrpc": "2.0", "method": method, "params": params, "id": 1}
if auth: payload["auth"] = auth
req = urllib.request.Request("https://zabbix.snarfnet.net/api_jsonrpc.php",
data=json.dumps(payload).encode('utf-8'),
headers={"Content-Type": "application/json-rpc"})
resp = urllib.request.urlopen(req, context=ctx, timeout=10)
return json.loads(resp.read().decode('utf-8'))
auth = zabbix_api("user.login", {"username": "Admin", "password": "zabbix"}).get("result")
print("Zabbix Auth OK:", bool(auth))
# 1. Update Template Macros for 10001 (Agent passive), 10343 (Agent active), 10285 (Prom)
# Find hostmacroids for these templates
template_ids = ["10001", "10343", "10285"]
for tid in template_ids:
macros = zabbix_api("usermacro.get", {
"output": ["hostmacroid", "macro", "value"],
"hostids": [tid]
}, auth=auth).get("result", [])
for m in macros:
mid = m["hostmacroid"]
macro_name = m["macro"]
if macro_name == "{$CPU.UTIL.CRIT}":
res = zabbix_api("usermacro.update", {"hostmacroid": mid, "value": "95"}, auth=auth)
print(f"Template {tid} macro {macro_name} updated to 95: {res.get('result')}")
elif macro_name == "{$MEMORY.UTIL.MAX}":
res = zabbix_api("usermacro.update", {"hostmacroid": mid, "value": "95"}, auth=auth)
print(f"Template {tid} macro {macro_name} updated to 95: {res.get('result')}")
elif macro_name == "{$LOAD_AVG_PER_CPU.MAX.WARN}":
res = zabbix_api("usermacro.update", {"hostmacroid": mid, "value": "2.5"}, auth=auth)
print(f"Template {tid} macro {macro_name} updated to 2.5: {res.get('result')}")
# 2. Update trigger expressions from 5m to 15m
# Triggers: 22370 (Passive CPU), 22376 (Passive Mem), 22415 (Active CPU), 22421 (Active Mem)
# In Zabbix 6.0/7.0, expression format is min(/<template_name>/<item_key>,15m)>{$MACRO}
# Let's inspect the exact expression syntax for trigger update
for tr_id in ["22370", "22376", "22415", "22421"]:
tr = zabbix_api("trigger.get", {
"output": ["triggerid", "description", "expression"],
"triggerids": [tr_id]
}, auth=auth).get("result", [{}])[0]
expr = tr.get("expression", "")
print(f"\nTrigger {tr_id} ({tr.get('description')}): current expr = {expr}")
# Replace 5m with 15m in the expression
if "5m" in expr:
new_expr = expr.replace("5m", "15m")
# In trigger.update, Zabbix expects expression with host/item syntax if updated
# Or description update: "Linux: High CPU utilization (>95% for 15m)"
desc = tr.get("description", "")
# Update trigger
up_res = zabbix_api("trigger.update", {
"triggerid": tr_id,
"expression": new_expr
}, auth=auth)
print(f" Updated expr to 15m: {up_res}")
print("\nFinished updating Zabbix noise reduction thresholds!")

View File

@@ -0,0 +1,51 @@
import paramiko
import time
targets = [
{
"name": "Node 1 (140.44.44.121)",
"ip": "140.44.44.121",
"settings": [
("bios.IntegratedDevices.SriovGlobalEnable", "Enabled")
]
},
{
"name": "Node 4 (140.44.44.124)",
"ip": "140.44.44.124",
"settings": [
("bios.SysProfileSettings.SysProfile", "PerfOptimized")
]
}
]
for t in targets:
print(f"\n==========================================")
print(f"Applying BIOS settings to {t['name']}...")
try:
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect(t['ip'], username='root', password='Katzir476!', timeout=15)
for key, val in t['settings']:
cmd = f"racadm set {key} {val}"
stdin, stdout, stderr = c.exec_command(cmd)
out = stdout.read().decode().strip()
print(f" racadm set {key} {val}:\n {out}")
time.sleep(1)
# Create BIOS job
stdin, stdout, stderr = c.exec_command("racadm jobqueue create BIOS.Setup.1-1")
job_out = stdout.read().decode().strip()
print(f" racadm jobqueue create BIOS.Setup.1-1:\n {job_out}")
# View job queue
stdin, stdout, stderr = c.exec_command("racadm jobqueue view")
q_out = stdout.read().decode().strip()
print(f" Job Queue:\n {q_out[:200]}")
c.close()
except Exception as e:
print(f" Error on {t['name']}: {e}")
print("\nDone applying BIOS settings!")

37
scripts/apply_clean_cm.py Normal file
View File

@@ -0,0 +1,37 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
with open(r'c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\aerodrome\prometheus\prometheus.yaml', 'r', encoding='utf-8') as f:
text = f.read()
# Extract just the ConfigMap section
# It starts with "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: prometheus-config" and ends with "---"
parts = text.split("---")
cm_part = None
for p in parts:
if "name: prometheus-config" in p:
cm_part = p
break
if cm_part:
print("Found prometheus-config ConfigMap! Size:", len(cm_part))
stdin, stdout, stderr = c.exec_command("cat > /tmp/prom_cm_clean.yaml")
stdin.write(cm_part)
stdin.close()
stdout.channel.recv_exit_status()
_, o, e = c.exec_command("kubectl apply -f /tmp/prom_cm_clean.yaml")
print("Apply result:\n", o.read().decode().strip())
err = e.read().decode().strip()
if err:
print("Apply err:\n", err)
# Reload prometheus via lifecycle endpoint
_, o, _ = c.exec_command("kubectl exec deploy/prometheus -- wget -qO- --post-action='' http://localhost:9090/-/reload")
print("Reload endpoint output:", o.read().decode().strip())
c.close()

View File

@@ -0,0 +1,250 @@
import yaml
def str_presenter(dumper, data):
if '\n' in data:
return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')
return dumper.represent_scalar('tag:yaml.org,2002:str', data)
yaml.add_representer(str, str_presenter)
yaml.representer.SafeRepresenter.add_representer(str, str_presenter)
with open('homelab-gitops-repo/manifests/spaceport/homepage/homepage.yaml', 'r', encoding='utf-8') as f:
text = f.read()
docs = list(yaml.safe_load_all(text))
cm = docs[2]
services_yaml = """- Critical Services:
- Authentik:
icon: authentik.png
href: https://authentik.snarfnet.net
description: SSO & Identity Management
- PeaNUT UPS:
icon: peanut.png
href: https://portal.snarfnet.net
description: UPS Power & Battery Monitor
- Unifi Controller:
icon: unifi.png
href: https://unifi.ui.com
widget:
type: unifi
url: https://140.44.44.1
username: Snarf
password: Katzir476!!!
- GitOps & Automation:
- Gitea:
icon: gitea.png
href: https://git.snarfnet.net
description: Git Repositories
widget:
type: gitea
url: https://git.snarfnet.net
key: f19e772cf54e40ceb2befe452e2bc24dd81fdb71
- ArgoCD:
icon: argocd.png
href: https://argo.snarfnet.net
description: Declarative GitOps Reconciler
widget:
type: argocd
url: https://argo.snarfnet.net
key: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJhcmdvY2QiLCJzdWIiOiJob21lcGFnZTphcGlLZXkiLCJuYmYiOjE3OTQ1MDIwNzgsImlhdCI6MTc5NDUwMjA3OCwianRpIjoiZjQ2NzU0NjMtODRhOS00MjI0LWI3ODItNmY5ODQxYWEwY2UxIn0.aHhrny2RBG-vAJAyXMvan0jw65jkP3bguqEduaG78O0
- AWX Automation:
icon: awx.png
href: https://awx.snarfnet.net
description: Ansible Automation Controller
- Semaphore:
icon: semaphore.png
href: https://semaphore.snarfnet.net
description: Playbook Execution Engine
- Storage & Clusters:
- Longhorn Aerodrome:
icon: longhorn.png
href: https://longhorn.aerodrome.snarfnet.net
description: Aerodrome Distributed Flash Storage
- Longhorn Spaceport:
icon: longhorn.png
href: https://longhorn.spaceport.snarfnet.net
description: Spaceport Distributed Block Storage
- Rancher (Aerodrome):
icon: rancher.png
href: https://aerodrome.snarfnet.net
description: Kubernetes Management Console
widget:
type: customapi
url: https://aerodrome.snarfnet.net/v3/clusters/local
headers:
Authorization: "Bearer kubeconfig-u-aqzkufonck4hmbd:bw22b96nv8844k8w5r8lzpz8gnk22tjwj95j6nkzr7hd5dhm4bs6z7"
mappings:
- field: capacity.cpu
label: CPU Cores
- field: capacity.pods
label: Max Pods
- field: state
label: Cluster State
- Harvester HCI (Spacedock):
icon: harvester.png
href: https://harvester.spacedock.snarfnet.net
description: HCI Virtualization Hypervisors
widget:
type: customapi
url: https://aerodrome.snarfnet.net/v3/clusters/c-cclnt
headers:
Authorization: "Bearer kubeconfig-u-aqzkufonck4hmbd:bw22b96nv8844k8w5r8lzpz8gnk22tjwj95j6nkzr7hd5dhm4bs6z7"
mappings:
- field: capacity.cpu
label: CPU Cores
- field: capacity.pods
label: Max Pods
- field: state
label: HCI State
- Portainer Swarm:
icon: portainer.png
href: https://shipyard.snarfnet.net
description: Docker Swarm Container Fleet
widget:
type: portainer
url: https://140.44.4.71:9443
env: 3
key: ptr_TCVm8ujeB4zT0oRejyMCwKw5enaPG/uKj1KuF0uOWX8=
- Infrastructure & Observability:
- NGINX Proxy Manager:
icon: nginx-proxy-manager
href: https://nginx.snarfnet.net
description: Reverse Proxy & SSL Certificates
- Uptime Kuma:
icon: uptime-kuma
href: https://uptime.snarfnet.net
description: Multi-Endpoint Availability Probes
widget:
type: uptimekuma
url: http://140.44.4.28:448
slug: default
- Zabbix:
icon: zabbix.png
href: https://zabbix.snarfnet.net
description: Infrastructure Metrics & Alarms
- Grafana:
icon: grafana.png
href: https://grafana.snarfnet.net
description: Real-Time Metrics & AI Dashboards
widget:
type: grafana
url: https://grafana.snarfnet.net
username: admin
password: Katzir476!!!
- Core Node Glances:
icon: glances.png
href: http://140.44.4.11:61208
description: Host Resource Monitor
- Environment & Weather:
- Tempest Weather Station:
icon: weather.png
href: https://snarfhome.snarfnet.net
description: Backyard Weather Telemetry
widget:
type: homeassistant
url: http://14.4.4.20:8123
key: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiIzZWUyNTE1MDk0NGU0ZjRjYmJkMGM1MWI5MThmY2EzYiIsImlhdCI6MTc0NjgxMDM3NSwiZXhwIjoyMDYyMTcwMzc1fQ.Le6M7WL63nxtLTxccFacQR5GVUY5C-a_zGtmXT_uaz4
custom:
- template: "{{ states('sensor.st_00158555_temperature') | float(0) | round(1) }} °F"
label: Outdoor Temp
- template: "{{ states('sensor.st_00158555_feels_like') | float(0) | round(1) }} °F"
label: Feels Like
- template: "{{ states('sensor.st_00158555_humidity') | float(0) | round(1) }} %"
label: Humidity
- template: "{{ states('sensor.st_00158555_wind_speed') | float(0) | round(1) }} mph"
label: Wind Speed
"""
settings_yaml = """title: SnarfNet Operations & Telemetry
headerStyle: boxed
color: slate
theme: dark
statusStyle: badge
background:
image: https://i.imgur.com/hOzzkCY.jpeg
blur: sm
opacity: 50
brightness: 50
cardBlur: sm
quicklaunch:
searchDescriptions: true
hideInternetSearch: true
hideVisitURL: true
hideVersion: true
layout:
Critical Services:
icon: mdi-shield-check
style: row
columns: 3
GitOps & Automation:
icon: mdi-git
style: row
columns: 4
Storage & Clusters:
icon: mdi-server-network
style: row
columns: 3
Infrastructure & Observability:
icon: mdi-chart-timeline-variant
style: row
columns: 3
Environment & Weather:
icon: mdi-weather-partly-cloudy
style: row
columns: 1
showStats: true
providers:
longhorn:
url: http://longhorn-backend.longhorn-system.svc.cluster.local:9500
"""
widgets_yaml = """- kubernetes:
cluster:
show: true
cpu: true
memory: true
show_names: true
nodes:
show: false
- longhorn:
total: true
nodes: false
- resources:
label: System
backend: resources
cpu: true
expanded: true
memory: true
disk: /
- search:
provider: duckduckgo
target: _blank
"""
bookmarks_yaml = """- Quick Links:
- GitHub:
- abbr: GH
href: https://github.com/
- UniFi Site Manager:
- abbr: UI
href: https://unifi.ui.com/
- Cloudflare:
- abbr: CF
href: https://dash.cloudflare.com/
"""
cm['data']['services.yaml'] = services_yaml
cm['data']['settings.yaml'] = settings_yaml
cm['data']['widgets.yaml'] = widgets_yaml
cm['data']['bookmarks.yaml'] = bookmarks_yaml
cm['data']['kubernetes.yaml'] = "mode: cluster\n"
with open('homelab-gitops-repo/manifests/spaceport/homepage/homepage.yaml', 'w', encoding='utf-8') as f:
yaml.dump_all(docs, f, sort_keys=False, default_flow_style=False, allow_unicode=True)
print("Saved clean homelab-gitops-repo/manifests/spaceport/homepage/homepage.yaml successfully!")

View File

@@ -0,0 +1,33 @@
import paramiko
import time
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
out = o.read().decode().strip()
err = e.read().decode().strip()
return out + ("\nERR: " + err if err else "")
with open("homelab-gitops-repo/manifests/aerodrome/grafana/grafana.yaml", "r", encoding="utf-8") as f:
grafana_yaml = f.read()
stdin, stdout, stderr = c.exec_command("cat > /tmp/grafana_updated.yaml")
stdin.write(grafana_yaml)
stdin.close()
stdout.channel.recv_exit_status()
print("Applying Grafana manifests:")
print(run("kubectl apply -f /tmp/grafana_updated.yaml"))
print("\nRestarting Grafana:")
print(run("kubectl rollout restart deployment/grafana -n default"))
print(run("kubectl rollout status deployment/grafana -n default --timeout=60s"))
print("\nWaiting for Grafana to finish initialization...")
time.sleep(10)
c.close()

View File

@@ -0,0 +1,20 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
out = o.read().decode().strip()
err = e.read().decode().strip()
return out + ("\nERR: " + err if err else "")
print("Applying server-side:")
print(run("kubectl apply -f /tmp/grafana_full.yaml --server-side --force-conflicts"))
print("\nRollout restart:")
print(run("kubectl rollout restart deployment/grafana -n default"))
print(run("kubectl rollout status deployment/grafana -n default --timeout=60s"))
c.close()

View File

@@ -0,0 +1,149 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.71", username="root", key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
adv_config = """# Buffer tuning for Authentik headers
proxy_buffers 8 16k;
proxy_buffer_size 32k;
# Outpost verification route - plain HTTP to internal VIP
location /outpost.goauthentik.io {
auth_request off;
proxy_pass http://140.44.4.29:9000/outpost.goauthentik.io;
proxy_set_header Host authentik.snarfnet.net;
proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
add_header Set-Cookie $auth_cookie;
auth_request_set $auth_cookie $upstream_http_set_cookie;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
}
# Redirect to Authentik when unauthenticated
location @goauthentik_proxy_signin {
internal;
auth_request off;
add_header Set-Cookie $auth_cookie;
return 302 https://authentik.snarfnet.net/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
}
# Protected Topology HTML (Interactive)
location = /topology.html {
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;
auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;
add_header Cache-Control "no-cache, no-store, must-revalidate";
expires 0;
alias /data/topology/snarfnet_homelab_topology.html;
default_type text/html;
}
# Protected Topology PNG (High-Res Image)
location = /topology.png {
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;
auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;
add_header Cache-Control "no-cache, no-store, must-revalidate";
expires 0;
alias /data/topology/snarfnet_homelab_topology.png;
default_type image/png;
}
location = /snarfnet_homelab_topology.html {
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;
auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;
add_header Cache-Control "no-cache, no-store, must-revalidate";
expires 0;
alias /data/topology/snarfnet_homelab_topology.html;
default_type text/html;
}
location = /snarfnet_homelab_topology.png {
auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;
auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;
add_header Cache-Control "no-cache, no-store, must-revalidate";
expires 0;
alias /data/topology/snarfnet_homelab_topology.png;
default_type image/png;
}
"""
stdin, stdout, stderr = c.exec_command("cat > /tmp/adv_config_h21.txt")
stdin.write(adv_config)
stdin.close()
stdout.read()
remote_script = """import sqlite3, shutil, os
db_path = '/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/database.sqlite'
conf_path = '/var/lib/docker/volumes/nginx-proxy-manager_nginx-data/_data/nginx/proxy_host/21.conf'
with open('/tmp/adv_config_h21.txt', 'r') as f:
adv = f.read()
# Update DB
conn = sqlite3.connect(db_path)
cur = conn.cursor()
cur.execute("UPDATE proxy_host SET advanced_config = ? WHERE id = 21", (adv,))
conn.commit()
conn.close()
print("Updated database for host 21.")
# Update 21.conf
with open(conf_path, 'r') as f:
conf = f.read()
# Strip existing adv_config block if present
tag = '# Buffer tuning for Authentik headers'
if tag in conf:
idx_start = conf.find(tag)
idx_end = conf.find('location / {', idx_start)
if idx_start != -1 and idx_end != -1:
new_conf = conf[:idx_start] + adv + "\\n\\n " + conf[idx_end:]
else:
new_conf = conf
else:
idx = conf.find('location / {')
if idx != -1:
new_conf = conf[:idx] + adv + "\\n\\n " + conf[idx:]
else:
new_conf = conf.rstrip().rstrip('}') + "\\n\\n" + adv + "\\n}\\n"
with open(conf_path, 'w') as f:
f.write(new_conf)
print("Updated 21.conf.")
"""
stdin, stdout, stderr = c.exec_command("cat > /tmp/apply_h21.py")
stdin.write(remote_script)
stdin.close()
stdout.read()
stdin, stdout, stderr = c.exec_command("python3 /tmp/apply_h21.py")
print("Remote python stdout:", stdout.read().decode())
print("Remote python stderr:", stderr.read().decode())
stdin, stdout, stderr = c.exec_command("docker exec $(docker ps -q --filter 'name=nginx-proxy-manager_app') nginx -t")
out = stdout.read().decode()
err = stderr.read().decode()
print("nginx -t stdout:", out)
print("nginx -t stderr:", err)
if "successful" in out or "successful" in err:
print("Nginx syntax OK! Reloading nginx...")
stdin, stdout, stderr = c.exec_command("docker exec $(docker ps -q --filter 'name=nginx-proxy-manager_app') nginx -s reload")
print("Reload stdout:", stdout.read().decode())
print("Reload stderr:", stderr.read().decode())
c.close()

31
scripts/apply_homepage.py Normal file
View File

@@ -0,0 +1,31 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
print(f"\n>>> {cmd}")
_, o, e = c.exec_command(cmd)
out = o.read().decode().strip()
err = e.read().decode().strip()
if out:
print(out)
if err:
print("[stderr]", err)
return out
with open('homelab-gitops-repo/manifests/spaceport/homepage/homepage.yaml', 'r', encoding='utf-8') as f:
content = f.read()
stdin, stdout, stderr = c.exec_command('cat > /tmp/homepage.yaml')
stdin.write(content)
stdin.close()
stdout.read()
run("ls -lh /tmp/homepage.yaml")
run("kubectl apply -f /tmp/homepage.yaml")
run("kubectl rollout restart deployment/homepage-2-1744068158 -n infrastructure")
run("kubectl rollout status deployment/homepage-2-1744068158 -n infrastructure --timeout=60s")
c.close()

View File

@@ -0,0 +1,79 @@
import paramiko
import sys
sys.stdout.reconfigure(encoding='utf-8')
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run_sql(sql):
pod_cmd = "kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'"
_, o, _ = c.exec_command(pod_cmd)
pod = o.read().decode().strip()
cmd = f"kubectl exec -i -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root"
stdin, stdout, stderr = c.exec_command(cmd)
stdin.write(sql)
stdin.close()
return stdout.read().decode().strip(), stderr.read().decode().strip()
# Target monitors:
# 98: Spacedock Harvester Node 01
# 99: Spacedock Harvester Node 02
# 100: Spacedock Harvester Node 03
# 101: Spacedock Harvester Node 04
# 102: Dell PowerEdge R730 Node 1 iDRAC
# 103: Dell PowerEdge R730 Node 2 iDRAC
# 104: Dell PowerEdge R730 Node 3 iDRAC
# 105: Dell PowerEdge R730 Node 4 iDRAC
# Tags to attach:
# 5: ai
# 9: IP
# 11: Tier-2-Operational
monitors = [98, 99, 100, 101, 102, 103, 104, 105]
tag_ids = [5, 9, 11]
sql_commands = ["USE kuma;"]
for m_id in monitors:
for t_id in tag_ids:
# Check if already exists
check = f"SELECT id FROM monitor_tag WHERE monitor_id={m_id} AND tag_id={t_id};"
sql_commands.append(f"""
INSERT INTO monitor_tag (monitor_id, tag_id, value)
SELECT {m_id}, {t_id}, NULL
WHERE NOT EXISTS (SELECT 1 FROM monitor_tag WHERE monitor_id={m_id} AND tag_id={t_id});
""")
# Also ensure Harvester Console (57) and Harvester HCI Console VIP (94) have ai (5)
for m_id in [57, 94]:
sql_commands.append(f"""
INSERT INTO monitor_tag (monitor_id, tag_id, value)
SELECT {m_id}, 5, NULL
WHERE NOT EXISTS (SELECT 1 FROM monitor_tag WHERE monitor_id={m_id} AND tag_id=5);
""")
full_sql = "\n".join(sql_commands)
out, err = run_sql(full_sql)
if err:
print("SQL Error:", err)
else:
print("Tags applied successfully!")
# Verification query
verify_sql = """
USE kuma;
SELECT m.id, m.name, GROUP_CONCAT(t.name ORDER BY t.id SEPARATOR ', ') AS tags
FROM monitor m
JOIN monitor_tag mt ON m.id = mt.monitor_id
JOIN tag t ON mt.tag_id = t.id
WHERE m.id IN (57, 94, 98, 99, 100, 101, 102, 103, 104, 105)
GROUP BY m.id, m.name
ORDER BY m.id;
"""
res, _ = run_sql(verify_sql)
print("\nUpdated Monitors and Tags:\n", res)
c.close()

View File

@@ -0,0 +1,38 @@
import paramiko
import yaml
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
with open(r'c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\aerodrome\prometheus\prometheus.yaml', 'r') as f:
docs = list(yaml.safe_load_all(f))
# Find ConfigMap prometheus-config
cm_doc = None
for d in docs:
if d and d.get('kind') == 'ConfigMap' and d.get('metadata', {}).get('name') == 'prometheus-config':
cm_doc = d
break
if cm_doc:
cm_yaml = yaml.dump(cm_doc)
stdin, stdout, stderr = c.exec_command("cat > /tmp/prometheus_cm.yaml")
stdin.write(cm_yaml)
stdin.close()
stdout.channel.recv_exit_status()
_, o, e = c.exec_command("kubectl apply -f /tmp/prometheus_cm.yaml")
print("CM apply:\n", o.read().decode().strip())
err = e.read().decode().strip()
if err:
print("CM err:", err)
# Restart prometheus pod
_, o, _ = c.exec_command("kubectl rollout restart deployment/prometheus")
print("Prometheus restart:", o.read().decode().strip())
_, o, _ = c.exec_command("kubectl rollout status deployment/prometheus --timeout=60s")
print("Rollout status:", o.read().decode().strip())
c.close()

View File

@@ -0,0 +1,45 @@
import paramiko
import time
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
out = o.read().decode().strip()
err = e.read().decode().strip()
return out + ("\nERR: " + err if err else "")
with open("homelab-gitops-repo/manifests/aerodrome/prometheus/prometheus.yaml", "r") as f:
prom_yaml = f.read()
stdin, stdout, stderr = c.exec_command("cat > /tmp/prometheus.yaml")
stdin.write(prom_yaml)
stdin.close()
stdout.channel.recv_exit_status()
print("Applying Prometheus:")
print(run("kubectl apply -f /tmp/prometheus.yaml"))
with open("homelab-gitops-repo/manifests/aerodrome/grafana/grafana.yaml", "r") as f:
grafana_yaml = f.read()
stdin, stdout, stderr = c.exec_command("cat > /tmp/grafana_full.yaml")
stdin.write(grafana_yaml)
stdin.close()
stdout.channel.recv_exit_status()
print("\nApplying Grafana configuration & dashboards:")
print(run("kubectl apply -f /tmp/grafana_full.yaml"))
print("\nRestarting Grafana:")
print(run("kubectl rollout restart deployment/grafana -n default"))
print(run("kubectl rollout status deployment/grafana -n default --timeout=60s"))
print("\nChecking Prometheus pod:")
time.sleep(5)
print(run("kubectl get pods -l app.kubernetes.io/name=prometheus"))
c.close()

View File

@@ -0,0 +1,36 @@
import paramiko
import time
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
with open(r'c:\Users\Snarf\OneDrive\Desktop\code\SnarfCode\homelab-gitops-repo\manifests\aerodrome\prometheus\prometheus.yaml', 'r') as f:
prom_yaml = f.read()
stdin, stdout, stderr = c.exec_command("cat > /tmp/prometheus.yaml")
stdin.write(prom_yaml)
stdin.close()
stdout.channel.recv_exit_status()
_, o, e = c.exec_command("kubectl apply -f /tmp/prometheus.yaml")
print("Kubectl apply:\n", o.read().decode().strip())
err = e.read().decode().strip()
if err:
print("Err:", err)
# Find prometheus pod
_, o, _ = c.exec_command("kubectl get pod -l app.kubernetes.io/name=prometheus -o jsonpath='{.items[0].metadata.name}'")
pod = o.read().decode().strip()
print("Prometheus pod:", pod)
# Reload config
_, o, _ = c.exec_command(f"kubectl exec {pod} -- wget -qO- --post-action='' http://localhost:9090/-/reload || kubectl exec {pod} -- curl -X POST http://localhost:9090/-/reload")
print("Reload result:", o.read().decode().strip())
time.sleep(5)
# Check targets for shipyard
_, o, _ = c.exec_command(f"""kubectl exec {pod} -- wget -qO- 'http://localhost:9090/api/v1/targets' | grep -o '"cluster":"shipyard"[^}}]*' | head -10""")
print("Shipyard targets found in Prometheus:\n", o.read().decode().strip())
c.close()

View File

@@ -0,0 +1,36 @@
import paramiko
import time
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
out = o.read().decode().strip()
err = e.read().decode().strip()
return out + ("\nERR: " + err if err else "")
with open("homelab-gitops-repo/manifests/aerodrome/prometheus/prometheus.yaml", "r", encoding="utf-8") as f:
prom_yaml = f.read()
stdin, stdout, stderr = c.exec_command("cat > /tmp/prometheus_updated.yaml")
stdin.write(prom_yaml)
stdin.close()
stdout.channel.recv_exit_status()
print("Applying Prometheus:")
print(run("kubectl apply -f /tmp/prometheus_updated.yaml"))
print("\nRestarting Prometheus:")
print(run("kubectl rollout restart deployment/prometheus -n default"))
print(run("kubectl rollout status deployment/prometheus -n default --timeout=60s"))
print("\nWaiting 15 seconds for scrape loop...")
time.sleep(15)
print("\nPrometheus active targets count:")
print(run("kubectl exec deployment/prometheus -- wget -qO- http://localhost:9090/api/v1/targets | grep -o '\"health\":\"up\"' | wc -l"))
c.close()

View File

@@ -0,0 +1,75 @@
import paramiko
import time
import re
nodes = {
"Node 1 (snaihvs0lnx01)": "140.44.44.121",
"Node 2 (snaihvs0lnx02)": "140.44.44.122",
"Node 3 (snaihvs0lnx03)": "140.44.44.123",
"Node 4 (idrac-spacedock4)": "140.44.44.124",
}
results = {}
for name, ip in nodes.items():
print(f"Connecting to {name} ({ip})...")
node_data = {}
try:
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect(ip, username="root", password="Katzir476!", timeout=10)
chan = c.invoke_shell()
time.sleep(2)
if chan.recv_ready():
chan.recv(4096)
def send_and_recv(cmd, wait_time=2.5):
chan.send(cmd + "\r\n")
time.sleep(wait_time)
out = ""
while chan.recv_ready():
out += chan.recv(8192).decode('utf-8', errors='ignore')
time.sleep(0.5)
return out
# 1. BIOS Settings
raw_bios = send_and_recv("racadm get bios.SysProfileSettings", 2)
for key in ["SysProfile", "ProcCStates", "ProcC1E", "ProcTurboMode", "ProcPwrPerf"]:
m = re.search(rf"{key}\s*=\s*(\w+)", raw_bios)
node_data[key] = m.group(1) if m else "Unknown"
raw_proc = send_and_recv("racadm get bios.ProcSettings", 2)
for key in ["ProcVirtualization", "ProcExecuteDisable", "LogicalProc"]:
m = re.search(rf"{key}\s*=\s*(\w+)", raw_proc)
node_data[key] = m.group(1) if m else "Unknown"
raw_dev = send_and_recv("racadm get bios.IntegratedDevices", 2)
for key in ["SriovGlobalEnable"]:
m = re.search(rf"{key}\s*=\s*(\w+)", raw_dev)
node_data[key] = m.group(1) if m else "Unknown"
# 2. Storage / RAID
raw_ctrl = send_and_recv("racadm raid get controllers", 2)
controllers = [line.strip() for line in raw_ctrl.splitlines() if "RAID." in line or "PERC" in line or "AHCI." in line]
node_data["Controllers"] = ", ".join(controllers) if controllers else "None"
raw_vdisks = send_and_recv("racadm raid get vdisks", 3)
vdisks = [line.strip() for line in raw_vdisks.splitlines() if line.startswith("Disk.Virtual")]
node_data["VirtualDisks"] = len(vdisks)
node_data["VirtualDisksList"] = vdisks
raw_pdisks = send_and_recv("racadm raid get pdisks", 4)
pdisk_lines = [line.strip() for line in raw_pdisks.splitlines() if line.startswith("Disk.Bay")]
node_data["PhysicalDisks"] = len(pdisk_lines)
node_data["PDisksSummary"] = raw_pdisks
c.close()
except Exception as e:
node_data["Error"] = str(e)
results[name] = node_data
print("\n\n========================= SUMMARY MATRIX =========================")
import json
print(json.dumps(results, indent=2))

View File

@@ -0,0 +1,24 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.90', username='rancher', password='Katzir476!', timeout=10)
def run(cmd):
stdin, stdout, stderr = c.exec_command(f"echo Katzir476! | sudo -S sh -c '{cmd}'")
return stdout.read().decode().strip()
print("=== CPU & VIRTUALIZATION (lscpu) ===")
print(run("lscpu | grep -E 'Model name|CPU\(s\):|Thread|Socket|Virtualization|NUMA'"))
print("\n=== KERNEL PARAMETERS (/proc/cmdline) ===")
print(run("cat /proc/cmdline"))
print("\n=== STORAGE CONTROLLER (lspci) ===")
print(run("lspci | grep -E 'RAID|SAS|SATA|Ethernet|Network'"))
print("\n=== BLOCK DEVICES (lsblk) ===")
print(run("lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS"))
c.close()

View File

@@ -0,0 +1,84 @@
import paramiko
import json
import time
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
HARVESTER_VIP = "140.44.4.90"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
print(f"Connecting to Harvester VIP {HARVESTER_VIP}...")
c.connect(HARVESTER_VIP, username="rancher", key_filename=SSH_KEY, timeout=10)
print("Connected to Harvester VIP!")
def run_k8s(cmd):
full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'"
stdin, stdout, stderr = c.exec_command(full_cmd)
out = stdout.read().decode().strip()
err = stderr.read().decode().strip()
return out
print("\n=== 1. HARVESTER CLUSTER NODES ===")
print(run_k8s("kubectl get nodes -o wide"))
print("\n=== 2. NODE UPTIME ===")
print(run_k8s("uptime"))
print("\n=== 3. VIRTUAL MACHINES (KubeVirt) ===")
print(run_k8s("kubectl get vms,vmis -A -o wide"))
print("\n=== 4. HARVESTER SYSTEM POD STATUS (Not Running / Degraded) ===")
print(run_k8s("kubectl get pods -A | grep -v 'Running\\|Completed'"))
print("\n=== 5. LONGHORN STORAGE VOLUME STATUS ===")
print(run_k8s("kubectl get volumes.longhorn.io -n longhorn-system"))
c.close()
except Exception as e:
print(f"Error connecting to Harvester: {e}")
# Check iDRACs
idracs = {
"Spacedock Node 1": "140.44.44.121",
"Spacedock Node 2": "140.44.44.122",
"Spacedock Node 3": "140.44.44.123",
"Spacedock Node 4": "140.44.44.124",
}
print("\n=== 6. IDRAC SYSTEM EVENT LOGS & POWER STATUS ===")
for name, ip in idracs.items():
print(f"\n--- {name} ({ip}) ---")
try:
ic = paramiko.SSHClient()
ic.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ic.connect(ip, username="root", password="Katzir476!", timeout=8)
chan = ic.invoke_shell()
time.sleep(1.5)
if chan.recv_ready():
chan.recv(4096)
def racadm(cmd):
chan.send(cmd + "\r\n")
time.sleep(2)
out = ""
while chan.recv_ready():
out += chan.recv(8192).decode('utf-8', errors='ignore')
time.sleep(0.5)
return out
pwr = racadm("racadm serveraction powerstatus")
for line in pwr.splitlines():
if "power" in line.lower() or "server" in line.lower():
print(line.strip())
sel = racadm("racadm getsel -s 10")
print("Last 10 SEL Events:")
for line in sel.splitlines()[-12:]:
print(" ", line.strip())
ic.close()
except Exception as e:
print(f"Error querying {name}: {e}")

View File

@@ -0,0 +1,48 @@
import json, ssl, urllib.request
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
ZABBIX_URL = "http://140.44.4.60:8888/api_jsonrpc.php"
def zbx_req(method, params, token=None):
payload = {'jsonrpc': '2.0', 'method': method, 'params': params, 'id': 1}
headers = {'Content-Type': 'application/json-rpc'}
if token:
headers['Authorization'] = f'Bearer {token}'
req = urllib.request.Request(ZABBIX_URL, data=json.dumps(payload).encode('utf-8'), headers=headers, method='POST')
with urllib.request.urlopen(req, timeout=10, context=ctx) as r:
return json.loads(r.read().decode())['result']
token = zbx_req('user.login', {'username': 'Admin', 'password': 'zabbix'})
print("Zabbix Auth Token:", token)
# 1. Actions
actions = zbx_req('action.get', {'output': 'extend', 'selectOperations': 'extend', 'selectFilter': 'extend'}, token)
print("\n=== TRIGGER ACTIONS ===")
for a in actions:
print(f"ID: {a['actionid']}, Name: {a['name']}, Status (0=enabled, 1=disabled): {a['status']}, EventSource: {a['eventsource']}")
print(f" Operations: {a.get('operations')}")
print(f" Filter: {a.get('filter')}")
# 2. Media Types
media_types = zbx_req('mediatype.get', {'output': 'extend'}, token)
print("\n=== MEDIA TYPES ===")
for m in media_types:
print(f"ID: {m['mediatypeid']}, Name: {m['name']}, Type: {m['type']}, Status: {m['status']}")
# 3. User Media
users = zbx_req('user.get', {'output': ['userid', 'username'], 'selectMedias': 'extend'}, token)
print("\n=== USERS & MEDIA ===")
for u in users:
print(f"User: {u['username']} (ID: {u['userid']})")
for med in u.get('medias', []):
print(f" Media: mediatypeid={med['mediatypeid']}, sendto={med['sendto']}, active={med['active']}, severity={med['severity']}")
# 4. Recent Alerts
alerts = zbx_req('alert.get', {'output': 'extend', 'limit': 15, 'sortfield': 'clock', 'sortorder': 'DESC'}, token)
print("\n=== RECENT ALERTS ===")
for al in alerts:
print(f"Alert ID: {al['alertid']}, ActionID: {al['actionid']}, Clock: {al['clock']}, Error: {al.get('error')}, Status: {al.get('status')}, Subject: {al.get('subject')}")

View File

@@ -0,0 +1,467 @@
import json
import requests
import copy
import urllib3
urllib3.disable_warnings()
auth = ("admin", "Katzir476!!!")
base_url = "https://grafana.snarfnet.net"
# Load templates
with open("homelab-gitops-repo/dashboards/grafana/systems-dashboard.json") as f:
sys_template = json.load(f)
with open("homelab-gitops-repo/dashboards/grafana/container-metrics-dashboard.json") as f:
cnt_template = json.load(f)
def build_cluster_dashboard(cluster_key, cluster_title, folder_id, folder_uid, cluster_emoji):
dash = {
"annotations": {"list": []},
"editable": True,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": None,
"links": [],
"liveNow": False,
"refresh": "15s",
"schemaVersion": 39,
"tags": [cluster_key, "resource-monitor", "infrastructure"],
"time": {"from": "now-1h", "to": "now"},
"timepicker": {"refresh_intervals": ["5s", "10s", "15s", "30s", "1m", "5m"]},
"timezone": "browser",
"title": f"{cluster_emoji} {cluster_title} Resource Monitor",
"uid": f"{cluster_key}-resource-monitor",
"version": 1,
"panels": []
}
panels = []
current_y = 0
# ==========================================
# GROUP 1: Systems & Cluster Host Performance
# ==========================================
# Elevate Systems Row Header
panels.append({
"collapsed": False,
"gridPos": {"h": 1, "w": 24, "x": 0, "y": current_y},
"id": 100,
"title": f"🖥️ Systems & Host Performance — {cluster_title} Cluster Fleet",
"type": "row"
})
current_y += 1
# Systems Stat Panels (height 4)
stat_panels = [
{
"id": 1,
"title": f"Online Nodes ({cluster_title})",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'count(node_load1{{cluster="{cluster_key}"}})', "legendFormat": "Nodes", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": None}]}, "unit": "none"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
},
{
"id": 2,
"title": "Avg Cluster CPU Load (1m)",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 6, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'avg(node_load1{{cluster="{cluster_key}"}})', "legendFormat": "Load", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "blue", "value": None}, {"color": "orange", "value": 2.0}, {"color": "red", "value": 4.0}]}, "unit": "none"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
},
{
"id": 3,
"title": "Total Monitored RAM",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 12, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'sum(node_memory_MemTotal_bytes{{cluster="{cluster_key}"}})', "legendFormat": "RAM", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "purple", "value": None}]}, "unit": "bytes"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
},
{
"id": 4,
"title": "Cluster RAM In-Use %",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 18, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'(1 - (sum(node_memory_MemAvailable_bytes{{cluster="{cluster_key}"}}) / sum(node_memory_MemTotal_bytes{{cluster="{cluster_key}"}}))) * 100', "legendFormat": "In-Use", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": None}, {"color": "orange", "value": 75}, {"color": "red", "value": 90}]}, "unit": "percent"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
}
]
panels.extend(stat_panels)
current_y += 4
# Systems CPU Timeseries
panels.append({
"id": 5,
"title": "CPU Utilization by Host (%)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'100 - (avg by (instance) (rate(node_cpu_seconds_total{{mode="idle", cluster="{cluster_key}"}}[2m])) * 100)', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "percent", "min": 0, "max": 100}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}}
})
panels.append({
"id": 6,
"title": "1-Minute Load Average by Host",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'node_load1{{cluster="{cluster_key}"}}', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 10}, "unit": "none", "min": 0}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}}
})
current_y += 8
# Systems Memory Timeseries
panels.append({
"id": 7,
"title": "Memory Used % by Host",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'(1 - (node_memory_MemAvailable_bytes{{cluster="{cluster_key}"}} / node_memory_MemTotal_bytes{{cluster="{cluster_key}"}})) * 100', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "percent", "min": 0, "max": 100}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}}
})
panels.append({
"id": 8,
"title": "Available RAM by Host (GiB)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'node_memory_MemAvailable_bytes{{cluster="{cluster_key}"}}', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 10}, "unit": "bytes"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "min"]}}
})
current_y += 8
# Filesystem & Disk I/O
panels.append({
"id": 9,
"title": "Root (/) Filesystem Usage %",
"type": "bargauge",
"gridPos": {"h": 8, "w": 8, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'(1 - (node_filesystem_avail_bytes{{mountpoint="/", cluster="{cluster_key}"}} / node_filesystem_size_bytes{{mountpoint="/", cluster="{cluster_key}"}})) * 100', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": None}, {"color": "orange", "value": 75}, {"color": "red", "value": 90}]}, "unit": "percent", "min": 0, "max": 100}},
"options": {"displayMode": "gradient", "orientation": "horizontal", "showUnfilled": True}
})
panels.append({
"id": 10,
"title": "Disk Read Throughput",
"type": "timeseries",
"gridPos": {"h": 8, "w": 8, "x": 8, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'sum by (instance) (rate(node_disk_read_bytes_total{{cluster="{cluster_key}"}}[2m]))', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 10}, "unit": "Bps"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}
})
panels.append({
"id": 11,
"title": "Disk Write Throughput",
"type": "timeseries",
"gridPos": {"h": 8, "w": 8, "x": 16, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'sum by (instance) (rate(node_disk_written_bytes_total{{cluster="{cluster_key}"}}[2m]))', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 10}, "unit": "Bps"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}
})
current_y += 8
# Network RX/TX
panels.append({
"id": 12,
"title": "Host Network Receive (RX)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'sum by (instance) (rate(node_network_receive_bytes_total{{device!~"lo|veth.*|flannel.*|cali.*", cluster="{cluster_key}"}}[2m]))', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}
})
panels.append({
"id": 13,
"title": "Host Network Transmit (TX)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'sum by (instance) (rate(node_network_transmit_bytes_total{{device!~"lo|veth.*|flannel.*|cali.*", cluster="{cluster_key}"}}[2m]))', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}
})
current_y += 8
# ==========================================
# GROUP 2: Containers & Workloads
# ==========================================
if cluster_key in ["aerodrome", "spaceport"]:
cadvisor_filter = f'cluster="{cluster_key}"' if cluster_key == "spaceport" else 'instance=~"sninfr2.*"'
panels.append({
"collapsed": False,
"gridPos": {"h": 1, "w": 24, "x": 0, "y": current_y},
"id": 200,
"title": f"📦 Kubernetes Containers & Workloads — {cluster_title} Cluster",
"type": "row"
})
current_y += 1
# Container Stats
panels.extend([
{
"id": 14,
"title": "Monitored Containers",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'count(container_last_seen{{container!="", {cadvisor_filter}}})', "legendFormat": "Containers", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": None}]}, "unit": "none"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
},
{
"id": 15,
"title": "Active Kubernetes Pods",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 6, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'count(count by (pod, namespace) (container_last_seen{{container!="", {cadvisor_filter}}}))', "legendFormat": "Pods", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "blue", "value": None}]}, "unit": "none"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
},
{
"id": 16,
"title": "Total Container CPU Load",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 12, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'sum(rate(container_cpu_usage_seconds_total{{container!="", {cadvisor_filter}}}[2m]))', "legendFormat": "Cores", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "orange", "value": None}]}, "unit": "none"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
},
{
"id": 17,
"title": "Total Container Working RAM",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 18, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'sum(container_memory_working_set_bytes{{container!="", {cadvisor_filter}}})', "legendFormat": "RAM", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "purple", "value": None}]}, "unit": "bytes"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
}
])
current_y += 4
# Container CPU
panels.append({
"id": 18,
"title": "Top 10 CPU Consuming Pods (Cores)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'topk(10, sum by (pod, namespace) (rate(container_cpu_usage_seconds_total{{container!="", {cadvisor_filter}}}[2m])))', "legendFormat": "{{namespace}}/{{pod}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "none"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}}
})
panels.append({
"id": 19,
"title": "CPU Throttling by Namespace",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'sum by (namespace) (rate(container_cpu_cfs_throttled_periods_total{{{cadvisor_filter}}}[2m]))', "legendFormat": "{{namespace}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 10}, "unit": "none"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}}
})
current_y += 8
# Container Memory
panels.append({
"id": 20,
"title": "Top 10 Memory Consuming Pods",
"type": "timeseries",
"gridPos": {"h": 8, "w": 14, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'topk(10, sum by (pod, namespace) (container_memory_working_set_bytes{{container!="", {cadvisor_filter}}}))', "legendFormat": "{{namespace}}/{{pod}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "bytes"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}}
})
panels.append({
"id": 21,
"title": "Memory Working Set by Namespace",
"type": "bargauge",
"gridPos": {"h": 8, "w": 10, "x": 14, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'sum by (namespace) (container_memory_working_set_bytes{{container!="", {cadvisor_filter}}})', "legendFormat": "{{namespace}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "unit": "bytes"}},
"options": {"displayMode": "gradient", "orientation": "horizontal", "showUnfilled": True}
})
current_y += 8
# Container Network
panels.append({
"id": 22,
"title": "Top Pod Network RX (Receive)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'topk(10, sum by (pod, namespace) (rate(container_network_receive_bytes_total{{{cadvisor_filter}}}[2m])))', "legendFormat": "{{namespace}}/{{pod}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}
})
panels.append({
"id": 23,
"title": "Top Pod Network TX (Transmit)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": f'topk(10, sum by (pod, namespace) (rate(container_network_transmit_bytes_total{{{cadvisor_filter}}}[2m])))', "legendFormat": "{{namespace}}/{{pod}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}
})
current_y += 8
else:
# Shipyard (Docker Swarm)
panels.append({
"collapsed": False,
"gridPos": {"h": 1, "w": 24, "x": 0, "y": current_y},
"id": 200,
"title": "📦 Docker Swarm Containers & Service Stacks — Shipyard Fleet",
"type": "row"
})
current_y += 1
panels.extend([
{
"id": 14,
"title": "Monitored Swarm Containers",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": 'count(container_last_seen{cluster="shipyard", name!=""}) or count(container_last_seen{cluster="shipyard"})', "legendFormat": "Containers", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": None}]}, "unit": "none"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
},
{
"id": 15,
"title": "Active Swarm Nodes",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 6, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": 'count(up{cluster="shipyard", job="node-exporter"})', "legendFormat": "Swarm Nodes", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "blue", "value": None}]}, "unit": "none"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
},
{
"id": 16,
"title": "Total Container CPU Load",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 12, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": 'sum(rate(container_cpu_usage_seconds_total{cluster="shipyard"}[2m]))', "legendFormat": "Cores", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "orange", "value": None}]}, "unit": "none"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
},
{
"id": 17,
"title": "Total Container Working RAM",
"type": "stat",
"gridPos": {"h": 4, "w": 6, "x": 18, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": 'sum(container_memory_working_set_bytes{cluster="shipyard"})', "legendFormat": "RAM", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "thresholds"}, "thresholds": {"mode": "absolute", "steps": [{"color": "purple", "value": None}]}, "unit": "bytes"}},
"options": {"colorMode": "value", "graphMode": "area", "justifyMode": "center"}
}
])
current_y += 4
# Shipyard CPU & Memory
panels.append({
"id": 18,
"title": "Container CPU Usage by Service/Task",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": 'topk(10, sum by (instance, id) (rate(container_cpu_usage_seconds_total{cluster="shipyard"}[2m])))', "legendFormat": "{{instance}} ({{id}})", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "none"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}}
})
panels.append({
"id": 19,
"title": "Container Memory Working Set by Service/Task",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": 'topk(10, sum by (instance, id) (container_memory_working_set_bytes{cluster="shipyard"}))', "legendFormat": "{{instance}} ({{id}})", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "bytes"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "lastNotNull", "max"]}}
})
current_y += 8
# Shipyard Network RX/TX
panels.append({
"id": 20,
"title": "Container Network Receive (RX)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": 'topk(10, sum by (instance) (rate(container_network_receive_bytes_total{cluster="shipyard"}[2m])))', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}
})
panels.append({
"id": 21,
"title": "Container Network Transmit (TX)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": current_y},
"datasource": "Prometheus",
"targets": [{"expr": 'topk(10, sum by (instance) (rate(container_network_transmit_bytes_total{cluster="shipyard"}[2m])))', "legendFormat": "{{instance}}", "refId": "A"}],
"fieldConfig": {"defaults": {"color": {"mode": "palette-classic"}, "custom": {"drawStyle": "line", "lineInterpolation": "smooth", "lineWidth": 2, "fillOpacity": 15}, "unit": "Bps"}},
"options": {"legend": {"displayMode": "table", "placement": "bottom", "calcs": ["mean", "max"]}}
})
current_y += 8
dash["panels"] = panels
return dash
clusters = [
{"key": "aerodrome", "title": "Aerodrome", "folder_id": 5, "folder_uid": "folder-aerodrome", "emoji": "✈️"},
{"key": "spaceport", "title": "Spaceport", "folder_id": 6, "folder_uid": "folder-spaceport", "emoji": "🚀"},
{"key": "shipyard", "title": "Shipyard", "folder_id": 7, "folder_uid": "folder-shipyard", "emoji": "⚓"}
]
# 1. Generate and Upload Dashboards
for c in clusters:
dash_json = build_cluster_dashboard(c["key"], c["title"], c["folder_id"], c["folder_uid"], c["emoji"])
# Save to local gitops repo
filename = f"homelab-gitops-repo/dashboards/grafana/{c['key']}-resource-monitor.json"
with open(filename, "w", encoding="utf-8") as f:
json.dump(dash_json, f, indent=2)
print(f"Saved {filename}")
# Upload to Grafana API
payload = {
"dashboard": dash_json,
"folderId": c["folder_id"],
"folderUid": c["folder_uid"],
"overwrite": True
}
resp = requests.post(f"{base_url}/api/dashboards/db", json=payload, auth=auth, verify=False, timeout=10)
print(f"Uploaded {c['title']} dashboard to Grafana: {resp.status_code} - {resp.json().get('status')}")
# 2. Delete old dashboards from AI Stack folder
for old_uid in ["snarfnet-systems", "snarfnet-containers"]:
del_resp = requests.delete(f"{base_url}/api/dashboards/uid/{old_uid}", auth=auth, verify=False, timeout=10)
print(f"Deleted old dashboard {old_uid}: {del_resp.status_code}")
print("\nDashboard generation and deployment complete!")

View File

@@ -0,0 +1,27 @@
import subprocess
import os
edge_path = r"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe"
out_png = os.path.abspath("docs/snarfnet_homelab_topology.png")
html_url = "file:///" + os.path.abspath("docs/snarfnet_homelab_topology.html").replace("\\", "/")
cmd = [
edge_path,
"--headless",
"--disable-gpu",
"--no-sandbox",
"--hide-scrollbars",
"--window-size=1920,2400",
f"--screenshot={out_png}",
html_url
]
print("Running cmd:", " ".join(cmd))
res = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
print("Return code:", res.returncode)
print("Stdout:", res.stdout)
print("Stderr:", res.stderr)
print("File exists?", os.path.exists(out_png))
if os.path.exists(out_png):
print("File size:", os.path.getsize(out_png))

10
scripts/cat_21_conf.py Normal file
View File

@@ -0,0 +1,10 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.71', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
cmd = "docker exec $(docker ps -q --filter name=nginx-proxy-manager_app) cat /data/nginx/proxy_host/21.conf"
_, o, _ = c.exec_command(cmd)
print(o.read().decode())
c.close()

13
scripts/cat_prom_conf.py Normal file
View File

@@ -0,0 +1,13 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
print(run("kubectl exec deployment/prometheus -- cat /etc/prometheus/prometheus.yml"))
c.close()

15
scripts/check_30.py Normal file
View File

@@ -0,0 +1,15 @@
import paramiko
import sys
sys.stdout.reconfigure(encoding='utf-8')
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
c.connect('140.44.4.30', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=5)
print("Connected to 140.44.4.30 with snarf")
stdin, stdout, _ = c.exec_command("hostname && ip a")
print(stdout.read().decode())
c.close()
except Exception as e:
print("Failed 140.44.4.30:", e)

View File

@@ -0,0 +1,24 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
print("Nodes on Aerodrome:")
print(run("kubectl get nodes -o wide"))
print("\nDaemonSets on Aerodrome:")
print(run("kubectl get ds -A"))
print("\nServices matching metrics:")
print(run("kubectl get svc -A | grep -E 'metric|node|cadvisor|exporter'"))
print("\nKubelet metrics on node 140.44.4.11 (port 10250 or 10255):")
print(run("curl -sk https://localhost:10250/metrics/cadvisor | head -n 10"))
c.close()

View File

@@ -0,0 +1,19 @@
import json, ssl, urllib.request
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
ZABBIX_URL = 'http://140.44.4.60:8888/api_jsonrpc.php'
payload = {'jsonrpc': '2.0', 'method': 'user.login', 'params': {'username': 'Admin', 'password': 'zabbix'}, 'id': 1}
req = urllib.request.Request(ZABBIX_URL, data=json.dumps(payload).encode(), headers={'Content-Type': 'application/json-rpc'})
token = json.loads(urllib.request.urlopen(req, context=ctx).read())['result']
payload = {'jsonrpc': '2.0', 'method': 'alert.get', 'params': {'output': 'extend', 'limit': 20, 'sortfield': 'clock', 'sortorder': 'DESC'}, 'id': 2}
req = urllib.request.Request(ZABBIX_URL, data=json.dumps(payload).encode(), headers={'Content-Type': 'application/json-rpc', 'Authorization': f'Bearer {token}'})
alerts = json.loads(urllib.request.urlopen(req, context=ctx).read())['result']
for a in alerts:
print(f"ID: {a['alertid']} | Status: {a['status']} | User: {a['userid']} | Error: '{a.get('error')}' | Subject: {a.get('subject')}")

View File

@@ -0,0 +1,9 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
_, o, _ = c.exec_command("kubectl get pods -A")
print(o.read().decode().strip())
c.close()

View File

@@ -1,17 +1,34 @@
import os
import sys
import paramiko
import sys
sys.stdout.reconfigure(encoding="utf-8")
pkey = paramiko.Ed25519Key.from_private_key_file(os.path.expanduser('~/.ssh/id_ed25519_snarfnet'))
sys.stdout.reconfigure(encoding='utf-8')
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.75', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect("140.44.4.11", username="root", pkey=pkey, timeout=10)
def run(cmd):
stdin, stdout, stderr = c.exec_command(cmd)
return stdout.read().decode().strip(), stderr.read().decode().strip()
cmd = "curl -s http://140.44.4.21:80/main.105b4163.async.js | grep -o -E '\"/api/[a-zA-Z0-9_/]+\"' | sort -u | head -n 30"
stdin, stdout, stderr = client.exec_command(cmd)
print("API ROUTES ON 140.44.4.21:")
print(stdout.read().decode())
client.close()
cid, _ = run("echo 'Katzir476!' | sudo -S docker ps -q -f name=peanut_peanut")
# Check readdir in src/app or src
script = """
const fs = require('fs');
function walk(dir) {
let res = [];
for (let f of fs.readdirSync(dir)) {
let p = dir + '/' + f;
if (fs.statSync(p).isDirectory()) res.push(...walk(p));
else if (f.endsWith('.ts') || f.endsWith('.js') || f.endsWith('.tsx')) res.push(p);
}
return res;
}
console.log(walk('src/app/api'));
"""
run(f"cat << 'EOF' > /tmp/walk.js\n{script}\nEOF")
out, err = run(f"echo 'Katzir476!' | sudo -S docker exec -i {cid} node < /tmp/walk.js")
print(out)
if err: print(err)
c.close()

View File

@@ -0,0 +1,9 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.41', username='root', password='dietpi', timeout=10)
_, o, e = c.exec_command("kubectl get app aerodrome-prometheus -n infrastructure")
print("out:", o.read().decode().strip())
print("err:", e.read().decode().strip())
c.close()

View File

@@ -0,0 +1,16 @@
import paramiko
import base64
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
print("Argo initial secret:")
print(run("kubectl get secret argocd-initial-admin-secret -n infrastructure -o jsonpath='{.data.password}'"))
c.close()

View File

@@ -0,0 +1,8 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.41', username='root', password='dietpi', timeout=10)
_, o, _ = c.exec_command("kubectl get applications.argoproj.io -n infrastructure")
print(o.read().decode().strip())
c.close()

View File

@@ -0,0 +1,29 @@
import requests
import json
import urllib3
urllib3.disable_warnings()
token = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJhcmdvY2QiLCJzdWIiOiJob21lcGFnZTphcGlLZXkiLCJuYmYiOjE3OTQ1MDIwNzgsImlhdCI6MTc5NDUwMjA3OCwianRpIjoiZjQ2NzU0NjMtODRhOS00MjI0LWI3ODItNmY5ODQxYWEwY2UxIn0.aHhrny2RBG-vAJAyXMvan0jw65jkP3bguqEduaG78O0"
headers = {
"Authorization": f"Bearer {token}",
"Content-Type": "application/json"
}
resp = requests.get("https://argo.snarfnet.net/api/v1/applications", headers=headers, verify=False, timeout=10)
data = resp.json()
items = data.get("items", [])
for app in items:
name = app.get("metadata", {}).get("name")
sync = app.get("status", {}).get("sync", {}).get("status")
health = app.get("status", {}).get("health", {}).get("status")
resources = app.get("status", {}).get("resources", [])
out_resources = [r for r in resources if r.get("status") != "Synced" or r.get("health", {}).get("status") not in ["Healthy", None]]
print(f"[{name}] Sync: {sync}, Health: {health}, Total Resources: {len(resources)}")
if out_resources:
print(f" -> Out-of-sync / unhealthy resources:")
for r in out_resources:
print(f" {r.get('kind')}/{r.get('name')}: Sync={r.get('status')}, Health={r.get('health', {}).get('status')}")

View File

@@ -1,27 +1,35 @@
import paramiko
import sys
import json
sys.stdout.reconfigure(encoding='utf-8')
key_path = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.11", username="root", key_filename=key_path, timeout=5)
c.connect("140.44.4.41", username="root", key_filename=SSH_KEY, timeout=10)
cmd = "k3s kubectl -n infra exec authentik-postgresql-0 -- psql -U authentik -d authentik -c 'SELECT id, name, authorization_flow_id FROM authentik_core_application;'"
stdin, stdout, stderr = c.exec_command(cmd)
print("Authentik Applications:")
print(stdout.read().decode('utf-8'))
def k8s(cmd):
full_cmd = f"kubectl {cmd}"
_, o, e = c.exec_command(full_cmd)
return o.read().decode().strip(), e.read().decode().strip()
cmd2 = "k3s kubectl -n infra exec authentik-postgresql-0 -- psql -U authentik -d authentik -c 'SELECT id, name, external_host FROM authentik_providers_proxy_proxyprovider;'"
stdin, stdout, stderr = c.exec_command(cmd2)
print("Authentik Proxy Providers:")
print(stdout.read().decode('utf-8'))
print("Authentik pods:")
out, _ = k8s("get pods -A | grep -i authentik")
print(out)
cmd3 = "k3s kubectl -n infra exec authentik-postgresql-0 -- psql -U authentik -d authentik -c 'SELECT id, name, client_id FROM authentik_providers_oauth2_oauth2provider;'"
stdin, stdout, stderr = c.exec_command(cmd3)
print("Authentik OAuth2/OIDC Providers:")
print(stdout.read().decode('utf-8'))
# Check Authentik pod name
pod = ""
for line in out.splitlines():
if "server" in line:
pod = line.split()[1]
break
print("Authentik server pod:", pod)
if pod:
# Query Authentik applications and proxy providers via manage.py or python
out, err = k8s(f"exec -n authentik {pod} -- ak list_apps 2>/dev/null || kubectl exec -n authentik {pod} -- python -m authentik.manage shell -c 'from authentik.core.models import Application; print([(a.name, a.slug) for a in Application.objects.all()])'")
print("Applications:", out, err)
# Query Authentik proxy providers
out, err = k8s(f"exec -n authentik {pod} -- python -m authentik.manage shell -c 'from authentik.providers.proxy.models import ProxyProvider; print([(p.name, p.external_host) for p in ProxyProvider.objects.all()])'")
print("Proxy Providers:", out, err)
c.close()

View File

@@ -0,0 +1,35 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10)
_, o, _ = c.exec_command("kubectl exec -n infra authentik-postgresql-0 -- cat /opt/bitnami/postgresql/secrets/password")
pw = o.read().decode().strip()
def psql(query):
cmd = f'kubectl exec -n infra authentik-postgresql-0 -- sh -c "PGPASSWORD={pw} psql -U authentik -d authentik -c \\"{query}\\""'
_, o, e = c.exec_command(cmd)
res = o.read().decode().strip()
err = e.read().decode().strip()
if err and not res:
return f"[ERR: {err}]"
return res
print("--- APPLICATIONS ---")
print(psql("SELECT name, slug FROM authentik_core_application;"))
print("\n--- PROXY PROVIDERS ---")
print(psql("SELECT name, external_host, mode FROM authentik_providers_proxy_proxyprovider;"))
print("\n--- EXPRESSION POLICIES ---")
print(psql("SELECT name, expression FROM authentik_policies_expression_expressionpolicy;"))
print("\n--- POLICY BINDINGS ---")
print(psql("SELECT target_id, enabled FROM authentik_policies_policybinding;"))
print("\n--- AUTHENTIK SETTINGS / SESSIONS ---")
print(psql("SELECT key, value FROM authentik_tenants_tenant;"))
c.close()

View File

@@ -0,0 +1,32 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10)
_, o, _ = c.exec_command("kubectl exec -n infra authentik-postgresql-0 -- cat /opt/bitnami/postgresql/secrets/password")
pw = o.read().decode().strip()
sql = """
\\echo '=== APPS ==='
SELECT name, slug FROM authentik_core_application;
\\echo '=== PROVIDERS ==='
SELECT name, external_host, mode FROM authentik_providers_proxy_proxyprovider;
\\echo '=== POLICIES ==='
SELECT name, expression FROM authentik_policies_expression_expressionpolicy;
"""
stdin, stdout, stderr = c.exec_command("cat > /tmp/query.sql")
stdin.write(sql)
stdin.close()
stdout.channel.recv_exit_status()
c.exec_command("kubectl cp /tmp/query.sql infra/authentik-postgresql-0:/tmp/query.sql")
cmd = f"kubectl exec -n infra authentik-postgresql-0 -- env PGPASSWORD='{pw}' psql -U authentik -d authentik -f /tmp/query.sql"
_, o, e = c.exec_command(cmd)
print(o.read().decode())
print("ERR:", e.read().decode())
c.close()

View File

@@ -0,0 +1,46 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10)
def k8s(cmd):
full_cmd = f"kubectl {cmd}"
_, o, e = c.exec_command(full_cmd)
return o.read().decode().strip(), e.read().decode().strip()
pod = "authentik-server-cd7d9b949-vfl2m"
script = '''
from authentik.core.models import Application
from authentik.providers.proxy.models import ProxyProvider
from authentik.policies.expression.models import ExpressionPolicy
print("=== APPLICATIONS ===")
for app in Application.objects.all():
print(f"{app.name} (slug: {app.slug}) | Provider: {app.provider}")
print("=== PROXY PROVIDERS ===")
for p in ProxyProvider.objects.all():
print(f"{p.name} | Host: {p.external_host} | Mode: {p.mode}")
print("=== EXPRESSION POLICIES ===")
for ep in ExpressionPolicy.objects.all():
print(f"Policy: {ep.name}")
print(ep.expression)
'''
# Write script to /tmp/ak_query.py on 140.44.4.11
stdin, stdout, stderr = c.exec_command("cat > /tmp/ak_query.py")
stdin.write(script)
stdin.close()
stdout.channel.recv_exit_status()
# Copy into pod and execute
k8s(f"cp /tmp/ak_query.py infra/{pod}:/tmp/ak_query.py -c server")
out, err = k8s(f"exec -n infra {pod} -c server -- /lifecycle/ak shell < /tmp/ak_query.py")
print("Authentik query output:\n", out)
if err: print("ERR:\n", err)
c.close()

View File

@@ -0,0 +1,49 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10)
def k8s(cmd):
full_cmd = f"kubectl {cmd}"
_, o, e = c.exec_command(full_cmd)
return o.read().decode().strip(), e.read().decode().strip()
pod = "authentik-server-cd7d9b949-vfl2m"
# Inspect applications in Authentik
py_query = """
python3 - << 'PYEOF'
from authentik.core.models import Application
from authentik.providers.proxy.models import ProxyProvider
from authentik.policies.models import PolicyBindingModel, Policy
print("=== APPLICATIONS ===")
for app in Application.objects.all():
print(f"App: {app.name} (slug: {app.slug})")
print(f" Provider: {app.provider}")
print("\n=== PROXY PROVIDERS ===")
for p in ProxyProvider.objects.all():
print(f"Provider: {p.name}")
print(f" External host: {p.external_host}")
print(f" Internal host: {p.internal_host}")
print(f" Mode: {p.mode}")
print("\n=== POLICIES ===")
for pol in Policy.objects.all():
print(f"Policy: {pol.name} ({pol.__class__.__name__})")
print("\n=== POLICY BINDINGS ===")
for pb in PolicyBindingModel.objects.all():
print(f"Binding: {pb.target} -> {pb.policy} (enabled={pb.enabled}, order={pb.order})")
PYEOF
"""
out, err = k8s(f"exec -n infra {pod} -c authentik-server -- {py_query}")
print(out)
if err: print("ERR:", err)
c.close()

View File

@@ -0,0 +1,28 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10)
_, o, _ = c.exec_command("kubectl exec -n infra authentik-postgresql-0 -- cat /opt/bitnami/postgresql/secrets/password")
pw = o.read().decode().strip()
sql = """
SELECT p.name, ep.expression
FROM authentik_policies_expression_expressionpolicy ep
JOIN authentik_policies_policy p ON ep.policy_ptr_id = p.policy_uuid;
"""
stdin, stdout, stderr = c.exec_command("cat > /tmp/query_policies.sql")
stdin.write(sql)
stdin.close()
stdout.channel.recv_exit_status()
c.exec_command("kubectl cp /tmp/query_policies.sql infra/authentik-postgresql-0:/tmp/query_policies.sql")
cmd = f"kubectl exec -n infra authentik-postgresql-0 -- env PGPASSWORD='{pw}' psql -U authentik -d authentik -f /tmp/query_policies.sql"
_, o, e = c.exec_command(cmd)
print(o.read().decode())
c.close()

View File

@@ -0,0 +1,25 @@
import urllib.request, ssl, json
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
AK_BASE = "https://authentik.snarfnet.net"
AK_TOKEN = "SnarfNetPermanentAutomationToken2026"
def req(endpoint):
url = f"{AK_BASE}/api/v3/{endpoint}"
r = urllib.request.Request(url, headers={"Authorization": f"Bearer {AK_TOKEN}"})
with urllib.request.urlopen(r, timeout=10, context=ctx) as res:
return json.loads(res.read().decode())
print("=== Authentik Providers (Proxy) ===")
providers = req("providers/proxy/")
for p in providers.get("results", []):
print(p["pk"], p["name"], p.get("internal_host"), p.get("external_host"), p.get("mode"))
print("\n=== Authentik Outposts ===")
outposts = req("outposts/instances/")
for o in outposts.get("results", []):
print(o["pk"], o["name"], o["type"], [p for p in o.get("providers", [])])

View File

@@ -0,0 +1,28 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.11", username="root", key_filename=SSH_KEY, timeout=10)
_, o, _ = c.exec_command("kubectl exec -n infra authentik-postgresql-0 -- cat /opt/bitnami/postgresql/secrets/password")
pw = o.read().decode().strip()
def run_sql(sql):
cmd = f"kubectl exec -i -n infra authentik-postgresql-0 -- /bin/bash -c 'export PGPASSWORD={pw}; psql -U authentik -d authentik'"
stdin, stdout, stderr = c.exec_command(cmd)
stdin.write(sql + "\n")
stdin.close()
return stdout.read().decode().strip()
print("--- APPLICATIONS ---")
print(run_sql("SELECT name, slug FROM authentik_core_application;"))
print("\n--- PROXY PROVIDERS ---")
print(run_sql("SELECT name, external_host, mode FROM authentik_providers_proxy_proxyprovider;"))
print("\n--- EXPRESSION POLICIES ---")
print(run_sql("SELECT name, expression FROM authentik_policies_expression_expressionpolicy;"))
c.close()

View File

@@ -0,0 +1,16 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
pod = run("kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'")
print(run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "USE kuma; SELECT id, name, type, url, port, accepted_statuscodes_json FROM monitor WHERE name LIKE '%AWX%';" """))
print(run(f"""kubectl exec -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root -e "USE kuma; SELECT * FROM heartbeat WHERE monitor_id IN (SELECT id FROM monitor WHERE name LIKE '%AWX%') ORDER BY id DESC LIMIT 5;" """))
c.close()

View File

@@ -0,0 +1,22 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10)
def k8s(cmd):
full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'"
_, o, e = c.exec_command(full_cmd)
return o.read().decode().strip(), e.read().decode().strip()
print("Harvester Block Devices:")
out, _ = k8s("kubectl get blockdevices.harvesterhci.io -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,NODE:.spec.nodeName,DEV:.status.devicePath,FS:.status.fileSystem.type,MOUNT:.status.fileSystem.mountPoint,PROVISIONED:.status.provisioned")
print(out)
print("\nHarvester Node Disk status:")
out, _ = k8s("kubectl get nodes.harvesterhci.io -A -o yaml | grep -E 'nodeName|diskStatus|mountPath|provisioned'")
print(out[:1000])
c.close()

View File

@@ -0,0 +1,10 @@
import paramiko
import time
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
_, o, _ = c.exec_command("echo 'Katzir476!' | sudo -S docker stack ps monitoring")
print(o.read().decode().strip())
c.close()

17
scripts/check_cm_json.py Normal file
View File

@@ -0,0 +1,17 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
_, o, _ = c.exec_command('kubectl get cm homepage-2-1744068158 -n infrastructure -o json')
import json
cm = json.loads(o.read().decode())
print("Data keys in ConfigMap:", list(cm.get('data', {}).keys()))
print("\n--- settings.yaml ---")
print(cm.get('data', {}).get('settings.yaml'))
print("\n--- widgets.yaml ---")
print(cm.get('data', {}).get('widgets.yaml'))
c.close()

23
scripts/check_cm_live.py Normal file
View File

@@ -0,0 +1,23 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
_, o, _ = c.exec_command("kubectl get configmap prometheus-config -o json")
import json
data = json.loads(o.read().decode().strip())
prom_yml = data.get("data", {}).get("prometheus.yml", "")
print("Length of prometheus.yml in CM:", len(prom_yml))
print("Contains shipyard:", "shipyard" in prom_yml)
if "shipyard" in prom_yml:
for line in prom_yml.splitlines():
if "shipyard" in line:
print(" ", line)
else:
print("Last 20 lines of prometheus.yml:")
for line in prom_yml.splitlines()[-20:]:
print(" ", line)
c.close()

11
scripts/check_cm_meta.py Normal file
View File

@@ -0,0 +1,11 @@
import paramiko
import json
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
_, o, _ = c.exec_command("kubectl get configmap prometheus-config -o json")
data = json.loads(o.read().decode().strip())
print(json.dumps(data.get("metadata"), indent=2))
c.close()

11
scripts/check_cm_now.py Normal file
View File

@@ -0,0 +1,11 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
_, o, _ = c.exec_command("kubectl get configmap prometheus-config -o yaml")
yaml_str = o.read().decode().strip()
print("Length of CM:", len(yaml_str))
print("Contains shipyard:", "shipyard" in yaml_str)
c.close()

9
scripts/check_cm_tmp.py Normal file
View File

@@ -0,0 +1,9 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
_, o, _ = c.exec_command("cat /tmp/prometheus_cm.yaml | grep -A 25 'job_name: node-exporter'")
print(o.read().decode().strip())
c.close()

View File

@@ -0,0 +1,25 @@
import requests
import urllib3
import json
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
s = requests.Session()
s.verify = False
r = s.post('https://14.44.4.1/api/auth/login', json={'username': 'snarf', 'password': 'Katzir476!!!'})
csrf = r.headers.get('x-csrf-token') or r.cookies.get('csrf_token')
headers = {'X-CSRF-Token': csrf} if csrf else {}
res = s.get('https://14.44.4.1/proxy/network/v2/api/site/default/firewall-policies', headers=headers)
policies = res.json()
compute_net_id = "1748bb1d-f674-4dcd-85d0-f58109a02e77"
for p in policies:
src_nets = p.get('source', {}).get('network_ids', []) or []
dst_nets = p.get('destination', {}).get('network_ids', []) or []
if compute_net_id in src_nets or compute_net_id in dst_nets:
print(f"Policy: {p.get('name')} (Index {p.get('index')}) | Action: {p.get('action')} | Type: {p.get('connection_state_type')}")
print(f" Src: {p.get('source')}")
print(f" Dst: {p.get('destination')}")

View File

@@ -0,0 +1,31 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10)
def k8s(cmd):
full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'"
_, o, e = c.exec_command(full_cmd)
return o.read().decode().strip(), e.read().decode().strip()
print("Replica spec fields:")
out, _ = k8s("kubectl explain replica.longhorn.io.spec")
print(out)
print("\nVolume spec fields:")
out, _ = k8s("kubectl explain volume.longhorn.io.spec")
print(out)
print("\nEngine spec fields:")
out, _ = k8s("kubectl explain engine.longhorn.io.spec")
print(out)
# Check engine CR directly
out, _ = k8s("kubectl get engines.longhorn.io -n longhorn-system pvc-4da99c48-f52e-4adf-9b4c-16e8878eb495-e-0 -o yaml")
print("\nEngine YAML for pvc-4da99c48:")
print(out)
c.close()

View File

@@ -0,0 +1,30 @@
import paramiko
import sys
sys.stdout.reconfigure(encoding='utf-8')
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
stdin, stdout, stderr = c.exec_command(cmd)
return stdout.read().decode().strip(), stderr.read().decode().strip()
print("Ping 14.4.4.20 (HomeAssistant):")
out, _ = run("ping -c 2 -W 2 14.4.4.20")
print(out)
print("Ping 14.4.4.15:")
out, _ = run("ping -c 2 -W 2 14.4.4.15")
print(out)
print("Ping 14.4.4.44 (nutserver):")
out, _ = run("ping -c 2 -W 2 14.4.4.44")
print(out)
print("Traceroute to 14.4.4.44:")
out, _ = run("traceroute 14.4.4.44 || tracepath 14.4.4.44")
print(out)
c.close()

View File

@@ -0,0 +1,33 @@
import paramiko
import sys
sys.stdout.reconfigure(encoding='utf-8')
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run_sql(sql):
pod_cmd = "kubectl get pod -n monitoring -l app.kubernetes.io/name=uptime-kuma -o jsonpath='{.items[0].metadata.name}'"
_, o, _ = c.exec_command(pod_cmd)
pod = o.read().decode().strip()
cmd = f"kubectl exec -i -n monitoring {pod} -- mariadb --socket=/app/data/run/mariadb.sock -u root"
stdin, stdout, stderr = c.exec_command(cmd)
stdin.write(sql)
stdin.close()
return stdout.read().decode().strip(), stderr.read().decode().strip()
print("--- Monitor tags for 57, 94, 98-105 ---")
query = """
USE kuma;
SELECT m.id, m.name, t.name as tag_name
FROM monitor m
LEFT JOIN monitor_tag mt ON m.id = mt.monitor_id
LEFT JOIN tag t ON mt.tag_id = t.id
WHERE m.id IN (57, 94, 98, 99, 100, 101, 102, 103, 104, 105)
ORDER BY m.id, t.id;
"""
out, _ = run_sql(query)
print(out)
c.close()

View File

@@ -0,0 +1,21 @@
import paramiko
import time
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.90', username='rancher', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
stdin, stdout, stderr = c.exec_command(f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'")
out = stdout.read().decode().strip()
return out
time.sleep(5)
print("Pods status:")
print(run("kubectl get pods -n ai-stack -l app.kubernetes.io/name=dcgm-exporter -o wide"))
print("Logs from one pod:")
pod = run("kubectl get pod -n ai-stack -l app.kubernetes.io/name=dcgm-exporter -o jsonpath='{.items[0].metadata.name}'")
print(run(f"kubectl logs -n ai-stack {pod} --tail=20"))
c.close()

View File

@@ -0,0 +1,26 @@
import paramiko
import sys
sys.stdout.reconfigure(encoding='utf-8')
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.71', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
print(f"\n>>> {cmd}")
_, o, e = c.exec_command(cmd)
res = o.read().decode().strip()
if res:
print(res)
return res
run("""python3 -c "
import sqlite3
con = sqlite3.connect('/mnt/nfs_shares/heimdall/www/app.sqlite')
cur = con.cursor()
for r in cur.execute('SELECT id, user_id, title, pinned, \"order\", deleted_at FROM items LIMIT 10;').fetchall():
print(r)
" """)
c.close()

View File

@@ -0,0 +1,13 @@
import paramiko
import json
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
_, o, _ = c.exec_command("kubectl get deploy prometheus -o json")
data = json.loads(o.read().decode().strip())
volumes = data.get("spec", {}).get("template", {}).get("spec", {}).get("volumes", [])
for v in volumes:
print(v)
c.close()

View File

@@ -0,0 +1,14 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
print(run("kubectl get deployment grafana -o jsonpath='{.spec.template.spec.containers[0].volumeMounts}'"))
c.close()

View File

@@ -0,0 +1,24 @@
import paramiko
import sys
sys.stdout.reconfigure(encoding='utf-8')
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.41', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def test(cmd):
print(f"\n>>> {cmd}")
_, o, e = c.exec_command(cmd)
res = o.read().decode().strip()
err = e.read().decode().strip()
if res:
print(res[:300])
if err:
print("[stderr]", err[:200])
test("curl -m 3 -s -I http://140.44.4.28:448/ || echo 'Timeout 140.44.4.28:448'")
test("curl -m 3 -s -I http://140.44.4.31:448/ || echo 'Timeout 140.44.4.31:448'")
test("curl -m 3 -s http://140.44.4.70:8050/api/v1/devices || echo 'Failed PeaNUT'")
test("curl -m 3 -s -H 'Authorization: token 47c10c54574131c007302b246352f653facb5062' http://140.44.4.62:90/api/v1/user || echo 'Failed Gitea'")
c.close()

View File

@@ -0,0 +1,23 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10)
def run_cmd(cmd):
full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'"
_, o, e = c.exec_command(full_cmd)
return o.read().decode().strip(), e.read().decode().strip()
print("Mount /dev/sdf to /mnt and check contents:")
out, err = run_cmd("mkdir -p /mnt/test && mount /dev/sdf /mnt/test && ls -la /mnt/test")
print("sdf:", out, err)
# Check all devices sdb through sdh
for d in ["sdb", "sdc", "sdd", "sde", "sdf", "sdg", "sdh"]:
out, _ = run_cmd(f"mkdir -p /mnt/{d} && mount /dev/{d} /mnt/{d} 2>/dev/null && ls -la /mnt/{d} | grep -E 'longhorn|replicas'")
print(f"{d}: {out}")
c.close()

View File

@@ -0,0 +1,23 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10)
def run_cmd(cmd):
full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'"
_, o, e = c.exec_command(full_cmd)
return o.read().decode().strip(), e.read().decode().strip()
print("Mount /dev/sdf to /tmp/test:")
out, err = run_cmd("mkdir -p /tmp/test && mount /dev/sdf /tmp/test && ls -la /tmp/test; umount /tmp/test")
print("sdf:", out, err)
# Check all devices on Node 1:
for d in ["sdb", "sdc", "sdd", "sde", "sdf", "sdg", "sdh"]:
out, _ = run_cmd(f"mkdir -p /tmp/{d} && mount /dev/{d} /tmp/{d} 2>/dev/null && ls -la /tmp/{d}; umount /tmp/{d} 2>/dev/null")
print(f"--- {d} ---:\n{out}")
c.close()

View File

@@ -0,0 +1,35 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.90", username="rancher", key_filename=SSH_KEY, timeout=10)
def k8s(cmd):
full_cmd = f"echo Katzir476! | sudo -S sh -c 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml; PATH=$PATH:/var/lib/rancher/rke2/bin; {cmd}'"
_, o, e = c.exec_command(full_cmd)
return o.read().decode().strip(), e.read().decode().strip()
print("=== NODES.LONGHORN.IO DISK STATUS ===")
out, _ = k8s("kubectl get nodes.longhorn.io -n longhorn-system -o yaml | grep -E 'diskStatus|diskUUID|ready|schedulable|storageAvailable|storageMaximum|path:'")
print(out[:1500])
print("\n=== LSBLK & MOUNTS ON NODE 01 ===")
out, _ = k8s("lsblk -f; echo '--- DF ---'; df -h | grep -E 'Filesystem|extra-disks|harvester'")
print(out)
# Check all 4 nodes for extra-disks mount
for ip in ["140.44.4.81", "140.44.4.82", "140.44.4.83", "140.44.4.84"]:
print(f"\n--- Node {ip} ---")
try:
nc = paramiko.SSHClient()
nc.set_missing_host_key_policy(paramiko.AutoAddPolicy())
nc.connect(ip, username="rancher", key_filename=SSH_KEY, timeout=5)
_, o, _ = nc.exec_command("echo Katzir476! | sudo -S lsblk | grep -E 'sd|nvme'; df -h | grep extra")
print(o.read().decode().strip())
nc.close()
except Exception as e:
print("Error connecting:", e)
c.close()

View File

@@ -0,0 +1,27 @@
import urllib.request
import urllib.parse
import ssl
import json
import http.client
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
def do_req(ip, host_header, path):
conn = http.client.HTTPSConnection(ip, 443, context=ctx, timeout=8)
conn.request("GET", path, headers={"Host": host_header})
resp = conn.getresponse()
data = resp.read().decode('utf-8')
conn.close()
return json.loads(data)
try:
token = do_req("140.44.4.62", "dns2.snarfnet.net", "/api/user/login?user=admin&pass=snarfnetadmin&includeInfo=true")['token']
rec_heimdall = do_req("140.44.4.62", "dns2.snarfnet.net", f"/api/zones/records/get?token={token}&zone=snarfnet.net&domain=heimdall.snarfnet.net")
print("DNS2 heimdall:", rec_heimdall)
rec_portal = do_req("140.44.4.62", "dns2.snarfnet.net", f"/api/zones/records/get?token={token}&zone=snarfnet.net&domain=portal.snarfnet.net")
print("DNS2 portal:", rec_portal)
except Exception as e:
print("DNS check err:", e)

View File

@@ -0,0 +1,19 @@
import paramiko
import sys
sys.stdout.reconfigure(encoding='utf-8')
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.71', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
stdin, stdout, stderr = c.exec_command(cmd)
return stdout.read().decode().strip(), stderr.read().decode().strip()
print("--- echo 'Katzir476!' | sudo -S docker service ls ---")
out, err = run("echo 'Katzir476!' | sudo -S docker service ls")
print("OUT:\n", out)
print("ERR:\n", err)
c.close()

View File

@@ -0,0 +1,41 @@
import requests
import json
import urllib3
urllib3.disable_warnings()
try:
r = requests.get("http://140.44.4.28:448/api/status-page/heartbeat/default", verify=False, timeout=10)
data = r.json()
heartbeats = data.get("heartbeatList", {})
down = []
up = []
# Also fetch status page monitor list to map IDs to names
r_sp = requests.get("http://140.44.4.28:448/api/status-page/default", verify=False, timeout=10)
sp_data = r_sp.json()
id_to_name = {}
for group in sp_data.get("publicGroupList", []):
for m in group.get("monitorList", []):
id_to_name[str(m.get("id"))] = m.get("name")
for mid, beats in heartbeats.items():
name = id_to_name.get(str(mid), f"Monitor #{mid}")
if beats:
last = beats[-1]
if last.get("status") == 1:
up.append((name, last.get("ping")))
else:
down.append((name, last.get("msg")))
print(f"Total Monitors: {len(heartbeats)}")
print(f"Up: {len(up)} | Down: {len(down)}")
if down:
print("\nDown Monitors:")
for name, msg in down:
print(f" ❌ {name}: {msg}")
else:
print("\nAll monitors are UP! 🎉")
except Exception as e:
print("Error:", e)

View File

@@ -0,0 +1,9 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
_, o, _ = c.exec_command("kubectl exec deploy/prometheus -- ls -la /etc/prometheus")
print("ls -la /etc/prometheus:\n", o.read().decode().strip())
c.close()

View File

@@ -0,0 +1,30 @@
import paramiko
SSH_KEY = r"C:\Users\Snarf\.ssh\id_ed25519_snarfnet"
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect("140.44.4.81", username="rancher", key_filename=SSH_KEY, timeout=10)
def run_cmd(cmd):
full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'"
_, o, e = c.exec_command(full_cmd)
return o.read().decode().strip(), e.read().decode().strip()
print("--- /etc/fstab on Node 1 ---")
out, _ = run_cmd("cat /etc/fstab")
print(out)
print("\n--- Systemd mount units on Node 1 ---")
out, _ = run_cmd("systemctl list-unit-files | grep -i var-lib-harvester")
print(out)
print("\n--- Blkid on Node 1 ---")
out, _ = run_cmd("blkid")
print(out)
print("\n--- Any harvester disk mount script or cloud-init ---")
out, _ = run_cmd("ls -la /oem; cat /oem/99_custom.yaml 2>/dev/null || cat /oem/harvester.config 2>/dev/null")
print(out[:1000])
c.close()

View File

@@ -0,0 +1,17 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
print("Ping 140.44.4.80:")
print(run("ping -c 2 -W 1 140.44.4.80 || true"))
print("ARP table for bc:24:11:67:16:cd:")
print(run("ip neigh | grep -i 'bc:24:11:67:16:cd' || arp -a | grep -i 'bc:24:11:67:16:cd' || true"))
c.close()

View File

@@ -0,0 +1,19 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
out = o.read().decode().strip()
return out
print("ps aux | grep glances:")
print(run("ps aux | grep glances"))
print("\nnetstat / ss listening:")
print(run("ss -tulpn | grep 61208 || ss -tulpn | grep glances"))
c.close()

View File

@@ -0,0 +1,28 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.71', username='snarf', password='Katzir476!', timeout=10)
def run(cmd):
full_cmd = f"echo Katzir476! | sudo -S sh -c '{cmd}'"
_, o, e = c.exec_command(full_cmd)
return o.read().decode().strip(), e.read().decode().strip()
print("=== GOTIFY DISCORD BRIDGE LOGS ===")
cid, _ = run("docker ps -q -f name=gotify-discord-bridge")
if cid:
logs, _ = run(f"docker logs {cid} --tail=100")
print(logs)
else:
print("Bridge container not found!")
print("\n=== GOTIFY SERVER RECENT MESSAGES ===")
# Check messages inside Gotify DB or logs
gid, _ = run("docker ps -q -f name=gotify_gotify")
if gid:
logs, _ = run(f"docker logs {gid} --tail=50")
print(logs)
c.close()

View File

@@ -0,0 +1,13 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.71', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
print(run("ls -la /mnt/swarm_shared/legacy_volumes/gotify/_data/"))
c.close()

View File

@@ -0,0 +1,22 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.71', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
py_script = """import sqlite3
conn = sqlite3.connect('/mnt/swarm_shared/legacy_volumes/gotify/gotify.db')
cur = conn.cursor()
indexes = cur.execute("SELECT name, sql FROM sqlite_master WHERE type='index' AND tbl_name='applications'").fetchall()
for idx in indexes:
print(idx)
"""
stdin, stdout, stderr = c.exec_command("cat > /tmp/check_idx.py && python3 /tmp/check_idx.py")
stdin.write(py_script)
stdin.close()
print(stdout.read().decode())
print(stderr.read().decode())
c.close()

View File

@@ -0,0 +1,16 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.71', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
out = o.read().decode().strip()
return out
print("Gotify inspect mounts:")
print(run("docker service inspect gotify_gotify --format '{{json .Spec.TaskTemplate.ContainerSpec.Mounts}}'"))
c.close()

View File

@@ -0,0 +1,19 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.71', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
out = o.read().decode().strip()
return out
print("Gotify service ps:")
print(run("docker service ps gotify_gotify --no-trunc | head -n 10"))
print("\nGotify service logs:")
print(run("docker service logs gotify_gotify --tail=25"))
c.close()

View File

@@ -0,0 +1,37 @@
import paramiko
nodes = {
"Node 1 (140.44.44.121)": "140.44.44.121",
"Node 2 (140.44.44.122)": "140.44.44.122",
"Node 3 (140.44.44.123)": "140.44.44.123",
"Node 4 (140.44.44.124)": "140.44.44.124",
}
for name, ip in nodes.items():
try:
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect(ip, username="root", password="Katzir476!", timeout=10)
# Check PCIe inventory for NVIDIA
stdin, stdout, stderr = c.exec_command("racadm hwinventory")
out = stdout.read().decode(errors="ignore")
gpus = []
for block in out.split("[InstanceID:"):
if "NVIDIA" in block or "Tesla" in block or "T4" in block:
lines = [l.strip() for l in block.splitlines() if any(k in l for k in ["DeviceDescription", "DeviceType", "FQDD", "PCI"])]
gpus.append(" | ".join(lines[:3]))
# Check Above 4G decoding
stdin, stdout, stderr = c.exec_command("racadm get bios.IntegratedDevices.MemOpMode || racadm get bios.IntegratedDevices.MemoryMappedIoAbove4GB")
a4g = stdout.read().decode(errors="ignore").strip()
print(f"=== {name} ===")
print(f" GPUs detected: {len(gpus)}")
for g in gpus:
print(f" {g}")
c.close()
except Exception as e:
print(f"=== {name} === ERROR: {e}")

View File

@@ -0,0 +1,27 @@
import requests
import json
import urllib3
urllib3.disable_warnings()
auth = ('admin', 'd0KJLZJXUrbIhW1sjU8lrdFYj7VXts26V7Jw7xQr')
try:
r = requests.get('https://grafana.snarfnet.net/api/datasources', auth=auth, verify=False, timeout=10)
print("Datasources Status:", r.status_code)
if r.status_code == 200:
for ds in r.json():
print(f" - ID: {ds.get('id')}, Name: {ds.get('name')}, Type: {ds.get('type')}, URL: {ds.get('url')}, isDefault: {ds.get('isDefault')}")
except Exception as e:
print("Error getting datasources:", e)
try:
r = requests.get('https://grafana.snarfnet.net/api/search', auth=auth, verify=False, timeout=10)
print("Dashboards Status:", r.status_code)
if r.status_code == 200:
dashboards = r.json()
print(f"Total dashboards: {len(dashboards)}")
for d in dashboards[:15]:
print(f" - {d.get('title')} (uri: {d.get('uri')}, type: {d.get('type')})")
except Exception as e:
print("Error getting dashboards:", e)

View File

@@ -0,0 +1,15 @@
import requests
import json
import urllib3
urllib3.disable_warnings()
auth = ("admin", "Katzir476!!!")
url = "https://grafana.snarfnet.net/api/search"
resp = requests.get(url, auth=auth, verify=False, timeout=10)
print("Status:", resp.status_code)
dashboards = resp.json()
print(f"Total Dashboards Found: {len(dashboards)}")
for d in dashboards:
print(f" - [{d.get('type')}] Title: {d.get('title')} | UID: {d.get('uid')} | URL: {d.get('url')}")

View File

@@ -0,0 +1,14 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
print(run("kubectl diff -f /tmp/grafana_full.yaml || true"))
c.close()

View File

@@ -0,0 +1,14 @@
import requests
import json
import urllib3
urllib3.disable_warnings()
auth = ("admin", "Katzir476!!!")
# Let's query Prometheus datasource from Grafana or directly from 140.44.4.11/140.44.4.34
url = "https://grafana.snarfnet.net/api/datasources"
resp = requests.get(url, auth=auth, verify=False, timeout=10)
datasources = resp.json()
print("Datasources:")
for ds in datasources:
print(f"ID: {ds['id']} | Name: {ds['name']} | Type: {ds['type']} | URL: {ds['url']}")

View File

@@ -0,0 +1,21 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
print("Namespaces on Aerodrome:")
print(run("kubectl get ns"))
print("Grafana pod:")
pod = run("kubectl get pod -l app.kubernetes.io/name=grafana -o jsonpath='{.items[0].metadata.name}'")
print(pod)
print("Grafana admin secret:")
print(run("kubectl get secret grafana -o jsonpath='{.data.admin-password}' | base64 -d"))
c.close()

View File

@@ -0,0 +1,23 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
pod = run("kubectl get pod -l app.kubernetes.io/name=grafana -o jsonpath='{.items[0].metadata.name}'")
print("Pod:", pod)
print("ConfigMap grafana:")
print(run("kubectl get cm grafana -o yaml"))
print("Testing local curl inside pod to localhost:3000/api/health:")
print(run(f"kubectl exec {pod} -- wget -qO- http://localhost:3000/api/health"))
print("Grafana env:")
print(run(f"kubectl exec {pod} -- env | grep -i grafana"))
c.close()

View File

@@ -0,0 +1,17 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
pod = run("kubectl get pod -l app.kubernetes.io/name=grafana -o jsonpath='{.items[0].metadata.name}'")
print("Grafana pod:", pod)
print("Grafana provisioning logs:")
print(run(f"kubectl logs {pod} | grep -i -E 'provision|datasource|dashboard' | tail -n 25"))
c.close()

View File

@@ -0,0 +1,19 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
pod = run("kubectl get pod -l app.kubernetes.io/name=grafana -o jsonpath='{.items[0].metadata.name}'")
print(run(f"kubectl exec {pod} -- ls -la /etc/grafana/provisioning/datasources"))
print(run(f"kubectl exec {pod} -- cat /etc/grafana/provisioning/datasources/datasources.yaml"))
print(run(f"kubectl exec {pod} -- ls -la /etc/grafana/provisioning/dashboards"))
print(run(f"kubectl exec {pod} -- cat /etc/grafana/provisioning/dashboards/dashboards.yaml"))
print(run(f"kubectl exec {pod} -- ls -la /var/lib/grafana-dashboards"))
c.close()

View File

@@ -0,0 +1,13 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
print(run("kubectl get pods -l app.kubernetes.io/name=grafana"))
c.close()

View File

@@ -0,0 +1,15 @@
import paramiko
import base64
import json
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
_, o, _ = c.exec_command('kubectl get secret grafana -n default -o json')
data = json.loads(o.read().decode()).get('data', {})
for k, v in data.items():
print(k, ":", base64.b64decode(v).decode('utf-8', errors='ignore'))
c.close()

View File

@@ -0,0 +1,30 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
py_script = """import sqlite3
conn = sqlite3.connect('/tmp/grafana_real.db')
cur = conn.cursor()
tables = [t[0] for t in cur.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall()]
for t in tables:
if 'source' in t or 'dash' in t or 'user' in t:
print('Matching table:', t)
print('--- data_source content ---')
for row in cur.execute('SELECT * FROM data_source').fetchall():
print(row)
print('--- dashboard content ---')
for row in cur.execute('SELECT id, title, slug, uid FROM dashboard').fetchall():
print(row)
"""
stdin, stdout, stderr = c.exec_command("cat > /tmp/inspect_grafana.py && python3 /tmp/inspect_grafana.py")
stdin.write(py_script)
stdin.close()
print(stdout.read().decode())
print(stderr.read().decode())
c.close()

View File

@@ -0,0 +1,17 @@
import paramiko
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
c.connect('140.44.4.11', username='root', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=10)
def run(cmd):
_, o, e = c.exec_command(cmd)
return o.read().decode().strip()
pod = run("kubectl get pod -l app.kubernetes.io/name=grafana -o jsonpath='{.items[0].metadata.name}'")
print("Grafana pod:", pod)
print("Provisioning logs:")
print(run(f"kubectl logs {pod} | grep -i -E 'provision|datasource|dashboard'"))
c.close()

17
scripts/check_ha.py Normal file
View File

@@ -0,0 +1,17 @@
import paramiko
import sys
sys.stdout.reconfigure(encoding='utf-8')
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
c.connect('14.4.4.20', username='snarf', key_filename=r'C:\Users\Snarf\.ssh\id_ed25519_snarfnet', timeout=5)
print("Connected to 14.4.4.20")
except Exception as e:
print("Failed key:", e)
try:
c.connect('14.4.4.20', username='root', password='dietpi', timeout=5)
print("Connected with root/dietpi")
except Exception as e2:
print("Failed root/dietpi:", e2)

Some files were not shown because too many files have changed in this diff Show More