Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add homelab topology documentation, infrastructure improvement plans, and management scripts

Browse Source
This commit is contained in:
Snarf
2026-10-31 21:52:01 -04:00
parent a793a04ffb
commit 502b00922b
12 changed files with 1395 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

37
docs/homelab_topology.md
View File

@@ -25,15 +25,16 @@ All Kubernetes resources are hosted on the **Compute VLAN (VLAN 20 — `140.44.4
### Cluster A: `aerodrome` (Local / Rancher Management Cluster)
* **Infrastructure role:** Hosts Rancher management plane, Authentik SSO, Grafana, AWX, and Uptime Kuma.
* **SSH Default User:** `root`
* **IP Allocation:**
* `sninfr2lnxctl01` $\rightarrow$ `140.44.4.11` (Master Node, Debian 12, Ready)
* `sninfr2lnxctl02` $\rightarrow$ `140.44.4.12` (Master Node, Debian 12, Ready)
* `sninfr2lnxctl03` $\rightarrow$ `140.44.4.13` (Master Node, Debian 12, Ready)
* `sninfr2lnxwrk01` $\rightarrow$ `140.44.4.21` (Worker Node, **NotReady/Offline**)
* `sninfr2lnxwrk01` $\rightarrow$ `140.44.4.21` (Worker Node, Debian 12, Ready)
* `sninfr2lnxwrk02` $\rightarrow$ `140.44.4.22` (Worker Node, Debian 12, Ready)
* `sninfr2lnxwrk03` $\rightarrow$ `140.44.4.23` (Worker Node, Debian 12, Ready)
* `sninfr2lnxwrk04` $\rightarrow$ `140.44.4.24` (Worker Node, Debian 12, Ready)
* `sninfr2lnxwrk05` $\rightarrow$ `140.44.4.25` (Worker Node, **NotReady/Offline**)
* `sninfr2lnxwrk05` $\rightarrow$ `140.44.4.25` (Worker Node, Debian 12, Ready)
* `sninfr2lnxwrk06` $\rightarrow$ `140.44.4.26` (Worker Node, Debian 12, Ready)
* `sninfr2lnxwrk07` $\rightarrow$ `140.44.4.27` (Worker Node, Debian 12, Ready)
* **MetalLB IP Range:** `140.44.4.28` - `140.44.4.39`
@@ -50,6 +51,7 @@ All Kubernetes resources are hosted on the **Compute VLAN (VLAN 20 — `140.44.4
### Cluster B: `spacedock` (Harvester HCI Virtualization Cluster)
* **Infrastructure role:** Physical hypervisors running Harvester v1.8.0. Handles virtual machine provisioning and GPU partition workloads.
* **SSH Default User:** `rancher`
* **IP Allocation:**
* `snaihvs0lnx01` $\rightarrow$ `140.44.4.81` (Master/HCI, amd64, Ready)
* `snaihvs0lnx02` $\rightarrow$ `140.44.4.82` (Master/HCI, amd64, Ready)
@@ -59,17 +61,18 @@ All Kubernetes resources are hosted on the **Compute VLAN (VLAN 20 — `140.44.4
### Cluster C: `spaceport` (Raspberry Pi Downstream Cluster)
* **Infrastructure role:** Hosts development and operational workloads: Gitea, Homepage, Jenkins, Semaphore (Ansible), Coder, and your primary Zabbix monitoring server.
* **SSH Default User:** `root` (Dropbear SSH on DietPi/Debian 12; password `dietpi` on `ctl01-04` & `wrk01-04, 06`, `Katzir476!` on `wrk05`)
* **IP Allocation:**
* `sninfr0lnxctl01` $\rightarrow$ `140.44.4.41` (Master Node, arm64 RPi 5, Ready)
* `sninfr0lnxctl02` $\rightarrow$ `140.44.4.42` (Master Node, arm64 RPi 5, Ready)
* `sninfr0lnxctl03` $\rightarrow$ `140.44.4.43` (Master Node, arm64 RPi 5, Ready)
* `sninfr0lnxctl04` $\rightarrow$ `140.44.4.44` (Master Node, arm64 RPi 5, Ready)
* `sninfr0lnxwrk01` $\rightarrow$ `140.44.4.51` (Worker Node, arm64, Ready)
* `sninfr0lnxwrk02` $\rightarrow$ `140.44.4.52` (Worker Node, arm64, Ready)
* `sninfr0lnxwrk03` $\rightarrow$ `140.44.4.53` (Worker Node, arm64, Ready)
* `sninfr0lnxwrk04` $\rightarrow$ `140.44.4.54` (Worker Node, arm64, Ready)
* `sninfr0lnxwrk05` $\rightarrow$ `140.44.4.55` (Worker Node, arm64, Ready)
* `sninfr0lnxwrk06` $\rightarrow$ `140.44.4.56` (Worker Node, arm64, Ready)
* `sninfr0lnxwrk01` $\rightarrow$ `140.44.4.51` (Worker Node, arm64 RPi 5, Ready)
* `sninfr0lnxwrk02` $\rightarrow$ `140.44.4.52` (Worker Node, arm64 RPi 5, Ready)
* `sninfr0lnxwrk03` $\rightarrow$ `140.44.4.53` (Worker Node, arm64 RPi 5, Ready)
* `sninfr0lnxwrk04` $\rightarrow$ `140.44.4.54` (Worker Node, arm64 RPi 5, Ready)
* `sninfr0lnxwrk05` $\rightarrow$ `140.44.4.55` (Worker Node, arm64 RPi 5, Ready)
* `sninfr0lnxwrk06` $\rightarrow$ `140.44.4.56` (Worker Node, arm64 RPi 5, Ready)
* **MetalLB IP Range:** `140.44.4.57` - `140.44.4.65`
* `140.44.4.57` $\rightarrow$ `monitoring/zabbix-zabbix-web` (Port 8888)
* `140.44.4.58` $\rightarrow$ `infrastructure/semaphore-lb` (Port 100)
@@ -85,21 +88,23 @@ All Kubernetes resources are hosted on the **Compute VLAN (VLAN 20 — `140.44.4
### Cluster D: `shipyard` (Docker Swarm / Portainer Legacy Cluster)
* **Infrastructure role:** Hosts legacy containerized stacks (Nginx Proxy Manager, Gotify, Heimdall dashboard, Homebox inventory, MeshCentral remote management, Peanut, Pi-hole, VS Code Server, and Watchtower).
* **SSH Default User:** `snarf`
* **IP Allocation:**
* `Shipyard VIP` $\rightarrow$ `140.44.4.70` (Keepalived Virtual IP)
* `Shipyard 01` $\rightarrow$ `140.44.4.71` (Manager Node, amd64, Ready)
* `Shipyard 02` $\rightarrow$ `140.44.4.72` (Worker Node, amd64, Ready)
* `Shipyard 03` $\rightarrow$ `140.44.4.73` (Worker Node, amd64, Ready)
* `Shipyard 04` $\rightarrow$ `140.44.4.74` (Worker Node / NPM Host, amd64, Ready)
* `Shipyard 05` $\rightarrow$ `140.44.4.75` (Worker Node, amd64, Ready)
* `Shipyard 01` $\rightarrow$ `140.44.4.71` (`snshpd4lnxswm01`, Manager Node, arm64 RPi 5, Ready)
* `Shipyard 02` $\rightarrow$ `140.44.4.72` (`snshpd4lnxswm02`, Worker Node, arm64 RPi 5, Ready)
* `Shipyard 03` $\rightarrow$ `140.44.4.73` (`snshpd4lnxswm03`, Worker Node, arm64 RPi 5, **Down/Offline**)
* `Shipyard 04` $\rightarrow$ `140.44.4.74` (`snshpd4lnxswm04`, Worker Node / NPM Host, arm64 RPi 5, Ready)
* `Shipyard 05` $\rightarrow$ `140.44.4.75` (`snshpd4lnxswm05`, Worker Node, arm64 RPi 5, **Down/Offline**)
---
### AI Stack (Raspberry Pis)
* **Infrastructure role:** Edge AI inference and local web interface.
* **SSH Default User:** `snarf`
* **IP Allocation:**
* `snnpud4lnxops01` $\rightarrow$ `140.44.4.7` (Raspberry Pi with AI Hat 2, arm64)
* `snnpud4lnxweb01` $\rightarrow$ `140.44.4.8` (Raspberry Pi running Open WebUI, arm64)
* `snnpud4lnxops01` $\rightarrow$ `140.44.4.7` (Raspberry Pi with AI Hat 2, arm64, Ready)
* `snnpud4lnxweb01` $\rightarrow$ `140.44.4.8` (Raspberry Pi running Open WebUI, arm64, Ready)
---
@@ -155,6 +160,6 @@ This table records how subdomains on `*.snarfnet.net` route to internal IP addre
| `spacedock1.snarfnet.net` | HTTPS | `140.44.44.121` | 80 | Harvester Hypervisor Node 1 UI |
| `spacedock3.snarfnet.net` | HTTPS | `140.44.44.123` | 80 | Harvester Hypervisor Node 3 UI |
| `spacedock4.snarfnet.net` | HTTP | `140.44.44.124` | 80 | Dakboard Display Node |
| `longhorn.aerodrome.snarfnet.net`| HTTP | `140.44.4.21` (K3s Node) | 80 | Longhorn Storage UI (Aerodrome) |
| `longhorn.aerodrome.snarfnet.net`| HTTP | `140.44.4.36` (K3s VIP) | 80 | Longhorn Storage UI (Aerodrome) |
| `longhorn.spaceport.snarfnet.net` | HTTP | `140.44.4.41` (K3s Node) | 80 | Longhorn Storage UI (Spaceport) |

16
docs/infrastructure_improvement_plan.md
View File

@@ -146,9 +146,19 @@ flowchart TD
- `Backups & Storage Health` (Default Priority: `6`)
- Provision dedicated **Client Tokens** (`http://140.44.4.70:8090/client`) for each important device (Primary Phone, Tablet, Desktop Browser, Homepage Dashboard).
2. **Discord Forwarding Architecture (Item 2):**
- **Gotify $\rightarrow$ Discord Bridge:** Deploy a lightweight, self-healing container service (`gotify-discord-bridge`) on Shipyard Swarm alongside Gotify. It connects internally to `ws://gotify:80/stream?token=<CLIENT_TOKEN>`, transforms every incoming Gotify message into a rich, color-coded Discord Embed (Red for priority $\ge 8$, Orange for $5-7$, Blue for $\le 4$), and POSTs it to your Discord Webhook URL.
- **Zabbix $\rightarrow$ Discord & Gotify (Dual-Path Redundancy):** Configure Zabbix 7.0.16 via JSON-RPC API to enable both the native **Discord** webhook media type and a **Gotify** webhook media type, linked to the `Admin` user, and **enable** Trigger Action `'Report problems to Zabbix administrators'` (`status=0`, currently disabled!). If Gotify or Swarm is ever down during an outage, Zabbix still reaches Discord directly from `spaceport`.
- **Home Assistant $\rightarrow$ Discord & Gotify:** Add a unified `notify.snarfnet_broadcast` group in Home Assistant (`14.4.4.20`) combining `notify.gotify` and the native Discord webhook integration.
- **Dedicated Discord Channel Webhooks (Verified):**
- `ups-power`: `https://discord.com/api/webhooks/1566076118535114813/2e_fCJeHb-mF8VwerzOORGRa-QUDFuwN6R2Gfz-MGUiIOudA0PiXGZSyd19PNFG-vR7D`
- `infra-alerts`: `https://discord.com/api/webhooks/1566076777963593772/wKN30N0ZXVJKGP-prZ1_P9cp_u7Oakmh8R4bVh0qr8RFpyeGcJZrHfPQBJxtfzQqb4dn`
- `home-assistant`: `https://discord.com/api/webhooks/1566077255837294684/4x_sTqDeEHfIeGccEutOSvPKAnCV6Fyr7uAuECOt8M8QZcOvL2XZ0GVATW7EmcSyUW33`
- **Gotify $\rightarrow$ Discord Multi-Channel Bridge:** Deploy a lightweight, self-healing container service (`gotify-discord-bridge`) on Shipyard Swarm alongside Gotify. It connects internally to `ws://gotify:80/stream?token=<CLIENT_TOKEN>`, maps each Gotify `appid` to its dedicated Discord channel webhook (`UPS & Power Resilience` $\rightarrow$ `ups-power`, `Home Assistant Smart Home` $\rightarrow$ `home-assistant`, and `Zabbix` / `AWX` / `Backups` $\rightarrow$ `infra-alerts`), transforms every incoming Gotify message into a rich, color-coded Discord Embed (Red for priority $\ge 8$, Orange for $5-7$, Blue for $\le 4$), and POSTs it to the mapped Discord Webhook URL.
- **Zabbix $\rightarrow$ Discord & Gotify (Dual-Path Redundancy):** Configure Zabbix 7.0.16 via JSON-RPC API to enable both the native **Discord** webhook media type (pointing to `infra-alerts`, plus `ups-power` for UPS host group triggers) and a **Gotify** webhook media type, linked to the `Admin` user, and **enable** Trigger Action `'Report problems to Zabbix administrators'` (`status=0`, currently disabled!). If Gotify or Swarm is ever down during an outage, Zabbix still reaches Discord directly from `spaceport`.
- **Home Assistant $\leftrightarrow$ Gotify & Discord (`14.4.4.20:8123` — Configured & Verified):**
- Provisioned reusable script **`script.snarfnet_broadcast`** ([scripts/setup_homeassistant_notifications.py](file:///c:/Users/Snarf/OneDrive/Desktop/code/SnarfCode/scripts/setup_homeassistant_notifications.py)) which records structured broadcast entries via `logbook.log`, creates a persistent UI notification, and pushes directly to `notify.mobile_app_pixel_10_pro_fold` and `notify.mobile_app_pixel_9_pro_fold`.
- `gotify-discord-bridge` polls Home Assistant's `/api/logbook` API using the Long-Lived Access Token, evaluates any Jinja2 state templates via `/api/template`, publishes new notifications to Gotify Application #3 (`Home Assistant Smart Home`), and routes them to the `#home-assistant` Discord webhook.
- Provisioned Home Assistant automations:
1. **`Snarfnet UPS Power & Exhaust Protocol` (`1793453000001`):** Monitors `sensor.server_closet_status_data` (`OL`/`OB`/`LB`), broadcasts live UPS telemetry (`battery_charge`, `load`, `input_voltage`, `server_room_temperature`), and automatically sets `fan.server_closet_switch` to `100%` when running on battery.
2. **`Snarfnet Server Room Thermal Guard` (`1793453000002`):** Engages `fan.server_closet_switch` at `100%` and broadcasts a high-temperature warning if `sensor.server_room_temperature_temperature` exceeds `82°F` for 2 minutes.
3. **Bambu H2D 3D Printer Automations (`1788367985433`, `1788368709567`, `1788370272217`, `1788370321997`):** Enhanced to broadcast `Started`, `Completed`, `Stalled`, and `Failed` events to Gotify and `#home-assistant` on Discord alongside existing camera snapshots.
3. **Complete Zabbix Sensors, Thresholds, & Templates (Item 7):**
- **Fix Missing Hosts (100% Coverage):**
- Deploy `zabbix-agent2` with shared PSK (`PSK_autoregister`) via Ansible across all missing Linux nodes: Spaceport Masters (`140.44.4.41-44`), Aerodrome Workers (`140.44.4.25`, `.27`), AI Pis (`140.44.4.7`, `.8`), and Harvester nodes (`140.44.4.81-83`).