Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add network blueprint for high-availability DNS, honeypot, and UDM firewall hardening

Browse Source
This commit is contained in:
Snarf
2026-11-05 11:44:43 -05:00
parent f1c197422d
commit fa969fc1d3
1 changed files with 122 additions and 125 deletions
Download Patch File Download Diff File Expand all files Collapse all files

247
docs/DNS_Setup_Plan.md
View File

@@ -1,185 +1,182 @@
# Project Blueprint: High-Availability DNS Stack, Network Honeypot & VLAN Security Architecture
# Project Blueprint: High-Availability DNS Stack, Network Honeypot & UDM Pro Network Security
## 1. Project Objectives & Current State Assessment
## 1. Executive Summary & Live UDM Pro Topology Assessment
### 1.1 Objectives
1. **Enterprise-Grade DNS Stack:**
- Deploy a resilient, scalable, dual-cluster DNS architecture across **Aerodrome** (`local`) and **Spaceport** (`c-m-4fgkl75b`).
- Implement **Pi-hole** as the front-end ad/malware filtering and caching engine.
- Implement **Technitium DNS Server** as the authoritative local DNS resolver (`snarfnet.net` split-horizon and PTR zones) and recursive DNS-over-HTTPS (DoH) upstream.
- Ensure Active-Active / High Availability across both clusters with zero rate-limiting bottlenecks.
2. **Network Security & Honeypot:**
- Deploy an automated, containerized honeypot (**OpenCanary** or **Cowrie**) with centralized logging and instant alerts to Gotify / Zabbix.
3. **VLAN Segmentation & Firewall Policy Hardening:**
- Audit and enforce inter-VLAN firewall rules on the **UniFi Dream Machine Pro (UDM Pro)** (`140.44.4.1`).
- Isolate Smart Home / IoT, Storage, and Guest traffic from the Management/Infrastructure plane.
- Apply Kubernetes `NetworkPolicy` resources to restrict east-west namespace traffic.
Based on the live UDM Pro controller diagnostics, we have mapped the exact network topography, identified critical security exposures, and tailored the implementation plan.
---
### 1.2 Inspection of Existing Cluster "Stubs"
### 1.1 Live Network & VLAN Topography
We probed both clusters and discovered why the existing DNS implementations stalled:
| Cluster | Component | Current State | Root Cause Failure | Recommendation |
|---|---|:---:|---|---|
| **Aerodrome** (`140.44.4.11`) | `pihole` (deploy) | `0/1 Pending` | PVC `pihole` does not exist (`FailedScheduling`) | **Delete and rebuild cleanly** |
| **Aerodrome** (`140.44.4.11`) | `technitium-dns` (deploy) | `0/1 Pending` | PVC `technitium-longhorn-pvc` does not exist | **Delete and rebuild cleanly** |
| **Aerodrome** (`140.44.4.11`) | `pihole-dns-tcp/udp` | NodePort | Mapped to ephemeral high ports (`32655`/`31195`); standard DNS clients cannot query NodePorts | **Replace with Port 53 LoadBalancer VIP** |
| **Spaceport** (`140.44.4.41`) | `pihole` (deploy) | `1/1 Running` | Uses unmanaged local storage, exposed via NodePort only | **Delete and replace with unified manifest** |
| **Spaceport** (`140.44.4.41`) | `technitium-dns` (deploy) | `0/1 Pending` | PVC `technitium-longhorn-pvc` missing | **Delete and rebuild cleanly** |
> [!IMPORTANT]
> **Decision: Clean Teardown & Declarative Deployment.** Rather than patching fractured, incomplete stubs, we will prune the orphaned resources in `infra` and `infrastructure` and deploy a unified, declarative GitOps stack with dedicated Longhorn storage volumes and MetalLB VIPs on standard port 53.
| VLAN ID | Network Name | Subnet / CIDR | Gateway IP | Active Leases | Security Classification |
|:---:|---|---|---|:---:|---|
| **1** | **LAN** | `14.44.4.0/25` | `14.44.4.1` | 14 / 117 | Trusted Internal Clients |
| **2** | **Management** | `140.44.44.0/24` | `140.44.44.1` | 9 / 51 | Out-of-band (iDRAC, Switches, APs) |
| **3** | **Spacedock** | `140.44.1.0/24` | `140.44.1.1` | 0 / 249 | Harvester VM / Hypervisor Tenant |
| **20** | **Compute** | `140.44.4.0/25` | `140.44.4.1` | 1 / 15 (Static Nodes) | Mission-Critical K3s Clusters, Swarm |
| **30** | **Media** | `192.168.1.0/25` | `192.168.1.1` | 1 / 91 | Streaming & Entertainment Devices |
| **40** | **Storage** | `14.10.10.0/25` | `14.10.10.1` | 2 / 125 | Synology NAS (`.71`, `.72`) NFS / iSCSI |
| **50** | **Smart Home** | `14.4.4.0/25` | `14.4.4.1` | 53 / 85 | IoT Devices, Gaming Host (`14.4.4.15`) |
| **60** | **HA** | `140.4.4.0/25` | `140.4.4.1` | 0 / 7 | High-Availability Interconnects |
| **VPN** | **WireGuard / Teleport** | `192.168.3.0/24` / `.2.0/24` | UDM Pro | Remote | Ingress Remote Access |
---
## 2. High-Availability DNS Architecture (Pi-hole + Technitium)
### 1.2 Critical Security Findings from UDM Pro Diagnostics
1. **Default Security Posture is `Allow All`:**
* Currently, the UDM Pro routes traffic freely between all subnets without restriction.
* **Risk:** The 53 IoT devices in Smart Home (`14.4.4.0/25`) and Media devices can directly initiate connections to Kubernetes nodes, etcd ports, SSH, and Synology NAS shares in `Compute` (`140.44.4.0/25`) and `Storage` (`14.10.10.0/25`).
2. **WAN Port Forwarding to Smart Home Host (`14.4.4.15`):**
* Port forwards for game servers (`FS25` ports 8443/10823, `Stationeers` ports 6443/27015) point directly from WAN (`174.106.252.96`) to host `14.4.4.15` on the Smart Home VLAN.
* If an exploit compromises this host, an attacker currently has unhindered lateral access to the entire homelab infrastructure.
3. **DNS Leaks & Lack of Local Filtering:**
* UDM Pro WAN is set to `Auto DNS Server` (Spectrum ISP DNS `0.0.0.0`).
* No DNS ad/malware filtering or split-horizon DNS caching is currently active across the subnets.
---
## 2. High-Availability DNS Stack (Pi-hole + Technitium)
```mermaid
flowchart TD
subgraph Clients["LAN Clients across all VLANs (VLAN 20, 40, 50, Guest)"]
ClientDevices["Laptops, Phones, Servers, IoT"]
subgraph Clients["LAN Clients across all VLANs (VLAN 1, 20, 30, 50, VPN)"]
Devices["IoT Devices (53), Laptops, Servers, Hypervisors"]
end
subgraph UDM["UniFi Dream Machine Pro (140.44.4.1)"]
DHCP["DHCP Options 6:<br>Primary DNS: 140.44.4.35 (Aerodrome VIP)<br>Secondary DNS: 140.44.4.65 (Spaceport VIP)"]
DHCP["DHCP Option 6 per VLAN:<br>Primary DNS: 140.44.4.35 (Aerodrome VIP)<br>Secondary DNS: 140.44.4.65 (Spaceport VIP)"]
NAT["DNS Redirection Rule:<br>Capture port 53 to hardcoded DNS (8.8.8.8) -> Redirect to VIPs"]
end
subgraph AeroCluster["Aerodrome K3s Cluster"]
subgraph AeroCluster["Aerodrome K3s Cluster (VLAN 20)"]
AeroVIP["MetalLB VIP: 140.44.4.35:53 (UDP/TCP)"]
AeroPihole["Pi-hole Instance 1<br>(Ad-blocking, Caching, FTL rate-limit: 0)"]
AeroTech["Technitium Instance 1 (Primary Authoritative)<br>(snarfnet.net Zone, PTR Records, DNSSEC)"]
AeroStorage["Longhorn Persistent Volume"]
AeroTech["Technitium Instance 1 (Primary Authoritative)<br>(snarfnet.net Zone, Reverse PTR, DNSSEC)"]
AeroStorage["Longhorn Storage Volume"]
end
subgraph SpaceCluster["Spaceport K3s Cluster"]
subgraph SpaceCluster["Spaceport K3s Cluster (VLAN 20)"]
SpaceVIP["MetalLB VIP: 140.44.4.65:53 (UDP/TCP)"]
SpacePihole["Pi-hole Instance 2<br>(Ad-blocking, Caching, FTL rate-limit: 0)"]
SpaceTech["Technitium Instance 2 (Secondary Authoritative)<br>(Zone Transfer AXFR/IXFR from Primary)"]
SpaceStorage["Longhorn Persistent Volume"]
SpaceStorage["Longhorn Storage Volume"]
end
subgraph Upstream["Encrypted Upstream Resolvers"]
DoH["DNS-over-HTTPS (Quad9 / Cloudflare / Mullvad)"]
end
ClientDevices -->|Queries 140.44.4.35| AeroVIP
ClientDevices -.->|Fallback 140.44.4.65| SpaceVIP
Devices -->|DNS Queries (UDP/TCP 53)| DHCP
DHCP -->|Primary| AeroVIP
DHCP -.->|Secondary / Failover| SpaceVIP
AeroVIP --> AeroPihole
SpaceVIP --> SpacePihole
AeroPihole -->|Recursive / Local Lookup| AeroTech
SpacePihole -->|Recursive / Local Lookup| SpaceTech
AeroTech <==>|Automatic Zone Replication (AXFR)| SpaceTech
AeroTech -->|Encrypted Upstream| DoH
SpaceTech -->|Encrypted Upstream| DoH
AeroPihole -->|Recursive / Local Lookups| AeroTech
SpacePihole -->|Recursive / Local Lookups| SpaceTech
AeroTech <==>|Automatic Zone Sync (AXFR)| SpaceTech
AeroTech -->|Encrypted DoH| DoH
SpaceTech -->|Encrypted DoH| DoH
AeroPihole --- AeroStorage
AeroTech --- AeroStorage
SpacePihole --- SpaceStorage
SpaceTech --- SpaceStorage
```
### 2.1 Solving the Historical Rate-Limiting & Performance Bottleneck
Your earlier Raspberry Pi Unbound/Pi-hole suffered when local devices (especially smart home polling, K8s CoreDNS forwards, and mobile apps) sent bursts of queries, tripping Pi-hole FTL's default rate-limit (`1000 queries in 60s per client`) and saturating single-threaded Unbound.
**How our cluster architecture resolves this:**
1. **Configurable Rate Limiting:**
- Set `FTLCONF_RATE_LIMIT=0` (or `10000/60` for WAN only) in the Pi-hole configuration so internal bursts are never blocked.
2. **Technitium Engine vs. Unbound:**
- Technitium is built on .NET 8/9 with an asynchronous, high-throughput DNS engine capable of handling tens of thousands of QPS without degrading.
- Built-in caching with negative caching and aggressive prefetching ensures queries are answered from RAM in `<0.5ms`.
3. **Clustered Failover:**
- MetalLB VIP on Aerodrome (`140.44.4.35`) and Spaceport (`140.44.4.65`) guarantees high availability: if an entire Kubernetes node or cluster is undergoing maintenance, clients seamlessly failover to the secondary VIP without DNS interruption.
4. **Zone Synchronization:**
- Aerodrome acts as the primary Technitium DNS server.
- Spaceport acts as the secondary Technitium DNS server, replicating zones via standard DNS NOTIFY / AXFR transfers automatically.
### 2.1 Addressing the Rate-Limiting & Sluggishness Issue
1. **FTL Rate Limit Bypass:** Set `FTLCONF_RATE_LIMIT=0` so local bursts (such as 53 Smart Home devices polling simultaneously) are never throttled.
2. **Technitium Engine:** Replaces single-threaded Unbound with a high-performance .NET 9 asynchronous resolver capable of servicing tens of thousands of queries per second with sub-millisecond RAM caching.
3. **High Availability:** MetalLB VIPs on Aerodrome (`140.44.4.35`) and Spaceport (`140.44.4.65`) allow continuous service even during node reboots or maintenance.
4. **Authoritative Split-Horizon:** Technitium will authoritatively host the internal `snarfnet.net` forward zone and reverse PTR zones (`4.44.140.in-addr.arpa`, `4.4.14.in-addr.arpa`, `10.10.14.in-addr.arpa`), eliminating slow mDNS or external lookup delays.
---
## 3. Network Security & Honeypot Architecture
## 3. Network Honeypot & UDM Pro Firewall Hardening
```mermaid
flowchart LR
subgraph VLANs["Network VLAN Segments (UDM Pro)"]
VLAN20["VLAN 20: Infra / Management<br>(140.44.4.0/24)"]
VLAN40["VLAN 40: Storage<br>(14.10.10.0/24)"]
VLAN50["VLAN 50: Smart Home / IoT<br>(14.4.4.0/24)"]
VLAN90["VLAN 90: Isolated DMZ / Honeypot<br>(14.99.99.0/24)"]
VLAN80["VLAN 80: Guest Network<br>(14.80.80.0/24)"]
subgraph WAN_Threats["External & Lateral Threats"]
Internet["Public Internet"]
CompromisedIoT["Compromised IoT / Host (e.g. 14.4.4.15)"]
end
subgraph UDM_FW["UDM Pro Firewall Rules"]
DropInterVLAN["Block Inter-VLAN Routing"]
AllowEstablished["Allow Established & Related"]
AllowDNS["Allow Port 53 to DNS VIPs"]
subgraph UDM_FW["UDM Pro Firewall Rules (LAN IN)"]
Rule1["Rule 1: Allow Established & Related"]
Rule2["Rule 2: Allow DNS (Port 53 to .35 & .65)"]
Rule3["Rule 3: Allow NTP (Port 123 to .11 & .44)"]
Rule4["Rule 4: Drop IoT -> Compute (VLAN 50 -> 20)"]
Rule5["Rule 5: Drop IoT -> Storage (VLAN 50 -> 40)"]
Rule6["Rule 6: Drop Media -> Compute (VLAN 30 -> 20)"]
end
subgraph SecurityCluster["Security Monitoring Services"]
Honeypot["OpenCanary / Cowrie Honeypot<br>(Simulating SSH, HTTP, SMB, RDP)"]
GotifyAlert["Gotify Alerting Bridge<br>(Instant Mobile Notification)"]
ZabbixLog["Zabbix / Syslog Telemetry"]
subgraph HoneypotNet["Honeypot Decoy (VLAN 50 / Decoy IP 14.4.4.99)"]
OpenCanary["OpenCanary Honeypot<br>(Simulating SSH:22, HTTP:80, SMB:445, RDP:3389)"]
end
VLAN50 -.->|Blocked from accessing| VLAN20
VLAN80 -.->|Blocked from accessing| VLAN20
VLAN80 -.->|Blocked from accessing| VLAN40
VLAN50 -->|Allowed only port 53| DropInterVLAN
VLAN90 --> Honeypot
Honeypot -->|Triggered Event| GotifyAlert
Honeypot -->|Audit Event| ZabbixLog
subgraph AlertSystem["Alerting & Audit Pipeline"]
Gotify["Gotify Push Notification<br>(Instant alert to your phone)"]
Zabbix["Zabbix Security Audit Log"]
end
CompromisedIoT -.->|Probes decoy IP| OpenCanary
OpenCanary -->|High Priority Webhook| Gotify
OpenCanary -->|Syslog Event| Zabbix
CompromisedIoT -->|Blocked by Rule 4| UDM_FW
```
### 3.1 Honeypot Implementation: OpenCanary / Cowrie
* **OpenCanary:**
* Low-interaction, highly modular honeypot that exposes simulated services:
* Port 22 (SSH fake login banner)
* Port 80 / 8080 (Fake router admin login)
* Port 445 (Fake SMB share)
* Port 3389 (Fake RDP prompt)
* Port 21 (Fake FTP)
* Any packet or connection attempt to these ports triggers an immediate high-priority alert via webhook to Gotify and logs to syslog/Zabbix.
* **Placement Options:**
* Option A: Deployed on a dedicated honeypot IP in the IoT/Guest subnet to detect lateral movement.
* Option B: Containerized pod in Spaceport with hostPort binding to a decoy IP.
### 3.2 UDM Pro Firewall Rules Matrix
| Rule Name | Direction | Source | Destination | Action | Purpose |
|---|---|---|---|:---:|---|
| `Allow-Established-Related` | `LAN IN` | Any | Any | **ACCEPT** | Allows responses to existing connections |
| `Allow-DNS-Traffic` | `LAN IN` | All VLANs | `140.44.4.35`, `140.44.4.65` (port 53) | **ACCEPT** | Allows all devices to query the DNS stack |
| `Allow-NTP-Traffic` | `LAN IN` | All VLANs | `140.44.4.11`, `14.4.4.44` (port 123) | **ACCEPT** | Allows Chrony time synchronization |
| `Isolate-IoT-to-Infra` | `LAN IN` | `VLAN 50 (SmartHome)` | `VLAN 20 (Infra)`, `VLAN 40 (Storage)` | **DROP** | Prevents compromised IoT devices from pivoting into servers |
| `Isolate-Guest-Network` | `GUEST IN` | `VLAN 80 (Guest)` | RFC1918 / Private Subnets | **DROP** | Isolates guest devices to Internet-only |
| `Isolate-Storage-VLAN` | `LAN IN` | Non-Storage Subnets | `VLAN 40 (14.10.10.0/24)` | **DROP** | Restricts Synology NFS/iSCSI to hypervisors and K8s storage nodes |
### 3.1 Network Honeypot Deployment
* **Tool:** **OpenCanary** (lightweight, zero false-positives).
* **Placement:**
* Assigned a decoy IP on the Smart Home subnet (e.g., `14.4.4.99`) where the gaming port forwards exist.
* Emulates vulnerable services: SSH (`22`), Telnet (`23`), Web admin (`80`/`8080`), SMB (`445`), and RDP (`3389`).
* **Alerting:**
* When any internal device attempts a port scan or lateral connection to `14.4.4.99`, OpenCanary immediately sends a critical push notification to **Gotify** (`gotify.snarfnet.net`) and generates an alert in **Zabbix**.
---
## 4. Phased Execution Roadmap
### 3.2 UDM Pro Firewall Rules Matrix (LAN IN)
### Phase 1: Clean Up Legacy Stubs
- Delete broken deployments, services, and lingering pods in namespace `infra` (Aerodrome) and `infrastructure` (Spaceport).
- Verify MetalLB IP pool allocations:
- Allocate `140.44.4.35` on Aerodrome for Primary DNS.
- Allocate `140.44.4.65` on Spaceport for Secondary DNS.
We will define firewall groups on the UDM Pro:
* **RFC1918_Subnets:** All internal private ranges (`140.44.0.0/16`, `14.0.0.0/8`, `192.168.0.0/16`).
* **K8s_DNS_VIPs:** `140.44.4.35`, `140.44.4.65`.
* **NTP_Servers:** `140.44.4.11`, `14.4.4.44`.
### Phase 2: Deploy Pi-hole + Technitium on Aerodrome (Primary Stack)
- Create Longhorn PVCs for Pi-hole config (`etc-pihole`, `etc-dnsmasq`) and Technitium config (`technitium-data`).
- Deploy Technitium DNS container (port 5353 internally).
- Deploy Pi-hole container (port 53 externally via MetalLB `140.44.4.35`, upstream pointing to Technitium `127.0.0.1:5353`).
- Configure `FTLCONF_RATE_LIMIT=0` and subscribe to standard curated blocklists.
- Configure `snarfnet.net` forward and reverse zones in Technitium.
| Order | Rule Name | Type | Source | Destination | Action | Purpose |
|:---:|---|:---:|---|---|:---:|---|
| **2001** | `Allow Established/Related` | LAN IN | Any | Any (State: Established, Related) | **ACCEPT** | Essential return traffic |
| **2002** | `Allow Cluster DNS` | LAN IN | All VLANs | `K8s_DNS_VIPs` (Port: 53 UDP/TCP) | **ACCEPT** | Clients resolve DNS |
| **2003** | `Allow Cluster NTP` | LAN IN | All VLANs | `NTP_Servers` (Port: 123 UDP) | **ACCEPT** | Chrony time sync |
| **2004** | `Allow Trusted to All` | LAN IN | `VLAN 1 (LAN)`, `VLAN 20 (Compute)` | RFC1918_Subnets | **ACCEPT** | Admins & servers access all VLANs |
| **2005** | `Isolate SmartHome to Compute` | LAN IN | `VLAN 50 (Smart Home)` | `VLAN 20 (Compute)` | **DROP** | Block IoT pivoting into K8s |
| **2006** | `Isolate SmartHome to Storage` | LAN IN | `VLAN 50 (Smart Home)` | `VLAN 40 (Storage)` | **DROP** | Protect Synology NAS from IoT |
| **2007** | `Isolate Media to Compute` | LAN IN | `VLAN 30 (Media)` | `VLAN 20 (Compute)` | **DROP** | Block media devices from cluster |
| **2008** | `Isolate Guest Network` | GUEST IN | Guest Network | RFC1918_Subnets | **DROP** | Restrict guest traffic to Internet only |
### Phase 3: Deploy Pi-hole + Technitium on Spaceport (Secondary Stack)
- Deploy identical stack on Spaceport bound to MetalLB VIP `140.44.4.65`.
- Establish Technitium zone replication between Primary (Aerodrome) and Secondary (Spaceport).
- Test dual-stack failover by querying both VIPs independently.
---
### Phase 4: Network Honeypot & Alerting Setup
- Deploy OpenCanary / Cowrie with active listeners on decoy ports.
- Configure Gotify webhook integration for instant mobile push notifications upon probe detection.
## 4. Phased Implementation Roadmap
### Phase 5: UDM Pro Network Firewall & VLAN Policy Enforcement
- Configure DHCP on UDM Pro to distribute `140.44.4.35` and `140.44.4.65` across all VLANs.
- Apply and verify LAN IN / GUEST IN firewall rules on UDM Pro.
- Apply Kubernetes `NetworkPolicy` objects for cluster namespace segmentation.
### Phase 1: Teardown Lingering Stubs
* Cleanly delete orphaned, broken deployments and services in `infra` (Aerodrome) and `infrastructure` (Spaceport):
* Remove `deployment/pihole`, `deployment/technitium-dns`, and non-functional NodePort services.
### Phase 2: Deploy Dual-Cluster Pi-hole + Technitium
* Create Longhorn storage volumes on both clusters for persistent configs and blocklists.
* Deploy **Aerodrome Primary Stack**:
* Technitium on internal port `5353`.
* Pi-hole with MetalLB LoadBalancer VIP **`140.44.4.35:53`** (UDP & TCP).
* Deploy **Spaceport Secondary Stack**:
* Technitium on internal port `5353` configured for AXFR zone transfers from Aerodrome.
* Pi-hole with MetalLB LoadBalancer VIP **`140.44.4.65:53`** (UDP & TCP).
* Populate `snarfnet.net` local records and reverse PTR zones.
### Phase 3: Network Honeypot Deployment
* Deploy containerized OpenCanary with virtual listener on `14.4.4.99`.
* Configure webhook bridge to Gotify server (`140.44.4.70`) for immediate push notification upon any probe.
### Phase 4: UDM Pro Network Configuration & Policy Hardening
* Configure DHCP Scope DNS servers on UDM Pro for all VLANs:
* Primary: `140.44.4.35`
* Secondary: `140.44.4.65`
* Set UDM Pro WAN DNS to `140.44.4.35` & `140.44.4.65` (or Quad9 fallback).
* Apply LAN IN firewall rule matrix on UDM Pro to enforce strict VLAN isolation.