# Project Blueprint: High-Availability DNS Stack, Network Honeypot & VLAN Security Architecture ## 1. Project Objectives & Current State Assessment ### 1.1 Objectives 1. **Enterprise-Grade DNS Stack:** - Deploy a resilient, scalable, dual-cluster DNS architecture across **Aerodrome** (`local`) and **Spaceport** (`c-m-4fgkl75b`). - Implement **Pi-hole** as the front-end ad/malware filtering and caching engine. - Implement **Technitium DNS Server** as the authoritative local DNS resolver (`snarfnet.net` split-horizon and PTR zones) and recursive DNS-over-HTTPS (DoH) upstream. - Ensure Active-Active / High Availability across both clusters with zero rate-limiting bottlenecks. 2. **Network Security & Honeypot:** - Deploy an automated, containerized honeypot (**OpenCanary** or **Cowrie**) with centralized logging and instant alerts to Gotify / Zabbix. 3. **VLAN Segmentation & Firewall Policy Hardening:** - Audit and enforce inter-VLAN firewall rules on the **UniFi Dream Machine Pro (UDM Pro)** (`140.44.4.1`). - Isolate Smart Home / IoT, Storage, and Guest traffic from the Management/Infrastructure plane. - Apply Kubernetes `NetworkPolicy` resources to restrict east-west namespace traffic. --- ### 1.2 Inspection of Existing Cluster "Stubs" We probed both clusters and discovered why the existing DNS implementations stalled: | Cluster | Component | Current State | Root Cause Failure | Recommendation | |---|---|:---:|---|---| | **Aerodrome** (`140.44.4.11`) | `pihole` (deploy) | `0/1 Pending` | PVC `pihole` does not exist (`FailedScheduling`) | **Delete and rebuild cleanly** | | **Aerodrome** (`140.44.4.11`) | `technitium-dns` (deploy) | `0/1 Pending` | PVC `technitium-longhorn-pvc` does not exist | **Delete and rebuild cleanly** | | **Aerodrome** (`140.44.4.11`) | `pihole-dns-tcp/udp` | NodePort | Mapped to ephemeral high ports (`32655`/`31195`); standard DNS clients cannot query NodePorts | **Replace with Port 53 LoadBalancer VIP** | | **Spaceport** (`140.44.4.41`) | `pihole` (deploy) | `1/1 Running` | Uses unmanaged local storage, exposed via NodePort only | **Delete and replace with unified manifest** | | **Spaceport** (`140.44.4.41`) | `technitium-dns` (deploy) | `0/1 Pending` | PVC `technitium-longhorn-pvc` missing | **Delete and rebuild cleanly** | > [!IMPORTANT] > **Decision: Clean Teardown & Declarative Deployment.** Rather than patching fractured, incomplete stubs, we will prune the orphaned resources in `infra` and `infrastructure` and deploy a unified, declarative GitOps stack with dedicated Longhorn storage volumes and MetalLB VIPs on standard port 53. --- ## 2. High-Availability DNS Architecture (Pi-hole + Technitium) ```mermaid flowchart TD subgraph Clients["LAN Clients across all VLANs (VLAN 20, 40, 50, Guest)"] ClientDevices["Laptops, Phones, Servers, IoT"] end subgraph UDM["UniFi Dream Machine Pro (140.44.4.1)"] DHCP["DHCP Options 6:
Primary DNS: 140.44.4.35 (Aerodrome VIP)
Secondary DNS: 140.44.4.65 (Spaceport VIP)"] end subgraph AeroCluster["Aerodrome K3s Cluster"] AeroVIP["MetalLB VIP: 140.44.4.35:53 (UDP/TCP)"] AeroPihole["Pi-hole Instance 1
(Ad-blocking, Caching, FTL rate-limit: 0)"] AeroTech["Technitium Instance 1 (Primary Authoritative)
(snarfnet.net Zone, PTR Records, DNSSEC)"] AeroStorage["Longhorn Persistent Volume"] end subgraph SpaceCluster["Spaceport K3s Cluster"] SpaceVIP["MetalLB VIP: 140.44.4.65:53 (UDP/TCP)"] SpacePihole["Pi-hole Instance 2
(Ad-blocking, Caching, FTL rate-limit: 0)"] SpaceTech["Technitium Instance 2 (Secondary Authoritative)
(Zone Transfer AXFR/IXFR from Primary)"] SpaceStorage["Longhorn Persistent Volume"] end subgraph Upstream["Encrypted Upstream Resolvers"] DoH["DNS-over-HTTPS (Quad9 / Cloudflare / Mullvad)"] end ClientDevices -->|Queries 140.44.4.35| AeroVIP ClientDevices -.->|Fallback 140.44.4.65| SpaceVIP AeroVIP --> AeroPihole SpaceVIP --> SpacePihole AeroPihole -->|Recursive / Local Lookup| AeroTech SpacePihole -->|Recursive / Local Lookup| SpaceTech AeroTech <==>|Automatic Zone Replication (AXFR)| SpaceTech AeroTech -->|Encrypted Upstream| DoH SpaceTech -->|Encrypted Upstream| DoH AeroPihole --- AeroStorage AeroTech --- AeroStorage SpacePihole --- SpaceStorage SpaceTech --- SpaceStorage ``` ### 2.1 Solving the Historical Rate-Limiting & Performance Bottleneck Your earlier Raspberry Pi Unbound/Pi-hole suffered when local devices (especially smart home polling, K8s CoreDNS forwards, and mobile apps) sent bursts of queries, tripping Pi-hole FTL's default rate-limit (`1000 queries in 60s per client`) and saturating single-threaded Unbound. **How our cluster architecture resolves this:** 1. **Configurable Rate Limiting:** - Set `FTLCONF_RATE_LIMIT=0` (or `10000/60` for WAN only) in the Pi-hole configuration so internal bursts are never blocked. 2. **Technitium Engine vs. Unbound:** - Technitium is built on .NET 8/9 with an asynchronous, high-throughput DNS engine capable of handling tens of thousands of QPS without degrading. - Built-in caching with negative caching and aggressive prefetching ensures queries are answered from RAM in `<0.5ms`. 3. **Clustered Failover:** - MetalLB VIP on Aerodrome (`140.44.4.35`) and Spaceport (`140.44.4.65`) guarantees high availability: if an entire Kubernetes node or cluster is undergoing maintenance, clients seamlessly failover to the secondary VIP without DNS interruption. 4. **Zone Synchronization:** - Aerodrome acts as the primary Technitium DNS server. - Spaceport acts as the secondary Technitium DNS server, replicating zones via standard DNS NOTIFY / AXFR transfers automatically. --- ## 3. Network Security & Honeypot Architecture ```mermaid flowchart LR subgraph VLANs["Network VLAN Segments (UDM Pro)"] VLAN20["VLAN 20: Infra / Management
(140.44.4.0/24)"] VLAN40["VLAN 40: Storage
(14.10.10.0/24)"] VLAN50["VLAN 50: Smart Home / IoT
(14.4.4.0/24)"] VLAN90["VLAN 90: Isolated DMZ / Honeypot
(14.99.99.0/24)"] VLAN80["VLAN 80: Guest Network
(14.80.80.0/24)"] end subgraph UDM_FW["UDM Pro Firewall Rules"] DropInterVLAN["Block Inter-VLAN Routing"] AllowEstablished["Allow Established & Related"] AllowDNS["Allow Port 53 to DNS VIPs"] end subgraph SecurityCluster["Security Monitoring Services"] Honeypot["OpenCanary / Cowrie Honeypot
(Simulating SSH, HTTP, SMB, RDP)"] GotifyAlert["Gotify Alerting Bridge
(Instant Mobile Notification)"] ZabbixLog["Zabbix / Syslog Telemetry"] end VLAN50 -.->|Blocked from accessing| VLAN20 VLAN80 -.->|Blocked from accessing| VLAN20 VLAN80 -.->|Blocked from accessing| VLAN40 VLAN50 -->|Allowed only port 53| DropInterVLAN VLAN90 --> Honeypot Honeypot -->|Triggered Event| GotifyAlert Honeypot -->|Audit Event| ZabbixLog ``` ### 3.1 Honeypot Implementation: OpenCanary / Cowrie * **OpenCanary:** * Low-interaction, highly modular honeypot that exposes simulated services: * Port 22 (SSH fake login banner) * Port 80 / 8080 (Fake router admin login) * Port 445 (Fake SMB share) * Port 3389 (Fake RDP prompt) * Port 21 (Fake FTP) * Any packet or connection attempt to these ports triggers an immediate high-priority alert via webhook to Gotify and logs to syslog/Zabbix. * **Placement Options:** * Option A: Deployed on a dedicated honeypot IP in the IoT/Guest subnet to detect lateral movement. * Option B: Containerized pod in Spaceport with hostPort binding to a decoy IP. ### 3.2 UDM Pro Firewall Rules Matrix | Rule Name | Direction | Source | Destination | Action | Purpose | |---|---|---|---|:---:|---| | `Allow-Established-Related` | `LAN IN` | Any | Any | **ACCEPT** | Allows responses to existing connections | | `Allow-DNS-Traffic` | `LAN IN` | All VLANs | `140.44.4.35`, `140.44.4.65` (port 53) | **ACCEPT** | Allows all devices to query the DNS stack | | `Allow-NTP-Traffic` | `LAN IN` | All VLANs | `140.44.4.11`, `14.4.4.44` (port 123) | **ACCEPT** | Allows Chrony time synchronization | | `Isolate-IoT-to-Infra` | `LAN IN` | `VLAN 50 (SmartHome)` | `VLAN 20 (Infra)`, `VLAN 40 (Storage)` | **DROP** | Prevents compromised IoT devices from pivoting into servers | | `Isolate-Guest-Network` | `GUEST IN` | `VLAN 80 (Guest)` | RFC1918 / Private Subnets | **DROP** | Isolates guest devices to Internet-only | | `Isolate-Storage-VLAN` | `LAN IN` | Non-Storage Subnets | `VLAN 40 (14.10.10.0/24)` | **DROP** | Restricts Synology NFS/iSCSI to hypervisors and K8s storage nodes | --- ## 4. Phased Execution Roadmap ### Phase 1: Clean Up Legacy Stubs - Delete broken deployments, services, and lingering pods in namespace `infra` (Aerodrome) and `infrastructure` (Spaceport). - Verify MetalLB IP pool allocations: - Allocate `140.44.4.35` on Aerodrome for Primary DNS. - Allocate `140.44.4.65` on Spaceport for Secondary DNS. ### Phase 2: Deploy Pi-hole + Technitium on Aerodrome (Primary Stack) - Create Longhorn PVCs for Pi-hole config (`etc-pihole`, `etc-dnsmasq`) and Technitium config (`technitium-data`). - Deploy Technitium DNS container (port 5353 internally). - Deploy Pi-hole container (port 53 externally via MetalLB `140.44.4.35`, upstream pointing to Technitium `127.0.0.1:5353`). - Configure `FTLCONF_RATE_LIMIT=0` and subscribe to standard curated blocklists. - Configure `snarfnet.net` forward and reverse zones in Technitium. ### Phase 3: Deploy Pi-hole + Technitium on Spaceport (Secondary Stack) - Deploy identical stack on Spaceport bound to MetalLB VIP `140.44.4.65`. - Establish Technitium zone replication between Primary (Aerodrome) and Secondary (Spaceport). - Test dual-stack failover by querying both VIPs independently. ### Phase 4: Network Honeypot & Alerting Setup - Deploy OpenCanary / Cowrie with active listeners on decoy ports. - Configure Gotify webhook integration for instant mobile push notifications upon probe detection. ### Phase 5: UDM Pro Network Firewall & VLAN Policy Enforcement - Configure DHCP on UDM Pro to distribute `140.44.4.35` and `140.44.4.65` across all VLANs. - Apply and verify LAN IN / GUEST IN firewall rules on UDM Pro. - Apply Kubernetes `NetworkPolicy` objects for cluster namespace segmentation.