Snarf/SnarfCode
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add master infrastructure improvement and resilience plan

Browse Source
This commit is contained in:
Snarf
2026-10-31 22:24:50 -04:00
parent 08c6c76d89
commit d1a3ba988a
1 changed files with 7 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
docs/infrastructure_improvement_plan.md
View File

@@ -57,8 +57,8 @@ Our automated live probe across the homelab APIs identified the following baseli
1. **Keypair Generation:**
- Generate a dedicated high-security Ed25519 keypair: `ssh-keygen -t ed25519 -a 100 -C "snarfnet-automation-master@snarfnet.net" -f id_ed25519_snarfnet`.
2. **Standardized Automation Account (`snarf-admin` / `root`):**
- Bootstrap all 30+ hosts across VLANs 1, 2, 20, 40, and 50 with a single enrollment script that:
- Installs the public key into `/root/.ssh/authorized_keys` and `/home/snarf/.ssh/authorized_keys` (plus `rancher` on Harvester nodes and `admin` on Synology DSM).
- Bootstrap all 28 target hosts across VLANs 1, 20, 40, and 50 (skipping `spacedock` Harvester nodes `140.44.4.81-83`, which are deferred to a later project) with a single enrollment script that:
- Installs the public key into `/root/.ssh/authorized_keys` and `/home/snarf/.ssh/authorized_keys` (plus `admin` on Synology DSM).
- Configures `/etc/sudoers.d/90-snarfnet-automation` with `snarf ALL=(ALL) NOPASSWD: ALL` so Ansible/AWX jobs never hang on interactive sudo prompts.
- Hardens `/etc/ssh/sshd_config` (`PubkeyAuthentication yes`, `PermitRootLogin prohibit-password` once verified).
3. **Secure Key Storage (Zero Plaintext in Git):**
@@ -68,8 +68,8 @@ Our automated live probe across the homelab APIs identified the following baseli
- **GitOps Secret Encryption:** Use **Ansible Vault** (for playbooks in Gitea) and **SOPS + Age** (for Kubernetes secrets in ArgoCD) so the Gitea repository contains zero plaintext credentials.
#### Predicted Roadblocks & Mitigations
- **Roadblock 1.1 — Harvester HCI Immutable Root (`/home/rancher`):** Harvester v1.8.0 uses an elemental immutable filesystem; writing to `/etc/ssh` directly can be wiped on upgrade.
- *Mitigation:* Persist the SSH public key via Harvester's `/oem/99_custom.yaml` cloud-init configuration and the `Harvester Setting -> ssh-keys` CRD in Kubernetes (`spacedock`).
- **Roadblock 1.1 — Harvester HCI Immutable Root (`/home/rancher`):**
- *Status:* Skipped / Deferred. Harvester nodes (`140.44.4.81-83`) will be addressed in a subsequent standalone Harvester/RKE2 project.
- **Roadblock 1.2 — Synology DSM Home Directory Permissions:** Synology DSM (`14.10.10.71` & `.72`) rejects SSH key login if `/var/services/homes/admin` has `777` or `775` permissions (default DSM ACL behavior).
- *Mitigation:* Run `chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys && chmod 755 ~` on both NAS appliances during enrollment.
- **Roadblock 1.3 — Currently Offline Nodes (`140.44.4.21`, `.25`/`.27`, `.73`, `.75`):**
@@ -95,10 +95,9 @@ flowchart TD
NTP_SEC["Secondary Redundant NTP Server\naerodrome-ctl01 (140.44.4.11) / nutserver (14.4.4.44)\nChrony Engine (Port UDP 123)"]
end
subgraph InternalClients ["Homelab Infrastructure Clients (31+ Hosts)"]
subgraph InternalClients ["Homelab Infrastructure Clients (28 Target Hosts)"]
K8S_A["aerodrome K3s (10 Nodes)\n140.44.4.11-13, 21-27"]
K8S_S["spaceport K3s (10 Nodes)\n140.44.4.41-44, 51-56"]
HVS["spacedock Harvester (3 Nodes)\n140.44.4.81-83"]
SWM["shipyard Swarm (5 Nodes)\n140.44.4.71-75"]
NAS["Synology NAS (2 Appliances)\n14.10.10.71, 72"]
SMART["Smart Home & AI Pis (4 Hosts)\n14.4.4.20, 14.4.4.44, 140.44.4.7-8"]
@@ -118,7 +117,6 @@ flowchart TD
NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| K8S_A
NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| K8S_S
NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| HVS
NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| SWM
NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| NAS
NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| SMART
@@ -167,7 +165,8 @@ flowchart TD
- **DietPi Nodes (`spaceport-ctl01-04`, `spaceport-wrk01-04, 06`):**
Configure `/etc/systemd/timesyncd.conf` or `/etc/chrony/chrony.conf` to prioritize the internal server, disabling DietPi's delayed hourly cron sync in favor of persistent daemon-based synchronization.
- **Harvester HCI Nodes (`spacedock` `140.44.4.81-83`):**
Inject the internal NTP server configuration via Harvester's cloud-init OEM layer (`/oem/99_custom.yaml` or Harvester NTP settings UI), ensuring hypervisor node clocks match K8s downstreams.
> [!NOTE]
> **Project Scope Exclusion:** The baremetal Harvester HCI nodes (`spacedock` cluster `140.44.4.81-83`) are explicitly skipped and deferred to a separate subsequent project. All steps target the 28 active hosts across Aerodrome, Spaceport, Shipyard, Synology NAS, AI Pis, and NUT Master.
- **Synology DSM (`14.10.10.71` & `14.10.10.72`):**
Set Regional Options $\rightarrow$ NTP Server to `140.44.4.1` with daily synchronization schedule.