From 08c6c76d890fd585d098758c4f7382dc2609acfb Mon Sep 17 00:00:00 2001 From: Snarf Date: Sat, 31 Oct 2026 22:16:38 -0400 Subject: [PATCH] docs: add homelab topology and infrastructure improvement documentation with hardening script --- docs/homelab_topology.md | 8 +- docs/infrastructure_improvement_plan.md | 156 ++++++++++++++++++++---- scripts/harden_online_rpi_boot.py | 11 +- 3 files changed, 140 insertions(+), 35 deletions(-) diff --git a/docs/homelab_topology.md b/docs/homelab_topology.md index 10566f6..8c751b0 100644 --- a/docs/homelab_topology.md +++ b/docs/homelab_topology.md @@ -92,10 +92,10 @@ All Kubernetes resources are hosted on the **Compute VLAN (VLAN 20 — `140.44.4 * **IP Allocation:** * `Shipyard VIP` $\rightarrow$ `140.44.4.70` (Keepalived Virtual IP) * `Shipyard 01` $\rightarrow$ `140.44.4.71` (`snshpd4lnxswm01`, Manager Node, arm64 RPi 5, Ready) - * `Shipyard 02` $\rightarrow$ `140.44.4.72` (`snshpd4lnxswm02`, Worker Node, arm64 RPi 5, Ready) - * `Shipyard 03` $\rightarrow$ `140.44.4.73` (`snshpd4lnxswm03`, Worker Node, arm64 RPi 5, **Down/Offline**) - * `Shipyard 04` $\rightarrow$ `140.44.4.74` (`snshpd4lnxswm04`, Worker Node / NPM Host, arm64 RPi 5, Ready) - * `Shipyard 05` $\rightarrow$ `140.44.4.75` (`snshpd4lnxswm05`, Worker Node, arm64 RPi 5, **Down/Offline**) + * `Shipyard 02` $\rightarrow$ `140.44.4.72` (`snshpd4lnxswm02`, Manager Node, arm64 RPi 5, Ready) + * `Shipyard 03` $\rightarrow$ `140.44.4.73` (`snshpd4lnxswm03`, Manager Node, arm64 RPi 5, Ready) + * `Shipyard 04` $\rightarrow$ `140.44.4.74` (`snshpd4lnxswm04`, Manager Node / NPM Host, arm64 RPi 5, Ready) + * `Shipyard 05` $\rightarrow$ `140.44.4.75` (`snshpd4lnxswm05`, Manager Node, arm64 RPi 5, Ready) --- diff --git a/docs/infrastructure_improvement_plan.md b/docs/infrastructure_improvement_plan.md index d07e5ac..b862a62 100644 --- a/docs/infrastructure_improvement_plan.md +++ b/docs/infrastructure_improvement_plan.md @@ -6,25 +6,28 @@ This master architectural plan defines the prioritized roadmap, implementation b ## 1. Executive Summary & Engineering Prioritization -Following Site Reliability Engineering (SRE) best practices, the eight infrastructure initiatives and the power resilience architecture are ordered into a **6-Phase Dependency Graph**: +Following Site Reliability Engineering (SRE) best practices, the nine infrastructure initiatives and the power resilience architecture are ordered into a **7-Phase Dependency Graph**: 1. **Phase 1 — Cryptographic Access & Secret Foundation (Item 8):** Unify SSH key access (`id_ed25519_snarfnet`) across all hosts and secure private keys inside AWX, Semaphore, and GitOps (SOPS/Age). *Prerequisite for all non-interactive automation, patching, and UPS shutdown scripts.* -2. **Phase 2 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3):** Resolve dual-default StorageClasses, clear stuck `Terminating` PVCs, add Longhorn recurring snapshots on `spaceport`, configure Synology NAS (`14.10.10.71`) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. *Prerequisite before upgrading kernels or control planes.* -3. **Phase 3 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.* -4. **Phase 4 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates. -5. **Phase 5 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using `iac-reverse` Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD ApplicationSets (`spaceport` & `aerodrome`) for automated drift remediation. -6. **Phase 6 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment. +2. **Phase 2 — Centralized Internal NTP Time Synchronization Architecture (Item 9):** Deploy a hardened, high-availability internal Chrony NTP server syncing against diverse external Stratum 1/2 pools (Cloudflare, Google, NIST, pool.ntp.org) with local stratum fallback. Reconfigure all 31+ nodes (`systemd-timesyncd` / `chrony`) to pull time from the internal server. *Prevents Raft etcd election splits, Docker Swarm TLS cert expiration, Authentik JWT rejections, and Raspberry Pi post-outage clock drift.* +3. **Phase 3 — Persistent Storage Stabilization & Off-Cluster Backups (Item 3):** Resolve dual-default StorageClasses, clear stuck `Terminating` PVCs, add Longhorn recurring snapshots on `spaceport`, configure Synology NAS (`14.10.10.71`) as the NFSv4 BackupTarget for all Longhorn clusters, and migrate Docker Swarm off degraded GlusterFS bricks to hardened NFSv4 + atomic SQLite backup sidecars. *Prerequisite before upgrading kernels or control planes.* +4. **Phase 4 — Observability, Sensors & Multi-Channel Alerting (Items 1, 2, & 7):** Configure Gotify broadcast channels, deploy the Gotify/Zabbix/Home Assistant $\rightarrow$ Discord forwarding pipeline, enable Zabbix problem actions, and complete Zabbix Agent 2 + SNMP + Kubernetes sensor coverage across all 30+ nodes. *Ensures full visibility before running automated upgrades.* +5. **Phase 5 — AWX Configuration & Automated Infrastructure Patching (Items 5 & 6):** Remediate the critical Kubernetes control-plane version skew on `aerodrome` (`v1.26.10` Masters vs. `v1.30.6` Workers) and configure AWX Projects, Credentials, Inventories, Job Templates, and recurring Schedules for rolling OS, K3s, and container updates. +6. **Phase 6 — Declarative IaC & GitOps Continuity (Item 4):** Codify live infrastructure into local Gitea (`git.snarfnet.net`) using `iac-reverse` Terraform plans (Harvester, Authentik, Docker Swarm, Synology) and ArgoCD ApplicationSets (`spaceport` & `aerodrome`) for automated drift remediation. +7. **Phase 7 — Power Resilience & Automated UPS Shutdown (Core Resilience Pillar):** Execute the 6-tier graceful shutdown and bottom-up boot recovery orchestrated by the NUT Master (`14.4.4.44`), utilizing the unified SSH key, direct internal Gotify/Discord alerts, NFS-safe Swarm scaling, and clean Longhorn PVC detachment. ```mermaid flowchart LR - P1["Phase 1: Unified SSH Key\n& Secret Vault (Item 8)"] --> P2["Phase 2: Storage & NFS\nStabilization (Item 3)"] - P1 --> P3["Phase 3: Gotify, Discord &\nZabbix Sensors (Items 1, 2, 7)"] - P2 --> P4["Phase 4: AWX & Rolling\nInfra Upgrades (Items 5, 6)"] - P3 --> P4 - P4 --> P5["Phase 5: Terraform &\nArgoCD GitOps (Item 4)"] - P1 --> P6["Phase 6: Power Resilience\n& UPS Shutdown"] - P2 --> P6 - P3 --> P6 + P1["Phase 1: Unified SSH Key\n& Secret Vault (Item 8)"] --> P2["Phase 2: Central Internal\nNTP Sync Server (Item 9)"] + P2 --> P3["Phase 3: Storage & NFS\nStabilization (Item 3)"] + P2 --> P4["Phase 4: Gotify, Discord &\nZabbix Sensors (Items 1, 2, 7)"] + P3 --> P5["Phase 5: AWX & Rolling\nInfra Upgrades (Items 5, 6)"] + P4 --> P5 + P5 --> P6["Phase 6: Terraform &\nArgoCD GitOps (Item 4)"] + P1 --> P7["Phase 7: Power Resilience\n& UPS Shutdown"] + P2 --> P7 + P3 --> P7 + P4 --> P7 ``` --- @@ -38,7 +41,7 @@ Our automated live probe across the homelab APIs identified the following baseli | **Aerodrome K3s (`local`)** | Masters (`ctl01-03`): `v1.26.10+k3s1`
Workers (`wrk01-07`): `v1.30.6+k3s1` | **Critical Version Skew:** Masters are 4 minor versions behind workers (`wrk01` & `wrk07` `Ready=Unknown`). Dual default StorageClasses (`local-path` & `longhorn`). 2 PVCs stuck `Terminating` (`pihole`, `technitium`). Longhorn `backup-target` is empty (`''`). | | **Spaceport K3s (`c-m-4fgkl75b`)** | 4 Masters + 6 Workers: `v1.31.4+k3s1` (All 10 `Ready`) | Dual default StorageClasses (`local-path` & `longhorn`). 1 PVC stuck `Terminating` (`technitium-longhorn-pvc`). **Zero Longhorn recurring snapshot jobs (`[]`)** and empty `backup-target` (`''`) despite hosting Gitea, Zabbix Postgres, Jenkins, and Semaphore DBs. | | **Spacedock Harvester (`c-m-xgrxqfxh`)** | 3 Baremetal Nodes (`snaihvs0lnx01-03`): Harvester `v1.8.0` / `v1.35.2+rke2r1` | Healthy `harvester-longhorn` default StorageClass. 2 active PVCs in `snarfai` (`longhorn-pvc-20tb`, `open-webui-pvc`). Not yet monitored in Zabbix. | -| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); `.73` & `.75` offline | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. Legacy stacks (`gotify`, `homebox`, `peanut`, `pihole`) use `/mnt/swarm_shared` (GlusterFS across `.71-.73`), which is degraded when `.73` is offline. | +| **Shipyard Docker Swarm** | 5 Nodes (`140.44.4.71-75`); all 5 online and `Ready` | NPM pinned in `PublishMode: host` on `.74` with daily NAS backup. GlusterFS (`/mnt/swarm_shared`) and NFS (`/mnt/nfs_shares`) healthy with active peers across all nodes. Hardened with `usb_max_current_enable=1`, `tune2fs -e continue`, and `nofail`. | | **Zabbix (`zabbix.snarfnet.net`)** | Version `7.0.16` (`140.44.4.57:8888`, Trapper `140.44.4.61:10051`) | 22 hosts registered. **Missing:** Spaceport masters (`ctl01-04`), Aerodrome `wrk05`/`wrk07`, Harvester nodes (`.81-.83`), Synology NAS (`.71`/`.72`), AI Pis (`.7`/`.8`), UDM Pro (`.1`), Home Assistant (`.20`). **Enabled MediaTypes: `[]` (None)**; **Problem Action: Disabled (`status=1`)**. | | **Gotify (`gotify.snarfnet.net`)** | Version `2.9.1` (`140.44.4.70:8090`) | Healthy behind Authentik Forward Auth externally; direct internal VIP (`http://140.44.4.70:8090`) reachable for low-level webhooks. | | **AWX (`awx.snarfnet.net`)** | Operator `2.19.1` / AWX `24.6.1` (`aerodrome`) | Running in `awx` namespace (`awx-web`, `awx-task`, `awx-postgres-15-0`). Ready for credential, inventory, project, and schedule provisioning via `awx-manage` / REST API. | @@ -70,11 +73,122 @@ Our automated live probe across the homelab APIs identified the following baseli - **Roadblock 1.2 — Synology DSM Home Directory Permissions:** Synology DSM (`14.10.10.71` & `.72`) rejects SSH key login if `/var/services/homes/admin` has `777` or `775` permissions (default DSM ACL behavior). - *Mitigation:* Run `chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys && chmod 755 ~` on both NAS appliances during enrollment. - **Roadblock 1.3 — Currently Offline Nodes (`140.44.4.21`, `.25`/`.27`, `.73`, `.75`):** - - *Mitigation:* Enrollment playbook records unreachable nodes in a retry queue and AWX automatically applies the key via a post-boot callback or scheduled convergence job once they come online. + - *Mitigation:* Enrollment playbook records unreachable nodes in a retry queue and AWX automatically applies the key via a post-boot callback or scheduled convergence job once they come online. *(Resolved: All nodes are now 100% online and ready for key enrollment).* --- -### Phase 2: Persistent Storage Stabilization & NAS NFS Architecture (Item 3) +### Phase 2: Centralized Internal NTP Time Synchronization Architecture (Item 9) + +#### Architecture & Implementation Steps + +```mermaid +flowchart TD + subgraph ExteriorNTP ["Public Exterior Stratum 1 & 2 NTP Sources"] + NIST["NIST Atomic Clock\n(time.nist.gov)"] + CF["Cloudflare Anycast NTP\n(time.cloudflare.com)"] + GOOG["Google Public NTP\n(time.google.com)"] + POOL["NTP Pool Project\n(0.us.pool.ntp.org / 1.us.pool.ntp.org)"] + end + + subgraph InternalNTP ["Central Internal Time Authority"] + NTP_PRI["Primary Internal NTP Server\nUDM-Pro Gateway (140.44.4.1 / 14.44.4.1)\nChrony Engine (Port UDP 123)"] + NTP_SEC["Secondary Redundant NTP Server\naerodrome-ctl01 (140.44.4.11) / nutserver (14.4.4.44)\nChrony Engine (Port UDP 123)"] + end + + subgraph InternalClients ["Homelab Infrastructure Clients (31+ Hosts)"] + K8S_A["aerodrome K3s (10 Nodes)\n140.44.4.11-13, 21-27"] + K8S_S["spaceport K3s (10 Nodes)\n140.44.4.41-44, 51-56"] + HVS["spacedock Harvester (3 Nodes)\n140.44.4.81-83"] + SWM["shipyard Swarm (5 Nodes)\n140.44.4.71-75"] + NAS["Synology NAS (2 Appliances)\n14.10.10.71, 72"] + SMART["Smart Home & AI Pis (4 Hosts)\n14.4.4.20, 14.4.4.44, 140.44.4.7-8"] + end + + NIST --> NTP_PRI + CF --> NTP_PRI + GOOG --> NTP_PRI + POOL --> NTP_PRI + + NIST --> NTP_SEC + CF --> NTP_SEC + GOOG --> NTP_SEC + POOL --> NTP_SEC + + NTP_PRI <-->|"Peer Exchange"| NTP_SEC + + NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| K8S_A + NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| K8S_S + NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| HVS + NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| SWM + NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| NAS + NTP_PRI -->|"Sub-millisecond NTP Broadcast / Unicast"| SMART + + NTP_SEC -.->|"Automatic Failover"| InternalClients +``` + +1. **Strategic Necessity for Homelab High Availability:** + - **Raft & etcd Consensus:** Both K3s clusters (`aerodrome` and `spaceport`), Docker Swarm (`shipyard`), and Harvester RKE2 require Raft heartbeat stability; clock skew $> 500\text{ms}$ triggers spurious leader elections, split-brain states, and quorum loss. + - **Docker Swarm & Kubernetes TLS Node Certificates:** Docker Swarm and K3s node TLS certificates have finite validity periods (e.g. 90-day Swarm certificates). A desynchronized clock causes immediate certificate expiration errors and daemon failures upon boot. + - **Authentik SSO & Token Lifetimes:** Authentik OAuth2, OIDC, and SAML assertions rely on strict `nbf` (not before) and `exp` (expires at) timestamp validation; clock skew $> 60\text{s}$ leads to total login authentication rejection across all homelab dashboards. + - **Lack of Hardware Real-Time Clocks (RTC) on Raspberry Pis:** All 24 Raspberry Pi nodes lack battery-backed hardware RTCs. When power returns following an outage, Pi clocks revert to their last filesystem unmount timestamp or 1970 until network time sync succeeds. + +2. **Central Internal Chrony Server Deployment:** + - **Primary Server:** Configure Chrony on the central routing authority (`140.44.4.1` on UDM-Pro / Gateway), or deploy an authoritative dual-homed Chrony service on `aerodrome-ctl01` (`140.44.4.11`) and `nutserver` (`14.4.4.44`). + - **Diverse Exterior Stratum 1 & 2 Sources:** + ```chrony + server time.cloudflare.com iburst minpoll 4 maxpoll 8 + server time.google.com iburst minpoll 4 maxpoll 8 + server time.nist.gov iburst minpoll 4 maxpoll 8 + server 0.us.pool.ntp.org iburst minpoll 4 maxpoll 8 + server 1.us.pool.ntp.org iburst minpoll 4 maxpoll 8 + ``` + - **Orphan Mode & Local Stratum Fallback:** + ```chrony + local stratum 10 + allow 140.44.4.0/24 + allow 14.4.4.0/24 + allow 14.10.10.0/24 + allow 140.44.44.0/24 + allow 14.44.4.0/24 + ``` + *Benefit:* If an extended storm or ISP fiber cut takes the WAN offline, the internal Chrony server maintains local stratum 10 authority, locking all 31+ homelab hosts to identical drift-free time with zero inter-cluster divergence. + +3. **Homelab-Wide Client Rollout (All 31+ Nodes):** + - **Debian / Ubuntu / Raspberry Pi OS Clients (`systemd-timesyncd`):** + Deploy standardized `/etc/systemd/timesyncd.conf` via Ansible: + ```ini + [Time] + NTP=140.44.4.1 140.44.4.11 + FallbackNTP=time.cloudflare.com time.google.com 0.us.pool.ntp.org + PollIntervalMinSec=32 + PollIntervalMaxSec=512 + ``` + Execute `systemctl restart systemd-timesyncd && timedatectl set-ntp true`. + - **DietPi Nodes (`spaceport-ctl01-04`, `spaceport-wrk01-04, 06`):** + Configure `/etc/systemd/timesyncd.conf` or `/etc/chrony/chrony.conf` to prioritize the internal server, disabling DietPi's delayed hourly cron sync in favor of persistent daemon-based synchronization. + - **Harvester HCI Nodes (`spacedock` `140.44.4.81-83`):** + Inject the internal NTP server configuration via Harvester's cloud-init OEM layer (`/oem/99_custom.yaml` or Harvester NTP settings UI), ensuring hypervisor node clocks match K8s downstreams. + - **Synology DSM (`14.10.10.71` & `14.10.10.72`):** + Set Regional Options $\rightarrow$ NTP Server to `140.44.4.1` with daily synchronization schedule. + +4. **Observability & Clock Drift Alerting in Zabbix:** + - Link the Zabbix `System time synchronization by Zabbix agent` template across all hosts. + - Configure alert triggers: + - **Warning:** `system.localtime` clock drift $> 1.0\text{s}$ against Zabbix Server. + - **High:** `system.localtime` clock drift $> 5.0\text{s}$ (critical risk to Raft consensus). + - **High:** `systemd-timesyncd.service` or `chronyd.service` in failed state. + +#### Predicted Roadblocks & Mitigations +- **Roadblock 2.1 — Negative Time Jumps on Raspberry Pi Boot (systemd-timesyncd vs fake-hwclock):** + If `fake-hwclock` restores a timestamp from the future (e.g. from an unclean shutdown), `systemd-timesyncd` can refuse to jump backward without a hard step, or a backward step can corrupt etcd WAL logs. + - *Mitigation:* Ensure `systemd-timesyncd` is configured with `timedatectl set-ntp true` and that the systemd service starts before clustering engines (K3s, Docker Swarm), stepping time forward cleanly at boot. +- **Roadblock 2.2 — VLAN Segmentation Blocking UDP Port 123:** + Nodes reside across VLAN 1, 2, 20, 40, and 50. If UniFi firewall rules drop inter-VLAN UDP 123, cross-VLAN nodes (e.g. Synology NAS on VLAN 40, Home Assistant on VLAN 50) fail to sync. + - *Mitigation:* Enable a stateful UniFi firewall rule allowing `LAN IN: Any $\rightarrow$ UDP 123` to the internal NTP server IP, or serve NTP natively on each VLAN gateway interface on the UDM-Pro (`140.44.4.1`, `14.4.4.1`, `14.10.10.1`, `140.44.44.1`). + +--- + +### Phase 3: Persistent Storage Stabilization & NAS NFS Architecture (Item 3) #### Architecture & Implementation Steps 1. **Kubernetes StorageClass & Stuck PVC Remediation (`aerodrome` & `spaceport`):** @@ -100,7 +214,7 @@ Our automated live probe across the homelab APIs identified the following baseli --- -### Phase 3: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7) +### Phase 4: Notification Broadcasts, Discord Forwarding & Complete Zabbix Monitoring (Items 1, 2, & 7) #### Architecture & Implementation Steps @@ -177,7 +291,7 @@ flowchart TD --- -### Phase 4: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6) +### Phase 5: Infrastructure Upgrades, Ansible Playbooks & AWX Automation (Items 5 & 6) #### Architecture & Implementation Steps 1. **AWX Provisioning (`https://awx.snarfnet.net` on `aerodrome`):** @@ -218,7 +332,7 @@ flowchart TD --- -### Phase 5: Terraform Plans & ArgoCD GitOps Integrity (Item 4) +### Phase 6: Terraform Plans & ArgoCD GitOps Integrity (Item 4) #### Architecture & Implementation Steps 1. **Local GitOps Repository Structure on Gitea (`https://git.snarfnet.net/snarf/homelab-gitops`):** @@ -251,7 +365,7 @@ flowchart TD --- -### Phase 6: Power Resilience & UPS Automation Plan (Updated Core Pillar) +### Phase 7: Power Resilience & UPS Automation Plan (Updated Core Pillar) #### 6.1 Core Power Architecture & IP Assignments diff --git a/scripts/harden_online_rpi_boot.py b/scripts/harden_online_rpi_boot.py index 660aed6..959e761 100644 --- a/scripts/harden_online_rpi_boot.py +++ b/scripts/harden_online_rpi_boot.py @@ -58,11 +58,7 @@ if [[ "$ROOT_DEV" == /dev/* ]]; then echo " [EXT4] $ROOT_DEV -> Errors behavior: $ERR_BEHAV" fi -# 2. Ensure fsck.repair=yes (and usb-storage.quirks=152d:0583:u if JMicron bridge present) in active cmdline.txt files -HAS_JMICRON=0 -if lsusb 2>/dev/null | grep -qi "152d:0583"; then - HAS_JMICRON=1 -fi +# 2. Ensure fsck.repair=yes in active cmdline.txt files for cmdfile in /boot/firmware/cmdline.txt /boot/cmdline.txt; do if [ -f "$cmdfile" ] && ! grep -q "DO NOT EDIT" "$cmdfile"; then if ! grep -q "fsck.repair=yes" "$cmdfile"; then @@ -72,11 +68,6 @@ for cmdfile in /boot/firmware/cmdline.txt /boot/cmdline.txt; do else echo " [CMDLINE] Verified fsck.repair=yes already in $cmdfile" fi - if [ "$HAS_JMICRON" -eq 1 ] && ! grep -q "usb-storage.quirks=152d:0583:u" "$cmdfile"; then - cp "$cmdfile" "${cmdfile}.bak" - sed -i 's/^/usb-storage.quirks=152d:0583:u /' "$cmdfile" - echo " [CMDLINE] Added usb-storage.quirks=152d:0583:u to $cmdfile" - fi fi done