diff --git a/reference_guide.md b/reference_guide.md new file mode 100644 index 0000000..b36fbb8 --- /dev/null +++ b/reference_guide.md @@ -0,0 +1,179 @@ +# Reference Guide: Open WebUI & Hailo-10H NPU Stack + +This document serves as a complete technical reference for the architecture, service definitions, API schemas, and deployment designs implemented to connect **Open WebUI** (on Pi 4) with the **Hailo-10H NPU HAT+** (on Pi 5). + +--- + +## 1. System Architecture Layout + +```mermaid +graph TD + subgraph Host_Pi4 ["Debian 13 Pi 4 (Frontend Host)"] + UI["Open WebUI (Native Python)
Port 8080"] + Bridge["Pipelines Service (uvicorn)
Port 9099
Module: hailo_pipeline.py"] + end + + subgraph Host_Pi5 ["Raspberry Pi 5 (NPU Host)"] + NPU_Server["hailo-ollama Server (oatpp)
Port 8000"] + NPU_HW["Hailo-10H NPU HAT+
(8GB VRAM / 40 TOPS)"] + end + + %% Communications Flow + User -->|Browser HTTP| UI + UI -->|OpenAI Chat Request| Bridge + Bridge -->|Ollama Chat JSON Payload| NPU_Server + NPU_Server -->|Model Loading & Graph Execution| NPU_HW +``` + +--- + +## 2. Host 1: Raspberry Pi 5 (NPU Inference Engine) + +The Pi 5 hosts the physical PCIe Hailo HAT+ hardware and runs the bare-metal, native C++ compiled server `hailo-ollama` powered by the Oatpp framework. + +### Service Definition: `/etc/systemd/system/hailo-ollama.service` +```ini +[Unit] +Description=Hailo-Ollama NPU Server +After=network-online.target + +[Service] +User=root +Group=root +ExecStart=/usr/bin/hailo-ollama serve +Restart=always +RestartSec=3 +Environment="OLLAMA_HOST=0.0.0.0:8000" +Environment="HAILO_MONITOR=1" + +[Install] +WantedBy=multi-user.target +``` + +### Critical Paths & Directories +* **Binary Path**: `/usr/bin/hailo-ollama` +* **Configuration**: `/etc/xdg/hailo-ollama/hailo-ollama.json` +* **Model Manifests (JSON Templates)**: `/usr/share/hailo-ollama/models/manifests/` + * Contains system templates, repetition penalties, stop tokens, and target HEF hashes for models. +* **Model Blob Storage (Downloaded HEF Files)**: `/usr/share/hailo-ollama/models/blob/` + * Downloaded model binaries compiled for the Hailo NPU. Named by their SHA-256 hash. + +--- + +## 3. Host 2: Raspberry Pi 4 (WebUI & Pipelines Bridge) + +The Pi 4 runs a native Python 3.11 environment hosting Open WebUI and the Pipelines server. The custom Pipeline script bridges the API mismatch and translates OpenAI requests to Oatpp. + +### Service Definition: `/etc/systemd/system/pipelines.service` +```ini +[Unit] +Description=Open WebUI Pipelines Bridge +After=network.target + +[Service] +User=root +Group=root +WorkingDirectory=/root/open-webui/pipelines +Environment="PATH=/root/open-webui/venv/bin:/usr/local/bin:/usr/bin:/bin" +Environment="PIPELINES_API_KEY=hailo-bridge" +ExecStart=/root/open-webui/venv/bin/uvicorn main:app --host 0.0.0.0 --port 9099 --workers 1 +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +### Script Location: `/root/open-webui/pipelines/pipelines/hailo_pipeline.py` + +This script implements: +1. **Manifold Pipeline Spec**: Exposes sub-models cleanly to Open WebUI's dropdown. +2. **Namespace Stripping**: Cleans model identifiers prepended by WebUI. +3. **Response Key Hierarchy**: Safely extracts inference text from different key structures (Ollama nested, direct content/response keys, OpenAI choices). +4. **Graceful Streaming Loops**: Handles SSE line-by-line buffers and yields tokens before evaluating completion flags. + +--- + +## 4. API & Communication Schemas (Oatpp) + +The `hailo-ollama` service mimics the Ollama REST API. However, it requires explicit model preparation. + +### 4.1. Pull Model Endpoint (Downloads weights to local NPU blob store) +* **URL**: `POST http://140.44.4.7:8000/api/pull` +* **Request Payload**: + ```json + { + "model": "llama3.2:3b" + } + ``` +* **Behavior**: Downloads the model over the internet and compiles/saves the hardware HEF binary to `/usr/share/hailo-ollama/models/blob/`. If the model is not pulled, `/api/chat` returns a 200 OK with `"error": "model not found"`. + +### 4.2. Chat Inference Endpoint +* **URL**: `POST http://140.44.4.7:8000/api/chat` +* **Request Payload**: + ```json + { + "model": "llama3.2:3b", + "messages": [ + {"role": "user", "content": "Hi"} + ], + "stream": true + } + ``` + +#### Non-Streaming Response (`"stream": false`) +```json +{ + "model": "llama3.2:3b", + "created_at": "2026-07-09T14:00:30.299821983Z", + "message": { + "role": "assistant", + "content": "Hello! How can I help you today?" + }, + "done": true, + "done_reason": "stop", + "total_duration": 22737674048, + "eval_count": 58 +} +``` + +#### Streaming Response Chunks (`"stream": true`) +Sent as a series of JSON strings separated by newlines: +```json +{"model":"llama3.2:3b","message":{"role":"assistant","content":"Why"},"done":false} +{"model":"llama3.2:3b","message":{"role":"assistant","content":" couldn"},"done":false} +... +{"model":"llama3.2:3b","message":{"role":"assistant","content":""},"done":true,"done_reason":"stop"} +``` + +--- + +## 5. Operations & Troubleshooting Commands + +### Tail Service Logs +```bash +# On Pi 4 (WebUI / Pipelines Bridge) +journalctl -u open-webui -f +journalctl -u pipelines -f + +# On Pi 5 (NPU Engine) +journalctl -u hailo-ollama -f +``` + +### Restart Services +```bash +# On Pi 4 +systemctl restart pipelines +systemctl restart open-webui + +# On Pi 5 +systemctl restart hailo-ollama +``` + +### Clean Stuck NPU Resource Locks +If the C++ server hangs while compiling or loading a model, it can lock `/dev/hailo0`. Use `kill -9` to force-release the NPU: +```bash +# On Pi 5 +sudo kill -9 $(pgrep hailo-ollama) +sudo systemctl restart hailo-ollama +```