Snarf/homelab-gitops
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add ai-stack Kubernetes manifests for Qdrant, vLLM, and Triton

Browse Source
This commit is contained in:
Snarf
2026-11-08 15:03:24 -05:00
parent be820c6b92
commit 7b2dd24526
1 changed files with 7 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
manifests/spacedock/ai-stack/ai-stack.yaml
View File

@@ -249,6 +249,13 @@ spec:
args: args:
- | - |
rm -rf /usr/local/lib/python3.12/dist-packages/torchcodec* rm -rf /usr/local/lib/python3.12/dist-packages/torchcodec*
python3 -c '
p = "/usr/local/lib/python3.12/dist-packages/vllm/v1/attention/ops/triton_unified_attention.py"
s = open(p).read()
s = s.replace("return 32", "return 16")
s = s.replace("launch_num_stages: int | None = None", "launch_num_stages: int | None = 1")
open(p, "w").write(s)
'
exec vllm serve google/gemma-4-12B-it-qat-w4a16-ct \ exec vllm serve google/gemma-4-12B-it-qat-w4a16-ct \
--served-model-name Gemma-4-12B-it \ --served-model-name Gemma-4-12B-it \
--dtype half \ --dtype half \