From 20b81926b668179f4ae4731952947fa54ff0053f Mon Sep 17 00:00:00 2001 From: techjarves Date: Sun, 21 Jun 2026 17:00:09 +0530 Subject: [PATCH] add LLM configuration and backend management components with automated setup scripts --- .gitignore | 1 + PLAN.md | 192 ---------- app/frontend/src/components/Settings.jsx | 63 +++- app/frontend/src/services/api.js | 27 ++ linux.sh | 4 +- scripts/llm-backends.json | 26 ++ scripts/serve.cjs | 424 +++++++++++++++++++++-- scripts/setup-llama.ps1 | 28 +- scripts/setup-llama.sh | 40 ++- windows.bat | 4 +- 10 files changed, 575 insertions(+), 234 deletions(-) delete mode 100644 PLAN.md create mode 100644 scripts/llm-backends.json diff --git a/.gitignore b/.gitignore index d2265da..54e4ee9 100644 --- a/.gitignore +++ b/.gitignore @@ -9,6 +9,7 @@ app/llm-backend/ app/tools/ # Generated files +app/config/ app/dist/ app/outputs/ diff --git a/PLAN.md b/PLAN.md deleted file mode 100644 index 401a6e6..0000000 --- a/PLAN.md +++ /dev/null @@ -1,192 +0,0 @@ -# Linux Port Plan for Local AI Image Generator - -> **Plan location**: This plan will be copied to the project root as `PLAN.md` upon approval, per user request. - -> **Implementation Update (post-verification)** -> -> During end-to-end testing, the third-party Linux CUDA binary (`leaxer-ai/leaxer-stable-diffusion`) was found to **segfault during generation** after successfully loading the model. As a result, Linux CUDA support was **removed** from the implementation. Linux NVIDIA GPUs now use the **Vulkan** backend. The setup script also gained a glibc version check, because the official leejet Linux binaries require **glibc 2.38+** (Ubuntu 24.04). The frontend's misleading "mock SVG" fallback was replaced with a real error message when the backend crashes. See the updated README for build-from-source instructions. - -## 1. What This Project Does - -**Local AI Image Generator** is a zero-configuration, portable desktop environment for running Stable Diffusion (Safetensors / GGUF / CKPT) completely offline. The Windows flow is: - -1. **Entry Point**: `start.bat` double-click launcher. -2. **Setup**: `scripts/setup.ps1` auto-downloads portable Node.js for Windows, detects the GPU vendor, downloads the matching stable-diffusion.cpp backend (CUDA for NVIDIA, Vulkan for AMD/Intel), installs npm dependencies, and builds the Vite+React frontend into `app/dist/`. -3. **Runtime**: `scripts/serve.cjs` (a Node.js HTTP server) serves the static frontend on `localhost:1420`, manages the backend process lifecycle, proxies generation requests, handles model downloads/imports, tracks telemetry, and manages the gallery. -4. **All sandboxed**: Everything lives under the project folder (`app/tools/`, `app/backend/`, `app/models/`, `app/outputs/`). No global PATH changes, no system installers. - -## 2. Research Findings - -### 2.1 stable-diffusion.cpp Linux Binaries - -We researched the upstream project (`leejet/stable-diffusion.cpp`) and third-party builders. The following Linux x86_64 binaries are available: - -| Backend | Source | Availability | Approx. Size | Notes | -|---|---|---|---|---| -| **CPU** | leejet official releases | ✅ Available | ~20–30 MB | Generic x86_64 CPU binary | -| **Vulkan** | leejet official releases | ✅ Available | ~100 MB | Cross-vendor GPU (AMD, Intel, NVIDIA) | -| **ROCm** | leejet official releases | ✅ Available | ~1.2 GB | AMD GPU; requires host ROCm kernel driver | -| **CUDA** | leejet official releases | ❌ **Not published** | N/A | See section 2.2 | -| **CUDA** | leaxer-ai/leaxer-stable-diffusion (third-party) | ❌ **Rejected** | ~115 MB | Loads models but **segfaults during generation**; removed from setup. | - -**Critical Finding**: Upstream does **not** publish Linux CUDA binaries. There is an open feature request ([leejet/stable-diffusion.cpp#1291](https://github.com/leejet/stable-diffusion.cpp/issues/1291)) but no official support yet. - -### 2.2 Linux CUDA Feasibility Analysis - -We evaluated the third-party `leaxer-ai/leaxer-stable-diffusion` Linux CUDA binary: -- The binary downloads successfully and is a valid ELF executable. -- It loads models and initializes CUDA without errors. -- **However, it segfaults as soon as image generation starts** (reproduced on NVIDIA L4 with `TXT2IMG`). -- Because the binary is non-functional for its intended purpose, the project does **not** download or use it. - -**Decision**: Linux CUDA support is **disabled** until a reliable prebuilt binary becomes available (either from upstream leejet or a verified third-party builder). Linux NVIDIA systems fall back to the **Vulkan** backend, which uses the same GPU via the NVIDIA Vulkan driver. - -Users who need native CUDA performance on Linux can build `stable-diffusion.cpp` from source with `-DSD_CUDA=ON` and copy the resulting binary into `app/backend/linux/cuda/`; see README for instructions. - -### 2.3 Portable Node.js for Linux -- Node.js distributes `node-v22.12.0-linux-x64.tar.xz` (matching the Windows LTS version already used). -- Fully portable: extract with `tar` and run `bin/node` / `bin/npm` directly. - -### 2.4 Zero-Dependency GPU Detection on Linux -To avoid requiring `lspci`, `lshw`, or other host packages, we can parse PCI vendor IDs directly from sysfs: -- Read `/sys/bus/pci/devices/*/vendor` -- **NVIDIA**: `0x10de` -- **AMD**: `0x1002` -- **Intel**: `0x8086` -This is pure file I/O and works in any Linux environment with sysfs. - -### 2.5 Extraction Strategy (No Host `unzip` Required) -- Node.js `.tar.xz` → extracted with `tar` (universally available on Linux). -- Backend `.zip` files → extracted using Node.js after bootstrapping (e.g., via `adm-zip` or a small inline script), or fall back to `python3 -m zipfile` if needed. -- NVIDIA CUDA redist `.tar.xz` → extracted with `tar`. - -## 3. Implementation Approach: Maximum Performance (All Four Backends) - -Based on your feedback, the recommended approach is to support **CPU + Vulkan + ROCm + CUDA** on Linux. - -### 3.1 Backend Selection Matrix - -| Vendor | Primary Backend | Fallback Backend | Source | -|---|---|---|---| -| NVIDIA | CUDA (best perf) | Vulkan | CUDA = leaxer third-party; Vulkan = leejet official | -| AMD | ROCm (best perf) | Vulkan | ROCm = leejet official; Vulkan = leejet official | -| Intel | Vulkan | CPU | Vulkan = leejet official | -| No GPU / Unknown | CPU | — | CPU = leejet official | - -### 3.2 Backend Directory Layout (Linux) - -``` -app/backend/ -├── win/ -│ ├── cuda/sd-cuda.exe -│ └── vulkan/sd-vulkan.exe -└── linux/ - ├── cpu/sd-cpu - ├── vulkan/sd-vulkan - ├── rocm/sd-rocm + libstable-diffusion.so - └── cuda/sd-cuda + libstable-diffusion.so + libcudart.so.12 + libcublas.so.12 + libcublasLt.so.12 -``` - -## 4. Files to Create / Modify - -### New Files -1. **`start.sh`** — Bash entry point equivalent to `start.bat`. - - Checks for portable Node.js and frontend build. - - Runs `scripts/setup.sh` if anything is missing. - - Starts `scripts/serve.cjs` with the portable Node.js binary. - - Opens browser (`xdg-open`) or prints URL. - - Handles graceful shutdown (traps SIGINT/SIGTERM to kill backend). - -2. **`scripts/setup.sh`** — Bash setup equivalent to `setup.ps1`. - - Downloads portable Node.js Linux tarball with `curl` + progress bar. - - Extracts with `tar`. - - Detects GPU via sysfs PCI vendor IDs. - - Downloads appropriate backends: - - **CPU + Vulkan** for everyone (baseline). - - **ROCm** if AMD GPU detected. - - **CUDA** if NVIDIA GPU detected (optional, due to large download). - - Downloads NVIDIA CUDA redist libraries if CUDA is selected. - - Extracts all archives using `tar` / Node.js / Python fallback. - - Renames binaries to predictable names. - - Runs `npm install` and `npm run build` using portable npm. - -3. **`scripts/reset.sh`** — Bash reset equivalent to `reset.ps1`. - - Deletes `app/tools/`, `app/backend/`, `app/dist/`, `app/frontend/node_modules/` while preserving `app/models/` and `app/outputs/`. - -### Modified Files -4. **`scripts/serve.cjs`** — Enhance Linux support. - - **Add Linux backend paths**: - ```js - linuxCpu: path.join(ROOT, "app", "backend", "linux", "cpu", "sd-cpu"), - linuxVulkan: path.join(ROOT, "app", "backend", "linux", "vulkan", "sd-vulkan"), - linuxRocm: path.join(ROOT, "app", "backend", "linux", "rocm", "sd-rocm"), - linuxCuda: path.join(ROOT, "app", "backend", "linux", "cuda", "sd-cuda"), - ``` - - **Add Linux GPU detection** via sysfs (`/sys/bus/pci/devices/*/vendor`). - - **Update `getBackendOptions()`**: For Linux, return `cpu`, `vulkan`, `rocm` (if binary exists), and `cuda` (if binary + libraries exist). - - **Update `selectBackendPath()`**: Choose Linux backend based on resolved type; set `LD_LIBRARY_PATH` for CUDA/ROCm subprocesses so they find bundled `.so` files. - - **Update `getHealth()` / `getSetupPaths()`**: Include Linux backend checks. - - **Update `getGpuInfo()` / `hasNvidiaGpu()`**: Add Linux sysfs parsing. - - **CUDA VRAM polling**: Keep existing `nvidia-smi` approach; on Linux it works if drivers are installed. - -5. **`README.md`** — Add Linux section with compatibility matrix, `./start.sh` usage, and known limitations. - -## 5. Download Sources - -| Component | URL Pattern | -|---|---| -| Portable Node.js | `https://nodejs.org/dist/v22.12.0/node-v22.12.0-linux-x64.tar.xz` | -| CPU Backend | `https://github.com/leejet/stable-diffusion.cpp/releases/download/master-XXX/sd-master-XXX-bin-Linux-Ubuntu-24.04-x86_64.zip` | -| Vulkan Backend | `https://github.com/leejet/stable-diffusion.cpp/releases/download/master-XXX/sd-master-XXX-bin-Linux-Ubuntu-24.04-x86_64-vulkan.zip` | -| ROCm Backend | `https://github.com/leejet/stable-diffusion.cpp/releases/download/master-XXX/sd-master-XXX-bin-Linux-Ubuntu-24.04-x86_64-rocm-7.13.0.zip` | -| CUDA Binary (third-party) | `https://github.com/leaxer-ai/leaxer-stable-diffusion/releases/download/v0.1.0/sd-x86_64-unknown-linux-gnu-cuda` | -| CUDA Server (third-party) | `https://github.com/leaxer-ai/leaxer-stable-diffusion/releases/download/v0.1.0/sd-server-x86_64-unknown-linux-gnu-cuda` | -| CUDA Runtime (NVIDIA redist) | `https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/linux-x86_64/cuda_cudart-linux-x86_64-12.X.YZ-archive.tar.xz` | -| cuBLAS (NVIDIA redist) | `https://developer.download.nvidia.com/compute/cuda/redist/libcublas/linux-x86_64/libcublas-linux-x86_64-12.X.YZ-archive.tar.xz` | - -## 6. Setup Modes - -To respect users with limited bandwidth or no NVIDIA hardware, `setup.sh` should support two modes: - -1. **Quick Setup (default)** — Downloads CPU + Vulkan only. - - Works on all Linux systems. - - NVIDIA users get Vulkan acceleration (good, but not best). - - ~120–150 MB total download. - -2. **Maximum Performance Setup** — Downloads CPU + Vulkan + ROCm + CUDA. - - Best performance for each vendor. - - ~1.5 GB total download (mostly ROCm + cuBLAS). - - User can opt-in via `./start.sh --max-perf` or `./scripts/setup.sh --max-perf`. - -## 7. Self-Contained Checklist - -| Host Dependency | How We Eliminate It | -|---|---| -| Node.js installed globally | Download portable `node-v22.12.0-linux-x64.tar.xz` into `app/tools/node-linux/` | -| npm installed globally | Use portable npm from the tarball | -| `unzip` command | Extract ZIPs using Node.js or Python `zipfile` | -| `lspci` / `lshw` | Parse `/sys/bus/pci/devices/*/vendor` directly | -| C++ compiler / CUDA toolkit | Use pre-built binaries only | -| System package manager (apt/yum) | Never used | -| CUDA runtime libs | Bundle NVIDIA redistributable `.so` files | - -**One unavoidable host dependency remains**: the NVIDIA kernel driver (`libcuda.so.1`) for CUDA mode. Without it, CUDA cannot run. The app will detect this and automatically fall back to Vulkan for NVIDIA. - -## 8. Known Limitations - -1. **Third-party CUDA binary**: The Linux CUDA build comes from `leaxer-ai/leaxer-stable-diffusion`, which has only one release (v0.1.0) and may become stale. We should pin to a specific version and periodically evaluate alternatives. -2. **Large ROCm download**: The official ROCm backend ZIP is ~1.2 GB, which is a significant one-time download. -3. **Large cuBLAS download**: NVIDIA's redistributable cuBLAS package is ~400–500 MB. -4. **NVIDIA driver requirement for CUDA**: `libcuda.so.1` must be present on the host (provided by the NVIDIA proprietary driver). If missing, CUDA mode falls back to Vulkan. -5. **ROCm host driver requirement**: ROCm requires a compatible AMD GPU and ROCm kernel driver. If unavailable, ROCm falls back to Vulkan. -6. **glibc compatibility**: Official Linux binaries are built on Ubuntu 24.04. They may fail on very old distributions (e.g., CentOS 7, Ubuntu 18.04) due to glibc version mismatches. - -## 9. Recommended Implementation Order - -1. Create `start.sh`, `scripts/setup.sh`, and `scripts/reset.sh` with CPU + Vulkan support first. -2. Update `scripts/serve.cjs` with Linux paths, sysfs GPU detection, and backend selection. -3. Add ROCm download path (behind `--max-perf` flag). -4. Add CUDA download path using leaxer binary + NVIDIA redist libraries (behind `--max-perf` flag). -5. Add fallback logic so that if ROCm/CUDA fails to initialize, the app automatically switches to Vulkan. -6. Update `README.md` with Linux instructions and performance notes. -7. Test on Linux machines with NVIDIA (CUDA/Vulkan), AMD (ROCm/Vulkan), Intel (Vulkan), and CPU-only configurations. diff --git a/app/frontend/src/components/Settings.jsx b/app/frontend/src/components/Settings.jsx index d7b3099..02e4c72 100644 --- a/app/frontend/src/components/Settings.jsx +++ b/app/frontend/src/components/Settings.jsx @@ -5,7 +5,7 @@ import { Monitor, HardDrive, MemoryStick, Thermometer, Hash, Layers, ChevronRight, Box, Wand2, Lightbulb, RotateCcw, Check, Palette } from "lucide-react"; -import { stopServer, formatBytes, getLlmStatus, startLlm, stopLlm } from "../services/api"; +import { stopServer, formatBytes, getLlmBackends, getLlmStats, getLlmStatus, benchmarkLlm, startLlm, stopLlm } from "../services/api"; import { THEMES } from "../themes"; const ASPECT_RATIOS = [ @@ -187,6 +187,9 @@ function Settings({ setTheme, }) { const [llmStatus, setLlmStatus] = useState({ ready: false, settings: {} }); + const [llmBackends, setLlmBackends] = useState({ available: [], candidates: [] }); + const [llmStats, setLlmStats] = useState({ benchmarks: [] }); + const [benchmarkBusy, setBenchmarkBusy] = useState(false); useEffect(() => { let cancelled = false; @@ -195,6 +198,13 @@ function Settings({ const status = await getLlmStatus(); if (!cancelled) setLlmStatus(status); } catch (_) {} + try { + const [backends, stats] = await Promise.all([getLlmBackends(), getLlmStats()]); + if (!cancelled) { + setLlmBackends(backends); + setLlmStats(stats); + } + } catch (_) {} }; fetchLlmStatus(); const interval = setInterval(fetchLlmStatus, 3000); @@ -247,6 +257,37 @@ function Settings({ performanceProfile: settings?.performanceProfile, }); + const handleBenchmarkTextBackend = async () => { + const model = llmStatus.settings?.model; + if (!model) { + showAlert({ title: "No Text Model Loaded", message: "Load a GGUF text model before benchmarking.", danger: true }); + return; + } + setBenchmarkBusy(true); + try { + const result = await benchmarkLlm(model, { + contextSize: Math.min(2048, Number(textSettings?.contextSize) || 2048), + gpuLayers: textSettings?.gpuLayers ?? -1, + includeCpu: true, + }); + const winner = result.winner; + showAlert({ + title: winner ? "Benchmark Complete" : "Benchmark Finished", + message: winner + ? `Fastest backend: ${winner.backendMode} at ${Number(winner.predicted_per_second || 0).toFixed(1)} tokens/sec.` + : "Benchmark finished, but no backend returned usable timing data.", + }); + const [status, backends, stats] = await Promise.all([getLlmStatus(), getLlmBackends(true), getLlmStats()]); + setLlmStatus(status); + setLlmBackends(backends); + setLlmStats(stats); + } catch (err) { + showAlert({ title: "Benchmark Failed", message: err.message || String(err), danger: true }); + } finally { + setBenchmarkBusy(false); + } + }; + const handleThinkingToggle = async (enabled) => { const nextSettings = { ...textSettings, enableThinking: enabled }; @@ -924,6 +965,26 @@ function Settings({ GPU Layers: {llmStatus.settings?.gpuLayers === -1 ? "All" : llmStatus.settings?.gpuLayers} )} +
+ Available: {(llmBackends.available || []).map((backend) => backend.key.toUpperCase()).join(", ") || "None"}
+ Fastest saved: {(() => { + const model = llmStatus.settings?.model; + const winner = (llmStats.benchmarks || []).find((item) => item.ok && (!model || item.model === model)); + return winner + ? `${winner.backendMode} (${Number(winner.predicted_per_second || 0).toFixed(1)} tok/s)` + : "Not benchmarked"; + })()} +
+ diff --git a/app/frontend/src/services/api.js b/app/frontend/src/services/api.js index 32aa79f..61f3a7c 100644 --- a/app/frontend/src/services/api.js +++ b/app/frontend/src/services/api.js @@ -316,6 +316,32 @@ export async function getLlmStatus() { } } +export async function getLlmBackends(refresh = false) { + const res = await fetch(`/api/llm/backends${refresh ? "?refresh=1" : ""}`); + return await readJsonResponse(res, "The local server returned invalid text backend data."); +} + +export async function getLlmStats() { + const res = await fetch("/api/llm/stats"); + return await readJsonResponse(res, "The local server returned invalid text runtime stats."); +} + +export async function benchmarkLlm(model, options = {}) { + const res = await fetch("/api/llm/benchmark", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + model, + backends: options.backends, + includeCpu: options.includeCpu, + prompt: options.prompt, + contextSize: options.contextSize, + gpuLayers: options.gpuLayers, + }), + }); + return await readJsonResponse(res, "The local server returned invalid benchmark data."); +} + export async function listLlmModels() { const res = await fetch("/api/llm/models"); const data = await readJsonResponse(res, "The local server returned invalid text model data."); @@ -358,6 +384,7 @@ export async function startLlm(model, options = {}) { batchSize: options.batchSize, ubatchSize: options.ubatchSize, performanceProfile: options.performanceProfile, + preferredBackend: options.preferredBackend, }), }); return await readJsonResponse(res, "The local server returned an invalid text backend response."); diff --git a/linux.sh b/linux.sh index 828d285..866b322 100755 --- a/linux.sh +++ b/linux.sh @@ -129,12 +129,14 @@ fi CPU_SERVER_PATH="$APP_DIR/backend/linux/cpu/sd-server-cpu" VULKAN_SERVER_PATH="$APP_DIR/backend/linux/vulkan/sd-server-vulkan" LLM_CUDA_PATH="$APP_DIR/llm-backend/linux/cuda/llama-server" +LLM_ROCM_PATH="$APP_DIR/llm-backend/linux/rocm/llama-server" +LLM_SYCL_PATH="$APP_DIR/llm-backend/linux/sycl/llama-server" LLM_VULKAN_PATH="$APP_DIR/llm-backend/linux/vulkan/llama-server" LLM_CPU_PATH="$APP_DIR/llm-backend/linux/cpu/llama-server" if [[ ! -x "$CPU_BACKEND_PATH" || ! -x "$CPU_SERVER_PATH" ]] && [[ ! -x "$BACKEND_PATH" || ! -x "$VULKAN_SERVER_PATH" ]]; then SETUP_REASON="Linux backend binaries are missing or not executable." fi -if [[ ! -x "$LLM_CUDA_PATH" && ! -x "$LLM_VULKAN_PATH" && ! -x "$LLM_CPU_PATH" ]]; then +if [[ ! -x "$LLM_CUDA_PATH" && ! -x "$LLM_ROCM_PATH" && ! -x "$LLM_SYCL_PATH" && ! -x "$LLM_VULKAN_PATH" && ! -x "$LLM_CPU_PATH" ]]; then SETUP_REASON="Linux llama.cpp text backend is missing or not executable." fi diff --git a/scripts/llm-backends.json b/scripts/llm-backends.json new file mode 100644 index 0000000..eb55a54 --- /dev/null +++ b/scripts/llm-backends.json @@ -0,0 +1,26 @@ +{ + "llamaRelease": "b9668", + "notes": "Portable llama.cpp backend manifest. Setup scripts may override the release with LLAMA_RELEASE or a script parameter.", + "windows": { + "cuda": "llama-{release}-bin-win-cuda-12.4-x64.zip", + "vulkan": "llama-{release}-bin-win-vulkan-x64.zip", + "sycl": "llama-{release}-bin-win-sycl-x64.zip", + "hip": "llama-{release}-bin-win-hip-x64.zip", + "cpu": "llama-{release}-bin-win-cpu-x64.zip", + "cudaRuntime": "cudart-llama-bin-win-cuda-12.4-x64.zip" + }, + "linux": { + "cuda": "llama-{release}-bin-ubuntu-cuda-12.4-x64.tar.gz", + "rocm": "llama-{release}-bin-ubuntu-rocm-x64.tar.gz", + "vulkanX64": "llama-{release}-bin-ubuntu-vulkan-x64.tar.gz", + "vulkanArm64": "llama-{release}-bin-ubuntu-vulkan-arm64.tar.gz", + "syclFp32": "llama-{release}-bin-ubuntu-sycl-fp32-x64.tar.gz", + "syclFp16": "llama-{release}-bin-ubuntu-sycl-fp16-x64.tar.gz", + "cpuX64": "llama-{release}-bin-ubuntu-x64.tar.gz", + "cpuArm64": "llama-{release}-bin-ubuntu-arm64.tar.gz" + }, + "mac": { + "arm64": "llama-{release}-bin-macos-arm64.tar.gz", + "x64": "llama-{release}-bin-macos-x64.tar.gz" + } +} diff --git a/scripts/serve.cjs b/scripts/serve.cjs index 404cfb7..867dc9f 100644 --- a/scripts/serve.cjs +++ b/scripts/serve.cjs @@ -89,14 +89,22 @@ const LLM_BACKEND_PATHS = { winCuda: path.join(ROOT, "app", "llm-backend", "win", "cuda", "llama-server.exe"), winVulkan: path.join(ROOT, "app", "llm-backend", "win", "vulkan", "llama-server.exe"), winSycl: path.join(ROOT, "app", "llm-backend", "win", "sycl", "llama-server.exe"), + winHip: path.join(ROOT, "app", "llm-backend", "win", "hip", "llama-server.exe"), winCpu: path.join(ROOT, "app", "llm-backend", "win", "cpu", "llama-server.exe"), linuxCuda: path.join(ROOT, "app", "llm-backend", "linux", "cuda", "llama-server"), + linuxRocm: path.join(ROOT, "app", "llm-backend", "linux", "rocm", "llama-server"), linuxVulkan: path.join(ROOT, "app", "llm-backend", "linux", "vulkan", "llama-server"), linuxSycl: path.join(ROOT, "app", "llm-backend", "linux", "sycl", "llama-server"), linuxCpu: path.join(ROOT, "app", "llm-backend", "linux", "cpu", "llama-server"), macArm64: path.join(ROOT, "app", "llm-backend", "mac", "arm64", "llama-server"), macX64: path.join(ROOT, "app", "llm-backend", "mac", "x64", "llama-server"), }; +const LLM_CONFIG_DIR = path.join(ROOT, "app", "config"); +const LLM_MODEL_SETTINGS_PATH = path.join(LLM_CONFIG_DIR, "llm-model-settings.json"); +const LLM_BENCHMARK_PATH = path.join(LLM_CONFIG_DIR, "llm-benchmarks.json"); +if (!fs.existsSync(LLM_CONFIG_DIR)) { + fs.mkdirSync(LLM_CONFIG_DIR, { recursive: true }); +} const OPENVINO_MODELS = path.join(ROOT, "app", "openvino-models"); if (!fs.existsSync(OPENVINO_MODELS)) { fs.mkdirSync(OPENVINO_MODELS, { recursive: true }); @@ -832,10 +840,12 @@ function getLlamaTelemetryBackendPath() { if (osPlatform === "win32") { if (fs.existsSync(LLM_BACKEND_PATHS.winVulkan)) return LLM_BACKEND_PATHS.winVulkan; if (fs.existsSync(LLM_BACKEND_PATHS.winSycl)) return LLM_BACKEND_PATHS.winSycl; + if (fs.existsSync(LLM_BACKEND_PATHS.winHip)) return LLM_BACKEND_PATHS.winHip; if (fs.existsSync(LLM_BACKEND_PATHS.winCuda)) return LLM_BACKEND_PATHS.winCuda; } else if (osPlatform === "linux") { if (fs.existsSync(LLM_BACKEND_PATHS.linuxVulkan)) return LLM_BACKEND_PATHS.linuxVulkan; if (fs.existsSync(LLM_BACKEND_PATHS.linuxSycl)) return LLM_BACKEND_PATHS.linuxSycl; + if (fs.existsSync(LLM_BACKEND_PATHS.linuxRocm)) return LLM_BACKEND_PATHS.linuxRocm; if (fs.existsSync(LLM_BACKEND_PATHS.linuxCuda)) return LLM_BACKEND_PATHS.linuxCuda; } return null; @@ -876,6 +886,89 @@ function getLlamaVram() { return cachedLlamaVramInfo; } +function readJsonFile(filePath, fallback) { + try { + if (!fs.existsSync(filePath)) return fallback; + return JSON.parse(fs.readFileSync(filePath, "utf8")); + } catch (_) { + return fallback; + } +} + +function writeJsonFile(filePath, value) { + fs.mkdirSync(path.dirname(filePath), { recursive: true }); + fs.writeFileSync(filePath, JSON.stringify(value, null, 2) + "\n", "utf8"); +} + +function getPersistedLlmModelSettings() { + return readJsonFile(LLM_MODEL_SETTINGS_PATH, { models: {} }); +} + +function savePersistedLlmModelSettings(settings) { + writeJsonFile(LLM_MODEL_SETTINGS_PATH, settings && typeof settings === "object" ? settings : { models: {} }); +} + +function getLlmModelSettings(filename) { + const safeFilename = path.basename(String(filename || "")); + if (!safeFilename) return {}; + const settings = getPersistedLlmModelSettings(); + return settings.models?.[safeFilename] || {}; +} + +function updateLlmModelSettings(filename, patch) { + const safeFilename = path.basename(String(filename || "")); + if (!safeFilename) return {}; + const allSettings = getPersistedLlmModelSettings(); + if (!allSettings.models || typeof allSettings.models !== "object") allSettings.models = {}; + const previous = allSettings.models[safeFilename] || {}; + const next = { + ...previous, + ...patch, + updatedAt: new Date().toISOString(), + }; + allSettings.models[safeFilename] = next; + savePersistedLlmModelSettings(allSettings); + return next; +} + +function getPersistedLlmBenchmarks() { + return readJsonFile(LLM_BENCHMARK_PATH, { results: [] }); +} + +function savePersistedLlmBenchmarks(benchmarks) { + writeJsonFile(LLM_BENCHMARK_PATH, benchmarks && typeof benchmarks === "object" ? benchmarks : { results: [] }); +} + +function recordLlmBenchmark(result) { + const benchmarks = getPersistedLlmBenchmarks(); + const results = Array.isArray(benchmarks.results) ? benchmarks.results : []; + results.unshift({ + ...result, + createdAt: new Date().toISOString(), + }); + benchmarks.results = results.slice(0, 100); + savePersistedLlmBenchmarks(benchmarks); + return benchmarks.results[0]; +} + +function parseLlamaDeviceList(output) { + const devices = []; + const devicePattern = /(CUDA|Vulkan|SYCL|HIP|ROCm)\d*\s*:\s*(.+?)\s+\(([\d.]+)\s+MiB(?:,\s*([\d.]+)\s+MiB free)?\)/gi; + let match; + while ((match = devicePattern.exec(output || "")) !== null) { + const totalMiB = Number(match[3]); + const freeMiB = match[4] ? Number(match[4]) : NaN; + if (!Number.isFinite(totalMiB) || totalMiB <= 0) continue; + devices.push({ + type: match[1], + name: match[2].trim(), + vram_gb: Math.round((totalMiB / 1024) * 100) / 100, + free_vram_gb: Number.isFinite(freeMiB) ? Math.round((freeMiB / 1024) * 100) / 100 : null, + }); + } + return devices.sort((a, b) => b.vram_gb - a.vram_gb); +} + let cachedMacRamUsedGb = null; function pollMacRam() { if (osPlatform !== "darwin") return; @@ -1307,43 +1400,120 @@ function getLlmBackend() { return getLlmBackendCandidates()[0] || null; } -function getLlmBackendCandidates() { - const candidates = []; - const includeUndetectedGpu = process.env.LOCALAI_LLM_TRY_UNDETECTED_GPU === "1"; - const add = (key, backendPath, mode, detected = true) => { - if (!backendPath || !fs.existsSync(backendPath)) return; - if (key !== "cpu" && !detected && !includeUndetectedGpu) return; - if (candidates.some((item) => item.path === backendPath)) return; - candidates.push({ key, path: backendPath, mode, detected }); - }; +let cachedLlmBackendProbe = null; +let cachedLlmBackendProbeAt = 0; - if (osPlatform === "win32") { - add("cuda", LLM_BACKEND_PATHS.winCuda, "Auto (CUDA/CPU)", Boolean(detectLlamaCudaGpu())); - add("vulkan", LLM_BACKEND_PATHS.winVulkan, "Auto (Vulkan/CPU)", Boolean(detectLlamaVulkanGpu())); - add("sycl", LLM_BACKEND_PATHS.winSycl, "Auto (SYCL/CPU)", Boolean(detectLlamaSyclGpu())); - add("cpu", LLM_BACKEND_PATHS.winCpu, "CPU", true); - } else if (osPlatform === "darwin") { - add("metal", process.arch === "arm64" ? LLM_BACKEND_PATHS.macArm64 : LLM_BACKEND_PATHS.macX64, process.arch === "arm64" ? "Metal GPU" : "CPU", true); - } else { - add("cuda", LLM_BACKEND_PATHS.linuxCuda, "Auto (CUDA/CPU)", Boolean(detectLlamaCudaGpu())); - add("sycl", LLM_BACKEND_PATHS.linuxSycl, "Auto (SYCL/CPU)", Boolean(detectLlamaSyclGpu())); - add("vulkan", LLM_BACKEND_PATHS.linuxVulkan, "Auto (Vulkan/CPU)", Boolean(detectLlamaVulkanGpu())); - add("cpu", LLM_BACKEND_PATHS.linuxCpu, "CPU", true); +function resolveSystemLlamaServer() { + const configured = process.env.LOCALAI_LLM_SYSTEM_SERVER || process.env.LLAMA_SERVER_PATH || ""; + if (configured && fs.existsSync(configured)) return configured; + return ""; +} + +function probeLlmBackend(backend) { + if (!backend?.path || !fs.existsSync(backend.path)) { + return { + ...backend, + installed: false, + detected: false, + devices: [], + error: "Backend binary not installed.", + }; } - const priority = osPlatform === "win32" - ? { cuda: 0, vulkan: 1, sycl: 2, metal: 3, cpu: 9 } - : osPlatform === "darwin" - ? { metal: 0, cpu: 9 } - : { cuda: 0, sycl: 1, vulkan: 2, cpu: 9 }; + try { + const result = spawnSync(backend.path, ["--list-devices"], { + encoding: "utf8", + windowsHide: true, + timeout: 10000, + maxBuffer: 1024 * 1024, + }); + const output = `${result.stdout || ""}\n${result.stderr || ""}`; + const devices = parseLlamaDeviceList(output); + const detected = backend.key === "cpu" || backend.key === "metal" || backend.key === "system" || devices.length > 0; + return { + ...backend, + installed: true, + detected, + devices, + device: devices[0] || null, + error: result.status && !detected ? output.trim().slice(-600) : "", + }; + } catch (err) { + return { + ...backend, + installed: true, + detected: backend.key === "cpu", + devices: [], + device: null, + error: err.message || String(err), + }; + } +} - return candidates.sort((a, b) => { - if (a.key === "cpu" && b.key !== "cpu") return 1; - if (b.key === "cpu" && a.key !== "cpu") return -1; - const detectedDelta = Number(b.detected) - Number(a.detected); - if (detectedDelta) return detectedDelta; - return (priority[a.key] ?? 8) - (priority[b.key] ?? 8); - }); +function getAllLlmBackendCandidates() { + const systemPath = resolveSystemLlamaServer(); + const candidates = []; + const add = (key, backendPath, mode, portable = true) => { + if (!backendPath) return; + if (candidates.some((item) => item.path === backendPath)) return; + candidates.push({ key, path: backendPath, mode, portable }); + }; + + if (systemPath) add("system", systemPath, "System llama.cpp", false); + if (osPlatform === "win32") { + add("cuda", LLM_BACKEND_PATHS.winCuda, "Auto (CUDA/CPU)"); + add("hip", LLM_BACKEND_PATHS.winHip, "Auto (HIP/CPU)"); + add("vulkan", LLM_BACKEND_PATHS.winVulkan, "Auto (Vulkan/CPU)"); + add("sycl", LLM_BACKEND_PATHS.winSycl, "Auto (SYCL/CPU)"); + add("cpu", LLM_BACKEND_PATHS.winCpu, "CPU"); + } else if (osPlatform === "darwin") { + add("metal", process.arch === "arm64" ? LLM_BACKEND_PATHS.macArm64 : LLM_BACKEND_PATHS.macX64, process.arch === "arm64" ? "Metal GPU" : "CPU"); + } else { + add("cuda", LLM_BACKEND_PATHS.linuxCuda, "Auto (CUDA/CPU)"); + add("rocm", LLM_BACKEND_PATHS.linuxRocm, "Auto (ROCm/CPU)"); + add("sycl", LLM_BACKEND_PATHS.linuxSycl, "Auto (SYCL/CPU)"); + add("vulkan", LLM_BACKEND_PATHS.linuxVulkan, "Auto (Vulkan/CPU)"); + add("cpu", LLM_BACKEND_PATHS.linuxCpu, "CPU"); + } + return candidates; +} + +function getLlmBackendProbe(force = false) { + const now = Date.now(); + if (!force && cachedLlmBackendProbe && now - cachedLlmBackendProbeAt < 60000) return cachedLlmBackendProbe; + + const probed = getAllLlmBackendCandidates().map(probeLlmBackend); + const priority = osPlatform === "win32" + ? { system: -1, cuda: 0, hip: 1, vulkan: 2, sycl: 3, metal: 4, cpu: 9 } + : osPlatform === "darwin" + ? { system: -1, metal: 0, cpu: 9 } + : { system: -1, cuda: 0, rocm: 1, sycl: 2, vulkan: 3, cpu: 9 }; + + const available = probed + .filter((item) => item.installed && item.detected) + .sort((a, b) => { + if (a.key === "cpu" && b.key !== "cpu") return 1; + if (b.key === "cpu" && a.key !== "cpu") return -1; + return (priority[a.key] ?? 8) - (priority[b.key] ?? 8); + }); + + cachedLlmBackendProbe = { + generatedAt: new Date().toISOString(), + candidates: probed, + available, + selected: available[0] || null, + }; + cachedLlmBackendProbeAt = now; + return cachedLlmBackendProbe; +} + +function getLlmBackendCandidates() { + const includeUndetectedGpu = process.env.LOCALAI_LLM_TRY_UNDETECTED_GPU === "1"; + const probe = getLlmBackendProbe(); + if (includeUndetectedGpu) { + return probe.candidates.filter((item) => item.installed); + } + return probe.available; } function getLlmModels() { @@ -2647,13 +2817,37 @@ async function startLlm(settings = {}) { if (candidates.length === 0) { throw new Error("llama.cpp is not installed. Run the platform setup script to install the text backend."); } + const persistedSettings = getLlmModelSettings(filename); + const preferredBackend = String(settings.preferredBackend || persistedSettings.preferredBackend || "").toLowerCase(); + const sortedCandidates = preferredBackend + ? [...candidates].sort((a, b) => { + if (a.key === preferredBackend && b.key !== preferredBackend) return -1; + if (b.key === preferredBackend && a.key !== preferredBackend) return 1; + return 0; + }) + : candidates; const failures = []; - for (const backend of candidates) { + for (const backend of sortedCandidates) { for (const profile of buildLlmLoadProfiles(settings, backend)) { try { await startLlmWithBackend({ ...settings, __loadProfile: profile }, backend); llmSettings.backendFallbacks = failures; + updateLlmModelSettings(filename, { + preferredBackend: backend.key, + lastBackendMode: llmSettings.backendMode, + lastBackendBinary: llmSettings.backendBinary, + lastSettings: { + threads: llmSettings.threads, + contextSize: llmSettings.contextSize, + gpuLayers: llmSettings.gpuLayers, + cacheTypeK: llmSettings.cacheTypeK, + cacheTypeV: llmSettings.cacheTypeV, + batchSize: llmSettings.batchSize, + ubatchSize: llmSettings.ubatchSize, + flashAttn: llmSettings.flashAttn, + }, + }); return; } catch (err) { const message = err.message || String(err); @@ -2833,8 +3027,9 @@ async function startLlmWithBackend(settings = {}, backend) { } // CPU mode: lock memory to prevent OS paging (critical for consistent speed) - const isGpuMode = backend.mode.includes("GPU") || backend.mode.includes("CUDA") || - backend.mode.includes("Vulkan") || backend.mode.includes("Metal") || + const isGpuMode = backend.mode.includes("GPU") || backend.mode.includes("CUDA") || + backend.mode.includes("HIP") || backend.mode.includes("ROCm") || + backend.mode.includes("Vulkan") || backend.mode.includes("Metal") || backend.mode.includes("SYCL"); if (!isGpuMode && llmSettings.mlock) { args.push("--mlock"); // ✅ Prevent OS swapping on CPU @@ -2902,6 +3097,7 @@ async function startLlmWithBackend(settings = {}, backend) { process.stderr.write(" [llm-err] " + output); if (/Vulkan\d+\s*:/i.test(output)) llmSettings.backendMode = "Vulkan GPU"; else if (/CUDA\d+\s*:/i.test(output)) llmSettings.backendMode = "CUDA GPU"; + else if (/(HIP|ROCm)\d*\s*:/i.test(output)) llmSettings.backendMode = "ROCm GPU"; else if (/SYCL\d+\s*:/i.test(output)) llmSettings.backendMode = "SYCL GPU"; else if (/Metal/i.test(output) && /GPU|device/i.test(output)) llmSettings.backendMode = "Metal GPU"; else if (/\-\s+CPU\s+:/i.test(output) && llmSettings.backendMode.startsWith("Auto")) llmSettings.backendMode = "CPU"; @@ -3983,6 +4179,7 @@ const server = http.createServer(async (req, res) => { if (req.url === "/api/llm/status" && req.method === "GET") { const backend = getLlmBackend(); + const probe = getLlmBackendProbe(); return json(res, 200, { ready: llmReady, running: llmProc !== null, @@ -3993,9 +4190,35 @@ const server = http.createServer(async (req, res) => { backendInstalled: Boolean(backend), backendMode: backend?.mode || "", backendPath: backend?.path || "", + selectedBackend: probe.selected, + availableBackends: probe.available, }); } + if (req.url.startsWith("/api/llm/backends") && req.method === "GET") { + const parsed = new URL(req.url, `http://${req.headers.host || "localhost"}`); + return json(res, 200, { ok: true, ...getLlmBackendProbe(parsed.searchParams.get("refresh") === "1") }); + } + + if (req.url === "/api/llm/stats" && req.method === "GET") { + return json(res, 200, getLlmRuntimeStats()); + } + + if (req.url === "/api/llm/model-settings" && req.method === "GET") { + return json(res, 200, { ok: true, ...getPersistedLlmModelSettings() }); + } + + if (req.url === "/api/llm/model-settings" && req.method === "POST") { + const body = await readJsonBody(req, res); + if (!body) return; + const filename = path.basename(String(body.model || "")); + if (!filename) return json(res, 400, { ok: false, error: "model is required" }); + const updated = updateLlmModelSettings(filename, { + preferredBackend: body.preferredBackend ? String(body.preferredBackend).toLowerCase() : undefined, + }); + return json(res, 200, { ok: true, model: filename, settings: updated }); + } + if (req.url === "/api/llm/models" && req.method === "GET") { return json(res, 200, { models: getLlmModels() }); } @@ -4044,6 +4267,17 @@ const server = http.createServer(async (req, res) => { return json(res, 200, { ok: true }); } + if (req.url === "/api/llm/benchmark" && req.method === "POST") { + const body = await readJsonBody(req, res); + if (!body) return; + try { + const result = await runExclusiveLlmOperation(() => benchmarkLlmModel(body)); + return json(res, 200, result); + } catch (err) { + return json(res, 500, { ok: false, error: err.message || String(err) }); + } + } + function isOomError(msg) { if (!msg) return false; const lower = String(msg).toLowerCase(); @@ -4191,6 +4425,124 @@ async function doLlmChat(req, res, body, retryCount = 0) { } } +function getLlmRuntimeStats() { + const benchmarks = getPersistedLlmBenchmarks(); + return { + ok: true, + ready: llmReady, + running: llmProc !== null, + selectedBackend: getLlmBackendProbe().selected, + settings: llmSettings, + modelSettings: getPersistedLlmModelSettings(), + benchmarks: (benchmarks.results || []).slice(0, 20), + }; +} + +async function benchmarkLlmBackend(model, backend, baseSettings = {}) { + const prompt = String(baseSettings.prompt || "Reply with one short sentence about local AI performance."); + const startedAt = Date.now(); + await startLlmWithBackend({ + ...baseSettings, + model, + contextSize: Math.min(2048, Number(baseSettings.contextSize) || 2048), + gpuLayers: Number.isFinite(Number(baseSettings.gpuLayers)) ? Number(baseSettings.gpuLayers) : -1, + enableThinking: false, + }, backend); + + const result = await requestJson(`http://127.0.0.1:${PORT_LLM}/v1/chat/completions`, { + model: llmSettings.model || "local-model", + messages: [ + { role: "system", content: "You are a concise local benchmark assistant." }, + { role: "user", content: prompt }, + ], + temperature: 0.2, + max_tokens: 64, + stream: false, + }, 180000); + const finishedAt = Date.now(); + const timings = result.timings || {}; + return { + backendKey: backend.key, + backendMode: llmSettings.backendMode || backend.mode, + backendBinary: llmSettings.backendBinary || path.basename(backend.path), + model, + ok: true, + total_ms: finishedAt - startedAt, + prompt_ms: Number(timings.prompt_ms) || null, + predicted_ms: Number(timings.predicted_ms) || null, + predicted_n: Number(timings.predicted_n) || null, + predicted_per_second: Number(timings.predicted_per_second) || null, + prompt_per_second: Number(timings.prompt_per_second) || null, + }; +} + +async function benchmarkLlmModel(settings = {}) { + const filename = path.basename(String(settings.model || llmSettings.model || "")); + const modelPath = path.join(LLM_MODELS, filename); + if (!filename || !pathInside(modelPath, LLM_MODELS) || !fs.existsSync(modelPath)) { + throw new Error("Select a downloaded GGUF text model before benchmarking."); + } + + const previous = { + ready: llmReady, + model: llmSettings.model, + settings: { ...llmSettings }, + }; + const requestedBackends = Array.isArray(settings.backends) + ? settings.backends.map((item) => String(item).toLowerCase()) + : []; + const candidates = getLlmBackendCandidates() + .filter((backend) => backend.key !== "cpu" || settings.includeCpu !== false) + .filter((backend) => requestedBackends.length === 0 || requestedBackends.includes(backend.key)); + + if (candidates.length === 0) { + throw new Error("No available text backends were detected for benchmarking."); + } + + const results = []; + for (const backend of candidates) { + try { + const result = await benchmarkLlmBackend(filename, backend, settings); + results.push(recordLlmBenchmark(result)); + } catch (err) { + results.push(recordLlmBenchmark({ + backendKey: backend.key, + backendMode: backend.mode, + backendBinary: path.basename(backend.path), + model: filename, + ok: false, + error: (err.message || String(err)).slice(-800), + })); + await killLlm(); + } + } + + const successful = results + .filter((item) => item.ok && Number(item.predicted_per_second) > 0) + .sort((a, b) => Number(b.predicted_per_second) - Number(a.predicted_per_second)); + const winner = successful[0] || null; + if (winner) { + updateLlmModelSettings(filename, { + preferredBackend: winner.backendKey, + benchmarkWinner: { + backendMode: winner.backendMode, + predicted_per_second: winner.predicted_per_second, + createdAt: winner.createdAt, + }, + }); + } + + if (previous.ready && previous.model && previous.model === filename && winner) { + try { + await startLlm({ ...previous.settings, model: filename, preferredBackend: winner.backendKey }); + } catch (err) { + console.warn(" [llm] Failed to restore text model after benchmark:", err.message || err); + } + } + + return { ok: true, model: filename, winner, results }; +} + // ── llmfit Integration ────────────────────────────────────────────────────────── let cachedLlmfitResults = new Map(); diff --git a/scripts/setup-llama.ps1 b/scripts/setup-llama.ps1 index e44eab2..185ae35 100644 --- a/scripts/setup-llama.ps1 +++ b/scripts/setup-llama.ps1 @@ -140,6 +140,17 @@ function Install-LlamaArchive { Write-Host " OK llama.cpp $Variant backend installed." } +function Try-InstallLlamaArchive { + param([string]$Variant, [string]$AssetName, [string]$Reason) + + try { + Install-LlamaArchive -Variant $Variant -AssetName $AssetName + } catch { + Write-Host " !! Skipping optional llama.cpp $Variant backend: $Reason" -ForegroundColor Yellow + Write-Host " $($_.Exception.Message)" -ForegroundColor DarkYellow + } +} + $hasNvidia = $false try { $gpus = Get-CimInstance Win32_VideoController -ErrorAction SilentlyContinue @@ -157,7 +168,20 @@ if (-not $hasNvidia) { } if ($hasNvidia) { - Install-LlamaArchive -Variant "cuda" -AssetName "llama-$Release-bin-win-cuda-12.4-x64.zip" + Try-InstallLlamaArchive -Variant "cuda" -AssetName "llama-$Release-bin-win-cuda-12.4-x64.zip" -Reason "NVIDIA CUDA acceleration" +} + +$hasAmd = $false +try { + $gpus = Get-CimInstance Win32_VideoController -ErrorAction SilentlyContinue + foreach ($gpu in $gpus) { + if ($gpu.Name -like "*AMD*" -or $gpu.Name -like "*Radeon*") { + $hasAmd = $true + } + } +} catch {} +if ($hasAmd) { + Try-InstallLlamaArchive -Variant "hip" -AssetName "llama-$Release-bin-win-hip-x64.zip" -Reason "AMD HIP acceleration" } # Intel Arc/Graphics detection for SYCL backend $hasIntel = $false @@ -170,7 +194,7 @@ try { } } catch {} if ($hasIntel) { - Install-LlamaArchive -Variant "sycl" -AssetName "llama-$Release-bin-win-sycl-x64.zip" + Try-InstallLlamaArchive -Variant "sycl" -AssetName "llama-$Release-bin-win-sycl-x64.zip" -Reason "Intel SYCL acceleration" } Install-LlamaArchive -Variant "vulkan" -AssetName "llama-$Release-bin-win-vulkan-x64.zip" Install-LlamaArchive -Variant "cpu" -AssetName "llama-$Release-bin-win-cpu-x64.zip" diff --git a/scripts/setup-llama.sh b/scripts/setup-llama.sh index e0acb1a..635ec4e 100644 --- a/scripts/setup-llama.sh +++ b/scripts/setup-llama.sh @@ -31,10 +31,39 @@ download_and_extract() { if [[ ! -x "$dest/llama-server" ]]; then echo " XX llama-server was not found after extracting $asset" >&2 - exit 1 + return 1 fi } +download_and_extract_optional() { + local asset="$1" + local dest="$2" + local reason="$3" + + if [[ -x "$dest/llama-server" ]]; then + echo " OK optional llama.cpp backend already ready: $dest" + return 0 + fi + + if download_and_extract "$asset" "$dest"; then + return 0 + fi + + echo " !! Skipping optional llama.cpp backend for $reason ($asset)" >&2 + rm -rf "$dest" + return 0 +} + +has_command() { + command -v "$1" >/dev/null 2>&1 +} + +has_linux_gpu_vendor() { + local vendor="$1" + [[ -d /sys/bus/pci/devices ]] || return 1 + grep -Ril "$vendor" /sys/bus/pci/devices/*/vendor >/dev/null 2>&1 +} + if [[ "$PLATFORM" == "Darwin" ]]; then if [[ "$ARCH" == "arm64" ]]; then download_and_extract "llama-$RELEASE-bin-macos-arm64.tar.gz" "$APP_DIR/llm-backend/mac/arm64" @@ -46,6 +75,15 @@ elif [[ "$PLATFORM" == "Linux" ]]; then download_and_extract "llama-$RELEASE-bin-ubuntu-vulkan-arm64.tar.gz" "$APP_DIR/llm-backend/linux/vulkan" download_and_extract "llama-$RELEASE-bin-ubuntu-arm64.tar.gz" "$APP_DIR/llm-backend/linux/cpu" else + if has_command nvidia-smi || has_linux_gpu_vendor "0x10de"; then + download_and_extract_optional "llama-$RELEASE-bin-ubuntu-cuda-12.4-x64.tar.gz" "$APP_DIR/llm-backend/linux/cuda" "NVIDIA CUDA acceleration" + fi + if has_command rocminfo || has_linux_gpu_vendor "0x1002"; then + download_and_extract_optional "llama-$RELEASE-bin-ubuntu-rocm-x64.tar.gz" "$APP_DIR/llm-backend/linux/rocm" "AMD ROCm acceleration" + fi + if has_linux_gpu_vendor "0x8086"; then + download_and_extract_optional "llama-$RELEASE-bin-ubuntu-sycl-fp32-x64.tar.gz" "$APP_DIR/llm-backend/linux/sycl" "Intel SYCL acceleration" + fi download_and_extract "llama-$RELEASE-bin-ubuntu-vulkan-x64.tar.gz" "$APP_DIR/llm-backend/linux/vulkan" download_and_extract "llama-$RELEASE-bin-ubuntu-x64.tar.gz" "$APP_DIR/llm-backend/linux/cpu" fi diff --git a/windows.bat b/windows.bat index 04a105d..b63d6fb 100644 --- a/windows.bat +++ b/windows.bat @@ -10,7 +10,9 @@ set SETUP=%~dp0scripts\setup.ps1 set CUDA_BACKEND=%APP%\backend\win\cuda\sd-cuda.exe set VULKAN_BACKEND=%APP%\backend\win\vulkan\sd-vulkan.exe set LLM_CUDA_BACKEND=%APP%\llm-backend\win\cuda\llama-server.exe +set LLM_HIP_BACKEND=%APP%\llm-backend\win\hip\llama-server.exe set LLM_VULKAN_BACKEND=%APP%\llm-backend\win\vulkan\llama-server.exe +set LLM_SYCL_BACKEND=%APP%\llm-backend\win\sycl\llama-server.exe set LLM_CPU_BACKEND=%APP%\llm-backend\win\cpu\llama-server.exe set SERVE=%~dp0scripts\serve.cjs if "%FRONTEND_PORT%"=="" set FRONTEND_PORT=1420 @@ -32,7 +34,7 @@ if not exist "%DIST%" ( set SETUP_REASON=Frontend build is missing. goto :run_setup ) -if not exist "%LLM_CUDA_BACKEND%" if not exist "%LLM_VULKAN_BACKEND%" if not exist "%LLM_CPU_BACKEND%" ( +if not exist "%LLM_CUDA_BACKEND%" if not exist "%LLM_HIP_BACKEND%" if not exist "%LLM_VULKAN_BACKEND%" if not exist "%LLM_SYCL_BACKEND%" if not exist "%LLM_CPU_BACKEND%" ( set SETUP_REASON=llama.cpp text backend is missing. goto :run_setup )