add LLM configuration and backend management components with automated setup scripts

.gitignore

@@ -9,6 +9,7 @@ app/llm-backend/
 app/tools/
 
 # Generated files
+app/config/
 app/dist/
 app/outputs/
 

PLAN.md

@@ -1,192 +0,0 @@
-# Linux Port Plan for Local AI Image Generator
-
-> **Plan location**: This plan will be copied to the project root as `PLAN.md` upon approval, per user request.
-
-> **Implementation Update (post-verification)**
->
-> During end-to-end testing, the third-party Linux CUDA binary (`leaxer-ai/leaxer-stable-diffusion`) was found to **segfault during generation** after successfully loading the model. As a result, Linux CUDA support was **removed** from the implementation. Linux NVIDIA GPUs now use the **Vulkan** backend. The setup script also gained a glibc version check, because the official leejet Linux binaries require **glibc 2.38+** (Ubuntu 24.04). The frontend's misleading "mock SVG" fallback was replaced with a real error message when the backend crashes. See the updated README for build-from-source instructions.
-
-## 1. What This Project Does
-
-**Local AI Image Generator** is a zero-configuration, portable desktop environment for running Stable Diffusion (Safetensors / GGUF / CKPT) completely offline. The Windows flow is:
-
-1. **Entry Point**: `start.bat` double-click launcher.
-2. **Setup**: `scripts/setup.ps1` auto-downloads portable Node.js for Windows, detects the GPU vendor, downloads the matching stable-diffusion.cpp backend (CUDA for NVIDIA, Vulkan for AMD/Intel), installs npm dependencies, and builds the Vite+React frontend into `app/dist/`.
-3. **Runtime**: `scripts/serve.cjs` (a Node.js HTTP server) serves the static frontend on `localhost:1420`, manages the backend process lifecycle, proxies generation requests, handles model downloads/imports, tracks telemetry, and manages the gallery.
-4. **All sandboxed**: Everything lives under the project folder (`app/tools/`, `app/backend/`, `app/models/`, `app/outputs/`). No global PATH changes, no system installers.
-
-## 2. Research Findings
-
-### 2.1 stable-diffusion.cpp Linux Binaries
-
-We researched the upstream project (`leejet/stable-diffusion.cpp`) and third-party builders. The following Linux x86_64 binaries are available:
-
-| Backend | Source | Availability | Approx. Size | Notes |
-|---|---|---|---|---|
-| **CPU** | leejet official releases | ✅ Available | ~20–30 MB | Generic x86_64 CPU binary |
-| **Vulkan** | leejet official releases | ✅ Available | ~100 MB | Cross-vendor GPU (AMD, Intel, NVIDIA) |
-| **ROCm** | leejet official releases | ✅ Available | ~1.2 GB | AMD GPU; requires host ROCm kernel driver |
-| **CUDA** | leejet official releases | ❌ **Not published** | N/A | See section 2.2 |
-| **CUDA** | leaxer-ai/leaxer-stable-diffusion (third-party) | ❌ **Rejected** | ~115 MB | Loads models but **segfaults during generation**; removed from setup. |
-
-**Critical Finding**: Upstream does **not** publish Linux CUDA binaries. There is an open feature request ([leejet/stable-diffusion.cpp#1291](https://github.com/leejet/stable-diffusion.cpp/issues/1291)) but no official support yet.
-
-### 2.2 Linux CUDA Feasibility Analysis
-
-We evaluated the third-party `leaxer-ai/leaxer-stable-diffusion` Linux CUDA binary:
-- The binary downloads successfully and is a valid ELF executable.
-- It loads models and initializes CUDA without errors.
-- **However, it segfaults as soon as image generation starts** (reproduced on NVIDIA L4 with `TXT2IMG`).
-- Because the binary is non-functional for its intended purpose, the project does **not** download or use it.
-
-**Decision**: Linux CUDA support is **disabled** until a reliable prebuilt binary becomes available (either from upstream leejet or a verified third-party builder). Linux NVIDIA systems fall back to the **Vulkan** backend, which uses the same GPU via the NVIDIA Vulkan driver.
-
-Users who need native CUDA performance on Linux can build `stable-diffusion.cpp` from source with `-DSD_CUDA=ON` and copy the resulting binary into `app/backend/linux/cuda/`; see README for instructions.
-
-### 2.3 Portable Node.js for Linux
-- Node.js distributes `node-v22.12.0-linux-x64.tar.xz` (matching the Windows LTS version already used).
-- Fully portable: extract with `tar` and run `bin/node` / `bin/npm` directly.
-
-### 2.4 Zero-Dependency GPU Detection on Linux
-To avoid requiring `lspci`, `lshw`, or other host packages, we can parse PCI vendor IDs directly from sysfs:
-- Read `/sys/bus/pci/devices/*/vendor`
-- **NVIDIA**: `0x10de`
-- **AMD**: `0x1002`
-- **Intel**: `0x8086`
-This is pure file I/O and works in any Linux environment with sysfs.
-
-### 2.5 Extraction Strategy (No Host `unzip` Required)
-- Node.js `.tar.xz` → extracted with `tar` (universally available on Linux).
-- Backend `.zip` files → extracted using Node.js after bootstrapping (e.g., via `adm-zip` or a small inline script), or fall back to `python3 -m zipfile` if needed.
-- NVIDIA CUDA redist `.tar.xz` → extracted with `tar`.
-
-## 3. Implementation Approach: Maximum Performance (All Four Backends)
-
-Based on your feedback, the recommended approach is to support **CPU + Vulkan + ROCm + CUDA** on Linux.
-
-### 3.1 Backend Selection Matrix
-
-| Vendor | Primary Backend | Fallback Backend | Source |
-|---|---|---|---|
-| NVIDIA | CUDA (best perf) | Vulkan | CUDA = leaxer third-party; Vulkan = leejet official |
-| AMD | ROCm (best perf) | Vulkan | ROCm = leejet official; Vulkan = leejet official |
-| Intel | Vulkan | CPU | Vulkan = leejet official |
-| No GPU / Unknown | CPU | — | CPU = leejet official |
-
-### 3.2 Backend Directory Layout (Linux)
-
-```
-app/backend/
-├── win/
-│   ├── cuda/sd-cuda.exe
-│   └── vulkan/sd-vulkan.exe
-└── linux/
-    ├── cpu/sd-cpu
-    ├── vulkan/sd-vulkan
-    ├── rocm/sd-rocm + libstable-diffusion.so
-    └── cuda/sd-cuda + libstable-diffusion.so + libcudart.so.12 + libcublas.so.12 + libcublasLt.so.12
-```
-
-## 4. Files to Create / Modify
-
-### New Files
-1. **`start.sh`** — Bash entry point equivalent to `start.bat`.
-   - Checks for portable Node.js and frontend build.
-   - Runs `scripts/setup.sh` if anything is missing.
-   - Starts `scripts/serve.cjs` with the portable Node.js binary.
-   - Opens browser (`xdg-open`) or prints URL.
-   - Handles graceful shutdown (traps SIGINT/SIGTERM to kill backend).
-
-2. **`scripts/setup.sh`** — Bash setup equivalent to `setup.ps1`.
-   - Downloads portable Node.js Linux tarball with `curl` + progress bar.
-   - Extracts with `tar`.
-   - Detects GPU via sysfs PCI vendor IDs.
-   - Downloads appropriate backends:
-     - **CPU + Vulkan** for everyone (baseline).
-     - **ROCm** if AMD GPU detected.
-     - **CUDA** if NVIDIA GPU detected (optional, due to large download).
-   - Downloads NVIDIA CUDA redist libraries if CUDA is selected.
-   - Extracts all archives using `tar` / Node.js / Python fallback.
-   - Renames binaries to predictable names.
-   - Runs `npm install` and `npm run build` using portable npm.
-
-3. **`scripts/reset.sh`** — Bash reset equivalent to `reset.ps1`.
-   - Deletes `app/tools/`, `app/backend/`, `app/dist/`, `app/frontend/node_modules/` while preserving `app/models/` and `app/outputs/`.
-
-### Modified Files
-4. **`scripts/serve.cjs`** — Enhance Linux support.
-   - **Add Linux backend paths**:
-     ```js
-     linuxCpu:    path.join(ROOT, "app", "backend", "linux", "cpu",    "sd-cpu"),
-     linuxVulkan: path.join(ROOT, "app", "backend", "linux", "vulkan", "sd-vulkan"),
-     linuxRocm:   path.join(ROOT, "app", "backend", "linux", "rocm",   "sd-rocm"),
-     linuxCuda:   path.join(ROOT, "app", "backend", "linux", "cuda",   "sd-cuda"),
-     ```
-   - **Add Linux GPU detection** via sysfs (`/sys/bus/pci/devices/*/vendor`).
-   - **Update `getBackendOptions()`**: For Linux, return `cpu`, `vulkan`, `rocm` (if binary exists), and `cuda` (if binary + libraries exist).
-   - **Update `selectBackendPath()`**: Choose Linux backend based on resolved type; set `LD_LIBRARY_PATH` for CUDA/ROCm subprocesses so they find bundled `.so` files.
-   - **Update `getHealth()` / `getSetupPaths()`**: Include Linux backend checks.
-   - **Update `getGpuInfo()` / `hasNvidiaGpu()`**: Add Linux sysfs parsing.
-   - **CUDA VRAM polling**: Keep existing `nvidia-smi` approach; on Linux it works if drivers are installed.
-
-5. **`README.md`** — Add Linux section with compatibility matrix, `./start.sh` usage, and known limitations.
-
-## 5. Download Sources
-
-| Component | URL Pattern |
-|---|---|
-| Portable Node.js | `https://nodejs.org/dist/v22.12.0/node-v22.12.0-linux-x64.tar.xz` |
-| CPU Backend | `https://github.com/leejet/stable-diffusion.cpp/releases/download/master-XXX/sd-master-XXX-bin-Linux-Ubuntu-24.04-x86_64.zip` |
-| Vulkan Backend | `https://github.com/leejet/stable-diffusion.cpp/releases/download/master-XXX/sd-master-XXX-bin-Linux-Ubuntu-24.04-x86_64-vulkan.zip` |
-| ROCm Backend | `https://github.com/leejet/stable-diffusion.cpp/releases/download/master-XXX/sd-master-XXX-bin-Linux-Ubuntu-24.04-x86_64-rocm-7.13.0.zip` |
-| CUDA Binary (third-party) | `https://github.com/leaxer-ai/leaxer-stable-diffusion/releases/download/v0.1.0/sd-x86_64-unknown-linux-gnu-cuda` |
-| CUDA Server (third-party) | `https://github.com/leaxer-ai/leaxer-stable-diffusion/releases/download/v0.1.0/sd-server-x86_64-unknown-linux-gnu-cuda` |
-| CUDA Runtime (NVIDIA redist) | `https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/linux-x86_64/cuda_cudart-linux-x86_64-12.X.YZ-archive.tar.xz` |
-| cuBLAS (NVIDIA redist) | `https://developer.download.nvidia.com/compute/cuda/redist/libcublas/linux-x86_64/libcublas-linux-x86_64-12.X.YZ-archive.tar.xz` |
-
-## 6. Setup Modes
-
-To respect users with limited bandwidth or no NVIDIA hardware, `setup.sh` should support two modes:
-
-1. **Quick Setup (default)** — Downloads CPU + Vulkan only.
-   - Works on all Linux systems.
-   - NVIDIA users get Vulkan acceleration (good, but not best).
-   - ~120–150 MB total download.
-
-2. **Maximum Performance Setup** — Downloads CPU + Vulkan + ROCm + CUDA.
-   - Best performance for each vendor.
-   - ~1.5 GB total download (mostly ROCm + cuBLAS).
-   - User can opt-in via `./start.sh --max-perf` or `./scripts/setup.sh --max-perf`.
-
-## 7. Self-Contained Checklist
-
-| Host Dependency | How We Eliminate It |
-|---|---|
-| Node.js installed globally | Download portable `node-v22.12.0-linux-x64.tar.xz` into `app/tools/node-linux/` |
-| npm installed globally | Use portable npm from the tarball |
-| `unzip` command | Extract ZIPs using Node.js or Python `zipfile` |
-| `lspci` / `lshw` | Parse `/sys/bus/pci/devices/*/vendor` directly |
-| C++ compiler / CUDA toolkit | Use pre-built binaries only |
-| System package manager (apt/yum) | Never used |
-| CUDA runtime libs | Bundle NVIDIA redistributable `.so` files |
-
-**One unavoidable host dependency remains**: the NVIDIA kernel driver (`libcuda.so.1`) for CUDA mode. Without it, CUDA cannot run. The app will detect this and automatically fall back to Vulkan for NVIDIA.
-
-## 8. Known Limitations
-
-1. **Third-party CUDA binary**: The Linux CUDA build comes from `leaxer-ai/leaxer-stable-diffusion`, which has only one release (v0.1.0) and may become stale. We should pin to a specific version and periodically evaluate alternatives.
-2. **Large ROCm download**: The official ROCm backend ZIP is ~1.2 GB, which is a significant one-time download.
-3. **Large cuBLAS download**: NVIDIA's redistributable cuBLAS package is ~400–500 MB.
-4. **NVIDIA driver requirement for CUDA**: `libcuda.so.1` must be present on the host (provided by the NVIDIA proprietary driver). If missing, CUDA mode falls back to Vulkan.
-5. **ROCm host driver requirement**: ROCm requires a compatible AMD GPU and ROCm kernel driver. If unavailable, ROCm falls back to Vulkan.
-6. **glibc compatibility**: Official Linux binaries are built on Ubuntu 24.04. They may fail on very old distributions (e.g., CentOS 7, Ubuntu 18.04) due to glibc version mismatches.
-
-## 9. Recommended Implementation Order
-
-1. Create `start.sh`, `scripts/setup.sh`, and `scripts/reset.sh` with CPU + Vulkan support first.
-2. Update `scripts/serve.cjs` with Linux paths, sysfs GPU detection, and backend selection.
-3. Add ROCm download path (behind `--max-perf` flag).
-4. Add CUDA download path using leaxer binary + NVIDIA redist libraries (behind `--max-perf` flag).
-5. Add fallback logic so that if ROCm/CUDA fails to initialize, the app automatically switches to Vulkan.
-6. Update `README.md` with Linux instructions and performance notes.
-7. Test on Linux machines with NVIDIA (CUDA/Vulkan), AMD (ROCm/Vulkan), Intel (Vulkan), and CPU-only configurations.

app/frontend/src/components/Settings.jsx

@@ -5,7 +5,7 @@ import {
   Monitor, HardDrive, MemoryStick, Thermometer, Hash, Layers,
   ChevronRight, Box, Wand2, Lightbulb, RotateCcw, Check, Palette
 } from "lucide-react";
-import { stopServer, formatBytes, getLlmStatus, startLlm, stopLlm } from "../services/api";
+import { stopServer, formatBytes, getLlmBackends, getLlmStats, getLlmStatus, benchmarkLlm, startLlm, stopLlm } from "../services/api";
 import { THEMES } from "../themes";
 
 const ASPECT_RATIOS = [
@@ -187,6 +187,9 @@ function Settings({
   setTheme,
 }) {
   const [llmStatus, setLlmStatus] = useState({ ready: false, settings: {} });
+  const [llmBackends, setLlmBackends] = useState({ available: [], candidates: [] });
+  const [llmStats, setLlmStats] = useState({ benchmarks: [] });
+  const [benchmarkBusy, setBenchmarkBusy] = useState(false);
 
   useEffect(() => {
     let cancelled = false;
@@ -195,6 +198,13 @@ function Settings({
         const status = await getLlmStatus();
         if (!cancelled) setLlmStatus(status);
       } catch (_) {}
+      try {
+        const [backends, stats] = await Promise.all([getLlmBackends(), getLlmStats()]);
+        if (!cancelled) {
+          setLlmBackends(backends);
+          setLlmStats(stats);
+        }
+      } catch (_) {}
     };
     fetchLlmStatus();
     const interval = setInterval(fetchLlmStatus, 3000);
@@ -247,6 +257,37 @@ function Settings({
     performanceProfile: settings?.performanceProfile,
   });
 
+  const handleBenchmarkTextBackend = async () => {
+    const model = llmStatus.settings?.model;
+    if (!model) {
+      showAlert({ title: "No Text Model Loaded", message: "Load a GGUF text model before benchmarking.", danger: true });
+      return;
+    }
+    setBenchmarkBusy(true);
+    try {
+      const result = await benchmarkLlm(model, {
+        contextSize: Math.min(2048, Number(textSettings?.contextSize) || 2048),
+        gpuLayers: textSettings?.gpuLayers ?? -1,
+        includeCpu: true,
+      });
+      const winner = result.winner;
+      showAlert({
+        title: winner ? "Benchmark Complete" : "Benchmark Finished",
+        message: winner
+          ? `Fastest backend: ${winner.backendMode} at ${Number(winner.predicted_per_second || 0).toFixed(1)} tokens/sec.`
+          : "Benchmark finished, but no backend returned usable timing data.",
+      });
+      const [status, backends, stats] = await Promise.all([getLlmStatus(), getLlmBackends(true), getLlmStats()]);
+      setLlmStatus(status);
+      setLlmBackends(backends);
+      setLlmStats(stats);
+    } catch (err) {
+      showAlert({ title: "Benchmark Failed", message: err.message || String(err), danger: true });
+    } finally {
+      setBenchmarkBusy(false);
+    }
+  };
+
   const handleThinkingToggle = async (enabled) => {
     const nextSettings = { ...textSettings, enableThinking: enabled };
 
@@ -924,6 +965,26 @@ function Settings({
                     GPU Layers: {llmStatus.settings?.gpuLayers === -1 ? "All" : llmStatus.settings?.gpuLayers}
                   </div>
                 )}
+                <div style={{ marginTop: "10px", fontSize: "0.78rem", lineHeight: "1.5" }}>
+                  Available: {(llmBackends.available || []).map((backend) => backend.key.toUpperCase()).join(", ") || "None"}<br />
+                  Fastest saved: {(() => {
+                    const model = llmStatus.settings?.model;
+                    const winner = (llmStats.benchmarks || []).find((item) => item.ok && (!model || item.model === model));
+                    return winner
+                      ? `${winner.backendMode} (${Number(winner.predicted_per_second || 0).toFixed(1)} tok/s)`
+                      : "Not benchmarked";
+                  })()}
+                </div>
+                <button
+                  className="m3-button secondary"
+                  type="button"
+                  onClick={handleBenchmarkTextBackend}
+                  disabled={benchmarkBusy || !llmStatus.ready}
+                  style={{ marginTop: "12px", width: "100%", justifyContent: "center" }}
+                >
+                  <Gauge size={15} />
+                  {benchmarkBusy ? "Benchmarking..." : "Benchmark Text Backends"}
+                </button>
               </div>
             </div>
           </div>

app/frontend/src/services/api.js

@@ -316,6 +316,32 @@ export async function getLlmStatus() {
   }
 }
 
+export async function getLlmBackends(refresh = false) {
+  const res = await fetch(`/api/llm/backends${refresh ? "?refresh=1" : ""}`);
+  return await readJsonResponse(res, "The local server returned invalid text backend data.");
+}
+
+export async function getLlmStats() {
+  const res = await fetch("/api/llm/stats");
+  return await readJsonResponse(res, "The local server returned invalid text runtime stats.");
+}
+
+export async function benchmarkLlm(model, options = {}) {
+  const res = await fetch("/api/llm/benchmark", {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify({
+      model,
+      backends: options.backends,
+      includeCpu: options.includeCpu,
+      prompt: options.prompt,
+      contextSize: options.contextSize,
+      gpuLayers: options.gpuLayers,
+    }),
+  });
+  return await readJsonResponse(res, "The local server returned invalid benchmark data.");
+}
+
 export async function listLlmModels() {
   const res = await fetch("/api/llm/models");
   const data = await readJsonResponse(res, "The local server returned invalid text model data.");
@@ -358,6 +384,7 @@ export async function startLlm(model, options = {}) {
       batchSize: options.batchSize,
       ubatchSize: options.ubatchSize,
       performanceProfile: options.performanceProfile,
+      preferredBackend: options.preferredBackend,
     }),
   });
   return await readJsonResponse(res, "The local server returned an invalid text backend response.");

linux.sh

@@ -129,12 +129,14 @@ fi
 CPU_SERVER_PATH="$APP_DIR/backend/linux/cpu/sd-server-cpu"
 VULKAN_SERVER_PATH="$APP_DIR/backend/linux/vulkan/sd-server-vulkan"
 LLM_CUDA_PATH="$APP_DIR/llm-backend/linux/cuda/llama-server"
+LLM_ROCM_PATH="$APP_DIR/llm-backend/linux/rocm/llama-server"
+LLM_SYCL_PATH="$APP_DIR/llm-backend/linux/sycl/llama-server"
 LLM_VULKAN_PATH="$APP_DIR/llm-backend/linux/vulkan/llama-server"
 LLM_CPU_PATH="$APP_DIR/llm-backend/linux/cpu/llama-server"
 if [[ ! -x "$CPU_BACKEND_PATH" || ! -x "$CPU_SERVER_PATH" ]] && [[ ! -x "$BACKEND_PATH" || ! -x "$VULKAN_SERVER_PATH" ]]; then
   SETUP_REASON="Linux backend binaries are missing or not executable."
 fi
-if [[ ! -x "$LLM_CUDA_PATH" && ! -x "$LLM_VULKAN_PATH" && ! -x "$LLM_CPU_PATH" ]]; then
+if [[ ! -x "$LLM_CUDA_PATH" && ! -x "$LLM_ROCM_PATH" && ! -x "$LLM_SYCL_PATH" && ! -x "$LLM_VULKAN_PATH" && ! -x "$LLM_CPU_PATH" ]]; then
   SETUP_REASON="Linux llama.cpp text backend is missing or not executable."
 fi
 

scripts/llm-backends.json

@@ -0,0 +1,26 @@
+{
+  "llamaRelease": "b9668",
+  "notes": "Portable llama.cpp backend manifest. Setup scripts may override the release with LLAMA_RELEASE or a script parameter.",
+  "windows": {
+    "cuda": "llama-{release}-bin-win-cuda-12.4-x64.zip",
+    "vulkan": "llama-{release}-bin-win-vulkan-x64.zip",
+    "sycl": "llama-{release}-bin-win-sycl-x64.zip",
+    "hip": "llama-{release}-bin-win-hip-x64.zip",
+    "cpu": "llama-{release}-bin-win-cpu-x64.zip",
+    "cudaRuntime": "cudart-llama-bin-win-cuda-12.4-x64.zip"
+  },
+  "linux": {
+    "cuda": "llama-{release}-bin-ubuntu-cuda-12.4-x64.tar.gz",
+    "rocm": "llama-{release}-bin-ubuntu-rocm-x64.tar.gz",
+    "vulkanX64": "llama-{release}-bin-ubuntu-vulkan-x64.tar.gz",
+    "vulkanArm64": "llama-{release}-bin-ubuntu-vulkan-arm64.tar.gz",
+    "syclFp32": "llama-{release}-bin-ubuntu-sycl-fp32-x64.tar.gz",
+    "syclFp16": "llama-{release}-bin-ubuntu-sycl-fp16-x64.tar.gz",
+    "cpuX64": "llama-{release}-bin-ubuntu-x64.tar.gz",
+    "cpuArm64": "llama-{release}-bin-ubuntu-arm64.tar.gz"
+  },
+  "mac": {
+    "arm64": "llama-{release}-bin-macos-arm64.tar.gz",
+    "x64": "llama-{release}-bin-macos-x64.tar.gz"
+  }
+}

scripts/serve.cjs

@@ -89,14 +89,22 @@ const LLM_BACKEND_PATHS = {
   winCuda: path.join(ROOT, "app", "llm-backend", "win", "cuda", "llama-server.exe"),
   winVulkan: path.join(ROOT, "app", "llm-backend", "win", "vulkan", "llama-server.exe"),
   winSycl: path.join(ROOT, "app", "llm-backend", "win", "sycl", "llama-server.exe"),
+  winHip: path.join(ROOT, "app", "llm-backend", "win", "hip", "llama-server.exe"),
   winCpu: path.join(ROOT, "app", "llm-backend", "win", "cpu", "llama-server.exe"),
   linuxCuda: path.join(ROOT, "app", "llm-backend", "linux", "cuda", "llama-server"),
+  linuxRocm: path.join(ROOT, "app", "llm-backend", "linux", "rocm", "llama-server"),
   linuxVulkan: path.join(ROOT, "app", "llm-backend", "linux", "vulkan", "llama-server"),
   linuxSycl: path.join(ROOT, "app", "llm-backend", "linux", "sycl", "llama-server"),
   linuxCpu: path.join(ROOT, "app", "llm-backend", "linux", "cpu", "llama-server"),
   macArm64: path.join(ROOT, "app", "llm-backend", "mac", "arm64", "llama-server"),
   macX64: path.join(ROOT, "app", "llm-backend", "mac", "x64", "llama-server"),
 };
+const LLM_CONFIG_DIR = path.join(ROOT, "app", "config");
+const LLM_MODEL_SETTINGS_PATH = path.join(LLM_CONFIG_DIR, "llm-model-settings.json");
+const LLM_BENCHMARK_PATH = path.join(LLM_CONFIG_DIR, "llm-benchmarks.json");
+if (!fs.existsSync(LLM_CONFIG_DIR)) {
+  fs.mkdirSync(LLM_CONFIG_DIR, { recursive: true });
+}
 const OPENVINO_MODELS = path.join(ROOT, "app", "openvino-models");
 if (!fs.existsSync(OPENVINO_MODELS)) {
   fs.mkdirSync(OPENVINO_MODELS, { recursive: true });
@@ -832,10 +840,12 @@ function getLlamaTelemetryBackendPath() {
   if (osPlatform === "win32") {
     if (fs.existsSync(LLM_BACKEND_PATHS.winVulkan)) return LLM_BACKEND_PATHS.winVulkan;
     if (fs.existsSync(LLM_BACKEND_PATHS.winSycl)) return LLM_BACKEND_PATHS.winSycl;
+    if (fs.existsSync(LLM_BACKEND_PATHS.winHip)) return LLM_BACKEND_PATHS.winHip;
     if (fs.existsSync(LLM_BACKEND_PATHS.winCuda)) return LLM_BACKEND_PATHS.winCuda;
   } else if (osPlatform === "linux") {
     if (fs.existsSync(LLM_BACKEND_PATHS.linuxVulkan)) return LLM_BACKEND_PATHS.linuxVulkan;
     if (fs.existsSync(LLM_BACKEND_PATHS.linuxSycl)) return LLM_BACKEND_PATHS.linuxSycl;
+    if (fs.existsSync(LLM_BACKEND_PATHS.linuxRocm)) return LLM_BACKEND_PATHS.linuxRocm;
     if (fs.existsSync(LLM_BACKEND_PATHS.linuxCuda)) return LLM_BACKEND_PATHS.linuxCuda;
   }
   return null;
@@ -876,6 +886,89 @@ function getLlamaVram() {
   return cachedLlamaVramInfo;
 }
 
+function readJsonFile(filePath, fallback) {
+  try {
+    if (!fs.existsSync(filePath)) return fallback;
+    return JSON.parse(fs.readFileSync(filePath, "utf8"));
+  } catch (_) {
+    return fallback;
+  }
+}
+
+function writeJsonFile(filePath, value) {
+  fs.mkdirSync(path.dirname(filePath), { recursive: true });
+  fs.writeFileSync(filePath, JSON.stringify(value, null, 2) + "\n", "utf8");
+}
+
+function getPersistedLlmModelSettings() {
+  return readJsonFile(LLM_MODEL_SETTINGS_PATH, { models: {} });
+}
+
+function savePersistedLlmModelSettings(settings) {
+  writeJsonFile(LLM_MODEL_SETTINGS_PATH, settings && typeof settings === "object" ? settings : { models: {} });
+}
+
+function getLlmModelSettings(filename) {
+  const safeFilename = path.basename(String(filename || ""));
+  if (!safeFilename) return {};
+  const settings = getPersistedLlmModelSettings();
+  return settings.models?.[safeFilename] || {};
+}
+
+function updateLlmModelSettings(filename, patch) {
+  const safeFilename = path.basename(String(filename || ""));
+  if (!safeFilename) return {};
+  const allSettings = getPersistedLlmModelSettings();
+  if (!allSettings.models || typeof allSettings.models !== "object") allSettings.models = {};
+  const previous = allSettings.models[safeFilename] || {};
+  const next = {
+    ...previous,
+    ...patch,
+    updatedAt: new Date().toISOString(),
+  };
+  allSettings.models[safeFilename] = next;
+  savePersistedLlmModelSettings(allSettings);
+  return next;
+}
+
+function getPersistedLlmBenchmarks() {
+  return readJsonFile(LLM_BENCHMARK_PATH, { results: [] });
+}
+
+function savePersistedLlmBenchmarks(benchmarks) {
+  writeJsonFile(LLM_BENCHMARK_PATH, benchmarks && typeof benchmarks === "object" ? benchmarks : { results: [] });
+}
+
+function recordLlmBenchmark(result) {
+  const benchmarks = getPersistedLlmBenchmarks();
+  const results = Array.isArray(benchmarks.results) ? benchmarks.results : [];
+  results.unshift({
+    ...result,
+    createdAt: new Date().toISOString(),
+  });
+  benchmarks.results = results.slice(0, 100);
+  savePersistedLlmBenchmarks(benchmarks);
+  return benchmarks.results[0];
+}
+
+function parseLlamaDeviceList(output) {
+  const devices = [];
+  const devicePattern = /(CUDA|Vulkan|SYCL|HIP|ROCm)\d*\s*:\s*(.+?)\s+\(([\d.]+)\s+MiB(?:,\s*([\d.]+)\s+MiB free)?\)/gi;
+  let match;
+  while ((match = devicePattern.exec(output || "")) !== null) {
+    const totalMiB = Number(match[3]);
+    const freeMiB = match[4] ? Number(match[4]) : NaN;
+    if (!Number.isFinite(totalMiB) || totalMiB <= 0) continue;
+    devices.push({
+      type: match[1],
+      name: match[2].trim(),
+      vram_gb: Math.round((totalMiB / 1024) * 100) / 100,
+      free_vram_gb: Number.isFinite(freeMiB) ? Math.round((freeMiB / 1024) * 100) / 100 : null,
+    });
+  }
+  return devices.sort((a, b) => b.vram_gb - a.vram_gb);
+}
+
 let cachedMacRamUsedGb = null;
 function pollMacRam() {
   if (osPlatform !== "darwin") return;
@@ -1307,43 +1400,120 @@ function getLlmBackend() {
   return getLlmBackendCandidates()[0] || null;
 }
 
-function getLlmBackendCandidates() {
-  const candidates = [];
-  const includeUndetectedGpu = process.env.LOCALAI_LLM_TRY_UNDETECTED_GPU === "1";
-  const add = (key, backendPath, mode, detected = true) => {
-    if (!backendPath || !fs.existsSync(backendPath)) return;
-    if (key !== "cpu" && !detected && !includeUndetectedGpu) return;
-    if (candidates.some((item) => item.path === backendPath)) return;
-    candidates.push({ key, path: backendPath, mode, detected });
-  };
+let cachedLlmBackendProbe = null;
+let cachedLlmBackendProbeAt = 0;
 
-  if (osPlatform === "win32") {
-    add("cuda", LLM_BACKEND_PATHS.winCuda, "Auto (CUDA/CPU)", Boolean(detectLlamaCudaGpu()));
-    add("vulkan", LLM_BACKEND_PATHS.winVulkan, "Auto (Vulkan/CPU)", Boolean(detectLlamaVulkanGpu()));
-    add("sycl", LLM_BACKEND_PATHS.winSycl, "Auto (SYCL/CPU)", Boolean(detectLlamaSyclGpu()));
-    add("cpu", LLM_BACKEND_PATHS.winCpu, "CPU", true);
-  } else if (osPlatform === "darwin") {
-    add("metal", process.arch === "arm64" ? LLM_BACKEND_PATHS.macArm64 : LLM_BACKEND_PATHS.macX64, process.arch === "arm64" ? "Metal GPU" : "CPU", true);
-  } else {
-    add("cuda", LLM_BACKEND_PATHS.linuxCuda, "Auto (CUDA/CPU)", Boolean(detectLlamaCudaGpu()));
-    add("sycl", LLM_BACKEND_PATHS.linuxSycl, "Auto (SYCL/CPU)", Boolean(detectLlamaSyclGpu()));
-    add("vulkan", LLM_BACKEND_PATHS.linuxVulkan, "Auto (Vulkan/CPU)", Boolean(detectLlamaVulkanGpu()));
-    add("cpu", LLM_BACKEND_PATHS.linuxCpu, "CPU", true);
+function resolveSystemLlamaServer() {
+  const configured = process.env.LOCALAI_LLM_SYSTEM_SERVER || process.env.LLAMA_SERVER_PATH || "";
+  if (configured && fs.existsSync(configured)) return configured;
+  return "";
+}
+
+function probeLlmBackend(backend) {
+  if (!backend?.path || !fs.existsSync(backend.path)) {
+    return {
+      ...backend,
+      installed: false,
+      detected: false,
+      devices: [],
+      error: "Backend binary not installed.",
+    };
   }
 
-  const priority = osPlatform === "win32"
-    ? { cuda: 0, vulkan: 1, sycl: 2, metal: 3, cpu: 9 }
-    : osPlatform === "darwin"
-      ? { metal: 0, cpu: 9 }
-      : { cuda: 0, sycl: 1, vulkan: 2, cpu: 9 };
+  try {
+    const result = spawnSync(backend.path, ["--list-devices"], {
+      encoding: "utf8",
+      windowsHide: true,
+      timeout: 10000,
+      maxBuffer: 1024 * 1024,
+    });
+    const output = `${result.stdout || ""}\n${result.stderr || ""}`;
+    const devices = parseLlamaDeviceList(output);
+    const detected = backend.key === "cpu" || backend.key === "metal" || backend.key === "system" || devices.length > 0;
+    return {
+      ...backend,
+      installed: true,
+      detected,
+      devices,
+      device: devices[0] || null,
+      error: result.status && !detected ? output.trim().slice(-600) : "",
+    };
+  } catch (err) {
+    return {
+      ...backend,
+      installed: true,
+      detected: backend.key === "cpu",
+      devices: [],
+      device: null,
+      error: err.message || String(err),
+    };
+  }
+}
 
-  return candidates.sort((a, b) => {
-    if (a.key === "cpu" && b.key !== "cpu") return 1;
-    if (b.key === "cpu" && a.key !== "cpu") return -1;
-    const detectedDelta = Number(b.detected) - Number(a.detected);
-    if (detectedDelta) return detectedDelta;
-    return (priority[a.key] ?? 8) - (priority[b.key] ?? 8);
-  });
+function getAllLlmBackendCandidates() {
+  const systemPath = resolveSystemLlamaServer();
+  const candidates = [];
+  const add = (key, backendPath, mode, portable = true) => {
+    if (!backendPath) return;
+    if (candidates.some((item) => item.path === backendPath)) return;
+    candidates.push({ key, path: backendPath, mode, portable });
+  };
+
+  if (systemPath) add("system", systemPath, "System llama.cpp", false);
+  if (osPlatform === "win32") {
+    add("cuda", LLM_BACKEND_PATHS.winCuda, "Auto (CUDA/CPU)");
+    add("hip", LLM_BACKEND_PATHS.winHip, "Auto (HIP/CPU)");
+    add("vulkan", LLM_BACKEND_PATHS.winVulkan, "Auto (Vulkan/CPU)");
+    add("sycl", LLM_BACKEND_PATHS.winSycl, "Auto (SYCL/CPU)");
+    add("cpu", LLM_BACKEND_PATHS.winCpu, "CPU");
+  } else if (osPlatform === "darwin") {
+    add("metal", process.arch === "arm64" ? LLM_BACKEND_PATHS.macArm64 : LLM_BACKEND_PATHS.macX64, process.arch === "arm64" ? "Metal GPU" : "CPU");
+  } else {
+    add("cuda", LLM_BACKEND_PATHS.linuxCuda, "Auto (CUDA/CPU)");
+    add("rocm", LLM_BACKEND_PATHS.linuxRocm, "Auto (ROCm/CPU)");
+    add("sycl", LLM_BACKEND_PATHS.linuxSycl, "Auto (SYCL/CPU)");
+    add("vulkan", LLM_BACKEND_PATHS.linuxVulkan, "Auto (Vulkan/CPU)");
+    add("cpu", LLM_BACKEND_PATHS.linuxCpu, "CPU");
+  }
+  return candidates;
+}
+
+function getLlmBackendProbe(force = false) {
+  const now = Date.now();
+  if (!force && cachedLlmBackendProbe && now - cachedLlmBackendProbeAt < 60000) return cachedLlmBackendProbe;
+
+  const probed = getAllLlmBackendCandidates().map(probeLlmBackend);
+  const priority = osPlatform === "win32"
+    ? { system: -1, cuda: 0, hip: 1, vulkan: 2, sycl: 3, metal: 4, cpu: 9 }
+    : osPlatform === "darwin"
+      ? { system: -1, metal: 0, cpu: 9 }
+      : { system: -1, cuda: 0, rocm: 1, sycl: 2, vulkan: 3, cpu: 9 };
+
+  const available = probed
+    .filter((item) => item.installed && item.detected)
+    .sort((a, b) => {
+      if (a.key === "cpu" && b.key !== "cpu") return 1;
+      if (b.key === "cpu" && a.key !== "cpu") return -1;
+      return (priority[a.key] ?? 8) - (priority[b.key] ?? 8);
+    });
+
+  cachedLlmBackendProbe = {
+    generatedAt: new Date().toISOString(),
+    candidates: probed,
+    available,
+    selected: available[0] || null,
+  };
+  cachedLlmBackendProbeAt = now;
+  return cachedLlmBackendProbe;
+}
+
+function getLlmBackendCandidates() {
+  const includeUndetectedGpu = process.env.LOCALAI_LLM_TRY_UNDETECTED_GPU === "1";
+  const probe = getLlmBackendProbe();
+  if (includeUndetectedGpu) {
+    return probe.candidates.filter((item) => item.installed);
+  }
+  return probe.available;
 }
 
 function getLlmModels() {
@@ -2647,13 +2817,37 @@ async function startLlm(settings = {}) {
   if (candidates.length === 0) {
     throw new Error("llama.cpp is not installed. Run the platform setup script to install the text backend.");
   }
+  const persistedSettings = getLlmModelSettings(filename);
+  const preferredBackend = String(settings.preferredBackend || persistedSettings.preferredBackend || "").toLowerCase();
+  const sortedCandidates = preferredBackend
+    ? [...candidates].sort((a, b) => {
+        if (a.key === preferredBackend && b.key !== preferredBackend) return -1;
+        if (b.key === preferredBackend && a.key !== preferredBackend) return 1;
+        return 0;
+      })
+    : candidates;
 
   const failures = [];
-  for (const backend of candidates) {
+  for (const backend of sortedCandidates) {
     for (const profile of buildLlmLoadProfiles(settings, backend)) {
       try {
         await startLlmWithBackend({ ...settings, __loadProfile: profile }, backend);
         llmSettings.backendFallbacks = failures;
+        updateLlmModelSettings(filename, {
+          preferredBackend: backend.key,
+          lastBackendMode: llmSettings.backendMode,
+          lastBackendBinary: llmSettings.backendBinary,
+          lastSettings: {
+            threads: llmSettings.threads,
+            contextSize: llmSettings.contextSize,
+            gpuLayers: llmSettings.gpuLayers,
+            cacheTypeK: llmSettings.cacheTypeK,
+            cacheTypeV: llmSettings.cacheTypeV,
+            batchSize: llmSettings.batchSize,
+            ubatchSize: llmSettings.ubatchSize,
+            flashAttn: llmSettings.flashAttn,
+          },
+        });
         return;
       } catch (err) {
         const message = err.message || String(err);
@@ -2833,8 +3027,9 @@ async function startLlmWithBackend(settings = {}, backend) {
   }
   
   // CPU mode: lock memory to prevent OS paging (critical for consistent speed)
-  const isGpuMode = backend.mode.includes("GPU") || backend.mode.includes("CUDA") || 
-                    backend.mode.includes("Vulkan") || backend.mode.includes("Metal") || 
+  const isGpuMode = backend.mode.includes("GPU") || backend.mode.includes("CUDA") ||
+                    backend.mode.includes("HIP") || backend.mode.includes("ROCm") ||
+                    backend.mode.includes("Vulkan") || backend.mode.includes("Metal") ||
                     backend.mode.includes("SYCL");
   if (!isGpuMode && llmSettings.mlock) {
     args.push("--mlock");                    // ✅ Prevent OS swapping on CPU
@@ -2902,6 +3097,7 @@ async function startLlmWithBackend(settings = {}, backend) {
     process.stderr.write("  [llm-err] " + output);
     if (/Vulkan\d+\s*:/i.test(output)) llmSettings.backendMode = "Vulkan GPU";
     else if (/CUDA\d+\s*:/i.test(output)) llmSettings.backendMode = "CUDA GPU";
+    else if (/(HIP|ROCm)\d*\s*:/i.test(output)) llmSettings.backendMode = "ROCm GPU";
     else if (/SYCL\d+\s*:/i.test(output)) llmSettings.backendMode = "SYCL GPU";
     else if (/Metal/i.test(output) && /GPU|device/i.test(output)) llmSettings.backendMode = "Metal GPU";
     else if (/\-\s+CPU\s+:/i.test(output) && llmSettings.backendMode.startsWith("Auto")) llmSettings.backendMode = "CPU";
@@ -3983,6 +4179,7 @@ const server = http.createServer(async (req, res) => {
 
   if (req.url === "/api/llm/status" && req.method === "GET") {
     const backend = getLlmBackend();
+    const probe = getLlmBackendProbe();
     return json(res, 200, {
       ready: llmReady,
       running: llmProc !== null,
@@ -3993,9 +4190,35 @@ const server = http.createServer(async (req, res) => {
       backendInstalled: Boolean(backend),
       backendMode: backend?.mode || "",
       backendPath: backend?.path || "",
+      selectedBackend: probe.selected,
+      availableBackends: probe.available,
     });
   }
 
+  if (req.url.startsWith("/api/llm/backends") && req.method === "GET") {
+    const parsed = new URL(req.url, `http://${req.headers.host || "localhost"}`);
+    return json(res, 200, { ok: true, ...getLlmBackendProbe(parsed.searchParams.get("refresh") === "1") });
+  }
+
+  if (req.url === "/api/llm/stats" && req.method === "GET") {
+    return json(res, 200, getLlmRuntimeStats());
+  }
+
+  if (req.url === "/api/llm/model-settings" && req.method === "GET") {
+    return json(res, 200, { ok: true, ...getPersistedLlmModelSettings() });
+  }
+
+  if (req.url === "/api/llm/model-settings" && req.method === "POST") {
+    const body = await readJsonBody(req, res);
+    if (!body) return;
+    const filename = path.basename(String(body.model || ""));
+    if (!filename) return json(res, 400, { ok: false, error: "model is required" });
+    const updated = updateLlmModelSettings(filename, {
+      preferredBackend: body.preferredBackend ? String(body.preferredBackend).toLowerCase() : undefined,
+    });
+    return json(res, 200, { ok: true, model: filename, settings: updated });
+  }
+
   if (req.url === "/api/llm/models" && req.method === "GET") {
     return json(res, 200, { models: getLlmModels() });
   }
@@ -4044,6 +4267,17 @@ const server = http.createServer(async (req, res) => {
     return json(res, 200, { ok: true });
   }
 
+  if (req.url === "/api/llm/benchmark" && req.method === "POST") {
+    const body = await readJsonBody(req, res);
+    if (!body) return;
+    try {
+      const result = await runExclusiveLlmOperation(() => benchmarkLlmModel(body));
+      return json(res, 200, result);
+    } catch (err) {
+      return json(res, 500, { ok: false, error: err.message || String(err) });
+    }
+  }
+
 function isOomError(msg) {
   if (!msg) return false;
   const lower = String(msg).toLowerCase();
@@ -4191,6 +4425,124 @@ async function doLlmChat(req, res, body, retryCount = 0) {
   }
 }
 
+function getLlmRuntimeStats() {
+  const benchmarks = getPersistedLlmBenchmarks();
+  return {
+    ok: true,
+    ready: llmReady,
+    running: llmProc !== null,
+    selectedBackend: getLlmBackendProbe().selected,
+    settings: llmSettings,
+    modelSettings: getPersistedLlmModelSettings(),
+    benchmarks: (benchmarks.results || []).slice(0, 20),
+  };
+}
+
+async function benchmarkLlmBackend(model, backend, baseSettings = {}) {
+  const prompt = String(baseSettings.prompt || "Reply with one short sentence about local AI performance.");
+  const startedAt = Date.now();
+  await startLlmWithBackend({
+    ...baseSettings,
+    model,
+    contextSize: Math.min(2048, Number(baseSettings.contextSize) || 2048),
+    gpuLayers: Number.isFinite(Number(baseSettings.gpuLayers)) ? Number(baseSettings.gpuLayers) : -1,
+    enableThinking: false,
+  }, backend);
+
+  const result = await requestJson(`http://127.0.0.1:${PORT_LLM}/v1/chat/completions`, {
+    model: llmSettings.model || "local-model",
+    messages: [
+      { role: "system", content: "You are a concise local benchmark assistant." },
+      { role: "user", content: prompt },
+    ],
+    temperature: 0.2,
+    max_tokens: 64,
+    stream: false,
+  }, 180000);
+  const finishedAt = Date.now();
+  const timings = result.timings || {};
+  return {
+    backendKey: backend.key,
+    backendMode: llmSettings.backendMode || backend.mode,
+    backendBinary: llmSettings.backendBinary || path.basename(backend.path),
+    model,
+    ok: true,
+    total_ms: finishedAt - startedAt,
+    prompt_ms: Number(timings.prompt_ms) || null,
+    predicted_ms: Number(timings.predicted_ms) || null,
+    predicted_n: Number(timings.predicted_n) || null,
+    predicted_per_second: Number(timings.predicted_per_second) || null,
+    prompt_per_second: Number(timings.prompt_per_second) || null,
+  };
+}
+
+async function benchmarkLlmModel(settings = {}) {
+  const filename = path.basename(String(settings.model || llmSettings.model || ""));
+  const modelPath = path.join(LLM_MODELS, filename);
+  if (!filename || !pathInside(modelPath, LLM_MODELS) || !fs.existsSync(modelPath)) {
+    throw new Error("Select a downloaded GGUF text model before benchmarking.");
+  }
+
+  const previous = {
+    ready: llmReady,
+    model: llmSettings.model,
+    settings: { ...llmSettings },
+  };
+  const requestedBackends = Array.isArray(settings.backends)
+    ? settings.backends.map((item) => String(item).toLowerCase())
+    : [];
+  const candidates = getLlmBackendCandidates()
+    .filter((backend) => backend.key !== "cpu" || settings.includeCpu !== false)
+    .filter((backend) => requestedBackends.length === 0 || requestedBackends.includes(backend.key));
+
+  if (candidates.length === 0) {
+    throw new Error("No available text backends were detected for benchmarking.");
+  }
+
+  const results = [];
+  for (const backend of candidates) {
+    try {
+      const result = await benchmarkLlmBackend(filename, backend, settings);
+      results.push(recordLlmBenchmark(result));
+    } catch (err) {
+      results.push(recordLlmBenchmark({
+        backendKey: backend.key,
+        backendMode: backend.mode,
+        backendBinary: path.basename(backend.path),
+        model: filename,
+        ok: false,
+        error: (err.message || String(err)).slice(-800),
+      }));
+      await killLlm();
+    }
+  }
+
+  const successful = results
+    .filter((item) => item.ok && Number(item.predicted_per_second) > 0)
+    .sort((a, b) => Number(b.predicted_per_second) - Number(a.predicted_per_second));
+  const winner = successful[0] || null;
+  if (winner) {
+    updateLlmModelSettings(filename, {
+      preferredBackend: winner.backendKey,
+      benchmarkWinner: {
+        backendMode: winner.backendMode,
+        predicted_per_second: winner.predicted_per_second,
+        createdAt: winner.createdAt,
+      },
+    });
+  }
+
+  if (previous.ready && previous.model && previous.model === filename && winner) {
+    try {
+      await startLlm({ ...previous.settings, model: filename, preferredBackend: winner.backendKey });
+    } catch (err) {
+      console.warn("  [llm] Failed to restore text model after benchmark:", err.message || err);
+    }
+  }
+
+  return { ok: true, model: filename, winner, results };
+}
+
 // ── llmfit Integration ──────────────────────────────────────────────────────────
 let cachedLlmfitResults = new Map();
 

scripts/setup-llama.ps1

@@ -140,6 +140,17 @@ function Install-LlamaArchive {
     Write-Host "   OK  llama.cpp $Variant backend installed."
 }
 
+function Try-InstallLlamaArchive {
+    param([string]$Variant, [string]$AssetName, [string]$Reason)
+
+    try {
+        Install-LlamaArchive -Variant $Variant -AssetName $AssetName
+    } catch {
+        Write-Host "   !!  Skipping optional llama.cpp $Variant backend: $Reason" -ForegroundColor Yellow
+        Write-Host "       $($_.Exception.Message)" -ForegroundColor DarkYellow
+    }
+}
+
 $hasNvidia = $false
 try {
     $gpus = Get-CimInstance Win32_VideoController -ErrorAction SilentlyContinue
@@ -157,7 +168,20 @@ if (-not $hasNvidia) {
 }
 
 if ($hasNvidia) {
-    Install-LlamaArchive -Variant "cuda" -AssetName "llama-$Release-bin-win-cuda-12.4-x64.zip"
+    Try-InstallLlamaArchive -Variant "cuda" -AssetName "llama-$Release-bin-win-cuda-12.4-x64.zip" -Reason "NVIDIA CUDA acceleration"
+}
+
+$hasAmd = $false
+try {
+    $gpus = Get-CimInstance Win32_VideoController -ErrorAction SilentlyContinue
+    foreach ($gpu in $gpus) {
+        if ($gpu.Name -like "*AMD*" -or $gpu.Name -like "*Radeon*") {
+            $hasAmd = $true
+        }
+    }
+} catch {}
+if ($hasAmd) {
+    Try-InstallLlamaArchive -Variant "hip" -AssetName "llama-$Release-bin-win-hip-x64.zip" -Reason "AMD HIP acceleration"
 }
 # Intel Arc/Graphics detection for SYCL backend
 $hasIntel = $false
@@ -170,7 +194,7 @@ try {
     }
 } catch {}
 if ($hasIntel) {
-    Install-LlamaArchive -Variant "sycl" -AssetName "llama-$Release-bin-win-sycl-x64.zip"
+    Try-InstallLlamaArchive -Variant "sycl" -AssetName "llama-$Release-bin-win-sycl-x64.zip" -Reason "Intel SYCL acceleration"
 }
 Install-LlamaArchive -Variant "vulkan" -AssetName "llama-$Release-bin-win-vulkan-x64.zip"
 Install-LlamaArchive -Variant "cpu" -AssetName "llama-$Release-bin-win-cpu-x64.zip"

scripts/setup-llama.sh

@@ -31,10 +31,39 @@ download_and_extract() {
 
   if [[ ! -x "$dest/llama-server" ]]; then
     echo "   XX   llama-server was not found after extracting $asset" >&2
-    exit 1
+    return 1
   fi
 }
 
+download_and_extract_optional() {
+  local asset="$1"
+  local dest="$2"
+  local reason="$3"
+
+  if [[ -x "$dest/llama-server" ]]; then
+    echo "   OK   optional llama.cpp backend already ready: $dest"
+    return 0
+  fi
+
+  if download_and_extract "$asset" "$dest"; then
+    return 0
+  fi
+
+  echo "   !!   Skipping optional llama.cpp backend for $reason ($asset)" >&2
+  rm -rf "$dest"
+  return 0
+}
+
+has_command() {
+  command -v "$1" >/dev/null 2>&1
+}
+
+has_linux_gpu_vendor() {
+  local vendor="$1"
+  [[ -d /sys/bus/pci/devices ]] || return 1
+  grep -Ril "$vendor" /sys/bus/pci/devices/*/vendor >/dev/null 2>&1
+}
+
 if [[ "$PLATFORM" == "Darwin" ]]; then
   if [[ "$ARCH" == "arm64" ]]; then
     download_and_extract "llama-$RELEASE-bin-macos-arm64.tar.gz" "$APP_DIR/llm-backend/mac/arm64"
@@ -46,6 +75,15 @@ elif [[ "$PLATFORM" == "Linux" ]]; then
     download_and_extract "llama-$RELEASE-bin-ubuntu-vulkan-arm64.tar.gz" "$APP_DIR/llm-backend/linux/vulkan"
     download_and_extract "llama-$RELEASE-bin-ubuntu-arm64.tar.gz" "$APP_DIR/llm-backend/linux/cpu"
   else
+    if has_command nvidia-smi || has_linux_gpu_vendor "0x10de"; then
+      download_and_extract_optional "llama-$RELEASE-bin-ubuntu-cuda-12.4-x64.tar.gz" "$APP_DIR/llm-backend/linux/cuda" "NVIDIA CUDA acceleration"
+    fi
+    if has_command rocminfo || has_linux_gpu_vendor "0x1002"; then
+      download_and_extract_optional "llama-$RELEASE-bin-ubuntu-rocm-x64.tar.gz" "$APP_DIR/llm-backend/linux/rocm" "AMD ROCm acceleration"
+    fi
+    if has_linux_gpu_vendor "0x8086"; then
+      download_and_extract_optional "llama-$RELEASE-bin-ubuntu-sycl-fp32-x64.tar.gz" "$APP_DIR/llm-backend/linux/sycl" "Intel SYCL acceleration"
+    fi
     download_and_extract "llama-$RELEASE-bin-ubuntu-vulkan-x64.tar.gz" "$APP_DIR/llm-backend/linux/vulkan"
     download_and_extract "llama-$RELEASE-bin-ubuntu-x64.tar.gz" "$APP_DIR/llm-backend/linux/cpu"
   fi

windows.bat

@@ -10,7 +10,9 @@ set SETUP=%~dp0scripts\setup.ps1
 set CUDA_BACKEND=%APP%\backend\win\cuda\sd-cuda.exe
 set VULKAN_BACKEND=%APP%\backend\win\vulkan\sd-vulkan.exe
 set LLM_CUDA_BACKEND=%APP%\llm-backend\win\cuda\llama-server.exe
+set LLM_HIP_BACKEND=%APP%\llm-backend\win\hip\llama-server.exe
 set LLM_VULKAN_BACKEND=%APP%\llm-backend\win\vulkan\llama-server.exe
+set LLM_SYCL_BACKEND=%APP%\llm-backend\win\sycl\llama-server.exe
 set LLM_CPU_BACKEND=%APP%\llm-backend\win\cpu\llama-server.exe
 set SERVE=%~dp0scripts\serve.cjs
 if "%FRONTEND_PORT%"=="" set FRONTEND_PORT=1420
@@ -32,7 +34,7 @@ if not exist "%DIST%" (
     set SETUP_REASON=Frontend build is missing.
     goto :run_setup
 )
-if not exist "%LLM_CUDA_BACKEND%" if not exist "%LLM_VULKAN_BACKEND%" if not exist "%LLM_CPU_BACKEND%" (
+if not exist "%LLM_CUDA_BACKEND%" if not exist "%LLM_HIP_BACKEND%" if not exist "%LLM_VULKAN_BACKEND%" if not exist "%LLM_SYCL_BACKEND%" if not exist "%LLM_CPU_BACKEND%" (
     set SETUP_REASON=llama.cpp text backend is missing.
     goto :run_setup
 )