# Platforms & runtimes
Source: https://geniex.aihub.qualcomm.com/en/get-started/platforms
Snapdragon platforms supported by GenieX and which runtime to pick on each.
GenieX runs exclusively on Qualcomm Snapdragon — no x86 or non-Snapdragon ARM build. To get going, pick the **Snapdragon platform** you'll run on, then pick a **runtime** that matches the model you want to run.
## **Snapdragon platforms**
GenieX is supported across three Snapdragon families — compute, mobile, and IoT — covering Windows ARM64, Android, and Linux ARM64.
### Supported chipsets
These are the chipsets GenieX is validated on. Each row lists the **SoC identifier** you'd see from auto-detection, and the **AI Hub chipset id** used when pulling Qualcomm AI Hub Models.
| Family | Chipset | SoC id | AI Hub chipset id |
| ------------------------------------------------------- | --------------------------------- | --------- | ----------------------------------- |
| **Compute** *(Windows ARM64 / Copilot+ PC)* | **Snapdragon® X Elite** | `X1E*` | `qualcomm-snapdragon-x-elite` |
| | **Snapdragon® X2 Elite** | `X2E*` | `qualcomm-snapdragon-x2-elite` |
| **Mobile** *(Android)* | **Snapdragon® 8 Elite** | `SM8750` | `qualcomm-snapdragon-8-elite` |
| | **Snapdragon® 8 Elite Gen 5** | `SM8850` | resolved from `SM8850` *(see note)* |
| **IoT** *(Linux ARM64 — cameras, robotics, industrial)* | **Qualcomm® Dragonwing™ IQ-9075** | `QCS9075` | `qualcomm-qcs9075` |
| | **Qualcomm® Dragonwing™ IQ-8275** | `QCS8275` | `qualcomm-qcs8275` |
**Chipset vs. SoC id.** Snapdragon X-series parts are identified by their Oryon CPU SKU (`X1E80100`, `X2E80100`, …); every part within a generation shares one NPU architecture, so **all X Elite SKUs map to the same AI Hub asset** — including the X Plus and X2 Plus parts. Android reports its SoC through `ro.soc.model`, and Dragonwing boards through the device tree.
On Android, GenieX passes the SoC id (e.g. `SM8850`) straight to Qualcomm AI Hub, which resolves it through its own alias table — GenieX deliberately keeps no second mapping. Pass the **SoC id**, not an AI Hub chipset name, and the right asset is selected. Variant suffixes are not exposed by `ro.soc.model`: a Galaxy S25 reports `SM8750`, not `SM8750-AC` (the alias for `qualcomm-snapdragon-8-elite-for-galaxy`), so pass the chipset explicitly if you need the variant asset.
GenieX **auto-detects the chipset** on Windows on Snapdragon, Dragonwing Linux, and Android. Check what it found, or set it explicitly:
```bash theme={"dark"}
geniex config get chipset # show the detected (or configured) chipset
geniex config set chipset # launch an interactive picker
```
**Android requires an explicit chipset** for Qualcomm AI Hub pulls — auto-detect via `ro.soc.model` covers the CLI/SDK path, but the Android SDK needs `ModelPullInput.chipset` set to `"SM8750"` or `"SM8850"`. See [Android API reference](/en/run/android/api-reference#modelpullinput).
### Interfaces per OS
| OS | Interfaces |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Windows ARM64** *(Compute / Copilot+ PC)* | - [CLI](/en/run/cli/quickstart)
- [Python SDK](/en/run/python/quickstart)
- [Local server](/en/run/cli/local-server)
|
| **Android** *(Mobile)* | - [Android SDK](/en/run/android/quickstart) (Kotlin, Maven Central)
|
| **Linux ARM64** *(Dragonwing IoT)* | - [Native install](/en/run/cli/install)
- [Docker](/en/run/linux/install)
|
The chipsets above are the validated set. GenieX may run on other Snapdragon parts within the same families — for everything Qualcomm AI Hub can compile for, see the [Qualcomm AI Hub device list](https://workbench.aihub.qualcomm.com/docs/hub/devices.html).
**No device on hand?** Sign in to [Qualcomm Developer Cloud (QDC)](https://qdc.qualcomm.com/) for remote sessions on Snapdragon X Elite / X2 Elite, Snapdragon 8 Elite / 8 Elite Gen 5, and Dragonwing IQ-9075. See the [QDC walkthrough in the FAQ](/en/resources/faq#qdc-qualcomm-device-cloud).
## **GenieX runtimes**
GenieX ships with two runtimes so you get both **broad model coverage** and **peak Snapdragon performance** in one stack:
* **`llama_cpp`** — any GGUF model on Hugging Face, running on Hexagon NPU, Adreno GPU, or CPU through Qualcomm's GGML Hexagon backend. The widest model selection.
* **`qairt`** ([Qualcomm® AI Engine Direct](https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct-sdk)) — pre-compiled bundles from [Qualcomm AI Hub](https://aihub.qualcomm.com/models/), compiled and quantized per chipset and pinned to the Hexagon NPU. The fastest path when your model is on Qualcomm AI Hub.
**Qualcomm AI Engine Direct** (also known as the *Qualcomm AI Engine Direct SDK*, *Qualcomm AI Runtime*, and historically *QAIRT*) is the official name. Throughout these docs we use the official name.
| | **llama.cpp** | **Qualcomm AI Engine Direct** |
| ---------------------------------------- | ---------------------------------------- | -------------------------------------------------- |
| **Model format** | GGUF (any community model) | Qualcomm AI Hub pre-compiled bundles |
| **Compute units** | NPU / GPU / CPU | NPU only |
| **Precisions (Quantizations) picked by** | You (`Q4_0`, `Q8_0`, `F16`, …) | Pre-quantized in the bundle |
| **Best for** | Bringing your own GGUF from Hugging Face | Highest NPU performance on Qualcomm® AI Hub Models |
Pick **`llama_cpp`** for any GGUF from Hugging Face, or when you need CPU/GPU fallback (e.g. IoT devices without HTP). Pick **`qairt`** for the fastest NPU path on models published to Qualcomm AI Hub.
### Defaults
If you don't pass a compute unit:
| Runtime | Default compute unit |
| ----------- | ------------------------ |
| `llama_cpp` | `npu` (pinned to `HTP0`) |
| `qairt` | `npu` |
For llama.cpp's HTP + CPU per-tensor scheduling (the faster path on Snapdragon), pass `hybrid` explicitly.
***
## **llama.cpp**
The `llama_cpp` runtime executes **any GGUF model** through llama.cpp with Qualcomm's GGML Hexagon backend. Pull any community GGUF from Hugging Face and run it on Snapdragon NPU, Adreno GPU, or pure CPU.
### Compute units
`--compute` maps to the underlying hardware as follows:
| Alias | Effect |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `npu` *(default)* | Pin to Hexagon NPU (`HTP0`). Best NPU-only path. |
| `gpu` | Adreno GPU via OpenCL. |
| `cpu` | Pure CPU. Forces `nGpuLayers = 0`. |
| `hybrid` | Empty `device_id` + `n_gpu_layers=-1` (all layers) — llama.cpp's per-tensor HTP+CPU scheduler. **The fast path on Snapdragon.** |
The precision you pick at `geniex pull` time also determines where the model lands — see [Precisions (Quantizations) Supported](/en/models/supported#precisions-quantizations-supported).
***
## **Qualcomm AI Engine Direct**
The `qairt` runtime executes pre-compiled bundles from **Qualcomm AI Hub** through [Qualcomm® AI Engine Direct](https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct-sdk). NPU-only, with the bundle compiled and quantized for a specific Snapdragon chipset — typically the fastest NPU path when your model is on Qualcomm AI Hub.
### Compute units
Qualcomm AI Engine Direct is **NPU only**.
| Alias | Effect |
| ----------------- | ------------------------------------------------ |
| `npu` *(default)* | Pin `HTP0` — the only supported path. |
| `cpu` / `gpu` | Coerced to `npu` with a warning. Never an error. |
### Runtime constraints
The bundle has its **precision, context length, and KV cache size baked in** — none can be changed at runtime. On Android, `nGpuLayers != 0` and `nCtx != 0` are rejected with `PARAM_NOT_SUPPORTED`; leave both at defaults and tune `max_tokens` / `enable_thinking` only. To change precision or context length, get a different bundle from [Qualcomm AI Hub](https://aihub.qualcomm.com/models/).
# Quickstart
Source: https://geniex.aihub.qualcomm.com/en/get-started/quickstart
Pick your interface and get to first inference in minutes.
Choose the interface that matches what you're building. Each path takes you from install to a working model in a few minutes.
Run models from a terminal on Windows ARM64, or via Docker on Linux ARM64. Best for trying things out fast.
OpenAI-compatible API on Windows ARM64 and Linux ARM64.
Huggingface style API for scripting and notebooks, available on Windows ARM64 and Linux ARM64.
Docker image for Linux ARM64 with NPU access. Best for Dragonwing IoT and similar platforms.
Kotlin SDK from Maven Central, plus a prebuilt demo APK for Snapdragon 8 Elite.
## **Before you start**
* **Platform & runtime** — pick where you'll run (Windows ARM64, Android, Linux ARM64) and which runtime fits your model. See [Platforms & runtimes](/en/get-started/platforms).
* **Models** — see [Models](/en/models/supported) for tested examples on each runtime.
* **Bring your own model** — already have a GGUF or AI Hub bundle on disk? See [Run a local model](/en/run/cli/quickstart#run-a-local-model).
# What is GenieX
Source: https://geniex.aihub.qualcomm.com/en/get-started/what-is-geniex
On-device AI inference runtime for Qualcomm Snapdragon — run frontier LLMs and VLMs across CLI, Python, Android, and Docker.
GenieX is an on-device Gen AI inference runtime built for Qualcomm platforms. It is the easiest way to run frontier language and vision-language models locally on Hexagon NPU, Adreno GPU, or CPU with a few lines of code. It is the community version of Qualcomm GENIE.
**GenieX is in developer preview.** We'd love your feedback — [open an issue](https://github.com/qualcomm/GenieX/issues) or [join Slack](https://aihub.qualcomm.com/community/slack).
## **Architecture**
![GenieX architecture stack: CLI, Python API, Java API, Docker, and Serve interfaces sit on the GenieX SDK, which dispatches to the llama.cpp runtime (GGML over CPU/GPU/HTP kernels) or the Qualcomm AI Engine Direct runtime on NPU. Targets Windows, Android, and Linux.]()
GenieX exposes **five entry points**, all over a single SDK:
* **CLI** — run and serve models straight from the terminal.
* **Python** — embed inference in your apps with the Python SDK.
* **Java/Kotlin** — the Android SDK for on-device mobile apps.
* **Docker** — a containerized image for reproducible deployments.
* **OpenAI-compatible server** — a drop-in local server for existing OpenAI clients.
Under the hood, that SDK dispatches to either the **llama.cpp runtime** (GGML kernels for CPU / GPU / Hexagon HTP) or the **[Qualcomm® AI Engine Direct](https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct-sdk) runtime** (NPU-only). The same SDK runs on Windows ARM64, Android, and Linux ARM64.
**Qualcomm AI Engine Direct** is the official name of what is also known as the *Qualcomm AI Engine Direct SDK*, *Qualcomm AI Runtime*, and *QAIRT*. Throughout these docs we use the official name.
## **Why two runtimes?**
So you get both **broad model coverage** and **optimal performance** in one stack:
* **Most models just work** — point GenieX at almost any GGUF on Hugging Face and it runs on CPU / GPU / NPU via llama.cpp.
* **Qualcomm® AI Hub Models run optimally** — models published to [Qualcomm AI Hub](https://aihub.qualcomm.com/) are pre-compiled per chipset and run through Qualcomm AI Engine Direct on the Hexagon NPU for peak on-device performance.
See [Platforms & runtimes](/en/get-started/platforms#geniex-runtimes) for when to pick which.
## **What you can do with GenieX**
* **Run models locally** on Snapdragon X (Windows ARM64), Snapdragon 8 Elite (Android), and Dragonwing IoT chipsets.
* **Pick a runtime** — `llama.cpp` for any community GGUF model, Qualcomm AI Engine Direct (`qairt`) for Qualcomm AI Hub pre-compiled NPU bundles.
* **Build apps** through the CLI, an OpenAI-compatible local server, the Python SDK, the Android SDK, or a Docker image.
## **Pick where to start**
Choose your interface and get to first inference in minutes.
Snapdragon platforms GenieX supports, and when to pick llama.cpp vs Qualcomm AI Engine Direct.
Tested LLMs and VLMs across the llama.cpp and Qualcomm AI Engine Direct runtimes.
## **Community**
File a bug, request a feature, or browse open issues on GitHub.
Collaborate with the GenieX team and other developers.
## **Legal**
GenieX is released under the BSD 3-Clause License.
Qualcomm site terms of use.
# Models
Source: https://geniex.aihub.qualcomm.com/en/models/supported
Where to find models, how to run them, and which precisions land on the Snapdragon NPU.
There are two places to get models, matching GenieX's two [runtimes](/en/get-started/platforms#geniex-runtimes):
* **[Qualcomm AI Hub](https://aihub.qualcomm.com/models/)** — pre-compiled bundles for Qualcomm AI Engine Direct, plus curated GGUF models for llama.cpp.
* **[Hugging Face](https://huggingface.co/models?library=gguf)** — any GGUF model for llama.cpp. See [Run a GGUF model from Hugging Face](#run-a-gguf-model-from-hugging-face).
## **Input modalities**
| Modality | Supported by | Notes |
| -------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Text | LLM, VLM | Every model. |
| Image | VLM | Any VLM (QAIRT bundle or GGUF with an image mmproj). |
| Audio | VLM (llama.cpp only) | A GGUF VLM whose mmproj carries a conformer encoder — e.g. `google/gemma-4-E2B-it-qat-q4_0-gguf`. QAIRT bundles do not support audio. See the [audio input tutorial](/en/tutorials/audio-input). |
A single VLM turn can mix text, image, and audio. See the [audio input tutorial](/en/tutorials/audio-input) for verified CLI, server, and Python examples.
## **Run a Qualcomm AI Hub Model**
For a model in the `ai-hub-models/*` namespace, just `geniex infer` it:
```powershell windows theme={"dark"}
geniex infer ai-hub-models/Qwen3-4B
```
Qualcomm AI Engine Direct is NPU-only. Pass `--compute npu` or omit the flag.
## **Run a GGUF model from Hugging Face**
Any compatible GGUF repo on Hugging Face works. Copy the repo path and pass it to `geniex infer`:
```powershell windows theme={"dark"}
geniex infer /-GGUF
```
For example:
```powershell windows theme={"dark"}
geniex infer unsloth/Phi-4-mini-instruct-GGUF
```
When prompted:
* **Model type** — `vlm` for vision-language models (Qwen3-VL family), `llm` otherwise.
* **Precision (Quantization)** — `Q4_0` for best Hexagon NPU support. See [Precisions (Quantizations) Supported](#precisions-quantizations-supported).
### Set up a Hugging Face token
Some models on Hugging Face are **gated** — you must accept the model's license on the Hugging Face website and provide an access token before GenieX can download them.
Go to [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens) and create a new token with **Read** access.
Visit the model's page on Hugging Face (e.g. `https://huggingface.co//`) and accept the license/agreement if prompted.
GenieX checks the following sources in order (first non-empty value wins):
1. `GENIEX_HFTOKEN` environment variable
2. `HF_TOKEN` environment variable
3. `~/.cache/huggingface/token` file
**Option A — environment variable (recommended):**
```powershell windows theme={"dark"}
# Standard HF variable (works with other HF tools too)
$env:HF_TOKEN = "hf_..."
# Or GenieX-specific (takes priority over HF_TOKEN)
$env:GENIEX_HFTOKEN = "hf_..."
```
```bash linux theme={"dark"}
# Standard HF variable (works with other HF tools too)
export HF_TOKEN="hf_..."
# Or GenieX-specific (takes priority over HF_TOKEN)
export GENIEX_HFTOKEN="hf_..."
```
**Option B — Hugging Face CLI login (persists to disk):**
```bash theme={"dark"}
pip install huggingface_hub
huggingface-cli login
```
This writes the token to `~/.cache/huggingface/token`, which GenieX reads automatically.
`GENIEX_HFTOKEN` takes highest priority, then `HF_TOKEN`, then the cached token file. Use `GENIEX_HFTOKEN` if you need a separate token for GenieX without affecting other Hugging Face tools.
## **Run a local Qualcomm AI Engine Direct bundle**
Bundles outside the `ai-hub-models/*` namespace — whether self-converted from Hugging Face or already sitting on disk — are imported once, then run like any other model.
Register the bundle with `geniex pull --local-path`, then `geniex infer` it.
**Self-converted from Hugging Face**
Make sure the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli) is installed, then download a bundle and pull it:
```powershell windows theme={"dark"}
# Set model name (must match the HF subfolder name exactly)
$model = "llama_v3_2_3b_instruct"
# Download the model from Hugging Face
hf download yichqian/geniex-qairt-models `
--local-dir $model `
--include "$model/**"
# Pull using the downloaded folder (absolute path)
geniex pull local/$model --local-path (Resolve-Path "$model\$model").Path
# Run inference
geniex infer local/$model
```
```bash linux theme={"dark"}
# Set model name (must match the HF subfolder name exactly)
model="llama_v3_2_3b_instruct"
# Download the model from Hugging Face
hf download yichqian/geniex-qairt-models \
--local-dir "$model" \
--include "$model/**"
# Pull using the downloaded folder (absolute path)
geniex pull "local/$model" --local-path "$(realpath "$model/$model")"
# Run inference
geniex infer "local/$model"
```
**Already on disk**
Point `--local-path` at the extracted bundle directory (containing `.bin` shards and `metadata.json`):
```powershell windows theme={"dark"}
geniex pull local/my-bundle --local-path C:\models\my-bundle
geniex infer local/my-bundle
```
```bash linux theme={"dark"}
geniex pull local/my-bundle --local-path /home/user/models/my-bundle
geniex infer local/my-bundle
```
You can also pull directly from an AI Hub `.zip` archive without extracting first:
```powershell windows theme={"dark"}
geniex pull local/my-bundle --local-path C:\downloads\model.zip
```
```bash linux theme={"dark"}
geniex pull local/my-bundle --local-path /home/user/downloads/model.zip
```
`geniex pull` copies model files into its local cache. After a successful pull, you can safely delete the original download to reclaim disk space. Use `geniex list` to confirm the model is cached.
Point `from_pretrained` at the bundle directory and force the Qualcomm AI Engine Direct runtime with `device_map="qairt"`:
```python theme={"dark"}
from geniex import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
r"C:\\Qwen3-4B-Instruct-2507", # any .bin file or the bundle folder
model_name="qwen3_4b_instruct_2507", # matches the Qualcomm AI Hub model id
device_map="qairt", # required for Qualcomm AI Engine Direct
)
```
See the [Python API reference](/en/run/python/api-reference) for all parameters.
Set `hub = HubSource.LOCALFS` and point `local_path` at an extracted AI Hub directory (`metadata.json` plus one or more `.bin` shards) or the AI Hub `.zip`. `pullFlow` imports it into the SDK cache (no network); no `chipset` is required — the bundle is already compiled for one. The modality (LLM vs VLM) is read from `metadata.json`.
```kotlin theme={"dark"}
ModelManagerWrapper.pullFlow(
ModelPullInput(
model_name = "local/qwen3-4b-2507",
hub = HubSource.LOCALFS,
local_path = "/data/local/tmp/Qwen3-4B-Instruct-2507", // extracted dir or .zip
)
).collect { /* Progress / Completed / Error */ }
```
Then load with `runtime_id = "qairt"` — identical to the downloaded-model flow:
```kotlin theme={"dark"}
val paths = ModelManagerWrapper.getPaths("local/qwen3-4b-2507")
?: error("Model not imported")
val llm = LlmWrapper.builder()
.llmCreateInput(
LlmCreateInput(
model_path = paths.model_path,
config = ModelConfig(),
runtime_id = "qairt",
compute_unit = null, // qairt is NPU-only
)
)
.build()
.getOrThrow()
```
The `model_name` is just the cache key — any `org/repo`-style string works; a `local/...` prefix is a common convention. You don't author a `geniex.json` for a local import — the model manager generates one during import. A QAIRT folder is recognized by `metadata.json` + `.bin` shards (or a `.zip`); a directory of loose `.bin` files with no `metadata.json` won't be detected. When importing models of mixed types, prefer `paths.runtime_id` (authoritative) over hard-coding `runtime_id`.
## **Run a local GGUF model**
A local GGUF model is a directory (or file) holding your `.gguf` weights — side-loaded, produced by another tool, or already on disk. Import it once, then run it like any other model.
Point `--local-path` at the directory containing your `.gguf` file:
```powershell windows theme={"dark"}
geniex pull local/my-model --local-path C:\models\my-model
geniex infer local/my-model
```
```bash linux theme={"dark"}
geniex pull local/my-model --local-path /home/user/models/my-model
geniex infer local/my-model
```
Pass the path to a local `.gguf` file straight to `from_pretrained`:
```python theme={"dark"}
from geniex import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
r"C:\models\my-model\model.gguf", # path to a local .gguf file
device_map="auto", # auto -> npu for llama_cpp
)
```
See the [Python API reference](/en/run/python/api-reference) for all parameters.
Set `hub = HubSource.LOCALFS` and point `local_path` at a directory containing the `.gguf` file(s). A `geniex.json` manifest is used if present; otherwise the layout is inferred from the file names. For a VLM, drop the `mmproj-*.gguf` into the same directory.
```kotlin theme={"dark"}
ModelManagerWrapper.pullFlow(
ModelPullInput(
model_name = "local/qwen3-0.6b",
hub = HubSource.LOCALFS,
local_path = "/data/local/tmp/qwen3-0.6b", // dir containing *.gguf
)
).collect { /* Progress / Completed / Error */ }
```
Then load with `runtime_id = "llama_cpp"` — identical to the downloaded-model flow:
```kotlin theme={"dark"}
val paths = ModelManagerWrapper.getPaths("local/qwen3-0.6b")
?: error("Model not imported")
val llm = LlmWrapper.builder()
.llmCreateInput(
LlmCreateInput(
model_path = paths.model_path,
config = ModelConfig(nCtx = 4096),
runtime_id = "llama_cpp",
compute_unit = null, // null → NPU on Snapdragon (recommended)
)
)
.build()
.getOrThrow()
```
## **Precisions (Quantizations) Supported**
Which precision you pick determines where the model runs.
### llama.cpp
llama.cpp accepts the full range of GGML quantization formats. The CLI prompts you to pick one when you `geniex pull` a GGUF model:
```powershell theme={"dark"}
Choose a precision version to download
> Q4_0 [1.2 GiB] (default)
Q8_0 [2.0 GiB]
F16 [3.8 GiB]
```
| Precision (Quantization) | Runs on | Notes |
| ------------------------ | --------------- | ------------------------------------------------------------ |
| **`Q4_0`** *(default)* | **Hexagon NPU** | Best NPU support in llama.cpp. Recommended for most models. |
| `Q8_0` | GPU / CPU | Better quality at \~2× the disk and memory cost. |
| `F16` | GPU / CPU | Reference precision. Mainly for evaluation — large and slow. |
| `Q4_K_M`, `Q5_K_M`, etc. | GPU / CPU | Mixed-precision K-quants. Not optimized for Hexagon NPU. |
Stick with `Q4_0` if you want the model to land on the Hexagon NPU. Other precisions will work but typically run on GPU or CPU.
### Qualcomm AI Engine Direct
Qualcomm AI Engine Direct bundles are **statically pre-quantized** — there is no precision choice at runtime. To use a different precision, get a different bundle from [Qualcomm AI Hub](https://aihub.qualcomm.com/models/).
| Precision (Quantization) | Runs on | Notes |
| --------------------------- | --------------- | ------------------------------------------------------------------------------------------------- |
| **`w4a16`** *(most common)* | **Hexagon NPU** | Weights int4, activations int16. Best balance of size and accuracy for on-device LLMs. |
| `w4` | **Hexagon NPU** | Weights int4, activations float. Slightly higher accuracy than w4a16 at the cost of more compute. |
Most pre-built bundles on Qualcomm AI Hub use `w4a16`. The quantization level is baked into the bundle at compile time — pick the bundle that matches your quality/performance target.
For other bundle constraints (context length, KV cache, Android `nCtx`), see [Qualcomm AI Engine Direct runtime constraints](/en/get-started/platforms#runtime-constraints).
# FAQ
Source: https://geniex.aihub.qualcomm.com/en/resources/faq
Common questions about GenieX, its runtimes, and supported chipsets.
## **General**
A multi-platform AI inference SDK built for Qualcomm Snapdragon. GenieX runs frontier LLMs and VLMs on-device — Hexagon NPU, Adreno GPU, or CPU compute units — across Windows ARM64, Android, and Linux ARM64. See [What is GenieX](/en/get-started/what-is-geniex).
* **Trying things out, scripting** — [CLI](/en/run/cli/install) on Windows ARM64 or Linux ARM64.
* **Building an app** — [Local server](/en/run/cli/local-server) (OpenAI-compatible HTTP) or [Python SDK](/en/run/python/install).
* **Mobile** — [Android SDK](/en/run/android/install) (Kotlin, Maven Central).
* **Reproducible IoT deployment** — [Docker image](/en/run/cli/install) on Linux ARM64 EVKs, pinned to a release tag.
The [Quickstart](/en/get-started/quickstart) is a router into all four.
[Supported platforms](/en/get-started/platforms) — Snapdragon X on Windows ARM64, Snapdragon 8 Elite on Android, Dragonwing QCS9075 on Linux ARM64.
## **Runtimes**
* **llama.cpp** — runs any GGUF model, supports NPU/GPU/CPU compute units. Best for trying community models.
* **[Qualcomm® AI Engine Direct](https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct-sdk)** (`qairt`) — runs pre-compiled Qualcomm AI Hub Models, NPU only. Typically the fastest path when your model is on Qualcomm AI Hub.
See [Platforms & runtimes](/en/get-started/platforms#geniex-runtimes).
If you don't pass a compute unit, both `llama_cpp` and `qairt` default to `npu`. For llama\_cpp's faster HTP + CPU per-tensor scheduler path, pass `hybrid` explicitly.
For llama.cpp on Snapdragon NPU: **`Q4_0`**. It has the best Hexagon NPU support. Qualcomm AI Hub Models are pre-quantized — no choice. See [Precisions (Quantizations) Supported](/en/models/supported#precisions-quantizations-supported).
## **Models**
Any **GGUF** model on the llama.cpp runtime. See [Run a GGUF model from Hugging Face](/en/models/supported#run-a-gguf-model-from-hugging-face).
Qualcomm AI Engine Direct requires a pre-compiled Qualcomm AI Hub Model — adding a new one means registering it on the C++ side first.
Some Hugging Face models require you to accept a license and authenticate. Set one of these environment variables:
* **Windows:** `$env:HF_TOKEN = "hf_..."` (or `$env:GENIEX_HFTOKEN = "hf_..."`)
* **Linux:** `export HF_TOKEN="hf_..."` (or `export GENIEX_HFTOKEN="hf_..."`)
Or run `huggingface-cli login` to persist the token to `~/.cache/huggingface/token`.
Priority: `GENIEX_HFTOKEN` > `HF_TOKEN` > token file. Get your token at [huggingface.co/settings/tokens](https://huggingface.co/settings/tokens). For full setup steps, see [Set up a Hugging Face token](/en/models/supported#set-up-a-hugging-face-token).
[Models](/en/models/supported) — split by runtime (Qualcomm AI Engine Direct, llama.cpp).
## **Chipsets & devices**
Yes. GenieX targets Qualcomm Snapdragon chipsets — Hexagon NPU, Adreno GPU, and Snapdragon ARM CPU compute units. There's no x86 or non-Snapdragon ARM build. If you don't have a device on hand, use [Qualcomm Developer Cloud / Device Cloud](/en/get-started/platforms).
No. Qualcomm AI Engine Direct is NPU-only by design. `cpu` and `gpu` aliases are coerced to NPU with a warning. Use the llama.cpp runtime (with a GGUF) if you need CPU/GPU fallback.
## **QDC (Qualcomm Device Cloud)**
No. Sign in to [Qualcomm Developer Cloud](https://qdc.qualcomm.com/), pick a Snapdragon device, and start an Interactive Session — that's the whole flow. See [Supported platforms](/en/get-started/platforms) for which chipsets are covered.
In the QDC portal, navigate to **Interactive Sessions** and click **Connect** to view the SSH connection instructions.
![QDC portal showing the Connect button for SSH instructions]()
Open a terminal and run the SSH tunnel command from QDC, replacing `` with your private key path.
![SSH tunneling setup with private key and port forwarding]()
**`WARNING: UNPROTECTED PRIVATE KEY FILE!`** — your `.pem` file permissions are too open. SSH requires private keys to be readable only by the owner:
```bash theme={"dark"}
chmod 600 /path/to/your-key.pem
```
In a new terminal, SSH into the forwarded port. The default password is `oelinux123`:
```bash theme={"dark"}
ssh -o StrictHostKeychecking=no -o UserKnownHostsFile=/dev/null -p 2222 root@localhost
```
![Successful SSH connection to the QDC device]()
Use a **Snapdragon 8 Elite** or **8 Elite Gen 5** Interactive Session on QDC and install the demo APK you build from source.
Build the sample app from [`qualcomm/ai-hub-apps`](https://github.com/qualcomm/ai-hub-apps/blob/release/geniex_chat_android/README.md) in Android Studio (**Build → Build APK(s)**) to produce a `.apk`.
Sign in to [Qualcomm Device Cloud](https://qdc.qualcomm.com/), choose **Snapdragon 8 Elite** or **Snapdragon 8 Elite Gen 5**, and select **Interactive session**.
Enable **Wi-Fi** (so the demo can download models) and **Keep screen on** (so the screen doesn't lock mid-download). SSH is **not** required.
Use **Upload file** in the session setup page to push your built `.apk` onto the device — the file must be uploaded **before** the session begins.
Start the session. In the mirrored screen, tap the uploaded APK to install, then open **GenieX Demo** from the app drawer.
Upload a test image via the QDC terminal:
```bash theme={"dark"}
curl -L "https://s7d1.scene7.com/is/image/dmqualcommprod/Qualcomm_AIHub_image2-1?$QC_Responsive$&fmt=png-alpha" -o /data/local/tmp/qualcomm.png
cp /data/local/tmp/qualcomm.png /sdcard/Download/
```
The image will appear in the app's image picker.
## **Python**
GenieX requires **ARM64** Python. x86\_64 / AMD64 builds are not supported, even under emulation.
QDC devices (Qualcomm embedded Linux / Yocto) ship with ARM64 Python pre-installed — verify with `python3 --version`. If your device does not have Python, use one of the methods below.
**Option 1 — [Miniconda](https://docs.anaconda.com/miniconda/) (recommended; works on Yocto and Ubuntu ARM64)**
```bash theme={"dark"}
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh
# CONDA_OVERRIDE_GLIBC: Yocto images often ship without `ldd`, which makes the
# installer's glibc probe fail with "Installer requires GLIBC >=2.28, but system has .".
# The override skips the probe; the real glibc on Qualcomm Linux 1.7+ is already 2.39.
CONDA_OVERRIDE_GLIBC=2.39 bash Miniconda3-latest-Linux-aarch64.sh -b -p $HOME/miniconda
eval "$($HOME/miniconda/bin/conda shell.bash hook)"
# Newer conda releases require explicit ToS acceptance for the default channels
# in non-interactive mode; without these, `conda create` errors out.
conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/main
conda tos accept --override-channels --channel https://repo.anaconda.com/pkgs/r
conda create -n geniex python=3.13 -y
conda activate geniex
python --version
```
**Option 2 — apt (Ubuntu ARM64 only)**
```bash theme={"dark"}
sudo apt update && sudo apt install -y python3 python3-pip python3-venv
python3 --version
```
QDC Yocto images do not ship `apt` — use Miniconda above.
No. There is currently no native Windows ARM64 Miniconda installer. On Windows ARM64, download the official [Python 3.13.3 ARM64 installer](https://www.python.org/ftp/python/3.13.3/python-3.13.3-arm64.exe) from python.org — see [Python Install](/en/run/python/install). Do not install the x86 / AMD64 build: GenieX wheels only target ARM64.
## **Server / API**
Yes — see [Local server](/en/run/cli/local-server). You can point the official `openai` Python client at `http://127.0.0.1:18181/v1` and reuse OpenAI code unchanged.
No. Pull models with `geniex pull` before starting the server.
# Troubleshooting
Source: https://geniex.aihub.qualcomm.com/en/resources/troubleshooting
Common errors and how to fix them.
## **Install / pip**
Common on networks with TLS inspection — including **Qualcomm Developer Cloud (QDC)**. Pre-download the SDK and point pip at the local file. See the full PowerShell snippet in [Python install](/en/run/python/install#install-via-pip).
The `.exe` is not yet code-signed. Click **More info → Run anyway** in the SmartScreen dialog.
## **CLI**
The installer doesn't add itself to `PATH`. Run:
```powershell theme={"dark"}
Set-Alias geniex (where.exe geniex)
```
Qualcomm AI Engine Direct is NPU-only. Use `--compute npu` (or omit the flag — `npu` is the default for `qairt`). To run on CPU/GPU, switch to a GGUF model on the [llama.cpp runtime](/en/get-started/platforms#llamacpp).
The conversation outgrew the context window (`--nctx`, default 4096).
* **llama.cpp (GGUF):** raise it at runtime, e.g. `geniex infer --nctx 8192`, up to the model's trained maximum. A larger window uses more memory.
* **Qualcomm AI Engine Direct (NPU):** the window is fixed in the compiled bundle and `--nctx` has no effect. Add `--sliding-window` to keep chatting (evicts the oldest context), or pull a bundle built for a longer context.
See [Increasing the context length](/en/run/cli/reference#increasing-the-context-length).
## **Server**
The server doesn't auto-download. Pull the model first:
```bash theme={"dark"}
geniex pull ai-hub-models/Qwen3-4B-Instruct-2507
```
Then restart `geniex serve`.
The `--privileged` flag is required for NPU access. Make sure your `docker run` includes it, plus the volume mounts for `/usr/lib`. See [CLI install (Docker)](/en/run/cli/install).
## **Linux**
Two separate things have to be right:
1. **Registry login.** Docker Hub (`docker.io/qualcomm/geniex`) is public and needs no login. For the Qualcomm Container Registry, log in first:
```bash bash theme={"dark"}
# Qualcomm Container Registry:
docker login docker-registry.qualcomm.com -u '$app' -p GB2S6KXMJXTPV8VHNFNS7Q6LVH75LOOBTLT8D723WUX6PSFZMTX95GIQG4EFWH5C021ONZ5763VI9IDHU96Q7VAZJ2830CLX3NPI6STQOJWRYXLLA2ZYTL1S
```
You should see `Login Succeeded`.
2. **Docker group membership.** If `docker pull` returns `permission denied while trying to connect to the docker API at unix:///var/run/docker.sock`, your user isn't in the `docker` group:
```bash bash theme={"dark"}
sudo usermod -aG docker $USER
newgrp docker # apply the new group in the current shell
```
Then retry the pull.
The container can't reach the NPU. Make sure your `docker run` includes `--privileged` and the `/usr/lib` mount, and that the host's Qualcomm driver packages (`qcom-adreno1`, `qcom-fastrpc1`) are installed — see [Linux install → Install host dependencies](/en/run/linux/install#install-host-dependencies).
The llama.cpp Hexagon backend `dlopen`s the **unversioned** `libcdsprpc.so`, but `qcom-fastrpc1` only ships `libcdsprpc.so.1`. On bare metal `install.sh` creates the symlink; if you deployed the release tarball directly (no `install.sh`), create it yourself:
```bash bash theme={"dark"}
sudo ln -sf /usr/lib/aarch64-linux-gnu/libcdsprpc.so.1 /usr/lib/aarch64-linux-gnu/libcdsprpc.so
sudo ln -sf /usr/lib/aarch64-linux-gnu/libadsprpc.so.1 /usr/lib/aarch64-linux-gnu/libadsprpc.so
sudo ldconfig
```
If the Hexagon driver can't load (see the entry above), `hybrid` still produces correct output — it just runs entirely on CPU, so the failure is silent. Confirm the NPU is actually engaged by running with `GGML_HEX_VERBOSE=1`; you should see `Hexagon Arch version vNN` and a `libggml-htp-vNN.so` session. No such lines means it fell back to CPU.
The aarch64 Linux build is compiled for **armv8.2-a** with the `fp16`, `dotprod`, `lse` (atomics), and `rdm` extensions. Baseline armv8.0 boards — some Dragonwing IoT SoCs with no NPU — don't implement these, so `geniex` stops at startup with a clear error rather than crashing with a raw `SIGILL: illegal instruction` partway through a run.
This check is global: every backend needs those instructions, so switching `--compute` to `cpu`, `gpu`, or `npu` won't help on such a device. Check what your CPU reports:
```bash bash theme={"dark"}
LD_SHOW_AUXV=1 /bin/true | grep AT_HWCAP # look for atomics, asimdrdm, asimddp, fphp, asimdhp
cat /proc/cpuinfo | grep Features # same features, under the kernel's names
```
If those features are absent, the device can't run the current build. Report it in [GitHub Issues](https://github.com/qualcomm/GenieX/issues) or Slack with the output above.
## **Android**
The demo (or your code) is passing the **raw user text** into `generateStreamFlow` instead of the chat-templated prompt. Qualcomm AI Engine Direct pipelines treat their input as already-templated — pass `applyChatTemplate().formattedText`, not the raw user message.
Android needs an explicit `chipset` for Qualcomm AI Hub pulls — auto-detect only runs on Windows on Snapdragon. Set `ModelPullInput.chipset` to `"SM8750"` (Snapdragon 8 Elite) or `"SM8850"` (Snapdragon 8 Elite Gen 5). See [Android API reference → ModelPullInput](/en/run/android/api-reference#modelpullinput).
The model id returned by Qualcomm AI Hub must match an entry in the Qualcomm AI Engine Direct runtime's registry (`qwen3_4b_instruct_2507`, `qwen2_5_vl_7b_instruct`, etc.). To add a new Qualcomm AI Hub Model, register it on the C++ side first — see `third-party/geniex-qairt/models/{llm,vlm}_model_registry.h`.
Qualcomm AI Hub Models are compiled with fixed KV cache and context length. Leave both `nGpuLayers` and `nCtx` at their defaults; tune `max_tokens` and `enable_thinking` instead.
## **Still stuck?**
File a bug, request a feature, or browse open issues.
Developer collaboration.
# API reference
Source: https://geniex.aihub.qualcomm.com/en/run/android/api-reference
GenieX Android SDK — runtime / compute-unit selection, model management, and inference APIs for LLM and VLM.
## **Runtime & compute unit selection**
### Runtime
Choose the inference runtime via `runtime_id`:
```kotlin theme={"dark"}
val runtime_id: String? // "llama_cpp" | "qairt" | null
```
| `runtime_id` | Runtime | Model format | Compute units |
| ------------- | -------------------------------- | --------------------------------- | ------------------------------ |
| `"llama_cpp"` | llama.cpp + GGML Hexagon backend | GGUF | CPU / Adreno GPU / Hexagon NPU |
| `"qairt"` | Qualcomm® AI Engine Direct | Qualcomm AI Hub pre-compiled bins | Hexagon NPU only |
| `null` | SDK picks based on model paths | — | — |
Constants are exposed as [`RuntimeIdValue`](https://github.com/qualcomm/GenieX/blob/main/bindings/android/app/src/main/java/com/geniex/sdk/bean/InputPluginBase.kt) (`LLAMA_CPP`, `QAIRT`).
### Compute unit
Friendly compute-unit aliases forwarded to `geniex_resolve_device` in the native SDK.
```kotlin theme={"dark"}
val compute_unit: String? // "cpu" | "gpu" | "npu" | null
```
| Alias | Effect |
| ------- | ----------------------------------------------------------- |
| `null` | Runtime default — `npu` for `llama_cpp`, `npu` for `qairt`. |
| `"npu"` | Hexagon NPU acceleration. Recommended on Snapdragon. |
| `"gpu"` | Adreno GPU via OpenCL (`llama_cpp` only). |
| `"cpu"` | Pure CPU. Forces `nGpuLayers = 0`. |
Qualcomm AI Engine Direct only supports NPU. Passing `"cpu"` or `"gpu"` with a Qualcomm AI Hub Model logs a warning and falls back to NPU — it won't error.
***
## **Model manager**
Models are pulled on-device through the bundled Rust model manager. Do **not** manually `adb push` weights — use `ModelManagerWrapper`.
### `ModelManagerWrapper`
```kotlin theme={"dark"}
// Init (idempotent — safe to call on every Activity.onCreate).
GenieXSdk.getInstance().init(context)
// Pull with streaming progress.
ModelManagerWrapper.pullFlow(
ModelPullInput(
model_name = "unsloth/Qwen3-0.6B-GGUF",
precision = "Q4_0",
hub = HubSource.HUGGINGFACE,
)
).collect { event ->
when (event) {
is ModelManagerWrapper.PullEvent.Progress -> /* update UI */
ModelManagerWrapper.PullEvent.Completed -> /* done */
is ModelManagerWrapper.PullEvent.Error -> /* show error */
}
}
// Resolve on-disk paths for a previously-pulled model.
val paths: ModelPaths? = ModelManagerWrapper.getPaths("unsloth/Qwen3-0.6B-GGUF")
// Inventory / cleanup.
ModelManagerWrapper.list() // List
ModelManagerWrapper.remove("org/repo") // 0 = success
ModelManagerWrapper.clean() // wipe all cached models
```
### `ModelPullInput`
```kotlin theme={"dark"}
data class ModelPullInput(
val model_name: String, // "org/repo" or alias
val precision: String? = null, // precision (quantization) e.g. "Q4_0", "Q4_K_M"
val hub: HubSource = HubSource.AUTO, // AUTO routes by model_name
val local_path: String? = null, // only when hub == LOCALFS
val hf_token: String? = null, // falls back to GENIEX_HFTOKEN env
val chipset: String? = null, // required for Qualcomm AI Hub on Android (e.g. "SM8750")
val display_name: String? = null,
)
```
### `HubSource`
```kotlin theme={"dark"}
enum class HubSource(val value: Int) {
AUTO(0), // routes by prefix (e.g. ai-hub-models/* → AIHUB)
HUGGINGFACE(1),
MODELSCOPE(2),
AIHUB(3),
VOLCES(4),
LOCALFS(127),
}
```
### `ModelPaths`
Returned by `getPaths()`. Feed fields directly into `LlmCreateInput` / `VlmCreateInput`:
```kotlin theme={"dark"}
data class ModelPaths(
val model_path: String,
val model_dir: String,
val model_name: String,
val runtime_id: String, // authoritative — prefer over UI selection
val mmproj_path: String? = null, // VLM projection weights
val tokenizer_path: String? = null,
val compute_unit: String? = null,
)
```
Qualcomm AI Hub pulls on Android **require** an explicit `chipset`. The Rust side only auto-detects on Windows on Snapdragon. Use `"SM8750"` for Snapdragon 8 Elite or `"SM8850"` for Snapdragon 8 Elite Gen 5.
***
## **Data structures**
### `LlmCreateInput`
```kotlin theme={"dark"}
data class LlmCreateInput(
val model_path: String,
val tokenizer_path: String? = null,
val config: ModelConfig,
val runtime_id: String? = null,
val compute_unit: String? = null,
)
```
### `VlmCreateInput`
```kotlin theme={"dark"}
data class VlmCreateInput(
val model_path: String,
val mmproj_path: String? = null, // vision projection weights (GGUF VLMs)
val config: ModelConfig,
val runtime_id: String? = null,
val compute_unit: String? = null,
)
```
### `ModelConfig`
```kotlin theme={"dark"}
data class ModelConfig(
var nCtx: Int = 2048, // context size; 0 = model default
var nThreads: Int = 8,
var nThreadsBatch: Int = 8,
var nBatch: Int = 2048,
var nUBatch: Int = 512,
var nSeqMax: Int = 1,
var nGpuLayers: Int = -1, // -1 = all layers
val chat_template_path: String = "",
val chat_template_content: String = "",
)
```
`nGpuLayers` is rewritten by the JNI based on `compute_unit`: `cpu` forces 0; `gpu` / `npu` / `hybrid` pass the value through (`-1` = all layers).
### `ChatMessage`
```kotlin theme={"dark"}
data class ChatMessage(
var role: String, // "system" | "user" | "assistant"
var content: String,
)
```
### `VlmChatMessage` / `VlmContent`
```kotlin theme={"dark"}
data class VlmChatMessage(
val role: String?, // "system" | "user" | "assistant"
val contents: List,
)
data class VlmContent(
val type: String?, // "text" | "image"
val text: String?, // text content, or absolute file path for image
)
```
### `GenerationConfig`
```kotlin theme={"dark"}
data class GenerationConfig(
var maxTokens: Int = 32,
var stopWords: Array? = null,
var stopCount: Int = 0,
var samplerConfig: SamplerConfig? = null,
var imagePaths: Array? = null,
var imageCount: Int = 0,
var audioPaths: Array? = null,
var audioCount: Int = 0,
)
```
The default `maxTokens` is **32**. Most use cases should set a higher value (e.g. `maxTokens = 2048`).
### `LlmStreamResult`
```kotlin theme={"dark"}
sealed class LlmStreamResult {
data class Token(val text: String) : LlmStreamResult()
data class Completed(val profile: ProfilingData) : LlmStreamResult()
data class Error(val throwable: Throwable) : LlmStreamResult()
}
```
***
## **llama.cpp (GGUF models)**
Runs any GGUF model on CPU, Adreno GPU, or Hexagon NPU. Compute-unit selection is controlled by `compute_unit`.
### LLM
```kotlin theme={"dark"}
val paths = ModelManagerWrapper.getPaths("unsloth/Qwen3-0.6B-GGUF")
?: error("Model not downloaded")
LlmWrapper.builder()
.llmCreateInput(
LlmCreateInput(
model_path = paths.model_path,
config = ModelConfig(nCtx = 4096),
runtime_id = "llama_cpp",
compute_unit = null, // null → npu (recommended on Snapdragon)
)
)
.build()
.onSuccess { llmWrapper = it }
.onFailure { println("Error: ${it.message}") }
val chat = arrayListOf(ChatMessage("user", "What is AI?"))
llmWrapper.applyChatTemplate(chat.toTypedArray(), null, false).onSuccess { t ->
llmWrapper.generateStreamFlow(t.formattedText, GenerationConfig(maxTokens = 2048)).collect { result ->
when (result) {
is LlmStreamResult.Token -> print(result.text)
is LlmStreamResult.Completed -> println("\nDone")
is LlmStreamResult.Error -> println("Error: ${result.throwable}")
}
}
}
```
#### Compute-unit variants
| Goal | `compute_unit` | Notes |
| ---------------------------- | ----------------- | ------------------------------------------- |
| Snapdragon NPU (recommended) | `"npu"` or `null` | Hexagon NPU acceleration. |
| Adreno GPU (OpenCL) | `"gpu"` | Defaults to `nGpuLayers = -1` (all layers). |
| Pure CPU | `"cpu"` | Works on any ARM64 chipset. |
### VLM
GGUF VLMs need two artifacts: the LLM weights (`model_path`) and the vision projection (`mmproj_path`). Both come from `getPaths()`:
```kotlin theme={"dark"}
val paths = ModelManagerWrapper.getPaths("unsloth/Qwen3-VL-2B-Instruct-GGUF")
?: error("Model not downloaded")
VlmWrapper.builder()
.vlmCreateInput(
VlmCreateInput(
model_path = paths.model_path,
mmproj_path = paths.mmproj_path,
config = ModelConfig(nCtx = 4096),
runtime_id = "llama_cpp",
compute_unit = null,
)
)
.build()
.onSuccess { vlmWrapper = it }
val msg = VlmChatMessage(
role = "user",
contents = listOf(
VlmContent("image", "/storage/emulated/0/Pictures/example.jpg"),
VlmContent("text", "Describe this image."),
),
)
val chat = arrayListOf(msg)
vlmWrapper.applyChatTemplate(chat.toTypedArray(), null, false).onSuccess { t ->
val gen = vlmWrapper.injectMediaPathsToConfig(chat.toTypedArray(), GenerationConfig(maxTokens = 2048))
vlmWrapper.generateStreamFlow(t.formattedText, gen).collect { result ->
when (result) {
is LlmStreamResult.Token -> print(result.text)
is LlmStreamResult.Completed -> println("\nDone")
is LlmStreamResult.Error -> println("Error: ${result.throwable}")
}
}
}
```
Always pass `t.formattedText` (the chat-templated prompt) into `generateStreamFlow`, **not** the raw user text. The native pipeline treats the prompt as already-templated.
***
## **Qualcomm® AI Hub Models (NPU via Qualcomm AI Engine Direct)**
Pre-compiled models from Qualcomm AI Hub. NPU-only, pinned to a specific chipset (`SM8750` = Snapdragon 8 Elite, `SM8850` = Snapdragon 8 Elite Gen 5).
### Downloading Qualcomm AI Hub Models
```kotlin theme={"dark"}
ModelManagerWrapper.pullFlow(
ModelPullInput(
model_name = "ai-hub-models/Qwen2.5-VL-7B-Instruct",
hub = HubSource.AUTO, // AUTO routes `ai-hub-models/*` to Qualcomm AI Hub
chipset = "SM8750", // REQUIRED on Android
)
).collect { event ->
when (event) {
is ModelManagerWrapper.PullEvent.Progress -> updateProgressBar(event.files)
ModelManagerWrapper.PullEvent.Completed -> println("done")
is ModelManagerWrapper.PullEvent.Error -> println("err ${event.code}: ${event.message}")
}
}
```
### Supported models
| Modality | Hub repo |
| -------- | -------------------------------------- |
| LLM | `ai-hub-models/Qwen3-4B-Instruct-2507` |
| VLM | `ai-hub-models/Qwen2.5-VL-7B-Instruct` |
### LLM
```kotlin theme={"dark"}
val paths = ModelManagerWrapper.getPaths("ai-hub-models/Qwen3-4B-Instruct-2507")
?: error("Model not downloaded")
LlmWrapper.builder()
.llmCreateInput(
LlmCreateInput(
model_path = paths.model_path,
config = ModelConfig(),
runtime_id = "qairt",
compute_unit = null, // null → NPU (only option for Qualcomm AI Engine Direct)
)
)
.build()
.onSuccess { llmWrapper = it }
.onFailure { println("Error: ${it.message}") }
val chat = arrayListOf(ChatMessage("user", "What is AI?"))
llmWrapper.applyChatTemplate(chat.toTypedArray(), null, false).onSuccess { t ->
llmWrapper.generateStreamFlow(t.formattedText, GenerationConfig()).collect { result ->
when (result) {
is LlmStreamResult.Token -> print(result.text)
is LlmStreamResult.Completed -> println("\nDone")
is LlmStreamResult.Error -> println("Error: ${result.throwable}")
}
}
}
```
Qualcomm AI Engine Direct rejects `nGpuLayers != 0` and `nCtx != 0` with `PARAM_NOT_SUPPORTED` — the KV cache and context length are fixed at compile time by the Qualcomm AI Hub bundle. Leave both at defaults and use `max_tokens` / `enable_thinking` only.
### VLM
```kotlin theme={"dark"}
val paths = ModelManagerWrapper.getPaths("ai-hub-models/Qwen2.5-VL-7B-Instruct")
?: error("Model not downloaded")
VlmWrapper.builder()
.vlmCreateInput(
VlmCreateInput(
model_path = paths.model_path,
mmproj_path = paths.mmproj_path,
config = ModelConfig(),
runtime_id = "qairt",
compute_unit = null,
)
)
.build()
.onSuccess { vlmWrapper = it }
val msg = VlmChatMessage(
role = "user",
contents = listOf(
VlmContent("image", "/storage/emulated/0/Pictures/cat.jpg"),
VlmContent("text", "What's in this image?"),
),
)
val chat = arrayListOf(msg)
vlmWrapper.applyChatTemplate(chat.toTypedArray(), null, false).onSuccess { t ->
val gen = vlmWrapper.injectMediaPathsToConfig(chat.toTypedArray(), GenerationConfig(maxTokens = 2048))
vlmWrapper.generateStreamFlow(t.formattedText, gen).collect { result ->
when (result) {
is LlmStreamResult.Token -> print(result.text)
is LlmStreamResult.Completed -> println("\nDone")
is LlmStreamResult.Error -> println("Error: ${result.throwable}")
}
}
}
```
Pass the **chat-templated** prompt (`t.formattedText`) to `generateStreamFlow`, never raw user text. Qualcomm AI Engine Direct VLM treats its prompt as already-templated — raw text produces degenerate output.
***
## **Need help?**
File a bug, request a feature, or browse open issues.
Developer collaboration and resources.
# Android Install
Source: https://geniex.aihub.qualcomm.com/en/run/android/install
Add the GenieX Android SDK to your Gradle project from Maven Central.
Add the GenieX SDK to an Android Studio project so your app can pull weights from Hugging Face / Qualcomm AI Hub and run them on the Hexagon NPU, Adreno GPU, or CPU compute units — all in Kotlin.
## **Try the sample app first**
See GenieX running on Android before you write any code. The reference chat app — with model picker, resumable downloads, and VLM support — lives in `qualcomm/ai-hub-apps`. Clone it, open it in Android Studio, and hit **Run ▶**.
Build the reference chat app from source — follow the README to clone, build, and run it.
Pick a model from the dropdown and choose NPU, GPU, or CPU on load. Tap the image button for VLMs. Stay on Wi-Fi for the first download.
No phone? See [Testing without a physical device](/en/resources/faq#how-do-i-test-the-android-demo-without-a-physical-device).
## **Prerequisites**
* **Android Studio** [Hedgehog (2023.1.1)](https://developer.android.com/studio/releases/past-releases/as-hedgehog-release-notes) or newer.
* **`minSdk = 27`** (Android 8.1) in your app module.
* A phone running **Snapdragon 8 Elite** (`SM8750`) or **Snapdragon 8 Elite Gen 5** (`SM8850`) — see [Supported platforms](/en/get-started/platforms).
## **Add the SDK to your app**
In `settings.gradle.kts` (or your top-level `build.gradle.kts` for older projects):
```kotlin theme={"dark"}
dependencyResolutionManagement {
repositories {
google()
mavenCentral()
}
}
```
In your **app module**'s `build.gradle.kts`:
```kotlin theme={"dark"}
dependencies {
implementation("com.qualcomm.qti:geniex-android:0.3.1")
}
```
The artifact ships native `arm64-v8a` libraries — no NDK or CMake on your side.
The SDK pulls weights at runtime. In `AndroidManifest.xml`:
```xml theme={"dark"}
```
For VLMs that load images from the gallery, also declare `READ_EXTERNAL_STORAGE` (or scoped media permissions on Android 13+).
Click **Sync Now** in Android Studio.
Next, head to the [Quickstart](/en/run/android/quickstart) to download and run your first model.
# Quickstart
Source: https://geniex.aihub.qualcomm.com/en/run/android/quickstart
Run your first model from the GenieX Android SDK in Kotlin.
This page walks through running your first model from a Kotlin app, then swapping in a different model. For a complete reference app with chat UI, model picker, and VLM support, see the [sample app](https://github.com/qualcomm/ai-hub-apps/blob/release/geniex_chat_android/README.md).
## **Prerequisites**
* The SDK added to your Gradle project — see [Install](/en/run/android/install).
* A phone running **Snapdragon 8 Elite** or **Snapdragon 8 Elite Gen 5**.
* `INTERNET` permission in your `AndroidManifest.xml` (the SDK pulls weights from Hugging Face / Qualcomm AI Hub on first use).
## **Run your first model**
The flow is the same regardless of model: **init the SDK → pull weights → load → generate**. Below is a minimal end-to-end example using `unsloth/Qwen3-0.6B-GGUF` — a small Qwen3 0.6B chat model that runs on any supported chipset.
Call once on app startup (idempotent — safe inside `Activity.onCreate`):
```kotlin theme={"dark"}
GenieXSdk.getInstance().init(context)
```
`pullFlow` streams progress events. Run inside a coroutine on `Dispatchers.IO`:
```kotlin theme={"dark"}
ModelManagerWrapper.pullFlow(
ModelPullInput(
model_name = "unsloth/Qwen3-0.6B-GGUF",
precision = "Q4_0",
hub = HubSource.HUGGINGFACE,
)
).collect { event ->
when (event) {
is ModelManagerWrapper.PullEvent.Progress -> /* update UI */
ModelManagerWrapper.PullEvent.Completed -> /* done */
is ModelManagerWrapper.PullEvent.Error -> /* show error */
}
}
```
Downloads are resumable — killing the app mid-pull and re-running picks up where it left off.
Resolve the on-disk paths and build an `LlmWrapper`:
```kotlin theme={"dark"}
val paths = ModelManagerWrapper.getPaths("unsloth/Qwen3-0.6B-GGUF")
?: error("Model not downloaded")
val llm = LlmWrapper.builder()
.llmCreateInput(
LlmCreateInput(
model_path = paths.model_path,
config = ModelConfig(nCtx = 4096),
runtime_id = "llama_cpp",
compute_unit = null, // null → NPU on Snapdragon (recommended)
)
)
.build()
.getOrThrow()
```
Apply the chat template, then collect tokens from the streaming flow:
```kotlin theme={"dark"}
val chat = arrayListOf(ChatMessage("user", "What is AI?"))
val templated = llm.applyChatTemplate(chat.toTypedArray(), null, false).getOrThrow()
llm.generateStreamFlow(
templated.formattedText,
GenerationConfig(maxTokens = 2048),
).collect { result ->
when (result) {
is LlmStreamResult.Token -> print(result.text)
is LlmStreamResult.Completed -> println("\nDone")
is LlmStreamResult.Error -> println("Error: ${result.throwable}")
}
}
```
Always pass `templated.formattedText` (the chat-templated prompt) into `generateStreamFlow`, **not** the raw user text. The native pipeline expects an already-templated prompt.
## **Switching models**
Swapping models is mostly a matter of changing the `model_name` and the `runtime_id`. There are two runtimes:
* **`llama_cpp`** — runs any GGUF model. Supports NPU / GPU / CPU compute units via `compute_unit`.
* **`qairt`** (Qualcomm AI Engine Direct) — runs Qualcomm AI Hub Models. NPU-only, requires an explicit `chipset` on Android.
### Another GGUF model (llama.cpp)
Just change the `model_name` (and `precision` if you want a different one) — the rest of the flow is identical:
```kotlin theme={"dark"}
ModelPullInput(
model_name = "unsloth/Qwen3-VL-2B-Instruct-GGUF",
precision = "Q4_0",
hub = HubSource.HUGGINGFACE,
)
```
For VLMs, also pass `paths.mmproj_path` into `VlmCreateInput` — see [API reference → VLM](/en/run/android/api-reference#vlm).
### A Qualcomm AI Hub Model (NPU via Qualcomm AI Engine Direct)
Qualcomm AI Hub Models are pre-compiled per chipset and only run on the NPU. You **must** pass `chipset` on Android:
```kotlin theme={"dark"}
ModelManagerWrapper.pullFlow(
ModelPullInput(
model_name = "ai-hub-models/Qwen3-4B-Instruct-2507",
hub = HubSource.AUTO, // routes ai-hub-models/* to Qualcomm AI Hub
chipset = "SM8750", // SM8750 = 8 Elite, SM8850 = 8 Elite Gen 5
)
).collect { /* … */ }
```
Then switch `runtime_id = "qairt"` in `LlmCreateInput`. See the supported Qualcomm AI Hub repos in the [API reference](/en/run/android/api-reference#supported-models).
### Switching compute unit (NPU / GPU / CPU)
For `llama_cpp` only — set `compute_unit` on `LlmCreateInput`:
| `compute_unit` | Compute unit |
| ----------------- | ---------------------------------------- |
| `null` or `"npu"` | Hexagon NPU (recommended on Snapdragon). |
| `"gpu"` | Adreno GPU via OpenCL. |
| `"cpu"` | Pure CPU. Works on any ARM64 chipset. |
Qualcomm AI Engine Direct ignores this — `cpu`/`gpu` are coerced to NPU with a warning.
## **Using a local model**
If the weights are already on the device — side-loaded via `adb push`, bundled in your app's files dir, or produced by another tool — point the model manager at that directory instead of a hub: set `hub = HubSource.LOCALFS` and `local_path` to the on-disk location. `pullFlow` imports it into the SDK cache (no network), after which `getPaths` / `LlmWrapper` work exactly as they do for a downloaded model.
The full Android snippets for importing a local GGUF model and a local Qualcomm AI Engine Direct bundle live on the Models page:
* [Run a local Qualcomm AI Engine Direct bundle → Android](/en/models/supported#run-a-local-qualcomm-ai-engine-direct-bundle)
* [Run a local GGUF model → Android](/en/models/supported#run-a-local-gguf-model)
## **Using the sample app**
The [sample app](https://github.com/qualcomm/ai-hub-apps/blob/release/geniex_chat_android/README.md) is a fully wired chat client built on top of the snippets above. A few patterns worth borrowing when you build your own UI:
* **Model picker UI** — the dropdown is driven by `app/src/main/assets/model_list.json`. Each entry pins a `model_name`, `hub`, and (for Qualcomm AI Engine Direct) a `chipset`. Edit this file to add new models without touching code.
* **Resumable downloads with progress** — the `Progress` events from `pullFlow` carry per-file byte counts; the sample wires them straight into a `LinearProgressIndicator`.
* **Runtime-aware compute-unit picker** — when the selected model uses Qualcomm AI Engine Direct, the picker hides GPU/CPU options. See `LoadDialog.kt`.
* **VLM image picker** — for VLMs, the sample passes the absolute file path into `VlmContent("image", path)`. Don't pass content URIs — the native side reads the file directly.
Clone [`qualcomm/ai-hub-apps`](https://github.com/qualcomm/ai-hub-apps/blob/release/geniex_chat_android/README.md), open it in Android Studio, and hit **Run ▶**.
## **Next steps**
Wrapper classes, runtime / compute-unit selection, and data structures.
Snapdragon platforms and when to pick llama.cpp vs Qualcomm AI Engine Direct.
# CLI Install
Source: https://geniex.aihub.qualcomm.com/en/run/cli/install
Install the GenieX CLI on Windows ARM64, or run it via Docker on Linux ARM64.
The CLI ships as a native Windows ARM64 executable, and as a Linux ARM64 Docker image for Dragonwing QCS9075 and similar IoT devices.
### **Prerequisites**
* Snapdragon X-series chipset (X Elite or X2 Elite). See [Supported platforms](/en/get-started/platforms).
* PowerShell or any shell available on Windows ARM64.
### **Download**
Latest GenieX CLI for Windows ARM64. Click the downloaded `.exe` and follow the prompts.
The installer is not yet code-signed — Windows SmartScreen will warn you. Click **More info → Run anyway**.
### **Add to PATH**
Make `geniex` available in your shell:
```powershell windows theme={"dark"}
Set-Alias geniex (where.exe geniex)
```
### **Verify**
```powershell windows theme={"dark"}
geniex --help
```
Now jump to the [Quickstart](/en/run/cli/quickstart).
Install `geniex` directly on a Dragonwing ARM64 Linux host (EVK or any device with a Qualcomm BSP). If you'd rather run it inside a container, see [Linux (Docker) Install](/en/run/linux/install) instead.
### **Prerequisites**
* Linux ARM64 host.
* Dragonwing IoT chipset — see [Supported platforms](/en/get-started/platforms).
### **Install host dependencies**
`geniex` reaches the NPU through the host's Qualcomm driver libraries. Install them before running `geniex`, in two steps.
**1. Standard APT packages**
Provides `libatomic1`, `libglib2.0-0`, and a generic OpenCL ICD loader:
```bash bash theme={"dark"}
sudo apt update
sudo apt install -y \
libatomic1 \
libglib2.0-0 \
ocl-icd-libopencl1
```
`ocl-icd-libopencl1` is replaced by `qcom-adreno1` in the next step.
**2. Qualcomm driver packages (via `ubuntu-qcom-iot` PPA)**
Provides the Qualcomm-proprietary libraries the NPU needs:
| Library | Provided by |
| ---------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| `libOpenCL_adreno.so.1`, `libCB.so.1`, `libadreno_utils.so.1`, `libgsl.so.1`, `libllvm-*.so.1` | `qcom-adreno1` |
| `libdmabufheap.so.0.0.0` | `qcom-libdmabufheap` *(auto-dep of `qcom-adreno1`)* |
| `libpropertyvault.so.0.0.0` | `qcom-property-vault` *(auto-dep of `qcom-adreno1`)* |
| `libcdsprpc.so.1.0.0` | `qcom-fastrpc1` |
```bash bash theme={"dark"}
sudo apt-get install -y qcom-adreno1 qcom-fastrpc1 libqnn1
```
The PPA (`ppa:ubuntu-qcom-iot/qcom-ppa`) is pre-configured on the IQ9075 EVK Ubuntu image. `qcom-adreno1` automatically pulls in `qcom-libdmabufheap` and `qcom-property-vault`.
### **Install**
If `HOME` is not set (common in minimal containers or `sudo -s` sessions), export it first so the installer knows where to place binaries:
```bash bash theme={"dark"}
export HOME=/root
```
Then run the install script:
```bash bash theme={"dark"}
curl -fsSL https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-geniex/install.sh | sh
```
The script downloads the latest stable release, verifies its SHA256, and installs without `sudo`. If the launcher directory isn't already on your `PATH`, it prints the exact export line to append — for example:
```bash bash theme={"dark"}
echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc # or ~/.zshrc / ~/.profile
```
Open a new shell or `source` that file to pick up the change.
If the script reports missing shared libraries (`libCB.so.1`, `libOpenCL_adreno.so.1`, `libcdsprpc.so.1.0.0`, etc.), install the Qualcomm driver packages first — see [Install host dependencies](#install-host-dependencies) above.
### **Verify**
```bash bash theme={"dark"}
geniex --help
```
If `geniex` isn't found, open a new shell — or follow the `PATH` line the installer printed at the end of its output.
### **Advanced**
The script runs without `sudo` and picks a location based on who you are:
* **Root** (EVK / container) → binary in `/usr/local/lib/geniex/`, launcher at `/usr/local/bin/geniex`.
* **Non-root** → binary in `${XDG_DATA_HOME:-~/.local/share}/geniex/`, launcher at `~/.local/bin/geniex`.
It downloads the latest stable release, verifies its SHA256, and installs atomically.
Pass flags after `sh -s --`:
```bash bash theme={"dark"}
# Pin a specific version
curl -fsSL .../install.sh | sh -s -- --version v0.1.8
# Custom install prefix
curl -fsSL .../install.sh | sh -s -- --prefix /opt/geniex
```
If the launcher directory isn't on your `PATH`, add the line the installer printed — typically:
```bash bash theme={"dark"}
echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc # or ~/.zshrc / ~/.profile
```
Open a new shell or `source` that file.
# Local server
Source: https://geniex.aihub.qualcomm.com/en/run/cli/local-server
Run an OpenAI-compatible HTTP API on localhost backed by Snapdragon NPU/GPU/CPU acceleration.
GenieX includes a built-in inference server that exposes an **OpenAI-compatible API**. Run models on-device and connect them to any application or framework that speaks the OpenAI protocol — agentic frameworks like **LangChain**, AI-native apps like **OpenClaw**, or your own code. No cloud dependency.
## **Prerequisites**
* The CLI installed — see [Install](/en/run/cli/install).
* Interactive shell from container (Docker only) — see [Run interactively](/en/run/linux/install#run-interactively).
* A model pulled. `geniex serve` does **not** auto-download models.
## **Start the server**
Pull a model:
```bash bash theme={"dark"}
geniex pull ai-hub-models/Qwen3-4B-Instruct-2507
```
Start the server:
```bash bash theme={"dark"}
geniex serve
```
The server runs on `http://127.0.0.1:18181` by default. Keep this terminal open and make requests from another one. Run `geniex serve -h` for all configurable options.
## **POST /v1/chat/completions**
Creates a model response for a conversation. Supports LLM (text-only) and VLM (text + image or audio).
### **LLM request**
```json Example Value theme={"dark"}
{
"model": "ai-hub-models/Qwen3-4B-Instruct-2507",
"messages": [
{"role": "user", "content": "Hello! Briefly introduce yourself."}
],
"max_tokens": 256,
"temperature": 0.7,
"stream": false
}
```
### **Try it from Swagger UI**
Open `http://127.0.0.1:18181` in your browser to access the built-in Swagger UI.
**Step 1.** Expand the `POST /v1/chat/completions` endpoint to view the example request body and schema.
![Swagger UI showing the chat completions endpoint with example request body]()
**Step 2.** Click **Try it out**, edit the request body as needed, then click **Execute**.
![Editing the request body in Try-it-out mode before executing]()
**Step 3.** View the response — a `200` status with the model's generated reply.
![Response body showing a successful 200 response with the model's reply]()
### **VLM request**
A VLM accepts multimodal content parts alongside text. Use `image_url` for images, and `input_audio` for audio on a model whose mmproj carries a conformer encoder (e.g. `google/gemma-4-E2B-it-qat-q4_0-gguf`). Both `image_url.url` and `input_audio.data` accept the same three formats:
| Format | Image example | Audio example |
| -------------------------------------------------- | -------------------------------------- | -------------------------------------- |
| Local file path (the `file://` prefix is optional) | `C:/Users/Username/Pictures/photo.jpg` | `/data/jfk.wav`, `file:///tmp/jfk.wav` |
| HTTP / HTTPS URL — fetched by the server | `https://example.com/image.jpg` | `https://example.com/clip.mp3` |
| Base64 data URL — inline bytes | `data:image/png;base64,iVBORw0KGgo...` | `data:audio/wav;base64,UklGR...` |
**Running in Docker?** Local paths are resolved **inside the container**, not on your host. The install command already mounts `$PWD/data` to `/data` — drop your images and audio files there and pass `/data/cat.jpg` / `/data/jfk.wav`. Alternatively, use an HTTP URL or base64 data URL to skip the filesystem entirely.
Audio input runs on the **llama.cpp** backend only. QAIRT models report `audio: false` and a QAIRT model given audio fails with `GenieXError(-201201): Multimodal generation failed`.
A single message can mix `image_url` and `input_audio` parts. Pull an audio-capable model and grab a sample clip:
```bash theme={"dark"}
geniex pull google/gemma-4-E2B-it-qat-q4_0-gguf
curl -L -o jfk.wav https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav
curl -L -o landmark.jpg "https://images.pexels.com/photos/402028/pexels-photo-402028.jpeg?w=1024"
```
```bash theme={"dark"}
curl http://127.0.0.1:18181/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-4-E2B-it-qat-q4_0-gguf",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the image, then transcribe the audio."},
{"type": "image_url", "image_url": {"url": "/full/path/to/landmark.jpg"}},
{"type": "input_audio", "input_audio": {"data": "/full/path/to/jfk.wav"}}
]
}
],
"max_tokens": 256
}'
```
The same request from the Python `openai` client:
```python theme={"dark"}
resp = client.chat.completions.create(
model="google/gemma-4-E2B-it-qat-q4_0-gguf",
messages=[
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the image, then transcribe the audio."},
{"type": "image_url", "image_url": {"url": "/full/path/to/landmark.jpg"}},
{"type": "input_audio", "input_audio": {"data": "/full/path/to/jfk.wav"}},
],
}
],
max_tokens=256,
)
print(resp.choices[0].message.content)
```
Output (verified on `--compute npu`, Snapdragon X Elite):
```text theme={"dark"}
This is a photograph featuring a traditional Japanese temple ... In the background, there are softer, blue-toned mountains and a distant urban skyline.
**Transcription:**
And so my fellow Americans ask not what your country can do for you, ask what you can do for your country.
```
In Swagger UI, replace the request body with a VLM payload like the one above, point the paths at local files, then click **Execute**.
![Editing the VLM request body in Try-it-out mode before executing]()
![Response body showing a successful 200 response with the VLM's image description]()
## **POST /v1/completions**
Generates a raw continuation of `prompt` with no chat template applied — the prompt reaches the model verbatim. Use this for clients that build the full prompt themselves, such as editor fill-in-the-middle (FIM) code autocompletion: put the model's FIM tokens directly in `prompt`. LLM models only.
```json Example Value theme={"dark"}
{
"model": "unsloth/Qwen2.5-Coder-3B-GGUF:Q4_0",
"prompt": "<|fim_prefix|>def fibonacci(n):\n <|fim_suffix|>\n return a<|fim_middle|>",
"max_tokens": 64,
"temperature": 0.2,
"stop": ["<|endoftext|>"],
"stream": false
}
```
The response `choices[0].text` is the raw completion, ready to insert at the cursor. `stream`, `stop`, `echo` and the sampler knobs (`temperature`, `top_p`, `top_k`, `min_p`, `repetition_penalty`, `seed`) work as on `/v1/chat/completions`; `suffix` is not supported — encode the suffix with the model's FIM tokens inside `prompt` instead.
## **Python client (OpenAI SDK)**
Because the server speaks the OpenAI protocol, you can point the official `openai` Python client at the local endpoint and reuse any existing OpenAI code. Install with `pip install openai`, then create a client:
```python python theme={"dark"}
from openai import OpenAI
client = OpenAI(
base_url="http://127.0.0.1:18181/v1",
api_key="geniex", # any non-empty string; the server does not check it
)
```
The examples below reuse this `client`. Replace the `model` value with a model you have already pulled. The optional `:` suffix (e.g. `Q4_0`, `Q4_K_M`, `Q8_0`) selects a quantization variant — `Q4_0` is recommended for llama.cpp on Hexagon NPU. See [Precisions (Quantizations) Supported](/en/models/supported#precisions-quantizations-supported).
### **Streaming**
Print each delta as it arrives:
```python python theme={"dark"}
stream = client.chat.completions.create(
model="unsloth/Qwen3-4B-GGUF:Q4_0",
messages=[
{"role": "user", "content": "Hello! Briefly introduce yourself."},
],
max_tokens=256,
temperature=0.7,
stream=True,
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end="", flush=True)
print()
```
### **Chat completion (non-streaming)**
Single request, single response, no streaming — the standard OpenAI `chat.completions.create` shape. The `enable_think=False` extra parameter turns off Qwen3's default `…` reasoning prefix so the reply content stays clean.
```python python theme={"dark"}
resp = client.chat.completions.create(
model="unsloth/Qwen3-4B-GGUF:Q4_0",
messages=[
{"role": "user", "content": "Hello! Briefly introduce yourself."},
],
max_tokens=128,
temperature=0.7,
extra_body={"enable_think": False},
)
print(resp.choices[0].message.content)
print("finish_reason:", resp.choices[0].finish_reason)
print("usage:", resp.usage)
```
Output:
```text theme={"dark"}
Hello! I'm Qwen, a large language model developed by Alibaba Cloud. I can help with a wide range of tasks, including answering questions, writing articles, creating stories, and more. I'm here to assist you in any way I can! How can I help you today?
finish_reason: stop
usage: CompletionUsage(completion_tokens=58, prompt_tokens=19, total_tokens=77, ...)
```
### **Separating reasoning (reasoning\_content)**
By default a thinking model leaves its chain-of-thought inline in `message.content` (`…`), mixed in with the final reply. Pass `reasoning_format="deepseek"` to make the server move the chain-of-thought into the OpenAI-standard `message.reasoning_content` field, leaving `content` clean.
`reasoning_format` is not the same as `enable_think`: `enable_think=False` makes the model **not produce** a chain-of-thought at all, while `reasoning_format="deepseek"` lets it think as usual but **moves** the thinking out of `content`. Values are `none` (default, kept inline) and `deepseek` / `deepseek-legacy` / `auto` (all separate). Tool-call requests ignore this parameter (tool parsing needs the raw tagged text).
```python python theme={"dark"}
resp = client.chat.completions.create(
model="unsloth/Qwen3-4B-GGUF:Q4_0",
messages=[
{"role": "user", "content": "What is 2+2? Answer briefly."},
],
max_tokens=128,
extra_body={"reasoning_format": "deepseek"},
)
msg = resp.choices[0].message
print("reasoning:", msg.model_extra.get("reasoning_content"))
print("content:", msg.content)
```
Streaming works the same way — the chain-of-thought arrives as `delta.reasoning_content` deltas and the final reply as `delta.content`.
### **Tool calling**
Function/tool calling uses the standard OpenAI `tools` schema. The server extracts the tool call from the model's generated text (`…` tags or a fenced ` ```json ` block) and re-emits it as OpenAI `tool_calls`. The flow works with **VLMs too** — the model can look at an image, decide what to search for, and call a tool.
The example below walks through a two-step agentic loop with `qualcomm/Qwen3-VL-4B-Instruct`: (1) the VLM identifies a landmark from a photo and calls `web_search`, (2) you execute the search locally and feed the results back so the VLM writes a grounded reply.
Only one tool call per assistant turn is parsed — parallel tool calls in a single response are not supported.
Two Qwen3-VL specifics for reliable tool calls:
1. Prime the model with a system message that spells out the `…` shape (Qwen3-VL's chat template does not enforce it as strongly as Qwen3's text-only template).
2. On the follow-up turn, drop `tools=` and drop the image content from `messages` — this stops the VLM from re-invoking the tool and avoids re-running the vision encoder on the same image.
Install the search library used by the tool (`pip install ddgs` — DuckDuckGo, no API key required), then:
```python python theme={"dark"}
import json
from ddgs import DDGS
tools = [
{
"type": "function",
"function": {
"name": "web_search",
"description": "Search the web for travel information about a location.",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query, e.g. 'things to do in Kyoto'"},
},
"required": ["query"],
},
},
}
]
def web_search(query: str) -> list[dict]:
return [
{"title": r["title"], "snippet": r["body"], "url": r["href"]}
for r in DDGS().text(query, max_results=3)
]
IMAGE_URL = "https://images.pexels.com/photos/402028/pexels-photo-402028.jpeg?w=1024"
system_prompt = (
"You are a travel assistant. Call the web_search tool to look up any location "
"the user asks about before answering. Once you receive the tool results, do not "
"call the tool again - use them to write a short, friendly reply for the user. "
"Emit tool calls in the exact format: "
'{"name": "web_search", "arguments": {"query": ""}}'
)
messages = [
{"role": "system", "content": system_prompt},
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": IMAGE_URL}},
{"type": "text", "text": "Identify the landmark, then call web_search for travel tips about it."},
],
},
]
# Step 1 - VLM identifies the landmark and requests a web_search call.
first = client.chat.completions.create(
model="qualcomm/Qwen3-VL-4B-Instruct",
messages=messages,
tools=tools,
tool_choice="auto",
max_tokens=512,
extra_body={"enable_think": False},
)
call = first.choices[0].message.tool_calls[0]
print("finish_reason:", first.choices[0].finish_reason) # -> "tool_calls"
print("call:", call.function.name, call.function.arguments)
# Step 2 - run the tool, feed the result back as a fresh text-only conversation.
result = web_search(**json.loads(call.function.arguments))
followup = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": "Summarize travel tips using the search results below."},
first.choices[0].message,
{"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)},
]
final = client.chat.completions.create(
model="qualcomm/Qwen3-VL-4B-Instruct",
messages=followup,
max_tokens=256,
extra_body={"enable_think": False},
)
print(final.choices[0].message.content)
```
Output (grounded on the live DuckDuckGo results — the exact prose varies with what the web returns that day):
```text theme={"dark"}
finish_reason: tool_calls
call: web_search {"query": "Kiyomizu-dera travel tips"}
Here's a quick summary of travel tips for Kiyomizu-dera in Kyoto:
**1. Best Times to Visit:**
Early morning (around 7-8 AM) or late afternoon (4-6 PM) are ideal to avoid the crowds.
Peak hours (11 AM–2 PM) can be very busy, so plan accordingly.
**2. How to Get There:**
Take the Kiyomizu-dera train or bus from Kyoto Station to the temple.
The temple is located on a hill with a wooden stage built without nails, and the walkways are accessible, though narrow.
**3. What to See:**
- The great wooden stage (without nails)
- Otawa waterfall and its three streams
- Jishu love shrine
- The Sannenzaka and Ninenzaka approach streets
- Night illuminations (best experienced at night)
...
```
## **Other endpoints**
* `GET /v1/models` — list available models.
* `GET /v1/models/{model}` — get info about a specific model.
# Quickstart
Source: https://geniex.aihub.qualcomm.com/en/run/cli/quickstart
Run your first model from the GenieX CLI.
## **Prerequisites**
* The CLI installed — see [Install](/en/run/cli/install).
* Interactive shell from container (Docker only) — see [Run interactively](/en/run/linux/install#run-interactively).
* Familiarity with [runtime choice](/en/get-started/platforms#geniex-runtimes) — `qairt` (Qualcomm AI Engine Direct) for Qualcomm AI Hub Models, `llama_cpp` for any GGUF.
## **Run your first model**
### **Qualcomm AI Engine Direct runtime (Qualcomm AI Hub)**
**Language model:**
```powershell windows theme={"dark"}
geniex infer ai-hub-models/Qwen3-4B
```
**Multimodal model:**
```powershell windows theme={"dark"}
geniex infer ai-hub-models/Qwen2.5-VL-7B-Instruct
```
### **llama.cpp runtime (GGUF)**
Pick `Q4_0` when prompted — it has the best Hexagon NPU support.
**Language model:**
```powershell windows theme={"dark"}
geniex infer unsloth/Qwen3.5-0.8B-GGUF
```
**Multimodal model:**
```powershell windows theme={"dark"}
geniex infer Qwen/Qwen3-VL-2B-Instruct-GGUF
```
When prompted:
* **Model type** — `vlm` for vision-language models, `llm` for text-only models. For `Qwen3.5` and `Gemma4`, pick `llm` for now (multimodal support coming soon).
* **Precision (Quantization)** — `Q4_0` for best Hexagon NPU performance.
To try other GGUF models, copy any compatible GGUF path from Hugging Face and substitute it into the command above. See [Run a GGUF model from Hugging Face](/en/models/supported#run-a-gguf-model-from-hugging-face).
## **Run a local model**
Already have a model on disk, or want to self-convert a bundle from Hugging Face? Use `geniex pull` with `--local-path` to register it, then run it like any other model. See:
* [Run a local Qualcomm AI Engine Direct bundle](/en/models/supported#run-a-local-qualcomm-ai-engine-direct-bundle) — self-converted from Hugging Face, an extracted bundle directory, or an AI Hub `.zip`.
* [Run a local GGUF model](/en/models/supported#run-a-local-gguf-model) — a directory containing your `.gguf` file.
## **Next steps**
Expose an OpenAI-compatible HTTP API on `localhost:18181`.
Every command, every flag.
# CLI reference
Source: https://geniex.aihub.qualcomm.com/en/run/cli/reference
Every GenieX CLI command and flag, with usage examples.
## **Model inference**
### **`geniex pull`**
Download a model and store it locally.
```powershell theme={"dark"}
geniex pull [:]
```
| Flag | Description |
| -------------- | ------------------------------------------------------------------------------- |
| `--model-hub` | Model source: `aihub` \| `hf` \| `localfs`. Auto-detected when omitted. |
| `--local-path` | Path to a local directory or AI Hub `.zip` file. Implies `--model-hub localfs`. |
| `--model-type` | Model type: `llm` \| `vlm`. Auto-detected when omitted. |
**Pulling from a local path:**
```powershell theme={"dark"}
geniex pull local/my-model --local-path /path/to/model-dir
```
`pull` copies files into the GenieX cache. After a successful pull you can safely delete the source to avoid keeping two copies.
**Precision (Quantization) (llama.cpp only)**
For GGUF models the CLI prompts you to pick a precision:
```powershell theme={"dark"}
Choose a precision version to download
> Q4_0 [1.2 GiB] (default)
Q8_0 [2.0 GiB]
F16 [3.8 GiB]
```
`Q4_0` has the best Hexagon NPU support. See [Precisions (Quantizations) Supported](/en/models/supported#precisions-quantizations-supported).
Qualcomm AI Hub Models are pre-quantized — no choice needed.
### **`geniex infer` — LLM**
Launch an interactive chat session with a language model.
```powershell theme={"dark"}
geniex infer ai-hub-models/Qwen3-4B
```
**Thinking mode** — control whether the model shows reasoning before responding:
```powershell theme={"dark"}
geniex infer ai-hub-models/Qwen3-4B --think # show reasoning steps
geniex infer ai-hub-models/Qwen3-4B --think=false # respond directly
```
**Compute unit selection** (via `--compute`) — pick which compute unit runs the model (default: `npu`):
```powershell theme={"dark"}
# llama.cpp models support all compute units
geniex infer unsloth/Qwen3.5-0.8B-GGUF --compute npu
geniex infer unsloth/Qwen3.5-0.8B-GGUF --compute gpu
geniex infer unsloth/Qwen3.5-0.8B-GGUF --compute cpu
# Qualcomm AI Hub Models only support NPU
geniex infer ai-hub-models/Qwen3-4B --compute npu
```
Qualcomm AI Hub Models run on NPU only. Using `--compute cpu` or `--compute gpu` returns an error.
### **`geniex infer` — VLM**
Run vision-language inference with text-only, image, or audio input:
```bash theme={"dark"}
geniex infer google/gemma-4-E2B-it-qat-q4_0-gguf
```
For text-only, just launch and chat. For image or audio input, provide the **absolute path** or drag the file into your terminal — image (`.jpg`, `.jpeg`, `.png`, `.webp`) and audio (`.wav`, `.mp3`) paths are auto-detected, and a single prompt can carry both. `google/gemma-4-E2B-it-qat-q4_0-gguf` ships an audio-capable mmproj (\~4.0 GiB total: `Q4_0` weights + conformer mmproj, pulled together):
```bash theme={"dark"}
geniex pull google/gemma-4-E2B-it-qat-q4_0-gguf
curl -L -o jfk.wav https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav
curl -L -o landmark.jpg "https://images.pexels.com/photos/402028/pexels-photo-402028.jpeg?w=1024"
geniex infer google/gemma-4-E2B-it-qat-q4_0-gguf \
-p "Describe the image and transcribe the audio. Image: /full/path/to/landmark.jpg Audio: /full/path/to/jfk.wav"
```
Output (verified on `--compute npu`, Snapdragon X Elite):
```text theme={"dark"}
**Image Description:**
This is a scenic, panoramic photograph that features a traditional Japanese temple ... The overall mood of the image is serene and beautiful.
**Audio Transcription:**
"And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country."
```
Audio input runs on the **llama.cpp** backend only and needs an audio-capable model (mmproj with a conformer encoder). Audio also runs on `--compute cpu` / `gpu`, but QAIRT models report `audio: false` and a QAIRT model given audio fails with `GenieXError(-201201): Multimodal generation failed`.
In an interactive session with an audio-capable model, `/mic` records a clip instead of loading one from disk — `Ctrl-C` stops the recording and transcribes it. This needs **SoX** on your `PATH` (`sudo apt install sox`, or `winget install --id=ChrisBagwell.SoX -e` on Windows); without it GenieX warns `SoX is not installed` at startup. See the [audio input tutorial](/en/tutorials/audio-input) for the full walkthrough.
## **`geniex serve`**
Start the OpenAI-compatible local server. See [Local server](/en/run/cli/local-server) for the API.
```bash theme={"dark"}
geniex serve
```
To move a thinking model's chain-of-thought out of `message.content` into `message.reasoning_content`, set the per-request `reasoning_format` field — see [Separating reasoning](/en/run/cli/local-server#separating-reasoning-reasoning_content).
## **Logging**
`--log` is a global flag controlling the CLI's log output. It is equivalent to the `GENIEX_LOG` environment variable and takes precedence over it when both are set.
```bash theme={"dark"}
geniex --log debug list
```
| Value | Emits |
| ------- | -------------------------------- |
| `none` | nothing (**CLI default**) |
| `error` | errors only |
| `warn` | warnings + errors |
| `info` | info + warnings + errors |
| `debug` | debug + info + warnings + errors |
| `trace` | everything |
## **Configuration flags**
These flags can be passed to `geniex infer` to control model loading and generation.
### Sampler flags
Control how the model selects tokens during generation.
| Flag | Type | Default | Description |
| ---------------------- | ------ | ------- | ----------------------------------------------------------- |
| `--temperature` | float | — | Sampling temperature. Higher values increase randomness. |
| `--top-p` | float | — | Top-p (nucleus) sampling threshold. |
| `--top-k` | int | — | Top-k sampling. Only consider the top-k most likely tokens. |
| `--min-p` | float | — | Min-p sampling threshold. |
| `--repetition-penalty` | float | `1` | Penalize repeated tokens. Values > 1 reduce repetition. |
| `--presence-penalty` | float | — | Penalize tokens that have appeared at all. |
| `--frequency-penalty` | float | — | Penalize tokens proportional to their frequency. |
| `--seed` | int | — | Random seed for reproducible outputs. |
| `--grammar-path` | string | — | Path to a GBNF grammar file for constrained generation. |
| `--grammar-string` | string | — | Inline grammar in GBNF string format. |
| `--enable-json` | — | — | Force JSON-only output. |
### Model flags
Control model loading, context, and generation limits.
| Flag | Type | Default | Description |
| --------------------------- | --------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `-n`, `--ngl` | int | `-1` | Number of layers to offload to GPU/NPU, `-1` = all (llama\_cpp only). |
| `--nctx` | int | `4096` | Context window size (max input + output tokens). |
| `--max-tokens` | int | `2048` | Maximum tokens to generate per response. |
| `--stop` | string\[] | — | Stop sequences (can be specified multiple times). |
| `--stop-file` | string | — | File containing stop sequences (one per line). |
| `--think` / `--think=false` | bool | `true` | Enable or disable thinking mode for reasoning models. |
| `-s`, `--system-prompt` | string | — | System prompt to set model behavior. |
| `--sliding-window` | — | `false` | (`qairt` only) Evict the oldest context above a small anchored prefix instead of erroring when the context length is exceeded, letting the conversation continue. |
### Increasing the context length
The context window (`--nctx`) is how much the model can hold at once. When a conversation grows past it you get a *context length exceeded* error. How to raise it depends on the runtime:
* **llama.cpp (GGUF):** `--nctx ` raises the window at runtime, up to the model's trained maximum. A larger window uses more KV-cache memory.
```bash theme={"dark"}
geniex infer unsloth/Qwen3-8B-GGUF --nctx 8192
```
* **Qualcomm AI Engine Direct (NPU):** the context length is **baked into the compiled bundle** and cannot be raised at runtime — `--nctx` has no effect. Instead:
* Add `--sliding-window` to keep chatting past the limit: the oldest context above a small anchored prefix is evicted instead of erroring.
* For a genuinely larger window, pull a bundle compiled for a longer context from [Qualcomm AI Hub](https://aihub.qualcomm.com/models/). See [Qualcomm AI Engine Direct runtime constraints](/en/get-started/platforms#runtime-constraints).
## **Utility commands**
| Command | Description | Example |
| ----------------------- | --------------------------------------------------------- | --------------------------------------- |
| `geniex list` | Display all downloaded models with their names and sizes. | `geniex list` |
| `geniex remove ` | Remove a specific local model by name. | `geniex remove unsloth/Qwen3-0.6B-GGUF` |
| `geniex clean` | Delete all locally cached models. | `geniex clean` |
| `geniex infer -h` | Show help for `geniex infer`. | `geniex infer -h` |
| `geniex serve -h` | Show help for `geniex serve`. | `geniex serve -h` |
# Linux (Docker) Install
Source: https://geniex.aihub.qualcomm.com/en/run/linux/install
Run GenieX via Docker on Linux ARM64 with NPU access.
The Linux Docker path runs through a container image built for Linux ARM64 with NPU access (Dragonwing QCS9075 and similar IoT platforms).
This is the containerized path. If you'd rather install `geniex` directly on the host without Docker, see [CLI Install → Linux ARM64 (Native)](/en/run/cli/install).
### **Prerequisites**
* Linux ARM64 host with Docker installed — if you don't have Docker yet, follow [Docker's Ubuntu install guide](https://docs.docker.com/engine/install/ubuntu/).
* Dragonwing IoT chipset — see [Supported platforms](/en/get-started/platforms).
* Credentials with read access to one of the registries below.
### **Install host dependencies**
The container reaches the NPU through the host's Qualcomm driver libraries, mounted in via `-v /usr/lib:/opt/qcom-lib:ro`. Install them **on the host** before `docker run` — follow [CLI Install → Install host dependencies](/en/run/cli/install#install-host-dependencies).
### **Pull the image**
The image is published to two registries — pick whichever you have access to and set `IMAGE` accordingly, then pull:
```bash bash theme={"dark"}
# Option A — Docker Hub (public, no login required):
IMAGE=docker.io/qualcomm/geniex:latest
# Option B — Qualcomm Container Registry:
docker login docker-registry.qualcomm.com -u '$app' -p GB2S6KXMJXTPV8VHNFNS7Q6LVH75LOOBTLT8D723WUX6PSFZMTX95GIQG4EFWH5C021ONZ5763VI9IDHU96Q7VAZJ2830CLX3NPI6STQOJWRYXLLA2ZYTL1S
IMAGE=docker-registry.qualcomm.com/qcom-ai-hub/geniex-cli:latest
docker pull "$IMAGE"
```
If `docker login` fails or `docker pull` returns `permission denied ... /var/run/docker.sock`, see [Troubleshooting → Linux](/en/resources/troubleshooting#linux) for the credential and `docker` group fixes.
### **Run interactively**
Drop into an interactive shell inside the container:
```bash bash theme={"dark"}
docker run -it --rm --privileged \
-v "$PWD/data:/data" \
-v /usr/lib:/opt/qcom-lib:ro \
"$IMAGE"
```
### **Verify**
Then run a test command from the container shell:
```bash bash theme={"dark"}
geniex --help
```
The `--privileged` flag is required for NPU access.
For server mode, see [Local server](/en/run/cli/local-server).
## **CLI**
For the full CLI Docker install guide, see [CLI Install — Linux ARM64 (Docker)](/en/run/cli/install).
## **Python**
For the full Python install guide on native Linux ARM64, see [Python Install — Linux ARM64](/en/run/python/install).
The Python SDK can be installed inside the Docker container or directly on the Linux ARM64 host. See the [Linux ARM64 tab](/en/run/python/install) for native install instructions.
# API reference
Source: https://geniex.aihub.qualcomm.com/en/run/python/api-reference
Complete API documentation for the GenieX Python SDK.
The GenieX Python SDK follows the same design patterns as Hugging Face `transformers` — use `AutoModel*.from_pretrained()` to load a model, then call `.generate()` for inference.
***
## AutoModelForCausalLM
Factory for loading causal language models — both text-only and multimodal. Returns a [`GenieXLLM`](#geniexllm) for text-only models, or a [`GenieXVLM`](#geniexvlm) when a multimodal model is detected (e.g. `phi4_multimodal`, `qwen3.5-vl`, `gemma4`).
```python theme={"dark"}
from geniex import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"ai-hub-models/Qwen3-4B-Instruct",
device_map="auto",
)
```
### `from_pretrained()`
| Parameter | Type | Default | Description |
| -------------------- | ------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `model_name_or_path` | `str` | *required* | HuggingFace repo id, short alias (e.g. `"qwen3"`), or local path. |
| `device_map` | `str` | `"auto"` | `"auto"` picks the first available runtime + compute unit. Also accepts `""` (runtime) or `":"` (runtime + compute unit). |
| `precision` | `str \| None` | `None` | Precision (quantization) variant (e.g. `"Q4_K_M"`). Filters files when downloading from Hub. |
| `mmproj_path` | `str \| None` | `None` | Path to the multimodal projector file. Auto-resolved from Hub; pass explicitly to force VLM mode. |
| Parameter | Type | Default | Description |
| ------------ | ------------- | ------- | ----------------------------------------------------------------------------------- |
| `model_name` | `str \| None` | `None` | Override the registry model name (e.g. `"granite4"` for Qualcomm AI Engine Direct). |
| `hf_token` | `str \| None` | `None` | HuggingFace bearer token for gated models. |
| `max_tokens` | `int` | — | Maximum tokens for generation. |
**Returns:** [`GenieXLLM`](#geniexllm) or [`GenieXVLM`](#geniexvlm) (auto-detected based on the model)
***
## AutoModelForVision2Seq
Factory for loading vision-language / multimodal models. Returns a [`GenieXVLM`](#geniexvlm) instance.
```python theme={"dark"}
from geniex import AutoModelForVision2Seq
model = AutoModelForVision2Seq.from_pretrained("ai-hub-models/Qwen2.5-7B-Instruct", device_map="qairt")
```
### `from_pretrained()`
Accepts all the same parameters as [`AutoModelForCausalLM.from_pretrained()`](#from_pretrained) plus:
| Parameter | Type | Default | Description |
| ------------- | ------------- | ------- | ------------------------------------------------------------------------------- |
| `mmproj_path` | `str \| None` | `None` | Path to the multimodal projector file. Auto-resolved when downloading from Hub. |
**Returns:** [`GenieXVLM`](#geniexvlm)
***
## GenieXLLM
Text-only language model instance returned by `AutoModelForCausalLM.from_pretrained()`.
### `generate()`
Run text generation from a formatted prompt string.
```python theme={"dark"}
output = model.generate(prompt, max_new_tokens=256)
print(output.text)
```
| Parameter | Type | Default | Description |
| ---------------- | ------- | ---------- | -------------------------------------------------------------------------------------- |
| `prompt` | `str` | *required* | The formatted prompt string (use `model.tokenizer.apply_chat_template()` to build it). |
| `max_new_tokens` | `int` | `512` | Maximum number of tokens to generate. |
| `temperature` | `float` | `0.7` | Sampling temperature. |
| `top_p` | `float` | `0.9` | Nucleus sampling threshold. |
| `top_k` | `int` | `40` | Top-k sampling. |
| `min_p` | `float` | `0.0` | Minimum probability threshold. |
| `stream` | `bool` | `False` | If `True`, returns a [`TextIteratorStreamer`](#textiteratorstreamer) instead. |
| Parameter | Type | Default | Description |
| -------------------- | ------------------- | ------- | ----------------------------------------------- |
| `repetition_penalty` | `float` | `1.1` | Repetition penalty. |
| `presence_penalty` | `float` | `0.0` | Presence penalty. |
| `frequency_penalty` | `float` | `0.0` | Frequency penalty. |
| `seed` | `int` | `-1` | Random seed (`-1` = random). |
| `stop` | `list[str] \| None` | `None` | Stop sequences. |
| `grammar` | `str \| None` | `None` | GBNF grammar string for constrained generation. |
**Returns:** [`GenerateOutput`](#generateoutput) (or [`TextIteratorStreamer`](#textiteratorstreamer) when `stream=True`)
### `reset()`
Resets conversation state and clears the KV cache.
### `save_kv_cache(path)` / `load_kv_cache(path)`
Save or load the key-value cache to/from a file path (`str`).
### `close()`
Releases the model handle and frees resources. Also supports context-manager usage:
```python theme={"dark"}
with AutoModelForCausalLM.from_pretrained("qwen3") as model:
output = model.generate(prompt)
```
***
## GenieXVLM
Vision-language model instance returned by `AutoModelForVision2Seq.from_pretrained()`.
### `generate()`
Same parameters as [`GenieXLLM.generate()`](#generate) plus:
| Parameter | Type | Default | Description |
| --------- | ------------------- | ------- | -------------------------------------------------- |
| `images` | `list[str] \| None` | `None` | List of image file paths for the model to process. |
| `audios` | `list[str] \| None` | `None` | List of audio file paths for the model to process. |
```python theme={"dark"}
output = model.generate(prompt, images=["/path/to/image.jpg"], max_new_tokens=256)
print(output.text)
```
**Returns:** [`GenerateOutput`](#generateoutput) (or [`TextIteratorStreamer`](#textiteratorstreamer) when `stream=True`)
### `reset()` / `close()`
Same as [`GenieXLLM`](#geniexllm).
***
## ModelTokenizer
Accessed via `model.tokenizer`. Provides a `transformers`-compatible chat template interface.
### `apply_chat_template()`
Formats a list of chat messages using the model's built-in chat template.
```python theme={"dark"}
messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = model.tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True,
)
```
| Parameter | Type | Default | Description |
| ----------------------- | --------------------------- | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `messages` | `list[dict]` | *required* | List of message dicts with `"role"` and `"content"` keys. |
| `tokenize` | `bool` | `False` | Must be `False` — standalone tokenization is not supported. |
| `add_generation_prompt` | `bool` | `True` | Whether to append the generation prompt suffix. |
| `enable_thinking` | `bool \| None` | `None` | Enable thinking mode. `None` (default) and `True` both let a thinking-capable model think (and are no-ops on non-thinking models). `False` asks a thinking-capable model to skip its thinking turn — forced to `True` with a warning on non-thinking models, where the suppression block is OOD. Capability is auto-detected and exposed as `model.supports_thinking`. |
| `tools` | `list[dict] \| str \| None` | `None` | Tool definitions as a list of dicts or pre-serialised JSON string. |
**Returns:** `str` — formatted prompt ready for `model.generate()`.
***
## Output classes
### GenerateOutput
Returned by `model.generate()`.
| Attribute | Type | Description |
| ---------- | ----------------------------- | ------------------------------------------------------- |
| `text` | `str` | The generated text (thinking tags stripped if present). |
| `thinking` | `str \| None` | The model's reasoning content, or `None`. |
| `profile` | [`ProfileData`](#profiledata) | Performance metrics. |
### ProfileData
| Attribute | Type | Description |
| ------------------ | ------------- | ------------------------------------------------- |
| `ttft` | `int` | Time to first token (ms). |
| `prompt_tokens` | `int` | Number of prompt tokens. |
| `generated_tokens` | `int` | Number of generated tokens. |
| `prefill_speed` | `float` | Prefill speed (tokens/s). |
| `decode_speed` | `float` | Decode speed (tokens/s). |
| `stop_reason` | `str \| None` | Why generation stopped (e.g. `"eos"`, `"limit"`). |
| Attribute | Type | Description |
| ------------- | ----- | ---------------------------- |
| `prompt_time` | `int` | Prompt processing time (ms). |
| `decode_time` | `int` | Decode time (ms). |
### TextIteratorStreamer
Returned by `model.generate(..., stream=True)`. Yields decoded text chunks as they are generated.
```python theme={"dark"}
streamer = model.generate(prompt, max_new_tokens=256, stream=True)
for chunk in streamer:
print(chunk, end="", flush=True)
final = streamer.output # GenerateOutput available after iteration
```
| Method / Property | Description |
| ----------------- | -------------------------------------------------------------- |
| `__iter__()` | Yields `str` chunks as they are generated. |
| `output` | `GenerateOutput \| None` — available after iteration finishes. |
| `cancel()` | Stop generation at the next token boundary. |
***
## Model manager
The same model manager the CLI uses is available programmatically via `geniex.model_manager`.
```python theme={"dark"}
from geniex import model_manager as mm
MODEL = "Qwen/Qwen3-0.6B-GGUF"
mm.pull(MODEL)
print(f"pull complete: {MODEL}")
paths = mm.get_paths(MODEL)
print(f"model path: {paths}")
local_models = mm.list_models()
print(f"local models: {local_models}")
mm.remove(MODEL)
print(f"model removed: {MODEL}")
```
| Function | Description |
| ----------------------- | ---------------------------------------------------- |
| `pull(model_name, ...)` | Download a model by alias or `org/repo[:precision]`. |
| `list_models()` | Returns `list[str]` of cached model names. |
| `get_paths(model_name)` | Returns `ModelPaths` with resolved local file paths. |
| `get_type(model_name)` | Returns `"llm"` or `"vlm"`. |
| `resolve_alias(alias)` | Resolves a short alias to canonical `org/repo`. |
| `remove(model_name)` | Delete a cached model from disk. |
| `clean()` | Remove all cached models. Returns count removed. |
| Parameter | Type | Default | Description |
| ------------- | ------------------ | ---------- | ------------------------------------------------------------------------- |
| `model_name` | `str` | *required* | `org/repo` or a short alias. |
| `precision` | `str \| None` | `None` | Precision (quantization) hint (e.g. `"Q4_K_M"`). |
| `hub` | `str` | `"auto"` | `"auto"` \| `"hf"` \| `"localfs"`. |
| `local_path` | `str \| None` | `None` | Source directory (required when `hub="localfs"`). |
| `hf_token` | `str \| None` | `None` | HuggingFace bearer token. |
| `on_progress` | `Callable \| None` | `None` | Callback `(files: list[FileProgress]) -> bool`; return `False` to cancel. |
| Attribute | Type | Description |
| ---------------- | ------------- | ------------------------------------- |
| `model_path` | `str` | Path to the main model file. |
| `model_dir` | `str` | Directory containing the model. |
| `model_name` | `str` | Canonical model name. |
| `runtime` | `str` | Runtime identifier. |
| `mmproj_path` | `str \| None` | Multimodal projector path (VLM only). |
| `tokenizer_path` | `str \| None` | Tokenizer file path. |
| `compute_unit` | `str \| None` | Compute unit identifier. |
***
## SDK functions
| Function | Description |
| --------------------------------------- | --------------------------------------------------------------------------------------------------- |
| `geniex.init()` | Initialize the SDK. Called automatically on first model load. |
| `geniex.deinit()` | Shut down the SDK and release resources. |
| `geniex.version()` | Returns the SDK version string. |
| `geniex.get_runtime_list()` | Returns `list[str]` of available runtime IDs. |
| `geniex.get_compute_unit_list(runtime)` | Returns `list[tuple[str, str]]` of `(compute_unit, compute_unit_name)` pairs for the given runtime. |
# Python Install
Source: https://geniex.aihub.qualcomm.com/en/run/python/install
Install the GenieX Python SDK on Windows ARM64 or Linux ARM64.
The GenieX Python SDK ships as an ARM64 wheel for Windows ARM64 and Linux ARM64.
## **Prerequisites**
* Snapdragon X-series chipset (X Elite or X2 Elite). See [Supported platforms](/en/get-started/platforms).
* **ARM64 Python 3.10+** — x86\_64 / AMD64 builds are not supported.
## **Install via pip**
If Python is not installed, download [Python 3.13.3 for ARM64](https://www.python.org/ftp/python/3.13.3/python-3.13.3-arm64.exe).
Confirm Python is the ARM64 build — **must print `ARM64`** (not `AMD64`):
```powershell theme={"dark"}
python -c "import platform; print(platform.machine())"
```
Create a virtual environment and install:
```powershell theme={"dark"}
python -m venv geniex-env
.\geniex-env\Scripts\Activate.ps1
pip install -U geniex
```
This pulls the package from [PyPI](https://pypi.org/project/geniex/).
The Qualcomm embedded Linux (Yocto) ships with Python pre-installed. Verify with:
```bash theme={"dark"}
python3 --version
```
Directly pip install:
```bash theme={"dark"}
python3 -m pip install -U geniex
```
This pulls the package from [PyPI](https://pypi.org/project/geniex/) and auto-fetches the Linux ARM64 SDK libraries.
If Python is not pre-installed, see [How do I install Python on a Linux ARM64 device?](/en/resources/faq#how-do-i-install-python-on-a-linux-arm64-device) for a verified Miniconda recipe (covers Qualcomm Yocto and Ubuntu ARM64).
## **Verify**
```python theme={"dark"}
import geniex
print(geniex.version())
```
Now jump to the [Quickstart](/en/run/python/quickstart).
# Quickstart
Source: https://geniex.aihub.qualcomm.com/en/run/python/quickstart
Run your first model from the GenieX Python SDK on Windows ARM64.
## **Prerequisites**
* The Python SDK installed — see [Install](/en/run/python/install).
* Familiarity with [runtime choice](/en/get-started/platforms#geniex-runtimes) — `qairt` for Qualcomm AI Hub Models, `llama_cpp` for any GGUF.
The SDK follows the same design as Hugging Face `transformers` — load with `AutoModelForCausalLM.from_pretrained()`, then call `.generate()`.
## **LLM inference (GGUF)**
Any GGUF model from Hugging Face runs via `llama_cpp`. Model weights are downloaded on first use.
```python theme={"dark"}
from geniex import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-0.6B-GGUF", # HF repo id of a GGUF model, or a local .gguf path
device_map="auto", # "auto" | "cpu" | "gpu" | "npu" | "hybrid"
# | "" | ":"
# auto -> npu for both llama_cpp and qairt
)
messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = model.tokenizer.apply_chat_template(
messages, add_generation_prompt=True,
)
# One-shot
output = model.generate(prompt, max_new_tokens=256)
print(output.text)
print(f"[{output.profile.generated_tokens} tok, "
f"{output.profile.decode_speed:.1f} tok/s, stop={output.profile.stop_reason}]")
# Streaming
streamer = model.generate(prompt, max_new_tokens=256, stream=True)
for chunk in streamer:
print(chunk, end="", flush=True)
model.close()
```
## **LLM inference (QAIRT)**
Pre-compiled bundles from [Qualcomm AI Hub](https://aihub.qualcomm.com/models/) run entirely on the Hexagon NPU via the `qairt` runtime. Use `device_map="qairt"` (or `"npu"`). Model weights are downloaded on first use.
```python theme={"dark"}
from geniex import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"ai-hub-models/Qwen3-4B", # Qualcomm AI Hub model id
device_map="qairt", # NPU-only
)
messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = model.tokenizer.apply_chat_template(
messages, add_generation_prompt=True,
)
# One-shot
output = model.generate(prompt, max_new_tokens=256)
print(output.text)
print(f"[{output.profile.generated_tokens} tok, "
f"{output.profile.decode_speed:.1f} tok/s, stop={output.profile.stop_reason}]")
# Streaming
streamer = model.generate(prompt, max_new_tokens=256, stream=True)
for chunk in streamer:
print(chunk, end="", flush=True)
model.close()
```
## **VLM inference (QAIRT)**
Download a sample image first:
```bash theme={"dark"}
curl -o demo.jpg https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-geniex/demo.jpg
```
Then run inference:
```python theme={"dark"}
import os
from geniex import AutoModelForCausalLM
image_path = os.path.abspath("demo.jpg")
model = AutoModelForCausalLM.from_pretrained(
"ai-hub-models/Qwen2.5-VL-7B-Instruct", # Qualcomm AI Hub VLM bundle
device_map="qairt",
)
messages = [{
"role": "user",
"content": [
{"type": "image", "image": image_path},
{"type": "text", "text": "Describe the image."},
],
}]
prompt = model.tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True,
)
streamer = model.generate(prompt, images=[image_path], max_new_tokens=256, stream=True)
for chunk in streamer:
print(chunk, end="", flush=True)
model.close()
```
## **Audio inference (GGUF)**
Audio input runs on the **llama.cpp** backend with an audio-capable model — its mmproj must carry a conformer encoder. `google/gemma-4-E2B-it-qat-q4_0-gguf` is one such model. Load it with `AutoModelForVision2Seq`, then pass file paths to `generate(images=[...], audios=[...])` — a single call can carry both.
Audio input runs on the **llama.cpp** backend only. QAIRT bundles report `capabilities()['audio'] == False` and raise `GenieXError(-201201): Multimodal generation failed` if given audio.
Download a sample clip and photo:
```bash theme={"dark"}
curl -L -o jfk.wav https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav
curl -L -o landmark.jpg "https://images.pexels.com/photos/402028/pexels-photo-402028.jpeg?w=1024"
```
Then run inference:
```python theme={"dark"}
import os
from geniex import AutoModelForVision2Seq
image_path = os.path.abspath("landmark.jpg")
audio_path = os.path.abspath("jfk.wav")
model = AutoModelForVision2Seq.from_pretrained(
"google/gemma-4-E2B-it-qat-q4_0-gguf", # GGUF VLM with an audio mmproj
device_map="npu", # audio runs on npu / gpu / cpu, not qairt
)
# capabilities() confirms the loaded mmproj handles audio.
print(model.capabilities()) # -> {'vision': True, 'audio': True}
messages = [{
"role": "user",
"content": [
{"type": "image", "image": image_path},
{"type": "audio", "audio": audio_path},
{"type": "text", "text": "Describe the image, then transcribe the audio."},
],
}]
prompt = model.tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True,
)
output = model.generate(prompt, images=[image_path], audios=[audio_path], max_new_tokens=256)
print(output.text)
model.close()
```
Output (verified on `device_map="npu"`, Snapdragon X Elite):
```text theme={"dark"}
A photograph captures a stunning landscape featuring a traditional Japanese temple ... framed by softer, muted blue and green mountains under a clear, pale sky.
**Audio Transcription:**
"And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country."
```
## **Jupyter notebook walkthrough**
For laptop users, follow the step-by-step Jupyter notebook at [examples/python/windows.ipynb](https://github.com/qualcomm/GenieX/blob/main/examples/python/windows.ipynb) — it covers environment setup and inference end-to-end.
## **Next steps**
All classes, methods, and parameters for the Python SDK.
Supported models, GGUF on Hugging Face, and self-converted Qualcomm AI Engine Direct bundles.
# Audio input
Source: https://geniex.aihub.qualcomm.com/en/tutorials/audio-input
Transcribe and reason over audio on-device with an audio-capable VLM, in the GenieX CLI, local server, and Python SDK.
A vision-language model whose multimodal projector carries a **conformer audio encoder** can take audio alongside text and images. GenieX feeds the clip through llama.cpp's mtmd (multimodal) path — the same mechanism that handles images — so a single turn can mix text, image, and audio.
Audio is the **`llama_cpp`** path only. QAIRT bundles report `audio: false` and a QAIRT model given audio fails with `GenieXError(-201201): Multimodal generation failed`. Audio runs on `--compute npu` / `gpu` / `cpu`; the NPU is the default and fast path on Snapdragon.
## **Step 1: Pull an audio-capable model**
Audio support is not a metadata flag — it means the mmproj GGUF actually contains an audio encoder. `google/gemma-4-E2B-it-qat-q4_0-gguf` ships one: the `Q4_0` weights (≈3.1 GiB) and the conformer mmproj (≈0.9 GiB) are pulled together, \~4.0 GiB total.
```bash theme={"dark"}
geniex pull google/gemma-4-E2B-it-qat-q4_0-gguf
```
Grab a sample clip and photo to use in the examples below:
```bash theme={"dark"}
curl -L -o jfk.wav https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav
curl -L -o landmark.jpg "https://images.pexels.com/photos/402028/pexels-photo-402028.jpeg?w=1024"
```
## **Step 2: Run it with `geniex infer`**
Drop an **absolute** `.wav` / `.mp3` path into the prompt (or drag the file into your terminal). Audio and image paths are auto-detected, and one prompt can carry both:
```bash theme={"dark"}
geniex infer google/gemma-4-E2B-it-qat-q4_0-gguf \
-p "Describe the image and transcribe the audio. Image: /full/path/to/landmark.jpg Audio: /full/path/to/jfk.wav"
```
Output (verified on `--compute npu`, Snapdragon X Elite):
```text theme={"dark"}
**Image Description:**
This is a scenic, panoramic photograph that features a traditional Japanese temple ... The overall mood of the image is serene and beautiful.
**Audio Transcription:**
"And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country."
```
### Record from the microphone with `/mic`
In an interactive session (launch `geniex infer ` with no `-p`), the `/mic` command records a clip and feeds it straight in — handy when you don't have a file on disk. It only appears when the loaded model supports audio.
```text theme={"dark"}
> /mic
Recording is going on, press Ctrl-C to stop
```
`Ctrl-C` stops the recording; GenieX saves it to a temp `.wav` and transcribes it.
`/mic` needs **SoX** on your `PATH`. Without it, GenieX prints `SoX is not installed, some features may not work` at startup. Install it:
```bash theme={"dark"}
sudo apt install sox # Debian/Ubuntu
sudo yum install sox # RHEL/CentOS/Fedora
sudo pacman -S sox # Arch Linux
```
```powershell theme={"dark"}
winget install --id=ChrisBagwell.SoX -e
# then restart your terminal so sox is on PATH
```
## **Step 3: Use it on the local server**
Start the server and send an OpenAI-compatible `input_audio` content part. `input_audio.data` takes the same three formats as an image URL — a local path, an HTTP/HTTPS URL, or a base64 data URL. A single message can mix `image_url` and `input_audio`:
```bash theme={"dark"}
geniex serve
```
```bash theme={"dark"}
curl http://127.0.0.1:18181/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "google/gemma-4-E2B-it-qat-q4_0-gguf",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Describe the image, then transcribe the audio."},
{"type": "image_url", "image_url": {"url": "/full/path/to/landmark.jpg"}},
{"type": "input_audio", "input_audio": {"data": "/full/path/to/jfk.wav"}}
]
}
],
"max_tokens": 256
}'
```
**Running in Docker?** Local paths are resolved **inside the container**. The install command mounts `$PWD/data` to `/data` — drop your files there and pass `/data/jfk.wav`, or use an HTTP URL / base64 data URL to skip the filesystem entirely.
See [Local server](/en/run/cli/local-server#vlm-request) for the full request shape and the Python `openai`-client equivalent.
## **Step 4: Use it from the Python SDK**
Load with `AutoModelForVision2Seq` and pass paths to `generate(images=[...], audios=[...])` — a single call can carry both. `capabilities()` confirms the mmproj handles audio before you send anything:
```python theme={"dark"}
import os
from geniex import AutoModelForVision2Seq
image_path = os.path.abspath("landmark.jpg")
audio_path = os.path.abspath("jfk.wav")
model = AutoModelForVision2Seq.from_pretrained(
"google/gemma-4-E2B-it-qat-q4_0-gguf",
device_map="npu", # audio runs on npu / gpu / cpu, not qairt
)
print(model.capabilities()) # -> {'vision': True, 'audio': True}
messages = [{
"role": "user",
"content": [
{"type": "image", "image": image_path},
{"type": "audio", "audio": audio_path},
{"type": "text", "text": "Describe the image, then transcribe the audio."},
],
}]
prompt = model.tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True,
)
output = model.generate(prompt, images=[image_path], audios=[audio_path], max_new_tokens=256)
print(output.text)
model.close()
```
## **Supported formats and preprocessing**
| | Details |
| ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **CLI / prompt auto-detection** | `.wav` and `.mp3` extensions only. Any other path in the prompt is treated as an image. |
| **Server** | No extension allowlist — `input_audio.data` bytes are decoded by content, so anything llama.cpp can decode works. |
| **Decoder (llama.cpp)** | Recognizes WAV, MP3, and FLAC by magic bytes. Note the mismatch: a `.flac` **path** in a CLI prompt is misrouted to images even though the decoder itself handles FLAC — pass FLAC via the server or a `.wav`/`.mp3` on the CLI. |
| **Channels / sample rate** | Audio is always down-mixed to **mono** and resampled to the encoder's target rate (typically **16 kHz**; some architectures use 24 kHz). Handled automatically — no need to pre-convert. |
| **Duration** | No maximum. Clips shorter than the encoder's chunk length are zero-padded; longer clips are chunked automatically. |
Channel/sample-rate/duration handling lives in the bundled llama.cpp mtmd audio path (third-party), not in GenieX — behavior may shift with the pinned llama.cpp version.
## **Failure modes**
| Situation | What happens |
| --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Audio sent to a QAIRT model** | `capabilities()` reports `audio: false`; generation fails with `GenieXError(-201201): Multimodal generation failed`. QAIRT has no audio decode path. |
| **Audio sent to a vision-only VLM** (mmproj without an audio encoder) | The audio is **silently skipped** with a `model does not support audio input; skipping N audio file(s)` warning, and generation continues text-only. Check `capabilities()['audio']` first. |
| **Unreadable / unsupported audio file** | The file is dropped with a debug log. If that leaves fewer media than the chat template expects, generation fails later with `-201201`. |
## **Next steps**
* [CLI reference](/en/run/cli/reference#geniex-infer-vlm) — every `geniex infer` flag.
* [Local server](/en/run/cli/local-server#vlm-request) — the full OpenAI-compatible API.
* [Python quickstart](/en/run/python/quickstart) — VLM and audio inference from Python.
# Speculative decoding with MTP
Source: https://geniex.aihub.qualcomm.com/en/tutorials/speculative-decoding-mtp
Speed up decoding with a Multi-Token Prediction draft model, in the GenieX CLI and the local server.
Speculative decoding speeds up a large model without changing what it produces. A cheap **draft** proposes several tokens ahead, the large **target** verifies them all in one forward pass, and the accepted prefix is committed at once. Because verification costs one pass no matter how many tokens it checks, accepted tokens after the first are nearly free — and the output stays identical to what the target would have generated alone.
**MTP (Multi-Token Prediction)** is the strongest variant. Instead of a separately-trained small model whose guesses drift from the target's, MTP uses prediction heads trained *alongside* the target that read its own hidden states, so proposals come from the same distribution the target samples from.
Speculative decoding is **`llama_cpp`-only** and **text-only**. On the `qairt` runtime these settings are ignored with a warning. If you enable it on a multimodal model, GenieX runs the LLM path and drops image / audio content with a warning.
## **Step 1: Pull the model pair**
MTP needs a draft trained against **the exact target** you're running — an arbitrary small GGUF can't be substituted, and a mismatched pair fails at load time.
| Role | Model |
| ---------- | ---------------------------------------------------------- |
| **Target** | `google/gemma-4-26B-A4B-it-qat-q4_0-gguf:Q4_0` |
| **Draft** | `RachidAR/gemma-4-26B-A4B-it-qat-assistant-q4_0-gguf:Q4_0` |
```bash theme={"dark"}
geniex pull google/gemma-4-26B-A4B-it-qat-q4_0-gguf:Q4_0
geniex pull RachidAR/gemma-4-26B-A4B-it-qat-assistant-q4_0-gguf:Q4_0
```
Both models are resident at once, so plan for roughly 20 GB of free disk and enough RAM to hold the pair.
## **Step 2: Run it with `geniex infer`**
Three new flags turn it on — the type, the draft model, and how far ahead to draft:
```bash theme={"dark"}
geniex infer google/gemma-4-26B-A4B-it-qat-q4_0-gguf:Q4_0 --spec-type draft-mtp --draft-model RachidAR/gemma-4-26B-A4B-it-qat-assistant-q4_0-gguf:Q4_0 --draft-tokens 3
```
| Flag | Default | Effect |
| ---------------- | ------- | ------------------------------------------------------------------------------------ |
| `--spec-type` | *(off)* | Set to `draft-mtp` to enable MTP. |
| `--draft-model` | — | Catalogue name `org/repo[:precision]` or a local GGUF path. Required by `draft-mtp`. |
| `--draft-tokens` | `3` | Max draft tokens per verification step. |
| `--draft-min` | `0` | Min draft tokens per step. `0` = llama.cpp default. |
| `--draft-p-min` | `0` | Draft stops proposing below this confidence. `0` = llama.cpp default. |
When speculation is active, the profiling block gains a `draft accept` line — accepted draft tokens over total proposed. That line is your confirmation it engaged:
```text theme={"dark"}
decode speed: 56.4 tok/s
stop reason: eos
draft accept: 14/75 (18.7%)
```
Setup failure is non-fatal: if the draft context can't be built, GenieX logs `speculative decoding setup failed; falling back to plain decoding` and continues at normal speed.
## **Step 3: Use it on the local server**
Start the server, then send the MTP fields on the request body — speculation is configured **per request**, so `geniex serve` has no `--spec-type` flag.
```bash theme={"dark"}
geniex serve
```
`POST /v1/chat/completions` takes the same three settings as `spec_*` fields. Everything else is the standard OpenAI-compatible body:
```json Request body {8-10} theme={"dark"}
{
"model": "google/gemma-4-26B-A4B-it-qat-q4_0-gguf:Q4_0",
"messages": [
{
"role": "user",
"content": "Hello! Briefly introduce yourself."
}
],
"spec_type": "draft-mtp",
"spec_draft_model": "RachidAR/gemma-4-26B-A4B-it-qat-assistant-q4_0-gguf:Q4_0",
"spec_n_max": 3,
"nctx": 8192,
"max_completion_tokens": 2048,
"temperature": 0.8,
"top_p": 0.95,
"stream": false
}
```
The three highlighted fields are the only additions. `spec_type` is what enables MTP; `spec_draft_model` names the draft you pulled in Step 1; `spec_n_max` is `--draft-tokens` under another name. `spec_n_min` and `spec_p_min` are accepted too, matching `--draft-min` and `--draft-p-min`.
Paste this straight into the built-in Swagger UI at `http://127.0.0.1:18181` to try it — see [Local server](/en/run/cli/local-server) for the full API.
**The server never auto-downloads a draft model.** Unlike `geniex infer`, it only uses what's already cached, so a missing draft errors mid-request — complete [Step 1](#step-1-pull-the-model-pair) first.
The `spec_*` fields are part of the model cache key, so changing any of them rebuilds the model on the next request. Keep them stable across a run.
## **Next steps**
* [CLI reference](/en/run/cli/reference) — every `geniex infer` flag.
* [Local server](/en/run/cli/local-server) — the full OpenAI-compatible API.