Quickstart

Prerequisites

The Python SDK installed — see Install.
Familiarity with runtime choice — qairt for Qualcomm AI Hub Models, llama_cpp for any GGUF.

The SDK follows the same design as Hugging Face transformers — load with AutoModelForCausalLM.from_pretrained(), then call .generate().

LLM inference (GGUF)

Any GGUF model from Hugging Face runs via llama_cpp. Model weights are downloaded on first use.

from geniex import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B-GGUF",     # HF repo id of a GGUF model, or a local .gguf path
    device_map="auto",          # "auto" | "cpu" | "gpu" | "npu" | "hybrid"
                                # | "<runtime>" | "<runtime>:<compute-unit>"
                                # auto -> npu for both llama_cpp and qairt
)

messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = model.tokenizer.apply_chat_template(
    messages, add_generation_prompt=True,
)

# One-shot
output = model.generate(prompt, max_new_tokens=256)
print(output.text)
print(f"[{output.profile.generated_tokens} tok, "
      f"{output.profile.decode_speed:.1f} tok/s, stop={output.profile.stop_reason}]")

# Streaming
streamer = model.generate(prompt, max_new_tokens=256, stream=True)
for chunk in streamer:
    print(chunk, end="", flush=True)

model.close()

LLM inference (QAIRT)

Pre-compiled bundles from Qualcomm AI Hub run entirely on the Hexagon NPU via the qairt runtime. Use device_map="qairt" (or "npu"). Model weights are downloaded on first use.

from geniex import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "ai-hub-models/Qwen3-4B",   # Qualcomm AI Hub model id
    device_map="qairt",         # NPU-only
)

messages = [{"role": "user", "content": "What is 2+2?"}]
prompt = model.tokenizer.apply_chat_template(
    messages, add_generation_prompt=True,
)

# One-shot
output = model.generate(prompt, max_new_tokens=256)
print(output.text)
print(f"[{output.profile.generated_tokens} tok, "
      f"{output.profile.decode_speed:.1f} tok/s, stop={output.profile.stop_reason}]")

# Streaming
streamer = model.generate(prompt, max_new_tokens=256, stream=True)
for chunk in streamer:
    print(chunk, end="", flush=True)

model.close()

VLM inference (QAIRT)

Download a sample image first:

curl -o demo.jpg https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-geniex/demo.jpg

Then run inference:

import os
from geniex import AutoModelForCausalLM

image_path = os.path.abspath("demo.jpg")

model = AutoModelForCausalLM.from_pretrained(
    "ai-hub-models/Qwen2.5-VL-7B-Instruct",  # Qualcomm AI Hub VLM bundle
    device_map="qairt",
)
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image_path},
        {"type": "text", "text": "Describe the image."},
    ],
}]
prompt = model.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
)

streamer = model.generate(prompt, images=[image_path], max_new_tokens=256, stream=True)
for chunk in streamer:
    print(chunk, end="", flush=True)

model.close()

Audio inference (GGUF)

Audio input runs on the llama.cpp backend with an audio-capable model — its mmproj must carry a conformer encoder. google/gemma-4-E2B-it-qat-q4_0-gguf is one such model. Load it with AutoModelForVision2Seq, then pass file paths to generate(images=[...], audios=[...]) — a single call can carry both.

Audio input runs on the llama.cpp backend only. QAIRT bundles report capabilities()['audio'] == False and raise GenieXError(-201201): Multimodal generation failed if given audio.

Download a sample clip and photo:

curl -L -o jfk.wav https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav
curl -L -o landmark.jpg "https://images.pexels.com/photos/402028/pexels-photo-402028.jpeg?w=1024"

Then run inference:

import os
from geniex import AutoModelForVision2Seq

image_path = os.path.abspath("landmark.jpg")
audio_path = os.path.abspath("jfk.wav")

model = AutoModelForVision2Seq.from_pretrained(
    "google/gemma-4-E2B-it-qat-q4_0-gguf",  # GGUF VLM with an audio mmproj
    device_map="npu",                       # audio runs on npu / gpu / cpu, not qairt
)
# capabilities() confirms the loaded mmproj handles audio.
print(model.capabilities())  # -> {'vision': True, 'audio': True}

messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image_path},
        {"type": "audio", "audio": audio_path},
        {"type": "text", "text": "Describe the image, then transcribe the audio."},
    ],
}]
prompt = model.tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
)

output = model.generate(prompt, images=[image_path], audios=[audio_path], max_new_tokens=256)
print(output.text)

model.close()

Output (verified on device_map="npu", Snapdragon X Elite):

A photograph captures a stunning landscape featuring a traditional Japanese temple ... framed by softer, muted blue and green mountains under a clear, pale sky.

**Audio Transcription:**
"And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country."

Jupyter notebook walkthrough

For laptop users, follow the step-by-step Jupyter notebook at examples/python/windows.ipynb — it covers environment setup and inference end-to-end.

Next steps

API reference

All classes, methods, and parameters for the Python SDK.

Models

Supported models, GGUF on Hugging Face, and self-converted Qualcomm AI Engine Direct bundles.

Was this page helpful?

Yes

Get Started

Run

Tutorials

Resources

Prerequisites

LLM inference (GGUF)

LLM inference (QAIRT)

VLM inference (QAIRT)

Audio inference (GGUF)

Jupyter notebook walkthrough

Next steps

API reference

Models

​Prerequisites

​LLM inference (GGUF)

​LLM inference (QAIRT)

​VLM inference (QAIRT)

​Audio inference (GGUF)

​Jupyter notebook walkthrough

​Next steps

API reference

Models

Prerequisites

LLM inference (GGUF)

LLM inference (QAIRT)

VLM inference (QAIRT)

Audio inference (GGUF)

Jupyter notebook walkthrough

Next steps