Running OpenWhispr's AI on the Snapdragon NPU

OpenWhispr is an open-source, privacy-first voice-to-text app for macOS, Windows, and Linux — press a hotkey, speak, and your words appear at your cursor. Its local mode keeps everything on-device: speech-to-text via Whisper, Parakeet, and Orukeet, on-device speaker diarization, semantic note search, and local LLMs for AI text processing. Audio never leaves your machine. No telemetry, no data collection, MIT licensed.

Until now, "local" on Windows meant running those models on the CPU or GPU. The newest work in the repo goes further: on Windows-on-Snapdragon machines, OpenWhispr can now run inference on the Hexagon NPU — the dedicated AI accelerator inside Snapdragon X-series chips — for both transcription and language models.

Part 1: NPU-accelerated Whisper

Local Whisper transcription now runs through a small Python sidecar (resources/qualcomm/whisper_npu_sidecar.py) that loads the Whisper encoder and decoder as ONNX graphs and executes them on the Hexagon NPU via ONNX Runtime's QNN Execution Provider. A few details make it fast and reliable:

  • Precompiled chipset exports. The models are fetched as precompiled QNN exports for your specific Snapdragon chipset, so the NPU runs the graph natively rather than JIT-compiling it.
  • Tuned HTP settings. The session enables HTP fp16 precision, burst performance mode, and aggressive graph-finalization optimization — mirroring the settings Qualcomm uses in its own AI Hub reference apps, which the implementation was adapted from.
  • A clean process boundary. The sidecar speaks a tiny line-based JSON protocol over stdin/stdout ({"id": 1, "wav": "…", "language": "auto"} → {"id": 1, "ok": true, "text": "…"}). The Electron main process converts audio to 16 kHz mono WAV with ffmpeg, hands it over, and gets text back — keeping the heavyweight Python/QNN stack isolated from the app.

Setup is handled for the user: a QualcommRuntimeManager provisions an app-owned Python venv with pinned versions (onnxruntime-qnn==2.7.0, NumPy, Hugging Face transformers) and verifies the runtime before any NPU model is offered.

Part 2: LLMs on the NPU via GenieX

The same branch extends NPU acceleration to AI reasoning. Qualcomm's GenieX runtime — a CLI that exposes an OpenAI-compatible HTTP API backed by the Hexagon NPU (llama.cpp with Q4_0 GGUFs, or QAIRT bundles) — is managed by a geniexServer.js helper. It mirrors the interface of the existing llama.cpp server manager, so any model flagged runtime: "geniex" in the model registry runs on the NPU through the exact same inference path users already know. Details like routing streaming chat through the GenieX server, serving with the model ID the server actually hosts, and suppressing "thinking" tokens on that transport are all handled in the branch.

Gating: only offer what the hardware can run

NPU features are strictly gated so the pickers never offer something a machine can't execute. The check requires all of:

  1. Windows on ARM64 — the QNN runtime and precompiled exports are native ARM64; x64-under-emulation can't reach the NPU.
  2. A real Qualcomm processor — an ARM64 Windows machine isn't necessarily a Snapdragon, so the app queries WMI for the CPU name and matches for "Snapdragon"/"Qualcomm". A false negative just falls back to the CPU/GPU tab; a false positive would offer inference that can't run.

Note: the two pull requests stack — #3037 builds on #3032, so the Whisper commits appear in both until #3032 merges.


This write-up was generated by GLM 5.3 Flash (Z.ai), based on a review of the OpenWhispr repository and its pull requests. Code references and links were verified against the repo at the time of writing; details may change as the pull requests evolve.

Screenshot 2026-10-312 145012.png
Screenshot 2026-10-312 145012.png

Rejoining the server...

Rejoin failed... trying again in seconds.

Failed to rejoin.
Please retry or reload the page.

The session has been paused by the server.

Failed to resume the session.
Please reload the page.

An unhandled error has occurred. Reload 🗙