On-device dictation: engines & what open source uses

A quick look at how open-source dictation apps do speech-to-text - model + CPU/GPU/NPU - plus a small hands-on benchmark of the engines relevant to Jot. Compiled 2026-07-23.

1. Benchmark - the engines that matter for us

Same 3 short English clips, one Windows PC, CPU only, model warmed (first-run load excluded). "RTF" = seconds of compute per second of audio (lower is faster).

Engine (quant)SizeSpeed (RTF)vs realtime Accuracy*Punct/capsNumbersStreamingMultilingualLicense
Nemotron 3.5 (int4)
- Jot today
~750 MB~0.48~2� faster perfectyesspoken ("twelve")yes (cache-aware) yesNVIDIA
Parakeet v3 (int8)
- Handy's default
~500 MB~0.07~14� faster perfectyesdigits ("12")no (offline)? 25 EU langsCC-BY-4.0
Whisper base (int8)
- common baseline
~150 MB~0.11~9� faster perfect (clean)yesdigitsno 99 langsMIT

Read this honestly. The clips are clean text-to-speech, which is optimistic and flattens accuracy differences - all three scored ~perfect. Real accented/noisy speech would separate the 600M models (Parakeet v3, Nemotron) from tiny Whisper-base. Both Parakeet and Whisper auto-convert numbers to digits (this made a spelled-out "twenty first" look like an error - it isn't). Nemotron was timed in batch mode, same as the others, so the speed comparison is fair - but note Jot actually runs Nemotron streaming for live captions. ? Parakeet v3 here is offline; a streaming Parakeet exists via sherpa-onnx. Single run each on one CPU - treat as rough, not lab-grade.

Takeaway: on CPU, Parakeet v3 is ~7� faster than our Nemotron 3.5, smaller, just as accurate on this test, and CC-BY-4.0 (shippable). Nemotron's advantage is that we run it as a real streaming engine. A Parakeet-v3 move would want a streaming export (sherpa-onnx) to keep live captions.

2. What the open-source dictation apps actually use

AppModel(s)EngineCPUGPU (Windows)NPULicenseUI
HandyWhisper, Parakeet v2/v3whisper.cpp + ONNX (transcribe-rs)?Vulkan (Whisper only); Parakeet CPU-only-MITTauri/Rust
VibeWhisper, Nemotron 3.5, Parakeet v3whisper.cpp + ONNX?CUDA + Vulkan-MITTauri/Rust
BuzzWhisperwhisper.cpp + faster-whisper?CUDA + Vulkan-MITPython/Qt
WhisperingWhisper, Parakeet, Moonshine, cloudtranscribe-rs?Vulkan (Whisper); Parakeet CPU-MITTauri/Svelte
VoiceInk (mac)Whisper, Parakeet v3whisper.cpp + FluidAudio?Metal? ANEGPL-3Swift
WhisperWriterWhisperfaster-whisper (CTranslate2)?CUDA only-GPL-3Python/PyQt
nerd-dictationVosk / KaldiVosk-API?--GPL-3Python
Jot (us)Nemotron 3.5 int4 / FP16ONNX Runtime?DirectML (FP16)-closedC#/WPF

Underlying engines: whisper.cpp (CPU, CUDA, Metal, Vulkan, CoreML?ANE, OpenVINO, Ascend) is the broadest; faster-whisper/CTranslate2 is CPU + NVIDIA-CUDA only (no AMD/Vulkan/NPU); sherpa-onnx runs Parakeet/Whisper/Zipformer on ONNX Runtime with CPU/CUDA/CoreML/DirectML + embedded NPUs (QNN, RKNN, Ascend); FluidAudio runs Parakeet on the Apple Neural Engine.

3. What we can learn