On-device dictation: engines & what open source uses
A quick look at how open-source dictation apps do speech-to-text - model + CPU/GPU/NPU - plus a small hands-on benchmark of the engines relevant to Jot. Compiled 2026-07-23.
1. Benchmark - the engines that matter for us
Same 3 short English clips, one Windows PC, CPU only, model warmed (first-run load excluded). "RTF" = seconds of compute per second of audio (lower is faster).
| Engine (quant) | Size | Speed (RTF) | vs realtime | Accuracy* | Punct/caps | Numbers | Streaming | Multilingual | License |
|---|---|---|---|---|---|---|---|---|---|
| Nemotron 3.5 (int4) - Jot today |
~750 MB | ~0.48 | ~2� faster | perfect | yes | spoken ("twelve") | yes (cache-aware) | yes | NVIDIA |
| Parakeet v3 (int8) - Handy's default |
~500 MB | ~0.07 | ~14� faster | perfect | yes | digits ("12") | no (offline)? | 25 EU langs | CC-BY-4.0 |
| Whisper base (int8) - common baseline |
~150 MB | ~0.11 | ~9� faster | perfect (clean) | yes | digits | no | 99 langs | MIT |
Read this honestly. The clips are clean text-to-speech, which is optimistic and flattens accuracy differences - all three scored ~perfect. Real accented/noisy speech would separate the 600M models (Parakeet v3, Nemotron) from tiny Whisper-base. Both Parakeet and Whisper auto-convert numbers to digits (this made a spelled-out "twenty first" look like an error - it isn't). Nemotron was timed in batch mode, same as the others, so the speed comparison is fair - but note Jot actually runs Nemotron streaming for live captions. ? Parakeet v3 here is offline; a streaming Parakeet exists via sherpa-onnx. Single run each on one CPU - treat as rough, not lab-grade.
Takeaway: on CPU, Parakeet v3 is ~7� faster than our Nemotron 3.5, smaller, just as accurate on this test, and CC-BY-4.0 (shippable). Nemotron's advantage is that we run it as a real streaming engine. A Parakeet-v3 move would want a streaming export (sherpa-onnx) to keep live captions.
2. What the open-source dictation apps actually use
| App | Model(s) | Engine | CPU | GPU (Windows) | NPU | License | UI |
|---|---|---|---|---|---|---|---|
| Handy | Whisper, Parakeet v2/v3 | whisper.cpp + ONNX (transcribe-rs) | ? | Vulkan (Whisper only); Parakeet CPU-only | - | MIT | Tauri/Rust |
| Vibe | Whisper, Nemotron 3.5, Parakeet v3 | whisper.cpp + ONNX | ? | CUDA + Vulkan | - | MIT | Tauri/Rust |
| Buzz | Whisper | whisper.cpp + faster-whisper | ? | CUDA + Vulkan | - | MIT | Python/Qt |
| Whispering | Whisper, Parakeet, Moonshine, cloud | transcribe-rs | ? | Vulkan (Whisper); Parakeet CPU | - | MIT | Tauri/Svelte |
| VoiceInk (mac) | Whisper, Parakeet v3 | whisper.cpp + FluidAudio | ? | Metal | ? ANE | GPL-3 | Swift |
| WhisperWriter | Whisper | faster-whisper (CTranslate2) | ? | CUDA only | - | GPL-3 | Python/PyQt |
| nerd-dictation | Vosk / Kaldi | Vosk-API | ? | - | - | GPL-3 | Python |
| Jot (us) | Nemotron 3.5 int4 / FP16 | ONNX Runtime | ? | DirectML (FP16) | - | closed | C#/WPF |
Underlying engines: whisper.cpp (CPU, CUDA, Metal, Vulkan, CoreML?ANE, OpenVINO, Ascend) is the broadest; faster-whisper/CTranslate2 is CPU + NVIDIA-CUDA only (no AMD/Vulkan/NPU); sherpa-onnx runs Parakeet/Whisper/Zipformer on ONNX Runtime with CPU/CUDA/CoreML/DirectML + embedded NPUs (QNN, RKNN, Ascend); FluidAudio runs Parakeet on the Apple Neural Engine.
3. What we can learn
- Parakeet v3 is the field's rising default (Handy, Whispering, VoiceInk, Vibe) - and our benchmark shows why: fastest on CPU, small, accurate, permissive license. Worth seriously considering as Jot's engine (with a streaming export to keep live captions).
- On Windows, the field GPU-accelerates with Vulkan, not DirectML - and nobody GPU-accelerates Parakeet in shipping builds (Handy explicitly leaves it CPU-only). Jot's DirectML path is unusual; a Parakeet-on-GPU (or NPU) Jot would be ahead of the pack.
- NPU is Apple-only in practice. VoiceInk/FluidAudio run Parakeet on the Apple Neural Engine. No shipping Windows dictation app uses the NPU (QNN/OpenVINO are library-level, experimental). If Jot shipped NPU, it'd be close to first.
- Nemotron 3.5 is niche - only Vibe also ships it. Our benchmark suggests it's not obviously worth its size/speed cost over Parakeet v3 on CPU, unless its streaming quality is the deciding factor.
- Ideas worth borrowing: one interface over two engines (whisper.cpp for Whisper + ONNX for Parakeet, � la
transcribe-rs); model download with a manual mirror for firewall/corporate networks (Handy); and configurable paste method + clipboard fallback - Handy uses synthetic input (enigo) withpaste_method/clipboardsettings, i.e. they hit the same paste problem we did and solved it the same way.