Whisper for local speech recognition
Whisper is a speech recognition model family from OpenAI, released with open weights, and the speech-to-text entry in the audio models index. You give it recorded audio and it gives you text, with rough timings per segment. It was trained on a very large and very varied pile of audio, which is why it copes with accents, background noise and phone-quality recordings.
Whisper is multilingual. It detects the spoken language and transcribes it, and it can translate speech in other languages into English text. Under the hood it is an ordinary encoder-decoder transformer, so the same attention machinery that powers text models applies here, with audio spectrogram slices going in instead of words.
Sizes and hardware
Whisper ships as a ladder of sizes rather than a single model, from tiny through to large. Each step up increases the parameter count, the download and the accuracy, and each step costs more time and memory per minute of audio. Most sizes also have an English-only variant, which tends to do better on English than the multilingual model of the same size and is lighter to load.
The small end is genuinely small. The tiny and base models transcribe on an ordinary laptop CPU with no GPU involved. They make mistakes on proper nouns and unclear speech, but for searchable notes on your own recordings they are often enough.
The large end wants a GPU. It will still run on CPU, and if you are transcribing overnight that may be fine, but on a card the difference is dramatic. Its VRAM appetite is modest, so almost any discrete GPU covered on the hardware pages will hold the largest Whisper. Apple machines with unified memory run it well through Metal-backed tooling.
Quantized builds of each size are widely available and cut memory and load time further at a small cost in accuracy. On CPU they are usually the version you want.
How people run it
Nobody much runs the original research code. The common paths are:
- A C++ port. A self-contained program that loads a converted weight file and transcribes on CPU, with optional GPU acceleration. This is the lightest way to run Whisper and the usual choice for embedding transcription into something else or running it on a small always-on machine.
- A fast inference engine. Reimplementations built on optimised inference libraries, which batch aggressively and run considerably faster on a GPU than the reference implementation while producing the same kind of output.
- A wrapper with extras. Tools that run Whisper and then add word-level timing alignment and speaker diarisation, which is what you need for subtitles or meeting notes where who-said-what matters.
- Desktop apps and media tools that bundle one of the above behind a drag-and-drop interface.
All of them take an audio file, split it into short windows, and emit segments of text with start and end times. Turning those into a subtitle file, a transcript or a search index is downstream work your tool may or may not do for you.
Fit
Whisper is the right tool when you have recorded audio and want words out of it: interviews, lectures, podcasts, voice memos, video you want to subtitle, or a personal archive you want to search. It holds up on noisy real-world recordings and on languages other than English, though quality varies by language and drops off for those with less training data behind them.
There are places it does not belong. It is built for recorded files rather than live streaming, and while people do chunk audio and feed it in near real time, dedicated streaming recognisers handle that better. Separating voices from each other takes a second model alongside it. It produces no speech, so for the opposite direction see Piper and the wider text to speech options.
One failure mode is worth knowing before it bites you: Whisper hallucinates on silence and non-speech audio, confidently transcribing words nobody said. Trim long stretches of music or room tone, or put a voice activity filter in front of it.