Getting started with local audio AI
Audio is a friendly place to start with local AI. Many useful audio models are small enough to run on a laptop CPU, and the results are easy to judge: you either understand the transcript or you do not. What follows is the three jobs audio models do, how their hardware needs differ, and what your first run will actually look like.
The three jobs
Almost everything people do with local audio models falls into one of three buckets.
Speech to text. You give the model recorded audio and it gives you words, usually with rough timings. This is transcription, subtitling, meeting notes, and search over your own recordings. It is the most mature and most reliable of the three.
Text to speech. You give the model written text and it gives you spoken audio. Used for narration, screen readers, notification voices, and giving a local assistant a voice. Copying a specific person's voice from a sample is a related but separate job, covered under voice cloning; for synthesis in general, see text to speech.
Music and sound generation. You describe a piece of music or a sound and the model composes audio from scratch. This is the least predictable of the three and the most expensive to run. Music generation goes deeper.
Pick one job to start. The tooling, the file formats and the quality bar are different enough that trying all three at once mostly produces confusion.
Hardware, and why it varies so much
The three jobs sit at very different points on the hardware curve, so whatever machine you already own is probably enough for at least one of them.
Speech recognition and most text-to-speech models are small. Their parameter counts run to tens or hundreds of millions rather than billions, so they fit in ordinary system memory and run on a CPU. A GPU speeds transcription up considerably and is worth using if you have one, but you can start without. Small text-to-speech models in particular are built to run faster than real time on modest hardware, single-board computers included.
Music and sound generation is a different story. These models are larger, they generate in a loop over many steps, and they want a GPU with real VRAM behind it. They will usually run on a CPU, but a short clip can take long enough that you stop experimenting, and experimenting is the work. If you plan to spend most of your time here, read the hardware pages before you buy anything.
The rule that follows from all this: if a job involves understanding audio, assume it is cheap. If it involves inventing audio, assume it is expensive.
Pick a job, then a model
Once you know which job you want, choose a model to try first. The audio models index lists the common starting points for each job, with a page each on what they are, what sizes they come in and what hardware they want. Start with the smallest variant of whichever model you pick. It downloads faster, it tells you whether your setup works, and if the quality is good enough you have saved yourself the larger download.
Expect the first run to be slow for reasons that have nothing to do with speed. The model weights have to be downloaded, which can take a while on a domestic connection, and most tools do this silently the first time you ask for something. Some stacks also fetch a separate audio decoder or language pack. After that the weights are cached on disk and startup is quick.
A few other first-run expectations worth setting:
- Transcription output is text plus timings. Turning that into a finished subtitle file is a separate step your tool may or may not do.
- Synthesised speech sounds flat until you match the model to the voice and language you actually want. The default voice is rarely the one you were hoping for.
- Generated music arrives as a short clip. Length is a hard constraint on most models, and stitching clips into something longer is your job.
- Everything runs offline once the weights are local, which is the main reason to do this at all. Audio is personal data, and keeping recordings on your own disk beats handing them to a hosted service.
When your first model works end to end, the useful next step is either a larger variant of the same model or one of the other two jobs. Both paths start from the same index.