MusicGen for local music generation
MusicGen is a text-to-music model from Meta's AI research group, released with open weights as part of their audio generation toolkit. You describe a piece of music in a sentence and it composes a short instrumental clip to match. It is the music entry in the audio models index, and the heaviest of the three models covered there.
Mechanically it is closer to a language model than to an image generator. A separate neural codec compresses audio into discrete codes, MusicGen predicts those codes the way a text model predicts the next token, and the codec decodes them back into a waveform. That is why generation time scales with clip length, and why there is a hard ceiling on how much audio comes out of one pass.
Sizes and hardware
MusicGen comes in a few sizes, from a small model of a few hundred million parameters up to a large one in the low billions. There are variants too: a melody-conditioned version that takes a reference tune alongside your text prompt and follows its shape, and stereo versions of the main sizes.
This is a GPU model. It will run on CPU, but generating a clip that way takes long enough that experimenting stops being fun, and experimenting is most of the work with generative audio. On a discrete GPU the small model asks for a modest slice of VRAM and the large model a good deal more, especially in stereo and at longer clip lengths, since the decoder holds the whole generated sequence. A mid-range card of the kind covered on the hardware pages will run the smaller sizes comfortably, while the large model wants more headroom than an entry-level card gives you. Cards like the RTX 3090 and RTX 4090 have room to spare. Apple machines with unified memory run it through Metal-backed builds, generally slower than an equivalent discrete card but without the memory ceiling.
Bigger is not automatically better here. The larger models follow prompts more closely and sound more coherent, while the small model lets you try a pile of prompts in the time the large one spends on a handful. For generative work that iteration speed often matters more than per-clip quality.
How people run it
The reference route is the research toolkit its authors published: a Python library that loads the weights, takes a text prompt and a duration, and returns audio. Around that sit the usual layers:
- A simple web UI with a prompt box, a duration slider and a play button, which is how most people first meet it.
- Node graph editors. Audio generation nodes exist for the same node-based tools people use for image work, letting you chain generation with post-processing; the ComfyUI page covers that style of interface.
- Batch scripts that run a list of prompts overnight and write numbered WAV files, which is the practical way to build a library of usable clips.
Generation is driven by a text prompt describing genre, instrumentation, mood and tempo in words, optionally plus a reference audio clip for the melody-conditioned variant. Longer pieces come from generating a clip and then continuing from its tail, which works, though the music tends to drift over several extensions.
Check the licence on the weights before you publish anything made with them. The code and the weights are released under different terms, and the weights carry restrictions that the permissively licensed code does not.
Fit
MusicGen is the right tool for short instrumental beds: background music for a video, a loop for a game prototype, placeholder scoring, ambient texture, or a sketch of an idea you will re-record later. It responds well to genre and instrumentation cues, and it produces something usable quickly and offline.
Several things people ask of it are outside what it does. It does not sing: there are no vocals and no lyrics, and prompting for them produces vocal-like noise at best. It hands back a finished mix, with no stems and no MIDI score, so editing what comes out is awkward. Its sense of song structure runs to a short clip, which leaves verses, choruses and arrangement to you. And sound effects are a different job, though related models in the same family target it. Music generation covers the wider set of options and how people fit them into a workflow.