Image to video
Image to video starts from a picture you already have and puts it in motion. Of the three approaches on the techniques hub it is the one most people settle on, because it splits the hard problem in two: you decide what the shot looks like in a still workflow, then ask a local video model for one thing only, which is how it moves.
How it works
The model is given your still as a condition, not as a suggestion. It denoises a block of frames the way a text to video model does, but the first frame is pinned to the picture you supplied, and every other frame is generated to be consistent with it. Some models accept a target frame as well and fill in the motion between the two, which is a different kind of control again: you say where the shot starts and where it ends, and the model works out the middle.
A prompt is usually still accepted, and it is best used to describe motion rather than content. The content argument is already over: it was settled by the image. Extra motion modules, often distributed as LoRA-style add-ons, can push a model towards a particular kind of movement such as a camera push or an orbit. See video models for the models currently shaped around a conditioning image, and image generation for producing the still in the first place.
What it is good at
- Anything that has to look like a specific thing. Character, product, location, palette: if it is right in the still, it starts right in the clip.
- Cheap iteration. Rerolling a still costs a fraction of a video run, so you can fix composition before spending anything on motion.
- Ambient motion: hair, cloth, water, steam, firelight, foliage, a slow camera move over a static scene.
- Turning existing artwork, photographs or renders into short clips without recreating them.
- Reusing a look across shots, because every clip can start from a frame of the same still.
Where it struggles
- Drift away from the source. The first frame matches your still exactly. By the end of the clip, colours have shifted, details have been rewritten and the framing has crept.
- Morphing on articulated subjects. Hands, legs, tools and anything that crosses in front of something else deform as the model guesses at what is behind them.
- Too little or too much motion. The two common failure modes are a clip that barely moves and a clip where the whole scene boils. Most models expose a motion strength control, and it is coarse.
- Motion that ignores the prompt. Asking for a specific action rarely works. The model mostly infers plausible movement from the picture, so a still that implies no action tends to produce none.
- Subjects that were never separate. If the still has a subject fused into the background, motion will smear the two together rather than separating them.
The usual fix for most of these is a shorter clip. If the source still is complicated, generate less motion and stretch it later with frame interpolation.
Cost
A video run costs roughly what an equivalent text to video run costs, since the same block of frames is being denoised. The saving is in attempts, not in any one run: a conditioning image removes most of the rerolls, so the total spend on a finished shot is far lower. VRAM is still the limit, and clip length is still the first dial to turn when you run out. See GPUs for local AI for what fits where.