Audio and Text Tuning for Video Motion Editing
Motion-centric video editing remains difficult for large generative video models, which often respond well to appearance changes but struggle to produce specific, localized actions or state transitions in an existing clip. We introduce Sound Sparks Motion, a training-free framework that enables motion editing in an audio-visual video generation model by tuning its internal multimodal conditioning signals at test time.
Rather than modifying model weights, our method tunes only two lightweight variables: an audio latent derived from the source video and a residual perturbation in the text-conditioning. We find that this combination can encourage motion edits that the underlying model often struggles to realize under prompt-only control.
Since there is no direct way to evaluate temporal alignment between text and motion, we guide the tuning process using a vision–language model that provides feedback indicating whether the intended motion appears in the generated video. This simple supervision yields an effective semantic objective for motion editing, while regularization and perceptual-temporal constraints help preserve content and visual quality. Beyond per-video tuning, we show that the learned latent controls are transferable across videos, suggesting that they capture reusable motion-edit directions rather than overfitting to a single example.
Our method turns a frozen retake-style video editor into a motion-aware editor through test-time latent tuning — no fine-tuning required.
The LTX-2 video generation model weights remain entirely frozen. We never backpropagate into the model parameters.
We tune two lightweight variables — an audio latent and a text residual — via gradient descent. Qwen2.5-VL provides a differentiable motion-alignment signal as supervision.
The optimized conditioning embeddings are injected into the Retake pipeline to regenerate only the target time window, preserving the rest of the video.
The full supplementary video and the baseline comparison reel.
Hit Spark the motion! on any card to apply its edit.
Percentages are from a human study of 25 scenarios with 15 participants (375 judgements): on each card, how often ours was preferred, judged more natural, and judged the stronger edit, against the same for H3 alone. Ties make up the rest. These clips carry sound.
The audio latent alone, the text residual alone, and the two together, beside H3 with no editing. The reel opens with the human study over all 12 scenarios (15 participants), then shows the 5 where the combination won clearly.
Twelve scenarios against seven baselines. Pick a scenario to compare. In a study of 21 participants, our result was ranked first in 56.3% of trials, against 21.4% for the next best method.
Measured against seven recent video-editing systems on the same 12 motion-edit scenarios — judged both by people and by a vision–language model.
Fraction of first-place selections.
Fraction of scenarios where the edit was achieved.
The audio latent and text residual tuned on one clip, applied to another. Hit Transfer the motion! to carry them across.
Average cross-attention for the two tuned signals, over the first frame of the result. A2V is audio-to-video, T2V is text-to-video; switch between them on any card.
Every variant of a scenario plays at once. Each reel replays its first scenario at half speed.
Temporarily anonymized for review.