Audio and Text Tuning for Video Editing
Motion-centric video editing remains difficult for large generative video models, which often respond well to appearance changes but struggle to produce specific, localized actions or state transitions in an existing clip. We introduce Sound Sparks Motion, a training-free framework that enables motion editing in an audio-visual video generation model by tuning its internal multimodal conditioning signals at test time.
Rather than modifying model weights, our method tunes only two lightweight variables: an audio latent derived from the source video and a residual perturbation in the text-conditioning. We find that this combination can encourage motion edits that the underlying model often struggles to realize under prompt-only control.
Since there is no direct way to evaluate temporal alignment between text and motion, we guide the tuning process using a vision–language model that provides feedback indicating whether the intended motion appears in the generated video. This simple supervision yields an effective semantic objective for motion editing, while regularization and perceptual-temporal constraints help preserve content and visual quality. Beyond per-video tuning, we show that the learned latent controls are transferable across videos, suggesting that they capture reusable motion-edit directions rather than overfitting to a single example.
Our method turns a frozen retake-style video editor into a motion-aware editor through test-time latent tuning — no fine-tuning required.
The LTX-2 video generation model weights remain entirely frozen. We never backpropagate into the model parameters.
We tune two lightweight variables — an audio latent and a text residual — via gradient descent. Qwen2.5-VL provides a differentiable motion-alignment signal as supervision.
The optimized conditioning embeddings are injected into the Retake pipeline to regenerate only the target time window, preserving the rest of the video.
Every clip below starts as its untouched source. Hit Spark the motion! and the edit is applied in place — same shot, same framing, new motion. Transfer results reuse controls learned on a different video.
Measured against seven recent video-editing systems on the same 12 motion-edit scenarios — judged both by people and by a vision–language model.
Fraction of first-place selections.
Fraction of scenarios where the edit was achieved.
Temporarily anonymized for review.