Sync music to the beat of a short video with AI

Read the beat times from the music, then start each cut on one of them. On Sume that means running reference ingest on a clip whose soundtrack is the music, which returns beat_times under audio.music, and placing Timeline 1.0 video[] starts on those times. No single Sume call does the whole job.

YouTube names "syncing music to beat" as one of the refinements in its Gemini editing flow (Made on YouTube, read 2026-09-29). The steps below are built from Sume's reference ingest and Timeline docs.

Where do beat times come from?

Reference ingest reads one media.sume.com clip of up to 300 seconds and returns a manifest whose audio facts include librosa beats when the track is music. The manifest's music block carries present, tempo_bpm, beat_times[] and confidence, in seconds on the source clock. The route is dest first: production lists it only when the operator opts in, so check tools_list before you build on it.

Can I ask Music 1.0 for a tempo and trust it?

You can write a BPM in the prompt, but the Music docs call tempo information from the provider model-reported metadata, not an audio measurement. Music 1.0 also rejects a duration field. Measure the beats of the audio you actually have.

How do I place cuts on the beats?

Detach the clip's audio as the Timeline spine with POST /v1/audio-detach, then give each video[] slot a start on a beat. The first slot must start at 0, starts must increase, and each slot lasts at least 0.2 seconds. Cutting every fourth beat gives one cut a bar in 4/4:

def slots(beats, clips, total, every=4):
    cuts = [0.0] + [round(b, 3) for b in beats[every - 1::every]
                    if 0.2 < b < total - 0.2]
    out = []
    for i, start in enumerate(cuts):
        end = cuts[i + 1] if i + 1 < len(cuts) else total
        out.append({
            "source_url": clips[i % len(clips)],
            "start": start,
            "duration": round(end - start, 3),
        })
    return out

What should I check before rendering?

Send the body to the unbilled POST /v1/timeline-1.0/plan first; it reports duration_seconds and segment_count. Set audio.duration_seconds to the track length, and keep video coverage within 0.5 seconds of it. Timeline costs $0.10 per output minute, rounded up.

What if the track has no beat?

Reference ingest marks a silent track with audio.silent: true, and the docs' advice for that case is to plan new music and discard the source audio. When music.present is false, beat_times gives you nothing to cut on. Fall back to a fixed interval you choose, or generate a track with Music 1.0 and run ingest on a clip that carries it.

Should every cut land on a beat?

Not necessarily. Cutting every beat at a fast tempo can leave slots near the 0.2 second minimum, which reads as flicker. The every argument in the function above lets you move to every second, fourth or eighth beat. Preview the result and pick the count by eye; Sume does not choose it.