Check a product-swap video edit for the old product with frames

To check a product-swap edit, pull stills at the same times from the source clip and the edited clip with Sume's video-frames, then compare each pair for the old product, its label or its shape. A prompted edit with gemini-omni-flash-1.1 rewrites the video from a text instruction, so a bottle can survive in a frame you did not look at. Video frames takes up to 24 times per call, so one request per clip gives you a full contact sheet at the cost of its Modal compute.

How does the swap edit work?

The Video Router docs describe gemini-omni-flash-1.1 as one catalog id that Sume routes by the shape of the request. Send video_url and a prompt and it is a video_to_video edit; the prompt describes the change, resolution is optional with a default of 720p, and there is no aspect_ratio or duration on an edit. The docs' own example prompt is Replace the bottle with an apple. Keep everything else the same.

Native audio is always on for this model, and billing is the provider list price times 1.25 per output second by resolution. video_url is the edit source, so it cannot be combined with image_url, end_image_url or reference_*_urls. For a swap that needs your own product photo as a reference, read the Runway comparison first, since that route differs.

What should the check compare?

Choose times that cover the whole clip, with extra points where the product moves, is handled or turns toward the camera. The first and last seconds are where an edit might drift, but I have no measurement of that; treat it as a place to look, not a rule.

How do you build the list of times?

Every time must be at least 0 and less than the clip's duration, or the worker fails the job with frame_time_out_of_range and names the probed duration. The script below makes eight mid-bin times for a clip of a known length and prints the request body to send for both clips. The fps parameter does the same inside the API (mid-bin samples, 0 < fps <= 2, capped at 24 frames), but explicit times let you reuse the exact same list on both files.

import json

def sample_times(duration, n=8):
    n = min(n, 24)
    return [round((i + 0.5) * duration / n, 2) for i in range(n)]

duration = 12.0
for name, url in {
    "source": "https://media.sume.com/artifacts/artf_source/clip.mp4",
    "edit": "https://media.sume.com/artifacts/artf_edit/clip.mp4",
}.items():
    body = {"video_url": url, "at": sample_times(duration), "format": "png"}
    print(name, json.dumps(body))

What do you do with the stills?

Submit one POST /v1/video-frames per clip with an Idempotency-Key. Each submit returns 202; poll GET /v1/video-frames/:id until resource_status is ready, then read frames[{t,url,width,height}]. If the edited clip's duration differs from the source, use the shorter duration to build the list so every time is valid on both.

For a quick overview of one clip without choosing times, video inspect returns eight mid-bin stills by default at a max_edge of 768, plus a probe. Use it for a first look, then switch to video frames for the pairs you compare. If a frame still shows the old product, rerun the edit with a more specific prompt that names where it appears; each rerun is billed again, so check a few key times before you render a long version.