Change a video's background with a prompt: Gemini Omni on Sume

To change the background of a video with a prompt, send the clip to Gemini Omni Flash 1.1 as a video_url and describe the new setting and what to keep. On Sume that is one call to the Video Router; there is no mask, rotoscope or green screen step.

Google's I/O 2026 roundup lists cinematic zooms and background changes via prompts as things the Gemini app does with Gemini Omni (read 2026-10-02). The Gemini app needs a Google AI Plus, Pro or Ultra plan; the API route below needs a Sume key.

What does the request look like?

The Video Router routes by request shape. A video_url selects edit mode. The prompt describes the change, resolution is optional (default 720p), and aspect_ratio and duration are not accepted because the output follows the source. Native audio stays on.

curl -X POST https://api.sume.com/v1/video-router/generate \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: new-background-001" \
  -d '{
    "model": "gemini-omni-flash-1.1",
    "prompt": "Replace the office background with a sunlit greenhouse. Keep the speaker, their clothes, their voice and the camera framing exactly the same.",
    "video_url": "https://example.com/talking-head.mp4",
    "resolution": "1080p",
    "mode": "async"
  }'

How do I keep the subject stable?

Name what must not change, in the same sentence as the change. "Keep the speaker, their clothes, their voice and the framing" does more than a long scene description. Ask for one background per request: a second background in the same prompt tends to blend them.

Match the light. If the subject is lit warm from the left, say the new setting is lit from the left too. A mismatch is the most common reason a swap looks pasted on.

What are the limits?

Edit mode rewrites the whole clip rather than compositing a cut-out, so small changes to the subject can appear. Sume's docs do not publish a length cap for the edit source, so try a short section first.

video_url cannot be combined with reference images in the same request. If you need a specific background from a photo, generate a new clip with the photo as <IMAGE_REF_0> instead, or use a text description of the place.

For long recordings, cut a section with Video trim first. It takes a Sume-hosted clip and returns a new MP4 for $0.02 per job.

How do I get the file back?

The submit returns an async job. Poll GET /v1/jobs/:id/status with backoff and read GET /v1/jobs/:id/result when it completes, as described in Jobs and results. Store the Sume media URL from the result; the docs say to keep the Sume URL, not a provider link.

Billing is provider list times 1.25 per output second by resolution, so a 1080p edit costs more than a 720p one. Check the 720p result first, then re-run at 1080p.