Port a reference-image video prompt to Sume with <IMAGE_REF_0> tags

On Sume, use image_url when the picture is the opening frame, and use reference_image_urls with <IMAGE_REF_0>, <IMAGE_REF_1> tags in the prompt when pictures are subjects or props that must appear somewhere in the clip. Both run on gemini-omni-flash-1.1 through the Video Router.

OpenAI's deprecations page lists the Sora video models and Videos API as removed on 2026-09-24. If your prompts were written around an input image, the port is not only a model id change: you now choose between pinning the first frame and naming references.

Two modes, two intents

Sume's Video Router doc lists four capabilities for Omni. Image to video takes image_url, with an optional end_image_url. Reference to video takes up to 10 reference_image_urls and up to 3 reference_video_urls of at most 3 seconds each, and the prompt addresses them as <IMAGE_REF_0> and <VIDEO_REF_0>, counted from zero in list order.

The two are not interchangeable. A first frame fixes how the clip starts and the model animates from there. References describe who or what should appear, and the prompt tells the model where. A product shot that must begin on the exact packshot wants image_url. A scene where a character picks up an object wants references.

A tagged reference request

The request below sends two reference images and names both in the prompt. It prints the raw response so you can see the job envelope before you write a parser. Tags are zero-based and follow the order of the list, so swapping the two URLs swaps what the tags mean.

import json, os, urllib.request
body = {
    "model": "gemini-omni-flash-1.1",
    "prompt": "<IMAGE_REF_0> walks into frame and sets down the bag from <IMAGE_REF_1>, soft window light",
    "reference_image_urls": ["https://example.com/hero.png", "https://example.com/bag.png"],
    "resolution": "720p",
    "duration": 6,
    "aspect_ratio": "16:9",
    "mode": "async",
}
req = urllib.request.Request(
    "https://api.sume.com/v1/video-router/generate",
    data=json.dumps(body).encode(),
    headers={"Authorization": "Bearer " + os.environ["SUME_API_KEY"],
             "Content-Type": "application/json",
             "Idempotency-Key": "ref-demo-001"})
try:
    with urllib.request.urlopen(req, timeout=60) as r:
        print(r.status, r.read().decode())
except urllib.error.HTTPError as e:
    print(e.code, e.read().decode())

Rules that cause 400s

You cannot combine video_url (the edit source) with image_url, end_image_url or any reference list. Native audio is always on, so do not send generate_audio false. Keep the clip between 3 and 10 seconds, and use 16:9 or 9:16. Those are documented limits, and the API rejects values outside them rather than rounding.

If a prompt mentions a tag with no matching list entry, that is your bug, not the model's. Count the list before you send, or generate the prompt from the same array that builds reference_image_urls.

• Use public HTTPS URLs for every image.
• Write one tag per subject. Do not reuse IMAGE_REF_0 for two people.
• Keep each prompt about what happens, not about the picture's contents; the model already sees it.

What to test after the port

Run each old prompt twice, once as a first frame and once with tags, and keep whichever one matches what the shot was for. The comparison costs two short renders at the Omni 720p rate, which the repo docs derive from a $0.10 list price times 1.25, or $0.125 a second.