A/B test one first frame across four video models in Python, under $1

The short answer

To A/B one first frame across four video models, loop over four model ids with the same frame_images payload against POST /v1/videos, poll each job and read usage.cost. At 5 seconds, Grok at 720p costs $0.0625, Wan 3.0 at 480p $0.3125, MiniMax H3 at 480p $0.3125 and Gemini Omni Flash at 360p $0.1875, totaling $0.875.

The script

This script submits all four jobs first, so they run at once, then polls each. It needs SUME_API_KEY in the environment and a public image URL. Each job has its own Idempotency-Key, so a rerun of the same script returns the same jobs and does not bill twice.

import os, time, requests
H = {"Authorization": "Bearer " + os.environ["SUME_API_KEY"]}
IMG = "https://example.com/product.png"
RUNS = {"grok-imagine-video-1.5": "720p", "wan-3.0": "480p",
        "minimax-h3": "480p", "gemini-omni-flash-1.1": "360p"}
jobs = {}
for model, res in RUNS.items():
    body = {"model": model, "prompt": "Slow push-in on the product",
            "duration": 5, "resolution": res,
            "frame_images": [{"type": "image_url",
                              "image_url": {"url": IMG},
                              "frame_type": "first_frame"}]}
    r = requests.post("https://api.sume.com/v1/videos", json=body,
                      headers={**H, "Idempotency-Key": "ab-" + model})
    r.raise_for_status()
    jobs[model] = r.json()["polling_url"]
total = 0.0
for model, url in jobs.items():
    while True:
        s = requests.get(url, headers=H).json()
        if s["status"] in ("completed", "failed", "cancelled"):
            break
        time.sleep(10)
    cost = s.get("usage", {}).get("cost", 0)
    total += cost
    print(model, s["status"], cost, s.get("unsigned_urls"))
print("total", round(total, 4))

The cost cap is the table

Each price is five times the per-second rate, and the sum is what you should expect to see.

Keeping it under a dollar

Sume does not cap a batch for you, so the cap is in how you build it: fixed models, fixed duration and fixed resolution. Raising duration to 10 doubles the total to $1.75. The loop above makes four jobs only.

What to compare

Open the four unsigned_urls side by side. Judge motion, fidelity to the first frame and whether sound matters. Grok is silent, while Omni and H3 return audio. The right pick is the cheapest one you would publish, which you can then run at a higher resolution.

If a job fails

A failed status carries an error field, and the loop prints the status so a failure shows up. Download links in unsigned_urls point at GET /v1/videos/{id}/content. See the Videos API docs for the full list of statuses.