How to compare AI video models fairly: one prompt, three models, 480p

A fair model comparison on Sume holds everything fixed except the model: the same prompt, 480p, 5 seconds and the same aspect ratio. Seedance 2.0 Mini, Wan 3.0 and MiniMax H3 all accept that envelope, so one script can submit three jobs and give you three clips to review without knowing which is which.

Fix the variables

Choose values every candidate accepts: 5 seconds is inside all three models' ranges, 480p is a rung on all three, and 16:9 is accepted by all three. There is no seed on any of these models, so run each prompt a few times if the decision matters, and judge more than one take per model.

The harness

It submits one job per model with its own Idempotency-Key and prints the job ids. Poll each with the usual status endpoint afterwards.

import asyncio
import os

import httpx

MODELS = ["seedance-2-mini", "wan-3.0", "minimax-h3"]
PROMPT = "A barista pours oat milk into a latte, slow push-in, warm light"


async def submit(client: httpx.AsyncClient, model: str) -> tuple[str, str]:
    body = {"model": model, "prompt": PROMPT, "resolution": "480p",
            "duration": 5, "aspect_ratio": "16:9"}
    headers = {"Idempotency-Key": f"compare-001-{model}"}
    response = await client.post("/v1/videos", json=body, headers=headers)
    response.raise_for_status()
    return model, response.json()["id"]


async def main() -> None:
    key = os.environ["SUME_API_KEY"]
    headers = {"Authorization": f"Bearer {key}"}
    async with httpx.AsyncClient(base_url="https://api.sume.com", headers=headers) as client:
        for model, job_id in await asyncio.gather(*(submit(client, m) for m in MODELS)):
            print(model, job_id)


asyncio.run(main())

Judge it blind

Rename the downloaded files to A, B and C and have someone who did not run the job rank them on the things you care about: product fidelity, motion, text, audio. Then rerun the winner at the resolution you will ship. At the catalog's dated Wan list rate of $0.05 per second, a 5-second 480p test is $0.25 at list.