Bulk run says completed but UGC variants failed: read counts

When a Sume Format bulk queue reports status: completed, it means every item is terminal, not that every item succeeded. Some of your UGC variants can be failed or canceled while the queue still reads completed. Branch on counts.failed and counts.canceled, then list the items whose status is not completed and retry only those.

This follows Sume's Bulk runs page, which says plainly that completed is not "all succeeded". The queue is a server-side list of ordinary Format runs, up to 100 items with a concurrency window of 1 to 16.

What does each queue field tell me?

The docs define each field as follows.

Where do I read why an item failed?

On the child run, not only on the queue. When a child settles as failed, the queue item error is the fixed format_run_failed with the message "The Format run failed." The docs say to read why on the run receipt: GET /v1/format-runs/{run_id}. An item that failed before a child run started keeps its index with run_id: null and an error from the start attempt, for example format_run_failed_to_start.

How do I check a queue in code?

This reads the queue once and prints the items to retry. Call it from a loop with your own delay; the queue has no webhook, so polling status_url is the queue-level way to follow progress.

import json, os, urllib.request

KEY = os.environ.get("SUME_API_KEY", "")
if not KEY:
    raise SystemExit("SUME_API_KEY is empty")

def read(url):
    req = urllib.request.Request(url, headers={"Authorization": f"Bearer {KEY}"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)["data"]

def report(queue_id):
    q = read(f"https://api.sume.com/v1/format-run-queues/{queue_id}")
    if q["status"] != "completed":
        return "still running"
    bad = [i for i in q["items"] if i["status"] != "completed"]
    for item in bad:
        print(item["index"], item["status"], item["run_id"], item["error"])
    return f"{q['counts']['completed']} ok, {len(bad)} to retry"

How do I retry the failed variants?

Create a new bulk request containing only the failed items, with a fresh Idempotency-Key. Replaying the old key with the same payload returns the old queue, and a different payload with the same key is a 409 idempotency_conflict; that behavior has its own post. Each retried item goes through ordinary Format-run admission again, including the wallet and spend caps, so a failure caused by an empty wallet will fail again.