Wallet runs dry mid-bulk: items fail with a null run_id, rest go on

A Sume bulk queue does not pause when the wallet runs dry. The create call returned 202 long ago, and each child run goes through ordinary admission when its turn comes, wallet included. A child that cannot start becomes that item's failed status with run_id: null and an error carrying the create failure, and the queue carries on with the rest. At the end the queue reads completed, which only means every item is terminal. Your job is to separate the items that never started, which cost nothing and can be resubmitted, from the ones that started and failed, which need a look at their receipts.

What the docs say happens

The bulk runs page is explicit: child runs still go through ordinary Format-run admission (wallet, workspace generation concurrency, spend caps), and a child that fails to start is that item failed. A failed item that never started still occupies its index with run_id: null and an error set to the create-run failure. The rest of the queue continues, and a freed slot starts the next queued item immediately.

There is also no cancel or list endpoint for queues, and no queue-level webhook, so you cannot stop the drain once it starts. Plan for the shortfall before you submit. Read GET /v1/balance first, estimate the batch, and consider submitting in slices that your balance clearly covers.

Sort the outcome

The queue object gives you counts (total, queued, running, completed, failed, canceled) and an items array with index, status, run_id and error. The script below splits failed items in two. Items with a null run_id never started a run; those are safe to put into a new queue once credits are available. Items with a run_id did start; read the receipt at GET /v1/format-runs/{run_id} for the real reason, because the queue item only says format_run_failed. The sample queue is inline so it runs as-is.

queue = {"status": "completed", "counts": {"total": 5, "completed": 2, "failed": 3, "canceled": 0},
 "items": [
  {"index": 0, "status": "completed", "run_id": "arun_a", "error": None},
  {"index": 1, "status": "completed", "run_id": "arun_b", "error": None},
  {"index": 2, "status": "failed", "run_id": "arun_c", "error": {"code": "format_run_failed"}},
  {"index": 3, "status": "failed", "run_id": None, "error": {"code": "insufficient_credits"}},
  {"index": 4, "status": "failed", "run_id": None, "error": {"code": "insufficient_credits"}},
]}

never_started = [i["index"] for i in queue["items"] if i["status"] == "failed" and i["run_id"] is None]
started_failed = [i["run_id"] for i in queue["items"] if i["status"] == "failed" and i["run_id"]]

print("resubmit as a new queue after top-up:", never_started)
print("read these receipts first:", started_failed)
assert queue["counts"]["failed"] == len(never_started) + len(started_failed)

Resubmitting without double work

Build the new queue from the never-started indexes only, and give the new create a different Idempotency-Key from the original, because replaying the original key returns the old queue with a 202. Derive it from your batch id plus a retry number so that a crash during the resubmit is itself safe to repeat. For the items that started and failed, decide individually: continue with previous_run_id where the run left clips behind, or retry with a new key where it did not.

A skipped child is recorded on the queue as failed, not skipped, and bulk items always run with on_active_run allow, so you will not see skipped items from your own queue.

Guard the next batch

Check the balance before each slice, size the slice from it with some margin, and set a run spend cap on the items if the Format's own cap is higher than you want. The queue will still happily run a batch your wallet cannot finish; the preflight is what keeps a 100-item launch from turning into 60 videos and 40 failures.

Estimating the shortfall

Before a big batch, read the balance and multiply the per-item estimate by the item count with some margin. If the wallet covers only part of it, submit only that part. Sume does not offer a queue pause, but you can keep each queue small, so the unfunded remainder is a list in your own system rather than failed items on a queue. Top-ups happen in the dashboard, so the handover to a person is a message with the counts, not an API call.