Python: poll a Sume bulk queue, then list the item indexes to redo

To know which products of a holiday bulk queue need a redo, poll GET /v1/format-run-queues/{queue_id} until status is completed, then collect the index of every item whose status is not completed. completed on a queue only means that every item is terminal, so the redo list comes from items[], not from the queue status. The script below does exactly that with the Python standard library and backs off on 429 and 503.

The script

Set SUME_API_KEY and QUEUE_ID in the environment. The key needs formats:read. Responses wrap the queue in data.

import json, os, time, urllib.error, urllib.request

KEY = os.environ["SUME_API_KEY"]
QUEUE = os.environ["QUEUE_ID"]
URL = f"https://api.sume.com/v1/format-run-queues/{QUEUE}"

def get():
    req = urllib.request.Request(URL, headers={"Authorization": "Bearer " + KEY})
    with urllib.request.urlopen(req, timeout=30) as r:
        return json.load(r)["data"]

delay = 5
while True:
    try:
        q = get()
        if q["status"] == "completed":
            break
    except urllib.error.HTTPError as e:
        if e.code not in (429, 503):
            raise
    time.sleep(delay)
    delay = min(delay * 2, 60)

redo = [i["index"] for i in q["items"] if i["status"] != "completed"]
print(q["counts"])
print("redo indexes:", redo)

Why it is shaped this way

The runs docs recommend doubling the gap up to one minute, and they say a 429 or 503 during polling is temporary because the work continues. The loop therefore sleeps and tries again for those two codes and raises for anything else, such as a 404 for a queue that belongs to another owner.

Steps to use it

Run it after you create the queue, with the id from the 202 response.

• Create the queue and save its frq_ id next to your product list.
• Run the script and wait for the redo list.
• For each redo index, read the child receipt if it has a run_id, since the queue item only carries a short error.
• Fix the cause, then build a new, smaller items list from those rows and create a new queue with a new idempotency key.

What Sume does not do

Sume does not retry a failed item for you, and a replay of the old idempotency key would return the old queue and not a new one. The queue also has no webhook, so this loop, or per-item webhooks, is how your code learns the outcome.

Using the redo list

The list of indexes is only useful if you can turn it back into rows. Because items come back in submitted order, index 41 of a queue is the 42nd entry of the list you sent, so keep that list in a file next to the queue id. A redo queue is then a plain slice of that file, and it gets its own key, such as the original name with a suffix for the second pass.

Decide before you start what counts as a redo. A failed item and a canceled item both appear in the list above, but they mean different things: a failed one usually needs a fix first, while a canceled one may be fine to resubmit unchanged. The child receipt for a failed item has an error field and, on a run that produced files before failing, an artifacts list that may already hold usable media.