Black Friday bulk queue: reconcile webhooks received against counts

If you armed a webhook on every item of a Sume bulk queue, the number of deliveries you receive can legitimately be lower than the item count. The queue has no webhook, and a canceled or skipped run sends none, so reconcile against the queue instead of waiting for N events.

Where the two numbers come from

• Webhooks: each child registers its own communication.webhook_url and gets one signed format.run.terminal POST when it completes or fails.
• Queue: GET /v1/format-run-queues/{id} returns counts with total, completed, failed and canceled.
• Gap: canceled runs, and deliveries that are still retrying or exhausted.

A reconcile loop

Keep a set of request_id values from verified deliveries. In the webhook body request_id equals the run id, and the queue item holds the same run_id. Once the queue is completed, subtract.

def missing(queue, seen_request_ids):
    out = []
    for it in queue["items"]:
        if it["status"] == "canceled":
            continue  # canceled runs send no webhook
        if it["run_id"] and it["run_id"] not in seen_request_ids:
            out.append(it["run_id"])
    return out

What to do with the leftovers

Read each missing run with GET /v1/format-runs/{run_id} and look at webhook_delivery.status: pending and retrying mean Sume is still trying, exhausted means all ten attempts failed. For those, POST /v1/format-runs/{run_id}/webhook/redeliver replays a real run to your URL. Items with run_id: null failed to start; the queue item error says why.