Bulk queue key from a payload hash: same body 202, new body 409

To make a bulk queue safe to retry, derive its Idempotency-Key from a sha256 of the exact { concurrency, items } body. Sending the same body again returns 202 and the queue that already exists, so a lost response costs nothing. Sending a changed body under the same key returns 409 idempotency_conflict, and details.queue_id names the original queue.

The replay rules

The docs give the key a scope of one Format. The header wins if you also send idempotency_key in the body.

Why hash the body

A random key per attempt does not protect you, because a retry after a timeout has a new random key and starts a second queue. A key that is a function of the body is the same on every attempt. If the first call reached the server, the retry gets the old queue back. If it did not, the retry creates the queue.

The side effect is useful: editing one item changes the hash, so you get a new key and a new queue and not a 409. A 409 then only means that something reused a key by hand. Note the cost of the other direction. If you want to run an identical batch again on purpose, the same key returns the old queue, so add a run label to the hash input, such as the date.

The unspent key and the `completed` queue

A replay returns the queue as it is now, with current counts. It does not re-run failed items. After a queue is completed, check counts.failed and counts.canceled; completed means that every item is terminal, not that all succeeded. To retry only the failed rows, build a new body from those rows and let its hash make a new key.

import hashlib, json

def bulk_key(handle: str, slug: str, body: dict, label: str = "") -> str:
    if not handle or not slug:
        raise ValueError("handle and slug are required")
    canon = json.dumps(body, sort_keys=True, separators=(",", ":"))
    digest = hashlib.sha256((f"{handle}/{slug}|{label}|" + canon).encode()).hexdigest()
    return "bulk-" + digest[:32]

body = {"concurrency": 4, "items": [{"instruction": "clip 1"}, {"instruction": "clip 2"}]}
same = {"concurrency": 4, "items": [{"instruction": "clip 1"}, {"instruction": "clip 2"}]}
edit = {"concurrency": 4, "items": [{"instruction": "clip 1"}, {"instruction": "clip 3"}]}
print(bulk_key("demo", "promo", body) == bulk_key("demo", "promo", same))
print(bulk_key("demo", "promo", body) == bulk_key("demo", "promo", edit))

A worked retry

Say a script posts a 40-item queue and the connection drops before the reply arrives. With a random key, the script retries, and the server may now hold two queues of 40 items, 80 runs. With the hashed key, the retry sends the same key and the same body, and the server answers 202 with the queue it already made. The script reads id and status_url and carries on to polling.

Now say that the script is changed to drop item 17 and is run again against the same key by hand. The hash changes, so the key changes, and a new queue is made. If a person reuses the old key with the new body, the answer is 409 and details.queue_id points to the first queue. That is the signal to look, not to retry.

What the key does not cover

The docs say the queue has no webhook, no list-queues call and no cancel-queue call. Keep the returned id (frq_...) in your own store, since you cannot list queues to find it later. If you lose it, replay the same body with the same key to get the queue back as a 202.