Celery task for a Sume video: task id as the Idempotency-Key

Use the Celery task id as the Idempotency-Key on POST /v1/videos, and let the task call self.retry(countdown=30) while the job is pending or in_progress. A retry runs the function again, which sends the same POST with the same key. Sume answers with the original job, not a second one, so the retry is a poll.

Why this shape works

Sume documents the replay rule on this route: you send Idempotency-Key to make retries safe, and a replay returns the original job. A different body under the same key is rejected with 409 idempotency_conflict. Celery keeps the task id the same across retries of one task, so the key is stable without any storage on your side.

That removes the usual two-step design (a submit task that saves a job id, then a poll task that reads it). One task, one key, one place to log. The cost is one extra POST per poll, and that POST is a replay and does not reserve funds again.

The task

The code reads the key from self.request.id. It returns the download URL when the status is completed, raises on failed or cancelled, and retries otherwise. Set max_retries so the task gives up after a bound you choose; 60 retries at 30 seconds is half an hour.

import os
import requests
from celery import Celery

app = Celery("video", broker=os.environ["CELERY_BROKER_URL"])
BASE = "https://api.sume.com/v1/videos"

@app.task(bind=True, max_retries=60)
def render(self, payload: dict) -> str:
    h = {
        "Authorization": f"Bearer {os.environ['SUME_API_KEY']}",
        "Idempotency-Key": self.request.id,
    }
    r = requests.post(BASE, json=payload, headers=h, timeout=30)
    r.raise_for_status()
    job = requests.get(r.json()["polling_url"], headers=h, timeout=30).json()
    if job["status"] == "completed":
        return job["unsigned_urls"][0]
    if job["status"] in ("failed", "cancelled"):
        raise RuntimeError(job.get("error") or job["status"])
    raise self.retry(countdown=30)

Failure paths

If the worker dies after the POST but before the poll, Celery redelivers the task with the same id, and the replay returns the original job. That is the case the key exists for. If the job ends as failed, the task raises, and Celery marks it failed; whether to resubmit is a business choice, and it needs a new task id, because a replay of the old key would return the failed job.

queue_full on the POST is a 429 that means the workspace has no room for another paid job. Use retry-after, not a fixed delay.

Things to decide

The docs do not state how long Sume keeps an idempotency key, so do not design a loop that depends on a replay days later. Keep the retry bound short and store the final URL when the task returns.

A 429 on the POST should also retry: use the retry-after header when present. See reading the 429, and 409 idempotency_conflict if a payload changes between retries.

• Pass the payload as an argument so every retry sends the same body.
• Do not build the payload from the clock or a random value inside the task.
• Prefer callback_url plus a verifier if you run many tasks, so most polls never happen.