100 items, concurrency 16: size your webhook receiver for the burst

A Format bulk run keeps up to 16 child runs in flight, so up to 16 webhooks can land within moments of each other, and your receiver should expect that. A 100-item queue at a concurrency of 16 is 7 waves (6 full waves of 16 plus a last wave of 4), and each finished child starts the next item at once, so completions arrive in clusters rather than a steady trickle. The right design is the one Stripe's webhook guide describes: return a 2xx quickly, then process events asynchronously through a queue at a rate your system supports.

The numbers

The window never goes above concurrency, but your plan's generation concurrency still applies to the children, so the effective burst is often smaller than 16. The receiver should be built for the ceiling. Sume's delivery timeout is 10 seconds per attempt and a slow endpoint spends the retry budget, so a handler that calls a database, a model and an invoicing API inline will cause duplicate deliveries just when the load is highest.

A receiver that holds up

Structure the endpoint in three steps: verify the signature on the raw body, insert a row keyed by request_id (ignore conflicts), return 200. A separate worker reads unprocessed rows with a fixed pool size and fetches each run's result from result_url. If the pool is smaller than the burst, rows wait in your table, not in Sume's retry loop.

Because a run webhook fires once when the run completes or fails, and a canceled run never sends one, the queue itself is the source of truth for completion. Poll GET /v1/format-run-queues/{id} and reconcile counts against the number of webhook rows you hold; a gap means a missing delivery, which you fix with redeliver or a direct read of the child run.

• Store first, process later. Never do model or media work in the request.
• Cap your worker pool so a burst of 16 cannot fan out into 16 downstream calls.
• Reconcile queue counts with received rows after the queue reports completed.
• Remember that completed means every item is terminal, not that every item succeeded.

Why this gets more important

Cheaper clips and bigger batches push more completions into the same minute. The receiver is usually the part nobody sized, because it looked trivial when the batch was ten items.