Bash progress line for a Sume bulk queue from counts with jq

A bulk queue has no webhook, so the way to watch one is to poll GET /v1/format-run-queues/{id} and read counts. Done items are completed + failed + canceled, out of total. The script below prints one line per poll, backs off from 10 seconds to a minute, treats 429 and 503 as transient, and ends with exit status 1 if any item failed or was canceled, so a shell can branch on it.

It needs curl and jq. The key must have formats:read; the queue id is in the receipt that the create call returned.

The script

Set SUME_API_KEY and QUEUE_ID. SUME_BASE is optional and defaults to the production API.

#!/usr/bin/env bash
set -euo pipefail
: "${SUME_API_KEY:?set SUME_API_KEY}" "${QUEUE_ID:?set QUEUE_ID}"
BASE="${SUME_BASE:-https://api.sume.com/v1}"
gap=10
while :; do
  code=$(curl -sS -o q.json -w '%{http_code}' -H "Authorization: Bearer $SUME_API_KEY" \
    "$BASE/format-run-queues/$QUEUE_ID")
  if [ "$code" = 200 ]; then
    jq -r '.data | "\(.status): \(.counts.completed + .counts.failed + .counts.canceled)/\(.counts.total) done, \(.counts.running) running, \(.counts.failed) failed"' q.json
    [ "$(jq -r .data.status q.json)" = completed ] && break
  elif [ "$code" != 429 ] && [ "$code" != 503 ]; then
    echo "poll failed: HTTP $code" >&2; exit 2
  fi
  sleep "$gap"; gap=$(( gap * 2 > 60 ? 60 : gap * 2 ))
done
[ "$(jq '.data.counts.failed + .data.counts.canceled' q.json)" -eq 0 ]

Details that matter

The HTTP status is read separately with -w '%{http_code}' and the body goes to a file. That keeps a 429 with an empty or odd body from reaching jq, and it lets the script tell a transient status from a final one. A 401, 403 or 404 is final: the key, its scope or the id is wrong and waiting will not fix it, so the script exits with status 2.

The loop stops when the queue's own status is completed, which means every item is terminal. It does not stop on a count that looks like 100 percent, because running can briefly include a child the API has claimed but not yet given a run id. The last line, failed + canceled == 0, becomes the exit status through [ ... ].

Reads cost less than writes. A plan's read allowance is 40 times its write allowance, so a poll every few seconds is fine on any plan, but the doubling gap keeps it cheap for a batch that takes an hour.

After it finishes

Exit 1 means look at q.json, which the script leaves behind: the items with failed or canceled have an error and, if the child started, a run_id whose receipt says why. Retry only those rows, in a new queue under a new idempotency key.

If you start this script in a second terminal while another process is also polling, nothing breaks: polling does not change the queue. Cancel is per child, never for the queue.

To run it from cron or a CI job, give the job a hard timeout longer than the batch should take and treat a timeout as a failure to investigate, not as a reason to start a second queue. Starting the same chunk again under a new key renders every row twice, and the idempotency key is what protects you only when you reuse it.