Go client for a Sume bulk queue: transient polls and exit status

A Go program can create a Sume bulk queue and wait for it using only net/http and encoding/json. The shape is: read items.json, POST it to /v1/formats/{handle}/{slug}/bulk-runs with an Idempotency-Key, require 202, then poll status_url until status is completed, and exit non-zero when counts.failed or counts.canceled is not zero.

Go decodes JSON keys case-insensitively onto exported fields, so counts.total fills Total without tags. The only tag needed is on StatusURL, whose wire name is status_url. The code needs Go 1.21 or later for the built-in min.

The program

Set SUME_BASE to https://api.sume.com/v1, along with SUME_API_KEY, SUME_FORMAT (handle/slug) and BATCH_KEY. It is written dense to fit the page; run gofmt on it.

package main
import ("bytes"; "encoding/json"; "fmt"; "net/http"; "os"; "time")
type queue struct { ID, Status string; StatusURL string `json:"status_url"`; Counts struct{ Total, Failed, Canceled int } }
func call(method, url, idem string, body []byte) (queue, int) {
	req, _ := http.NewRequest(method, url, bytes.NewReader(body))
	req.Header.Set("Authorization", "Bearer "+os.Getenv("SUME_API_KEY"))
	req.Header.Set("Content-Type", "application/json")
	if idem != "" { req.Header.Set("Idempotency-Key", idem) }
	res, err := (&http.Client{Timeout: 30 * time.Second}).Do(req)
	if err != nil { return queue{}, 503 } // transport error: treat as transient
	defer res.Body.Close()
	var env struct{ Data queue }
	json.NewDecoder(res.Body).Decode(&env)
	return env.Data, res.StatusCode
}
func main() {
	items, _ := os.ReadFile("items.json")
	body := fmt.Appendf(nil, `{"concurrency":4,"items":%s}`, items)
	q, code := call("POST", os.Getenv("SUME_BASE")+"/formats/"+os.Getenv("SUME_FORMAT")+"/bulk-runs", os.Getenv("BATCH_KEY"), body)
	if code != 202 { fmt.Println("create failed:", code); os.Exit(2) }
	fmt.Println("queue", q.ID)
	for gap := 15 * time.Second; q.Status != "completed"; gap = min(gap*2, time.Minute) {
		time.Sleep(gap)
		next, code := call("GET", q.StatusURL, "", nil)
		if code == 200 { q = next } else if code != 429 && code != 503 { fmt.Println("poll failed:", code); os.Exit(2) }
	}
	fmt.Println(q.Counts)
	os.Exit(min(q.Counts.Failed+q.Counts.Canceled, 1))
}

The decisions in it

A transport error returns 503 from call, so a dropped connection while polling is handled like the documented transient 503. On the create call that turns into create failed: 503, which is correct to retry, but only with the same BATCH_KEY: if the request reached Sume before the connection broke, the replay returns the queue that already exists with 202 and starts nothing new.

Only 429 and 503 are retried during polling. Everything else exits with code 2, because a 401, 403 or 404 will not fix itself, and a 404 format_run_queue_not_found also covers a queue owned by someone else. The gap doubles up to a minute. The poll counts against the read budget, which is forty times the write budget and does not affect your creates.

What it leaves out

It prints Counts only. For a per-item report, add Items []struct{ Index int; Status, RunID string } to the queue type with a json:"run_id" tag on RunID, and print the failed ones. The reason a child failed is on the run receipt, not on the queue item.

The body is built with concurrency fixed at 4, written into the format string in main. That is the one number that changes how fast the batch drains: the window of children running at once, from 1 to 16. Change it there. It is separate from the per-minute request limit, and one bulk create counts as a single write however many items it carries.

The program prints the counts as Go's default struct form, total then failed then canceled. Swap in a %+v verb if you want field names in a CI log. Build it once with go build and ship the binary to the runner, so the batch step has no toolchain to install.

It also does not set generation_spend_cap_usd. Put that inside each item in items.json: a queue has no cap of its own, so the worst case is the sum of the item caps, and an item without one inherits the Format's cap.