GitLab resource_group: one Sume bulk queue at a time

To keep two GitLab pipelines from starting Sume bulk runs at the same time, put the submit job in a resource_group. GitLab then lets only one job per group run at once, and the process mode decides which waiting job goes next. For a batch submit, oldest_first keeps order, while newest_first skips stale batches but requires idempotent jobs.

The GitLab behaviour here is from Resource group, read 2026-10-10. The Sume side is from Bulk runs and Generation admission.

What are the process modes?

GitLab's page lists four modes. unordered is the default and runs whichever job is ready. oldest_first runs the oldest waiting job. newest_first runs the newest waiting job first and the page says it requires jobs to be idempotent. newest_ready_first picks the newest job that is ready. The page says nothing about how a job retry interacts with the group, so do not assume a retry is exempt from the lock; test it in your own project.

Why would two bulk queues collide?

A Sume bulk run takes between 1 and 100 items and a concurrency between 1 and 16 that limits how many items the server runs at once. Two overlapping batches therefore share the same account admission. Sume's admission page gives per-plan processing concurrency of 1, 4, 8 and 20 and queue capacity of 5, 20, 40 and 100, and returns 429 queue_full when both are full. A second 100-item batch started while the first is still draining is the easy way to hit that.

A resource group does not make the queue go away. It makes your pipeline the one place that decides when the next batch may start, so a nightly job and a manual job stop racing.

How do I write the job?

Keep the lock around the submit and the wait, not just the submit. If the job releases the group the moment the request returns, the next batch starts while the first is still in flight, and the group did nothing. Poll the queue until it is completed, then check the counts; the bulk docs say a completed queue means every item is terminal, so a green job needs a check of counts.failed.

Use a fresh Idempotency-Key for each batch. The bulk page states that replaying a bulk key returns 202 with the earlier queue, so a key reused across nights would return last night's queue and submit nothing new.

render_batch:
  stage: deploy
  resource_group: sume-bulk
  script:
    - |
      Q=$(curl -sS -X POST "https://api.sume.com/v1/formats/acme/promo/bulk-runs" \
        -H "Authorization: Bearer $SUME_API_KEY" -H "Content-Type: application/json" \
        -H "Idempotency-Key: batch-$CI_PIPELINE_ID" \
        -d @items.json | jq -r '.data.id')
      echo "queue $Q"
    - ./wait-and-check.sh "$Q"

Which mode should I choose?

If every batch holds different products, use oldest_first so nothing is skipped and pipelines drain in order. If the batch is a full snapshot that replaces the last one, newest_first can save a render, but then the job must be safe to run in any order: stable key per snapshot, no side effect on partial output.

Either way, keep the run budget explicit. Each item sits under its own spend cap. The lock prevents duplicate batches; the cap and the counts check catch the rest.

If a batch is large, split it. A bulk run takes at most 100 items, so a 250 item catalog becomes three batches of 100, 100 and 50, each with its own key such as the pipeline id and a part number. The resource group then runs the three one after another, and the whole sequence stays within one queue's admission.