How many new Sume jobs can I submit now? Read generation_limits

The number of new jobs you can add right now is the smaller of two values: the free concurrency slots (concurrency_limit minus active and queued generation jobs) and queue_capacity_remaining. The function below returns it from the generation_limits object of a submit response.

def new_work_budget(limits):
    """In-flight budget from the generation_limits object of a submit response."""
    open_slots = max(
        0,
        limits["concurrency_limit"]
        - limits["active_generation_jobs"]
        - limits["queued_generation_jobs"],
    )
    return min(open_slots, limits["queue_capacity_remaining"])


limits = {
    "concurrency_limit": 100,
    "active_generation_jobs": 30,
    "queued_generation_jobs": 10,
    "queue_capacity_remaining": 560,
}
print(new_work_budget(limits))
print(new_work_budget({**limits, "active_generation_jobs": 100, "queued_generation_jobs": 0}))

What it prints

With the example numbers, it prints 60: 100 minus 30 active minus 10 queued is 60 open slots, and the 560 queue capacity left is larger. When all 100 slots are busy it prints 0. The field names come from Generation admission.

Fields

The table explains the fields.

A guide, not a rule

Sume does not reject a valid job when concurrency is full: while queue capacity remains, it accepts the job as queued. The formula is a pacing guide for how many to send in one wave, not a limit that Sume enforces on that number.

Re-read each time

The docs also return a wave_size_hint. Treat it as a suggestion and prefer the live numbers from the latest response, since they change as jobs finish.

If you hit queue_full

If a submit does return 429 with queue_full, stop sending, wait for a job to finish, and resend with the same Idempotency-Key.