Sume job failed worker_job_stalled or worker_lease_expired: what now

If a Sume job ends as failed with the code worker_job_stalled or worker_lease_expired, no worker is still running it. The job is terminal, a job.failed event was written for it, and its usage reservation is refunded like any other failure. Neither code is retryable on the same job, so submit the work again as a new job and stop polling the old one.

Both codes come from the same sweep in the job store, and they describe two different ways a job can get stranded in processing. Knowing which one you hold tells you whether anything of yours is at fault. Usually nothing is.

What is the difference between the two codes?

A processing job holds a lease while a worker runs it. If the lease runs out and the job has used all of its attempts, the sweep fails it with worker_lease_expired and the message "Worker lease expired after maximum attempts." That is a job a worker abandoned mid-run.

worker_job_stalled is the other case. The job has no lease at all, it has no attempts left, and its available_at time is more than 5 minutes in the past. Nobody abandoned it. It was parked, waiting for work that could never resume. The message reads "Job stalled in processing with no lease and no attempts remaining." The 5-minute grace matches the default lease, because each provider callback pushes available_at forward and the sweep must not remove a job that is still waiting on an in-flight round trip.

Does a stranded job always fail?

No. The same sweep has a second half. A lease-less processing job that still has attempts left, past the same 5-minute grace, goes back to queued with its attempt count raised by one. It runs again without any action from you. Only a job that has spent every attempt is failed.

Am I charged, and will I be told?

The sweep writes a job.failed terminal event in the same statement that fails the job. The Sume operations notes say failed events refund their reservations. If you submitted with mode webhook, you get a job.failed delivery with the public error, and the webhooks page lists job.failed as one of exactly three terminal events along with job.completed and job.canceled.

Webhooks are not the only path. The docs say a delivery problem never changes the job's real state, so a status read shows the same failed job. Poll GET /v1/jobs/{id}/status and stop on completed, failed or canceled.

How should my client handle it?

A non-retryable internal failure carries the next action contact_support in the public error, so a generic retry loop will not retry it by itself. Decide in your own code. The one rule from the jobs docs still holds: do not resend the original paid request only because a local process timed out. Here the job really is terminal, so a fresh submit is correct, and it needs a new Idempotency-Key. Reusing the old key returns the old failed job.

import os, requests

BASE = "https://api.sume.com/v1"
H = {"Authorization": "Bearer " + os.environ["SUME_API_KEY"]}

def settled(job_id):
    r = requests.get(f"{BASE}/jobs/{job_id}/status", headers=H, timeout=30)
    r.raise_for_status()
    body = r.json()
    if body.get("status") == "failed":
        err = body.get("error") or {}
        print("failed:", err.get("public_reason") or err.get("code"))
        return "resubmit-with-new-key"
    return body.get("status")

If the same job type keeps ending this way, send the request id from the error body to Sume support, as the errors page asks. Leave out API keys and signed URLs.