Redis lease so one worker polls each Sume job

Short answer

Take a short lease per job with SET key token NX EX, and release it only if the stored value still equals your token. The Redis SET docs describe this recipe for simple locks: acquire when the reply is OK, retry later on nil, set a non-guessable random token, and remove the key with a script that deletes only on a matching value. For Sume polling the lease is a cost saver, not a safety requirement, because status reads are idempotent.

Why bother if reads are safe

Reading a Sume job twice cannot break anything. Each key has a read budget, forty times its write budget, so duplicate pollers are rarely a rate-limit problem either. The reason for a lease is tidiness at scale: many workers, one job id each, and you would rather have exactly one of them waking up every few seconds for it. A lease also keeps logs and callbacks from firing twice for the same terminal transition.

Acquire, renew, release

Choose the lease time a little longer than one poll interval plus the request timeout, and renew it each loop with a script that extends the expiry only while the stored value still equals your token, the same check as the release script. If the worker dies, the lease expires and another worker picks the job up. The release script exists because of exactly that case: if your lease expired while you were slow and a second worker took it, a plain delete would remove their lease.

Sume's status reply may carry next_poll_after_seconds, so sleep for that long between reads and extend the lease by that amount plus a margin.

import os
import uuid

import redis

r = redis.Redis.from_url(os.environ.get("REDIS_URL", "redis://localhost:6379"))
UNLOCK = """
if redis.call('get', KEYS[1]) == ARGV[1] then
  return redis.call('del', KEYS[1])
else
  return 0
end
"""

key, token = "sume:poll:job_123", uuid.uuid4().hex
if r.set(key, token, nx=True, ex=60):
    print("lease held")
    print("released:", r.eval(UNLOCK, 1, key, token))
else:
    print("another worker polls this job")

What the docs caution about

The Redis page marks the single-instance SET NX EX pattern as discouraged in favor of the Redlock algorithm when you need stronger guarantees and fault tolerance. That is the right lens: if a rare double-poll would cost you something, design the consumer to be idempotent and treat the lease as an optimization. The terminal transition should be handled once through a state change in your own database, not by trusting the lease alone.

Also keep the lease out of the money path. Never use it to decide whether to submit a paid job. Use an Idempotency-Key for that, so a replay returns the original job instead of billing twice.

Stopping cleanly

When the status reply reports terminal true, release the lease, record the outcome and stop. For a completed job, fetch the result when result_ready is true; for failed or canceled, read the error from the job record. If the lease holder crashes mid-poll, the expiry hands the job to a new poller without any manual cleanup, which is the reason to prefer an expiry over a lock with no timeout.