Log Sume ratelimit-remaining on every Python urllib response

The Sume docs say not to count requests yourself: read ratelimit-remaining and, on a 429, wait for retry-after. Every response carries ratelimit-limit, ratelimit-remaining and ratelimit-reset. With only the Python standard library, a small BaseHandler is enough to log them for every call.

The handler runs before urllib turns an error status into HTTPError, so a 429 is logged too.

The handler

import logging, urllib.request

log = logging.getLogger("sume.rate")

class RateLogger(urllib.request.BaseHandler):
    def http_response(self, request, response):
        left = response.headers.get("ratelimit-remaining")
        limit = response.headers.get("ratelimit-limit")
        if left is not None and limit:
            level = logging.WARNING if int(left) < int(limit) * 0.1 else logging.DEBUG
            log.log(level, "%s %s remaining %s/%s reset %ss", request.get_method(),
                    request.selector, left, limit, response.headers.get("ratelimit-reset"))
        return response
    https_response = http_response

opener = urllib.request.build_opener(RateLogger())
# opener.open(req) now logs the budget on every response; no counting on your side.

What it logs

Normal responses log at DEBUG. When the remaining count drops under 10 percent of the limit it logs a WARNING such as GET /v1/jobs/job_1/status remaining 0/120 reset 9s. The header describes the bucket the request spent from, so a submit shows the write bucket and a status poll shows the read bucket.

Notes

• ratelimit-limit on the response is the authority for the deployment you call; the plan table is only the shipped default.
• Logging does not replace backoff: on 429, honor retry-after and read error.details.scope.
• Keep the API key out of these lines. The handler logs method, path and numbers only.