Grafana alert webhook with HMAC to a Sume run: incident explainer

A Grafana alert can start a Sume Format run through the webhook contact point: point it at a small receiver, verify the HMAC Grafana can add, and call POST /v1/formats/{handle}/{slug}/runs once per alert you care about. Grafana's own docs do not describe retries or a timeout for this notifier, so design for a missed call.

A natural use is an incident explainer: a short clip or image that turns a firing alert and its labels into something an on-call person can scan.

What Grafana documents

Grafana's webhook notifier page says the contact point sends POST or PUT, supports basic auth or an Authorization header with a configurable scheme, and can add an HMAC-SHA256 signature computed over timestamp:body when you configure a timestamp header, and over the body alone when you do not (read 2026-10-10). It has a max-alerts setting, and the payload carries fields such as receiver, status and alerts.

Notice the signed string joins timestamp and body with a colon, while Sume joins them with a period. Reusing a Sume verifier for Grafana, or the reverse, fails without any clear error.

One call, many alerts

Grafana can put several alerts in one payload, so loop over the alerts array. Key each run on the alert fingerprint if your payload carries one, plus the startsAt time, so a re-sent firing alert maps to the same Sume key, while the same alert firing again tomorrow gets a new one. If you need more than a handful, use the bulk endpoint, which accepts 1 to 100 items and returns a queue to poll.

Skip resolved alerts unless you want a second Format for the all-clear. Keep the two paths apart by branching on status before any Sume call.

Cost and trust

An alert storm can look like a spend storm. Set generation_spend_cap_usd on every create, and cap the number of runs per receiver minute in your own code. Treat labels and annotations as untrusted: put them in input, which Sume treats as caller data rather than instructions, and never let them choose the Format slug.

Return a 2xx to Grafana once the alert is stored and the create call has been accepted. If the Sume create fails with 429 rate_limited, wait for retry-after; for 402 insufficient_credits, alert a person instead of retrying.

Receive the result

Use communication.webhook_url for the finished media and verify format.run.terminal with the secret from GET /v1/webhooks/signing-secret. Branch on outcome. Canceled and skipped runs never deliver an event, and the run record expires 90 minutes after creation or earlier if silent, so poll status_url for any run without an event.

Verify the Grafana HMAC

Grafana's page names the signed string as timestamp:body when a timestamp header is configured and the body alone otherwise, with X-Grafana-Alerting-Signature as the default signature header; set the timestamp header so replays can be rejected, and read the exact names on that page rather than guessing. Compute HMAC-SHA256 over the raw bytes you received, compare in constant time, and reject a request whose timestamp is outside a window you choose. Refuse to run at all when the shared secret is empty.

Keep this check in its own function and name it for Grafana. The Sume verifier on the return path joins timestamp and body with a period and reads x-sume-webhook-signature. Giving them different names, and different tests, prevents the quiet failure where a copied helper rejects every genuine call.