Test a webhook endpoint before go-live: a Sume CI gate (Python)

To test a webhook endpoint before going live, make the sender fire a real, signed request at your deployed URL and fail the deploy if it does not answer 2xx. Sume has an endpoint for exactly this: POST /v1/webhooks/test-deliveries with {"webhook_url": "https://..."} sends a dummy webhook.test event, signed like a real delivery, and returns the HTTP status your receiver gave back. It needs an API key with account:write.

That makes it a one-file CI gate. The point of running it from CI rather than from the dashboard button is that the test runs against the URL you just deployed, with the secret that deploy actually holds, every time. Coding agents now scaffold and ship receivers on their own; the Neuron digest of October 1 reported Claude Code mods and GitHub Copilot computer use that same day (reported, not a vendor spec). A gate that proves the receiver answers is what keeps an agent-written handler honest.

What does a Sume test delivery send and return?

The request body has one field, webhook_url, which must be a public HTTPS URL. Localhost, private-network and non-HTTPS URLs are rejected, so a gate cannot point at http://localhost:3000; it needs a deployed or tunnelled URL. The delivery is a dummy body with event: "webhook.test", no job_id and no run id, and it is never appended to your Requests list. It is also not a replay of a real call: for that, use the job or Format-run redeliver endpoints.

The HTTP response describes the POST Sume made to you, so read it instead of assuming a 200 from Sume means your endpoint answered. Both status_code and error are nullable in the schema.

What does the CI script look like?

This script takes the receiver URL as its argument and exits non-zero unless the receiver answered 2xx. It uses requests and the key in SUME_API_KEY.

import os
import sys
import requests

API = "https://api.sume.com"
H = {"Authorization": f"Bearer {os.environ['SUME_API_KEY']}"}
url = sys.argv[1]  # your receiver's public HTTPS URL

r = requests.post(
    f"{API}/v1/webhooks/test-deliveries",
    headers=H, json={"webhook_url": url}, timeout=30,
)
r.raise_for_status()
sent = r.json()["data"]

code = sent["status_code"]
if code is None or not 200 <= code < 300:
    sys.exit(f"receiver not ready: status={code} error={sent['error']}")
print("ok", code, "secret fingerprint", sent["signing_secret_fingerprint"])

What must the receiver do with a webhook.test event?

Route on event, and answer an unknown or test event with a fast 2xx. The docs say a verifier covers both job and run webhooks and that you should treat unrecognized events as 204 so a newly added event type never turns into a 500 and a retry storm. A handler that does event["job_id"] without checking will throw on the test body, which has neither job_id nor run_id, and your gate will correctly go red.

Verify the signature before you look at the event. The test delivery uses the same sume-v1 scheme, the same workspace secret, and the same headers (x-sume-webhook-timestamp, x-sume-webhook-signature, x-sume-webhook-secret-fingerprint) as a real one, so a receiver that passes the gate is exercising the real verification path, secret loading included.

• Send the gate only after the new version is serving traffic, not before the deploy finishes.
• Keep the key that runs it in CI secrets: account:write can also rotate the signing secret.
• A failure with a null status_code usually means the URL never answered; read error.
• Do not retry in a tight loop on failure. One red run is the signal; read the error, fix the handler, redeploy.

What a green gate does not prove

A 2xx on webhook.test proves the URL, the TLS, the secret and the handler's fast path. It does not prove your job logic, because the dummy body carries no job. Pair it with a real mode: "webhook" submit in staging, and keep status polling available for deliveries that never arrive, as the generation-job webhook docs recommend. Job webhooks retry up to 10 attempts with a 10-second timeout each, so a slow handler that passes a quiet test can still fail under load; return 2xx after durably recording the event, then do the work.

The same call works from a shell when you only want a one-off check: curl -X POST https://api.sume.com/v1/webhooks/test-deliveries -H "Authorization: Bearer $SUME_API_KEY" -H "Content-Type: application/json" -d '{"webhook_url":"https://hooks.example.com/sume"}'. Put the Python version in CI because it turns the answer into an exit code.

If the gate is red after a secret rotation, remember the 24-hour overlap: during that window deliveries carry two comma-separated signatures, and a verifier that compares the header for equality fails. See Verifying webhooks.