Summarize a video with an API: transcript, stills, then JSON

To summarize a video with the Sume API, run POST /v1/video-inspect on the clip with transcribe: true to get timed stills and a transcript. Then send the stills as input_image attachments and the transcript in input to POST /v1/agent/completions, with an output_schema that shapes the summary and chapters as JSON.

The facts below come from Sume's Video inspect and Agent Completions docs, the structured output rules they share, and the inspect result schema in the Sume API reference, read on 2026-09-27. Each half has its own post: Video inspect API and Agent API with image input and JSON output.

Why stills and a transcript instead of the video file?

Agent Completions attaches images, not video: input_image is the only attachment type today. So this recipe hands the agent what inspect extracts from the clip: stills at known times and the words spoken.

Inspect reads one clip already on your workspace's media.sume.com, such as the output of an earlier Sume job, up to 1,800 seconds long. It has no open-internet fetch: an off-host URL is rejected at admit.

How do I pull the stills and the transcript?

Send POST /v1/video-inspect with the clip's video_url and an Idempotency-Key. The default sync mode waits up to 30 seconds and answers 200 with the finished inspect, or 202 with a job; read it later at GET /v1/video-inspect/:id.

• frames: omit it for 8 mid-bin stills, or pass { "fps": n } (more than 0, up to 2) for a still every 1/n seconds. A call returns at most 24 stills, each { t, url, width, height } on media.sume.com.
• In the current code an fps program keeps its first 24 stills, so size n to the clip: fps: 0.02, as below, spaces 24 stills 50 seconds apart across a 20-minute clip.
• transcribe: true adds a transcript with text and words[]. segmentation: { "mode": "sentence" } also returns gapless sentence segments[], each with index, text, start, end, and duration_seconds.
• A clip with no audio track fails with inspect_source_has_no_audio. A frames: false inspect is enough to read probe.has_audio first, and its probe also reports duration_seconds.

curl -X POST https://api.sume.com/v1/video-inspect \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: webinar-inspect-001" \
  -d '{
    "video_url": "https://media.sume.com/artifacts/artf_demo/webinar.mp4",
    "frames": { "fps": 0.02 },
    "transcribe": true,
    "segmentation": { "mode": "sentence" }
  }'

How do I get the summary back as JSON?

Send the stills and the transcript to POST /v1/agent/completions with a key that has agent_completions:write:

• attachments: each still as an input_image, up to 30 per run. A URL already on media.sume.com is not re-copied. filename is the label the agent sees, so put the still's time in it.
• input: the transcript segments. It is written whole to a file and treated as data, never as instructions.
• output_schema: your summary and chapter shape, inside the strict subset. Every object sets additionalProperties: false, and every property is listed in required.
• generation_spend_cap_usd: required, with no default. Omit it and the request fails with 400 invalid_request.

curl -sS -X POST "https://api.sume.com/v1/agent/completions" \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: webinar-summary-v1" \
  -d '{
    "instruction": "Summarize the video from the attached stills and the transcript in input. List chapters with start times.",
    "input": { "segments": [{ "text": "Welcome to the demo.", "start": 0, "end": 2.4 }] },
    "attachments": [
      { "type": "input_image", "image_url": "https://media.sume.com/artifacts/artf_demo/still-1.jpg", "filename": "t-25s.jpg" }
    ],
    "output_schema": {
      "name": "acme/video-summary/v1",
      "schema": {
        "type": "object", "additionalProperties": false,
        "required": ["summary", "chapters"],
        "properties": {
          "summary": { "type": "string" },
          "chapters": { "type": "array", "items": { "$ref": "#/$defs/chapter" } }
        },
        "$defs": { "chapter": { "type": "object", "additionalProperties": false, "required": ["title", "start_seconds"],
          "properties": { "title": { "type": "string" }, "start_seconds": { "type": "number" } } } }
      }
    },
    "generation_spend_cap_usd": 2
  }'

How do I read the result?

Poll the agent.run receipt or take its agent.run.terminal webhook, as in Agent API with image input and JSON output. A completed run fills output in your schema. If nothing the run made satisfies the schema, output is null, output_error says why, and over the API the run ends failed.

The schema fixes the shape, not the facts: the docs say values other than URLs and durations are the run's own account of its work, not verified measurements. Keep the transcript in input rather than instruction, since input is treated as data; pass customer data to an AI agent safely covers what that boundary does and does not stop.

What are the limits, and what does it cost?

Probe and stills are unbilled; only the transcript reserves, at the STT 1.0 public rate of $0.01 per audio minute on API pricing. The completion's cost is debited_usd_micros from GET /v1/usage?run_id=, the agent's own turns included.