Agent API with image input and JSON output: Sume Agent Completions

To send images to the Sume agent API and get JSON back, call POST /v1/agent/completions with input_image content parts (or top-level attachments) that carry a public HTTPS image_url, add an output_schema, and read the run's output once it completes. Attachments and output_schema compose.

The details come from Sume's Agent Completions docs page and the attachment rules it points to, read on 2026-09-26. For the endpoint in general, see Run the Sume video agent from your backend.

How do I put an image in a message?

In messages[], content accepts a string or an OpenAI-style array of { "type": "text" } parts; input_text is an alias for text, and input_image parts carry images. Turns are joined, in order, into one prompt. An image-only turn is fine: omit the text part and the agent is told to use the attached files. assistant turns are rejected, because every completion runs in a fresh thread.

The same item shape also works at the top level as attachments, and the two sources are merged into one list. An item takes image_url or asset_id, plus an optional filename label. The Assets routes behind asset_id are hidden from the public OpenAPI document, so this example uses image_url:

curl -sS -X POST "https://api.sume.com/v1/agent/completions" \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: sku-4411-caption-v1" \
  -d '{
    "messages": [{
      "role": "user",
      "content": [
        { "type": "input_text", "text": "Describe this product shot." },
        { "type": "input_image", "image_url": "https://example.com/shot.jpg" }
      ]
    }],
    "output_schema": {
      "name": "caption",
      "schema": {
        "type": "object",
        "properties": { "caption": { "type": "string" }, "alt_text": { "type": ["string", "null"] } },
        "required": ["caption", "alt_text"],
        "additionalProperties": false
      }
    },
    "generation_spend_cap_usd": 2
  }'

How do I get JSON back instead of text?

Bind output_schema and the run's output follows your schema; the images still reach the agent, and output is parsed against your schema after the run completes. It is the same contract as scheduled runs, which share the structured-output rules of Format runs. Without a schema, output uses the default sume/action-run-output/v1 shape, with the agent's closing text in output.text.

The schema must fit the strict subset, or it is rejected before any run starts:

• The root is an object, and every object sets "additionalProperties": false.
• Every property appears in required; make a field optional with a nullable union such as "type": ["string", "null"].
• At most 10 nesting levels, 5000 properties, and 1000 enum values.
• $ref may only point at #/$defs/<name> or the registered SumeMediaFile#.

How do I read the result?

The create call returns 202 with an agent.run receipt. Poll GET /v1/agent-runs/{run_id} or its status_url until next_action stops being poll_status, or send communication.webhook_url and receive one signed agent.run.terminal POST. A completed run fills output, artifacts, and usage. If nothing the run made satisfies your schema, output is null, output_error says why, and over the API the run ends failed. Schema design is covered in Sume Format structured output.

curl -sS "https://api.sume.com/v1/agent-runs/$RUN_ID" \
  -H "Authorization: Bearer $SUME_API_KEY" \
  | jq '.data | {status, output, output_error}'

Which images can I send?

Sume fetches every image when you create the run, checks its real type and size, and copies it into durable storage, so a broken or private image fails the create instead of the run. An image_url must be public HTTPS and reachable without auth. A URL already on media.sume.com, such as an earlier Sume output, is not re-copied.

Why was my image request refused?

These refusals come back on the create call:

• 400 invalid_attachment: wrong type, a missing or non-HTTPS URL, both image_url and asset_id, or a source that is not an allowed image.
• 413 attachment_too_large: an image over 30 MB, or a set over 500 MB.
• 502 attachment_fetch_failed: Sume could not fetch the image because of an unreachable host, hotlink protection, or a non-2xx response.
• 400 invalid_request: a missing generation_spend_cap_usd, neither or both of instruction and messages, or an assistant turn.
• 409 idempotency_conflict: an Idempotency-Key reused with a different payload, such as a new image list.

What does the agent API not accept yet?

input_image is the only attachment type today, so PDFs and other files cannot be attached yet; the endpoint's other gaps, such as streaming and a synchronous choices[] response, are listed in Run the Sume video agent from your backend.