What is a video agent? How Sume defines and runs one

In Sume's docs, a video agent is a sandbox Agent that composes individual generation tools (Image, Video, Avatar, TTS, timeline, and more) into a post-ready video. You describe the deliverable; the agent picks the models, asks before it spends, and assembles the result, so you are not stitching raw model calls together yourself.

This post defines the term only as Sume's own docs use it, starting from Sume basics, which calls Sume “fundamentally a video agent platform.”

What can a video agent make that one model call cannot?

The docs' answer is composition. Because the agent combines tools, you can ship deliverables that a single generated clip cannot: multi-minute host footage, B-roll, voiceover, and timeline assembly into a post-ready video.

The individual models still exist as atomic HTTP endpoints: create an avatar, render a talking clip, generate an image or a short video clip, add captions. The docs describe them as supporting pieces, relative to Formats. A Format decides which tools to call, in what order, and how to assemble the result. If you only need a single clip or image, call the model; if you need a packaged workflow, call a Format.

How do you work with Sume's video agent?

In the Agents chat, a person works with the sandbox Agent: write a brief, approve spend, inspect artifacts, and shape a house style over a few turns. The thread keeps every artifact and approval. Interactive chat is the right surface when a human should stay in the loop.

Describe the deliverable, not the tool calls. This is the example brief from the quick start:

Make a 15-second vertical product promo with a presenter avatar and captions.
Show me a draft before the final render.

How does a video agent become an API call?

You save the recipe. Ask the agent to “Save this as a Format called product-promo,” or edit a Format in the library. A Format is a saved authoring recipe (SKILL.md plus references), addressable by handle and slug. When a partner calls it, Sume boots a fresh sandbox, loads the recipe, runs the Agent with generation tools, and returns artifacts plus optional structured JSON. The docs call the Format API the surface most partners should integrate.

Your backend sends POST https://api.sume.com/v1/formats/{handle}/{slug}/runs with an Idempotency-Key header, using a key with the formats:read and formats:write scopes. The call returns 202 with a run receipt; poll status_url until the run leaves queued or processing, then read result_url. More in What is a Sume Format?

Format run = fresh sandbox + recipe (SKILL) + instruction/input + tools
           → artifacts + optional structured output

Which Sume surface runs the agent for which job?

Every surface except the model endpoints runs the Agent. Agent Completions are ad-hoc, Scheduled is recurring, and Formats stay the path when the recipe is fixed and only the inputs change.

How do I keep an unattended video agent in bounds?

The best-practices and basics pages give these rules for calling Formats and Agent Completions:

• Cap spend on every run. Always set generation_spend_cap_usd, and an agent cap when relevant; treat a missing cap as a bug in the client.
• Prefer a Format over raw model calls when you have a saved recipe. The Format owns the tooling, spend gates, and house style; your client only sends the brief.
• Bind output_schema when another service will read the result, and require only fields the Format actually produces.
• Keep credentials scoped: a team API key for Formats on a team handle, the narrowest scopes your client needs, and key rotation from the dashboard.
• Let the key carry the workspace. Do not send workspace_id in request bodies.

Can other agents use Sume's tools?

Yes. Sume's developer tools share one boundary: the public API is the source of truth, and wrappers stay thin. Besides the workspace-scoped Developer API, there is a CLI for shells and scripts, hosted MCP at https://mcp.sume.com/mcp for remote MCP clients, and dashboard tools. The basics page notes that the CLI and hosted MCP still work but are not the primary path today.

The docs ask agent tools to use catalog, jobs, media schema, and usage reads for planning before requesting confirmation for write or paid actions, and not to ask the user for a workspace id: that context comes from the API key or the authenticated app session.