Agentic video editing: what an AI editing agent does

Agentic video editing means an AI agent plans an edit and carries it out by calling editing tools (inspect, trim, filter, detach audio, assemble), then looks at the result and fixes what is off. A one-click AI editor applies one fixed effect; an editing agent chains many steps toward a goal you describe in words.

The tool facts below come from Sume's MCP tools and gates page and the docs for each tool, read on 2026-09-28. For Sume's own definition of the broader term, see What is a video agent?.

How does an AI video editing agent work?

It runs a loop, the same one a human editor runs, except each step is a tool call:

• Look: read the clip's facts (duration, streams, size) and sample stills, and a transcript if the edit depends on what is said.
• Plan: turn your brief into concrete decisions, such as cut from 3.2 s to 9.8 s, crop to vertical, join these four clips with a fade.
• Act: call one tool per decision. A trim, filter, or audio detach makes a new file and leaves the source untouched.
• Check: inspect the output, compare it with the brief, and repeat a step if needed.

Which tools does an editing agent call?

One per step of the loop. Each tool hands the agent facts or a new file, never a verdict, so every step also has a limit the agent has to work around. On Sume these are hosted MCP tools with matching REST routes; Video editing MCP server lists every tool and its call order.

What footage can an editing agent work on?

Only what its tools can read, so where your footage lives matters as much as the prompt. Sume's editing tools read only this workspace's media.sume.com files, such as the output of an earlier Sume generation or edit. There is no public route for uploading a local file, so footage on your laptop is out of reach today.

Video captions differ: POST /v1/video-captions takes a fetchable public HTTPS video URL. In current code the caption job refuses a source longer than 60 seconds or one with no audio stream.

Sound needs a plan too. A Timeline render takes its sound only from the audio spine and an optional soundtrack, so an agent that wants to keep dialogue detaches it first and places the file on the spine.

Can the agent check an edit before paying for it?

Partly. On Timeline 1.0, POST /v1/timeline-1.0/plan validates a document without billing and returns its duration, segment count, and estimated cost, and the filter check is unbilled too. Neither can tell whether the cut is good. The agent can inspect its own output afterwards, but pacing and taste still need a person, so ask for a draft and review it.

How do I run agentic video editing without watching it?

Send the brief to the Sume agent from your backend. An Agent Completion runs the same agent as the Agents chat, with its sandbox and tools, from your own backend with an API key and nobody watching. It needs a generation_spend_cap_usd, which has no default. To drive the tools from your own agent instead, connect it to https://mcp.sume.com/mcp; Video editing MCP server lists every tool, and Can Claude edit videos? walks through one client.