Remotion with Claude Code: add AI video, voice and music

Remotion works with Claude Code by writing the video as React code: you create a Remotion project, install Remotion's Agent Skills, and start Claude Code in the project folder to prompt the composition. Remotion's <Video> and <Audio> components play files, so AI footage, narration and music have to be generated first and loaded by URL or from the project's public/ folder.

Remotion facts are quoted from its docs on coding agents, Agent Skills, <Video>, <Audio> and Caption; Sume facts come from the Sume API reference and its Media inputs and MCP quickstart docs. All were read on 2026-09-28. Sume has no official Remotion connector; this is plain HTTPS calls plus files.

How do I set up Remotion with Claude Code?

You need Node.js and a coding agent such as Claude Code; Remotion notes that most coding agents require a paid subscription. Then, per Remotion's guide:

• npx create-video --yes --blank my-video creates a blank project; cd my-video and npm install.
• npx remotion skills add installs the Remotion Agent Skills. The Skills page gives npx skills add remotion-dev/skills as another way.
• npm run dev starts the preview; in a separate terminal, run claude in the same folder and prompt the video.
• Remotion has its own license and terms for commercial use, including pricing; read them before you ship.

Where do the AI clips, voiceover and music come from?

From a generation API, one job per asset. On Sume, each finished job's output is a Sume-owned media URL, and the docs say integrations should store that Sume URL (Media inputs):

• Footage: POST /v1/videos for a generated clip; read the media.sume.com artifact from GET /v1/jobs/{id}/result once the job is completed (Jobs and results).
• Narration: POST /v1/tts-1.0/generate with timestamps.words: true; the completed result includes words[] with start and end seconds (Sume API reference).
• Music: POST /v1/music-router/generate, with the length and "instrumental" in the prompt (Music Router).

How do I load those files in the composition?

Remotion's docs say src on <Video> and <Audio> from @remotion/media can be a remote URL or a local file referenced with staticFile(), which turns a file in your public/ folder into a URL. Downloading the three files into public/ keeps the project self-contained. muted drops a clip's own audio, volume sets a level, and loop repeats the bed:

import { AbsoluteFill, staticFile } from "remotion";
import { Audio, Video } from "@remotion/media";

export const Promo: React.FC = () => (
  <AbsoluteFill>
    <Video src={staticFile("clip-1.mp4")} muted />
    <Audio src={staticFile("narration.mp3")} />
    <Audio src={staticFile("music.mp3")} volume={0.2} loop />
  </AbsoluteFill>
);

How do I turn TTS word timings into Remotion captions?

Map each word to Remotion's Caption type from @remotion/captions: text, startMs, endMs, timestampMs and confidence. Remotion says text is whitespace sensitive and should include a space, ideally before each word. In Sume's current code each TTS word carries word, start and end in seconds. From there, Remotion's createTikTokStyleCaptions() groups captions into pages:

import type { Caption } from "@remotion/captions";

type SumeWord = { word: string; start: number; end: number };

export const toCaptions = (words: SumeWord[]): Caption[] =>
  words.map((w) => ({
    text: ` ${w.word}`,
    startMs: Math.round(w.start * 1000),
    endMs: Math.round(w.end * 1000),
    timestampMs: null,
    confidence: null,
  }));

Can Claude Code call Sume directly while it builds?

Yes, two ways. Claude Code can write and run a script against the REST API, with SUME_API_KEY kept in the environment; that is Sume's primary path. Or it can connect to Sume's hosted MCP with claude mcp add --transport http sume https://mcp.sume.com/mcp (MCP quickstart). Sume's docs say MCP and the CLI still work but are not part of the primary path today (Sume basics).

Two MCP details matter here: default OAuth access is read-only until you turn Write on at consent, and one jobs_wait call holds at most 55 seconds; on wait_slice_expired, call jobs_wait again with the same ids rather than resubmitting the paid job. Sume MCP in Claude Code, Cursor and Codex covers the connection.