JSON to video API: render an MP4 from a timeline document

A JSON to video API takes a JSON document that describes an edit (which clips play, when, for how long, over which audio) and renders it into one video file on the server, so you don't run a renderer yourself. Your code writes the JSON; the API returns an MP4.

On Sume that document is the body of Timeline 1.0: one audio track plus ordered video[] slots, rendered by POST /v1/timeline-1.0/render. The facts below come from the Timeline 1.0 docs, read on 2026-09-29; anything described as current behavior is read from Sume's code.

What goes in the JSON?

The docs call it a declarative document: the server compiles FFmpeg itself, and callers never send filtergraphs, codecs, or shell fragments. A document has four parts; the limits are in the table further down.

• audio: the sound and the output length (audio.duration_seconds), from one audio.url, up to 20 audio.parts[], or audio.mode: "silence".
• video[]: the slots, each with a source_url (a clip or a still), a start on the timeline, a duration, and optionally a source_in into the file, a fit (cover, contain, stretch, or blur), and a transition on any slot after the first.
• output: optional size, frame rate, and edge fades. The default is 1080×1920.
• soundtrack: an optional music bed under the audio.

{
  "audio": {
    "url": "https://media.sume.com/artifacts/artf_demo/voice.wav",
    "duration_seconds": 24
  },
  "video": [
    {
      "source_url": "https://media.sume.com/artifacts/artf_demo/intro.mp4",
      "start": 0,
      "duration": 8
    },
    {
      "source_url": "https://media.sume.com/artifacts/artf_demo/talk.mp4",
      "start": 8,
      "duration": 16,
      "fit": "contain",
      "transition": { "type": "fade", "duration": 0.25 }
    }
  ]
}

How do I turn my own JSON data into a video?

Map it. A JSON to video API does not read arbitrary JSON; it reads its own schema. If your data is a list of products, scenes, or rows, your code loops over it and writes one Timeline document per video: each item becomes a slot with a start and a duration, and the next item's start is the sum of the durations before it. The first slot must start at 0, and later starts must increase.

Then send each document in two calls:

• POST /v1/timeline-1.0/plan is unbilled. It runs the schema, the URL checks, and the compiler, and returns duration_seconds, billable_minutes, and estimated_cost_usd_micros without creating a job. Validate a timeline before rendering covers it.
• POST /v1/timeline-1.0/render with an Idempotency-Key creates the job. Poll GET /v1/jobs/:id/status until it is terminal, then read video_url from GET /v1/jobs/:id/result, or pass a webhook_url to be called when it ends.

Can it render a Lottie JSON animation?

No. A Lottie (bodymovin) animation file is also JSON, but it describes vector shapes and keyframes. The fields in Sume's Timeline docs include no Lottie input, no text layer, and no template variable: its JSON arranges existing video clips and stills over audio. Titles and captions are a separate step, such as timed text over a video.

How much does a JSON to video render cost?

The render reserves $0.10 per output minute, counted as ceil(output minutes), and captures its own compute, never above the reservation. A 90-second video reserves 2 minutes, at most $0.20, plus a 5.5% agent fee by default. The plan call is free.

What are the catches?

• Every URL in the document must already be a file in your Sume workspace, such as an earlier Sume job's output. Off-host URLs like https://example.com/… are rejected.
• The render's sound comes only from audio and soundtrack. In current code each clip's own audio is dropped, so put speech or music on the audio track.
• A frame rate that differs from a source's is met by repeating or dropping frames, and the result warns output_fps_resamples_sources.