Media tools

Timeline 1.0

Assemble one audio spine plus ordered video slots into a long-form MP4. The only public assembly surface.

Timeline 1.0 takes a declarative document (one audio spine + ordered video[] slots) and returns one MP4. The server compiles ffmpeg on the worker media runtime (apps/api/src/routes.ts renderTimelineV1 / submitSumeTimelineRenderJob). Callers never send filtergraphs, codecs, or shell fragments.

text

POST /v1/timeline-1.0/render
POST /v1/timeline-1.0/plan                 # unbilled compile preflight
POST /v1/models/sume/timeline-1.0/runs     # same body as render, no extra `model` field
POST /v1/models/sume/timeline/v1.0/runs    # legacy model-run alias

This surface has no GET /v1/timeline-1.0/:id. Poll the job envelope:

text

GET /v1/jobs/:id/status
GET /v1/jobs/:id/result

The hosted MCP flow is timeline_create, then jobs_wait, then timeline_get (packages/mcp-server/src/mcp.ts). timeline_get is GET /v1/jobs/:id/result. Writes need idempotency_key (and mcp:write under OAuth).

Plan (unbilled)

POST /v1/timeline-1.0/plan (planTimelineV1) runs schema + Sume-host URL checks + the pure compiler. It returns object: timeline_plan with duration_seconds, segment_count, billable_minutes, and a filtergraph_summary. It does not create a job, reserve credits, or download media. Idempotency-Key is not required. A plan cannot predict short-source pad/loop warnings.

Render

The required fields are audio.duration_seconds (1–1800), plus either audio.url or audio.parts[] (unless audio.mode is "silence"), and video[] (1–200 slots). Every URL must already be this workspace’s media.sume.com artifact or asset. Import the files first (POST /v1/media-imports). Idempotency-Key is required.

The default mode is async (readCommunicationOptions). To wait up to 30 seconds for a 200 finished job, pass mode: "sync". Otherwise, you get 202. Then poll the job.

bash

curl -X POST https://api.sume.com/v1/timeline-1.0/render \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: timeline-001" \
  -d '{
    "audio": {
      "url": "https://media.sume.com/artifacts/artf_demo/voice.wav",
      "duration_seconds": 24
    },
    "video": [
      {
        "source_url": "https://media.sume.com/artifacts/artf_demo/intro.mp4",
        "start": 0,
        "duration": 8
      },
      {
        "source_url": "https://media.sume.com/artifacts/artf_demo/talk.mp4",
        "start": 8,
        "duration": 16,
        "transition": { "type": "fade", "duration": 0.25 }
      }
    ]
  }'

A successful submit returns a job (type: timeline_render, model: sume/timeline-1.0). When the job is result_ready, GET /v1/jobs/:id/result returns kind: timeline_render with video_url, duration_seconds, segment_count, billable_minutes, and optional warnings[]. Soft warnings (padded/looped short sources, snapped transitions, ignored still motion) are not failures.

Credits scale with the output minutes, rounded up to whole minutes. The API rate card lists the current credits. The reserve is ceil(audio.duration_seconds / 60) minutes. The job uses no provider inference, only worker ffmpeg.

The default output is 1080×1920 MP4. If you omit output.fps, the job renders at the rate of the sources (the longest video sources decide, stills have no rate, and 30 applies only when no source has a rate). If a rate is different from the rate of a source, the job repeats or drops a frame every few frames. This causes judder on motion. The job reports this as output_fps_resamples_sources with the rate that the sources wanted.

Program

Field	Effect
`audio.duration_seconds`	Output length. Required. 1–1800 s.
`audio.url`	One Sume-hosted spine. Not permitted with `parts`.
`audio.parts[]`	≤20 gapless slices (`url` + optional `source_in` / `duration`). Sample-domain join, no re-TTS. Not permitted with `url`.
`audio.mode`	`"silence"` — declared length with no spine file. In this mode, do not send `url` / `parts` / `gain_db` / `source_in`.
`audio.source_in`	In-point into a single `url` spine. Not permitted with `parts`. The output length is still `duration_seconds`.
`audio.gain_db`	−60…12. Not permitted with silence.
`video[].source_url`	Sume-hosted clip or still. Stills are static holds (the job accepts `motion`, ignores it, and reports `motion_ignored`).
`video[].start`	On-spine start. `video[0].start` must be 0. Later starts must increase. Declared starts are authoritative. The compiler compensates for xfade and never pre-shifts.
`video[].duration`	On-screen length, ≥ 0.2 s. Coverage can stop at most 0.5 s before the end of the spine.
`video[].source_in`	In-point into the file.
`video[].fit`	`cover` (default) \| `contain` \| `stretch` \| `blur`.
`video[].transition`	Use only on slots after the first. `type` ∈ `fade` \| `wipeleft` \| `wiperight` \| `slideup` \| `slidedown` \| `dissolve`. Duration ≤ 1 s, ≤ 50% of the shorter neighbor, and at least one output frame.
`output.width` / `height`	Even integers 256–2160.
`output.fps`	`24` \| `25` \| `30` \| `60`. Omit it to match the sources.
`output.fade_in_seconds` / `fade_out_seconds`	0–5 s. The sum must be ≤ output length.
`soundtrack`	Optional bed: `url`, `gain_db`, `loop`, `fade_out_seconds` ≤ 10, `duck_db` 0–20 (needs a real spine, not silence).
`render.strategy`	`auto` (default, chunks past 12 segments) \| `chunked` \| `single` (Sume refuses `single` above 12 slots: `render_strategy_unsafe`).

If you need sliced VO only inside this render, use audio.parts[]. For a reusable merged file, use timeline audio. For two sources on screen at the same time, use timeline compose. Then put that MP4 into video[].

Refusals (stable codes)

Code	When
`audio_url_required`	No `url` / `parts` and not silence.
`audio_url_and_parts_exclusive`	Both `url` and `parts`.
`silent_audio_takes_no_url` / `_parts` / `_gain` / `_source_in`	Silence plus a spine field.
`audio_source_in_requires_single_spine`	`source_in` with `parts`.
`audio_parts_shorter_than_duration`	The sum of the declared part lengths is less than `duration_seconds`.
`timeline_must_start_at_zero`	`video[0].start` ≠ 0.
`transition_on_first_segment`	`video[0].transition`.
`invalid_segment_timing` / `segment_overlap`	Starts do not increase, or slots overlap past the xfade.
`transition_too_long` / `transition_not_frame_aligned`	Duration compared with neighbors / fps.
`too_many_chained_transitions`	More than 8 adjacent fades. Insert a hard cut.
`edge_fades_exceed_output` / `soundtrack_fade_exceeds_output`	Fade longer than the spine.
`duck_requires_audio_spine`	`soundtrack.duck_db` with silence.
`render_strategy_unsafe`	`strategy: "single"` with more than 12 slots.
`unsupported_media_source` / `source_not_found`	Off-host or dead URL.
Provider / ffmpeg keys	400 — `model`, `filtergraph`, `ffmpeg_args`, `codec`, `crf`, and similar keys (`TIMELINE_REJECTED_PROVIDER_KEYS`).

Sume rejects off-host URLs (https://example.com/…) at admit. Import the files first.

A dim / crop / pixel pass is not a render option. Use video filter.

Not this surface

Need	Use
`[start, end)` of one clip	Video trim
Audio track as a durable wav / mp3	Audio detach
Concat / split audio into reusable files	Timeline audio
Still + video in one frame (반배너)	Timeline compose
Pixel pass (dim / crop)	Video filter
Probe / stills / optional STT	Video inspect

Media tools

Timeline 1.0

Assemble one audio spine plus ordered video slots into a long-form MP4. The only public assembly surface.

text

POST /v1/timeline-1.0/render
POST /v1/timeline-1.0/plan                 # unbilled compile preflight
POST /v1/models/sume/timeline-1.0/runs     # same body as render, no extra `model` field
POST /v1/models/sume/timeline/v1.0/runs    # legacy model-run alias

This surface has no GET /v1/timeline-1.0/:id. Poll the job envelope:

text

GET /v1/jobs/:id/status
GET /v1/jobs/:id/result

Plan (unbilled)

Render

The default mode is async (readCommunicationOptions). To wait up to 30 seconds for a 200 finished job, pass mode: "sync". Otherwise, you get 202. Then poll the job.

bash

curl -X POST https://api.sume.com/v1/timeline-1.0/render \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: timeline-001" \
  -d '{
    "audio": {
      "url": "https://media.sume.com/artifacts/artf_demo/voice.wav",
      "duration_seconds": 24
    },
    "video": [
      {
        "source_url": "https://media.sume.com/artifacts/artf_demo/intro.mp4",
        "start": 0,
        "duration": 8
      },
      {
        "source_url": "https://media.sume.com/artifacts/artf_demo/talk.mp4",
        "start": 8,
        "duration": 16,
        "transition": { "type": "fade", "duration": 0.25 }
      }
    ]
  }'

Program

Field	Effect
`audio.duration_seconds`	Output length. Required. 1–1800 s.
`audio.url`	One Sume-hosted spine. Not permitted with `parts`.
`audio.parts[]`	≤20 gapless slices (`url` + optional `source_in` / `duration`). Sample-domain join, no re-TTS. Not permitted with `url`.
`audio.mode`	`"silence"` — declared length with no spine file. In this mode, do not send `url` / `parts` / `gain_db` / `source_in`.
`audio.source_in`	In-point into a single `url` spine. Not permitted with `parts`. The output length is still `duration_seconds`.
`audio.gain_db`	−60…12. Not permitted with silence.
`video[].source_url`	Sume-hosted clip or still. Stills are static holds (the job accepts `motion`, ignores it, and reports `motion_ignored`).
`video[].start`	On-spine start. `video[0].start` must be 0. Later starts must increase. Declared starts are authoritative. The compiler compensates for xfade and never pre-shifts.
`video[].duration`	On-screen length, ≥ 0.2 s. Coverage can stop at most 0.5 s before the end of the spine.
`video[].source_in`	In-point into the file.
`video[].fit`	`cover` (default) \| `contain` \| `stretch` \| `blur`.
`video[].transition`	Use only on slots after the first. `type` ∈ `fade` \| `wipeleft` \| `wiperight` \| `slideup` \| `slidedown` \| `dissolve`. Duration ≤ 1 s, ≤ 50% of the shorter neighbor, and at least one output frame.
`output.width` / `height`	Even integers 256–2160.
`output.fps`	`24` \| `25` \| `30` \| `60`. Omit it to match the sources.
`output.fade_in_seconds` / `fade_out_seconds`	0–5 s. The sum must be ≤ output length.
`soundtrack`	Optional bed: `url`, `gain_db`, `loop`, `fade_out_seconds` ≤ 10, `duck_db` 0–20 (needs a real spine, not silence).
`render.strategy`	`auto` (default, chunks past 12 segments) \| `chunked` \| `single` (Sume refuses `single` above 12 slots: `render_strategy_unsafe`).

Refusals (stable codes)

Code	When
`audio_url_required`	No `url` / `parts` and not silence.
`audio_url_and_parts_exclusive`	Both `url` and `parts`.
`silent_audio_takes_no_url` / `_parts` / `_gain` / `_source_in`	Silence plus a spine field.
`audio_source_in_requires_single_spine`	`source_in` with `parts`.
`audio_parts_shorter_than_duration`	The sum of the declared part lengths is less than `duration_seconds`.
`timeline_must_start_at_zero`	`video[0].start` ≠ 0.
`transition_on_first_segment`	`video[0].transition`.
`invalid_segment_timing` / `segment_overlap`	Starts do not increase, or slots overlap past the xfade.
`transition_too_long` / `transition_not_frame_aligned`	Duration compared with neighbors / fps.
`too_many_chained_transitions`	More than 8 adjacent fades. Insert a hard cut.
`edge_fades_exceed_output` / `soundtrack_fade_exceeds_output`	Fade longer than the spine.
`duck_requires_audio_spine`	`soundtrack.duck_db` with silence.
`render_strategy_unsafe`	`strategy: "single"` with more than 12 slots.
`unsupported_media_source` / `source_not_found`	Off-host or dead URL.
Provider / ffmpeg keys	400 — `model`, `filtergraph`, `ffmpeg_args`, `codec`, `crf`, and similar keys (`TIMELINE_REJECTED_PROVIDER_KEYS`).

Sume rejects off-host URLs (https://example.com/…) at admit. Import the files first.

A dim / crop / pixel pass is not a render option. Use video filter.

Not this surface

Need	Use
`[start, end)` of one clip	Video trim
Audio track as a durable wav / mp3	Audio detach
Concat / split audio into reusable files	Timeline audio
Still + video in one frame (반배너)	Timeline compose
Pixel pass (dim / crop)	Video filter
Probe / stills / optional STT	Video inspect