Translate a video's voiceover by API: STT, TTS, Timeline

To translate a video's voiceover with the Sume API, transcribe the original speech with POST /v1/video-inspect, translate the lines yourself or with Agent Completions, speak the translation with POST /v1/tts-1.0/generate and its language field, then render the video over the new audio with POST /v1/timeline-1.0/render. TTS speaks the transcript you send, so the translation step is yours, and the speaker's lips do not move with the new voice.

Facts come from Sume's Video inspect, Timeline 1.0, Agent Completions, and Models overview docs and the TTS schema in the Sume API reference, read on 2026-09-27. Anything described as current behavior is read from Sume's code. To translate the subtitles instead of the voice, see Translate video subtitles by API.

How do I get the original lines and their timing?

Transcribe once with POST /v1/video-inspect: transcribe: true, segmentation: { "mode": "sentence" }, and language_code as a hint. Each sentence in segments[] has its index, text, start, and end in seconds; keep every start, because it becomes that sentence's source_in in the render. The video must already be in your workspace on media.sume.com, such as an earlier Sume job's output.

Who translates the text?

You do, or a Sume agent run you start. Send only the sentence texts to your translator, keep each start and end in your code, and check that the line count still matches.

To stay on Sume, send the lines to POST /v1/agent/completions as input, with an output_schema for the translated lines; Translate video subtitles by API shows that request. Review the translated lines before you pay for speech.

How do I speak the translation?

Send the translated lines as one transcript, up to 20,000 characters, with a voice and the target language. The voice is the avatar_id or avatar_handle of an avatar whose voice is ready, or a voice.id. Set language for every non-English transcript: omitted, it defaults to English, with Korean or Japanese inferred from a Hangul- or kana-only transcript as a fallback. timestamps.words: true plus segmentation.mode: "sentence" returns gapless sentence segments[] with the new timings.

In the current code, language: "ko" with no Hangul syllable in the transcript is refused with 400 tts_language_script_mismatch, which catches a script that was never translated.

curl -X POST https://api.sume.com/v1/tts-1.0/generate \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: dub-ko-001" \
  -d '{
    "transcript": "오늘은 새 대시보드를 소개합니다. 설정은 1분이면 끝납니다.",
    "avatar_handle": "product_host",
    "language": "ko",
    "output_format": { "container": "wav", "encoding": "pcm_s16le", "sample_rate": 44100 },
    "timestamps": { "words": true },
    "segmentation": { "mode": "sentence" }
  }'

What happens when the voice's language differs?

In the current code, TTS compares the voice's primary language, when Sume has one on record, with the requested language: your language, or else ko or ja inferred from the transcript, or else en. If they differ, the submit fails with 409 and error.code tts_voice_language_mismatch; error.details names voice_language and request_language, and the message warns that pronunciation may sound unnatural. No job or charge has been created at that point.

After the user confirms, resend the same request and Idempotency-Key with confirm_language_mismatch: true. The API reference says confirmation does not change the requested voice or language. A voice with no language on record is not checked.

How do I put the new voice under the video?

Render the original video over the new audio. audio.url is the TTS file's media.sume.com URL from the job result, and audio.duration_seconds its length, which the current code reports as the result's duration_seconds when timestamps.words is on. Give each sentence its own video[] slot so the pictures follow the new timing:

• source_url is the original video, and source_in the original sentence's start.
• start and duration come from the translated sentence's TTS segment. The first segment starts at 0 in the current code, as video[0].start must.
• Leave out transition, set output.width and output.height to the source's size (the default is 1080×1920), and remember the 200-slot cap: at most 200 sentences per render.
• In the current compiler a render plays only its spine and an optional soundtrack, so the original voice, and any music under it, is gone. Add a Sume-hosted bed as soundtrack if you have one.
• The new voice does not move anyone's lips: Sume's model docs say video models do not lip-sync to a later voice-over. For a speaker on camera, make new talking shots with VEED Fabric 1.0, which turns a still plus audio into a talking clip; its audio_url takes Sume-hosted audio up to 10 MB, typically a TTS segment, and with a wav container each TTS segment carries its own audio_url. See Lip sync API.

curl -X POST https://api.sume.com/v1/timeline-1.0/render \
  -H "Authorization: Bearer $SUME_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: dub-ko-render-001" \
  -d '{
    "audio": { "url": "https://media.sume.com/artifacts/artf_demo/tts-ko.wav", "duration_seconds": 7.4 },
    "output": { "width": 1920, "height": 1080 },
    "video": [
      { "source_url": "https://media.sume.com/artifacts/artf_demo/talk.mp4", "start": 0, "duration": 3.9, "source_in": 0 },
      { "source_url": "https://media.sume.com/artifacts/artf_demo/talk.mp4", "start": 3.9, "duration": 3.5, "source_in": 3.2 }
    ]
  }'

What does it cost, and what are the limits?

Each step bills on its own, plus a 5.5% agent fee by default. Synthesized audio longer than 1,200 seconds fails with tts_duration_exceeded and captures no credit.