Auto-Caption Your Videos with WriftAI

Whether it's a TikTok, Instagram Reel, YouTube Short, podcast clip, interview, or film, captions have become part of the video itself. And good captions are surprisingly hard to get right.

They need to keep up with the speaker, break at the right places, stay readable, and look good on top of whatever is happening in the video. A one-line caption that works for a fast-paced Reel might feel completely wrong in a long interview. Add word highlighting, different fonts, positioning, multiple languages, and suddenly there are a lot more decisions to make.

We shipped a model, sych/auto-video-captioning, to handle all of it.

Give it a video, choose how you want the captions to behave, and get back a captioned video along with the subtitle file. Here are a few examples of what you can do with it:

Start with simple captions

Sometimes you just need straightforward captions that follow the speaker without drawing too much attention to themselves.

These settings provide a straightforward starting point. Captions appear one line at a time, with up to 40 characters per line and a maximum duration of three seconds. The right timing depends on the pace of the speech. Shorter durations can make captions change more frequently, while longer durations give viewers more time to read.

Keeping captions readable

Four parameters matter most here:

• Max Characters per Line: controls how much text can fit on a line.
• Max Lines: controls how many lines a caption can use.
• Max Caption Duration: controls how long a caption can remain visible.
• Min Caption Duration: controls the minimum preferred duration for a caption.

Together, these parameters determine how much text the viewer sees at once and how quickly the captions move.

Highlighting spoken words

Using the same video clip, we can enable word highlighting without changing the other caption settings.

Captions as visual elements

Captions can do more than display what's being said. They can also contribute to a video's visual identity. For this example, we used a different 27-second segment from the same video and changed the typography, colors, outline, and line length to create a bolder style.

The short lines, large centered font, and orange text give the captions a much stronger visual presence. This configuration shows how much the presentation can change without requiring a different captioning workflow. You can experiment with fonts, colors, spacing, and timing to match the style of your video.

Captions for longer speech

Short, frequently changing captions aren't always the best fit for longer conversations. When a speaker delivers a longer phrase, allowing two lines can keep more of the text together and make it easier to follow. For this video, we used a 36-character line limit, two lines per caption, and longer display durations.

Compared with the short, bold captions in the previous example, this configuration prioritizes readability over visual emphasis. The two-line layout accommodates longer phrases, and the longer durations give viewers more time to read them.

The goal isn't to use one configuration for every video. It's to adjust the caption behavior to suit the content, whether that means short, prominent captions or longer subtitles that keep more of a conversation together.

Fonts for different styles and scripts

The model includes fonts for both visual styling and different writing systems:

This matters for multilingual videos because the font needs to support the writing system being rendered, not just the visual style you want.

Multilingual support

Captions aren't limited to English. The model supports 99 languages, including English, Urdu, Chinese, Korean, Japanese, Arabic, Hindi, and Bengali. If you already know the language of the video, you can set Language to English. If you don't, set it to Auto and the model will detect it.

Translating captions into English

There's also a separate Translate to English option. When it's on, captions are translated from the video's spoken language into English. Translation only goes into English. It doesn't translate from one non-English language into another.

Running the model

Once you've decided how you want the captions to behave, you can caption your video directly in the model's Playground without writing any code, or call the model from your own application.

sych/auto-video-captioning playground on WriftAI

WriftAI also provides SDKs for JavaScript/TypeScript, Python and Go, plus a CLI. Below is an example using the Python SDK:

1from pathlib import Path
2
3from wriftai import File, Client
4
5client = Client()
6
7prediction = client.predictions.create(
8    model="sych/auto-video-captioning",
9    input={
10        "video": File(Path("/path/to/video.mp4")),
11        "language": "auto",
12        "max_chars_per_line": 40,
13        "max_lines": 2,
14        "max_caption_duration": 3.0,
15        "min_caption_duration": 0.5,
16        "caption_position": "bottom",
17        "font_name": "Montserrat SemiBold",
18        "font_size": 48,
19        "font_color": "#FFFFFF",
20        "highlight_words": True,
21        "highlight_color": "#FFFF00",
22        "opacity": 1.0,
23        "stroke_color": "#000000",
24        "stroke_width": 3.0,
25        "letter_spacing": 0.0,
26		"translate_to_english": False
27    },
28    wait=True,
29)
30
31print(prediction.output)
32# {
33#   "video": "https://cdn.wrift.ai/temporary/video-6808ef49.mp4",
34#   "subtitles": "https://cdn.wrift.ai/temporary/captions-0b31f87b.ass"
35# }

You don't need to specify every parameter. The defaults are there for a straightforward captioning workflow, and you can override the settings that matter for your particular video. The full list of parameters and their defaults is on the model's API page.

What comes back

The model returns the captioned video and a separate subtitle file, so you can publish the video as is or edit the subtitles separately.

Auto video captioning pipeline

That's the idea behind the model: one captioning workflow, configured to fit each video.