Understanding Modern Text to Speech

Text-to-speech has moved far beyond simply converting text into a natural-sounding voice. Modern TTS systems can generate expressive speech, preserve a speaker's identity from reference audio, and even create entirely new voices from natural-language descriptions. This evolution has made TTS a much more flexible component of modern AI systems.

An AI voice agent needs low-latency speech generation to keep conversations responsive, an audiobook needs consistent and expressive narration across long-form content, while a virtual character may need a completely new voice designed around a specific personality or style.

The engineering challenge is therefore not simply finding a model that generates good audio. It is choosing the model whose quality, controllability, and inference performance match the application.

Inside a TTS Pipeline

At a high level, a TTS system progressively transforms textual input into speech. The text is first converted into a representation the model can process. A generative model then predicts an intermediate speech representation, such as acoustic features, mel-spectrograms, or discrete speech tokens, which is finally decoded into a waveform.

Modern TTS systems can additionally condition generation on a predefined voice, speaker embedding, reference audio, or natural-language voice description. This enables capabilities such as preset-voice synthesis, voice cloning, voice design, and controllable speech generation.

Figure 1: Simplified TTS workflow with voice conditioning.

Voice Generation Modes

Modern TTS systems can condition speech generation on more than just text. Depending on the model and inference mode, a voice can be selected from a predefined set, reproduced from reference audio, or generated from a natural-language description.

Figure 2: Modern TTS conditioning approaches for preset voices, voice cloning, and voice design.

Preset Voice Synthesis

Preset-voice synthesis generates speech using a predefined speaker identity, making it well suited for applications that require consistent voices and efficient generation without dynamically creating speaker identities. For example, hexgrad/kokoro-82m provides a set of predefined voices that users can select while supplying the text to be synthesized.

Voice Cloning

Voice cloning uses reference audio to capture the characteristics of an existing speaker and reproduce them in new speech. For example, qwen/qwen3-tts-12hz-1.7b-base can use a short voice sample to capture features such as timbre, pitch, and speaker identity, enabling personalized and consistent voice generation.

Voice Design

Voice design generates a new speaker by specifying characteristics instead of providing reference audio. For example, k2-fsa/omnivoice allows parameters such as age, gender, pitch, accent, and dialect to be configured to shape the generated voice. This provides greater control when creating a new speaker identity without relying on a preset voice or existing recording.

Understanding the Tradeoffs

Speech Quality

Speech quality is more than producing audio that simply sounds human. A production TTS system needs natural prosody, accurate pronunciation, intelligibility, and consistent speech characteristics. For voice cloning, speaker similarity becomes an additional consideration, since the generated speech must preserve the identity of the reference speaker while maintaining natural output.

The trade-off becomes more interesting when model footprint is considered alongside speech quality. hexgrad/kokoro-82m, with its compact 82M-parameter architecture, is attractive when high-quality speech needs to be generated with a lightweight model. Larger models such as k2-fsa/omnivoice and Qwen3-TTS models have a higher computational footprint, but their additional capacity can support more demanding speech-generation workloads. For example, qwen/qwen3-tts-12hz-1.7b-base reports a 0.789 speaker-similarity score and 1.835% WER in its published multilingual voice-cloning evaluation, highlighting its ability to preserve speaker identity while maintaining transcription accuracy.

Figure 3: Qualitative comparison of speech quality and model footprint across TTS models.

Voice Control

Controllability determines how much influence an application has over who is speaking and how that voice is generated. A fixed-voice system may only require selecting a speaker, while more flexible systems can condition generation on reference audio or specific voice attributes.

hexgrad/kokoro-82m keeps this control space relatively simple by working with predefined voices, making it well suited to applications where a consistent speaker is all that is required. Qwen3-TTS and k2-fsa/omnivoice provide a much broader control space, supporting different generation modes for preset voices, voice cloning, and voice design.

The key engineering trade-off is therefore simplicity versus control. If an application only needs a reliable predefined voice, the additional conditioning mechanisms may add unnecessary complexity. When speaker identity needs to be cloned or dynamically designed, the larger control space becomes much more valuable.

Figure 4: Qualitative comparison of voice controllability across TTS models.

Real-Time Factor

Once speech quality and controllability meet the application's requirements, inference speed becomes another key constraint.

Real-Time Factor (RTF) measures the time required to generate speech relative to the duration of the resulting audio. An RTF of 0.2 means 10 seconds of audio can be generated in roughly 2 seconds under the same benchmark conditions. Lower RTF means faster generation and is particularly important for high-throughput or latency-sensitive workloads.

Model size, hardware, precision, sequence length, and serving configuration all influence RTF. hexgrad/kokoro-82m's 82M-parameter architecture makes it naturally attractive for lightweight, high-throughput synthesis, while k2-fsa/omnivoice reports an RTF as low as 0.025 in its published configuration, equivalent to approximately 40× real-time.

These figures should not be treated as a universal ranking, since RTF can change significantly with hardware and inference configuration. The practical decision is therefore about finding the right balance between generation speed, speech quality, and the capabilities required by the application.

Choosing The right TTS model

The best TTS model ultimately depends on your workload, quality requirements, voice control, and inference constraints. The table below highlights some of the options available, WriftAI hosts many more than what's shown here.

Conclusion

Modern TTS is no longer limited to converting text into speech. With preset voices, voice cloning, and voice design, you can build systems with different levels of speaker control, while model size, speech quality, and inference speed introduce their own trade-offs.

The best way to choose the right TTS model for your pipeline is to test it on your own workload. Explore TTS models on WriftAI to find the one that best fits your requirements.