Introduction
In today's fast-paced world, speed and efficiency are crucial for text-to-speech (TTS) technologies. One of the major challenges is to reduce response time while ensuring impeccable audio quality. At Nari Labs, we have tackled this challenge with our Qwen3-TTS model, which now responds in under 50 ms. Here's how we achieved it.
Defining Real-time TTS
To understand our approach, it's essential to define what "real-time" means for a TTS server. It boils down to four elements:
- Low Audible TTFA: The time from request dispatch to the first audible sound must be minimal.
- Zero Underruns: Once playback starts, audio must remain uninterrupted.
- Capacity: The first two criteria must hold even as requests per second (RPS) increase.
- Non-malformed Output: Speech must be intelligible.
Performance of Other Engines
We compared the Qwen3-TTS implementation to other engines like vLLM-Omni and SGLang-Omni△. Our tests showed that our model is the only one to achieve a sub-50 ms p95 TTFA.
How We Optimized Qwen3-TTS
Optimizing Qwen3-TTS involved several key steps:
- Removing Initial Silence: We detected and removed silence at the beginning of the audio generated by the model.
- Improving the Processing Pipeline: By optimizing each step of the process, we reduced cumulative delays.
- Efficient Hardware Use: Utilizing an NVIDIA H100 SXM allowed us to maximize efficiency without sacrificing quality.
We also tested our model under Poisson open-loop traffic to simulate real workloads and found that we could maintain sub-50 ms performance even at 10 RPS.
What's Next?
We continue to fine-tune our model to further increase capacity while reducing costs. Our goal is to make this technology accessible and affordable for any business looking to integrate high-quality TTS solutions.
Let's discuss your project in 15 minutes.
Conclusion
Optimizing a TTS model to respond in under 50 ms is a significant challenge, but not impossible. With the right strategies and technologies, it's possible to push the boundaries of what's achievable.