/benchmarks/tts

TTS Universal Benchmark

A model-agnostic serving benchmark for TTS models in vllm-omni. One CLI (bench_tts.py) + one YAML registry (model_configs.yaml) drive perf and quality runs for every registered checkpoint: Qwen3-TTS (Base / CustomVoice) and VoxCPM2 today, more to come.

The same three task types — voice_clone, default_voice, voice_design — are wired into both the manual CLI and the DFX nightly CI matrix (tests/dfx/perf/tests/test_tts.json).

Quick start

1. Start the server

vllm serve Qwen/Qwen3-TTS-12Hz-1.7B-Base --omni --port 8000

The server auto-loads its Deploy YAML from vllm_omni/deploy/qwen3_tts.yaml (Pipeline + Deploy schema introduced in #2383). No --stage-configs-path or --deploy-config flag is needed for any registered model.

2. Run the benchmark (vllm bench serve --omni)

The primary, directly-controllable path. Copy-paste one of these and tweak any bench flag (sampling params, endpoint, extra body, warmups, etc.):

voice_clone (Qwen3-TTS-Base, seed-tts dataset)

vllm bench serve --omni \
    --host 127.0.0.1 --port 8000 \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --backend openai-audio-speech \
    --endpoint /v1/audio/speech \
    --dataset-name seed-tts \
    --dataset-path /path/to/seed-tts-eval \
    --seed-tts-locale en \
    --num-prompts 20 --num-warmups 2 \
    --extra-body '{"task_type":"Base"}' \
    --max-concurrency 1 --request-rate inf \
    --percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
    --save-result --result-dir ./results

default_voice (Qwen3-TTS-CustomVoice, bundled seed_tts_smoke)

vllm bench serve --omni \
    --host 127.0.0.1 --port 8000 \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --backend openai-audio-speech \
    --endpoint /v1/audio/speech \
    --dataset-name seed-tts-text \
    --dataset-path benchmarks/build_dataset/seed_tts_smoke \
    --seed-tts-locale en \
    --num-prompts 20 --num-warmups 2 \
    --extra-body '{"voice":"Vivian","language":"English","task_type":"CustomVoice"}' \
    --max-concurrency 1 --request-rate inf \
    --percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
    --save-result --result-dir ./results

voice_design (Qwen3-TTS-CustomVoice, bundled seed_tts_design)

vllm bench serve --omni \
    --host 127.0.0.1 --port 8000 \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice \
    --backend openai-audio-speech \
    --endpoint /v1/audio/speech \
    --dataset-name seed-tts-design \
    --dataset-path benchmarks/build_dataset/seed_tts_design \
    --seed-tts-locale en \
    --num-prompts 20 --num-warmups 2 \
    --extra-body '{"task_type":"VoiceDesign","language":"English"}' \
    --max-concurrency 1 --request-rate inf \
    --percentile-metrics ttft,e2el,audio_rtf,audio_ttfp,audio_duration,audio_underrun \
    --save-result --result-dir ./results

Streaming continuity (audio_underrun)

audio_underrun (seconds) is the per-request worst-case buffer deficit under a realtime-rate player simulation. The audio-speech backend records chunk arrival times during the SSE stream and surfaces:

• Mean / Median / P{50,99} AUDIO_UNDERRUN (s) per the standard percentile output - any positive value means at least one inter-chunk gap was longer than the chunk's audio duration.
• Streaming continuity OK rate - fraction of requests whose worst-case underrun stayed under the threshold (default 100 ms, the commonly cited "audible gap" budget). Override the threshold with VLLM_OMNI_BENCH_AUDIO_CONTINUITY_THRESHOLD_S=<float>.

This captures the failure mode where RTF_p50 < 1 (server keeps up in aggregate) but per-stream chunk arrival is bursty enough that listeners still hear gaps - common at high concurrency on streaming TTS pipelines.

Add WER / SIM / UTMOS to any of the above

Append --seed-tts-wer-eval (and optionally SEED_TTS_EVAL_DEVICE=cuda:0 in the env, per PR #2558). This triggers the seed-tts-eval protocol: Whisper-large-v3 ASR → WER, WavLM embeddings → SIM, balacoon/utmos → UTMOS.

3. Convenience wrapper (bench_tts.py)

If you're running the canonical configuration for a registered model, bench_tts.py loads the right defaults from model_configs.yaml and emits the exact vllm bench serve --omni command above — useful for concurrency sweeps and multi-task runs:

# Smallest smoke — 5 prompts, concurrency=1
python benchmarks/tts/bench_tts.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --task voice_clone \
    --dataset-path /path/to/seed-tts-eval \
    --concurrency 1 --num-prompts 5 \
    --output-dir ./results

# Full concurrency sweep
python benchmarks/tts/bench_tts.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --task voice_clone \
    --dataset-path /path/to/seed-tts-eval \
    --concurrency 1 2 4 8 16 32 \
    --num-prompts 20 \
    --output-dir ./results

# With WER / SIM / UTMOS quality eval (adds ASR + embedding compute)
python benchmarks/tts/bench_tts.py \
    --model Qwen/Qwen3-TTS-12Hz-1.7B-Base \
    --task voice_clone \
    --dataset-path /path/to/seed-tts-eval \
    --wer-eval \
    --concurrency 4 --num-prompts 200 \
    --output-dir ./results

4. Plot a sweep

python benchmarks/tts/plot_results.py \
    --results ./results/*.json \
    --output ./results/curve.png

Outputs TTFP / RTF / throughput curves (and a markdown table) for every (task, concurrency) combination in the result set.

Task types

Task	Dataset	Request body	Checkpoints that support it
voice_clone	seed-tts	ref_audio + ref_text + task_type=Base	Qwen3-TTS-*-Base, VoxCPM2
default_voice	seed-tts-text	voice=Vivian + task_type=CustomVoice	Qwen3-TTS-*-CustomVoice
voice_design	seed-tts-design	instructions=<natural-language description> + task_type=VoiceDesign	Qwen3-TTS-*-CustomVoice

-CustomVoice checkpoints do NOT ship speaker_encoder weights, so voice_clone requests raise ValueError at model runtime. Use -Base for voice_clone.

Adding a new TTS model

Drop an entry into model_configs.yaml — no Python changes required:

models:
  <org>/<model-id>:
    supported_tasks: [voice_clone]          # or default_voice / voice_design
    backend: openai-audio-speech            # vllm bench serve backend
    endpoint: /v1/audio/speech              # OpenAI-compatible endpoint
    task_extra_body:                        # merged into every request's body
      voice_clone:
        task_type: Base

Then add the model's Deploy YAML under vllm_omni/deploy/<model>.yaml (Pipeline + Deploy schema) and it's immediately benchable.

Datasets

Dataset	Bundled?	Format	Source
seed-tts-design	✅	5-field meta.lst	benchmarks/build_dataset/seed_tts_design/en/meta.lst (20 prompts)
seed_tts_smoke	✅	4-field meta.lst	benchmarks/build_dataset/seed_tts_smoke/en/meta.lst (20 text-only)
seed-tts	❌	4-field meta.lst + WAVs	Google-Drive: BytedanceSpeech/seed-tts-eval (~1.2 GB)
seed-tts-text	❌	4-field meta.lst	Same archive as seed-tts (wav column unused)

For manual voice_clone / default_voice runs against the full corpus, follow benchmarks/build_dataset/download_process_data_seedtts.md and point --dataset-path at the extracted seedtts_testset directory.

DFX nightly CI

tests/dfx/perf/tests/test_tts.json wires three perf regimes plus quality:

eval_phase	concurrency	purpose	Baseline metrics
latency	1	Single-request TTFP / RTF SLO	median_audio_ttfp_ms, median_audio_rtf
throughput	8	Codec-batching cliff sentinel (PDF #272 concurrency≥8)	median_audio_ttfp_ms, median_audio_rtf
quality	4	WER / SIM / UTMOS regression (disabled in CI by default)	mean_audio_rtf

Why median_* for latency/throughput and mean_* for quality: latency distributions have cold-start tails that drag the mean; quality aggregates over 200 prompts so single-request outliers don't matter.

Quality entries are enabled: false in CI because seed-tts-eval is not staged in the Buildkite container (matches the precedent in PR #2558 — quality runs are manual / release-validation, not nightly).

Concurrency cliff regression sentinel

Observed on H20-3e, Qwen3-TTS-1.7B (measured pre-merge on this branch):

Task	Model	c=1	c=4	c=8	c=16	c=32
voice_clone	1.7B-Base	RTF 0.15 / TTFP 165ms	0.28 / 412ms	0.49 / 1701ms	0.72 / 3355ms	0.77 / 3772ms
voice_design	1.7B-CustomVoice	RTF 0.08 / TTFP 53ms	0.11 / 154ms	0.21 / 872ms	0.33 / 1801ms	0.38 / 1989ms

Both models show a 4–6× TTFP jump from c=4 to c=8 while audio throughput saturates around c=4–8 — the codec-bs=1 bottleneck documented in vllm-project/vllm-omni#272. The throughput CI regime at c=8 is the sentinel for regressions in this area.

File layout

benchmarks/tts/
├── README.md                  (this file)
├── bench_tts.py               CLI — serve-mode benchmark driver
├── plot_results.py            Generate per-task / per-concurrency curves
└── model_configs.yaml         Model registry (supported tasks + extra body)

Related

• Upstream seed-tts-eval integration: vllm-project/vllm-omni#2558
• Pipeline + Deploy schema: vllm-project/vllm-omni#2383
• Concurrency cliff RFC: vllm-project/vllm-omni#272