/recipes/Tencent/Covo-Audio-Chat.md

Covo-Audio-Chat

Online serving for audio chat

Summary

• Vendor: Tencent
• Model: tencent/Covo-Audio-Chat
• Task: Audio chat — audio (and optional text) input, interleaved text + audio output
• Mode: Online serving with the OpenAI-compatible API (offline inference also supported)
• Maintainer: Community

When to use this recipe

Use this recipe when you want a known-good starting point for serving tencent/Covo-Audio-Chat with vLLM-Omni on a single 80 GB-class GPU and validate the deployment with the bundled client example.

References

• Related example under examples/: examples/online_serving/covo_audio/README.md
• Offline end-to-end script: examples/offline_inference/covo_audio/end2end.py
• Deploy config: vllm_omni/deploy/covo_audio.yaml
• Pipeline topology: vllm_omni/model_executor/models/covo_audio/pipeline.py

Hardware Support

This recipe currently documents one tested reference configuration for CUDA GPU serving. The fused thinker+talker stage and the BigVGAN code2wav stage both run on the same physical device.

GPU

1x A100 80GB

Environment

• OS: Linux
• Python: 3.12
• Driver / runtime: NVIDIA CUDA 12.x with an A100 80 GB GPU
• vLLM version: Match the repository requirements for your checkout
• vLLM-Omni version or commit: Use the commit you are deploying from
• Extra Python deps: torchdiffeq, librosa

Command

Install the two runtime-only deps that are not pulled in by vLLM-Omni itself:

pip install torchdiffeq librosa

Start the server from the repository root:

CUDA_VISIBLE_DEVICES=0 vllm serve tencent/Covo-Audio-Chat --omni \
    --trust-remote-code --port 18091

The deploy config at vllm_omni/deploy/covo_audio.yaml is loaded automatically by the model registry.

Verification

With the server ready, send a test request via the bundled client:

cd examples/online_serving/covo_audio
python openai_chat_completion_client.py

Expected: the client prints the generated text and writes an output WAV file to the current directory.

Notes

• Memory usage: Stage 0 (7B LLM) ~16 GiB + stage 1 (BigVGAN vocoder) ~2 GiB, both resident on the same device.
• Key flags: --omni and --trust-remote-code are both required.
• System prompt: the bundled openai_chat_completion_client.py and end2end.py already include COVO_AUDIO_SYSTEM_PROMPT (defined in vllm_omni/model_executor/models/covo_audio/prompt_utils.py) in the system message by default. When integrating your own client, make sure to include it — without it, stage 0 emits text-only tokens and the output audio is silent.
• Stage 1 runs with enforce_eager: true and dtype: float32 — see the header of deploy/covo_audio.yaml for the specific reasons.
• Known limitations: this starter recipe covers the single-GPU online-serving path only; multi-GPU layouts and platform-specific overrides are not documented here.