/recipes/Wan-AI/Wan2.2-S2V.md

Wan2.2 Speech-to-Video

Speech-to-video serving (Wan2.2 14B)

Summary

• Vendor: Wan-AI
• Model: Wan-AI/Wan2.2-S2V-14B
• Task: Speech-to-video generation (talking-head / singing from reference image + audio)
• Mode: Offline inference
• Maintainer: Community

When to use this recipe

Use this recipe when you want to generate talking-head or singing videos from a reference image and audio clip using the Wan2.2 S2V 14B model with vLLM-Omni. Two configurations are provided:

1. Multi-GPU with tensor parallelism — splits the DiT across GPUs for faster per-step latency.
2. Multi-GPU with tensor parallelism + CPU offload — enables generation on GPUs with limited VRAM by offloading idle model components to CPU.

References

• Upstream model card: https://huggingface.co/Wan-AI/Wan2.2-S2V-14B
• Example reference assets: https://github.com/Wan-Video/Wan2.2/tree/main/examples

Hardware Support

CUDA

2× NVIDIA A100/H100 (80 GB)

Environment

• OS: Linux
• Python: 3.10+
• Driver: NVIDIA driver with CUDA 12.x
• vLLM version: Match the repository requirements for your checkout
• vLLM-Omni version or commit: Use the commit you are deploying from

Prerequisites

None

Command

wget -O "Five Hundred Miles.png" \
  "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.png"
wget -O "Five Hundred Miles.MP3" \
  "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.MP3"

TP=2 (no CPU offload, requires ~60 GB VRAM per GPU):

VLLM_WORKER_MULTIPROC_METHOD=spawn \
python examples/offline_inference/speech_to_video/speech_to_video.py \
  --model Wan-AI/Wan2.2-S2V-14B \
  --image "Five Hundred Miles.png" \
  --audio "Five Hundred Miles.MP3" \
  --prompt "A person singing" \
  --height 720 --width 1280 --num-frames 81 \
  --num-inference-steps 40 \
  --fps 16 \
  --tensor-parallel-size 2 \
  --vae-use-slicing --vae-use-tiling \
  --output s2v_720p_tp2.mp4

TP=1 with CPU offload (reduces VRAM usage, adds ~12s overhead):

VLLM_WORKER_MULTIPROC_METHOD=spawn \
python examples/offline_inference/speech_to_video/speech_to_video.py \
  --model Wan-AI/Wan2.2-S2V-14B \
  --image "Five Hundred Miles.png" \
  --audio "Five Hundred Miles.MP3" \
  --prompt "A person singing" \
  --height 720 --width 1280 --num-frames 81 \
  --num-inference-steps 40 \
  --fps 16 \
  --tensor-parallel-size 1 \
  --enable-cpu-offload \
  --vae-use-slicing --vae-use-tiling \
  --output s2v_720p_tp1_offload.mp4

TP=1 * cfg=2 with CPU offload:

VLLM_WORKER_MULTIPROC_METHOD=spawn \
python examples/offline_inference/speech_to_video/speech_to_video.py \
  --model Wan-AI/Wan2.2-S2V-14B \
  --image "Five Hundred Miles.png" \
  --audio "Five Hundred Miles.MP3" \
  --prompt "A person singing" \
  --height 720 --width 1280 --num-frames 81 \
  --num-inference-steps 40 \
  --fps 16 \
  --tensor-parallel-size 1 \
  --enable-cpu-offload \
  --cfg-parallel-size 2 \
  --vae-use-slicing --vae-use-tiling \
  --output s2v_720p_tp1_cfg2_offload.mp4

XPU

4x or 8x Intel Arc B70 (32 GB)

Environment

• OS: Linux
• Python: 3.10+
• torch: 2.11.0+xpu
• triton-xpu: 3.7.0
• vLLM version: Match the repository requirements for your checkout
• vLLM-Omni version or commit: Use the commit you are deploying from

Prerequisites

None

Command

wget -O "Five Hundred Miles.png" \
  "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.png"
wget -O "Five Hundred Miles.MP3" \
  "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.MP3"

TP=4:

VLLM_WORKER_MULTIPROC_METHOD=spawn \
python examples/offline_inference/speech_to_video/speech_to_video.py \
  --model Wan-AI/Wan2.2-S2V-14B \
  --image "Five Hundred Miles.png" \
  --audio "Five Hundred Miles.MP3" \
  --prompt "A person singing" \
  --height 720 --width 1280 --num-frames 81 \
  --num-inference-steps 40 \
  --fps 16 \
  --tensor-parallel-size 4 \
  --vae-use-slicing --vae-use-tiling \
  --output s2v_720p_tp4.mp4

TP=4 cfg=2:

VLLM_WORKER_MULTIPROC_METHOD=spawn \
python examples/offline_inference/speech_to_video/speech_to_video.py \
  --model Wan-AI/Wan2.2-S2V-14B \
  --image "Five Hundred Miles.png" \
  --audio "Five Hundred Miles.MP3" \
  --prompt "A person singing" \
  --height 720 --width 1280 --num-frames 81 \
  --num-inference-steps 40 \
  --fps 16 \
  --tensor-parallel-size 4 \
  --cfg-parallel-size 2 \
  --vae-use-slicing --vae-use-tiling \
  --output s2v_720p_tp4_cfg2_offload.mp4

Online Serving

Server

vllm serve Wan-AI/Wan2.2-S2V-14B --omni \
  --model-class-name WanS2VPipeline \
  --tensor-parallel-size 2 \
  --flow-shift 3.0 \
  --vae-use-slicing --vae-use-tiling \
  --port 8091

Client

no_proxy=127.0.0.1 \
curl -X POST http://127.0.0.1:8091/v1/videos/sync \
  -F "prompt=A person singing" \
  -F 'image_reference={"image_url": "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.png"}' \
  -F 'audio_reference={"audio_url": "https://raw.githubusercontent.com/Wan-Video/Wan2.2/main/examples/Five%20Hundred%20Miles.MP3"}' \
  -F "width=832" -F "height=480" \
  -F "num_inference_steps=40" \
  -F "guidance_scale=4.5" \
  -F "fps=16" \
  --output s2v_480p_serve.mp4

Verification

The output video should show a talking/singing person matching the reference image with lip movements synchronized to the audio. Check:

1. Video file is generated at the specified output path.
2. Audio is muxed into the output MP4.
3. Lip sync quality is reasonable for the given inference steps.

Notes

• Key flags:
  • --tensor-parallel-size <N> — splits the DiT across N GPUs for TP.
  • --cfg-parallel-size <N> - splits the conditioned / unconditioned pred to 2 GPUs run parallel.
  • --enable-cpu-offload — enables model-level CPU offloading (transformer and text_encoder alternate on GPU). Adds ~12s overhead but reduces VRAM.
  • --vae-use-slicing --vae-use-tiling — reduces VAE peak memory by processing frames in slices and spatial tiles.
  • --num-frames 81 — generates ~5s of video at 16 fps.
  • --num-inference-steps — quality/speed tradeoff. 4 steps for fast preview, 40 steps for production quality.
• Performance tips:
  • Use --enforce-eager to skip torch.compile if you encounter recompilation issues or want deterministic profiling.
  • For profiling, add --enable-diffusion-pipeline-profiler for stage-level timing, or --profiler-config '{"profiler":"torch","torch_profiler_dir":"./perf","torch_profiler_record_shapes":true,"torch_profiler_with_stack":false}' for detailed op-level torch profiler traces.
  • The first clip in multi-clip generation includes warmup overhead; subsequent clips run at steady-state speed.
• Known limitations:
  • VLLM_WORKER_MULTIPROC_METHOD=spawn is required for multi-GPU offline inference to avoid CUDA context issues with forked processes.
  • CPU offload adds ~12s latency for model-level shuttling between CPU and GPU.
  • S2V self-attention is ~8.5× more expensive than T2V due to audio-visual conditioning fused into self-attn blocks.