vllm-project--vllm-omni
wehub-resource-sync
92 行
3.0 KiB
Markdown
92 行
3.0 KiB
Markdown
# Frame Interpolation
|
|
|
|
## Overview
|
|
|
|
vLLM-Omni supports post-generation frame interpolation for supported video
|
|
diffusion pipelines. This feature inserts synthesized intermediate frames
|
|
between adjacent generated frames to improve temporal smoothness without
|
|
rerunning the diffusion denoising loop.
|
|
|
|
Frame interpolation runs in the diffusion worker post-processing path instead
|
|
of the API server encoding path. This allows the interpolation step to reuse
|
|
the worker's current accelerator device and keeps the FastAPI event loop free
|
|
from heavy synchronous PyTorch work.
|
|
|
|
For an input video with `N` generated frames and interpolation exponent `exp`,
|
|
the output frame count is:
|
|
|
|
```text
|
|
(N - 1) * 2**exp + 1
|
|
```
|
|
|
|
The output FPS is multiplied by `2**exp` so the clip duration remains close to
|
|
the original generated video.
|
|
|
|
## Supported Pipelines
|
|
|
|
Frame interpolation is currently supported for:
|
|
|
|
- `WanPipeline` (Wan2.2 text-to-video)
|
|
- `WanImageToVideoPipeline`
|
|
|
|
## Request Parameters
|
|
|
|
The video APIs `/v1/videos` and `/v1/videos/sync` accept:
|
|
|
|
| Parameter | Type | Default | Description |
|
|
|-----------|------|---------|-------------|
|
|
| `enable_frame_interpolation` | bool | `false` | Enable post-generation frame interpolation |
|
|
| `frame_interpolation_exp` | int | `1` | Interpolation exponent. `1=2x`, `2=4x`, etc. |
|
|
| `frame_interpolation_scale` | float | `1.0` | RIFE inference scale |
|
|
| `frame_interpolation_model_path` | str | `None` | Local directory or Hugging Face repo ID containing `flownet.pkl` |
|
|
|
|
## Execution Flow
|
|
|
|
For supported Wan2.2 pipelines, the execution order is:
|
|
|
|
1. Diffusion worker finishes denoising and decodes the raw video tensor.
|
|
2. Worker-side model-specific post-processing runs.
|
|
3. If frame interpolation is enabled, RIFE interpolates the decoded video
|
|
tensor on the worker side and records a FPS multiplier in `custom_output`.
|
|
4. The API server receives the already-interpolated video and only performs
|
|
MP4 export.
|
|
|
|
This design keeps interpolation close to the generated tensor and avoids
|
|
introducing another heavyweight GPU context in the API server process.
|
|
|
|
## Example
|
|
|
|
Start the server:
|
|
|
|
```bash
|
|
vllm serve Wan-AI/Wan2.2-T2V-A14B-Diffusers --omni --port 8091
|
|
```
|
|
|
|
Run a sync request with interpolation enabled:
|
|
|
|
```bash
|
|
curl -X POST http://localhost:8091/v1/videos/sync \
|
|
-F "prompt=A dog running through a park" \
|
|
-F "num_frames=81" \
|
|
-F "width=832" \
|
|
-F "height=480" \
|
|
-F "fps=16" \
|
|
-F "num_inference_steps=40" \
|
|
-F "guidance_scale=1.0" \
|
|
-F "guidance_scale_2=1.0" \
|
|
-F "enable_frame_interpolation=true" \
|
|
-F "frame_interpolation_exp=1" \
|
|
-F "frame_interpolation_scale=1.0" \
|
|
-F "seed=42" \
|
|
-o sync_t2v_interpolated.mp4
|
|
```
|
|
|
|
## Notes
|
|
|
|
- This is a post-processing feature. It does not modify the diffusion denoising
|
|
schedule.
|
|
- Higher interpolation exponents increase post-processing time and memory usage.
|
|
- If the interpolation model weights are not available locally,
|
|
`frame_interpolation_model_path` may point to a Hugging Face repo containing
|
|
`flownet.pkl`.
|