mirrors / ray-project--ray

项目文件夹

文件

T

wehub-resource-sync f1825c8ceb chore: import upstream snapshot with attribution

2026-07-13 13:17:40 +08:00

4.8 KiB

原始文件 永久链接 Blame 文件历史

(sglang-integration)=

SGLang integration

Ray Serve LLM provides an OpenAI-compatible API that integrates with SGLang via the server_cls parameter on LLMConfig. Most engine_kwargs that work with sglang serve also work here, giving you SGLang's feature set through Ray Serve's distributed deployment capabilities.

The integration uses SGLangServer, a custom server class that wraps SGLang's in-process engine and exposes chat, completions, embeddings, tokenize, and detokenize endpoints through the standard Ray Serve LLM protocol.

This compatibility means you can:

Use SGLang's RadixAttention and other optimizations with Ray Serve's production features
Deploy SGLang models with autoscaling, multi-model serving, and advanced routing
Serve models across multiple nodes with tensor and pipeline parallelism

:::{note} Community SGLang support is in early development. Track progress and provide feedback at ray-project/ray#61114. :::

Prerequisites

pip install "ray[llm]" "sglang[all,ray]"

Set the following environment variable before running any example:

CUDA: RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=0
ROCm: RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES=0

Online serving (single node)

Deploy a single-node SGLang model with autoscaling. The server_cls parameter tells Ray Serve LLM to use the SGLangServer instead of the default vLLM engine.

::::{tab-set}

:::{tab-item} Server :sync: server

:language: python
:start-after: __sglang_single_node_start__
:end-before: __sglang_single_node_end__

:::

:::{tab-item} Python Client :sync: client

:language: python
:start-after: __sglang_query_start__
:end-before: __sglang_query_end__

:::

:::{tab-item} cURL :sync: curl

# Chat completions
curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Llama-3.1-8B-Instruct",
        "messages": [{"role": "user", "content": "List 3 countries and their capitals."}],
        "temperature": 0,
        "max_tokens": 64
    }'

# Text completions
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "Llama-3.1-8B-Instruct",
        "prompt": "San Francisco is a",
        "max_tokens": 30,
        "temperature": 0
    }'

:::

::::

Run:

RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=0 serve run serve_sglang_example:app

Online serving (multi-node with TP+PP)

Deploy a large model across multiple nodes using tensor parallelism (TP=4) and pipeline parallelism (PP=2). This requires 2 nodes with 4 GPUs each (8 GPUs total).

The placement_group_strategy: "PACK" fills GPUs on each node before moving to the next, so with 2 nodes (4 GPUs each) each node gets one pipeline stage. The SGLangServer.get_deployment_options() method constructs placement groups from the placement_group_config.

::::{tab-set}

:::{tab-item} Python :sync: python

:language: python
:start-after: __sglang_multinode_start__
:end-before: __sglang_multinode_end__

:::

::::

Run:

RAY_EXPERIMENTAL_NOSET_CUDA_VISIBLE_DEVICES=0 serve run serve_sglang_multinode_example:app

Limitations

The following SGLang features are available upstream but not yet integrated into Ray Serve LLM. Community contributions are welcome:

Engine replicas: Multiple engine replicas within a single deployment. See ray-project/ray#62480.
Observability: Engine-level metrics (e.g. KV cache utilization, request queue depth).
Prefill disaggregation: Separating prefill and decode phases across different workers.
Wide EP: Wide expert parallelism for Mixture-of-Experts models.
Elastic EP: Fault-tolerant expert parallelism with dynamic rank health tracking.
Transcriptions and score: The /v1/audio/transcriptions and /v1/score endpoints.

Dependencies

SGLang's in-process engine overrides Python signal handlers on startup. The SGLangServer.__init__ includes a workaround that saves and restores signal handlers around engine initialization. If you encounter issues with graceful shutdown, this is a known area of friction.

4.8 KiB 原始文件 永久链接 Blame 文件历史