SGLang

Performance-optimized LLM inference server, offering an OpenAI-compatible API and support for many model formats and quantization methods. Best used with large recent datacenter GPUs and may require more manual configuration and tuning than simpler inference servers.

LLM InferenceAdvanced

Versionlatest

LicenseApache 2.0

Python3.12

EnvironmentPython (uv)

InterfaceAPI

GPURequired

One-Click Install on Laboratory OS

GitHub

Securely deploy SGLang on Laboratory OS.

SGLang

Laboratory OS Installation Details

Environment

Git Repohttps://github.com/sgl-project/sglang.git

Working Directory/workspace/sglang

Python3.12

Env Manageruv

Environment Variables

APP_DIRECTORY

/workspace/sglang

OPENLAB_SGLANG_PORT

OPENLAB_SGLANG_MODELHugging Face model repo id or a local model path to serve.

cyankiwi/Qwen3.5-27B-AWQ-4bit

OPENLAB_SGLANG_MEM_FRACTION_STATICFraction of GPU memory SGLang should reserve for the model weights and KV cache.

0.8

OPENLAB_SGLANG_MAX_TOTAL_TOKENSMaximum total number of tokens across all sequences (model length).

32768

OPENLAB_SGLANG_MAX_RUNNING_REQUESTSMaximum number of requests to run concurrently.

OPENLAB_SGLANG_DOWNLOAD_DIR

${APP_DIRECTORY}/models

Install & Run

install-and-run.sh

#!/bin/bash
set -e

export PATH="/opt/conda/bin:/root/.local/bin:$PATH"

# Clone the GitHub repository into the directory when needed
mkdir -p "/workspace/sglang"
if [ -d "/workspace/sglang/.git" ]; then
  echo "Git repository already initialized in /workspace/sglang"
elif [ -z "$(ls -A "/workspace/sglang" 2>/dev/null)" ]; then
  git clone --progress "https://github.com/sgl-project/sglang.git" "/workspace/sglang"
else
  echo "Working directory /workspace/sglang already exists and is not a git repository."
  echo "Leaving existing files untouched and skipping clone of https://github.com/sgl-project/sglang.git."
fi
  
# Change to the app directory
cd "/workspace/sglang"

# Install the requested Python version and create a virtual environment when needed
uv python install 3.12
if [ ! -d ".venv" ]; then
  uv venv --seed --python 3.12 .venv
else
  echo "Virtual environment already exists at /workspace/sglang/.venv"
fi

# Activate the virtual environment
source .venv/bin/activate


# Environment setup complete
echo "Environment setup complete!"

uv pip install -e "python[all]"

mkdir -p "$OPENLAB_SGLANG_DOWNLOAD_DIR" "$HF_HOME"

# CUTLASS DSL (used by newer flashinfer) fails to detect GPU architecture on
# older GPUs (sm_75 and below), emitting an empty -arch=compute_ flag that
# causes NVVM to crash during CUDA graph capture. Force the legacy CUDA kernel
# path for norms, which works on all supported GPUs.
export FLASHINFER_USE_CUDA_NORM=1

# Detect GPU compute capability. On GPUs below sm_80 (pre-Ampere), bfloat16 is
# not natively supported. Sglang defaults the Mamba conv-state cache to bf16,
# which causes a Triton type mismatch (bf16 cache load vs fp16 input zeros) on
# those GPUs. Force float16 when the GPU can't do bf16.
CUDA_COMPUTE_CAP=$(python3 -c "import torch; c=torch.cuda.get_device_capability(); print(c[0]*10+c[1])" 2>/dev/null)
if [ -n "$CUDA_COMPUTE_CAP" ] && [ "$CUDA_COMPUTE_CAP" -lt 80 ]; then
  export SGLANG_MAMBA_CONV_DTYPE=float16
fi

SGLANG_ARGS=()

if [ -n "$OPENLAB_SGLANG_MEM_FRACTION_STATIC" ]; then
  SGLANG_ARGS+=(--mem-fraction-static "$OPENLAB_SGLANG_MEM_FRACTION_STATIC")
fi

if [ -n "$OPENLAB_SGLANG_MAX_TOTAL_TOKENS" ]; then
  SGLANG_ARGS+=(--max-total-tokens "$OPENLAB_SGLANG_MAX_TOTAL_TOKENS")
fi

if [ -n "$OPENLAB_SGLANG_MAX_RUNNING_REQUESTS" ]; then
  SGLANG_ARGS+=(--max-running-requests "$OPENLAB_SGLANG_MAX_RUNNING_REQUESTS")
fi

exec sglang serve --model-path "$OPENLAB_SGLANG_MODEL" --host 0.0.0.0 --port "$OPENLAB_SGLANG_PORT" --download-dir "$OPENLAB_SGLANG_DOWNLOAD_DIR" "${SGLANG_ARGS[@]}" $OPENLAB_EXTRA_ARGS