vLLM

High-performance OpenAI-compatible API, production-ready and optimized for serving many model responses in parallel. Supports a wide range of models and quantization formats. May require more manual configuration and tuning than simpler inference servers.

LLM InferenceAdvanced

Versionlatest

LicenseApache 2.0

Python3.12

EnvironmentPython (uv)

InterfaceAPI

GPURequired

One-Click Install on Laboratory OS

GitHub

Securely deploy vLLM on Laboratory OS.

vLLM

Laboratory OS Installation Details

Environment

Working Directory/workspace/vllm

Python3.12

Env Manageruv

Environment Variables

APP_DIRECTORY

/workspace/vllm

OPENLAB_VLLM_PORT

OPENLAB_VLLM_MODELHugging Face model repo id or a local model path to serve.

cyankiwi/Qwen3.5-27B-AWQ-4bit

OPENLAB_VLLM_GPU_MEMORY_UTILIZATIONFraction of GPU memory vLLM should reserve for the model and KV cache.

0.8

OPENLAB_VLLM_MAX_MODEL_LENMaximum context length to allocate for the server.

32768

OPENLAB_VLLM_MAX_NUM_SEQSMaximum number of sequences to warm up and schedule at once.

256

OPENLAB_VLLM_ENABLE_AUTO_TOOL_CHOICEPass --enable-auto-tool-choice so compatible models can decide when to call tools.

—

OPENLAB_VLLM_TOOL_CALL_PARSERParser name to pass to --tool-call-parser when automatic tool choice is enabled. The right parser depends on the model family.

—

OPENLAB_VLLM_DOWNLOAD_DIR

${APP_DIRECTORY}/models

Install & Run

install-and-run.sh

#!/bin/bash
set -e

export PATH="/opt/conda/bin:/root/.local/bin:$PATH"

# Create the directory if it doesn't exist
mkdir -p "/workspace/vllm"
  
# Change to the app directory
cd "/workspace/vllm"

# Install the requested Python version and create a virtual environment when needed
uv python install 3.12
if [ ! -d ".venv" ]; then
  uv venv --seed --python 3.12 .venv
else
  echo "Virtual environment already exists at /workspace/vllm/.venv"
fi

# Activate the virtual environment
source .venv/bin/activate


# Environment setup complete
echo "Environment setup complete!"

uv pip install vllm --torch-backend=auto

mkdir -p "$OPENLAB_VLLM_DOWNLOAD_DIR" "$HF_HOME"
VLLM_ARGS=()

if [ -n "$OPENLAB_VLLM_GPU_MEMORY_UTILIZATION" ]; then
  VLLM_ARGS+=(--gpu-memory-utilization "$OPENLAB_VLLM_GPU_MEMORY_UTILIZATION")
fi

if [ -n "$OPENLAB_VLLM_MAX_MODEL_LEN" ]; then
  VLLM_ARGS+=(--max-model-len "$OPENLAB_VLLM_MAX_MODEL_LEN")
fi

if [ -n "$OPENLAB_VLLM_MAX_NUM_SEQS" ]; then
  VLLM_ARGS+=(--max-num-seqs "$OPENLAB_VLLM_MAX_NUM_SEQS")
fi

if [ -n "$OPENLAB_VLLM_ENABLE_AUTO_TOOL_CHOICE" ]; then
  VLLM_ARGS+=(--enable-auto-tool-choice)
fi

if [ -n "$OPENLAB_VLLM_TOOL_CALL_PARSER" ]; then
  VLLM_ARGS+=(--tool-call-parser "$OPENLAB_VLLM_TOOL_CALL_PARSER")
fi

exec vllm serve "$OPENLAB_VLLM_MODEL" --host 0.0.0.0 --port "$OPENLAB_VLLM_PORT" --download-dir "$OPENLAB_VLLM_DOWNLOAD_DIR" "${VLLM_ARGS[@]}" $OPENLAB_EXTRA_ARGS