llama.cpp (CUDA)

GGUF model inference with a polished web UI and OpenAI-format API. CUDA build — GPU-accelerated for NVIDIA GPUs.

LLM InferenceChat UIsAdvanced

Versionlatest

LicenseMIT License

InterfaceWeb UI + API

GPURequired

One-Click Install on Laboratory OS

GitHub

Securely deploy llama.cpp (CUDA) on Laboratory OS.

Compatible Models

Moonshot AI /

llama.cpp (CUDA)

Laboratory OS Installation Details

Environment

Git Repohttps://github.com/ggml-org/llama.cpp.git

Working Directory/workspace/llama.cpp

Environment Variables

APP_DIRECTORY

/workspace/llama.cpp

OPENLAB_LLAMACPP_PORT

OPENLAB_LLAMACPP_MODELHuggingFace model repo to load via -hf, e.g. unsloth/Qwen3.5-27B-GGUF.

unsloth/Qwen3.5-0.8B-GGUF

OPENLAB_LLAMACPP_GENERATION_PARAMSGeneration parameters to pass to llama.cpp.

--repeat-penalty 1.0 --presence-penalty 1.5 --min-p 0.0 --top-k 20 --top-p 0.8 --temp 0.7

OPENLAB_LLAMACPP_REASONING_BUDGETCaps how many tokens a reasoning-capable model can spend thinking. Leave blank to use the model default, or set 0 to disable reasoning entirely.

—

OPENLAB_LLAMACPP_CONTEXT_SIZEMaximum context window to allocate for the server and CLI.

32768

LLAMA_ARG_CACHE_TYPE_KControls the precision of the attention key cache. Lower-precision K cache uses less RAM/VRAM and can make longer contexts fit, but may slightly reduce attention accuracy.

f16

LLAMA_ARG_CACHE_TYPE_VControls the precision of the attention value cache. Lower-precision V cache uses less RAM/VRAM and can make longer contexts fit, but may have a more noticeable effect on output quality.

f16

OPENLAB_LLAMACPP_PARALLEL_REQUESTSMaximum concurrent request slots for the OpenAI-compatible server.

Install & Run

install-and-run.sh

#!/bin/bash
set -e

export PATH="/opt/conda/bin:/root/.local/bin:$PATH"

# Clone the GitHub repository into the directory when needed
mkdir -p "/workspace/llama.cpp"
if [ -d "/workspace/llama.cpp/.git" ]; then
  echo "Git repository already initialized in /workspace/llama.cpp"
elif [ -z "$(ls -A "/workspace/llama.cpp" 2>/dev/null)" ]; then
  git clone --progress "https://github.com/ggml-org/llama.cpp.git" "/workspace/llama.cpp"
else
  echo "Working directory /workspace/llama.cpp already exists and is not a git repository."
  echo "Leaving existing files untouched and skipping clone of https://github.com/ggml-org/llama.cpp.git."
fi
  
# Change to the app directory
cd "/workspace/llama.cpp"


# Environment setup complete
echo "Environment setup complete!"

cmake -B /workspace/llama.cpp/build -S /workspace/llama.cpp -DGGML_NATIVE=OFF -DGGML_CUDA=ON
cmake --build /workspace/llama.cpp/build --config Release -j$(nproc)

exec /workspace/llama.cpp/build/bin/llama-server --host 0.0.0.0 --port "$OPENLAB_LLAMACPP_PORT" --ctx-size "$OPENLAB_LLAMACPP_CONTEXT_SIZE" --parallel "$OPENLAB_LLAMACPP_PARALLEL_REQUESTS" -hf "$OPENLAB_LLAMACPP_MODEL" $OPENLAB_LLAMACPP_GENERATION_PARAMS ${OPENLAB_LLAMACPP_REASONING_BUDGET:+--reasoning-budget "$OPENLAB_LLAMACPP_REASONING_BUDGET"} $OPENLAB_EXTRA_ARGS

CLI Command

cli.sh

# llama-cli
exec /workspace/llama.cpp/build/bin/llama-cli \
--ctx-size "$OPENLAB_LLAMACPP_CONTEXT_SIZE" \
-cnv \
-hf "$OPENLAB_LLAMACPP_MODEL" $OPENLAB_LLAMACPP_GENERATION_PARAMS ${OPENLAB_LLAMACPP_REASONING_BUDGET:+--reasoning-budget "$OPENLAB_LLAMACPP_REASONING_BUDGET"} $OPENLAB_EXTRA_ARGS

llama.cpp (CUDA)

Kimi K2

Gemma 3n E2B

Gemma 3n E4B

Mistral Small 3.2 (2506)

Anubis 70B v1.1

DeepSeek R1 (0528)