Skip to main content
App Library
ggml-org /

llama.cpp

GGUF model inference with a polished web UI and OpenAI-format API. CPU-only build — high hardware compatibility, works on any machine without a GPU.

LLM InferenceChat UIsAdvanced
Versionlatest
LicenseMIT License
InterfaceWeb UI + API
GPUOptional
One-Click Install on Laboratory OS
GitHub
Securely deploy llama.cpp on Laboratory OS.

Compatible Models

Moonshot AI /

Kimi K2

Google /

Gemma 3n E2B

Google /

Gemma 3n E4B

Mistral AI /

Mistral Small 3.2 (2506)

TheDrummer /

Anubis 70B v1.1

Deepseek AI /

DeepSeek R1 (0528)

View all

llama.cpp

Laboratory OS Installation Details

Environment

Git Repohttps://github.com/ggml-org/llama.cpp.git
Working Directory/workspace/llama.cpp

Environment Variables

APP_DIRECTORY
/workspace/llama.cpp
OPENLAB_LLAMACPP_PORT
<assigned at runtime>
OPENLAB_LLAMACPP_MODELHuggingFace model repo to load via -hf, e.g. unsloth/Qwen3.5-27B-GGUF.
unsloth/Qwen3.5-0.8B-GGUF
OPENLAB_LLAMACPP_GENERATION_PARAMSGeneration parameters to pass to llama.cpp.
--repeat-penalty 1.0 --presence-penalty 1.5 --min-p 0.0 --top-k 20 --top-p 0.8 --temp 0.7
OPENLAB_LLAMACPP_REASONING_BUDGETCaps how many tokens a reasoning-capable model can spend thinking. Leave blank to use the model default, or set 0 to disable reasoning entirely.
—
OPENLAB_LLAMACPP_CONTEXT_SIZEMaximum context window to allocate for the server and CLI.
32768
LLAMA_ARG_CACHE_TYPE_KControls the precision of the attention key cache. Lower-precision K cache uses less RAM/VRAM and can make longer contexts fit, but may slightly reduce attention accuracy.
f16
LLAMA_ARG_CACHE_TYPE_VControls the precision of the attention value cache. Lower-precision V cache uses less RAM/VRAM and can make longer contexts fit, but may have a more noticeable effect on output quality.
f16
OPENLAB_LLAMACPP_PARALLEL_REQUESTSMaximum concurrent request slots for the OpenAI-compatible server.
1

Install & Run

install-and-run.sh
#!/bin/bash
set -e

export PATH="/opt/conda/bin:/root/.local/bin:$PATH"

# Clone the GitHub repository into the directory when needed
mkdir -p "/workspace/llama.cpp"
if [ -d "/workspace/llama.cpp/.git" ]; then
  echo "Git repository already initialized in /workspace/llama.cpp"
elif [ -z "$(ls -A "/workspace/llama.cpp" 2>/dev/null)" ]; then
  git clone --progress "https://github.com/ggml-org/llama.cpp.git" "/workspace/llama.cpp"
else
  echo "Working directory /workspace/llama.cpp already exists and is not a git repository."
  echo "Leaving existing files untouched and skipping clone of https://github.com/ggml-org/llama.cpp.git."
fi
  
# Change to the app directory
cd "/workspace/llama.cpp"


# Environment setup complete
echo "Environment setup complete!"

cmake -B /workspace/llama.cpp/build -S /workspace/llama.cpp -DGGML_NATIVE=OFF
cmake --build /workspace/llama.cpp/build --config Release -j$(nproc)

exec /workspace/llama.cpp/build/bin/llama-server --host 0.0.0.0 --port "$OPENLAB_LLAMACPP_PORT" --ctx-size "$OPENLAB_LLAMACPP_CONTEXT_SIZE" --parallel "$OPENLAB_LLAMACPP_PARALLEL_REQUESTS" -hf "$OPENLAB_LLAMACPP_MODEL" $OPENLAB_LLAMACPP_GENERATION_PARAMS ${OPENLAB_LLAMACPP_REASONING_BUDGET:+--reasoning-budget "$OPENLAB_LLAMACPP_REASONING_BUDGET"} $OPENLAB_EXTRA_ARGS

CLI Command

cli.sh
# llama-cli
exec /workspace/llama.cpp/build/bin/llama-cli \
--ctx-size "$OPENLAB_LLAMACPP_CONTEXT_SIZE" \
-cnv \
-hf "$OPENLAB_LLAMACPP_MODEL" $OPENLAB_LLAMACPP_GENERATION_PARAMS ${OPENLAB_LLAMACPP_REASONING_BUDGET:+--reasoning-budget "$OPENLAB_LLAMACPP_REASONING_BUDGET"} $OPENLAB_EXTRA_ARGS