vLLM — High-Throughput LLM Serving

vLLM is an open-source LLM inference engine developed at UC Berkeley, optimized for maximum GPU throughput in production environments. It introduced PagedAttention — a technique that manages the KV cache like OS virtual memory, dramatically reducing GPU memory fragmentation and enabling much higher throughput than naive implementations. vLLM exposes an OpenAI-compatible REST API, making it a drop-in replacement for the OpenAI API in any existing application.

Quantization reduces model weight precision to lower VRAM requirements at a small quality cost. vLLM supports loading pre-quantized models from HuggingFace — no quantization step is needed on your end. Simply pass the --quantization flag matching the model's format.

cURL Examples

# Chat completions
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.2-3B-Instruct",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain PagedAttention in simple terms."}
    ],
    "temperature": 0.7,
    "max_tokens": 512
  }'

# Streaming response
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.2-3B-Instruct","messages":[{"role":"user","content":"Count to 10"}],"stream":true}'

# List available models
curl http://localhost:8000/v1/models

Python SDK — Drop-in OpenAI Replacement

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="none",  # or your --api-key value
)

# Streaming chat
stream = client.chat.completions.create(
    model="meta-llama/Llama-3.2-3B-Instruct",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is tensor parallelism?"},
    ],
    stream=True,
    temperature=0.7,
    max_tokens=256,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()

Tensor parallelism splits each transformer layer across N GPUs. Requires N GPUs of the same type on one machine. Enable with --tensor-parallel-size N:

# 2-GPU setup
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --port 8000

• For NCCL communication between GPUs, ensure --ipc=host in Docker
• Pipeline parallelism (across multiple nodes) is also supported but requires additional configuration