Deploy an Open-Source LLM on Runpod Serverless with vLLM (2026)
In this article
This guide is adapted from Runpod's official worker-vllm repository (MIT licence) and its vLLM get-started documentation. Commands and code are reproduced from those sources; where the upstream example has a bug, we say so and show the fix separately.
Disclosure: links to Runpod marked as such are affiliate links. If you sign up through them, AI Mastery may earn a commission at no cost to you. It does not change what this guide recommends, and every step works the same with a direct signup.
Running an open-weight model behind an API used to mean renting a GPU server, installing an inference engine, writing a web service around it and keeping all of it alive. Runpod Serverless collapses that into a configured endpoint: you choose a Hugging Face model, Runpod starts GPU workers running vLLM when requests arrive, and you call the result with the same OpenAI client code you already use.
This guide deploys a model with Runpod's official vLLM worker, tests it through both of the APIs the endpoint exposes, configures it with environment variables, and covers the failure modes that cost money when you get them wrong. It suits teams that want an OpenAI-compatible endpoint for an open model without operating servers, and anyone weighing serverless GPUs against a self-managed deployment like the one in our DeepSeek V4 self-hosting guide.
Prerequisites
- A Runpod account (sign up for Runpod, affiliate link) and a Runpod API key, created under your account's API key settings.
- A Hugging Face access token, only if your model is gated or private.
- A model that fits a GPU you can rent. Weights plus KV cache must fit in VRAM. Our VRAM calculator estimates this from parameter count, precision and context length, and our GPU cloud pricing table compares current hourly rates, including Runpod's.
Step 1: Choose a model
The vLLM worker serves most models on Hugging Face that vLLM supports. Runpod's documentation lists examples including meta-llama/Llama-3.2-3B-Instruct, Qwen/Qwen3-8B, google/gemma-3-1b-it, microsoft/Phi-4-mini-instruct and deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, and its tutorial uses openchat/openchat-3.5-0106.
Two properties decide whether a model will work. It must have an architecture vLLM supports, and, for chat requests, a chat template. Instruction-tuned models ship one; a base model needs one supplied through CUSTOM_CHAT_TEMPLATE. Start with a small instruct model for your first deployment: you will see the whole flow work before paying for a large GPU.
Step 2: Deploy from the Runpod Hub
Runpod's get-started guide deploys the worker from the Hub:
- Find the vLLM repo (
runpod-workers/worker-vllm) in the Runpod Hub. - Click Deploy, using the latest vLLM worker version.
- In the Model field, enter the model name, for example
openchat/openchat-3.5-0106. - Click Advanced to expand the vLLM settings.
- Set Max Model Length to
8192, or a context length appropriate for your model. - Leave the other settings at their defaults unless you need otherwise, then click Next.
- Click Create Endpoint.
The endpoint then initialises, which can take several minutes while Runpod provisions a worker and downloads the model. If you prefer to build from the raw image instead of the Hub, it is published as runpod/worker-v1-vllm:<version>, with versions listed on the repository's releases page.
Max Model Length is the setting to think about. It becomes vLLM's --max-model-len, and vLLM reserves KV-cache memory for it. A long context on a small GPU is the most common reason a worker never starts.
Step 3: Find your endpoint ID
When deployment completes, copy the Endpoint ID from the endpoint's page. Every API call below needs it, together with your Runpod API key.
Step 4: Test with Runpod's native API
The endpoint's Requests tab sends a default test request:
{
"input": {
"prompt": "Hello World"
}
}
From your own machine, the same request goes to the synchronous runsync route. This is Runpod's command, with your endpoint ID and API key substituted:
curl -X POST "https://api.runpod.ai/v2/YOUR_ENDPOINT_ID/runsync" \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"input": {"prompt": "Hello World"}}'
The response wraps the model output in Runpod's job envelope. Runpod's documentation shows this shape:
{
"delayTime": 638,
"executionTime": 3344,
"id": "f0706ead-c5ec-4689-937c-e21d5fbbca47-u1",
"output": [
{
"choices": [
{
"tokens": ["CHAT_RESPONSE"]
}
],
"usage": {
"input": 3,
"output": 100
}
}
],
"status": "COMPLETED",
"workerId": "0e7o8fgmm9xgty"
}
delayTime is how long the job waited for a worker, and executionTime is how long it ran. A large delayTime on the first request is the cold start: a worker had to be started and the model loaded. Expect it on the first call after the endpoint has been idle.
Step 5: Call it with the OpenAI client
The worker also serves OpenAI-compatible routes under /openai/v1: Chat Completions, Models, Responses and Messages, streaming and non-streaming. Existing OpenAI code needs three changes: the API key, the base URL and the model name. Runpod's example client:
from openai import OpenAI
import os
# Initialize the OpenAI Client with your Runpod API Key and Endpoint URL
client = OpenAI(
api_key=os.environ.get("RUNPOD_API_KEY"),
base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1",
)
A non-streaming chat completion, as published:
# Create a chat completion
response = client.chat.completions.create(
model="<YOUR DEPLOYED MODEL REPO/NAME>",
messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
temperature=0,
max_tokens=100,
)
# Print the response
print(response.choices[0].message.content)
The model value is the Hugging Face repo you deployed, such as openchat/openchat-3.5-0106, unless you set OPENAI_SERVED_MODEL_NAME_OVERRIDE. If you are not sure what name the endpoint expects, ask it:
models_response = client.models.list()
list_of_models = [model.id for model in models_response]
print(list_of_models)
The upstream streaming example has a bug. In the repository's README, the loop variable and the variable it prints do not match, so it raises NameError: name 'chunk' is not defined:
# Create a chat completion stream
response_stream = client.chat.completions.create(
model="<YOUR DEPLOYED MODEL REPO/NAME>",
messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
temperature=0,
max_tokens=100,
stream=True,
)
# Stream the response
for response in response_stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)
Our fix, not Runpod's code: name the loop variable chunk so it matches the line that uses it.
for chunk in response_stream:
print(chunk.choices[0].delta.content or "", end="", flush=True)
If you call the OpenAI route with curl, note that the README's "after" example still shows <YOUR OPENAI API KEY> in the Authorization header. The endpoint expects your Runpod API key there, exactly as in the runsync call above.
Step 6: Configure the worker with environment variables
Edit the endpoint (Manage, then Edit Endpoint, then Public Environment Variables) to change how vLLM runs. The variables that matter most:
| Variable | What it controls | Default | When to change it |
|---|---|---|---|
MODEL_NAME | Hugging Face repo ID or local path of the weights | (none) | Always set; this is the model you serve |
HF_TOKEN | Access token for gated or private models | (none) | Gated models such as Llama variants |
MAX_MODEL_LEN | Maximum context length (--max-model-len) | (model's own) | Lower it when the worker runs out of memory at startup |
GPU_MEMORY_UTILIZATION | Fraction of VRAM vLLM may use | 0.9 | Raise slightly for more KV cache, lower if something else shares the GPU |
MAX_NUM_SEQS | Sequences per scheduler iteration | 256 | Lower it to cut memory use; raise it for throughput |
TENSOR_PARALLEL_SIZE | Number of GPUs to split the model across | 1 | Models that do not fit one GPU |
QUANTIZATION | Quantization method | (none) | Serving an AWQ or GPTQ checkpoint |
MAX_CONCURRENCY | Concurrent requests per worker | 30 | Tune with load testing |
ENABLE_AUTO_TOOL_CHOICE / TOOL_CALL_PARSER | Tool calling and the parser for the model family | false / (none) | Agents that use function calling |
Beyond this list, the worker translates any environment variable whose name is the uppercased form of a vllm serve flag into that flag, so ENFORCE_EAGER=true becomes --enforce-eager. For flags it does not recognise, VLLM_EXTRA_ARGS is appended to the vllm serve command and overrides earlier settings. Runpod's example:
VLLM_EXTRA_ARGS="--override-generation-config '{\"max_new_tokens\": 512}' --uvicorn-log-level warning"
You can also mount a complete vllm serve config file and point VLLM_CONFIG_FILE at it. Environment variables still take precedence over values in the file.
When the worker will not start
Some startup failures cannot be fixed by restarting: the model does not fit the GPU, MAX_MODEL_LEN needs more KV cache than the GPU holds, vLLM rejects a flag, or a gated model has no HF_TOKEN. If the worker simply exited on these, it would crash-loop: every restart pays again to download and load the model, while the console shows only silent restarts.
The current worker recognises these cases, stays up, and answers every job with the cause and the fix. Runpod's README shows an out-of-memory example:
{
"error": {
"message": "Qwen/Qwen3-30B-A3B ran out of GPU memory during startup. This GPU has 19.57 GiB. Lower MAX_MODEL_LEN or MAX_NUM_SEQS, set ENFORCE_EAGER=true to skip CUDA graph capture, use a quantized checkpoint, or redeploy on a larger GPU (or more GPUs with TENSOR_PARALLEL_SIZE). ...",
"type": "startup_error",
"code": null
}
}
If your first test request returns a startup_error, read the message before changing anything: it names the remedy. The full vLLM traceback is in the worker logs. Errors the worker does not recognise, such as a failed download, still exit and retry, because those can succeed on another attempt.
What to watch out for
Cold starts cost time and money. A new worker downloads the model and loads it before it can answer. Large models make that slow, and any configuration that crash-loops makes it expensive. Confirm your settings with a small model first.
Size the context to the GPU, not the model card. A model that advertises a 128K context will not get it on a GPU with room for 8K of KV cache. Set MAX_MODEL_LEN to what your workload needs, and check the arithmetic with the VRAM calculator.
The model name must match. OpenAI-route requests must use the deployed repo name, or your OPENAI_SERVED_MODEL_NAME_OVERRIDE. When in doubt, call client.models.list().
Serverless is not always cheapest. Per-second serverless billing suits bursty or low-volume traffic. For steady high utilisation, a dedicated GPU instance or a self-managed vLLM deployment can cost less per token. Compare current hourly rates in our GPU cloud pricing table against your expected load before committing.
Where to go next
For serving a frontier-scale open model across several GPUs yourself, our DeepSeek V4 vLLM deployment guide covers tensor parallelism and cluster sizing. To run models on your own hardware instead, start with the complete guide to running LLMs locally. And if your application already uses the OpenAI SDK, the endpoint you just deployed is a drop-in replacement: change the key, the base URL and the model name.
Frequently asked questions
How do I call a Runpod vLLM endpoint with the OpenAI Python SDK?
Create the client with api_key set to your Runpod API key and base_url set to https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1, then pass your deployed model's Hugging Face repo name as the model parameter. The worker serves Chat Completions, Models, Responses and Messages routes.
What does MAX_MODEL_LEN do on a Runpod vLLM worker, and why does it cause startup failures?
It sets the model's maximum context length (vLLM's --max-model-len). If it is larger than the KV cache the GPU can hold, vLLM cannot start. The current worker detects this and answers each job with a startup_error explaining the cause, instead of silently restarting; lower MAX_MODEL_LEN or MAX_NUM_SEQS, or use a larger GPU.
How do I pass a vLLM flag that the Runpod worker does not list?
Any environment variable named as the uppercased form of a vllm serve flag is applied automatically, for example ENFORCE_EAGER=true becomes --enforce-eager. For anything else, put the raw flags in VLLM_EXTRA_ARGS; they are appended to the vllm serve command and override earlier settings.
Do I need a Hugging Face token to deploy a model on Runpod?
Only for gated or private models. Set it as HF_TOKEN in the endpoint's environment variables. Without it a gated model fails at startup, which the worker reports as a startup error rather than restarting repeatedly.
What is the difference between /runsync and the /openai/v1 routes on a Runpod endpoint?
/runsync is Runpod's native job API: you send {"input": {...}} and get a job result with delayTime, executionTime and the output. The /openai/v1 routes accept standard OpenAI request bodies, so existing OpenAI client code works by changing only the API key, base URL and model name.
Related Guides
Serve Multiple LoRA Adapters with SGLang's OpenAI API
Use one SGLang server to route OpenAI-compatible requests to different LoRA adapters via the model field or extra_body lora_path parameter.
Benchmark vLLM Speculative Decoding with EAGLE, n-gram, and MTP
Run vLLM's speculative decoding benchmark, configure EAGLE, EAGLE3, n-gram, custom draft models, and MTP, and interpret acceptance metrics.
Meta's Muse Spark Open Puts 816B MoE on Two Mac Studios
Meta's open-weight 816B MoE activates 42B parameters, holds 1M context, runs on two M5 Ultra Mac Studios, and leads on MCP-Atlas but trails on coding.