Deploy an Open-Source LLM on Runpod Serverless with vLLM (2026)

Published · Updated • guides
vLLMServerless

This guide is adapted from Runpod's official worker-vllm repository (MIT licence) and its vLLM get-started documentation. Commands and code are reproduced from those sources; where the upstream example has a bug, we say so and show the fix separately.

Disclosure: links to Runpod marked as such are affiliate links. If you sign up through them, AI Mastery may earn a commission at no cost to you. It does not change what this guide recommends, and every step works the same with a direct signup.

Running an open-weight model behind an API used to mean renting a GPU server, installing an inference engine, writing a web service around it and keeping all of it alive. Runpod Serverless collapses that into a configured endpoint: you choose a Hugging Face model, Runpod starts GPU workers running vLLM when requests arrive, and you call the result with the same OpenAI client code you already use.

This guide deploys a model with Runpod's official vLLM worker, tests it through both of the APIs the endpoint exposes, configures it with environment variables, and covers the failure modes that cost money when you get them wrong. It suits teams that want an OpenAI-compatible endpoint for an open model without operating servers, and anyone weighing serverless GPUs against a self-managed deployment like the one in our DeepSeek V4 self-hosting guide.

Prerequisites

  • A Runpod account (sign up for Runpod, affiliate link) and a Runpod API key, created under your account's API key settings.
  • A Hugging Face access token, only if your model is gated or private.
  • A model that fits a GPU you can rent. Weights plus KV cache must fit in VRAM. Our VRAM calculator estimates this from parameter count, precision and context length, and our GPU cloud pricing table compares current hourly rates, including Runpod's.

Step 1: Choose a model

The vLLM worker serves most models on Hugging Face that vLLM supports. Runpod's documentation lists examples including meta-llama/Llama-3.2-3B-Instruct, Qwen/Qwen3-8B, google/gemma-3-1b-it, microsoft/Phi-4-mini-instruct and deepseek-ai/DeepSeek-R1-Distill-Qwen-7B, and its tutorial uses openchat/openchat-3.5-0106.

Two properties decide whether a model will work. It must have an architecture vLLM supports, and, for chat requests, a chat template. Instruction-tuned models ship one; a base model needs one supplied through CUSTOM_CHAT_TEMPLATE. Start with a small instruct model for your first deployment: you will see the whole flow work before paying for a large GPU.

Step 2: Deploy from the Runpod Hub

Runpod's get-started guide deploys the worker from the Hub:

  1. Find the vLLM repo (runpod-workers/worker-vllm) in the Runpod Hub.
  2. Click Deploy, using the latest vLLM worker version.
  3. In the Model field, enter the model name, for example openchat/openchat-3.5-0106.
  4. Click Advanced to expand the vLLM settings.
  5. Set Max Model Length to 8192, or a context length appropriate for your model.
  6. Leave the other settings at their defaults unless you need otherwise, then click Next.
  7. Click Create Endpoint.

The endpoint then initialises, which can take several minutes while Runpod provisions a worker and downloads the model. If you prefer to build from the raw image instead of the Hub, it is published as runpod/worker-v1-vllm:<version>, with versions listed on the repository's releases page.

Max Model Length is the setting to think about. It becomes vLLM's --max-model-len, and vLLM reserves KV-cache memory for it. A long context on a small GPU is the most common reason a worker never starts.

Step 3: Find your endpoint ID

When deployment completes, copy the Endpoint ID from the endpoint's page. Every API call below needs it, together with your Runpod API key.

Step 4: Test with Runpod's native API

The endpoint's Requests tab sends a default test request:

{
    "input": {
        "prompt": "Hello World"
    }
}

From your own machine, the same request goes to the synchronous runsync route. This is Runpod's command, with your endpoint ID and API key substituted:

curl -X POST "https://api.runpod.ai/v2/YOUR_ENDPOINT_ID/runsync" \
     -H "Authorization: Bearer YOUR_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"input": {"prompt": "Hello World"}}'

The response wraps the model output in Runpod's job envelope. Runpod's documentation shows this shape:

{
  "delayTime": 638,
  "executionTime": 3344,
  "id": "f0706ead-c5ec-4689-937c-e21d5fbbca47-u1",
  "output": [
    {
      "choices": [
        {
          "tokens": ["CHAT_RESPONSE"]
        }
      ],
      "usage": {
        "input": 3,
        "output": 100
      }
    }
  ],
  "status": "COMPLETED",
  "workerId": "0e7o8fgmm9xgty"
}

delayTime is how long the job waited for a worker, and executionTime is how long it ran. A large delayTime on the first request is the cold start: a worker had to be started and the model loaded. Expect it on the first call after the endpoint has been idle.

Step 5: Call it with the OpenAI client

The worker also serves OpenAI-compatible routes under /openai/v1: Chat Completions, Models, Responses and Messages, streaming and non-streaming. Existing OpenAI code needs three changes: the API key, the base URL and the model name. Runpod's example client:

from openai import OpenAI
import os

# Initialize the OpenAI Client with your Runpod API Key and Endpoint URL
client = OpenAI(
    api_key=os.environ.get("RUNPOD_API_KEY"),
    base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1",
)

A non-streaming chat completion, as published:

# Create a chat completion
response = client.chat.completions.create(
    model="<YOUR DEPLOYED MODEL REPO/NAME>",
    messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
    temperature=0,
    max_tokens=100,
)
# Print the response
print(response.choices[0].message.content)

The model value is the Hugging Face repo you deployed, such as openchat/openchat-3.5-0106, unless you set OPENAI_SERVED_MODEL_NAME_OVERRIDE. If you are not sure what name the endpoint expects, ask it:

models_response = client.models.list()
list_of_models = [model.id for model in models_response]
print(list_of_models)

The upstream streaming example has a bug. In the repository's README, the loop variable and the variable it prints do not match, so it raises NameError: name 'chunk' is not defined:

  # Create a chat completion stream
  response_stream = client.chat.completions.create(
      model="<YOUR DEPLOYED MODEL REPO/NAME>",
      messages=[{"role": "user", "content": "Why is Runpod the best platform?"}],
      temperature=0,
      max_tokens=100,
      stream=True,
  )
  # Stream the response
  for response in response_stream:
      print(chunk.choices[0].delta.content or "", end="", flush=True)

Our fix, not Runpod's code: name the loop variable chunk so it matches the line that uses it.

for chunk in response_stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

If you call the OpenAI route with curl, note that the README's "after" example still shows <YOUR OPENAI API KEY> in the Authorization header. The endpoint expects your Runpod API key there, exactly as in the runsync call above.

Step 6: Configure the worker with environment variables

Edit the endpoint (Manage, then Edit Endpoint, then Public Environment Variables) to change how vLLM runs. The variables that matter most:

VariableWhat it controlsDefaultWhen to change it
MODEL_NAMEHugging Face repo ID or local path of the weights(none)Always set; this is the model you serve
HF_TOKENAccess token for gated or private models(none)Gated models such as Llama variants
MAX_MODEL_LENMaximum context length (--max-model-len)(model's own)Lower it when the worker runs out of memory at startup
GPU_MEMORY_UTILIZATIONFraction of VRAM vLLM may use0.9Raise slightly for more KV cache, lower if something else shares the GPU
MAX_NUM_SEQSSequences per scheduler iteration256Lower it to cut memory use; raise it for throughput
TENSOR_PARALLEL_SIZENumber of GPUs to split the model across1Models that do not fit one GPU
QUANTIZATIONQuantization method(none)Serving an AWQ or GPTQ checkpoint
MAX_CONCURRENCYConcurrent requests per worker30Tune with load testing
ENABLE_AUTO_TOOL_CHOICE / TOOL_CALL_PARSERTool calling and the parser for the model familyfalse / (none)Agents that use function calling

Beyond this list, the worker translates any environment variable whose name is the uppercased form of a vllm serve flag into that flag, so ENFORCE_EAGER=true becomes --enforce-eager. For flags it does not recognise, VLLM_EXTRA_ARGS is appended to the vllm serve command and overrides earlier settings. Runpod's example:

VLLM_EXTRA_ARGS="--override-generation-config '{\"max_new_tokens\": 512}' --uvicorn-log-level warning"

You can also mount a complete vllm serve config file and point VLLM_CONFIG_FILE at it. Environment variables still take precedence over values in the file.

When the worker will not start

Some startup failures cannot be fixed by restarting: the model does not fit the GPU, MAX_MODEL_LEN needs more KV cache than the GPU holds, vLLM rejects a flag, or a gated model has no HF_TOKEN. If the worker simply exited on these, it would crash-loop: every restart pays again to download and load the model, while the console shows only silent restarts.

The current worker recognises these cases, stays up, and answers every job with the cause and the fix. Runpod's README shows an out-of-memory example:

{
  "error": {
    "message": "Qwen/Qwen3-30B-A3B ran out of GPU memory during startup. This GPU has 19.57 GiB. Lower MAX_MODEL_LEN or MAX_NUM_SEQS, set ENFORCE_EAGER=true to skip CUDA graph capture, use a quantized checkpoint, or redeploy on a larger GPU (or more GPUs with TENSOR_PARALLEL_SIZE). ...",
    "type": "startup_error",
    "code": null
  }
}

If your first test request returns a startup_error, read the message before changing anything: it names the remedy. The full vLLM traceback is in the worker logs. Errors the worker does not recognise, such as a failed download, still exit and retry, because those can succeed on another attempt.

What to watch out for

Cold starts cost time and money. A new worker downloads the model and loads it before it can answer. Large models make that slow, and any configuration that crash-loops makes it expensive. Confirm your settings with a small model first.

Size the context to the GPU, not the model card. A model that advertises a 128K context will not get it on a GPU with room for 8K of KV cache. Set MAX_MODEL_LEN to what your workload needs, and check the arithmetic with the VRAM calculator.

The model name must match. OpenAI-route requests must use the deployed repo name, or your OPENAI_SERVED_MODEL_NAME_OVERRIDE. When in doubt, call client.models.list().

Serverless is not always cheapest. Per-second serverless billing suits bursty or low-volume traffic. For steady high utilisation, a dedicated GPU instance or a self-managed vLLM deployment can cost less per token. Compare current hourly rates in our GPU cloud pricing table against your expected load before committing.

Where to go next

For serving a frontier-scale open model across several GPUs yourself, our DeepSeek V4 vLLM deployment guide covers tensor parallelism and cluster sizing. To run models on your own hardware instead, start with the complete guide to running LLMs locally. And if your application already uses the OpenAI SDK, the endpoint you just deployed is a drop-in replacement: change the key, the base URL and the model name.

Frequently asked questions

How do I call a Runpod vLLM endpoint with the OpenAI Python SDK?

Create the client with api_key set to your Runpod API key and base_url set to https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1, then pass your deployed model's Hugging Face repo name as the model parameter. The worker serves Chat Completions, Models, Responses and Messages routes.

What does MAX_MODEL_LEN do on a Runpod vLLM worker, and why does it cause startup failures?

It sets the model's maximum context length (vLLM's --max-model-len). If it is larger than the KV cache the GPU can hold, vLLM cannot start. The current worker detects this and answers each job with a startup_error explaining the cause, instead of silently restarting; lower MAX_MODEL_LEN or MAX_NUM_SEQS, or use a larger GPU.

How do I pass a vLLM flag that the Runpod worker does not list?

Any environment variable named as the uppercased form of a vllm serve flag is applied automatically, for example ENFORCE_EAGER=true becomes --enforce-eager. For anything else, put the raw flags in VLLM_EXTRA_ARGS; they are appended to the vllm serve command and override earlier settings.

Do I need a Hugging Face token to deploy a model on Runpod?

Only for gated or private models. Set it as HF_TOKEN in the endpoint's environment variables. Without it a gated model fails at startup, which the worker reports as a startup error rather than restarting repeatedly.

What is the difference between /runsync and the /openai/v1 routes on a Runpod endpoint?

/runsync is Runpod's native job API: you send {"input": {...}} and get a job result with delayTime, executionTime and the output. The /openai/v1 routes accept standard OpenAI request bodies, so existing OpenAI client code works by changing only the API key, base URL and model name.

Related Guides