Serve Multiple LoRA Adapters with SGLang's OpenAI API

Serving one base model while swapping LoRA adapters per request is a cost-effective way to ship many specialised behaviours without running many full copies of the model. SGLang supports this directly in its OpenAI-compatible API: you launch one server with --enable-lora, register named adapter paths, and then choose which adapter a request uses through the model field or an extra_body parameter. Engineers who maintain SQL generators, code assistants, summarisation heads, or domain-specific fine-tunes can consolidate them onto a single GPU deployment and route traffic at request time.

The trade-off is mostly on the serving side. LoRA adapters are small relative to the base model, but the base model still occupies the same VRAM, and every registered adapter adds some additional memory and bookkeeping. The examples below use meta-llama/Llama-3.1-8B-Instruct; replace the example adapter paths with directories that exist in your environment. Setup time is minimal if you already have trained adapters; the work is in wiring adapter names and understanding the three request forms the API accepts.

Adapted from SGLang's lora.py example in the sgl-project/sglang repository (Apache-2.0). The source demonstrates four calls: two select an adapter with the model field suffix, one uses the older extra_body form, and one calls the base model.

Prerequisites

The source example establishes these requirements:

• A Python environment where sglang and openai are installed. The launch command uses python -m sglang.launch_server.
• Access to meta-llama/Llama-3.1-8B-Instruct or another base model. The examples below keep that exact model ID.
• One or more trained LoRA adapters on disk, each identified by a directory path.
• Enough GPU memory to load the base model plus the adapters you register at launch.
• An OpenAI-compatible client that can reach the server endpoint.

The adapter names you choose at launch are arbitrary strings; they become the identifiers client code uses later.

Step 1: Launch the server with named LoRA adapters

Start SGLang with --enable-lora and pass adapter mappings to --lora-paths. Each mapping is name=/path/to/adapter. Replace /path/to/sql and /path/to/python with directories that contain your trained adapter artefacts.

python -m sglang.launch_server \
    --model meta-llama/Llama-3.1-8B-Instruct \
    --enable-lora \
    --lora-paths sql=/path/to/sql python=/path/to/python

The sql and python names are not special; they are the keys your client will use. Without --enable-lora, any request that tries to select an adapter will fail.

Step 2: Create the OpenAI-compatible client

The server exposes an OpenAI-style endpoint, so you can use the openai Python package to talk to it.

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")

The base URL points at the local SGLang server. If you changed the host, port, or path at launch, update this value to match. The /v1 suffix is required for the OpenAI-compatible surface. The source uses api_key="EMPTY"; if your SGLang deployment has authentication enabled, replace it with the configured key.

The request snippets in the next sections keep the indentation they have inside the source example's main() function. If you want a single copy-paste file, use the complete runnable script at the end of this guide.

Step 3: Select a LoRA adapter through the model parameter

The current recommended form puts the adapter name after a colon in the model field. The part before the colon is the base model ID you used at server launch, and the part after is the registered adapter name.

    # Example 1: NEW - Adapter in model parameter (OpenAI-compatible)
    print("1. Chat with LoRA adapter in model parameter:")
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct:sql",  # ← adapter:name syntax
        messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
        max_tokens=50,
    )
    print(f"   Response: {response.choices[0].message.content}\n")

The :sql suffix tells SGLang to apply the adapter named sql for this request only. The same client can send another request with :python without restarting or reconfiguring the server. This is the cleanest integration path because adapter selection stays inside the standard model field.

Step 4: Use the same syntax with the completions endpoint

Adapter selection works the same way for the legacy completions API. Instead of a chat message list, you send a raw prompt. The adapter suffix still goes in model.

    # Example 2: Completions API with adapter
    print("2. Completion with LoRA adapter:")
    response = client.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct:python",
        prompt="def fibonacci(n):",
        max_tokens=50,
    )
    print(f"   Response: {response.choices[0].text}\n")

If your existing code already uses client.completions.create, this lets you adopt LoRA routing without a larger refactor. The response object differs: chat completions return response.choices[0].message.content, while completions return response.choices[0].text.

Step 5: Use the backward-compatible lora_path field

Older SGLang integrations may pass the adapter name in extra_body rather than inside model. That form still works, and the source example keeps it for compatibility.

    # Example 3: OLD - Backward compatible with explicit lora_path
    print("3. Backward compatible (explicit lora_path):")
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
        extra_body={"lora_path": "sql"},
        max_tokens=50,
    )
    print(f"   Response: {response.choices[0].message.content}\n")

Here the model field is just the base model, and extra_body carries the adapter name. This is useful when you cannot change the model string because it is controlled by another system. Prefer the model:adapter form for new code; it is more explicit and easier to log.

Step 6: Fall back to the base model

Sending a request with only the base model ID and no adapter suffix or lora_path gives you the unadapted model. This is important for A/B tests and fallback behaviour.

    # Example 4: Base model (no adapter)
    print("4. Base model without adapter:")
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": "Hello!"}],
        max_tokens=30,
    )
    print(f"   Response: {response.choices[0].message.content}\n")

Because adapters are selected per request, the same server can answer some requests with :sql, others with :python, and still others with no adapter. There is no global state to flip between calls.

The table below summarises the three adapter-selection forms and the base-model fallback.

Complete runnable script

The snippets above are excerpts. This is the full source file as shown in the SGLang example, with the indented calls kept inside main() and the original top-level error handling intact.

"""
OpenAI-compatible LoRA adapter usage with SGLang.

Server Setup:
    python -m sglang.launch_server \\
        --model meta-llama/Llama-3.1-8B-Instruct \\
        --enable-lora \\
        --lora-paths sql=/path/to/sql python=/path/to/python
"""

import openai

client = openai.Client(base_url="http://127.0.0.1:30000/v1", api_key="EMPTY")


def main():
    print("SGLang OpenAI-Compatible LoRA Examples\n")

    # Example 1: NEW - Adapter in model parameter (OpenAI-compatible)
    print("1. Chat with LoRA adapter in model parameter:")
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct:sql",  # ← adapter:name syntax
        messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
        max_tokens=50,
    )
    print(f"   Response: {response.choices[0].message.content}\n")

    # Example 2: Completions API with adapter
    print("2. Completion with LoRA adapter:")
    response = client.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct:python",
        prompt="def fibonacci(n):",
        max_tokens=50,
    )
    print(f"   Response: {response.choices[0].text}\n")

    # Example 3: OLD - Backward compatible with explicit lora_path
    print("3. Backward compatible (explicit lora_path):")
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": "Convert to SQL: show all users"}],
        extra_body={"lora_path": "sql"},
        max_tokens=50,
    )
    print(f"   Response: {response.choices[0].message.content}\n")

    # Example 4: Base model (no adapter)
    print("4. Base model without adapter:")
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[{"role": "user", "content": "Hello!"}],
        max_tokens=30,
    )
    print(f"   Response: {response.choices[0].message.content}\n")

    print("All examples completed!")


if __name__ == "__main__":
    try:
        main()
    except Exception as e:
        print(f"Error: {e}")
        print(
            "\nEnsure server is running:\n"
            "  python -m sglang.launch_server --model ... --enable-lora --lora-paths ..."
        )

What to watch out for

Adapter names must match exactly. The strings after --lora-paths are the keys. A request for base:sql will fail if the server was launched with --lora-paths sq=/path/to/sql or with a different case. Treat these names as part of your API contract and validate them at startup.

The extra_body field is not standard OpenAI. It works with SGLang's compatible server in this example, but not necessarily with other OpenAI-compatible proxies or gateways. If you plan to route requests through a third-party layer, test whether that layer preserves extra JSON fields. The model:adapter form is safer for portability because it stays inside a field every client already sends.

The source does not demonstrate mixed-adapter batching. The four example calls are issued synchronously one at a time. Before committing many adapters to one server, benchmark your own request mix; the number of registered adapters can affect throughput and latency even when each adapter is small.

Memory is not free. The base model dominates, but each registered adapter adds weights and routing metadata. Register only adapters you will actually use and check GPU memory after startup before adding more.

The source example catches all exceptions. The full lora.py wraps the calls in a broad try/except and prints a generic error. That is fine for a demo, but in production you should log the request model string, adapter name, and server response separately so a mistyped adapter name is easy to distinguish from a timeout or a malformed payload.

Gated base models still need access. The example uses a Meta Llama model. If your environment has not accepted the model terms or lacks a Hugging Face token, the server will fail at model load before any LoRA request is made. Resolve base-model access first.

Adapters are task-specific. A LoRA changes the model's behaviour only within the task it was trained for. Sending a :sql request with a general conversation prompt may produce worse output than the base model. Keep routing logic tied to the adapter's known competence.

Where to go next

The complete example script in SGLang's repository is the same material with the surrounding main() function and error handling. For deeper serving engineering around SGLang, including KV cache choices that affect mixed-workload latency, see Akv KV cache engineering in SGLang agents. A practical next step is to move adapter selection out of hard-coded examples and into a routing layer that maps each incoming request to an adapter name based on user context or task type.