pytorch backend for lora model

[nguoithichkhampha]

I know triton inference server support to serve the lora model but seems it only supports for LLM model with vLLM backend and TensorRTLLM backend.
Our use case is:
We have a foundation model for eeg data. then we fine tuning the model on different task with peft library from Hugging face.
the output is:

• base model
• many lora adapter for different tasks

How we can run inference with triton server without duplicating the weight of base model

[nathon-lee]

You're right that Triton's LoRA support is mainly for LLM backends like vLLM and TensorRT-LLM. For your model use case, here's what I'd recommend:

Use Python Backend with dynamic LoRA loading

The key idea is to load your base model once, then dynamically swap LoRA adapters based on the task. Here's the basic approach:

In your model.py

`
class TritonPythonModel:
def initialize(self, args):
# Load base model once
self.base_model = load_your_eeg_model()
self.adapters = {} # Cache for loaded adapters

def execute(self, requests):
    for request in requests:
        # Get task type from request parameters
        task = request.parameters()["task_type"]
        
        # Load adapter if not cached
        if task not in self.adapters:
            adapter_path = f"adapters/{task}"
            self.adapters[task] = PeftModel.from_pretrained(
                self.base_model, adapter_path
            )
        
        # Run inference with the right adapter
        model = self.adapters[task]
        result = model(input_data)
        # return response...

`
Why this works well:

Base model weights loaded only once
LoRA adapters are small and can be cached
You can pass the task type as a request parameter
Works with standard PEFT library
No need to duplicate the heavy base model
Alternative: If you have many adapters, implement an LRU cache to unload unused ones and save memory.

This gives you the flexibility you need without the weight duplication problem.