Different behavior when compared to HF

[danieltudosiu]

Hi all,

I am aiming to use LMDeploy to process big chunks for data, but sadly it seems it has different behavior when compared to pure HF.

I am expecting it to have the same output as the hardware is identical and I am setting the seeds accordingly.

Could you please help me get the same behavior as the HF forward pass?

HF Code:

import torch
from transformers import AutoModel, AutoTokenizer, GenerationConfig

model = AutoModel.from_pretrained(
    '/path/to/internlm/internlm-xcomposer2-vl-7b', 
    trust_remote_code=True,
).half().cuda().eval()

tokenizer = AutoTokenizer.from_pretrained(
    '/path/to/internlm/internlm-xcomposer2-vl-7b', 
    trust_remote_code=True
)

text = "<ImageHere> Please describe this image"

torch.manual_seed(4114)
generation_config = GenerationConfig(
    temperature=0.8,
    top_p=1.0,
    top_k=1,
    repetition_penalty=1,
    max_new_tokens=512,
    do_sample=True
)

image = '/path/to/toy/img.png'
with torch.cuda.amp.autocast(): 
    response, _ = model.chat(
        tokenizer, 
        query=text, 
        image=image, 
        history=[], 
        generation_config=generation_config,
        meta_instruction=""
    ) 
print(response)

LMDeploy Code:

from lmdeploy import pipeline, GenerationConfig
from lmdeploy.vl import load_image
from os.path import join

gen_config = GenerationConfig(
    temperature=0.8,
    top_p=1.0,
    top_k=1.0,
    random_seed=4114
)
    
pipe = pipeline('/path/to/internlm/internlm-xcomposer2-vl-7b')

sample = (
    "<ImageHere> Please describe this image", 
    load_image( '/path/to/toy/img.png')
) 
response = pipe(sample, gen_config=gen_config)

print(response)

[danieltudosiu]

The issue was me removing the system prompt on the HF side as outlined by meta_instruction="". After I fixed that the models behaved identically.