gpt4o inference

[Kartik-3004]

I setup the API key in VLMEvalkit and want to inference for gpt4o, using the command below. However, the results obtained are really bad. Did you face the issue ?

export PW_TEST_SCREENSHOT_NO_FONTS_READY=1 PYTHONPATH="$(dirname $0)/..":$PYTHONPATH \ python eval_requery.py \ --rank 0 \ --world-size 1 \ --model_type vlmevalkit_GPT4o_20241120 \ --save_path 'output/requery/vlmevalkit_GPT4o_20241120'

The results are:
{ "f1_score": { "total_dict": { "total_length": 300, "average": 0.003703703703703704 }, "area_dict": { "news": { "length": 221, "average": 0.005027652086475616 }, "knowledge": { "length": 79, "average": 0.0 } }, "subfield_dict": { "auto": { "length": 10, "average": 0.0 }, "fashion": { "length": 10, "average": 0.0 }, "anime": { "length": 10, "average": 0.0 }, "architecture": { "length": 14, "average": 0.0 }, "paper": { "length": 24, "average": 0.0 }, "art": { "length": 25, "average": 0.0 }, "traditional sports": { "length": 16, "average": 0.0 }, "entertainment": { "length": 41, "average": 0.0 }, "finance": { "length": 31, "average": 0.007168458781362007 }, "general": { "length": 50, "average": 0.0 }, "astronomy": { "length": 10, "average": 0.0 }, "technology": { "length": 18, "average": 0.012345679012345678 }, "falsepremise": { "length": 13, "average": 0.0 }, "esports": { "length": 28, "average": 0.023809523809523808 } } } }

[CaraJ7]

Hi @Kartik-3004 , would you please provide some visualization of the GPT4o response so the error can be better identified?

[Kartik-3004]

Hi,
Here is a sample output. It says "Failed to obtain answer via API".

{ "sample_id": "entertainment_0", "query": "Which city in Europe did Taylor Swift's concert in 2024 get forcibly cancelled in August 2024?", "prediction": "Failed to obtain answer via API.", "gt_answer": "Vienna", "f1_score": 0, "area": "news", "subfield": "entertainment" }

[Kartik-3004]

Hi,
Are you able to find a solution to this ?

[CaraJ7]

Hi, I have checked the code and everything works fine on my computer. The issue may be the wrong configuration of the API key of GPT4o. You could first try to obtain a meaningful response from GPT4o through the following code snippet:

os.environ['OPENAI_API_KEY'] = 'YOUR_API_KEY'

model = VLMEvalModel('vlmevalkit_GPT4o')
image_files = [
    'figs/fig1.png',
]
text_query = 'Please describe the image in detail. <image>'
response = model.infer(image_files, text_query)
print(response)

[Kartik-3004]

Hi,
I checked my API key is working.

I added some print statements in the models.vlmevalkit_model_api.py. Here is the output below.

Text Query: Please describe the image in detail. <image>
Query Input: [{'type': 'text', 'value': 'Please describe the image in detail. '}, {'type': 'image', 'value': 'image.jpg'}]
Response: Failed to obtain answer via API.

Is the input format wrong for VLMEvalkit ?

[Kartik-3004]

Hi !
After some debugging, I got the error.

In the models/vlmevalkit_model_api.py Line no. 46 should be:
self.model = supported_VLM[model_name](max_tokens=512)
and NOT
self.model = supported_VLM[model_name](max_new_tokens=512)