fix openai api (#1367)

Author: Jintao-Huang

docs/source/LLM/VLLM推理加速与部署.md

@@ -294,7 +294,12 @@ print(f'model_type: {model_type}')
 
 query = '浙江的省会在哪里?'
 request_config = XRequestConfig(seed=42)
-resp = asyncio.run(inference_client_async(model_type, query, request_config=request_config))
+tasks = [inference_client_async(model_type, query, request_config=request_config) for _ in range(5)]
+async def _batch_run(tasks):
+    return await asyncio.gather(*tasks)
+
+resp_list = asyncio.run(_batch_run(tasks))
+resp = resp_list[0]
 response = resp.choices[0].message.content
 print(f'query: {query}')
 print(f'response: {response}')
@@ -317,7 +322,7 @@ model_type: qwen-7b-chat
 query: 浙江的省会在哪里?
 response: 浙江省的省会是杭州市。
 query: 这有什么好吃的?
-response: 杭州有许多美食，例如西湖醋鱼、东坡肉、龙井虾仁、叫化童子鸡等。此外，杭州还有许多特色小吃，如西湖藕粉、杭州小笼包、杭州油条等。
+response: 浙江省有很多美食，比如杭州菜、宁波菜、绍兴菜、温州菜等。其中，杭州菜以清淡、鲜美、精致著称，而宁波菜则以鲜美、醇厚、香辣著称。此外，浙江省还有许多特色小吃，比如杭州的西湖醋鱼、宁波的汤圆、绍兴的酒酿圆子等。
 """
 ```
 

docs/source/Multi-Modal/MLLM部署文档.md

@@ -97,7 +97,7 @@ response: <ref>花</ref><box>(34,449),(368,981)</box><box>(342,456),(670,917)</b
 """
 ```
 
-使用openai:
+使用openai（方案1）:
 ```python
 from openai import OpenAI
 client = OpenAI(
@@ -162,6 +162,81 @@ response: <ref>花</ref><box>(34,449),(368,981)</box><box>(342,456),(670,917)</b
 """
 ```
 
+使用openai（方案2）:
+```python
+from openai import OpenAI
+client = OpenAI(
+    api_key='EMPTY',
+    base_url='http://localhost:8000/v1',
+)
+model_type = client.models.list().data[0].id
+print(f'model_type: {model_type}')
+
+# use base64
+# import base64
+# with open('rose.jpg', 'rb') as f:
+#     img_base64 = base64.b64encode(f.read()).decode('utf-8')
+# image_url = f'data:image/jpeg;base64,{img_base64}'
+
+# use local_path
+# from swift.llm import convert_to_base64
+# image_url = convert_to_base64(images=['rose.jpg'])['images'][0]
+# image_url = f'data:image/jpeg;base64,{image_url}'
+
+# use url
+image_url = 'https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/rose.jpg'
+
+query = '图中是什么花，有几只？'
+messages = [{
+    'role': 'user',
+    'content': [
+        {'type': 'text', 'text': 'Picture 1:'},
+        {'type': 'image_url', 'image_url': {'url': image_url}},
+        {'type': 'text', 'text': f'\n{query}'},
+    ]
+}]
+resp = client.chat.completions.create(
+    model=model_type,
+    messages=messages,
+    seed=42)
+response = resp.choices[0].message.content
+print(f'query: {query}')
+print(f'response: {response}')
+
+# 流式
+messages.append({'role': 'assistant', 'content': response})
+query = '框出图中的花'
+messages.append({'role': 'user', 'content': query})
+stream_resp = client.chat.completions.create(
+    model=model_type,
+    messages=messages,
+    stream=True,
+    seed=42)
+
+print(f'query: {query}')
+print('response: ', end='')
+for chunk in stream_resp:
+    print(chunk.choices[0].delta.content, end='', flush=True)
+print()
+"""Out[0]
+model_type: qwen-vl-chat
+query: Picture 1:<img>https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/rose.jpg</img>
+图中是什么花，有几只？
+response: 图中是三朵红玫瑰花。
+query: 框出图中的花
+response: <ref>花</ref><box>(34,449),(368,981)</box><box>(342,456),(670,917)</box><box>(585,508),(859,977)</box>
+"""
+
+"""Out[0]
+model_type: qwen-vl-chat
+query: Picture 1:<img>https://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/rose.jpg</img>
+图中是什么花，有几只？
+response: 图中是三朵红玫瑰花。
+query: 框出图中的花
+response: <ref>花</ref><box>(34,449),(368,981)</box><box>(342,456),(670,917)</box><box>(585,508),(859,977)</box>
+"""
+```
+
 ## yi-vl-6b-chat
 
 **服务端:**
@@ -233,13 +308,13 @@ print()
 """
 model_type: yi-vl-6b-chat
 query: 描述这张图片
-response: 图片显示一只小猫坐在地板上,眼睛睁开,凝视着前方。小猫有灰色和白色的毛皮,有黑色和白色的条纹。它的毛皮在图片中有些混乱,给它一个可爱和迷人的外观。小猫似乎正在注意周围环境,可能正在寻找一些东西或只是观察它的环境。
+response: 图片显示一只小猫坐在地板上,眼睛睁开,凝视着摄像机。小猫看起来很可爱,有灰色和白色的毛皮,以及蓝色的眼睛。小猫似乎正在看摄像机,可能被吸引到它正在被拍摄。
 query: 图中有几只羊
-response: 图中有四只羊.
+response: 图中有四只羊。
 """
 ```
 
-使用openai:
+使用openai（方案1）:
 ```python
 from openai import OpenAI
 client = OpenAI(
@@ -298,9 +373,76 @@ print()
 """
 model_type: yi-vl-6b-chat
 query: 描述这张图片
-response: 图片显示一只小猫坐在地板上,眼睛睁开,凝视着前方。小猫有灰色和白色的毛皮,有黑色和白色的条纹。它的毛皮在图片中有些混乱,给它一个可爱和迷人的外观。小猫似乎正在注意周围环境,可能正在寻找一些东西或只是观察它的环境。
+response: 图片显示一只小猫坐在地板上,眼睛睁开,凝视着摄像机。小猫看起来很可爱,有灰色和白色的毛皮,以及蓝色的眼睛。小猫似乎正在看摄像机,可能被吸引到它正在被拍摄。
 query: 图中有几只羊
-response: 图中有四只羊.
+response: 图中有四只羊。
+"""
+```
+
+使用openai（方案2）:
+```python
+from openai import OpenAI
+client = OpenAI(
+    api_key='EMPTY',
+    base_url='http://localhost:8000/v1',
+)
+model_type = client.models.list().data[0].id
+print(f'model_type: {model_type}')
+
+# use base64
+# import base64
+# with open('cat.png', 'rb') as f:
+#     img_base64 = base64.b64encode(f.read()).decode('utf-8')
+# image_url = f'data:image/jpeg;base64,{img_base64}'
+
+# use local_path
+# from swift.llm import convert_to_base64
+# image_url = convert_to_base64(images=['cat.png'])['images'][0]
+# image_url = f'data:image/jpeg;base64,{image_url}'
+
+# use url
+image_url = 'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png'
+
+query = '描述这张图片'
+messages = [{
+    'role': 'user',
+    'content': [
+        {'type': 'text', 'text': query},
+        {'type': 'image_url', 'image_url': {'url': image_url}},
+    ]
+}]
+resp = client.chat.completions.create(
+    model=model_type,
+    messages=messages,
+    temperature=0)
+response = resp.choices[0].message.content
+print(f'query: {query}')
+print(f'response: {response}')
+
+# 流式
+messages.append({'role': 'assistant', 'content': response})
+query = '图中有几只羊'
+messages.append({'role': 'user', 'content': [
+    {'type': 'text', 'text': query},
+    {'type': 'image_url', 'image_url': {'url': 'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png'}}
+]})
+stream_resp = client.chat.completions.create(
+    model=model_type,
+    messages=messages,
+    stream=True,
+    temperature=0)
+
+print(f'query: {query}')
+print('response: ', end='')
+for chunk in stream_resp:
+    print(chunk.choices[0].delta.content, end='', flush=True)
+print()
+"""
+model_type: yi-vl-6b-chat
+query: 描述这张图片
+response: 图片显示一只小猫坐在地板上,眼睛睁开,凝视着摄像机。小猫看起来很可爱,有灰色和白色的毛皮,以及蓝色的眼睛。小猫似乎正在看摄像机,可能被吸引到它正在被拍摄。
+query: 图中有几只羊
+response: 图中有四只羊。
 """
 ```
 
@@ -380,7 +522,7 @@ response: 这张图片看起来是用数字绘画技术创作的。艺术家使
 """
 ```
 
-使用openai:
+使用openai（方案1）:
 ```python
 from openai import OpenAI
 client = OpenAI(
@@ -444,6 +586,71 @@ response: 这张图片看起来是用数字绘画技术创作的。艺术家使
 """
 ```
 
+使用openai（方案2）:
+```python
+from openai import OpenAI
+client = OpenAI(
+    api_key='EMPTY',
+    base_url='http://localhost:8000/v1',
+)
+model_type = client.models.list().data[0].id
+print(f'model_type: {model_type}')
+
+# use base64
+# import base64
+# with open('cat.png', 'rb') as f:
+#     img_base64 = base64.b64encode(f.read()).decode('utf-8')
+# image_url = f'data:image/jpeg;base64,{img_base64}'
+
+# use local_path
+# from swift.llm import convert_to_base64
+# image_url = convert_to_base64(images=['cat.png'])['images'][0]
+# image_url = f'data:image/jpeg;base64,{image_url}'
+
+# use url
+image_url = 'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png'
+
+query = '描述这张图片'
+messages = [{
+    'role': 'user',
+    'content': [
+        {'type': 'text', 'text': query},
+        {'type': 'image_url', 'image_url': {'url': image_url}},
+    ]
+}]
+resp = client.chat.completions.create(
+    model=model_type,
+    messages=messages,
+    temperature=0)
+response = resp.choices[0].message.content
+print(f'query: {query}')
+print(f'response: {response}')
+
+# 流式
+messages.append({'role': 'assistant', 'content': response})
+query = '这张图是如何产生的？'
+messages.append({'role': 'user', 'content': query})
+stream_resp = client.chat.completions.create(
+    model=model_type,
+    messages=messages,
+    stream=True,
+    temperature=0)
+
+print(f'query: {query}')
+print('response: ', end='')
+for chunk in stream_resp:
+    print(chunk.choices[0].delta.content, end='', flush=True)
+print()
+
+"""
+model_type: minicpm-v-v2_5-chat
+query: 描述这张图片
+response: 这张图片展示了一只年轻的猫咪的特写，可能是一只小猫，具有明显的特征。它的毛皮主要是白色的，带有灰色和黑色的条纹，尤其是在脸部周围。小猫的眼睛很大，呈蓝色，给人一种好奇和迷人的表情。耳朵尖尖，竖立着，显示出警觉。背景模糊不清，突出了小猫的特征。整体的色调柔和，猫咪的毛皮与背景的柔和色调形成对比。
+query: 这张图是如何产生的？
+response: 这张图片看起来是用数字绘画技术创作的。艺术家使用数字绘图工具来模仿毛皮的纹理和颜色，眼睛的反射，以及整体的柔和感。这种技术使艺术家能够精确地控制细节和色彩，创造出逼真的猫咪形象。
+"""
+```
+
 ## qwen-vl
 
 **服务端:**

docs/source/Multi-Modal/vLLM推理加速文档.md

@@ -142,43 +142,47 @@ print(f'model_type: {model_type}')
 # import base64
 # with open('cat.png', 'rb') as f:
 #     img_base64 = base64.b64encode(f.read()).decode('utf-8')
-# images = [img_base64]
+# image_url = f'data:image/jpeg;base64,{img_base64}'
 
 # use local_path
 # from swift.llm import convert_to_base64
-# images = ['cat.png']
-# images = convert_to_base64(images=images)['images']
+# image_url = convert_to_base64(images=['cat.png'])['images'][0]
+# image_url = f'data:image/jpeg;base64,{image_url}'
 
 # use url
-images = ['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png']
+image_url = 'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/cat.png'
 
 query = 'Describe this image.'
 messages = [{
     'role': 'user',
-    'content': query
+    'content': [
+        {'type': 'text', 'text': query},
+        {'type': 'image_url', 'image_url': {'url': image_url}},
+    ]
 }]
+
 resp = client.chat.completions.create(
     model=model_type,
     messages=messages,
-    temperature=0,
-    extra_body={'images': images})
+    temperature=0)
 response = resp.choices[0].message.content
 print(f'query: {query}')
 print(f'response: {response}')
 
 # 流式
-images = ['http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png']
 query = 'How many sheep are in the picture?'
 messages = [{
     'role': 'user',
-    'content': query
+    'content': [
+        {'type': 'text', 'text': query},
+        {'type': 'image_url', 'image_url': {'url': 'http://modelscope-open.oss-cn-hangzhou.aliyuncs.com/images/animal.png'}}
+    ]
 }]
 stream_resp = client.chat.completions.create(
     model=model_type,
     messages=messages,
     stream=True,
-    temperature=0,
-    extra_body={'images': images})
+    temperature=0)
 
 print(f'query: {query}')
 print('response: ', end='')
@@ -188,9 +192,9 @@ print()
 """
 model_type: llava1_6-vicuna-13b-instruct
 query: Describe this image.
-response: The image shows a close-up of a person's hands playing a guitar. The hands are positioned on the neck of the guitar, with the fingers pressing down on the strings to produce music. The guitar has a wooden body and a glossy finish, and the strings are clearly visible. The background is blurred, but it appears to be an indoor setting with warm lighting. The focus of the image is on the hands and the guitar, with the background serving to highlight the subject. There are no visible texts or distinctive brands in the image. The style of the image is a realistic photograph with a shallow depth of field, which is a common technique in portrait photography to draw attention to the subject.
+response: In the image, a kitten with striking blue eyes is the main subject. The kitten, with its fur in shades of gray and white, is sitting on a white surface. Its head is slightly tilted to the left, giving it a curious and endearing expression. The kitten's eyes are wide open, and its mouth is slightly open, as if it's in the middle of a meow or perhaps just finished one. The background is blurred, drawing focus to the kitten, but it appears to be a room with a window, suggesting an indoor setting. The overall image gives a sense of warmth and cuteness.
 query: How many sheep are in the picture?
-response: There are two sheep in the picture.
+response: There are four sheep in the picture.
 """
 ```
 

docs/source_en/LLM/VLLM-inference-acceleration-and-deployment.md

@@ -235,7 +235,12 @@ print(f'model_type: {model_type}')
 
 query = 'Where is the capital of Zhejiang?'
 request_config = XRequestConfig(seed=42)
-resp = asyncio.run(inference_client_async(model_type, query, request_config=request_config))
+tasks = [inference_client_async(model_type, query, request_config=request_config) for _ in range(5)]
+async def _batch_run(tasks):
+    return await asyncio.gather(*tasks)
+
+resp_list = asyncio.run(_batch_run(tasks))
+resp = resp_list[0]
 response = resp.choices[0].message.content
 print(f'query: {query}')
 print(f'response: {response}')
@@ -256,9 +261,9 @@ asyncio.run(_stream())
 """Out[0]
 model_type: qwen-7b-chat
 query: Where is the capital of Zhejiang?
-response: The capital of Zhejiang Province is Hangzhou.
+response: The capital of Zhejiang is Hangzhou.
 query: What delicious food is there?
-response: Hangzhou has many delicious foods, such as West Lake Vinegar Fish, Dongpo Pork, Longjing Shrimp, Beggar's Chicken, etc. In addition, Hangzhou also has many specialty snacks, such as West Lake Lotus Root Powder, Hangzhou Xiao Long Bao, Hangzhou You Tiao, etc.
+response: Hangzhou is famous for its delicious food, such as West Lake Fish in Vinegar Gravy, Dongpo Pork, and Longjing Tea.
 """
 ```