1. modified inference.py to allow the use of LLM cloud service. 2. modified inference.py and evaluate.py to align the data structure.

taskbench/README.md

@@ -133,6 +133,8 @@ conda activate taskbench
 pip install -r requirements.txt
 ```
 
+Now you can use any LLM hosted in the cloud that follows the OpenAI API for inference.
+
 Additionally, if you wish to evaluate open-source large language models, you will also need to deploy the LLMs locally using an **OpenAI-compatible API**. We recommend using the `fastchat` tool to deploy the service to the `localhost:4000` endpoint.
 
 ```bash
@@ -147,17 +149,21 @@ python3 -m fastchat.serve.openai_api_server --host localhost --port 4000
 
 ### Inference
 
+When using an LLM cloud service, ensure that the correct `api_key` and `api_base` are set. In this case, `api_addr` and `api_host` can be omitted. If the model is hosted locally, provide the correct `api_addr` and `api_host`, and do not include `api_base`.
+
 For convenience, it is recommended to deploy all LLMs to the same endpoint, such as `localhost:4000`. To generate the prediction file on TaskBench, specify the name of the LLM using the following command:
 
 ```bash
 export YOUR_API_KEY=API_KEY
+export YOUR_API_BASE=API_BASE # not needed for local LLM
 python inference.py \
     --llm gpt-4 \
     --data_dir data_multimedia \
     --temperature 0.2 \
     --top_p 0.1 \
     --api_addr localhost \
     --api_port 4000 \
+    --api_base $YOUR_API_BASE \ # not needed for local LLM
     --api_key $YOUR_API_KEY \
     --multiworker 5 \
     --use_demos 0 \

taskbench/data_engine.py

@@ -32,6 +32,7 @@
 @click.option("--data_dir", type=str, default=None)
 @click.option("--graph_desc", type=str, default=None)
 @click.option("--tool_desc", type=str, default=None)
+@click.option("--api_base", type=str, default=None)
 @click.option("--api_addr", type=str, default="localhost")
 @click.option("--api_port", type=int, default=4000)
 @click.option("--api_key", type=str, default="your api key")
@@ -45,9 +46,12 @@
 @click.option("--llm", type=str, default="gpt-4")
 @click.option("--use_async", type=bool, default=False)
 @click.option("--dependency_type", type=str, default="resource")
-def main(temperature, top_p, check, graph_desc, tool_desc, api_addr, api_port, api_key, play, method, tool_number, number_of_samples, seed, data_dir, save_figure, multiworker, llm, use_async, dependency_type):
+def main(temperature, top_p, check, graph_desc, tool_desc, api_base, api_addr, api_port, api_key, play, method, tool_number, number_of_samples, seed, data_dir, save_figure, multiworker, llm, use_async, dependency_type):
     args = locals()
-    url = f"http://{api_addr}:{api_port}/v1/chat/completions"
+    if api_base is not None:
+        url = f"{api_base}/chat/completions"
+    else:
+        url = f"http://{api_addr}:{api_port}/v1/chat/completions"
     header = {
         "Authorization": f"Bearer {api_key}",
         "Content-Type": "application/json"