fix: main bug (#938)

* feat: parameterize cache paths with USER to avoid conflicts

* guide for missing training_hyperparameters

* guidance for  KeyError: 'concise_reason'

* fixed three bugs in the test

* fix general_model task bug

* fixed some bugs in the med_model scenario

* delete comments

* format with black

* fix mypy error

* fix ruff error

* fix isort error

* sync code

* revert cache_path code

* revert cache_path code

* delete data mining scenario

* fix factor report loop

* fix LiteLLMAPIBackend log_llm_chat_content setting

* refine fin factor report scenario

* remove unused LogColors

* fix UI

* remove medical scenario docs

* change **kaggle** to **data_science**

* remove default dataset_path in create_debug_data

* remove KAGGLE_SETTINGS in kaggle_crawler

* limit litellm versions

* reformat with black

* change README

* fix_data_science_docs

* make hypothesis observations string

* Hiding old versions of kaggle docs

* hidding kaggle agent docs

---------

Co-authored-by: Young <[email protected]>
Co-authored-by: Bowen Xian <[email protected]>
Co-authored-by: yuanteli <[email protected]>

Author: 4 people

README.md

@@ -71,7 +71,7 @@ You can learn more details about **RD-Agent(Q)** through the [paper](https://arx
 | MLE-Bench Results Released | R&D-Agent currently leads as the [top-performing machine learning engineering agent](#-the-best-machine-learning-engineering-agent) on MLE-bench |
 | Support LiteLLM Backend | We now fully support **[LiteLLM](https://github.com/BerriAI/litellm)** as a backend for integration with multiple LLM providers. |
 | General Data Science Agent | [Data Science Agent](https://rdagent.readthedocs.io/en/latest/scens/data_science.html) |
-| Kaggle Scenario release | We release **[Kaggle Agent](https://rdagent.readthedocs.io/en/latest/scens/kaggle_agent.html)**, try the new features!                  |
+| Kaggle Scenario release | We release **[Kaggle Agent](https://rdagent.readthedocs.io/en/latest/scens/data_science.html)**, try the new features!                  |
 | Official WeChat group release  | We created a WeChat group, welcome to join! (🗪[QR Code](https://github.com/microsoft/RD-Agent/issues/880)) |
 | Official Discord release  | We launch our first chatting channel in Discord (🗪[![Chat](https://img.shields.io/badge/chat-discord-blue)](https://discord.gg/ybQ97B6Jjy)) |
 | First release | **R&D-Agent** is released on GitHub |
@@ -249,22 +249,20 @@ The **[🖥️ Live Demo](https://rdagent.azurewebsites.net/)** is implemented b
   > 3. Join the competition: Click `Join the competition` -> `I Understand and Accept` at the bottom of the [competition details page](https://www.kaggle.com/competitions/sf-crime/data).
   ```bash
   # Generally, you can run the Kaggle competition program with the following command:
-  rdagent kaggle --competition <your competition name>
+  rdagent data_science --competition <your competition name>
 
   # Specifically, you need to create a folder for storing competition files (e.g., competition description file, competition datasets, etc.), and configure the path to the folder in your environment. In addition, you need to use chromedriver when you download the competition descriptors, which you can follow for this specific example:
   
   # 1. Install chromedriver.
 
   # 2. Add the competition description file path to the `.env` file.
   mkdir -p ./git_ignore_folder/kaggle_data
-  dotenv set KG_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/kaggle_data"
+  dotenv set DS_LOCAL_DATA_PATH "$(pwd)/git_ignore_folder/kaggle_data"
+  dotenv set DS_IF_USING_MLE_DATA True
 
   # 3. run the application
-  rdagent kaggle --competition sf-crime
+  rdagent data_science --competition sf-crime
   ```
-  > **Description of the above example:** <br />
-  > - Kaggle competition data is roughly divided into three sections: competition description file (json file) and complete dataset for the competition and simplified dataset for the competition. <br />
-  > - The Kaggle competition data will be downloaded automatically, the download process depends on `chromedriver`, installation instructions can be found in the [documentation](https://rdagent.readthedocs.io/en/latest/scens/kaggle_agent.html#example-guide). <br />
 
 ### 🖥️ Monitor the Application Results
 - You can run the following command for our demo program to see the run logs.
@@ -310,7 +308,7 @@ The supported scenarios are listed below:
 | **🩺 Medical**      | 🤖 [Iteratively Proposing Ideas & Evolving](https://rdagent.azurewebsites.net/dmm)[▶️YouTube](https://www.youtube.com/watch?v=VIaSTZuoZg4) | -                                                                                  |
 | **🏭 General**      | 🦾 [Auto paper reading & implementation](https://rdagent.azurewebsites.net/report_model)[▶️YouTube](https://www.youtube.com/watch?v=BiA2SfdKQ7o) <br/> 🤖 Auto Kaggle Model Tuning   | 🤖Auto Kaggle feature Engineering |
 
-- **[RoadMap](https://rdagent.readthedocs.io/en/latest/scens/kaggle_agent.html#roadmap)**: Currently, we are working hard to add new features to the Kaggle scenario.
+- **[RoadMap](https://rdagent.readthedocs.io/en/latest/scens/data_science.html#roadmap)**: Currently, we are working hard to add new features to the Kaggle scenario.
 
 Different scenarios vary in entrance and configuration. Please check the detailed setup tutorial in the scenarios documents.
 

constraints/3.10.txt

@@ -5,3 +5,4 @@ psutil==6.1.0
 rich==13.9.2
 scipy==1.14.1
 tqdm==4.66.5
+litellm==1.72.4

constraints/3.11.txt

@@ -5,3 +5,4 @@ psutil==6.1.0
 rich==13.9.2
 scipy==1.14.1
 tqdm==4.66.5
+litellm==1.72.4

docs/scens/catalog.rst

@@ -26,16 +26,10 @@ The supported scenarios are listed below:
         :ref:`🦾Auto reports reading & implementation <data_copilot_fin>`  
 
         :ref:`🤖Iteratively Proposing Ideas & Evolving <data_agent_fin>`
-    * - 🩺 Medical
-      - :ref:`🤖Iteratively Proposing Ideas & Evolving <model_agent_med>`
-      - *(no data yet)*
     * - 🏭 General
       - :ref:`🦾Auto paper reading & implementation <model_copilot_general>`  
 
-        :ref:`🤖Auto Kaggle Model Tuning <kaggle_agent>`
-      - :ref:`🤖Auto Kaggle feature Engineering <kaggle_agent>`  
-        
-        :ref:`🤖 Data Science <data_science_agent>`
+      - :ref:`🤖 Data Science <data_science_agent>`
 
 
 .. toctree::
@@ -47,8 +41,5 @@ The supported scenarios are listed below:
     data_agent_fin
     data_copilot_fin
     model_agent_fin
-    model_agent_med
     model_copilot_general
-    kaggle_agent
     data_science
-

docs/scens/data_science.rst

@@ -128,7 +128,7 @@ The Data Science Agent is an agent that can automatically perform feature engine
 
     .. code-block:: sh
 
-        rdagent kaggle --competition <Competition ID>
+        rdagent data_science --competition <Competition ID>
 
 - 📥 **Visualize the R&D Process**