Paper List for AutoML & LLM

Contents

• Paper List for AutoML & LLM
  • Contents
  • Introduction
    • Keywords Convention
  • LLM for AutoML
    • Survey
    • System
      • Search Space Compression
      • FE
    • BO
  • AutoML for LLM
    • PEFT

Introduction

This is a paper list (working in progress) about AutoML and LLM

Keywords Convention

abbreviation

main feature

conference

LLM for AutoML

Survey

1. AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks. , , 2023.06 Alexander Tornede, Difan Deng, Theresa Eimer, Joseph Giovanelli, Aditya Mohan, Tim Ruhkopf, Sarah Segel, Daphne Theodorakopoulos, Tanja Tornede, Henning Wachsmuth, Marius Lindauer. [pdf]

2. Large Language Models for Data Science: A Survey , , 2025.06 Song Wang, Zhen Tan, Zihan Chen, Dawei Li, Yaochen Zhu, Bohan Jiang, Yinhan He, Chengshuai Zhao, Zhenyu Lei, Paras Sheth, Lichi Li, Lo Pang-Yun Ting, Jundong Li, Huan Liu. [pdf], [project]

System

1. AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions. , , 2024.11 Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tuney Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang, Jiaheng Liu, Wanjun Zhong, Wangchunshu Zhou, Wenhao Huang, Ge Zhang. [pdf], [project]

2. AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML. , , 2025.06 Patara Trirat, Wonyong Jeong, Sung Ju Hwang. [pdf], [project]

3. SELA: TREE-SEARCH ENHANCED LLM AGENTS FOR AUTOMATED MACHINE LEARNING. , , 2024.10 Yizhou Chi, Yizhang Lin, Sirui Hong, Duyi Pan, Yaying Fei, Guanghao Mei, Bangbang Liu, Tianqi Pang, Jacky Kwok, Ceyao Zhang, Bang Liu, Chenglin Wu. [pdf], [project]

   • (Related further work) AFlow: Automating Agentic Workflow Generation. , , 2025.01 Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, Chenglin Wu. [pdf], [project]
   • (Related previous work) Data Interpreter: An LLM Agent For Data Science. , , 2024.02 Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zhibin Gou, Zongze Xu, Chenglin Wu. [pdf], [project]
   • (Related previous work) MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. , , 2024.01 Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, Jürgen Schmidhuber [pdf], [project]

4. AgentHPO: Large Language Model Agent for Hyper-Parameter Optimization. , , 2025.02 Siyi Liu, Chen Gao, Yong Li. [pdf]

Search Space Compression

1. Design Principle Transfer in Neural Architecture Search via Large Language Models , Xun Zhou, Xingyu Wu, Liang Feng, Zhichao Lu, Kay Chen Tan [pdf]

FE

1. Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. , Noah Hollmann, Samuel Müller, Frank Hutter. [pdf], [project]

BO

1. Position: Leverage Foundational Models for Black-Box Optimization. ,Xingyou Song, Yingtao Tian, Robert Tjarko Lange, Chansoo Lee, Yujin Tang, Yutian Chen. [pdf]
2. Large Language Models as Optimizers. ,Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, Xinyun Chen (Google DeepMind).[pdf],[project]
3. Large Language Models to Enhance Bayesian Optimization. ,Tennison Liu, Nicolás Astorga, Nabeel Seedat, Mihaela van der Schaar. [pdf],[project]
4. 𝙻𝙻𝙸𝙽𝙱𝙾: Trustworthy LLM-in-the-Loop Bayesian Optimization. ,Chih-Yu Chang, Milad Azvar, Chinedum Okwudire, Raed Al Kontar. [pdf],[project]
5. Searching For Optimal Solutions With LLMs Via Baysian Optimization. ,Dhruv Agarwal, Manoj Ghuhan Arivazhagan, Rajarshi Das, Sandesh Swamy, Sopan Khosla, Rashmi Gangadharaiah.[pdf]
6. A Sober Look at LLMs for Material Discovery-Are They Actually Good for Bayesian Optimization Over Molecules? ,Agustinus Kristiadi, Felix Strieth-Kalthoff, Marta Skreta, Pascal Poupart, Alán Aspuru-Guzik, Geoff Pleiss. [pdf],[project]
7. Language-Based Bayesian Optimization Research Assistant (BORA). ,Abdoulatif Cissé, Xenophon Evangelopoulos, Vladimir V. Gusev, Andrew I. Cooper.[pdf],[project]
8. Reasoning BO: Enhancing Bayesian Optimization with Long-Context Reasoning Power of LLMs.,Zhuo Yang, Lingli Ge, Dong Han, Tianfan Fu, Yuqiang Li.[pdf],[project]
9. Large Scale Multi-Task Bayesian Optimization with Large Language Models.,Yimeng Zeng, Natalie Maus, Haydn Thomas Jones, Jeffrey Tao, Fangping Wan, Marcelo Der Torossian Torres, Cesar de la Fuente-Nunez, Ryan Marcus, Osbert Bastani, Jacob R. Gardner.[pdf]
10. Evolve Cost-aware Acquisition Functions Using Large Language Models. ,Yiming Yao, Fei Liu, Ji Cheng, Qingfu Zhang. [pdf],[project]
11. FunBO:Discovering Acquisition Functions for Bayesian Optimization with FunSearch.,Virginia Aglietti, Ira Ktena, Jessica Schrouff, Eleni Sgouritsa, Francisco J. R. Ruiz, Alan Malek, Alexis Bellot, Silvia Chiappa.[pdf]
12. Active Learning with LLMs for Partially Observed and Cost-Aware Scenarios. ,Yiming Yao, Fei Liu, Ji Cheng, Qingfu Zhang.[pdf],[project]

AutoML for LLM

PEFT

1. AutoAdapt: On the Application of AutoML for Parameter-Efficient Fine-Tuning of Pre-Trained Code Models. ,, 2025.04 Amal Akli, Maxime Cordy, Mike Papadakis, Yves Le Traon.[pdf],[project]