CASH Simulation for post-hoc ensembling

[Heptasphere]

Hello @LennartPurucker and @Innixma,

I'm interested in simulating a CASH run from a subset of your experiment results, mostly to create the right EvaluationRepositoryCollection for some heterogeneous post-hoc ensembling.

I can't find in the code whether you've already implemented such a simulator.

Hence my questions:

• Have you already implemented such a simulator ?
• If yes, where should I look for it ?
• If no, which utilities would be useful to look into and do you have any advice ?

Thanks for any answer.

(By the way, this project rocks ! Thanks to the whole team for making it.)

[Innixma]

Hi @Heptasphere! Thanks for your interest in the project!

To plot the results of the already simulated CASH runs, you can run this script:

https://github.com/autogluon/tabarena/blob/main/examples/plots/run_plot_pareto_over_tuning_time.py

To evaluate a single ensemble after initializing an EvaluationRepositoryCollection, you can call:

EvaluationRepositoryCollection.evaluate_ensemble as shown here: https://github.com/autogluon/tabarena/blob/main/examples/meta/inspect_processed_data.py#L172-L178

All other techniques for calculating results revolve around calling the evaluate_ensemble method in various ways.
If you wanted to simulate approaches other than random search, you would write your own logic to select the configs that get passed into evaluate_ensemble.

For re-computing from scratch the simulated CASH runs for a given method, you can do:

hpo_trajectories_official = method_metadata.load_hpo_trajectories()  # cached official trajectories
# do this for each method to reproduce the tuning trajectory plots from scratch
hpo_trajectories = method_metadata.generate_hpo_trajectories(cache=True)  # Note: takes 2 hours per method on a beefy machine (96 vCPU, 384 GB VRAM), due to repeating with 20 different seeds for stability.

(https://github.com/autogluon/tabarena/blob/main/tabarena/tabarena/nips2025_utils/artifacts/method_metadata.py#L627)

Multi-family CASH simulation

Heterogenous CASH simulation is still being refined from an API perspective, so expect things to change in future. The current code is what I wrote for a recent ICML 2026 submission.

To do multi-family CASH, refer to TabArenaContext.combine_hpo:

https://github.com/autogluon/tabarena/blob/main/tabarena/tabarena/nips2025_utils/tabarena_context.py#L207

from tabarena.nips2025_utils.tabarena_context import TabArenaContext
tabarena_context = TabArenaContext()
hpo_trajectories_joint = tabarena_context.combine_hpo(methods=["LightGBM", "XGBoost"], new_config_type="GBM_XGB", ta_name="gbm_x_xgb", ta_suite="gbm_x_xgb")

Note that the mainline version of this method is missing some quality of life arguments like n_configs, time_limit, etc. This will be merged in soon over the course of the next few weeks.

To do multi-family CASH w/ tuning trajectories, refer to TabArenaContext.generate_hpo_trajectories:

This is now getting quite advanced, so I'll show a script I recently wrote that used this to merge method results together and calculate CASH trajectories: https://github.com/atschalz/tabarena/blob/icml/examples/icml2026/run_prep_tuning_trajectories.py

Note this comes from a WIP branch and the script won't work out of the box, but you can use it as a guideline for how to call TabArenaContext.generate_hpo_trajectories.

Cheers!