Benchmarks used for `HPOBench: A Collection of Reproducible Multi Fidelity Benchmark Problems for HPO`

For HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO we used the following

existing community benchmarks

family	container version	name in the paper	benchmarks	Reference
nas.nasbench_201	0.0.5	NB201	[Cifar10ValidNasBench201BenchmarkOriginal, Cifar100NasBench201BenchmarkOriginal, ImageNetNasBench201BenchmarkOriginal]	paper
nas.nasbench_101	0.0.4	NB101	[NASCifar10ABenchmark, NASCifar10BBenchmark, NASCifar10CBenchmark]	paper
nas.tabular_benchmarks	0.0.5	NBHPO	[SliceLocalizationBenchmarkOriginal, ProteinStructureBenchmarkOriginal, NavalPropulsionBenchmarkOriginal, ParkinsonsTelemonitoringBenchmarkOriginal]	paper
nas.nasbench_1shot1	0.0.4	NB1SHOT1	[NASBench1shot1SearchSpace1Benchmark, NASBench1shot1SearchSpace2Benchmark, NASBench1shot1SearchSpace3Benchmark]	paper
ml.pybnn	0.0.4	BNN	[BNNOnProteinStructure, BNNOnYearPrediction]	paper
rl.cartpole	0.0.4	Cartpole	[CartpoleReduced]	paper
surrogates.paramnet_benchmark	0.0.4	Net	[ParamNetReducedAdultOnTimeBenchmark, ParamNetReducedHiggsOnTimeBenchmark, ParamNetReducedLetterOnTimeBenchmark, ParamNetReducedMnistOnTimeBenchmark, ParamNetReducedOptdigitsOnTimeBenchmark, ParamNetReducedPokerOnTimeBenchmark]	paper

and new benchmarks

family	container version	name in the paper	benchmarks	Reference
ml.tabular_benchmark	-	LogReg	TabularBenchmark for model = 'lr'	-
ml.tabular_benchmark	-	SVM	TabularBenchmark for model = 'svm'	-
ml.tabular_benchmark	-	XGBoost	TabularBenchmark for model = 'xgb'	-
ml.tabular_benchmark	-	RF	TabularBenchmark for model = 'rf'	-
ml.tabular_benchmark	-	MLP	TabularBenchmark for model = 'nn'	-

We host all code to recreate the experiments in this repo: https://github.com/automl/HPOBenchExperimentUtils

The best-known values for all benchmarks we used to plot regret can be found here: https://github.com/automl/HPOBenchExperimentUtils/blob/master/HPOBenchExperimentUtils/utils/plotting_utils.py