Table of Contents
- About the troject
- Setting up the project
- Reproduce paper results
- Run CAMLEEB
- 1-core clustering AutoML framework execution experiment
- Run the experiment using a HPC SLURM cluster
- Run the experiment using a local setup
- Process the experiment results with HPC SLURM cluster
- Process the 1c experiment results in a local setup
- Clustering AutoML framework with-time-budget inference experiment
- Clustering AutoML 1-million-inferences experiment
Energy efficiency benchmarking for Clustering AutoML frameworks.
Hardware and Software requirements for running the experiments.
Two ways of setting up your environment: HPC or local setup.
- Running the jobs in a High-Performance Computing (HPC) Rocket Cluster managed by SLURM 23.02.8, equipped with an AMD EPYC 7763 64-Core Processor, 16GiB RAM, that runs a Redhat 9.4 OS with Linux Kernel version 5.14.0 for a 64-bit x86 architecture. The Conda 3.8.3 is included as module.
- Conda 3.8.3
- yq version 1.6 or above.
- In theory this should work properly on any PC that runs Linux. However, the system has been tested on a notebook HP 13th Gen Intel(R) Core(TM) i5-1335U, x64-based processor, 1.30 GHz, with 16GB RAM, and an Ubuntu OS 20.04.6 LTS (Focal Fossa). Also on a mini PC Lenovo ThinkCentre M710q, that comes with a tiny Intel 4-Core i3-6100T processor, 8GB RAM, and Debian 12 Linux OS for a 64-but x86 architecture.
- For the experiments done on the mini PC were also considered a power meter Shelly Plug S MTR Gen3 (Type F) for energy measurement.
- Lastest stable version of Docker. Important: make Docker run without root previleges:
sudo groupadd docker
sudo groupadd docker
newgrp docker- yq version 4.43.1 or above.
- gdown version 5.2.0. It may be used PIP.
- virtualenv version 20.17.1
- On the HPC SLURM cluster, the following module command imports requested modules. Search Conda 3.8.3 and find the path to the module.
module av conda- Replace the conda module's path line 40 of file schedule_slurm_job.sh. Ex:
module load any/python/3.8.3-conda- Clone the github repo and save the content in a specific folder, for example: /path/to/my_project/
- Download the related_work_offline_opt.csv file for the AutoCluster framework
cd /path/to/my_project
curl -L -o ./frameworks/autocluster/Experiments/RelatedWork/related_work/related_work_offline_opt.csv https://drive.google.com/uc?export=download&id=1Q9BGZ1vqwj-k4fsuonAOYfJ_sC-lycbQ- Or justdonwload the file and paste it in folder /<my_project>/frameworks/autocluster/Experiments/RelatedWork/related_work/
To run all the notebooks properly, your JupyterLab virtual environment should include the following packages:
- numpy
- pandas
- matplotlib
- seaborn
- autorank
- scikit-learn
Got to file ./experiments/reproduce_paper_results and find notebooks and python scripts for each figure listed by number of Figure. Just replace project_path variable.
project_path = '/path/to/my/project'- Fig. 1: Impact of search time and ARI on energy consumption.
- fig1a.ipynb and fig1b.ipynb
- Fig. 2: Comparing automl systems with respect to energy consumption for different scales of clustered assignment:
- fig2.py and fig2.sh. Just run the following command:
./fig2.sh- Fig. 3: CD plot for ARI and energy consumption during AutoML framework execution and cluster assignment:
- fig3.ipynb
- Fig. 4: Bradley-Terry tree of depth 5 for paired comparison between the AutoML framework's ARI results:
- fig4.1.ipynb, it outputs four CSV files that should be added to fig4.2.ipynb: ec_fmk_paired_comparisons.csv, ec_fmk_paired_comp_covs.csv, and fmk_paired_comparisons.csv, fmk_paired_comp_covs.csv.
- fig4.2.ipynb, it requires an R language kernel. It works by default with its own CSV files, but you may replace them with generated ones when running fig4.1.ipynb. It should look as follows:
# First paragraph line 13 and 14
fmk_pc <- read.csv(file="fmk_paired_comparisons.csv", row.names=1)
fmk_pc <- read.csv(file="fmk_paired_comp_covs.csv", row.names=1)
# Second paragraph line 13 and 14
fmk_pc <- read.csv(file="ec_fmk_paired_comparisons.csv", row.names=1)
fmk_pc <- read.csv(file="ec_fmk_paired_comp_covs.csv", row.names=1)- Fig. 5: Bradley-Terry tree of depth 5 for paired comparison between the AutoML framework's EC results:
- Same as Fig. 4.
- Fig. 6: Energy consumption versus clustering performance (ARI) of AutoML frameworks across varying cluster shapes:
- fig6.ipynb
- Fig. 7: ARI and CPU energy consumption during execution on cSmartML across CPU cores:
- fig7.ipynb
- Fig. 8: Energy consumption comparison between codecarbon measurements of experiments on an HPC and a mini PC, and real values obtained using a power meter attached to the mini PC:
- fig8.ipynb, it takes about 40mins, so be patient.
- Fig. 9 Energy consumption for AutoML frameworks measured with CodeCarbon and Power meter, 1min STB:
- fig9.ipynb
- Fig. 10: CPU utilization versus energy consumption for mini PC and HPC platforms under 1 and 10-minute STBs.:
- fig10.ipynb
- Table 5: Cost of 1 Trillion clustered instances across AutoML Clustering frameworks:
- table5and6.ipynb
- Table 6: Actual execution time for specified time budgets:
- table5and6.ipynb
The system that runs the benchmarking as being dubbed CAMLEEB (Clustering AutoML Energy Efficiency Benchmarking), so for you to run it just consider how the YAML files should be written. There is two YAML file types.
- Setting up the project: you are able to run experiments in a sequential and parallel way. With sequential expeiments, you run one experiment at a time. On the other side, you may want to run multiple experiments in a single run, that is using multiple cores of a node, so you should set your YAML project file to run experiment in parallel.
- Setting up the experiment: through this file you set proper parameters up for running the experiment. Make sure to add your project path to project_directory parameter. If you want to add power meter measurements, you should set parameter pm_sampling as true and also adding pm_api_service_key and pm_device_id of the Shelly power meter. Parameter all_datasets includes all datasets saved in dataset_directory, otherwise it might be set as false, so parameter dataset_names should include dataset names saved in dataset_directory separated by commas. In case you want to run experiments on a HPC, parameter hpc_or_pc should be hpc.
Either HPC cluster or a local setup may be used for running this experiment.
- After cloninig the underlying github repo, download the meta-knowledge csv file of the AutoCluster framework using the following link and paste it in the following folder /path/to/my_project/autocluster/autocluster/Experiments/RelatedWork/related_work
- Execute the following command in a terminal to open the folder where experiment files are located.
cd experiments- You have the following YAML files to edit, refer to project_directory, slurm_account and partition parameters:
- hpc_detailed_consumption_complete_exp.yaml: to complete experiments (all frameworks) using a single core.
- hpc_detailed_consumption_realdata_exp.yaml: to run AutoML4Clust and ML2DAC frameworks only with real data.
- hpc_detailed_consumption_synthdata_exp.yaml: to run AutoML4Clust and ML2DAC frameworks only with synthetic data.
-
You also have YAML files in folder ./experiments/experiment_yamls/minipc_synthd_autom_runs, this are meant to configure experiments on mini PC for each framework.
-
Modify the file project configuration file compound_round_1.yaml to run experiments in parallel. So for running experiments in parallel of all frameworks add the corresponding YAML path to parameter yaml_file_abs_path and set single as true, and multiple as false; for instance:
project_info:
- project_name: hpc_complete_runs
multiple_exps:
- multiple: false
- yaml_dir_abs_path: _
- run_times: 0
single_exp:
- single: true
- yaml_file_abs_path: /experiments/experiment_yamls/hpc_detailed_consumption_complete_exp.yaml- Same for other YAML file condifurations for HPC experiments.
- Experiments using the mini PC using a power meter, convenient is to run experiments sequentially so you make sure you have proper measurement on the power meter for one clustering AutoML framework at a time.
project_info:
- project_name: minipc_synthd_compound_runs
multiple_exps:
- multiple: true
- yaml_dir_abs_path: ./experiments/experiment_yamls/minipc_synthd_autom_runs
- run_times: 5
single_exp:
- single: false
- yaml_file_abs_path: _- You can run the experiment executing the following command:
./run_camleeb.sh- Got to your project directory (where you cloned the project and do similar step to download the AutoCluster metaknowledge file).
cd "/path/to/my/project/directory"- Same as running an experiment on the HPC, use YAML files and ./run_camleeb.sh script to run experiments on mini PC.
This ETL process pick all the .csv files from the resulting 1-core experiments, basically files like <framework_name>_experiments.csv and emissions.csv. These files are processed by the python script experiments_etl.py that ouputs a number of images and csv files of the ETL results in the unified_experiments folder.
- Open experiments folder
cd experiments- Provide file permissions to run the shell script.
chmod +x ./run_experiments_etl_job.sh- Start processing the experiment results. You should consider modifiying the following variables in sh file:
- env_type: variable that refer to running experiment on hpc or pc (same as mini pc).
- wdirectory: folder in which all the results of the experiment are saved.
- etlresults: folder to save ETL results.
./run_experiments_etl_job.sh $log_folder- This is pretty much similar as running in HPC (previous section).
Two ways of running this experiment, with HPC and in a local setup. As for the 1-core experiments, you must make sure that the Conda module any/python/3.8.3-conda works for your SLURM cluster in the following scheduler script. Also, make sure to change the account and partition names of the SLURM configuration in the scheduler script. This experiment outputs the csv file bfdf_iwtbe_res.csv with ARI and carbon emission results.
- Open the experiments folder
cd experiments- Grant permissions to the scheduler script.
chmod +x ./schedule_inference_wth_timebudget_experiment.sh- Run the the scheduler script. Note that it uses the environment variable of the 1-core execution experiment. Also, add you slurm account. Results of this job will be stored in current folder. The env variable $input_csv_file should be *best_framework_configs_per_tb.csv" generated by running the 1-core execution experiment.
./schedule_inference_wth_timebudget_experiment.sh $project_directory $my_slurm_account $input_csv_file- Open the experiments folder
cd experiments- Activate virtual environment etl_env that should be available running the following command.
source ../libs/virtualenv/etl_env/bin/activate- Run the python script of the experiment. It should be the csv file best_framework_configs_per_tb.csv that is generated after running the 1-core execution experiment.
python ./inference_wth_timebudget_experiment.py -f best_framework_configs_per_tb.csv -o . -d $project_directory This experiment is only designed to run on a HPC as it requires multiple cores and takes more computational resources.
- Open the experiments folder
cd experiments- Grant permissions to the experiment shell script file.
chmod +x ./trigger_1m_clust_asgmnt_exp_runner.sh
chmod +x ./1m_clust_asgmnt_exp_runner.sh
chmod +x ./schedule_1m_clust_asgmnt_exp.sh
chmod +x ./1m_clust_asgmnt_exp.sh- The environment variables and conda module should be properly setup as described in the first experiment.
- Run the experiment shell script for 72hrs repeating 10 times the whole process.
./trigger_1m_clust_asgmnt_exp_runner.sh- Results <framework_name>_experiment.csv and emissions.csv are saved in the folder with path /path/to/my_project_folder/experiments/million_pred_res/experiments/<framework_name>/<time_budget>
You can run this only utilizing a local setup. The script run in this ETL job outputs a number of images and a csv file called 1_million_exp_res_df.csv in the current folder experiments.
- Open the experiments folder
cd experiments- Run the shell script of the ETL process for the underlying experiment. The path in which the results were saved should be append to the command. Go to the folder /path/to/my_project_folder/experiments/trillion_pred_res and search for the proper experimnet result folder like experiments.
path_to_my_experiment_result=/path/to/my_project_folder/unified_experiments/trillion_pred_res/experiments<date-time>
python ./1_million_cluster_asgmnt_exp_etl.py -f $path_to_my_experiment_result -o ./