Running ADF on NIRD

[adagj]

The new atmospheric diagnostics package can be installed using conda. However, using a more recent conda version can lead to extended time (i.e. days) resolving dependencies. A more efficient approach is utilizing the conda module on NIRD. This method bypasses the need for personal conda installation and bashrc modifications, aligning with sigma2's recommendation against adding conda configurations to bashrc. I tested this method by removing my conda installation, and it proved efficient and effective:

1. On NIRD, load the conda module: $ module load Anaconda3/2022.05
2. create a directory where the conda files can be stored:
   $ mkdir /projects/NS____K/user/conda
3. set that as your conda storage space:
   $ my_conda_storage=/cluster/projects/NS____K/user/conda
$ export CONDA_PKGS_DIRS=${my_conda_storage}/package-cache
4. Clone the ADF repository:
   $ git clone https://github.com/NCAR/ADF ADF
$ cd ADF
5. create the conda environment (this will probably take some time):
   $ conda env create -f env/conda_environment.yaml
6. I needed to do a new git pull at this point:
   $ git pull

Steps 1 through 5 are necessary only once. Each subsequent time you want to run the ADF diagnostics, you can skip these steps.

For each time you want to run the ADF diagnostics:
7. Activate the environment. The first time you need to log out and in of NIRD. Then:
$ module load Anaconda3/2022.05
$ source ${EBROOTANACONDA3}/bin/activate
$ conda activate adf_v0.11

8. Running the diagnostics: ./run_adf_diag config_yourconfigname.yaml

9. To see some examples of the config_yourconfigname.yaml file, please have a look at /datalake/NS2345K/adagj/ADFout/config_ada_test.yaml or /datalake/NS2345K/mvertens/ADF/input_file//config_test_ghgmam4.yaml

10. For making the diagnostics available for all, please use the folder /nird/datapeak/NS9560K/www/diagnostics/. To be set in cam_diag_plot_loc in the config file. Output can be view from https://ns9560k.web.sigma2.no/datapeak/diagnostics/.

We recommend to run on the ipcc node in order to have sufficient available memory.

Useful links:

• https://justin-richling.github.io/ADF-Tutorial/README.html
• https://github.com/NCAR/ADF
• https://documentation.sigma2.no/software/userinstallsw/conda.html

[adagj]

Improvements might involve:

• Shared directory for observations. The observations used by the ADF diagnostics are located here at the moment: /projects/NS9560K/mvertens/obs_data/ADF_obs
• ADF repository for NorESM including specific variables from Oslo-Aero. Is the best option to make a fork?
• Add additional observations

[oyvindseland]

Does the diagnostics require the use of one specific variable or can it be a sum of variables. One example both for MAM4 and Oslo-Aero is the total amount of a chemical compound, e.g. sulfate. This can be found by summing up several tracers.

[Ovewh]

Cool Thanks Ada. Just, a comment no need for everyone install their own conda environment for running ADF. Just load the Anaconda module as you described and then if we agree on a location for the ADF conda environment one can just do, conda activate /path/to/adf/environment/. Having many conda environments can easily fill up the project space with too many files.

Also might be worth looking into running the diagnostics directly on the preproc queue on betzy and then copy output to nird? Atleast then there should be enough memory.

[adagj]

Tomas Torsvik is preparing the common conda environment, so that will be ready over the summer.

[Ovewh]

Sounds good, ok until that is ready, i think it is better to install conda environments under /nird/scratch/username. Not in a project directory.

[DirkOlivie]

@adagj
Hi Ada, I would like to run the ADF for the NorESM2.3 simulations. Is the recipe described here still the one to be followed to run the ADF? Or is there another place where it is described?
Currently I have some issues with:

• Solving the environment is taking already more than 1 hour. Is this exptected? Or has Tomas made a common environment?
• Passing the name of the experiment to be analyzed, does that happen via the config_yourconfigname.yaml file?
• Is the main branch of https://github.com/NorESMhub/ADF/ containing the modifications to treat specific CAM-OSLO variables (cb_OM, cb_SULFATE, ...)?
  Best regards, Dirk

[adagj]

Hi @DirkOlivie
Here is the noresm brancg you can use for the ADF https://github.com/NorESMhub/ADF/tree/adf_noresm_stuff

[adagj]

Hi @DirkOlivie

Yes, I had problems with the environment taking so long. That's why I wrote this recipe, because that worked for me. @YanchunHe Can you let Dirk know what's the best way to intall ADF or if it is a common environment installed on NIRD?
I need to merge my updates for you to run with column burdens and emissions. Will do that asap!

[maritsandstad]

@adagj and @YanchunHe , I guess we discussed this in the diagnostic meeting before Christmas and will discuss again in upcoming meeting (when will that be). But though having an environment definition to build from is good, a prebuilt environment for most users to just load and use so using diagnostics is more plug and play is probably what we want, right?

[adagj]

Yes! That is exactly what we want!

[maritsandstad]

For now, one solution can just be that you move/copy the environment that works for you for ADF, @adagj, to somewhere where everyone can load it, and then suggest a shortcurt for running that just includes loading Anaconda and the environment, these things can even be put in a bash-script or similar like I did for the land-diagnostic. Not terribly sophisticated, but it works quite well without requiring whoever wants to run ADF to build the environment, or indeed understand much of the setup.

[adagj]

I have made a new branch in the ADF, https://github.com/NorESMhub/ADF/tree/adf_noresm_stuff, with the noresm specific additions I have made. I suggest the main branch can follow the development at NCAR

[oyvindseland]

Any timing estimate for point 5? It did not finished within one hour. I stopped the processs and thought to try it on the IPCC node but that node does not have access to datalake. Returned to the standard nodes and will just run the set-up overnight.

[adagj]

That's not correct. The ipcc node has access to the datalake. E.g. /projects/NS9560K-datalake/

[adagj]

@YanchunHe has already installed an ADF version at NIRD available for all ines2 users, which probably is easier to use. Maybe he can help you with a recipe?

[oyvindseland]

OK , so the IPCC node has a different path to datalake.

[oyvindseland]

Point 5 has now taken more than 12 hours and still not finished

[YanchunHe]

The ADF is installed under:

IPCC node (ssh ipcc.nird.sigma2.no)

IPCC: /diagnostics/ADF

One can activate with:

conda activate /diagnostics/ADF/envs/adf_v0.11

The source code with currently Ada's branch activate at:

IPCC: /diagnostics/ADF/src

We will try to come up with a better recipe on how to run it soon later.

[oyvindseland]

I can confirm that the settings given by Yanchun works, thank you.
While waiting for a more detailed recipe, I just wanted to add a couple of points to the list.

Copy the file /diagnostics/ADF/src/config_noresm_default.yaml to a /my_local_directory/config_noresm_mycase.yaml

Replace the given username (adagj) with your username.

For model vs model comparison give your set the variable
cam_case_name twice, the upper for test the lower for control

set path to files if different from default:
cam_hist_loc: /projects/NS9560K-datalake/noresm3/cases

For every simulation set start_year and end_year No input = whole timeseries

Choose type of plots. Default only have tables and time-series.
Search for the term "List of plotting scripts being used" and edit in more categories as needed.

Run: /diagnostics/ADF/src/run_adf_diag --config-file /my_local_directory/config_noresm_mycase.yaml

Minor inconvenience: If you re-run the script with a different time-period you need to delete the netcdf files created for the timeseries.
ADFout/$casename/tseries/
The climo files used for other types of plots will be overwritten. The plots will be on a different catalogue so the original plots remains.

[YanchunHe]

Thanks @oyvindseland

So I made also a sample yaml config file config_noresm_ipcc.yaml (referred to config_noresm_default.yaml).

One needs to change the header 'your_user_name' once and some other case related settings.

So the shortest workflow is to:

1. conda activate /diagnostics/ADF/envs/adf_v0.11
2. cd /diagnostics/ADF/src
3. cp /diagnostics/ADF/src/config_noresm_ipcc.yaml /path/to/myconfig_for_a_case.yaml
4. ./run_adf_diag /path/to/myconfig_for_a_case.yaml