How to get more detailed output from the LFRic compile using CCE-fast?

[MichaelWhitall]

Hi folks,

I'm working on a quite-involved lfric_apps branch that really needs to preserve KGOs in order to prove it hasn't broken something. As is often the case, it preserves KGOs except in the rose stem tests that use the "cce_fast-debug" compile. After a lot of agonizing debugging, I've ascertained that the change in answers (which is only at the level of floating-point rounding error) appears in a block of code which I have moved into a different subroutine, but kept identical (along with everything that gets input to it). So this looks like a case of a change in answers caused by the CCE compiler optimisations changing behaviour when the same block of code appears in a different module (just to be sadistic, when I added a certain print statement to the trunk version of the code to see what it was doing, this caused it to change answers and give identical output to my branch!)

I Really want to put the new code structure on the trunk but not have it change answers, so I need to look in more detail at exactly what the Cray compiler is doing with this block of code in my branch versus the trunk.

So... does anyone know how I can alter the compile jobs in rose-stem to output full compiler listings? Then maybe I can figure out what compiler directives I need to add to force the Cray compiler optimisations to behave the same way they did before (I've been down this road before in the UM, but this is my first attempt to do it in LFRic).

Cheers!
Mike

[Adrian-Lock]

Or is there a file containing routine specific and compiler specific settings somewhere (as there was/is in the UM)? Perhaps you've moved you code between routines with different compiler settings and could "simply" ensure the transferred code keeps the same settings?

[MichaelWhitall]

Hi thanks Adrian,

Yeah I hoped it would be that simple! There is a file which sets specific optimisation changes for specific files when using CCE-fast:

applications/lfric_atm/build/fortran/crayftn/fast-debug.mk

But the file where the problematic block of code sits (calc_env_partitions.F90) isn't listed there (I tried grep-searching the whole lfric_apps directory for the filename but didn't find anything except references to it within the comorph source).

So it looks like the problem sits in some code which is just compiled using the CCE-fast default optimisations, and they change behaviour if I so-much-as breath on the code :P

Cheers!
Mike

[t00sa]

The compiler option you're looking for is -h list=a or possibly -h list=ad depending on how much detail you want. The a generates a summary report including loop optimisation information in a .lst file and the d generates a decompiled version of optimised source code in a .opt file.

[MichaelWhitall]

Awesome thanks Sam!

OK so in the file:
applications/lfric_atm/build/fortran/crayftn/fast-debug.mk
I've added a line:
%calc_env_partitions.o: private FFLAGS_EXTRA = -h list=a
(and similar for the file I've moved the problematic area of code into in the branch).
That should do it right? :)

My next question will probably be where should I look for the .lst file output by the LFRic compile? I'm painfully aware that various other files normally output by fcm_make (e.g. the pre-processed source), that I always knew where to find when compiling the UM, just don't seem to be there when I compile LFRic on the hpc. I have no idea where the compile directory actually is! Can anyone offer guidance on where the compile happens / where to find output files from it / how to stop it from housekeeping them or squirreling them away on a temporary disk that vanishes immediately?
(for the case of lfric_atm CCE-fast compile done on exa within the rose-stem infrastructure)

Cheers!
Mike

[MichaelWhitall]

I can confirm that, while the compile happily ran with the -h list=a flag supplied via fast-debug.mk as described above, I can fine no sign of the .lst file that should have been output (searching my cylc-run directories on exab where the compile ran for "*.lst" turns up nothing at all). Where should I look? / what do I need to change to make lfric rose-stem lfric_atm compile jobs output and retain this file?

Cheers!
Mike

[james-bruten-mo]

LFRic builds in a temp directory for performance benefits and so by default these files are removed after the build. I imagine these .lst files will reside in that location (although I might be wrong - Sam can confirm). You can edit the build location temporarily though by changing the BUILD_ROOT variable in rose-stem/site/meto/common/suite_config_ex1a.cylc. If you change it to $CYLC_TASK_WORK_DIR then the build files will appear in the work directory for the build task.

[MichaelWhitall]

That's great thanks James, I'll try that... Presumably this will also make it retain the pre-processed source, making it easier to interpret tracebacks? Probably something I'll want to use quite a bit so many thanks for sharing this :)

Cheers!
Mike