NaN trapping on ex1a

[iboutle]

I'm trying to track down where/how a NaN is generated in lfric_atm, but it would appear "f_shum_is_nan" doesn't work on the ex1a - is this known, and does anyone have any suggestions of how we can trap NaNs on the ex1a CCE compiler?

To be more specific, I have the following code:

  sum = 0.0_r_def
  count = 0
  do l = 1, n_profile
    do k = 1, n_surf_tile
      sum = sum + lw_diag%flux_up_tile(k,l)
      count = count + 1
      if (f_shum_is_nan(lw_diag%flux_up_tile(k,l))) then
        write(10,*)'IAB Nan here ',k,l,lw_diag%flux_up_tile(k,l)
      end if
    end do
  end do
  write(10,*)'IAB sum ',sum
  write(10,*)'IAB ave ',sum/real(count,r_def)
  write(10,*)'IAB min/max ',minval(lw_diag%flux_up_tile),maxval(lw_diag%flux_up_tile)

which returns the following output:

 IAB sum  NaN
 IAB ave  NaN
 IAB min/max  1.7820678,  454.95816

so the sum over the field is clearly showing a NaN is present, but f_shum_is_nan has never returned true, making it hard to track down exactly which l & k index the NaN exists at.

Does anyone have any clever ideas on how else to find where it is?

[stevemullerworth]

LFRic uses the ieee_arithmetic module to control how NaN values are handled, and it has commands for detecting NaN values.

https://fortranwiki.org/fortran/show/ieee_arithmetic

use, intrinsic :: ieee_arithmetic, only: ieee_is_nan

if (ieee_is_nan(x)) print *,'x is NaN'

I just checked it and it works as expected, returning true for both signalling and quiet NaN values.

[stevemullerworth]

PS Shumlib looks like it ought to use the same function if it is built with the right set of options...I will query this.

[iboutle]

Thanks Steve, will give it a try.

Yes, I guess it's an open question whether we still want to use shumlib for stuff like this in LFRic, or develop our own tools - I'd used the shumlib code fine in LFRic on the xc40, which is why it was my initial go-to, but obviously whatever set of flags/options were being used there are different on the ex1a (I've not tried to figure out which method it's actually trying to use!)

[stevemullerworth]

I did not realise till now that the Shumlib code is extracted as part of an lfric_atm build (though it is not used by lfric_atm). If you add a use statement to use the routines, they are built with the standard lfric_atm compile options, and these don't include the Shumlib-configured defs to make this NaN-detection function robust (in this case "HAS_IEEE_ARITHMETIC").

We'll have a conversation in the SimIT teams about whether lfric_atm should use Shumlib as a library - in the meantime, using its functions look risky.

[AdamVoysey]

As the original author of the Shumlib code for this I should add some insight. The reason Shumlib does not default to the HAS_IEEE_ARITHMETIC version is two-fold:

1. Some more obscure compilers had issues with it working correctly
2. (more importantly) The ieee_is_nan code is (was?) orders of magnitude slower with the Cray complier than the Shumlib EVAL_NAN_BY_BITS version.

For this reason we built Shumlib with both HAS_IEEE_ARITHMETIC and EVAL_NAN_BY_BITS on the XC40s.

I have not done the measurement for the EX machines, so don't know if this is still true on the newer compiler versions there. But this meant we defaulted to EVAL_NAN_BY_BITS for the XC40s.

[stevemullerworth]

Thanks for the context, Adam. We will have to think about this. We probably don't want core LFRic infrastructure to be dependent on Shumlib. Our current use of NaN values is only to enforce fields to be initialised with NaN when fast or full-debug settings are applied - I'm assuming these array initialisations would not be a performance issue as there are just a few calls to the IEEE lib to get the NaN value and switch off numerical checking while the field array initialisation takes place.

But we could properly include Shumlib in lfric_atm by linking to the installed libs so it gets the right defs. That would mean that lfric_atm developers see a consistent API relative to other Met Office apps. We can also try a quick fix of setting the defs in the lfric_atm build environment.

[MichaelWhitall]

Hi folks, I didn't know about the shumlib "is this number NaN" function, but I've always checked for bad values by just writing something like:

IF ( .NOT. ( x >= -HUGE(x) .AND. x <= HUGE(x) ) ) THEN
  WRITE(*,*) "Bad value detected...", x
END IF

I reckon this reliably flags NaNs in a portable way (and as a bonus, it also flags "Inf" values which are distinct from NaN with some compilers). The relational comparisons will always evaluate to false if x is not a valid number; then the .NOT. flips it to true... You can also easily change the min-allowed lower bound to zero instead of -HUGE, if you want to also flag negative values of things that should never be negative.

Cheers!
Mike

[stevemullerworth]

The ieee functionality also includes a function, ieee_is_normal for identifying "normal" values (not NaN nor infinity nor "denormal") or ieee_class that returns the type of a number.

https://fortranwiki.org/fortran/show/ieee_arithmetic

[iboutle]

A follow up to this - does anyone know how to get a useful traceback out of the ex1a CCE compiler?

Having been unable to track down where the NaN is coming from by printing input/output fields to the failing scheme (inputs look fine, outputs contain a NaN), I've switched to compiling with full-debug in the hope of getting the model to fail at the point of generating the NaN.

Sure enough it does fail where I expect it to, but the only output is:

nidb1341: rank 607 died from signal 8
nidb1349: rank 1185 died from signal 15
[FAIL] $CORE_ROOT_DIR/bin/tweak_iodef; \
[FAIL] $LAUNCH_SCRIPT/launch-exe # return-code=143
2025-04-14T13:10:37Z CRITICAL - failed/EXIT

(https://cylchub/services/cylc-review/view/ian.boutle?&suite=vn2.1_gc6_package%2Frun3&no_fuzzy_time=0&path=log/job/1/run_lfric_atm_nwp_gal9_copy-C224_MG_ex1a_cce_full-debug-64bit-rbl32-rdef32/02/job.err)

which is no real use. Is there a way to get routines/line-numbers of where the failure happens?

Thanks!

[mo-saracusworth]

It's not in the default environment because it (ATP) a bit of a mess on the new system but you could try this for file and line numbers on crashes:
https://www-hpc/ex-user-guide/dev-tools/debugging/#abnormal-termination-processing

[iboutle]

Amazing, thanks - that's much more helpful!!

[iboutle]

Hi Sara,

Is there an updated version of that link above, now old-spice has disappeared?

Also, is there any plan to include this at standard in rose-stem etc, as it's a complete pain trying to debug anything with no helpful traceback?

Cheers

Ian

[mo-saracusworth]

Here's a link for AzSpice
https://wwwspice/~hpcshared/ex-user-guide/dev-tools/debugging/#abnormal-termination-processing

I'll leave the question about the use of ATP in rose stem for the software system maintainers

[stevemullerworth]

As a followup for those interested, the shumlib code is now being included in lfric_atm as a library. Use of functions such as f_shum_is_nan should now work correctly on all platforms that support shumlib whereas before they probably didn't as the lfric_atm build options did not specify the platform-specific DEFS that shumlib requires.

This change has just gone on trunk so should be available from lfric_apps vn2.2.

Note that to use a shumlib function, the module it contains has to be listed in the IGNORE_DEPENDENCIES list in the lfric_atm Makefile so that the build system doesn't go hunting for it in the codebase. The changeset has included IGNORE_DEPENDENCIES settings just for the numerical checking modules of shumlib that are likely to be useful in LFRic (much of shumlib is UM specific).

https://code.metoffice.gov.uk/trac/lfric_apps/browser/main/trunk/applications/lfric_atm/Makefile?rev=11868#L27

It would be relatively easy to copy functionality to use in gungho_model and other application builds etc. but it is not being done by default.