Error tracebacks

[tommbendall]

I thought this would be worth starting a separate discussion, but this relates to the comments in this discussion: #371 (comment)

Model failures with the CCE compiler on the EX machines don't seem to provide tracebacks in the job.err files, which obviously presents a challenge for debugging!

In that other discussion it was suggested that using ATP could be a solution. Is this something that is feasible to include in rose-stem as standard? @stevemullerworth suggested it might be slow too... Or are there other possible ways of getting tracebacks?

[andrewcoughtrie]

If you're talking about the traceback that is returned when a log_event(LOG_LEVEL_ERROR, ...) is called then from what I remember when I implemented the functionality to return that there wasn't an option for the Cray compiler, this may change in the future with the new Fortran frontend if Cray pick it up.

[tommbendall]

Thanks for replying. I mean when the model fails ungracefully (e.g. a seg fault or arithmetic precision error), unlike the Gnu compiler or the Intel on the XC machines, the Cray compiler doesn't report any traceback, it just lists something like "Signal X error" which isn't very helpful. Adding ATP provides a traceback though, but it's annoying to have to recompile and restart a job once it has failed just to get a traceback

[mo-mguidolin]

Hi Tom. You can compile with ATP and have the followng env variable set (which if I don;t remeber wrong is the default):

export ATP_ENABLED=0

In this case ATP is not used and it should have no impact on your workflow. Then in case of a failure, you just need to set the env variable to 1.

By the way, ATP has a very small performance impact also if it is active all the time (only if a signal is generated).