NaN results with specific CMT file

[amkearns-usgs]

So I have been tasked with setting up a specfem3D globe deployment (GPU-based) for my organization/client. We have attempted to test its effectiveness in generating synthetic data with a specific low-magnitude event from 2024. (See the attached CMTSolution file.) We get NaN results in the generated synthetic data, and are not sure what the cause of it is; before this test, all our results have come from the QuickCMT set as distributed by Columbia's earth observatory (http://www.ldeo.columbia.edu/~gcmt/projects/CMT/catalog/NEW_QUICK/qcmt.ndk0).

Our primary researcher has suggested it could be an issue with the formatting of the CMTSOLUTION file itself, but I have seen no evidence of this (and the program seems to run normally). Since we're running the GPU-based implementation, there doesn't appear to be any useful debugging information printed to terminal or a file, and the program completes in the same amount of time as a working simulation without any obvious errors (i.e., what we get with QuickCMT data).

I have attached both the CMTSOLUTION file we have used for this, as well as the Par file currently being used by the program. This may be my relative lack of experience with the domain in question (I'm more a software engineer than a geophysicist) but nothing sticks out to me as obviously wrong. (Note, of course, that the .txt extensions are added solely to allow uploading to github; both files are normally extensionless.)

I can also provide some outputted sac files if those would be useful.

Par_file.txt
CMTSOLUTION.txt

[danielpeter]

what code version are you using? and it would be interesting to see the output_solver.txt file as well.

also not sure if it helps, but you would probably want to set
'USE_MONOCHROMATIC_CMT_SOURCE = .false.' for a simple forward simulation.

[amkearns-usgs]

OK. I consulted and that value was set to true for a different test and should have been set to false. We re-ran the test with the expected setting but haven't yet verified the result. (The output_solver.txt below should be from a the run with that value set false.)

I had forgotten about the output_solver.txt file. I have attached it. I suspect the version of the code we're running is rather out-of-date as we've had it sitting on a server not doing much for a while now; if you think a newer version will run better we can switch, but I don't really relish going through the process of recompiling the code, haha.

output_solver.txt

[amkearns-usgs]

Rerunning the solver continued to produce NaN output. Here's the output_solver when monochromatic cmt source is false.
output_solver_rerun.txt

Is there a way to tell if there is an issue with the event?

[danielpeter]

these outputs all look okay. there are no NaNs yet and the simulations seem to finish just fine. so, I would think that the source and wavefields are all okay too.

could you add a trace where you find these NaNs? it might be an issue with the SAC binary format. to check that, you could try also to output some other formats as well (OUTPUT_SEISMOS_ASCII_TEXT and/or OUTPUT_SEISMOS_SAC_ALPHANUM).

[amkearns-usgs]

OK. It will definitely be easier for me to figure out which traces have a NaN (they've not told me so I don't know if it's all of them or just some) if I look at ascii output. I'll have to rerun that, so I'll likely only have results to post at the end of the day.

[amkearns-usgs]

Do you know how a NaN value would be represented in the ascii output? I have attempted to grep for 'NaN' across all outputted files but haven't found anything. I presume the NaNs exist as our primary researcher says they are still in the sac output.

[amkearns-usgs]

Sorry. We've managed to figure out what the issue was: the sac files were perfectly valid, but some tapering operations done on the code were not built to handle low-magnitude events that might produce start and end ranges of data where most of the points were 0-valued. The code appears to have introduced a division-by-zero error in that case, which is where the NaNs came from.

Thank you for your assistance in helping me configure the output, as tweaking the param file made it possible for me to actually evaluate the data I was generating.