Session seems to hang - how to troubleshoot?

[ccgauvin94]

So if I start a session on a SLURM-enabled instance, I can see that the files import successfully. Then, after import, I can see the jobs spawn for the motion correction and CTF correction.

In the NextPYP UI, under "jobs" I see:

"Name: spr_session, Size: 5, Status: Running". However, if I click the "Logs" button under that line, there are no logs available to run.

I did snoop around the shared directory and under 'nextpyp/shared/sessions/fVpcv8myM877fWdg/Test-fVjFIJrvWTxl2BMJ/Test_6-QLhZfIwQ/log' I can see logs for what look like motion correction and CTF correction. From what I can tell from those outputs, they ran successfully.

However, the listed "spr_session" job never ends and no micrographs ever appear in the web UI. How do I go about troubleshooting this issue? Which log files should I look for? (There's quite a few).

Thanks

EDIT:

Btw, everything with this data seems to work fine in non-Session mode.

[abartesaghi]

The Logs button under the Jobs section will only populate with actual logs after the jobs finish running. So if there are any errors, you should see them there. Indeed, these logs should match the contents of the folder you mentioned. Seeing one of them would help with troubleshooting.

[ccgauvin94]

OK further investigation, the most obvious error I've found says:

Error : Error: location 1 of input file ctffind4.mrc is blank

I'm guessing this means that the aligned micrograph is not being written successfully. I've attached the log, and will keep investigating what the cause might be.

20231201_Sample_24_F10_T10_Dec05_13.45.42_pypd.log

[ccgauvin94]

A bit more information, here are the contents of the scratch dir:

When I try to download and examine the .webp file (which I assume should be the preview of the aligned frames) it's just black.

Weirdly, ctffind4.mrc does seem to exist, but, it appears to be the same size as the gain reference, which makes me think it's just the gain.

[ccgauvin94]

Seems to be related to linking, instead of copying. When linking files, nextPYP hangs, while when copying, things run OK.

[abartesaghi]

Thanks @ccgauvin94 for troubleshooting this. By "linking" you mean selecting the "link" option from the "raw data transfer" dropdown menu in the "Session settings" tab? If so, we haven't seen this before, but we can investigate.

[ccgauvin94]

Yes - I had chosen the link option initially. The storage that the raw data are on is a mounted NFS system, so my guess is it's an issue with my mounting options perhaps not supporting symlinks - probably not an issue with NextPYP.