Run freezes at BadPixStep

[suman-exo]

I'm reducing a NIRSpec/G395 time-series data, and it freezes at the following process in stage 2:

XXXX - exoTEDRF - INFO - Starting spatial outlier flagging...
0%| | 0/2038 [00:00<?, ?it/s]

The issue occurs for both NRS1 and NRS2 data. Is this a problem with the data, or something else? I'm using version 2.1.0 here. I tried updating to version 2.2.0, which resulted in a 'runtimerror' with multiprocessing.

[radicamc]

I've never encountered this sort of error with the bad pixel flagging with any dataset without any sort of external "influence". Is it possible that a higher priority task started on your CPU that froze the step? I've had that happen several times on various steps. Restarting the step generally fixes things though.
With regards to the runtime error, yes this is a bug that cropped up when upgrading the STScI pipeline version. See Issue #37 for a quick workaround for this. Alternatively, there should be a new release later this week which will patch this.

[suman-exo]

The workaround for the 'runtimerror' works, and with that, I tried running the reduction on version 2.2.0. But it still froze at the same specific step in all my trials. I didn't see this issue while reducing other data previously. I'm using a dedicated node in a cluster to run the job, and all remaining cores except one remain unused during this freeze.

Could a problem with the data file (uncal) itself cause such an issue? And do you have a suggested debug for this step?

[radicamc]

A potential idea could be, if you have an editable version of the code installed, to stick a print statement on line 1177 in stage2.py giving the current y-pixel. It could be that the step is running, only really slowly and/or is flagging many pixels and so is taking a long time.

 # Loop over whole deepstack and flag deviant pixels.
 for i in tqdm(range(5, dimx - 5)):
     for j in range(ymax):
         print(j)
         # If the pixel is known to be hot, add it to list to interpolate.

It's a long shot, but I can't see any reason why the step would just simply not run.
Have you inspected the data before it is being fed into the step and checked that there is nothing different or incorrect about it compared to other datasets that you have work with?

[suman-exo]

Hi, I couldn't get back to it for some time, so late to reply here. I ran the pipeline with two sectors now (1 and 5) from the same observation, and it ran without any issues. Earlier, I was trying to reduce only the sector 5 data, and that posed the freezing issue. Does this give any clue about a potential problem?

[radicamc]

That's odd. I wonder if segment 5 is particularly small compared to the other segments?

[suman-exo]

No, segment 5 is not particularly small; it is one of the middle segments. Maybe the issue arises from not having the first segment included in the reduction because of naming conventions, however, I haven't tested this theory with other data yet.