XY axes and EM/sCMOS mirroring - my head hurts

[cleterrier]

Hi,
I am somehow lost with the EMCCD mirroring vs sCMOS non-mirroring, in part because on top of that the default XY axes of DECODE don't match usual conventions (from top left, X toward the right, Y toward the bottom).

• I have used SMAP to generate spline-fitted PSFs. I usually ask for no mirroring in the SMAP options, even for the EMCCD camera. My calibration stacks are generated with EM gain on for EMCCD (so same situation as the acquired sequences, in contrast to your default situation). I checked that this generates average PSFs which, once exported as tif stacks, look similar in shape and orientation to the individual PSF in the Z calibration stacks when both are opened in ImageJ (so no rotation problem here).
• From the beginning I have used the decode.neuralfitter.frame_processing.Mirror2D(dims=-1) line in the fitting notebook. This works well, even if according to the explanations, I should comment this out as the Z calibration and acquired sequence are both with EM gain on. The resulting reconstructed image is rotated 90° left compared to acquired sequence, so I have to rotate the result 90° right to obtain the image as displayed by the camera. I assumed this was due to the strange XY axis convention of DECODE - there's no mirroring involved for the correction, just a rotation.
• So here there's a discrepancy between the rationale for decode.neuralfitter.frame_processing.Mirror2D(dims=-1) and the result I get (good result with this mirroring, even if the Z calibration and acquisition are done with EM gain on).
• Now I try to process images from another setup with an sCMOS, so here also Z calibration and acquired sequence have the same orientation (no EM gain for both). I commented out the decode.neuralfitter.frame_processing.Mirror2D(dims=-1) line in the fitting notebook, thinking this wasn't necessary. And now the resulting reconstructed image (after 90° right rotation), is mirrored along the Y axis compared to the camera image!

I really would like to understand what happens here because in my understanding, fitting with the wrong mirroring option will lead to bad fitting as the spline PSF will be "reversed" compared to what DECODE was trained with (as there is no mirroring in the Training notebook AFAIK). Yet I get good results with mirroring in one case where I shouldn't use it (the EMCCD case with EM gain on for both Z calibration and fitting), and mirrored results without mirroring in another case where I also shouldn't use it according to the explanations (the sCMOS case).

[cleterrier]

Re-reading issue #42 , indeed the coordinates of the DECODE output are inverted (x instead of y, y instead of x). So to correct that I should rotate 90° right and mirror along Y (that's the proper way to exchange X and Y). So with mirroring in fitting, I only have to do the 90° rotation, whereas without it, I also have to do the mirroring in localizations post-processing downstream of DECODE. But then, it does not explain why my fitting results are so good with mirroring in my EMCCD case, where I have the same EM gain on for both Z calibration and acquisition...

[Haydnspass]

Hey @cleterrier,

Sorry for your headache, I hope I can help:

• There are multiple possible places, where different conventions about the order of axes come into play. Moreover, there are multiple conventions on how to plot a matrix, e.g. is your (0, 0) origin in the top left/bottom left, and are rows of pixels in your plotting corresponding to rows or columns of your matrix (lot's of historical reasons this feels weird, at least to me). This is not limited to DECODE but plotting in general.
• The point above implies that even if the underlying data structure is correct and consistent, you have no guarantee that the visual output after plotting would be the same between any two programs
• When I implemented the PSF simulator and plotting functions I used an uncommon convention (i.e. the one you can see now). This will change, however in the next DECODE release we will not have implemented this, but have a axes_order='future' plotting flag, that will at least visually do what I think is the most common plotting style. See the documentation of the master branch https://decode.readthedocs.io/en/master/decode.plot.html#decode.plot.frame_coord.PlotFrame

The important points for fitting:

• If there is now mirroring between PSF calibration and experimental data (wherever this might or might not come from) you should not use Mirror2D. The fitting might still work, but would be slightly degraded in performance, depending on whether the PSF is somewhat symmetrical or not. Mirroring the last dimension will not lead to flipped z values
• Assuming that you have set Mirror or not Mirror correct, the transformation between DECODE and SMAP is: x_smap ~ y_decode, y_smap ~ x_decode, z_smap = z_decode
• Since our network requires input images of size multiples of 8, there can be an offset because the frame is cropped to an appropriate size before it runs through the decode pipeline

Stay tuned for our next release next week:

• h5 writer / loader in DECODE
• New DECODE loader in SMAP that does it automatically for you (on .h5 and .csv decode files).