changing img_size breaks because psf voxel size is incompatible

[slaide]

Hello,
I have been trying to increase the img_size from the default 40x40 to something larger (e.g. 128x128), but when i do that, i get an error:

in File "decode/neuralfitter/coord_transform.py", line 40, in _subpx_to_absolute
    x_coord = self._x_mesh.repeat(batch_size, 1, 1).to(x_offset.device) + x_offset
RuntimeError: The size of tensor a (40) must match the size of tensor b (128) at non-singleton dimension 2

which i believe originates from the PSF voxel size.
Is there a way to change the psf voxel size (in SMAP)? The only option that at least seems related is the ROI-size, but I am not quite sure if that does what i want.

[slaide]

ah, i checked the code again, and found out that the issue is that the test and training image sizes are specified seperately, and default to 40x40 when not provided, instead of inheriting the other value (if that is present). explicitely setting the param.TestSet.img_size to the same value as param.Simulation.img_size (and TestSet.frame_extent) fixed the issue.

(it would still be great to know how i can change the psf volume/voxel size in smap)

[Haydnspass]

Thanks for the report, I will create an issue and fix it :)
What do you mean by PSF volume?

[slaide]

I have been trying to simulate much larger frames than 40 x 40 pixels. we have an experiment where we take images from about 1k x 1k pixel ROI's on a flowcell, and I wanted to simulate those as closely as possible, so I increased the image size (and psf extent since that has to match the img_size). we also have an astigmatic psf, which, according to SMAP, does not work very well for |z|>500nm, but even when under (but close to) 500nm, we see certain artifacts during the simulation, which looks like the outline of some volume within which the PSF is contained (example: https://imgur.com/a/xExsE98). That is what I think is the limit of the size of the volume that SMAP approximates the PSF within. The current solution is just to constrain the z-range for emitter positions to a range that at least does not show easily visibile artifacts, but that of course is not a great/permanent solution either. (I have not checked what impact the presence of these artifacts has on the performance of the AI)

[Haydnspass]

Pinging Jonas @jries cause he is the expert on that

[jries]

I am not sure I understand the problem… First, as long as the simulation ROI is quite a bit larger than the PSF, that should be sufficient, even for very large experimental images. Also, the PSF size does not have to be equal the image size. What is the example image you shared? I just see some lines… You should check in the PSF if it has any offset (if you want, you can share the .fig file from the SMAP calibration). That of course can lead to edge effects. If you indeed have a PSF that is too large for the PSF calibration ROI you can increase the size of this calibration ROI in SMAP, and thus possibly the z-range. But in the end all of this might not matter. Did you try analysing your data with the standard DECODE settings?

…

On 10. Feb 2022, at 11:37, slaide ***@***.***> wrote: I have been trying to simulate much larger frames than 4040 pixels. we have an experiment where we take images from about 1k1k pixel ROI's on a flowcell, and I wanted to simulate those as closely as possible, so I increased the image size (and psf extent since that has to match the img_size). we also have an astigmatic psf, which, according to SMAP, does not work very well for |z|>500nm, but even when under (but close to) 500nm, we see certain artifacts during the simulation, which looks like the outline of some volume within which the PSF is contained (example: https://imgur.com/a/xExsE98 <https://imgur.com/a/xExsE98>). That is what I think is the limit of the size of the volume that SMAP approximates the PSF within. The current solution is just to constrain the z-range for emitter positions to a range that at least does not show easily visibile artifacts, but that of course is not a great/permanent solution either. (I have not checked what impact the presence of these artifacts has on the performance of the AI) — Reply to this email directly, view it on GitHub <#163 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABT5R7SDUQGGRIR55C23J33U2OIN7ANCNFSM5NXYCEWQ>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.

[slaide]

I think the problem might be solved by increasing the psf roi in smap, I was just unsure if that would be enough, but if you dont see issues why it would generally not work for larger frame sizes then I know the issue is on my side (and can probably be fixed).
but this line of code throws an exception of the psf extent and img_size are not equal (just a side note. thats what i did for the example image i have posted here, and code-wise it worked fine. edit: i realize we might mean different psf sizes. I have not looked into the connection of the configurable psf extent that is a decode input and the psf roi in smap).
We dont really have data to test the decode performance on, currently. We are planning to use it for tracking of proteins over long periods of time, and still working on essentially all aspects of that.