Using Simulated Data as Model Input

[ghost]

I want to pass some simulated frames through my trained network to see how robust it is by comparing the performance of a variety of datasets.

However, the simulated frames I'm using come from a sample of the sim_train or sim_test simulation objects, but in comparing tensor values with that of real data and the parameter file, it looks different from the real data tensor values. The sampled frames don't seem to have a baseline or emitter intensity matching that of the parameter file resulting in finding no emitters when passed through the model.

So, my question is what do these values inside the frame tensor of the simulation objects represent? Are they photon numbers or camera intensity values and have they already undergone some processing? How would one go about passing a simulated image through the model?

My workflow is below:
• Load parameter and model files as per decode/utils/examples/fit.ipynb
• Instead of loading a tiff image, frames are found via:
sim_train, sim_test = setup_random_simulation(param)
em_ref, frames, bg_frames = sim_test.sample()
• Setup frame processing and post processing as per the fit.ipynb
• Fit the frames using the infer module as per the fit.ipynb to get the predicted emitter set to use for evaluation.

Thank you.

[ASpeiser]

Hey,
thanks for sharing the question here.
Indeed the simulations are photon numbers. You can also check the Training.ipynb notebook where we explain the simulation procedure, specifically in this section:

The network always takes such frames containing photon numbers as input. That's why when you work with real images the frame processing includes the line
decode.neuralfitter.utils.processing.wrap_callable(camera.backward)
If you want to test it on simulated data that is already in the photon format you can just remove that line and it should work.

[ghost]

Thanks that does it - the model now finds roughly the correct number of emitters!

Adding to this, the evaluation results were still poor when predicted emitters were compared to the reference ones, but removing the following line from the frame processing resolves this:
"decode.neuralfitter.frame_processing.Mirror2D(dims=-1),"
Please could you explain to me why not mirroring the width dimension of simulated frames, gives the expected results of many more true positives, compared to the real frames where this is the procedure outlined in the fitting document?

Thanks again for all your help!

[Haydnspass]

Hey @Maxgamill
We mirror the last dimension of experimental frames because at least for the camera of our lab, when using EM Gain the last dimension is mirrored. So it's a hardware thing. Since the calibration with beads is not mirrored in our case, we need to account for that before fitting experimental frames.

But I agree, this might be different for other labs with other cameras and should be better documented. Will do that.

Bottom line: If you evaluate on simulated frames you have to disable the Mirror which I understand you did correctly :)