ELU/ReLU and MaxPool2D

[tsuijenk]

Hello,

Happy new year! I've been studying your code, particularly the ones like UNet_parts.py. I am sorry for the long message, but I would love to get a clear understanding of the code. I felt like the code does something that is not fully described in the paper. Thank you in advance! :)

Feature 1
In your paper, you said ELU is applied for all hidden units, but I only see ReLU being applied. I was wondering if you could clarify that.

Feature 2
According to original UNet paper, the MaxPool2D is completed after the 3 convolutional layers. But I see that MaxPool2D being added before the 3 convolutional layers.

I was thinking that you had a separate piece of code for the very first 3 convolutional layers after the image input. Hence, I suspect that it goes like this:
1). Image input.
2). 3 convolutional layers (code is somewhere)
3). The downsampling stage 1 (MaxPool2D, followed by 3 convolutional layers)
4). The downsampling stage 2 ...
5). etc.

Is this what you were doing?

Feature 3
I see that you are using PyTorch but not really PyTorch Lightning. Would you consider adopting PyTorch Lightning in the near future?

Additionally, I was wondering if the use of PyTorch Lightning would take over the duty of weight generator? For example, I know the adoption of PyTorch Lightning means we don't need to call backward() and zero_grad() explicitly.

Feature 4
I was wondering if you are backpropogating the sum of the three losses (gmm, bg, and count)? Because I see "loss_val.mean().backward". Does that mean your "loss_val" is a vector? Is it a vector over the three types of losses? Or a vector over the minibatch? I also see "loss_mean" being generated by "loss.log()". How is this value different from loss_val.mean()?

Then I see "loss_epoch.mean" being returned as part of training_step(). I guess we return this value, but we don't call backward() on this value, but loss_val.mean()?

Feature 5
In Model_speced_impl.py, I see that you are clamping the values into range (-8, 8) before applying sigmoid. I was wondering why should the values be clamped into range (-8, 8)? Wouldn't sigmoid do the job of clamping into (-1,1) already? What's the purpose of the additional step?

After sigmoid, do you then pass the sigmoid tensor into the post_processing.py to calculate the Bernoulli probability map?

[Haydnspass]

Hey @tsuijenk
Thanks for your great interest in decode. Answers below

1. The activation function is configurable and comes from the config file. If it is not set by the user individually you ELU is used because of the reference parameter.

   DECODE/decode/utils/reference_files/reference.yaml

   Line 31 in e57a1cc

   activation: ELU

2. Sure, in principal we are very open for any kind of improvement. Sometimes it can be a bit tricky that decode is not a "typical" deep learning project, but I have used lightning in other projects and do not see objections so far.

3. loss_val is a vector, I will have to look at the implementation myself again, we'll get back to you.

4. I think we did that because otherwise, gradients tend to be 0.