What is Fused spatial transforms?

[Jingnan-Jia]

From this docs:

I found that some transforms could be accelerated by GPU, like affine.

One of the reason is that this transform could receive torch.tensor as the input. But a great number of other transforms only receive numpy.Array as the input, which means that they can not be accelerated by GPU.

My question is:

1. Is there any rule about which transform could be accelerated by GPU, and which transform can not?
2. What does "Fused spatial transforms" mean? I googled "Fused spatial transforms" but I found this term is not commonly used. could you please give an explanication?

[rijobro]

Hi @Jingnan-Jia, some good questions!

1. There is no rule whatsoever. However, for simple transforms, it would probably be slower to copy the data to the gpu, run the transform, and then copy back. On top of that, we generally want to use the CPU for the loading of the data, to leave more the gpu available for training. this is useful when one process is preparing the batch whilst another is training the network (although this doesn't happen by default, admittedly). The affine transform is sufficiently compute-intensive, that we deemed it worthy of giving the user the option to execute on the GPU.

2. By fused transforms, we mean transforms that do multiple things at the same time. For example, the Rand2DElastic and Rand3DElastic also perform affine transformations at the same time. This minimises the number of resampling/interpolation steps required, which makes things faster.

Lastly, I've been working on trying to harmonise our transforms so that they'll all accept torch.Tensor or numpy.array as input. That requires a lot of changes, however, I don't think it'll be merged in the very immediate future.

[Jingnan-Jia]

Thanks for your reply.

Another question is:
Nowthat Affine could use GPU to accelerate the transform, if I set the workers = 3, will the occupied GPU memory by Affine(GPU) is 3 times the occupied GPU memory by Affine(CPU) when I set the workers = 1 in dataloader?

[wyli]

I remember the multi-worker loader doesn't work well with the GPU-based transforms, it'll cause some additional cpu-gpu communication overheads.

[Jingnan-Jia]

Okay I see, I will avoid the use of GPU-based transforms then. Just let GPU do deep learning and CPU do transform seem more elegent. I am afraid that the GPU-based transform will occupy the precious GPU memory during the training of networks .