Memory issues? Strange behaviour.

[dietzc]

Hi @psteinb & @StephanPreibisch,

I'm using these CUDA implementations now for a while (thank you!), but I figured that there seems to be something really strange going on concerning the memory footprint of the implementation.

I'm using a Titan Black X with 12GB Memory (single memory, not two graphic cards or stuff) and compiled your code for CUDA 7. I see the same behaviour on linux and windows.

I figured, that I can easily convolve a float image of size 400^3 (which is length=64000000 and ~244MB) with a kernel of size (31x31x91) in-place.

However, an image of size 406_400_400 and same kernel size (31x31x91) already fails. By fails I mean, that the array I passed over to CUDA to perform the in-place convolution nearly only contains zeros after convolution. Some values are very close to zero.

I know that the memory footprint FFT is high, however, not over 24 times as high as my input image size?

Any ideas??

Thanks!
Christian

[StephanPreibisch]

It is a little bit unforeseeable how much RAM the FFT will actually take (CUDA FFT documentation). Ideally use power-of-two values or at least sums of powers-of-two to avoid these issues. So if the image size is 406, use 512 or maybe 448 (256+128+64), it will most likely use way less RAM.

[dietzc]

But 24 times the RAM of the input? This is really surprising.

[StephanPreibisch]

Try 512 instead, I am sure it will just work. My 4GB card crashed on 357x415x214 (or so) but computed 1024x1024x512. It is somehow related to the way it is implemented, which is not so easy using GPU hardware apparently.

[psteinb]

I'll add a test on it once I find the time to and post my findings here.

[dietzc]

Thanks for the hints. My GPU doesn't crash. Just my output contains tons of zeros and some values close to zero.

Some findings which are even more confusing:

• 512x512x512 doesn't work!
• 768x768x256 works
• 768x768x768 works
• 512x512x768 works
• 512x512x640 works
• 640x640x640 works
• 528x528x528 works
• 611x623x600 doesn't work

and finally:

• 1024x1024x1024 doesn't work (no GPU crash)

?!!

[dietzc]

I found the following at http://docs.nvidia.com/cuda/cufft/index.html#axzz3zUNEcPXh:

Transform sizes up to 512 million elements in single precision and up to half that in double precision in any dimension, limited by the type of transform chosen and the available GPU memory

512x512x512 should work anyway

[psteinb]

Could you please specify the kernel you used to convolve the images with (shape is most important, content a bonsu)? I deem you filled the test 3D volumes with arbitrary content?

[dietzc]

kernel size: (31x31x91). content is not arbitrary, it's a sample of my SPIM image. However, it should work with arbitrary content, right? can you convolve a 512^3 img?

[dietzc]

http://docs.nvidia.com/cuda/cufft/index.html#axzz3zUNEcPXh

it' also mentoined that dimensions sizes which can be factored into distinct primes in the form of 2^a_3^b_5^c*7^d are more efficient and depending on the input dimension on or the other algorithm is being used. what I did is the following: I created all 2^i, 3^i, 5^i and 7^i where 0 < i < K with (K = 10 in my case) and all multiplications of theses numbers. I added all numbers < 1025 to the list. If I calculate a bounding box now for some sample, I use the next bigger number of the list compared to the dimension of the bounding-box for the new bounding-box dimension and I don't have these problems anymore. However, it's still strange that 512^3 doesn't work.

PS: potentially there is a more elegant way to find the next best suitable number. however, like this it works.

[psteinb]

@dietzc
did you apply the cufft patch to your cuda 7?

[dietzc]

running Cuda 7.5.

[psteinb]

cuda 7.5 sounds good to me. the kernel of shape 31x31x91, which dimension does 91 correspond to: x or z? I am preparing some test cases that I'll publish to a dedicated branch. I'll keep you posted.

[dietzc]

it's X,Y,Z so Z=91. and: thanks!

[psteinb]

I created some test cases and could reproduce the problem for 512^3 and even 128^3. I am looking into this now. the code is at
https://github.com/StephanPreibisch/FourierConvolutionCUDALib/tree/ticket11_memory_issues
more precisely:
https://github.com/StephanPreibisch/FourierConvolutionCUDALib/blob/ticket11_memory_issues/tests/test_gpu_memory_reach.cpp