Making sense of inference times using sliding_window_inference function

[naga-karthik]

Hello MONAI team! Thank you very much for maintaining this wonderful package.

I have been playing with sliding_window_inference function with the main goal of reduce inference times when run on a CPU.

I am using the following preprocessing transforms, where crop_size = (64, 160, -1) is fixed for all images.

def inference_transforms(crop_size, lbl_key="label"):
    return Compose([
            LoadImaged(keys=["image", lbl_key], image_only=False),
            EnsureChannelFirstd(keys=["image", lbl_key]),
            Orientationd(keys=["image", lbl_key], axcodes="RPI"),
            Spacingd(keys=["image", lbl_key], pixdim=(1.0, 1.0, 1.0), mode=(2, 1)),
            ResizeWithPadOrCropd(keys=["image", lbl_key], spatial_size=crop_size,),
            DivisiblePadd(keys=["image", lbl_key], k=2**5), 
            NormalizeIntensityd(keys=["image"], nonzero=False, channel_wise=False),
        ])

I am calling the sliding window function like this:

batch["pred"] = sliding_window_inference(test_input, inference_roi_size, mode="gaussian", sw_batch_size=4, predictor=net, overlap=0.5, progress=False)

I have a dataset of 294 images and my inference_roi_size = (64, 160, 320) is fixed. I am using a script that calculates the inference time per subject (in seconds) on a CPU. I have played around with the overlap, and sw_batch_size parameters and am getting the following results:

1. When sw_batch_size=4 and overlap=0.5 --> inference time per image is 22.56s
2. When sw_batch_size=4 and overlap=0.25 --> inference time per image is 23.91s
3. When sw_batch_size=8 and overlap=0.5 --> inference time per image is 24.81s

I am unable to understand how decreasing the overlap by half is increasing the inference time. If I understand correctly, having less overlap between patches should ultimately decrease the number of sliding windows, hence reduce the inference time?

And as per this comment, I increased sw_batch_size in (3) but still observing longer times.

Am I missing anything else that might be going on under the sliding_window_inference function that might contribute to these weird inference times? Please note that dataset caching and loading are not included in the times mentioned above.

Cheers,
Naga.

[KumoLiu]

Hi @naga-karthik, runtime does vary a bit on the CPU, maybe you could try inference on the GPU.

[naga-karthik]

Thanks for your reply, @KumoLiu !

runtime does vary a bit on the CPU

Can you say why this is the case?

maybe you could try inference on the GPU.

I have done that for my tests. But my goal is to deploy the inference script in a system that does have a GPU, hence my experiments and questions on CPU.

[KumoLiu]

Hi @naga-karthik, there are many factors that influence CPU test times so I said that it may be better benchmarked on GPU.
You are correct in your understanding that reducing the overlap should ideally decrease the number of sliding windows, potentially reducing inference time. However, in practice, the behavior can be more complex. The overlap affects the number of windows and their positions but can also impact the efficiency of processing each window. Increasing the sw_batch_size should allow for more parallel processing, which could lead to faster inference times. However, this might not always be the case, and the effectiveness of this change can be influenced by factors like CPU architecture and available resources. If your CPU has a limited number of cores or threads, increasing sw_batch_size beyond a certain point may not lead to further improvements. There's a point of diminishing returns where increasing parallelism doesn't necessarily speed up processing. Increasing sw_batch_size can also increase memory usage. If your system has limited RAM, it could lead to slower performance due to swapping data between RAM and disk.

Perhaps you could:

• Profiling your code to identify bottlenecks.
• Check CPU usage during inference to see if your CPU is fully utilized or if there's room for parallelism.
• Experiment with different combinations of parameters, including batch size, overlap, and sw_batch_size, while monitoring CPU and memory usage.

Thanks!

[naga-karthik]

Hey @KumoLiu, thanks for your detailed reply. It answers my main question.

Just a side question - do you have any suggestions on how best to set the inference_roi_size for maximum efficiency in sliding windows? I mean, would it be ideal to set it to the same value which was used during training? OR, is it best to set it crop size used in ResizeWithPadorCop transformation?

I am just trying to understand how to set inference_roi_size given the preprocessing crop size could be arbitrary (in my case). Thanks in advance!