BilateralFilter cuda implementation seems to fail on inputs with >16 channels?

[ajabri]

It seems that the BilateralFilter cuda kernel silently fails (returns zeros) on inputs with more than 16 channels. Here is an example of the 2D case:

# 16 channels
BilateralFilter.apply(torch.randn(10, 16, 60, 60).cuda().contiguous()).std()
tensor(1.0003, device='cuda:0')

# 17 channels
BilateralFilter.apply(torch.randn(10, 17, 60, 60).cuda().contiguous()).std()
tensor(0., device='cuda:0')

Is this expected?

Thanks!
Allan

[ajabri]

I wrote an extension of the kernel to the joint bilateral filter case that I'd like to contribute, just wanted to address this issue first :)

[charliebudd]

In the case of a bilateral filter, the PHL approximation is pretty much indistinguishable in my experience. If you're working with kernel sizes (spatial sigmas) greater than 8 ish (may differ for your hardware) then I would suggest the phl implementation as it really is much faster for practically no loss in quality. (As it happens the Bilateral filter currently defaults to this implementation any way. There is a boolean flag passed to the apply function if you wish to use the exact implementation)

As your use case is a joint bilateral filter then I would again suggest using the PHL implementation as the API allows for that already, to construct a bilateral filter just prepare you're feature vectors by concatinating the RGB components and the XY pixel coordinates.

Thankfully increasing the cap (which also applys to the PHL) shouldn't be an issue, 16 is a pretty arbitrary choice but as we dont want to increase our compile time I would suggest doing this in a fork. Like I said, we plan on introducing a JIT system and will move this code accross once it's in which will remove this issue in the future.

[ajabri]

I see, I'll try out the PHL implementation then. I've been using my own fork of the BilateralFilter cuda kernel (which allows for another feature tensor as input).

To increase the cap past 32, do I need to define new macros? Is there an easier way?

Thanks for your help!

[charliebudd]

The template function is explicitly instantiated for all configuration (capped at 16 data channels and 19 feature channels) here...

MONAI/monai/csrc/filtering/permutohedral/permutohedral_cuda.cu

Lines 534 to 537 in dde1100

	#define DECLARATION(dc, fc) \
	template void PermutohedralCuda<float, dc, fc>(float* values, float* positions, int elementCount, bool accurate); \
	template void PermutohedralCuda<double, dc, fc>(double* values, double* positions, int elementCount, bool accurate);
	DO_FOR_AB(DECLARATION, 16, 19)

you could just add your specific instatiations below e.g.

template void PermutohedralCuda<float, 128, 5>(float* values, float* positions, int elementCount, bool accurate);

Performance may not be too desirable as I didn't optimise for such usage, I'll try to take this into account when I come back to refactor this for the JIT system.

[ajabri]

Got it, thanks!

Does the PHL-backed implementation of the BilateralFilter support backpropagation? I noticed the PHLFilter backward seems to have been commented out.

[charliebudd]

Ahh yes I should of thought about this given your use case. The backpropergation needs some looking at, I actually intended to get this straightened out this week. I'll make an issue to track. Do you need gradients for the feature vectors and sigmas or just for the input being filtered?

[GHJg7]

hmm

[wyli]

The original post is 2 years old, more recently there's an extended implementation of bilateral/joint bilateral filtering

MONAI/monai/networks/layers/filtering.py

Line 345 in 65cf5fe

class TrainableJointBilateralFilter(torch.nn.Module):