Inverting sensitivity map

[sulaimanvesal]

Hi,

I need some help on how to apply inverse transformation for the output of sensitivity-map/grad-cam map. The code and prediction maps work fine, however for visualization purpose I must change back the prediction map to original image size. My input images have various image sizes, therefore in my transformation I just resize the images to 64x64x64. Now, I need to invert back.

The code based my my logic should work. The output of OcculsionSensitivity function is a tensor, therefore I convert tensor to metaTensor using "convert_to_tensor" function. Then I will copy the applied_operations and metadata information from data_batch[0] which is the input image. The final meta-tensor looks fine and then I feed it back as a dictionary to test_transform.inverse().

I have tried both dicationray_based transform and array_based transform, but no success and it throws the following error.

**Computing Sensitivity Map, and transform it back (inverse trasnform) to original input size: **

#Define transforms
  # test_transforms = Compose([EnsureChannelFirstd(keys=['image']),
  #                     Resized(keys=['image'], spatial_size=(64, 64, 64)),
  #                     ScaleIntensityRanged(keys=['image'],
  #                         a_min=0, 
  #                         a_max=255, 
  #                         b_min=0.0, 
  #                         b_max=1.0)])

test_transforms = Compose([EnsureChannelFirst(),Resize(spatial_size=image_size),ScaleIntensityRange(a_min=0, 
                            a_max=255, b_min=0.0, b_max=1.0)])

# since we're not training, we don't need to calculate the gradients for our outputs
with torch.no_grad():
    for data_batch in test_loader:
        val_images, val_labels = data_batch[0].to(device), data_batch[1].to(device)
        # calculate outputs by running images through the network
        outputs = model(val_images)
        # the class with the highest energy is what we choose as prediction
        _, predicted = torch.max(outputs.data, 1)

        total += val_labels.size(0)
        correct += (predicted == val_labels).sum().item()

        # Get the occlusion sensitivity map
        occ_sens = monai.visualize.OcclusionSensitivity(nn_module=model, mask_size=mask_size, n_batch=n_batch, stride=stride)
        occ_sens_b_box_all = [-1, -1, -1, -1, -1, -1, -1, -1]
        occ_result_all, _ = occ_sens(x=val_images, b_box=occ_sens_b_box_all)

        occ_result_all = occ_result_all[..., val_labels.argmax().item()]
        #occlusion output is not a meta tensor, so convert to metatensor.
        occ_result_all = convert_to_tensor(occ_result_all, track_meta = True)
        
        #copy the applied operation and meta data from the image input.
        occ_result_all.applied_operations = data_batch[0].to(device).applied_operations
        occ_result_all.meta= data_batch[0].to(device).meta

        seg_dict = {"image": occ_result_all}
        #with allow_missing_keys_mode(test_transforms):
        inverted_seg = test_transforms.inverse(seg_dict)
        print(inverted_seg.shape)

Error:

Traceback (most recent call last):
  File "C:\Users\sulai\anaconda3\envs\pytorch_env\lib\site-packages\monai\transforms\transform.py", line 90, in apply_transform
    return _apply_transform(transform, data, unpack_items)
  File "C:\Users\sulai\anaconda3\envs\pytorch_env\lib\site-packages\monai\transforms\transform.py", line 54, in _apply_transform
    return transform(parameters)
  File "C:\Users\sulai\anaconda3\envs\pytorch_env\lib\site-packages\monai\transforms\spatial\dictionary.py", line 600, in inverse
    d[key] = self.resizer.inverse(d[key])
  File "C:\Users\sulai\anaconda3\envs\pytorch_env\lib\site-packages\monai\transforms\spatial\array.py", line 890, in inverse
    transform = self.pop_transform(data)
  File "C:\Users\sulai\anaconda3\envs\pytorch_env\lib\site-packages\monai\transforms\inverse.py", line 206, in pop_transform
    return self.get_most_recent_transform(data, key, check, pop=True)
  File "C:\Users\sulai\anaconda3\envs\pytorch_env\lib\site-packages\monai\transforms\inverse.py", line 188, in get_most_recent_transform
    self.check_transforms_match(all_transforms[-1])
  File "C:\Users\sulai\anaconda3\envs\pytorch_env\lib\site-packages\monai\transforms\inverse.py", line 145, in check_transforms_match
    xform_id = transform.get(TraceKeys.ID, "")
AttributeError: 'list' object has no attribute 'get'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "patch_classification_inference.py", line 282, in <module>
    calc_accuracy(model, test_loader, test_transforms)
  File "patch_classification_inference.py", line 251, in calc_accuracy
    inverted_seg = test_transforms.inverse(seg_dict)
  File "C:\Users\sulai\anaconda3\envs\pytorch_env\lib\site-packages\monai\transforms\compose.py", line 183, in inverse
    data = apply_transform(t.inverse, data, self.map_items, self.unpack_items, self.log_stats)
  File "C:\Users\sulai\anaconda3\envs\pytorch_env\lib\site-packages\monai\transforms\transform.py", line 114, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <bound method Resized.inverse of <monai.transforms.spatial.dictionary.Resized object at 0x00000197B34D17C0>

[wyli]

the overall logic looks great, one detail is that data_batch[0] and occ_result_all are 'batches', you can try to decollate before the inverse:

for occ_result in monai.data.decollate_batch(occ_result_all):
# for occ_result in occ_result_all:  # this also implicitly decollate
    ...
    inverted = test_transforms.inverse(occ_result)
    ...

(I'm converting this to a discussion)

[sulaimanvesal]

Thank you @wyli, actually I tried this but the code throw the same error.

[wyli]

that's strange, could you please post the exact error message here?

[sulaimanvesal]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\anaconda3\envs\Pytorch_env\lib\site-packages\monai\transforms\transform.py in apply_transform(transform, data, map_items, unpack_items, log_stats)
     89             return [_apply_transform(transform, item, unpack_items) for item in data]
---> 90         return _apply_transform(transform, data, unpack_items)
     91     except Exception as e:

~\anaconda3\envs\Pytorch_env\lib\site-packages\monai\transforms\transform.py in _apply_transform(transform, parameters, unpack_parameters)
     53 
---> 54     return transform(parameters)
     55 

~\anaconda3\envs\Pytorch_env\lib\site-packages\monai\transforms\croppad\dictionary.py in inverse(self, data)
    150     def inverse(self, data: Mapping[Hashable, MetaTensor]) -> Dict[Hashable, MetaTensor]:
--> 151         d = dict(data)
    152         for key in self.key_iterator(d):

ValueError: dictionary update sequence element #0 has length 256; 2 is required

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-38-c22ad0b4a2b0> in <module>
     63 
     64                     print(val_outputs_list)
---> 65                     inverted_seg_list = [valid_transform_rev.inverse(item_dict) for item_dict in val_outputs_list]
     66                     print("This is invert", inverted_seg_list.shape)
     67                     #except:

<ipython-input-38-c22ad0b4a2b0> in <listcomp>(.0)
     63 
     64                     print(val_outputs_list)
---> 65                     inverted_seg_list = [valid_transform_rev.inverse(item_dict) for item_dict in val_outputs_list]
     66                     print("This is invert", inverted_seg_list.shape)
     67                     #except:

~\anaconda3\envs\Pytorch_env\lib\site-packages\monai\transforms\compose.py in inverse(self, data)
    181         # loop backwards over transforms
    182         for t in reversed(invertible_transforms):
--> 183             data = apply_transform(t.inverse, data, self.map_items, self.unpack_items, self.log_stats)
    184         return data
    185 

~\anaconda3\envs\Pytorch_env\lib\site-packages\monai\transforms\transform.py in apply_transform(transform, data, map_items, unpack_items, log_stats)
    112             else:
    113                 _log_stats(data=data)
--> 114         raise RuntimeError(f"applying transform {transform}") from e
    115 
    116 

RuntimeError: applying transform <bound method Padd.inverse of <monai.transforms.croppad.dictionary.SpatialPadd object at 0x000001B45B959E20>>

@wyli here is the output of "print(val_output_list)", if that could help you:

Applied operations
[ { class: 'SpatialResample',
    extra_info: { 'align_corners': False,
                  'dtype': 'float64',
                  'mode': 'bilinear',
                  'padding_mode': 'border',
                  'src_affine': tensor([[-4.9818e-01,  1.9138e-02, -2.2824e-01,  1.1340e+02],
        [-3.6558e-03, -4.6423e-01, -1.1141e+00,  6.5813e+01],
        [-4.2425e-02, -1.8473e-01,  2.7761e+00, -1.9855e+01],
        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  1.0000e+00]],
       dtype=torch.float64)},
    id: 1874720940816,
    orig_size: (384, 384, 21)},
  { class: 'CenterSpatialCrop',
    extra_info: {'cropped': [64, 64, 64, 64, 0, 1]},
    id: 1874720940912,
    orig_size: (384, 384, 21)},
  { class: 'SpatialPad',
    extra_info: {'padded': [(0, 0), (0, 0), (0, 0), (0, 0)]},
    id: 1874720941104,
    orig_size: (256, 256, 20)},
  {class: 'Lambda', id: 1874720941392, orig_size: (256, 256, 20)}]
Is batch?: False]

[wyli]

you need a dictionary input for the inverse, something like:

    seg.applied_operations = transformed_data["label"].applied_operations
    seg_dict = {"label": seg}
    with allow_missing_keys_mode(val_transforms):
        inverted_seg = val_transforms.inverse(seg_dict)

please find more details from this tutorial: https://github.com/Project-MONAI/tutorials/blob/main/modules/inverse_transforms_and_test_time_augmentations.ipynb

[sulaimanvesal]

@wyli Thank you for your help.

I finally managed this to work. A bit messy but this works:

with torch.no_grad():
    for idx, batch_data in enumerate(val_loader):
        data, target = batch_data["image"].to(device), batch_data["label"].to(device)
        logits = model_inferer_test(batch_data["image"].to(device))
        logits_update  = convert_to_tensor(torch.unsqueeze(logits[0], 0) , track_meta = True)
        logits_update.applied_operations = batch_data["label"].to(device).applied_operations
        logits_update.meta= batch_data["label"].to(device).meta
        seg_dict = {"label": logits_update}
        for occ_result in monai.data.decollate_batch(seg_dict): 
            with allow_missing_keys_mode(val_transform):
                inverted_seg = val_transform.inverse(occ_result)
                print("This is invert", inverted_seg['label'].shape)