Numpy array and Compose without LoadImaged

[pauljhin]

Hello,

I'm trying to apply the transformations using Compose in a loaded numpy array.

It seems to me that some transformation is not done correctly because the prediction result is different when the numpy array is already loaded in memory and I try to apply the transformations. When I use LoadImaged to load Nifti file it works correctly.

Is there a better way to use Compose to apply transformations to an array already loaded into memory?

The parte of code:

test_org_transforms = Compose(
    [
        #LoadImaged(keys="image"),
        EnsureChannelFirstd(keys="image"),
        Orientationd(keys=["image"], axcodes="RAS"),
        Spacingd(keys=["image"], pixdim=(
            1, 1, 1), mode="bilinear"),
        ScaleIntensityRanged(
            keys=["image"], a_min=700, a_max=4000,
            b_min=0.0, b_max=1.0, clip=True,
        ),
        CropForegroundd(keys=["image"], source_key="image"),
    ]
)

post_transforms = Compose([
    Invertd(
        keys="pred",
        transform=test_org_transforms,
        orig_keys="image",
        meta_keys="pred_meta_dict",
        orig_meta_keys="image_meta_dict",
        meta_key_postfix="meta_dict",
        nearest_interp=False,
        to_tensor=True,
    ),
    AsDiscreted(keys="pred", argmax=True, to_onehot=2),
    RemoveRepeatedChanneld(keys="pred", repeats=3),
])


test_data = [{"image":numpy_array_object}]

test_org_ds = Dataset(
    data=test_data, transform=test_org_transforms)

test_org_loader = DataLoader(test_org_ds, batch_size=1, num_workers=4)

with torch.no_grad():
    test_data = list(test_org_loader)[0]
    test_inputs = test_data["image"].to("cuda")
    roi_size = (96, 96, 96)
    sw_batch_size = 4
    test_data["pred"] = sliding_window_inference(
        test_inputs, roi_size, sw_batch_size, model)

    test_data = [post_transforms(i) for i in decollate_batch(test_data)]

    matrix = from_engine(["pred"])(test_data)[0] 
    numpy_matrix = matrix.cpu().detach().numpy()

[pauljhin]

UPDATE:

The problem seems to be that Spacingd needs some information that my dictionary with only numpy array doesn't have.

test_data = [{"image":numpy_array_object}]

What information is needed for Spacingd to work? What keys are needed to use in my dictionary?

[ebrahimebrahim]

Indeed as you've pointed out the transform SpacingD needs some additional meta data beyond just the image array; from the documentation:

This transform assumes the data dictionary has a key for the input data’s metadata and contains affine field. The key is formed by key_{meta_key_postfix}.

Based on this you should have a dictionary item like image_meta_dict in your data which has an affine field... however looking at the code this argument was deprecated since MetaTensor was introduced in MONAI 0.9. (the above documentation blurb needs to be updated-- would be good to open an issue for that)

assuming your monai version is >= 0.9, I think your input should be a MetaTensor rather than a numpy array. The MetaTensor carries the affine attribute that Spacing can operate on.

although now looking at the code again I think it should work even without MetaTensor, the affine would just be assumed to be identity

so I am guessing you have an older MONAI version and should be fine if you upgrade to at least 0.9

[pauljhin]

Hi @ebrahimebrahim ,

Thank you for your reply but I tried adding that key and it didn't work (it didn't have any effect). See how I did it:

My MONAI version is 1.0.1. I didn't get any warnings.

from torch import tensor

affine = {'affine': tensor([[  -0.4805,    0.0000,    0.0000,  122.7598],
         [   0.0000,    0.4805,    0.0000, -122.7598],
         [   0.0000,    0.0000,    0.7000,  -89.6000],
         [   0.0000,    0.0000,    0.0000,    1.0000]], dtype=torch.float64)}

test_data = [{"image":numpy_array_object, "image_meta_dict":affine}]

The affine values I copied by opening the image with LoadImaged.

PS: I built a dictionary with all values besides affine and it didn't have any effect either.

[ebrahimebrahim]

Sorry I think I didn't read your problem carefully enough-- you are not getting errors but rather getting different results with vs without LoadImage. I don't know why this would happen if you are putting in the same metadata that LoadImage would discover. But one thing to try is to represent your image as a metatensor rather than as a numpy array. So something like

test_data = [
  {
    "image" : monai.data.MetaTensor(numpy_array_object, affine = <your affine>),
  },
]

and see that gives the correct result. It could be that because the image_meta_dict approach is being depracated there are some transforms not handling it correctly.

[pauljhin]

Using MetaTensor it worked.

Thank you very much @ebrahimebrahim !