MetaTensor for Digital Pathology

[bhashemian]

MetaTensor is officially released in MONAI v0.9, which is a simple yet elegant way to handle metadata along with the image in the same object. Currently, the use cases are more focused on radiology transforms, and apart from some general purposed attributes and properties like meta and array, there are properties like affine that are most commonly used in 3D transformation and thus application in MRI and CT for instance. As we discussed in the last MONAI Pathology WG meeting, it would be interesting to take the same approach for digital pathology components. Below you can find more information about MetaTensor.

• Implementation: https://github.com/Project-MONAI/MONAI/blob/dev/monai/data/meta_tensor.py
• MetaTensor dev guide: https://colab.research.google.com/drive/1T4iAys-cC2qL80oJkIbAXAPlWNPwp4H7
• Migration guide: https://github.com/Project-MONAI/MONAI/wiki/MetaTensor-migration-guide

We have discussed about the most important metadata to carry on along the image for AI pipelines of digital pathology, and magnification of the image, and location of the patches came up as the must have in many use cases. Also magnification level is practically important although it can be inferred from the magnification. Moreover, encapsulating the cuImage object into MetaTensor were suggested to be of interest for pathology use cases.

I am opening up this discussion here to a broader community to get more feedback on how we can leverage the most MetaTensor in AI for digital pathology.

cc: @cooperlab @shaneahmed @manthey @JHancox @jeffbaumes

[cooperlab]

@drbeh I would distinguish between tile-specific and slide/case and metadata. For brightfield tiles I think minimum metadata is

1. slide/origin - to demultiplex tiles mixed from different slides
2. x,y - for reassembly of tile-level predictions
3. magnification - could be stored at the slide level, but resizing will typically be done at the tile level

Aside from magnification these are not necessary for most (all?) tile-level transforms I can think of. Other metadata that is not tile-specific can be stored at the slide level and linked using slide to avoid duplication. These include tile overlap (pixels), slide width/height (in tiles and/or pixels).

These answers are more complex for fluorescence imaging.

[shaneahmed]

I would strongly recommend adding mpp to the metadata. For tile-level, I think this can be kept as None with an option to specify if this information is available.

[shaneahmed]

One way could be to use meta data compatible with OpenSlide metadata. This doesn't need to include all the fields but some fields can be kept.

[cooperlab]

To note my point from today's meeting - "location" needs to be linked to a specific magnification / level. It could be the current tile magnification (possibly after transformations), the original tile magnification, or the scan magnification of the slide or something else. It just needs to be documented or returned in a way that makes this clear.

[bhashemian]

@cooperlab @shaneahmed here is the proposal for pathology MetaTensor properties. For now let's focus on patch metadata since it may cause a backward incompatibility and we want to capture it before the next release. For the slide-level information, we can make it as a feature request in the following release. What do you think of the patch-level metadata?

Here I have added path (path to the WSI file) as a patch property to be able to track patches from the same WSI for slide-wide transformation, merge them for post processing purposes, or any other slide-related work. What do you think of it? Do you have any other suggestion for this? Note that currently, it is not possible to carry around WSI objects (cucim.CuImage object), so we should have a work around.

meta
|
+-- patch
|   |
|   +-- location | torch.Tensor[int, int] --> (y, x)
|   +-- size | torch.Tensor[int, int] --> (height, width)
|   +-- downsample | float
|   +-- level | int (or float for inter-levels?)
|   +-- mpp | torch.Tensor[float, float] --> (mpp_y, mpp_x)
|   +-- path | str --> path to the WSI file
+-- slide
|   +-- level_count | int
|   +-- level_dimensions | list
|   +-- level_downsamples | list
|   +-- level_mpps | list
|   +-- level_tile_sizes | list
+-- affine --> not common in DP but can be automatically derived from other properties

[cooperlab]

@drbeh We need to clarify a few things in the event that the requested level does not exist and downsampling is performed.

-level- if downsampling, is this the original read level or a resulting non-existent downsampled level? I recommend the original read level, and then other things can be inferred from the combination of level and downsample. I don't see benefit to making level fractional - that requires users to perform more complex calculations to figure things out.
-location - If downsampling we should be clear whether location corresponds to pixels in the read level or in a resized non-existent level.

I also suggest a property for objective magnification. This can be inferred from mpp or can report what the vendor recorded in the file (adjusted for level_downsamples). The mappings of MPP -> objective magnification are not widely known.

[bhashemian]

@cooperlab thanks for the feedback.

• I agree to keep level as integer. Should the downsample be relative to the full resolution (level=0) image, the same as level_downsamples?
• Similar to OpenSlide and cuCIM, currently, location always refer to the location at the full resolution (level=0). Is it fine to keep it this way?

To decouple resampling from image loading, I would suggest to do all downsampling as a pre-processing transforms. We can discuss with cucim team (@gigony @grlee77) to see if there might be potential benefit to have the downsampling being done at cucim level but if it can be achieved similarly via a downsampling transform, the pipeline would be cleaner and we can track the transforms in MetaTensor in case we want to invert it. What do you think?

[cooperlab]

Let's take this up in the meeting. The last meeting we discussed if it was possible to specify an MPP / objective for the cuCIM reader. I agree that it is cleaner to make it part of the transforms, provided that the resizing happens on GPU.

[bhashemian]

Sure, I'll add it to the agenda for the next meeting. I also talked to cucim team and it seems that currently they are not able to do that and are dealing with other higher priority work. I made a feature request in MONIA #4980

[bhashemian]

For v1.0 release the metadata of WSI patch would look like below and can be access via image.meta.

Note that the only missing metadata that we discussed is magnification which is not being directly provided by any of the WSIReader backend and none of the operations in MONAI use it. We can add it once we have a use case for it, maybe when implementing reading at specific magnification. Also the metadata is flatten for more accessibility and easier manipulation.
Please let me know if you have any comment here.

[bhashemian]

Thank for your prompt feedback, Lee! Sure, I need to discuss with you on the most reliable way to define magnification but for location, following cucim and openslide convention, it is always referring to the full resolution image level=0.

[cooperlab]

Suggestions on magnification - we can move this over to a separate discussion when ready:

1. Use value if provided in the file (describes objective of level 0)
2. If not defined in the file, map the mpp to acceptable ranges for 20X (0.5 mpp +/- epsilon) and 40X (0.25 mpp)
3. If the mpp cannot be mapped to these ranges, it can be inferred using the standard above, or set to undefined

[shaneahmed]

Looks good to me. For location, I guess (y, x) is top left location of the required patch.

Just wanted to comment on (y, x). Usually the locations are specified as (x, y) instead of (y, x) as is followed by OpenSlide API.

We used the same convention as well here. It is easier to follow.

(y, x) I can understand is coming from (row, column) so I would suggest to use (row, column) as variable names if you would like to go with (y, x) otherwise I would suggest to follow OpenSlide convention of (x, y) that will make it consistent and compatible with existing pipelines.

[wyli]

that's a good point, there were some confusion about the coordinates for the other image readers as well #4862 (comment)

[bhashemian]

@shaneahmed thank you very much for your comments. I agree having (x, y) makes it more consistent with openslide or cucim but the output at the end should be DxHxW to be consistent with the rest of MONAI.