Volumetric Data for Multiple Instance Learning

[DocandBean]

Is there a best practice method for loading a volumetric data set for training a MIL attention model?

The example code for whole slide imaging loads a single large image and then creates bags using the RandGridPatchd and GridPatchd functions.

With a data set of k CTs, each with n images,
[pt_000_image_000, pt_000_image_000,...,pt_000_image_000],
[pt_001_image_001, pt_001_image_001,...,pt_001_image_n],
...,
[pt_k_image_000, pt_k_image_001,...,pt_k_image_n]

and each with a single label, [0,1,...m].

One (poor) method would be to tile the images for a given CT into a single image and then use the GridPatchd function with particular attention to the size and spacing equal to the original images.

What is the best method/function for creating a data set suitable for the attention model?

Thanks.

[KumoLiu]

Hi @DocandBean,
For MILModel, it only works for 2D images currently, refer to https://github.com/Project-MONAI/MONAI/blob/66df2b2a196bae116bfd5db9bf492df00b4b3e5f/monai/networks/nets/milmodel.py#L24-L25. From your description, I think this model is not suitable for your data because you are 3d images. If I misunderstood, free feel to correct me. Thanks!

[DocandBean]

Hi @KumoLiu,

Thank you for the response and apologies for my poor description of the problem. Each image in the kth CT is a 2D image of the form {'image' : 1xHxW}. The "volumetric" aspect is the set of k images together creates a 3D volume (or bag) of the form {'image' : Kx1xHxW}

The list_data_collate function in the example code stacks the individual 2D images into a bag of the form {'image' : Nx3xHxW}. It's not clear to me where to define the "N."

Any assistance would be greatly appreciated!

[KumoLiu]

Hi @DocandBean, "N" in the list_data_collate function means the num of patches. So in your case ,I think "K" is "N".

[DocandBean]

Hi @KumoLiu,

Thanks again for your response. I agree that k = N.

Is there a tutorial or example code that demonstrates how to create a data set with the expected input data shape [B, N, C, H, W] starting with CT studies of the form [N, C, H, W] where each individual image is of the form [C, H, W]?

The MIL example code takes images of the form [C, H, W] and then uses RandGridPatchd to create the form [N, C, H, W]. The API Reference for RandGridPatchd does not clearly define the return.

each of which contains the all the original key/value with the values for keys replaced by the patches

How can one take N images of the form [C, H, W] and get the same result?

Any assistance would be greatly appreciated.

[KumoLiu]

Hi @DocandBean,
After dataset from MIL attention model, it returns a list with N patches, after dataloader it returns data with shape [B, N, C, H, W]. So in your case if you have data like this

[pt_000_image_000, pt_000_image_001,...,pt_000_image_n],
[pt_001_image_000, pt_001_image_001,...,pt_001_image_n],
...,
[pt_k_image_000, pt_k_image_001,...,pt_k_image_n]

you can simply use SplitDim to convert 3D images into a stack of 2D images.

FYI, in MIL attention model each bag has only one label.

And Hi @drbeh, maybe we could enhance the API Reference for RandGridPatchd and GridPatchd to clearly define the return. What do you think?

[bhashemian]

Hi @KumoLiu @DocandBean, SplitDimd would not work here since its output is a dictionary with a key for each split. We will have a new component or will update this one by next week and then it should be usable.

[bhashemian]

Hi @DocandBean,

Thanks for your interest in using MIL pipeline. We are in the middle of making some changes to GridPatch and RandGridPatch, and consequently minor changes to MIL model. Hopefully, next week we have everything merged and you can utilize them in your pipeline. I will keep you posted.

[DocandBean]

Great. Thanks for the update. I'm looking forward to using this part of the MONAI code