Using Sliding window on a classifier

[Vivekrpg]

Hello,

We are working on using densenet121 model for training a classifier on 3D MRI images. We are using RandCropByPosNegLabel random crop (images/mask) for the purpose of training. However for testing, we wanted to use sliding window inference so as to cover the entire MRI image to find the region of interest.

When I am trying to use sliding_window_inference, I am getting following error

sw_batch_size=4 print(test_images.shape) test_outputs = sliding_window_inference(test_images, (96,96,96), sw_batch_size, model)

torch.Size([1, 1, 512, 504, 157])
~/monai_code/MONAI/monai/inferers/utils.py in sliding_window_inference(inputs, roi_size, sw_batch_size, predictor, overlap, mode, sigma_scale, padding_mode, cval, sw_device, device, *args, **kwargs)
140 # store the result in the proper location of the full output. Apply weights from importance map.
141 for idx, original_idx in zip(slice_range, unravel_slice):
--> 142 output_image[original_idx] += importance_map * seg_prob[idx - slice_g]
143 count_map[original_idx] += importance_map
144

RuntimeError: The size of tensor a (96) must match the size of tensor b (2) at non-singleton dimension 2

[Nic-Ma]

@wyli can you find any possible reason from the error message?
Maybe need to provide the whole test program for further debug.

Thanks.

[Vivekrpg]

is sliding_window_inference supposed to work with classifier or should this be handled through a transform? Here is what I am trying to do:

data["image"]: Cropped MRI image
data["label"]: Cropped mask
data["new_label"]: Classifier label. if(data["label"].sum()>0): new_label=1 else new_label=0

Instead of using RandSpatialCropSamples for Validation, I would like to use sliding window that scans the entire MRI image and run inferencing on each sample that gets generated by sliding window. I would then compare the output of the model with the similar sliding window over data["label"] which is the mask to confirm if each sample got predicted correctly.

Here is the relevant code (This code is very constructed from what is presented in the 3d_classification tutorial.

`
train_transforms = compose_transform([
"LoadImage",
"ScaleIntensity",
"AddChannel",
"Orientation",
"CropForeground",
"RandCropByPosNegLabel",
"set_class",
"ToTensor"
], roi_size,train_num_samples_per_image)

val_transforms = compose_transform([
"LoadImage",
"ScaleIntensity",
"AddChannel",
"Orientation",
"CropForeground",
"RandSpatialCropSamples",
"set_class",
"ToTensor"
], roi_size,val_num_samples_per_image)

Create DenseNet121, CrossEntropyLoss and Adam optimizer, Learning rate is 1e-5

model = monai.networks.nets.densenet.densenet121(spatial_dims=3,
in_channels=1,
out_channels=2
).to(device)
loss_function = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), 1e-5)

#Start training and validation
val_interval = 1
max_epochs = 2

for epoch in range(max_epochs):
print("-" * 10)
print(f"epoch {epoch + 1}/{max_epochs}")
model.train()
epoch_loss = 0
step = 0
for batch_data in train_loader:
print(f"Starting batch: {step}")
if(debug):
print_gpu_memory()
t0 = process_time()
step += 1
inputs, labels = batch_data["image"].to(device), batch_data["new_label"].to(device)
optimizer.zero_grad()
outputs = model(inputs)
loss = loss_function(outputs, labels.flatten())
loss.backward()
optimizer.step()
epoch_loss += loss.item()
batch_loss_values.append(loss.item())
epoch_len = len(train_ds) // train_loader.batch_size
print(f"{step}/{epoch_len}, train_loss: {loss.item():.4f}, batch_time: {(process_time()-t0):.4f}")
writer.add_scalar("train_loss", loss.item(), epoch_len * epoch + step)
epoch_loss /= step
epoch_loss_values.append(epoch_loss)
print(f"epoch {epoch + 1} average loss: {epoch_loss:.4f}")
torch.save(model.state_dict(), modelname+"_epoch"+str(epoch)+".pth")

# Run validation after val_intervals on the trained model
t2 = process_time() 
if (epoch + 1) % val_interval == 0:
    model.eval()
    with torch.no_grad():
        num_correct = 0.0
        metric_count = 0
        y_pred = torch.tensor([], dtype=torch.long, device=device)
        y = torch.tensor([], dtype=torch.long, device=device)
        for val_data in val_loader:
            val_images, val_labels = val_data["image"].to(device), val_data["new_label"].to(device)
            val_labels = val_labels.flatten()
            #val_outputs = model(val_images)
            sw_batch_size=200
            val_outputs = sliding_window_inference(val_images, roi_size, sw_batch_size, model)
            value = torch.eq(val_outputs.argmax(dim=1), val_labels)
            metric_count += len(value)
            num_correct += value.sum().item()
            y_pred = torch.cat([y_pred, val_outputs.argmax(dim=1)], dim=0)
            y = torch.cat([y, val_labels], dim=0)
        metric = num_correct / metric_count
        metric_values.append(metric)
        # print the confusion matrix for the validation data
        print_output_distribution(y,y_pred)
        print_confusion_matrix(y, y_pred)
        print_accuracy_on_each_sample_mri_image(y,y_pred,val_num_samples_per_image,0)
        print_accuracy_for_any_positive_sample_mri_image(y,y_pred,val_num_samples_per_image,0)
        if metric > best_metric:
            best_metric = metric
            best_metric_epoch = epoch + 1            
            print("saved new best metric model")                                              
        print("current epoch: {} current accuracy: {:.4f} best accuracy: {:.4f} at epoch {}".format(epoch + 1, metric, best_metric, best_metric_epoch))
        writer.add_scalar("val_accuracy", metric, epoch + 1)
print("Validation time: {:.4f} ".format(process_time()-t2))

print(f"train completed, best_metric: {best_metric:.4f} at epoch: {best_metric_epoch}")
`

[wyli]

this is a use case that is currently not directly supported by sliding_window_inference. perhaps after training, you could try to convert the classification model into a fully convolutional one

[wyli]

just a follow up of this answer, now we have

MONAI/monai/networks/nets/netadapter.py

Line 19 in fe559e5

class NetAdapter(torch.nn.Module):

a use case is with the torchvision model

MONAI/monai/networks/nets/torchvision_fc.py

Lines 23 to 25 in fe559e5

	class TorchVisionFCModel(NetAdapter):
	"""
	Customize the fully connected layer of TorchVision model or replace it by convolutional layer.

[dzenanz]

One possibility is to do tiling yourself. Here is an example:
https://github.com/dzenanz/miqa/blob/976e775dfc1030c33df4b83384814ac4ab2b9b0b/learning/nn_classifier.py#L130-L174