EnsembleEvaluator() produces no output

[ahxmeds]

Hi all,

I have trained 5 models on 5 folds of my data, and I have saved those 5 best models. I was looking into ensembling the 5 models via mean, weighted-mean and majority vote. I was roughly following the tutorial given here from the Define evaluation process based on EnsembleEvaluator section onwards. My models variable is initialized as a list of models from the 5 folds of training. But the evaluator.run() line doesn't produce any output, whether I run on CPU or GPU. Please let me know what could be the issue here. Some relevant portions of my code are given below:

mod_keys = ['CT', 'PT', 'label']
test_transforms = Compose(
    [
        LoadImaged(keys=mod_keys),
        EnsureChannelFirstd(keys=mod_keys),
        ScaleIntensityRanged(keys=['CT'], a_min=-1024, a_max=1024, b_min=0, b_max=1, clip=True),
        ScaleIntensityd(keys=['PT'], minv=0, maxv=1),
        CropForegroundd(keys=mod_keys, source_key='PT'),
        Orientationd(keys=mod_keys, axcodes="RAS"),
        Spacingd(keys=mod_keys, pixdim=(2.0, 2.0, 2.0), mode=('bilinear', 'bilinear', 'nearest')),
        ConcatItemsd(keys=['CT', 'PT'], name='image', dim=0),
        DeleteItemsd(keys=['CT', 'PT']),
        EnsureTyped(keys=["image", "label"]),
    ]
)

dataset_test = CacheDataset(data=test_data[0:2], transform=test_transforms, num_workers=16)
dataloader_test = DataLoader(dataset_test, batch_size=1, shuffle=False, num_workers=16)

mean_post_transforms = Compose(
    [
        EnsureTyped(keys=["pred0", "pred1", "pred2", "pred3", "pred4"]),
        MeanEnsembled(
            keys=["pred0", "pred1", "pred2", "pred3", "pred4"],
            output_key="pred",
            # in this particular example, we use validation metrics as weights
            weights=max_dscs,
        ),
        Activationsd(keys="pred", sigmoid=True),
        AsDiscreted(keys="pred", threshold=0.5),
    ]
)
evaluator = EnsembleEvaluator(
    device=torch.device('cpu'),
    val_data_loader=dataloader_test,
    pred_keys=["pred0", "pred1", "pred2", "pred3", "pred4"],
    networks=models,
    inferer=SlidingWindowInferer(roi_size=(192, 192, 192), sw_batch_size=4, overlap=0.25),
    postprocessing=mean_post_transforms,
    key_val_metric={
        "test_mean_dice": MeanDice(
            include_background=False,
            output_transform=from_engine(["pred", "label"]),
        )
    },
)
evaluator.run()

[KumoLiu]

Hi @ahxmeds, did you get any error messages? The code looks good to me. Thanks!

[ahxmeds]

It started working when I included logging.basicConfig(stream=sys.stdout, level=logging.INFO) line in the code.