How to do this test in a lightning way?

[kaare-mikkelsen]

My model has the property that I can prepare the test data in multiple different ways, which results in a set of equally plausible predictions for each data point (one prediction for each way of preparing the test data). By combining these predictions, it is possible to slightly boost overall performance on the test set. Right now, I do this in the following (abstract) way:

---

for iPrep in range(nPrep):
   preppedData=prepare_data(testData,iPrep)
   predictions[iPrep]=trainer.test(model,preppedData)

final_predictions=combinePredictions(predictions)

---

(obviously it is much longer in reality)

is there a proper, 'lightning' way of hiding this loop inside the model, so I can still use the trainer for this, but only call it once?

[awaelchli]

Maybe the prediction api can help you (currently beta, will be released in version 1.3).

You can have multiple predict dataloaders (your different test data). If you do

predictions = trainer.predict(model, predict_dataloaders=[data1, data2, data3, ...])

and it returns the predictions grouped by the dataloader index. Then you can combine them with your own function.

final_predictions=combinePredictions(predictions)

Not sure if this is 100% what you are looking for, but it can at least eliminate that one for loop you have.
Optionally, you can also override predict_step in the LightningModule.
If you install the latest version, you can use this predict feature already. The documentation will be included in the 1.3 release.