Warning with batch std when implementing LOGO

[mhking71]

Hi,

I have a very small dataset, so I was trying to implement Leave One Group Out cross-validation. When I do this, pytorch gives a user warning related to the computation of batch standard deviation in the forward method (my prediction set is just one bag).

I am not sure the best way to resolve this, but I think this is affecting inference. I think one way to do this is to set the unbiased kwarg to false, but I don't know how performance will change either when this is done.

Happy to share some of my code snippets if that is helpful.

Best,
Mike

[jan-engelmann]

Hi Mike,
Yes. Sharing code for this would be helpful.
Thanks!

[jan-engelmann]

Can you pass scaling=(0, 1) to the predict method?

Or alternatively look into the forward code how the scaling parameters are calculated there and calculate mean and std on the training dataset. Then again pass them to the scaling parameter of the predict method

[jan-engelmann]

What's your sample size? Likelihood/outcome? Embedding type (Xs)?

[mhking71]

My sample size is only 18 (very small hence the LOGO). I am predicting binary outcomes and am using scVI embeddings.

I'll play around with it using your suggestions and will update later.

Thanks!

[jan-engelmann]

yes. that's very few donors. a good check is using LogisticRegressionCV on covariates + mean scVI per donor embeddings. This is very close to the MixMIL mean model. First make sure you're able to predict from that.

Is it balanced cases vs controls?

[mhking71]

Hey Jan,

Sorry for taking so long -- I've been working on a different project for the last few weeks. I tried scaling=(0, 1) in the predict method and got this error:

'float' object has no attribute 'isnan'

I believe it is coming from this line:

if std.isnan().any():
std = 1
u = b * (u - mean) / std

[jan-engelmann]

That means you have to pass a tensor which has the .isna and .any method

[jan-engelmann]

Or just implement your own predict function