Impact of Imbalance validation dataset on threshold calculation

[lathashree01]

Hi,

I am performing a trial anomaly detection with patchcore and I see a performance drop than the previous iteration. There are no changes in model, however the data is quite imbalanced now i.e. I have higher normal examples (more than 2X) than anomalous.

As I am using F1 based threshold ((#1896) calculation so could this imbalance cause performance drop? but shouldn't the F1 be more robust than other metrics and help with this problem?

In a real-world case, usually we have more normal images than anomalous, how to get around this bottleneck?
Or is the imbalance not really the cause for this problem?

Thanks for help.

[alexriedel1]

What do you mean by "... performance drop than the previous iteration." ?
The number of anomalous samples shouldn't have an influence on your F1 Score, but maybe you have some harder to classify anomalous samples than before.

[lathashree01]

Hi @alexriedel1

Thanks for your reply.
I see a performance drop by 10% than my previous iteration of model. I am investigating the causes for this.

One straight forward cause could be the dataset is filtered and false anomalous are removed and hence performance is on-par with dataset I have.
When I said F1, I meant during F1Adaptivethreshold calculation will the imbalanced validation dataset impact the threshold calculation?