Performance issues with classwise method on large imbalanced dataset - seeking methodology recommendations

[jeandre-db]

Hello PyDVL team,

I am working on a data valuation project. I am encountering some challenges that I hope you can help me with.

Background:
I have implemented the classwise method for data valuation on my dataset. However, I am experiencing significant performance issues. The computation has been running for several hours and then it seems like it does not provide a value for all the samples in the dataset.

Dataset characteristics:

• Very large dataset
• Highly imbalanced classes (approximately 95/5 distribution)
• Currently using classwise methodology

Is the classwise method the most appropriate choice for imbalanced datasets of this size?

I apologize if these are basic questions. I want to make sure I am approaching this correctly.

Any guidance would be greatly appreciated. Thank you for your time.

[mdbenito]

Hi,

it's been a while since I worked on this, I'm not sure I can say something useful... FWIW:

• Generally speaking, game-theoretic data valuation methods like CW-Shapley are not suited for very large datasets (how many points are we talking about in your case?), especially with larger model sizes, because of the cost of training the model of interest on so many subsets of the training set. This is partially offset by Monte Carlo sampling strategies like PermutationSampler with a TruncationPolicy, but YMMV. What model are you fitting?
• CW-Shapley should in principle be a good choice for imbalanced datasets. We saw good performance in some cases like the "Click" dataset, but not for all experiments. It's hard to say something in general.
• (Random) things to watch out for are:
  • Your training/valuation sets are properly stratified by class
  • If you are running things in parallel you are not oversubscribing by parallelizing both model training and valuation fitting (just do one).
  • The inner and outer sampler in CW-Shapley are properly configured, which should be the case if you followed the documentation.
  • Your model is actually taking into account the class imbalance, and is not overfitting to the majority class. Recall that it is retrained on subsets of increasing size, and for some sizes, the fitting might be meaningless. You can investigate whether this is the case and try to use pydvl's stratified samplers (by set size) instead, but I don't think we ever actually tried that.

You can check the code that @kosmitive wrote for the paper I linked above here.

[jeandre-db]

Thanks for the prompt reply, I appreciate it!
To answer some of your questions, I think it is a reasonably large dataset of 65000 samples, the class imbalance is quite severe at 5% positive class. The model is a quite simple logistic regression model.
I did indeed see the good performance on the "Click" dataset and that was my inspiration to try it for this dataset.
The part I find quite challenging is guesstimating the right values for max_in_class_samples, for the PermuationSampler etc.
it can be rather difficult to iterate on these when the time to do one valuation can be quite long.