`PoolBasedActiveLearner.update()` does not update labels for `self.dataset`

[ahoho]

I'm not sure if this is deliberate, but PoolBasedActiveLearner.update() does not appear to update its self.dataset labels? Would it make sense to include an update_dataset() method?

[chschroeder]

Hi @ahoho, this is working as intended. You are probably using it in a practical setup, correct? There it makes sense that one would expect to update the dataset as well.

For the experimental setting we would override the labels if we set them during update. There I considered the dataset more of a "read-only" source of truth.

This is the history of this mechanism, but we can open this for discussion. Would you mind to elaborate where you encountered a problem and what you would expect / prefer to have?

Thanks for the input! I have been collecting a few items like these, and noticed there is still a lot of functionality where the convenience could be improved. I am thankful for any input, since it is much more diffcult to spot these issues, after having worked with the same code for quite some time.

[ahoho]

Thanks, your explanation makes sense! The issue was that I was computing some evaluation metrics post-training, and assumed that the queried labels would be updated in the data (I was doing something likenp.mean(active_learner.dataset.y[queried_indices] == pred) to compute train accuracy, then doing some post-hoc calibration, both of which silently failed because of the uncaught LABEL_UNLABELED). That said, it's a one-liner fix, so not really a big problem

Overall, the package is very nice, and I was able to implement what I wanted in less than a day! (Although my results aren't great, and I need to play around with the different querying strategies and SetFit hyperparameters)

[chschroeder]

Still, you are right and convenience is a topic where I already have quite a few issues that need to be improved. Maybe an additional flag like you suggested would be the best option? Just changing it to do the opposite might break the experiment setup of others, which is why I want to avoid that for now. For now, I will leave it open for others to comment.

And thank you for the kind words, happy to hear that! Nevertheless, any feedback is welcome. I can also improve what I am aware of ;).