Questions on Automatic Data Collection and Labeling in Data Drift

[qyy2003]

Data drift can degrade model performance. Small models deployed on mobile and edge devices suffer more than large foundation models.

A common mitigation strategy is to fine-tune and redeploy the model. However, relying on experts to manually collect, label, and fine-tune models at regular intervals is impractical. Both model monitoring and fine-tuning require access to ground truth, raising the critical question: How can we automate data collection and, more importantly, labeling?

Note: The approach discussed in Chapter 6: Data Engineering still appears to require human annotators, and Chapter 14: On-Device Learning does not address this issue.