Reaching out to mentors for project 11 (Leveraging Large Foundation Models (LFMs) for Automated Annotation and Edge-Deployable Model Training with Human-In-Loop).

[wx-yu-zu]

Dear Rajesh Gangireddy and Samet Akcay,

I am a master's student at Kyoto University and will be starting my PhD program this spring. I specialize in computer science and machine learning.

My master's thesis focuses on automated annotations with large multimodal models. I explored methods for image annotation in tasks such as object detection, segmentation, and optical character recognition (technically, I also focused on data privacy preservation throughout these processes). After submitting my master’s thesis, I have just started another research project on model improvement with human-in-the-loop.
I am genuinely passionate about contributing to OpenVINO with my research experience, as this project perfectly meets my research interests.

I would like to discuss this project further and would appreciate your guidance on the following points:

• Are there any existing prototypes, code resources, documents, or reference materials that would help me better understand the project? I would greatly appreciate it if you could share them.
• Do you have any preferred methods (or related work) for active learning or uncertainty estimation? I would like to familiarize myself with them.
• I would appreciate it if you could review my proposal when it is ready. I would like to know the key points to be aware of or any preparations to be done while drafting it.

Thank you so much for offering such an exciting project and this valuable opportunity.
I would appreciate your response.

[rajeshgangireddy]

Hi @wx-yu-zu
Thank you for your questions. Great that your master thesis and research interests aligns with this topic.

1. As for now, there is not much specific to this topic in our code repos.
   The idea is to integrate this project into our existing set of open source frameworks - (OpenVINO Training extensions (OTX) (for training and inference) and Datumaro (for dataset management).

2. For active learning, we have done a lot of research in terms of algorithms and have a set of algorithms that we can try out.
   We will start with the Feature Reconstruction Error (FRE) algorithm for sample selection (introduced in https://arxiv.org/pdf/2109.06873 ) and test out other algorithms based on time availability.

Let me know if this answers your questions. Feel free to ask more questions.
We look forward to your proposal.

Thanks!

[wx-yu-zu]

@rajeshgangireddy
Thank you so much for your helpful answers!
I have three additional questions as I work on my proposal:

1. Active learning aims to minimize human annotation costs and labor by efficiently selecting data for annotation. Since LFM-based annotation is highly scalable compared to human annotation, minimizing LFM-generated annotations is not necessarily required. Therefore, the sampling process in active learning should be employed to select data for label correction by human annotators. Is my understanding correct?

2. Are the lightweight models trained in edge environments or external environments? (I assume the latter, as training on edge devices requires consideration of various environmental factors.)

3. How can I share my draft proposal with you? Should I send it via email?


I would appreciate your response.

[rajeshgangireddy]

Hi @wx-yu-zu

1. Therefore, the sampling process in active learning should be employed to select data for label correction by human annotators.

Exactly. That is one way in which we think AL Algorithms to be useful. Another way in which we plan to use AL is to select a subset of more informative samples out of the thousands annotated ( automatically +manually) to make training more efficient without loosing any accuracy.

2. External Environments.

3. Please submit your FINAL proposal to be considered. You can see the details here : https://github.com/openvinotoolkit/openvino/wiki/Google-Summer-Of-Code

[wx-yu-zu]

Hi @rajeshgangireddy

Thank you for your quick response.

I'm glad to hear that my understanding was correct.
I should have provided more details regarding the third question.

I have written a document outlining a possible solution that I plan to include in my final proposal. I would greatly appreciate any feedback on my solution so that I can refine my proposal before the deadline.

For this reason, I would like to share my draft proposal with you. (This is not for official submission but to receive insightful feedback before the final submission.)

[rajeshgangireddy]

Hi @wx-yu-zu
Please email your document to @samet-akcay and me.
You can find our Work (Intel.com) emails in our respective Github profiles.

[wx-yu-zu]

Hi @rajeshgangireddy
I've sent my draft solution. Thank you for your time and consideration!