Interest in Project 17 – Optimized Serving Endpoint for ML Training & Inference

[darksapien23151]

Dear Dariusz Trawinski, Milosz Zeglarski,

This project sounds incredibly exciting, and I’d love to understand more about its scope and expectations. The idea of integrating oneDAL optimizations with scikit-learn and deploying a training + inference endpoint via OpenVINO Model Server is fascinating, and I see a lot of potential in making ML model serving more efficient and scalable.

However, I’d love some clarification on a few aspects:

• Is the goal to support multiple classic ML algorithms out of the box, or is there a specific subset that would be the focus?

• Will the model training endpoint include hyperparameter tuning and dataset pre-processing, or should the client handle that before sending requests?

• For versioned model storage, is there an existing structure in mind, or would defining an efficient model management system be part of the project scope?

• Additionally, is there flexibility for exploring adaptive model selection or smart caching to further enhance performance? Or would the direction be entirely up to the candidates?

Would love to hear your thoughts.
Looking forward to a smooth collaboration!
Harshitha Manne

[adrianboguszewski]

@dtrawins @mzegla

[dtrawins]

@darksapien23151 thank you for your interest and enthusiasm :-) The idea is to make this solution with built in several algorithms with option to extend it to be fully generic by updating the python script. So the expectation is so have at least several like SVM, Kmeans, linear regression, correlation.
Regarding training, I think hyper parameters could be a nice possible extension. My initial though is that the preprocessing should be on the client side do the exported model would accept the same format of the data like sent for the training.
OpenVINO Model Server already support version management for inference serving https://docs.openvino.ai/nightly/model-server/ovms_docs_models_repository_classic.html
The the training part would upload the model to sequential folder name
The high level proposal is to use:

• python node in the OVMS graph https://docs.openvino.ai/nightly/model-server/ovms_docs_python_support_reference.html similar but simplified to this demo https://docs.openvino.ai/nightly/model-server/ovms_demo_clip_image_classification.html
• https://github.com/uxlfoundation/scikit-learn-intelex in the python script to run the training
• exporting the model in ONNX format for serving and exposing KServer API https://docs.openvino.ai/nightly/model-server/ovms_docs_clients_kfs.html
  There is definitely flexibility to explore other performance optimization.

[darksapien23151]

Hey @dtrawins , @mzegla,

Okay, this sounds really cool! Thanks for the breakdown; it definitely helped me get a clearer picture of how things are structured. I’ll dig deeper into OpenVINO Model Server’s versioning and how we can best integrate it for smooth model management.

Also, I’ve previously worked on and contributed to an issue related to ONNX, so I assume I already have a good understanding of ONNX models and their deployment. That experience makes me even more excited about this project and how we can leverage ONNX for efficient inference.

This project aligns really well with my interests in optimizing ML workflows, and I’d love to contribute and get hands-on with implementing these ideas. It would be great to discuss potential next steps on how I can get involved!

Looking forward to hearing your thoughts!!

[darksapien23151]

@dtrawins , @mzegla,
I've mailed you the prototype for the model I've made and would like to hear more about it!

[darksapien23151]

Hey @adrianboguszewski,
Could you kindly follow up with the mentors regarding this? It’s been a while since their last response.
Thanks!

[adrianboguszewski]

I pinged them

[darksapien23151]

Good Day Mentors, @dtrawins , @mzegla
As you've mentioned on converting the model in ONNX format and exposing it to the clients through the KServe API, I wonder if we could further convert it into IR format, as it would further enhance the performance of the pipeline as mentioned here:
https://docs.openvino.ai/2025/openvino-workflow/model-preparation/convert-model-to-ir.html
So, according to the workflow I've reached upon, it would look something like ,
Client Input → Model Training (scikit-learn-intelex) → ONNX Conversion → IR Conversion → OVMS Deployment → Quantization → KServe API → Client Inference