[OpenVINO backend] suggesting disable reusing compiled model until the bug get fixed #21659
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @Mohamed-Ashraf273, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This change addresses a critical bug in the OpenVINO backend that affects the reliability of model predictions when compiled models are reused. By temporarily disabling the caching of these models, the PR ensures consistent and correct inference results, prioritizing accuracy over potential performance gains from caching until the underlying issue is resolved.
Highlights
- Disabling OpenVINO Compiled Model Reuse: This pull request temporarily disables the caching mechanism for OpenVINO compiled models. This is a workaround for a known bug (openvinotoolkit/openvino#32045) where reusing a compiled model can lead to inconsistent predictions for the same input, particularly observed with the Gemma model.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request disables the caching of compiled OpenVINO models as a temporary workaround for a bug causing inconsistent predictions. The change is straightforward and effectively prevents the bug by forcing re-compilation. I've suggested a minor improvement to the implementation to use a boolean flag instead of commenting out code, which should improve readability and make it easier to re-enable the cache in the future.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #21659 +/- ##
==========================================
- Coverage 82.55% 82.54% -0.01%
==========================================
Files 571 571
Lines 57626 57629 +3
Branches 9001 9002 +1
==========================================
Hits 47572 47572
- Misses 7759 7761 +2
- Partials 2295 2296 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
4 participants
@rkazants
@fchollet
Based on issue: openvinotoolkit/openvino#32045
Discovered from PR: keras-team/keras-hub#2389
I suggest disabling reusing compiled models until the bug gets fixed.
Note: the bug is not related to the PR: keras-team/keras-hub#2389 itself, it is related to getting the predictions with the same input for the same compiled model being different (for now, this issue has been found only on the Gemma model).