[OpenVINO] Add cache_position input inside prepare_inputs method for Mamba
#1517
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
rkazants
added
the
openvino-slow
Co-authored-by: Roman Kazantsev <[email protected]>
| if kwargs.get("past_key_values") is not None: | ||
| raise ValueError("`past_key_values` input is not supported for `OVModelWithMambaForCausalLM`") | ||
| if kwargs.get("position_ids") is not None: | ||
| raise ValueError("`position_ids` input is not supported for `OVModelWithMambaForCausalLM`") |
There was a problem hiding this comment.
is this really needed ?
There was a problem hiding this comment.
My motivation was to avoid possible confusion if anyone passes these arguments because parent's definition of prepare_inputs() has these arguments in its signature.
There was a problem hiding this comment.
hmm i'm honestly not sure why this function is public, imo it should be private (it serves one purpose, an internal one)
There was a problem hiding this comment.
It is used externally during quantization https://github.com/huggingface/optimum-intel/blob/main/optimum/intel/openvino/quantization.py#L675
There was a problem hiding this comment.
that's still internal though 😅.. what i mean is that it shouldn't be used by a user
the downside of a public method is that we need to maintain its behaviour and only change it through deprecation process (over multiple versions).
What does this PR do?
cache_positioninput is required to run inference. Needed for enabling data-aware quantization of Mamba model openvinotoolkit/nncf#3725 .Before submitting