You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fix string indexing bug with Phi-4 mm tokenization (microsoft#1751)
### Description
This PR brings in two new out-of-bounds checks when running each step
while matching the first Phi-4 multimodal pattern during tokenization.
### Motivation and Context
These checks were added to ORT extensions in [this
PR](microsoft/onnxruntime-extensions#997).
Without these checks, the following error can happen in some
environments.
```
/opt/rh/gcc-toolset-12/root/usr/include/c++/12/string_view:239: constexpr const std::basic_string_view<_CharT, _Traits>::value_type& std::basic_string_view<_CharT, _Traits>::operator[](size_type) const [with _CharT = char32_t; _Traits = std::char_traits<char32_t>; const_reference = const char32_t&; size_type = long unsigned int]: Assertion '__pos < this->_M_len' failed.
```
0 commit comments