Skip to content

Commit d77033c

Browse files
Fix string indexing bug with Phi-4 mm tokenization (microsoft#1751)
### Description This PR brings in two new out-of-bounds checks when running each step while matching the first Phi-4 multimodal pattern during tokenization. ### Motivation and Context These checks were added to ORT extensions in [this PR](microsoft/onnxruntime-extensions#997). Without these checks, the following error can happen in some environments. ``` /opt/rh/gcc-toolset-12/root/usr/include/c++/12/string_view:239: constexpr const std::basic_string_view<_CharT, _Traits>::value_type& std::basic_string_view<_CharT, _Traits>::operator[](size_type) const [with _CharT = char32_t; _Traits = std::char_traits<char32_t>; const_reference = const char32_t&; size_type = long unsigned int]: Assertion '__pos < this->_M_len' failed. ```
1 parent 27c591f commit d77033c

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

cmake/deps.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ pybind11;https://github.com/pybind/pybind11/archive/refs/tags/v2.13.6.zip;f78029
1414
googletest;https://github.com/google/googletest/archive/530d5c8c84abd2a46f38583ee817743c9b3a42b4.zip;5e3a61db2aa975cfd0f97ba92c818744e7fa7034
1515
microsoft_wil;https://github.com/microsoft/wil/archive/refs/tags/v1.0.230629.1.zip;e4a542a323c070376f7c2d1973d0f7ddbc1d2fa5
1616
directx_headers;https://github.com/microsoft/DirectX-Headers/archive/refs/tags/v1.613.1.zip;47653509a3371eabb156360f42faf582f314bf2e
17-
onnxruntime_extensions;https://github.com/microsoft/onnxruntime-extensions.git;9cd41a1df3d9918253f01ef90a431fba7e29c4ff
17+
onnxruntime_extensions;https://github.com/microsoft/onnxruntime-extensions.git;bd8fb6d86e98c17e397c42fc001913cc2e035597
1818

1919
# These two dependencies are for the optional constrained decoding feature (USE_GUIDANCE)
2020
llguidance;https://github.com/microsoft/llguidance.git;2d2f1de3c87e3289528affc346f734f7471216d9

0 commit comments

Comments
 (0)