File tree Expand file tree Collapse file tree 2 files changed +13
-1
lines changed
bindings/python/pyonmttok Expand file tree Collapse file tree 2 files changed +13
-1
lines changed Original file line number Diff line number Diff line change @@ -26,6 +26,18 @@ The project follows [semantic versioning 2.0.0](https://semver.org/). The API co
2626
2727### Fixes and improvements
2828
29+ ## [ v1.37.0] ( https://github.com/OpenNMT/Tokenizer/releases/tag/v1.37.0 ) (2023-02-28)
30+
31+ ### New features
32+
33+ * Add tokenization option ` allow_isolated_marks ` to allow combining marks to appear isolated in the tokenization output in specific conditions
34+
35+ ### Fixes and improvements
36+
37+ * Fix infinite loop when the text contains an invalid Unicode character
38+ * Fix segmentation fault when the ` BPELearner ` does not not find any pairs of characters in the tokenized data
39+ * [ Python] Update ICU to 72.1
40+
2941## [ v1.36.0] ( https://github.com/OpenNMT/Tokenizer/releases/tag/v1.36.0 ) (2023-01-11)
3042
3143### New features
Original file line number Diff line number Diff line change 11"""Version information."""
22
3- __version__ = "1.36 .0"
3+ __version__ = "1.37 .0"
You can’t perform that action at this time.
0 commit comments