0.2.7pre1

[xhluca]

What's Changed

• Fix query filtering and vocabulary dict by @xhluca in Fix query filtering and vocabulary dict #96 and @mossbee in fix: token ID in the query higher than the number of tokens in the index #92

Notes

• The behavior of tokenizers have changed wrt null token. Now, the null token will be added first to the vocab rather than at the end, as the previous approach is inconsistent with the general standard (the "" string should map to 0 in general). However, it is a backward compatible change because the tokenizers should work the same way as before, but expect the tokenizers before 0.2.7 to differ from the tokenizers in 0.2.7 and beyond in the behavior, even though both will work with the retriever object.

Full Changelog: 0.2.6...0.2.7

---

This discussion was created from the release 0.2.7pre1.

[xhluca]

Let me know if this new release works well for you! there's some changes to the tokenizer behavior, so i want to share this here before making an official release