Improve space efficiency for regex building by using a radix tree #9036
Conversation
Sources/Utils.php
Outdated
| $prefix = mb_substr($key, 0, $len, $encoding); | ||
| $key_remainder = mb_substr($key, $len, null, $encoding); | ||
| $remaining_remainder = mb_substr($remaining, $len, null, $encoding); |
There was a problem hiding this comment.
I have some code which cuts these string based on byte boundaries, but that will make this function UTF-8 only. Would that be acceptable?
There was a problem hiding this comment.
Splitting into bytes rather than characters is indeed more space efficient, but sadly it's not an option for us. That's because splitting into bytes will introduce byte sequences that are not valid UTF-8 into the overall regex. Although the preg_* functions themselves can handle that, having invalid byte sequences will cause problems when trying to store the compiled regular expressions.
For example, we store the TLD regex in the database. If the regex contains invalid byte sequences, the database will reject it.
Never mind. I misread the proposed code.
I'm pretty sure I did see the gist before, but now I get a 404 at that link. Do you have the privacy setting backwards? |
Sesquipedalian
dismissed
their stale review
Not relevant after all due to my own misreading of the proposed changes.
|
In my testing, these proposed changes are slower than the existing code. However, there are some good ideas in here for improving performance. |
3 participants
My test of over 1000 cities shows a sizeable memory reduction. Sidebar: This gist was accidentally left private in the last update, yet no one mentioned it.