How do I split two combined words that have no separators?

[moonman239]

I've got a dataset of forum posts. Sometimes users will forget to use the space bar, resulting in spaCy being unable to recognize the words. For example, sometimes they'll type "disneyworld" instead of "disney world."

I have a function that works well for n=2 words, but not necessarily for anything greater than that; in fact, I cannot be sure it works at all for n>2.

Here's the function:

def word_splitter(
    baseStr: str, nlp
):  # splits a string into recognizable words.
    stringLength = len(baseStr)
    if (stringLength == 1):
        if (baseStr == 'a'):
            return baseStr
        else:
            return None
    firstWords = []
    secondWords = []
    # split baseStr into two parts
    for i in range(stringLength):
        part1 = baseStr[0:i + 1]
        part2 = baseStr[i + 1:stringLength]
        part1IsWord = is_word(part1, nlp)
        part2IsWord = is_word(part2, nlp)
        if (part1IsWord):
            firstWords.append(part1)
        if (part2IsWord):
            secondWords.append(part2)
        if (part1IsWord and part2IsWord):
            return part1, part2
    # failed to split into two words; there may be more than two words.
    wordsToTry = firstWords + secondWords
    print("word {0} not adequately split, let's try {1}".format(
        baseStr, wordsToTry))
    return tuple(
        [word_splitter(baseStr.replace(word, ""), nlp) for word in wordsToTry])

[pmbaumgartner]

You could try using symspell, which I cover in a bit more detail in this discussion.