Clarification

[mineshmathew]

It seems you are computing all N grams instead of N gram prefixes given in the paper.
For example in case of a QA dataset , what we need is prefix n grams or say first N words of questions and then a sunburst of that.

But you seem to compute all possible ngrams from each sentence.

Please correct me If I am wrong.

[CCYChongyanChen]

to use the prefix instead of all n gram, we can probably update it to:

def computeNgramTrie(args, sentences: list, n: int=1, deliminator=" ") -> list:
    all_ngrams = Trie(args)
    preprocess_func = lambda l: l.split(deliminator) if deliminator else l
    all_ngrams.logger.info("constructing ngram Trie")
    for sentence in sentences:
        sentence = preprocess_func(sentence)
        ngrams = [sentence[i] for i in range(len(sentence))]
        # print(ngrams)
        all_ngrams.insert(ngrams)
    all_ngrams.postprocess()
    return all_ngrams