UnnaturalGrams Python Extension Module

[eddieantonio]

MITLM is kind of a dodgy piece of work. So, (un)naturally, we'll replace it.

What this will fix

• Remove our dependency on ZeroMQ
• Remove our dependency on a shifty outside process
• Allows us to use weighted n-grams

What needs to be present for a full replacement

• Compute the cross-entropy of some tokens against the corpus
• Predict what follows a given token prefix
• Train the corpus with tokens
• Create the Python extension module wrapper for the C library