Assignment 1.3 - Rare words

[daviddao]

In Assignment 1.3 it is written: "This will load the data in a bag-of-words representation where rare words (occurring less than 5 times in the training data) are removed". However, when I sum the word occurrences using the provided training dataset with

scr = srs.SentimentCorpus("books")

I get words, which doesn't appear at all (occurring less than 5 times)

>> scr.train_X.sum(0)
[..., 0.0, ...]

[negrinho]

Maybe the indexing has been built using also the development data. We will take a look.

[ChristopherBrix]

Yes, the whole corpus (training + dev) is used to discard rare words. This is because training and dev are not separated until after this filtering is performed.

>> srs.SentimentCorpus('books').X.sum(0)
[   5.  221. 1639. ...    5.    5.    6.]