spacy.TextCatBOW.v1 - further documentation

[hp52]

Hi,
I'm reading the documentation about the TextCatEnsemble.v2. It says that it's a stacked ensemble of a linear bag-of-words model and a neural network model. As a linear model, a TextCatBOW is used in the example which is described as an n-gram “bag-of-words” model. Since there is no more documentation about it, I'm wondering what kind of “bag-of-words” model it is?
Could you give an explanation?
Thanks for your help.

[svlandeg]

Hi! The TextCatBow architecture extracts n-grams from the text. If n is 3, it'll fetch combinations of 3 consecutive tokens. A true "bag of words" model is obtained when you set n to 1, then it'll extract each word separately. There's a linear layer following those n-grams. The final output layer depends on whether or not the classes of your textcat are exclusive: if they are, the output layer is a softmax activation, otherwise it's a sigmoid activation layer (also called Logistic in some of our code). You can find the code implementation here: https://github.com/explosion/spaCy/blob/master/spacy/ml/models/textcat.py

And if you run init config, you can see some default values for this implementation, e.g:

spacy init config -p textcat textcat.cfg -o accuracy

would give you

...
[components.textcat.model]
@architectures = "spacy.TextCatEnsemble.v2"
nO = null

[components.textcat.model.linear_model]
@architectures = "spacy.TextCatBOW.v1"
exclusive_classes = true
ngram_size = 1
no_output_layer = false
nO = null

[components.textcat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"
...

Where you see that by default, n is set to 1 (single tokens instead of n-grams)

[Joshmantova]

After reading through the implementation code, my understanding of this model is that it uses TF embeddings with n-grams before feeding that TF matrix to a single hidden layer neural net. Is that understanding accurate?

[polm]

By TF you mean term frequency, right? If so that's an accurate description.

[Joshmantova]

Yes, I do mean term frequency. It also sounds like the hidden layer has a linear activation function - is that correct as well?

[polm]

Yes, the hidden layer called SparseLinear has a linear activation function.