Is there any way to inference from hidden layers in SpaCy?

[mghmgh1281375]

I want to get the hidden layer outputs from the trained classifier in my application.
Is there any way to do that? does SpaCy capable of inference from hidden layers and just to returns sentence embeddings?

[svlandeg]

This is currently not available out-of-the-box in spaCy, but you could dive into the code to hack together what you need. Currently the Machine Learning architectures are defined in _ml.py, e.g. build_bow_text_classifier is called from TextCategorizer.Model when a new textcat pipe with the BOW architecture is made.

The machine learning models are created with Thinc. You have something like model = intermediate_network >> output_layer with >> meaning chain. Basically what you want to do, is get a hold of the intermediate_network reference (after training the full model) and use that to access the hidden representation. That should be entirely feasible if you feel like hacking around a bit ;-)

There's some changes happening though, which may be relevant for you as well.

The current spaCy 2.2.x versions rely on Thinc 7.x. However, there's been a complete overhaul of Thinc, and the new 8.0 version was recently released and documented at thinc.ai. We are currently in the process of also upgrading spaCy to use this new Thinc, which will make it much easier to define your own ML models for your pipeline components. Basically you will be able to define them in a config file that refers to registered functions. We are working on this on the develop branch, and you can see such an example config here. We've also moved & restructured the relevant ML code into a package here. So our example build_bow_text_classifier would now be living in models/textcat.py. You can define your own models in a similar fashion, register them, and refer to them from your custom config. I'm afraid much of this is still undocumented, though we're hoping to get spaCy 3.x out in the nearby future.

Thinc has this mechanism of keeping track of sublayers in a model by setting model.set_ref("hidden_layer", intermediate_network), which will allow you to access it later on.

If you do decide to start poking around on the develop branch, do realise that this is all very much a work in progress and we don't even have prebuilt models there yet. But we'd definitely be interested in hearing your feedback.

I'll close this issue as there is no immediate action for us right now, but please feel free to comment / ask further follow up questions and I'll do my best to address them!