Shared Vocab when saving a set of PhraseMatchers

[oterrier]

Hi all,
In my project I'm using a set of PhraseMatchers (one per label) sharing the same nlp.Vocab object
matchers: Dict[str,PhraseMatcher]
When I want to save/restore this I'm simply pickling the matchers dictionary and voila.
But then I'm wondering if there is a trick to avoid saving N times the big Vocab (which is the same for all PhraseMatchers) because the loading time can be huge so I try to optimize it as much as I can

Thanks in advance for your help

Olivier

[danieldk]

Thanks for your question! If you are using the regular pickle module, shared objects should only be pickled once:

The pickle module keeps track of the objects it has already serialized, so that later references to the same object won’t be serialized again.

So as long as you pickle the whole dictionary at once, there shouldn't be N vocabulary objects.