Questions on DocBin

[Phat-Loc]

I am curious about the design decisions behind DocBin

1. Why isn't the vocab stored with the DocBin object but instead I need to pass it in when I call get_docs? What happens if I use a different vocab from the one I used to save the DocBin?

2. How come there isn't a remove method?

I assume it is probably to save space and prevent duplication of data.

[adrianeboyd]

Rather than saving the whole vocab, the DocBin saves the strings used in each doc so it can reconstruct all tokens / annotation.

A Doc is always created with an associated Vocab, so when you provide a vocab to DocBin.get_docs, this is the vocab associated with the returned doc and it's where all the strings are added again.

To save space, the strings are saved as one set for all docs, so you can't easily extract just the strings associated with one doc, so you couldn't really remove a doc cleanly. It could be possible to have a slightly unsatisfactory remove that didn't clean up the strings.

It's totally fine to use a different vocab with DocBin.get_docs. To order to end up with the same lexical attributes in the vocab (like LIKE_NUM), you would normally want to use a vocab for the same language as your original docs, but it's not required.

[Phat-Loc]

Thanks!