How to pickle a ".spacy" dataset

[dave-espinosa]

Hello everyone.

I am currently implementing a Vertex AI Pipeline to train a spaCy model. Currently, I need to pass the "train.spacy" and "dev.spacy" datasets, from a "Preprocess" component, to a "Train" component (this is only general info, does not matter much if you're not familiar with Vertex AI Pipelines). However, the only way in which this can be achieved, is converting those ".spacy" files, into pickle. I have seen only similar approaches, however nothing close.

Do you have any idea about how to do that?

Thank you.

[koaning]

It surprises me that you could only use pickle files as artefacts in a tool like Google Vertex. Are you sure you're not simply able to store the .spacy files on Google Cloud Storage as an intermediate step? I'm not familiar with Google Vertex but transforming spaCy's binary format into something that Python can pickle sounds like a very inefficient middle step.

[dave-espinosa]

Hello @koaning,

Well, what you've just suggested, is my "experiment for today": theoretically, you can store a file at some "bucket + subdirectory" in GCS, and make each component in-between the pipeline, to output an OuputPath and input an InputPath; therefore, it should be feasible to pass around only the ".train" file storage destination. Now, the "conversion into pickle" is some sort of "Plan B", and takes inspiration in the subsection "Pass data to the next component that is not a dataset or model", in this article.

That's why I want to know if such conversion ".spacy -> pickle" can be done, and how.

Thank you.

[koaning]

A pickle is just a "Python way" to store Python objects on disk. They are a general method, one that's popular in the scikit-learn ecosystem, but they also come with security concerns. It's also the case that not everything in Python can be pickled. Lambda functions are a typical example of something that .pickle files cannot handle.

The .spacy files should be seen as the "spaCy way" of storing data. This format is lightweight, customised for the spaCy-usecase and has many benefits over a pickle file. I've never tried to pickle the DocBin object that the .spacy files represent, so I cannot comment on if it can be done. But if you can omit the pickle format, I highly recommend it.

[dave-espinosa]

Hello everyone,

I have found a workaround for my use case, and as forecasted in the discussion thread (by both @koaning and me), there is no need to pickle a .spacy file, in order to use it as input / output of a component, inside a Vertex AI Pipeline. If the project you are currently using, is the same to access both Vertex AI and Cloud Storage (as in my case), you can access ANY GCS Bucket from Vertex AI, by adding gcs/ first in your search path, as explained here.

I will include a very simple example about how to do it, borrowing the code from this post in Stack Overflow:

import spacy
from spacy.training import Example
from spacy.tokens import DocBin

td = [["Who is Shaka Khan?", {"entities": [(7, 17, "FRIENDS")]}],["I like London.", {"entities": [(7, 13, "LOC")]}],]

nlp = spacy.blank("en")
db = DocBin()

for text, annotations in td:
    example = Example.from_dict(nlp.make_doc(text), annotations)
    db.add(example.reference)

db.to_disk("gcs/your_bucket_name/your_path/td.spacy")  # <== THIS DOES THE TRICK

UPDATE: It is important to notice that, even when the previous implementation would allow you to store a .spacy file in some GCS destination, hard-coding paths is not recommended. Instead, try to use InputPath and OutputPath to let Kubeflow (through Vertex AI) decide the paths. I suggest to check this example, to see my suggestion at work.

Thank you.