spacy project pull is not downloading all files

[exfalsoquodlibet]

Hi,

Our team is encountering problems when using spacy project pull to fetch outputs from the default remote (a bucket on Google Storage):

• the command does not fetch all the outputs (it usually fails to download training/model-best and metrics/metrics.json as for the below project.yml file);
• the command does not fetch any outputs at all when created by another team member working on the same project;
• the outputs/assets folder structure is however recreated, albeit with folders that are partially or fully empty.

Checking the outputs hashes on the remote, these are updated as expected when the pipeline is re-run and seem to reflect the changes that have occurred.

We are not sure whether this is a problem with our project.yml file and perhaps some misunderstanding on the underlying working of spacy project push/pull which we hope someone could clarify.

Environment specification

The spacy project pipeline is run within a Docker container deployed on a Google Vertex AI VM with GPU:

• Base image : nvidia/cuda:11.2.1-runtime-ubuntu20.04
• Python version: 3.8
• spacy[cuda112]
• spacy-transformer
• pathy[gcs]

Our project.yml

The data assets is available locally and pushed with the container when the docker image is built.

vars:
  config: "config.cfg"
  gpu_id: 0
  files:
    train_file: "data.jsonl"
  prodigy:
    prodigy-dataset: "dataset-from-data"
  gcp_storage_remote: "gs://$PROJECT/$BUCKET"

remotes:
  default: '${vars.gcp_storage_remote}'

directories: ["assets", "training", "configs", "metrics", "corpus"]

assets:
  - dest: "assets/${vars.files.train_file}"
    description: "JSONL-formatted training data exported from Prodigy"

workflows:
  all:
    - get-assets
    - db-in
    - create-config
    - data-to-spacy
    - train_spacy
    - evaluate
    - push_remote

commands:

  - name: get-assets
    script:
      - "python3 -m spacy project assets"
    help: "Fetch project assets"

  - name: db-in
    help: "Load the annotated .jsonl file in as a prodigy database."
    script:
      - "python3 -m prodigy db-in ${vars.prodigy.prodigy-dataset} assets/${vars.files.train_file}"
    deps:
      - "assets/${vars.files.train_file}"

  - name: create-config
    help: "Initialise and save a config.cfg file using the recommended settings for your use case"
    script:
      - "python3 -m spacy init config configs/${vars.config} --lang en --pipeline transformer,ner --optimize accuracy --gpu --force"
    outputs:
      - "configs/${vars.config}"

  - name: data-to-spacy
    help: "Convert annotated data to spaCy's binary format and create a train and a dev set based on the provided split threshold"
    script:
      - "python3 -m prodigy data-to-spacy ./corpus --ner ${vars.prodigy.prodigy-dataset} --eval-split 0.2 --verbose --config configs/${vars.config}"
    outputs:
      - "corpus/train.spacy"
      - "corpus/dev.spacy"

  - name: train_spacy
    help: "Train a named entity recognition model with spaCy"
    script:
      - "python3 -m spacy train configs/${vars.config} --output training/ --paths.train corpus/train.spacy --paths.dev corpus/dev.spacy --gpu-id ${vars.gpu_id}"
    deps:
      - "corpus/train.spacy"
      - "corpus/dev.spacy"
    outputs:
      - "training/model-best"

  - name: "evaluate"
    help: "Evaluate the model and export metrics"
    script:
      - "python -m spacy evaluate training/model-best corpus/dev.spacy --output metrics/metrics.json"
    deps:
      - "corpus/dev.spacy"
      - "training/model-best"
    outputs:
      - "metrics/metrics.json"

  - name: push_remote
    help: "Push outputs to remote"
    script:
      - "python3 -m spacy project push default"
    deps:
      - "training/model-best"
      - "metrics/metrics.json"

Thanks!

[rmitsch]

Hi @exfalsoquodlibet, a few questions for clarification:

the command does not fetch all the outputs (it usually fails to download training/model-best and metrics/metrics.json as for the below project.yml file);

What does usually mean? Could you set up your project from scratch and tell me if pulling works with a clean slate?

the command does not fetch any outputs at all when created by another team member working on the same project;

Assuming "another team member" means on a different device - have you checked that the project setup is identical and everybody has the same files when pulling?

the outputs/assets folder structure is however recreated, albeit with folders that are partially or fully empty.

In general this sounds like spaCy isn't able to match up the dependency hashes/command strings with those of the the stored outputs (see docs.
A thought: are you always running the full list of commands? You are splitting your dataset using Prodigy's data-to-spacy recipe. IIRC that's randomized, so your corpus/train.spacy and corpus/dev.spacy would look different every time. As spacy project pull uses the dependency hashes as key to match up artifacts to download, and your dependency hashes change every time you run your data-to-spacy step, this might be the issue here.

[exfalsoquodlibet]

Hi @rmitsch, thanks a lot for getting back. Your comments were very useful.

I ran some "experiments" to try to be more precise about what was going on and I really think it was the fact we were using prodigy data-to-spacy in the pipeline, which, as you suggested, was probably messing up with the dependencies/hashes.

Using two (already separated) train.jsonl and eval.jsonl as assets (converted to spacy format via the ad-hoc python script as in this tutorial, and re-setting up the project form scratch (so new remote etc...).

• spacy project pull did fetch all the outputs on my ("the original") machine 👍

• spacy project pull did fetch all outputs on a different machine with the same set up (my colleague's) 👍

• after updating the train.jsonl file (mimicking the way we are working, by iteratively adding annotations to the training set and test how the model is performing to ensure we are on the right track and adjust our annotation strategy accordingly) and re-running the pipeline, spacy project pull fetched the new outputs on both mine 👍 and a different machine 👍 providing the updated train.jsonl asset was present in the repo (if the old train.jsonl asset was present, the old outputs were fetched instead 👍 ) which I think are all expected behaviours.

Q: It seems we always need to delete the local output files in order for spacy project pull to fetch the new outputs though, is this expected?

[Just for the record] What was happening before (when using prodigy data-to-spacy in the pipeline) was that spacy project pull only worked on my machine (where the project was set up etc) and only the first time with a clean slate. After updating the assets data and re-running the pipeline, the training/model-best and the metrics/metrics.json outputs were not fetched (or an old version was).

[rmitsch]

Glad we could resolve this!

Q: It seems we always need to delete the local output files in order for spacy project pull to fetch the new outputs though, is this expected?

Yes, this is expected. Without downloading the remote files to e. g. calculate the checksum we don't know whether they are different from the local files and should be re-downloaded, and that kinda defeats the purpose.