Evaluation step for textcat using transformer doesn't use GPU

[samehraban]

How to reproduce the behaviour

I noticed training a textcat using transformer is taking forever at the end of the first iteration. Turns out this loop takes about 40~50 seconds per iteration on my machine while the training itself takes about ~2.5 it/sec.

Your Environment

• spaCy version: 3.2.1
• Platform: Linux-4.15.0-162-generic-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.12

[adrianeboyd]

The evaluation does use GPU if you're using GPU for training, but if you have a large dev corpus, it can take a noticeable amount of time. You might want to raise eval_frequency in your config or lower the size of your dev corpus (for a quick test, you can truncate it with limit in the [corpora.dev] block).

The batch sizes and other settings that manage the length of the training and eval steps can be set separately in the config, see: #8600 (comment)

[pmbaumgartner]

@samehraban - I was able to replicate what you saw. What's happening is that evaluation involves several steps that are entirely on CPU, e.g., reading in the dev docs (which are loaded from scratch every time to ensure a clean eval) and running the scorer. When you see the GPU use drop to 0, this is what's happening.

[samehraban]

@pmbaumgartner Actually I'm totally confused here. After about 2000 steps, GPU utilization drops to almost zero in training. If I set discard_oversize to false for the minibatch_by_padded_size it gets good but then after a while COOM occurs.

[pmbaumgartner]

In general, I think it's going to be difficult to replicate this without the same data you're using.

What are you using to profile/monitor your GPU and CPU?

[samehraban]

I use nvtop.

[pmbaumgartner]

If there's a script/repo/project that we can run to replicate the issue we can investigate a bit further, otherwise it's hard to know exactly what's going on in your second issue (OOM after 2000 steps).