BlenderBot2 - Slow response

[linlinsam]

Hello, I am testing BlenderBot2 and the response is a quite slow.

I tried this: parlai interactive -mf zoo:blenderbot2/blenderbot2_400M/model --search-server relevant_search_server

And it took over 2 minutes to get an answer to my "Hello"
Enter Your Message: hello
[BlenderBot2Fid]: Hello, how are you today? I hope you have a great day! BINGO

My system information is the following:

[superchargez]

I encountered same problem on google colab with Tesla T4 (16 GB VRAM, out of which less than 11GB was used) and RAM that was under use was about 3GB.

[klshuster]

the model is optimized for GPU usage; responses in CPU-only modes may be quite slow (these are big models, after all)

[linlinsam]

Thank you for the answer. Then, how should I enable GPU when running ParlAI interactive command? Something like this?
parlai interactive -mf zoo:blenderbot2/blenderbot2_400M/model --search-server "https://www.google.com" -gpu -1

[klshuster]

if there is an available GPU, ParlAI will utilize it automatically. We only provide options for turning off gpu usage (--no-cuda)

[linlinsam]

Thank you for the help.

[stephenroller]

A common confounder here is when someone accidentally installs CPU-only pytorch

[madgrizzle]

I'm very new to all this and this was the very question I was looking for an answer on, but from a computational standpoint, what is blenderbot2 doing that blenderbot1 isn't? I can run blender_400Mdistilled with a few seconds response time on a GPU-less system but blenderbot2_400M takes 1-2 minutes?

[klshuster]

There are a couple things going on at the moment:

1. BB2 requires conditioning on (5) documents when generating a response, and thus requires extra computation there.
2. At the moment, BB2 does not have support for incremental decoding, which speeds up generation immensely (as it does not require recomputation of prior decoder states); this is something we should look into adding in the future

[stephenroller]

Oh yeah lack of incremental decoding is killing us then. I didn't realize.

[madgrizzle]

So is there hope BB2_400M (or smaller) might run on a GPU-less system? Long story short, I'm going through a midlife crisis and building a robot and can't accommodate a GPU in the design (size and power). I'm using BB1_400 at the moment as it runs ok without it.