The world's slowest chatbot

[JohnZolton]

I used whisper to detect questions in "real" time and then passed those questions to a question-answering model and return an answer based on a text file of information.

It technically worked but overall performance was pretty terrible, about 10 seconds between asking the question and outputting an answer.
Code here.

[skol101]

Is processing done in CPU?

[JohnZolton]

If you have a GPU it uses it, if not then CPU

[skol101]

Wouldn't that be a reason (CPU) why your bot is so slow to process spoken speech?

[JohnZolton]

Whisper processes audio files, not live input. So to get around this it transcribes 5 second chunks and sends that to the QA model. So there's a 0-5 second delay before the question gets detected and processed

[DonFlymoor]

If you could detect the end of a sentence it could be much faster to reply