BlenderBot2 vs RASA2

[Jawad1347]

Is blenderbot2 better than rasa2?
is it lighter (runs on small machine/without GPU)
is it faster (gives faster response)
is it trainable easily
is training self supervised?
does it contradict itself less? (blenderbot 1 does that a lot)

[stephenroller]

RASA and BlenderBot are not really comparable. RASA is more like tooling to build a task-oriented or customer-service type bot. BlenderBot1 and 2 are more like open-domain chit-chat bots that can talk with you about anything, but they won't help you accomplish anything. Additionally, BlenderBot is still research, and I wouldn't recommend it for any commercial applications.

BlenderBot2 does significantly improve on contradictions compared with BlenderBot1, but there's still room for improvement.

[Jawad1347]

So, blenderbot2 can't be used for commercial applications? I was thinking to provide it data (csv files a well as written documents like docs or pdfs) so it can give answers by reading them (and these files can be changed easily, perhaps in excel/any spreadsheet or document writer) and of course it could continue to talk by fetching information from internet.

First it would look for information from local data (database, csv files or documents) if required information is not found then go to net for that. Perhaps it is not possible with blenderbot2. Would this be possible with next generation blenderbot, though? If it is possible then I would learn ParlAI otherwise I don't think it is very useful framework.

[stephenroller]

Our mission is to advance the state of Artificial Intelligence and dialogue research through open source tooling and scientific papers that others can freely use and extend. Commercial uses are not prohibited by our LICENSE, but they are also not our primary objective.

We believe BlenderBot 1 and 2 aren't ready for commercial use due to their numerous limitations that include open-ended research problems. Please see our papers for descriptions of these limitations.

[superchargez]

Thank you Stephen, I also believe that models have potential to do better. Retrieving information from internet to augment knowledge base is really a great idea. I wonder if base model to be changed I mean why get stuck at blender, why not perhaps use GPT (2, or even Neo) while it functions the same way i.e. reinforces its knowledge with help of other models. Or is it just impossible?

[stephenroller]

We might get some improvements based on using a different base model, but the hallucination problem is difficult and unsolved. Retrieving using knowledge helps significantly, but not fully.

[jaseweston]

GPT2, and DialoGPT (GPT fine-tuned for dialogue), are quite a lot worse than even Blender 1, see e.g. these papers that compare them in automatic and human evaluations: https://arxiv.org/abs/2010.07079 https://aclanthology.org/2021.eacl-main.24/

…

On Sat, Aug 7, 2021 at 1:17 PM Stephen Roller ***@***.***> wrote: We might get some improvements based on using a different base model, but the hallucination problem is difficult and unsolved. Retrieving using knowledge helps significantly, but not fully. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3910 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACUOJ6CGBAA43HFZWKJ3UZLT3VTEJANCNFSM5BUCSP5A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[Jawad1347]

I would agree with you (and other human evaluators of GPT2 and models based on it) as I have tested it myself, however, I thought maybe my experience was based on smaller (or medium) model that is why it was bad. But thank you, I now know better. Anyway, what about GPT-Neo? I am sure it could/would be better, no? Sadly I could not test it (1.3B version) because colab crashes (out of memory -RAM), I have not tested the smaller version (256MB) but I am sure it would not be better than blenderbot 1.