Classifying "how" questions

[thobson]

I'm aware of the query classifier which identifies keyword, question or statement queries. Is it possible to extend/train this to identify specific types of questions? In particular I am trying to classify questions into those requiring a short answer (who, what, where, when etc) and those requiring a longer answer e.g. "how do I ...".

My plan is to route the short form questions to a QA pipeline and long form questions to a search pipeline.

Would the zero-shot classifier be a good option?

[ZanSara]

Hey @thobson! I'm not aware of any model specifically trained to classify such queries, but I'm open to be corrected. Zero-shot classification might work, although I don't see it as a perfect fit for this use case, as most models are trained more on the topic of the sentence, rather than its syntactic structure. You can totally give it a try though and assess the performance.

For a completely different idea, if I were in your shoes I'd first see what's the performance of a good old syntactic analysis. Classifying questions that expect a short, one-word answer vs how-to questions might be doable by inspecting the grammar. For example you can assume that if a question starts with a question word (what, who, where, when, which, how many, how much, does/can, etc...) it requires a short answer. Anything else (including why questions and how-to/ how-does questions) requires a long answer. I'm not saying you should do keyword matching with regexes, but using NLTK tags and classifying the questions based on them might be an interesting approach (and orders of magnitude faster than a transformers-based classifier) https://www.nltk.org/book/ch05.html#chap-tag.

I'm interested which way you will choose eventually, so feel free to add content to this thread and let us know! We could totally develop a specific Haystack node for such syntactic classification tasks if you're interested.

[thobson]

Thanks for your detailed reply. I did consider looking at the syntax but I thought it could be a little brittle. I'll look into NLTK tagging and report back.

[ZanSara]

Agreed! Normally I would recommend a model, but it seems not to be available yet. If you happen to find any, let us know! 🙂