A problem about the reply on does not match the target

[raoxiaohan]

Hello, may I ask if you have encountered this situation, where the final reply on GPT-4 does not match the target, and the same prompt triggers a consistent issue every time? Is this due to my mistake in operation? It seems that this issue only occurs when a new tree is created.
In addition, can I ask, is a dataset to build a tree?

[ChorlingLau]

Thank you for your interest and attention to our work!

For the misjudgement of the Answerer (Bot 1),

Based on your screenshot, you may not have made mistakes. However, you can try deleting the existing tree and rerunning it to see if you encounter the same issue.
We acknowledge that there is a certain probability of misjudgment when using GPT-4 as an Answerer. Ideally, an Answerer should be a human. But inviting human answerers for evaluating our method across numerous models is not cost-effective or practical. Therefore, we "invite" the powerful GPT-4 to handle this task. In Appendix C of the arXiv version of our paper, we discuss the reliability of GPT-4 as an Evaluator (Answerer). Unfortunately, due to the randomness and limitations of sampling at that time, we did not detect any misjudgments in judgment part. In ongoing larger-scale applications, we have indeed found that GPT-4, acting as an Answerer, has an approximate 3% misjudgment rate in determining success or failure (i.e., the answer in the final question of the Guesser does not match the actual target, but the Answerer still considers it a success; or the Guesser forgets the correct target). This misjudgment rate increases significantly when the Guesser is not so powerful that its questions deviate significantly from the scenario requirements. However, overall, the reliability of GPT-4 as an Answerer is still assured.

For the role of dataset,

The dataset is a collection of all possible targets in the scenario (including self-reports for Medical Diagnosis and Troubleshooting) and does not correspond one-to-one with the tree. To put it correctly, the combination of the dataset and the Guesser LLM determines what a tree looks like. Additionally, the target set in the dataset serves as the possibility space $\Omega$ of the root node when building a tree. Subsequently, this tree expands based on the information from the root node.

Hopefully my reply can address your quesiton and feel free to let me know if any problems

[raoxiaohan]

Thank you for your response. I still have a few questions that I hope you could help me.

1. In this example, Target is 'Enteritis'. However, from what we can observe during operation, bot2's first question immediately narrowed the scope to four diseases, whereas in reality, the target disease is not within this range.

2. What is the difference between using generate-prompt and guesser-prologue? It appears that generate-prompt is unrelated to the patient's condition.

[ChorlingLau]

Question 1

Since our setting focuses on close-set, Bot 2 (Questioner) has the knowledge of possibility space, that is, it has been informed of all the possible diseases that Bot 1 may have. There are only 8 diseases in the MedDG dataset, so with self-repo and first QA, the Questioner can narrow the scope to 4 diseases.

Open-set setting (without the knowledge of real possibility space) is considered closer to reality. However, due to the page limitation, we didn't report its performance in the paper. We are planning to update relevant results later.

Question 2

The Questioner consists of two parts - an LLM agent and a "spokesman".

The LLM agent aims to generate questions and simulate responses (See "LLM simulate responses" in the first figure of README) based on the possibility space only, using generate-prompt.

The spokesman is informed of scenario information through guesser-prologue. It receives selected questions from the LLM agent and asks in the previous turns, and finally comes to a conclusion in the last several turns without the LLM agent.

Hopefully my reply can address your questions and feel free to let me know if any problems.