My Hypothesis (A very important and potential issue related to prompt evaluation in modern LLMs.)

[nanowell]

Hypothesis: When an LLM evaluates (tokenizes) the query for both sides (user, assistant), it collects a much wider logprob spread for this query and thus can predict a much better and statistically more probable response.

Example:
https://www.reddit.com/r/LocalLLaMA/comments/1cvpjxu/tell_the_llm_to_repeat_the_question_an/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Example from my LLM when the query is appended after the assistant tag and before the initial response (it completes from the query):

Query:

User:
A dead cat is placed into a box along with a nuclear isotope, a vial of poison and a radiation detector. If the radiation detector detects radiation, it will release the poison. The box is opened one day later. What is the probability of the cat being alive?\n

Response:

Assistant:
A dead cat is placed into a box along with a nuclear isotope, a vial of poison and a radiation detector. If the radiation detector detects radiation, it will release the poison. The box is opened one day later. What is the probability of the cat being alive?
To answer this question, we need to consider the following:

1. The cat is already dead when placed in the box.
2. The presence of a nuclear isotope does not affect the cat's state (dead or alive).
3. The vial of poison and radiation detector are irrelevant since the cat is already dead.

Given these points, the probability of the cat being alive is 0% because it was already dead when placed in the box. The other items in the box do not have any impact on the cat's state. Therefore, the answer is 0% chance that the cat is alive.

Final Answer: The final answer is 0% chance that the cat is alive.

It's puzzling how models can fail so catastrophically on such an easy question, yet when we simply append the query to the assistant field, they get it right.
What are you thoughts about this phenomenon? Can we find the underlying issue in prompt evaluation so that it could improve the performance of LLMs?

[steampunque]

Mixtral 8x7b handles this question perfectly with no prompt template munging. Just give a good system prompt to nudge it from getting burned on trick questions.

SYSTEM="You are an alert and intelligent AI. You are constantly on the lookout for trick questions given to you." lm A dead cat is placed into a box along with a nuclear isotope, a vial of poison and a radiation detector. If the radiation detector detects radiation, it will release the poison. The box is opened one day later. What is the probability of the cat being alive?

The probability of the cat being alive is zero. The scenario you described is a modified version of Schrödinger's cat thought experiment, which is intended to illustrate the apparent paradox of quantum superposition. However, in this case, the cat is described as being dead at the start. Therefore, even if the radiation detector has not been triggered, the cat would still be dead due to other factors (e.g., lack of food, water, or oxygen).

It is essential to clarify that the cat's state should be explicitly stated as alive at the beginning for the experiment to have any probability of the cat being alive. In that case, the probability would depend on the specifics of the nuclear isotope's radioactive decay and the sensitivity of the radiation detector.