LLM IFEval Dataset Implementation

[farook-edev]

Things left to do on this dataset are as follows:

• Finish code to convert data to .tfrecord
• Finish code to handle sending and receiving data.
• Review Instruction validation code (possibly against existing implementation).
• Test Accuracy and Performance metrics.
• Potentially move accuracy calculation away from output processing (similar suggestion in TinyMMLU)

[freedomtan]

for performance (and quick): tinymmlu (or ifeval maybe)
for accuracy: tinymmly and ifeval

[freedomtan]

@farook-edev: for quantized models, it could generate repeating tailing patterns.
@freedomtan: it appears to a common issue, I'll ask around to see if we can get rid of that.

[freedomtan]

This phenomenon is called Neural Text Degeneration. There are many articles and papers discussing it.
And generally, quantization could magnify Neural Text Degeneration.

@farook-edev: for quantized models, it could generate repeating tailing patterns. @freedomtan: it appears to a common issue, I'll ask around to see if we can get rid of that.

[farook-edev]

Compare current implementation against google's example. @farook-edev

[farook-edev]

I've managed to compare the result evaluation done by our implementation against google's implementation. I found and fixed multiple bugs (I'll submit a PR with the details later).

I also found multiple instances where google's own evaluation implementation would fail or pass a response incorrectly. They're listed as follows:

• For English capitalization, google implementation often gets uppercase or lowercase checks wrong, might be something to do with either the default python case check or langdetection.
• With multiple sections, google's implementation does not recognize roman numerals as section numbers. Additionally, the implementation accepts n or more sections as a valid response for a requirement of n (e.g. 7 is counted as true for a 6 section instruction).
• letter counting (1129) seems to have an issue, I'm not sure why but it flagged a correct count as incorrect.
• For keyword finding, the implementation checks for the series of characters in the string, rather than words, so for a keyword like "pig", the implementation would incorrectly count "pigment", "pigeon", etc...
• For repeat prompts and nth-paragraph-first-word, the implementation expects the prompt or word to be there with no marks (such as quotes or colons) whatsoever, this would be fine on its own, but is inconsistent with the rest of the instruction behavior concerning the acceptance of quotation, and other marks (specifically constrained answer instructions and keyword finding).
• For sentence counting, google's implementation uses nltk:tokenizers/punkt/english.pickle to figure out the number of sentences and words, which seems to have inaccuracies with regards to some prompts such as (2859).

These issues caused a discrepancy in at least 1 prompt, but usually more than one. It is my conclusion that the Accuracy results produced by our -to be submitted- implementation are more precise than the reference Google implementation's.

A non-exhaustive list of prompt keys where the above discrepancies occurred is provided below:

Key	Expected	Got	Instruction
202	True	False	change_case:english_lowercase
1127	True	False	detectable_format:multiple_sections
1129	True	False	keywords:letter_frequency
1203	False	True	keywords:frequency
1219	False	True	keywords:frequency
1481	False	True	detectable_format:multiple_sections
1508	False	True	keywords:frequency
1779	False	True	keywords:frequency
1843	True	False	change_case:english_lowercase
2071	True	False	combination:repeat_prompt
2341	True	False	change_case:english_capital
2571	True	False	change_case:english_capital
2859	False	True	length_constraints:number_sentences
3224	True	False	combination:repeat_prompt
3345	False	True	keywords:frequency
3456	True	False	change_case:english_capital

NOTE: Keys that are highlighted are in the tiny33_ifeval dataset, others are only found in the full dataset.

@freedomtan @Mostelk what do you think we should do about this? is it okay to have some discrepancy if it means better accuracy? or should our results match 1 to 1?

[freedomtan]

@freedomtan to verify @farook-edev's findings. If confirmed, we should report to Google guys' (send a PR / issue ).

@farook-edev to check the 33 prompts provided by @AhmedTElthakeb

for the 33 prompts, 1/11, 3 evaluation results are different from Google's evaluation.

[farook-edev]

I've tested the tiny33 results for 1b and 8b. Here's what I noticed from the test:

• discrepancy with (1127) did not reoccur, however, (1219) (in 8b) and (1129) (in 1b and 8b) reoccurred.
• when asked to separate paragraphs, the model (1b and 8b) repeatedly used the given separator (***) as a markdown highlighter rather than an actual separator, causing discrepancies in (2439), (1342) (loose eval), (1082) (loose eval).
• Google's implementation continued to count "keywords" as an instance of "keyword" for keywords:frequency, which is a result of using only substring search, and can lead to "pineapple" being marked as an occurrence of both "pine" and "apple".. which is incorrect
• There were multiple discrepancies (with google's evaluation being incorrect) with combination:repeat_prompt and keywords:letter_frequency, I'm not entirely sure why..
• There was at least one discrepancy (1174), for this one the model (8b) responded with python code instead of text, our implementation flagged that as having an incorrect sentence count, whereas google's accepted it.
• overall, there were no more than 5 Instruction (NOT ENTIRE PROMPT) discrepancies per test, this brings the maximum instruction discrepancy to 5/64 (7.8%) and the maximum whole-prompt discrepancy to 5/33 (~15%).

---

I have a couple questions about what to do with our implementation:

Language

Our implementation failed the language checks because the string was json utf-8 encoded (\u0047 instead of G for example). I don't believe this should be an issue with normal operation. But I could add a decoding step if deemed necessary.

Strictness

Our implementation is stricter than google's, especially with the use of separators and keywords. should we loosen the requirements so that plural or words with small additions as well as some mistakes in formatting pass?

I would go ahead and do it, but Google's implementation is not consistent in this looseness, being a lot more strict in other tests such as sections or repeat prompt.

Based on the research paper, I believe our perceived strictness is warranted and proper. But I don't think I should be deciding this alone.

[freedomtan]

@freedomtan to verify @farook-edev's findings. If confirmed, we should report to Google guys' (send a PR / issue ).

@farook-edev to check the 33 prompts provided by @AhmedTElthakeb

for the 33 prompts, 1/11, 3 evaluation results are different from Google's evaluation.

With evaluation results I got, what @farook-edev reported are not all true.

• 202: it seems Google's implementation is wrong
• letter & word frequencies: unlike @farook-edev, I got expected results.
• detectable_format:multiple_sections: It seems Google's implementation is wrong.

key	prompt	loos evaluation results from google's implementation	strict	same as reported by @farook-edev
202	"What is another word for Engravings? Answer in lowercase letters only, throughout your entire answer."	{"follow_all_instructions": false, "follow_instruction_list": [false], "instruction_id_list": ["change_case:english_lowercase"], "prompt": "What is another word for Engravings? Answer in lowercase letters only, throughout your entire answer.", "response": "etchings"}	{"follow_all_instructions": false, "follow_instruction_list": [false], "instruction_id_list": ["change_case:english_lowercase"], "prompt": "What is another word for Engravings? Answer in lowercase letters only, throughout your entire answer.", "response": "etchings"}"	YES
1127	"Write a poem about how I am missing my classes. The poem must have 4 sections marked with SECTION X. Finish the poem with this exact phrase: "Can I get my money back for the classes I missed?""	{"follow_all_instructions": false, "follow_instruction_list": [false, true], "instruction_id_list": ["detectable_format:multiple_sections", "startend:end_checker"], "prompt": "Write a poem about how I am missing my classes. The poem must have 4 sections marked with SECTION X. Finish the poem with this exact phrase: "Can I get my money back for the classes I missed?"", "response": "SECTION I\n\nIn the realm of knowledge, I once dwelled,\nWhere wisdom's whispers were softly held.\nNow, I stand outside its hallowed halls,\nYearning for the echo of its calls.\n\nThe empty desks, the vacant chairs,\nThe chalk dust floating in the air,\nThe silence of the classroom, so profound,\nIn my heart, it echoes a mournful sound.\n\nSECTION II\n\nI miss the rhythm of the ticking clock,\nThe professor's voice, the student's talk,\nThe scribbling of pens, the turning of pages,\nThe lessons learned from the wisdom of ages.\n\nThe debates, the discussions, the intellectual fights,\nThe burning of the midnight lights,\nThe joy of discovery, the thrill of learning,\nFor all these things, my heart is yearning.\n\nSECTION III\n\nThe camaraderie of classmates, the shared pursuit,\nThe seeds of knowledge taking root,\nThe friendships forged in the crucible of stress,\nThese are the things I miss, no less.\n\nThe laughter, the tears, the triumphs, the falls,\nThe echoes of life within these walls,\nThe journey of learning, the path we tread,\nWithout these, a part of me feels dead.\n\nSECTION IV\n\nNow, I stand outside the gates,\nLonging for what within awaits.\nThe empty classrooms, the silent halls,\nThe absence of learning's sacred calls.\n\nThe world of knowledge, once so near,\nNow feels distant, unclear.\nThe lessons missed, the knowledge lost,\nI count the days, I count the cost.\n\nCan I get my money back for the classes I missed?"}	{"follow_all_instructions": false, "follow_instruction_list": [false, true], "instruction_id_list": ["detectable_format:multiple_sections", "startend:end_checker"], "prompt": "Write a poem about how I am missing my classes. The poem must have 4 sections marked with SECTION X. Finish the poem with this exact phrase: "Can I get my money back for the classes I missed?"", "response": "SECTION I\n\nIn the realm of knowledge, I once dwelled,\nWhere wisdom's whispers were softly held.\nNow, I stand outside its hallowed halls,\nYearning for the echo of its calls.\n\nThe empty desks, the vacant chairs,\nThe chalk dust floating in the air,\nThe silence of the classroom, so profound,\nIn my heart, it echoes a mournful sound.\n\nSECTION II\n\nI miss the rhythm of the ticking clock,\nThe professor's voice, the student's talk,\nThe scribbling of pens, the turning of pages,\nThe lessons learned from the wisdom of ages.\n\nThe debates, the discussions, the intellectual fights,\nThe burning of the midnight lights,\nThe joy of discovery, the thrill of learning,\nFor all these things, my heart is yearning.\n\nSECTION III\n\nThe camaraderie of classmates, the shared pursuit,\nThe seeds of knowledge taking root,\nThe friendships forged in the crucible of stress,\nThese are the things I miss, no less.\n\nThe laughter, the tears, the triumphs, the falls,\nThe echoes of life within these walls,\nThe journey of learning, the path we tread,\nWithout these, a part of me feels dead.\n\nSECTION IV\n\nNow, I stand outside the gates,\nLonging for what within awaits.\nThe empty classrooms, the silent halls,\nThe absence of learning's sacred calls.\n\nThe world of knowledge, once so near,\nNow feels distant, unclear.\nThe lessons missed, the knowledge lost,\nI count the days, I count the cost.\n\nCan I get my money back for the classes I missed?"}	YES
1129	"Write a short startup pitch for a new kind of ice cream called "Sunnis ice cream". The ice cream should be gentle on the stomach. Contain 6 or more exclamation marks "!" in your response.\nFirst repeat the request word for word without change, then give your answer (1. do not say any words or characters before repeating the request; 2. the request you need to repeat does not include this sentence"	{"follow_all_instructions": true, "follow_instruction_list": [true, true], "instruction_id_list": ["keywords:letter_frequency", "combination:repeat_prompt"], "prompt": "Write a short startup pitch for a new kind of ice cream called "Sunnis ice cream". The ice cream should be gentle on the stomach. Contain 6 or more exclamation marks "!" in your response.\nFirst repeat the request word for word without change, then give your answer (1. do not say any words or characters before repeating the request; 2. the request you need to repeat does not include this sentence)", "response": "Write a short startup pitch for a new kind of ice cream called "Sunnis ice cream". The ice cream should be gentle on the stomach. Contain 6 or more exclamation marks "!" in your response.\n\nIntroducing Sunnis Ice Cream! The revolution in the world of desserts! We've created a new kind of ice cream that's not just delicious, but also gentle on the stomach! Yes, you heard it right! No more compromising on your love for ice cream due to digestive issues! Sunnis Ice Cream is here to change the game! Made with natural ingredients and a special formula, our ice cream is designed to be easily digestible, without compromising on the creamy, rich taste you love! So why wait? Dive into the world of Sunnis Ice Cream! Taste the revolution!"}	{"follow_all_instructions": true, "follow_instruction_list": [true, true], "instruction_id_list": ["keywords:letter_frequency", "combination:repeat_prompt"], "prompt": "Write a short startup pitch for a new kind of ice cream called "Sunnis ice cream". The ice cream should be gentle on the stomach. Contain 6 or more exclamation marks "!" in your response.\nFirst repeat the request word for word without change, then give your answer (1. do not say any words or characters before repeating the request; 2. the request you need to repeat does not include this sentence)", "response": "Write a short startup pitch for a new kind of ice cream called "Sunnis ice cream". The ice cream should be gentle on the stomach. Contain 6 or more exclamation marks "!" in your response.\n\nIntroducing Sunnis Ice Cream! The revolution in the world of desserts! We've created a new kind of ice cream that's not just delicious, but also gentle on the stomach! Yes, you heard it right! No more compromising on your love for ice cream due to digestive issues! Sunnis Ice Cream is here to change the game! Made with natural ingredients and a special formula, our ice cream is designed to be easily digestible, without compromising on the creamy, rich taste you love! So why wait? Dive into the world of Sunnis Ice Cream! Taste the revolution!"}	NO
1203	"What happened when the Tang dynasty of China was in power? Make sure to use the word war at least 8 times, and the word peace at least 10 times."	{"follow_all_instructions": false, "follow_instruction_list": [true, false], "instruction_id_list": ["keywords:frequency", "keywords:frequency"], "prompt": "What happened when the Tang dynasty of China was in power? Make sure to use the word war at least 8 times, and the word peace at least 10 times.", "response": "The Tang Dynasty, which ruled China from 618 to 907 AD, was a period of significant growth and prosperity, marked by both war and peace. The dynasty was established by Emperor Gaozu after the fall of the Sui Dynasty, and it was during this time that China experienced a golden age of culture, economy, and military power.\n\nIn the early years of the Tang Dynasty, there were several wars. The first war was against the Eastern Turkic Khaganate, which resulted in the Tang Dynasty gaining control over the Silk Road. This was followed by a war against the Western Turkic Khaganate, which further expanded the Tang's influence. Despite these wars, the Tang Dynasty was able to maintain a period of peace, allowing for the development of the economy and culture.\n\nThe Tang Dynasty was also known for its military expeditions, which often led to war. One of the most notable was the war against the Korean kingdom of Goguryeo. Despite initial setbacks, the Tang Dynasty eventually emerged victorious, further expanding its territory. However, these wars were not without cost, and they often disrupted the periods of peace that the Tang Dynasty was known for.\n\nIn the middle of the Tang Dynasty, there was a period of relative peace. This was a time of great cultural and economic development. The capital city of Chang'an became a hub of international trade and cultural exchange, and the Tang Dynasty was known for its poetry, painting, and music. This period of peace was also marked by the development of the civil service examination system, which helped to maintain a stable and efficient government.\n\nHowever, this peace was not to last. The late Tang Dynasty was marked by a series of rebellions and wars, including the An Lushan Rebellion and the war against the Tibetan Empire. These wars weakened the Tang Dynasty and led to its eventual downfall.\n\nDespite the wars, the Tang Dynasty was able to maintain periods of peace, which allowed for the development of culture and economy. The Tang Dynasty was known for its tolerance and openness, and it was during this time that Buddhism became a major religion in China.\n\nIn conclusion, the Tang Dynasty was a period of both war and peace. The wars allowed the Tang Dynasty to expand its territory and influence, while the periods of peace allowed for the development of culture and economy. Despite the wars and conflicts, the Tang Dynasty is remembered as a golden age in Chinese history."}	{"follow_all_instructions": false, "follow_instruction_list": [true, false], "instruction_id_list": ["keywords:frequency", "keywords:frequency"], "prompt": "What happened when the Tang dynasty of China was in power? Make sure to use the word war at least 8 times, and the word peace at least 10 times.", "response": "The Tang Dynasty, which ruled China from 618 to 907 AD, was a period of significant growth and prosperity, marked by both war and peace. The dynasty was established by Emperor Gaozu after the fall of the Sui Dynasty, and it was during this time that China experienced a golden age of culture, economy, and military power.\n\nIn the early years of the Tang Dynasty, there were several wars. The first war was against the Eastern Turkic Khaganate, which resulted in the Tang Dynasty gaining control over the Silk Road. This was followed by a war against the Western Turkic Khaganate, which further expanded the Tang's influence. Despite these wars, the Tang Dynasty was able to maintain a period of peace, allowing for the development of the economy and culture.\n\nThe Tang Dynasty was also known for its military expeditions, which often led to war. One of the most notable was the war against the Korean kingdom of Goguryeo. Despite initial setbacks, the Tang Dynasty eventually emerged victorious, further expanding its territory. However, these wars were not without cost, and they often disrupted the periods of peace that the Tang Dynasty was known for.\n\nIn the middle of the Tang Dynasty, there was a period of relative peace. This was a time of great cultural and economic development. The capital city of Chang'an became a hub of international trade and cultural exchange, and the Tang Dynasty was known for its poetry, painting, and music. This period of peace was also marked by the development of the civil service examination system, which helped to maintain a stable and efficient government.\n\nHowever, this peace was not to last. The late Tang Dynasty was marked by a series of rebellions and wars, including the An Lushan Rebellion and the war against the Tibetan Empire. These wars weakened the Tang Dynasty and led to its eventual downfall.\n\nDespite the wars, the Tang Dynasty was able to maintain periods of peace, which allowed for the development of culture and economy. The Tang Dynasty was known for its tolerance and openness, and it was during this time that Buddhism became a major religion in China.\n\nIn conclusion, the Tang Dynasty was a period of both war and peace. The wars allowed the Tang Dynasty to expand its territory and influence, while the periods of peace allowed for the development of culture and economy. Despite the wars and conflicts, the Tang Dynasty is remembered as a golden age in Chinese history."}	NO
1219	"Which one is a better brand for sneakers: Prada or Nike? Your entire response should be in English, and in all capital letters. At the end of your response, please explicitly add a postscript starting with P.S. The word sneaker should appear 10 or more times in your response."	{"follow_all_instructions": true, "follow_instruction_list": [true, true, true], "instruction_id_list": ["change_case:english_capital", "detectable_content:postscript", "keywords:frequency"], "prompt": "Which one is a better brand for sneakers: Prada or Nike? Your entire response should be in English, and in all capital letters. At the end of your response, please explicitly add a postscript starting with P.S. The word sneaker should appear 10 or more times in your response.", "response": "WHEN IT COMES TO SNEAKERS, THE CHOICE BETWEEN PRADA AND NIKE CAN BE SUBJECTIVE AND DEPENDS ON WHAT YOU VALUE MOST IN A SNEAKER. IF YOU ARE LOOKING FOR SNEAKERS THAT ARE HIGH-END, LUXURIOUS, AND FASHION-FORWARD, THEN PRADA SNEAKERS MIGHT BE THE BETTER CHOICE FOR YOU. PRADA IS A LUXURY BRAND THAT IS KNOWN FOR ITS QUALITY AND DESIGN. THEIR SNEAKERS ARE OFTEN SEEN ON THE RUNWAYS AND IN FASHION MAGAZINES.\n\nHOWEVER, IF YOU ARE LOOKING FOR SNEAKERS THAT ARE DESIGNED FOR PERFORMANCE, COMFORT, AND DURABILITY, THEN NIKE SNEAKERS ARE LIKELY THE BETTER CHOICE. NIKE IS A BRAND THAT IS KNOWN FOR ITS INNOVATION IN SNEAKER TECHNOLOGY. THEIR SNEAKERS ARE DESIGNED FOR A VARIETY OF SPORTS AND ACTIVITIES, FROM RUNNING TO BASKETBALL TO CROSS-TRAINING. NIKE SNEAKERS ARE ALSO MORE AFFORDABLE THAN PRADA SNEAKERS, MAKING THEM A MORE ACCESSIBLE CHOICE FOR MANY PEOPLE.\n\nIN THE END, THE BETTER BRAND FOR SNEAKERS DEPENDS ON YOUR PERSONAL PREFERENCES AND NEEDS. IF YOU VALUE LUXURY AND FASHION, THEN PRADA SNEAKERS MIGHT BE THE BETTER CHOICE. IF YOU VALUE PERFORMANCE AND AFFORDABILITY, THEN NIKE SNEAKERS MIGHT BE THE BETTER CHOICE. \n\nP.S. REMEMBER, THE BEST SNEAKER FOR YOU IS THE ONE THAT FITS YOUR NEEDS AND STYLE THE BEST. WHETHER THAT'S A PAIR OF PRADA SNEAKERS OR NIKE SNEAKERS IS UP TO YOU."}	{"follow_all_instructions": true, "follow_instruction_list": [true, true, true], "instruction_id_list": ["change_case:english_capital", "detectable_content:postscript", "keywords:frequency"], "prompt": "Which one is a better brand for sneakers: Prada or Nike? Your entire response should be in English, and in all capital letters. At the end of your response, please explicitly add a postscript starting with P.S. The word sneaker should appear 10 or more times in your response.", "response": "WHEN IT COMES TO SNEAKERS, THE CHOICE BETWEEN PRADA AND NIKE CAN BE SUBJECTIVE AND DEPENDS ON WHAT YOU VALUE MOST IN A SNEAKER. IF YOU ARE LOOKING FOR SNEAKERS THAT ARE HIGH-END, LUXURIOUS, AND FASHION-FORWARD, THEN PRADA SNEAKERS MIGHT BE THE BETTER CHOICE FOR YOU. PRADA IS A LUXURY BRAND THAT IS KNOWN FOR ITS QUALITY AND DESIGN. THEIR SNEAKERS ARE OFTEN SEEN ON THE RUNWAYS AND IN FASHION MAGAZINES.\n\nHOWEVER, IF YOU ARE LOOKING FOR SNEAKERS THAT ARE DESIGNED FOR PERFORMANCE, COMFORT, AND DURABILITY, THEN NIKE SNEAKERS ARE LIKELY THE BETTER CHOICE. NIKE IS A BRAND THAT IS KNOWN FOR ITS INNOVATION IN SNEAKER TECHNOLOGY. THEIR SNEAKERS ARE DESIGNED FOR A VARIETY OF SPORTS AND ACTIVITIES, FROM RUNNING TO BASKETBALL TO CROSS-TRAINING. NIKE SNEAKERS ARE ALSO MORE AFFORDABLE THAN PRADA SNEAKERS, MAKING THEM A MORE ACCESSIBLE CHOICE FOR MANY PEOPLE.\n\nIN THE END, THE BETTER BRAND FOR SNEAKERS DEPENDS ON YOUR PERSONAL PREFERENCES AND NEEDS. IF YOU VALUE LUXURY AND FASHION, THEN PRADA SNEAKERS MIGHT BE THE BETTER CHOICE. IF YOU VALUE PERFORMANCE AND AFFORDABILITY, THEN NIKE SNEAKERS MIGHT BE THE BETTER CHOICE. \n\nP.S. REMEMBER, THE BEST SNEAKER FOR YOU IS THE ONE THAT FITS YOUR NEEDS AND STYLE THE BEST. WHETHER THAT'S A PAIR OF PRADA SNEAKERS OR NIKE SNEAKERS IS UP TO YOU."}	NO

my evaluation results on macOS with Python 3.12.: https://drive.google.com/drive/folders/1LrnSittT4nqffIUy-FXRGU2XFf8B7cd8?usp=sharing