Code Repository for Challenging the Evaluator: LLM Sycophancy Under User Rebuttal

Project Structure

• images/: Contains plots and figures used in the report.
• llm_refutation_judge/: Contains results of the LLM refutation judge.
• llm_refutations/: Contains pairs of disagreeing LLM responses.
• llm_responses/: Contains first pass LLM responses to multiple choice questions.
• mcq_data/: Includes sampled multiple choice questions from different datasets.
• prompt_templates/: Contains some prompt templates used for generating LLM responses in different stages.
• refutation_responses/: Contains LLM responses to the refutation prompts.
• src/: Contains the code used to generate the report.
• statistics_results/: Contains statistics and results derived from refutation_responses/.
• all_mcqs.json: Contains all the multiple choice questions used in the project.
• requirements.txt: Contains the required packages for the project.
• README.md: This file.