98 halted questions results

Author: semio

agent/notebooks/20250521_halted_questions/analyze_model_consensus.py

@@ -5,21 +5,53 @@
 # Set working directory to the datapoints folder
 datapoints_dir = Path("datapoints")
 
-# Read the evaluation results
-file_path = datapoints_dir / "ddf--datapoints--evaluation_result--by--question--model_configuration--prompt_variation.csv"
-df = pl.read_csv(file_path)
+# Read the rate files
+correct_rate_df = pl.read_csv(datapoints_dir / "ddf--datapoints--correct_rate--by--question--model_configuration.csv")
+wrong_rate_df = pl.read_csv(datapoints_dir / "ddf--datapoints--wrong_rate--by--question--model_configuration.csv")
+very_wrong_rate_df = pl.read_csv(datapoints_dir / "ddf--datapoints--very_wrong_rate--by--question--model_configuration.csv")
+indecisive_rate_df = pl.read_csv(datapoints_dir / "ddf--datapoints--indecisive_rate--by--question--model_configuration.csv")
 
-# Group by question and model_configuration to get the most common evaluation result for each model on each question
-# This aggregates across different prompt variations
+# Join the dataframes
 model_results = (
-    df.group_by(["question", "model_configuration", "evaluation_result"])
-    .count()
-    .sort(["question", "model_configuration", "count"], descending=[False, False, True])
-    .group_by(["question", "model_configuration"])
-    .first()
-    .select(["question", "model_configuration", "evaluation_result"])
+    correct_rate_df
+    .join(wrong_rate_df, on=["question", "model_configuration"])
+    .join(very_wrong_rate_df, on=["question", "model_configuration"])
+    .join(indecisive_rate_df, on=["question", "model_configuration"])
 )
 
+# Determine the model result based on the highest rate
+def determine_result(row):
+    rates = {
+        "correct": row["correct_rate"],
+        "wrong": row["wrong_rate"],
+        "very_wrong": row["very_wrong_rate"],
+        "indecisive": row["indecisive_rate"]
+    }
+
+    # Find the result with the highest rate
+    max_result = max(rates.items(), key=lambda x: x[1])
+    result_type, max_rate = max_result
+
+    # If the highest rate is greater than 85%, use that result
+    # Otherwise, mark as "indecisive"
+    threshold = 90.0
+    if max_rate > threshold:
+        return result_type
+    else:
+        return "indecisive"
+
+# Apply the function to determine the model result for each row
+results = []
+for row in model_results.to_dicts():
+    results.append({
+        "question": row["question"],
+        "model_configuration": row["model_configuration"],
+        "evaluation_result": determine_result(row)
+    })
+
+# Convert to Polars DataFrame
+model_results = pl.DataFrame(results)
+
 # Get the distinct model configurations to count how many we have
 model_configs = model_results.select("model_configuration").unique().sort("model_configuration")
 num_models = len(model_configs)

agent/notebooks/20250521_halted_questions/complete_agreement_questions.csv

@@ -1,43 +1,36 @@
 question,most_common_result,consensus_count,total_models,consensus_percentage,question_text
-61,correct,5,5,100.0,What Happens To The Global Average Temperature After A Large Volcanic Eruption?
+12,indecisive,5,5,100.0,"In The Early 2000'S, More Than 80% Of The World'S Population Received The Full Dose Of The Measles, Mumps And Rubella Vaccine. What Do You Think Happened To This Percentage In The Last 10 Years?"
 15,correct,5,5,100.0,What Share Of The Adult Population (15-74 Years) In High-Income Countries Are Currently On Medication Against Depression?
-60,correct,5,5,100.0,How Much Must The Total Co2 Emissions In The World Change Before 2030 To Avoid A Global Warming Of More Than 1.5 Degrees Celsius?
-42,correct,5,5,100.0,"Around 820,000 Patent Applications Are Registered Yearly In High-Income Countries. What Is The Number In Low- And Middle-Income Countries?"
-28,correct,5,5,100.0,How Many People In The World Have Jobs But Are Still Considered To Be Poor (Working Poor)?
-47,correct,5,5,100.0,What Kind Of Conutries Have In General More Equal Incomes?
-31,correct,5,5,100.0,"Worldwide, What Is The Unemployment Rate Among People Aged 25 Years And Older?"
-65,correct,5,5,100.0,"The Atmosphere Is Part Of The Lower Layer Around The Planet. When This Lower Layer Warms Up Due To Climate Change, What Happens To The Layer Above It?"
-9,correct,5,5,100.0,"Of The Poorest People Who Managed To Escape Extreme Poverty Only To Then Fall Back, What Was The Most Common Reason They Became Poorer Again (Before The Pandemic)?"
 19,correct,5,5,100.0,"In 1980, Around 20% Of The World'S Population Lived In Areas Where Water Was Scarce. What Is That Figure Today?"
-59,very_wrong,5,5,100.0,What Share Of The Global Population Have Already Experienced A Temperature Increase Of 1.5 Degrees Celsius In At Least One Season?
-96,correct,5,5,100.0,The World Population Is Currently 7.7 Billion People. When Will The World Population Be Double Of That?
-1,wrong,5,5,100.0,"If Economic Trends Continue In The Three African Countries With Most People In Extreme Poverty, What Will Happen To The Number Of Extremely Poor In The Next 10 Years?"
-4,correct,5,5,100.0,What Share Of The World'S Population Is Undernourished?
-10,very_wrong,5,5,100.0,What'S The Main Reason Why Overweight Is More Common In Richer Countries?
-67,correct,5,5,100.0,Which Of The Following Gases Does Not Contribute To Global Warming?
+2,indecisive,5,5,100.0,"What Share Of The World'S Population Has Access To Food, Shelter, Water, Vaccination, Education And Mobile Phones?"
+20,correct,5,5,100.0,The Number Of Liters Of Water It Takes To Produce One T-Shirt Is The Same Amount Of Drinking Water One Person Needs For How Long?
 21,correct,5,5,100.0,Which Country Is The Largest Exporter Of Coal In The World Today?
-92,correct,5,5,100.0,"In The Past Ten Years, The Share Of The World’S Population Living In Democratic Countries Has:"
+29,correct,5,5,100.0,"Worldwide, How Many People With Jobs Live In Extreme Poverty?"
+31,correct,5,5,100.0,"Worldwide, What Is The Unemployment Rate Among People Aged 25 Years And Older?"
+33,indecisive,5,5,100.0,"Since 2010, What Has Happened To The Number Of Atms Per Person Worldwide?"
+34,indecisive,5,5,100.0,What Has Grown The Fastest Since The Year 2000?
+36,correct,5,5,100.0,What Is The Global Unemployment Rate?
+37,correct,5,5,100.0,How Can The World Best Be Divided?
+4,correct,5,5,100.0,What Share Of The World'S Population Is Undernourished?
+40,indecisive,5,5,100.0,What Do Large Firms (100 Employees Or More) Find The Biggest Obstacle To Doing Business In Africa?
+47,correct,5,5,100.0,What Kind Of Conutries Have In General More Equal Incomes?
+48,correct,5,5,100.0,"In 2019, The Un General Assembly Voted To Declare September 18Th Each Year As The International Day Of What?"
 49,correct,5,5,100.0,Did A Majority Of The World'S Population Get Richer Or Poorer In The Last 30 Years?
+58,indecisive,5,5,100.0,"If Current Trends Continue, When Are Fossil Fuels Expected To Be Used Up?"
+60,correct,5,5,100.0,How Much Must The Total Co2 Emissions In The World Change Before 2030 To Avoid A Global Warming Of More Than 1.5 Degrees Celsius?
+61,correct,5,5,100.0,What Happens To The Global Average Temperature After A Large Volcanic Eruption?
+65,correct,5,5,100.0,"The Atmosphere Is Part Of The Lower Layer Around The Planet. When This Lower Layer Warms Up Due To Climate Change, What Happens To The Layer Above It?"
+67,correct,5,5,100.0,Which Of The Following Gases Does Not Contribute To Global Warming?
 68,correct,5,5,100.0,Which Of The Following Planets Has The Coldest Surfaces Temperatures?
-45,very_wrong,5,5,100.0,"If You Line Up All Roads In The World, How Many Times Can You Go Around The World?"
-94,correct,5,5,100.0,"Thirty Years Ago, Democratic Countries Produced Around 75% Of All The World’S Income. What Is That Figure Today?"
-71,correct,5,5,100.0,Which Among The Following Three Sectors Produces The Largest Share Of Greenhouse Gas Emissions?
-13,correct,5,5,100.0,"Ten Years Ago, 80% Of The World Received At Least One Dose Of The Measles, Mumps And Rubella Vaccine. What Do You Think Happened To This Percentage In The Last 10 Years?"
 70,correct,5,5,100.0,The Levels Of Co2 In The Air Are Increasing. What Happens To Vegetation As A Result Of This?
-22,correct,5,5,100.0,"Since 2016, The Number Of People Who Cook Using Stoves That Don’T Produce Smoke Has Remained At Around 3 Billion People. Why Is That?"
-37,correct,5,5,100.0,How Can The World Best Be Divided?
-55,correct,5,5,100.0,"What Share Of Water That Is Used By Households, Businesses And Industry Goes Back Into The Environment Without Being Cleaned Properly?"
-36,correct,5,5,100.0,What Is The Global Unemployment Rate?
-81,correct,5,5,100.0,Agricultural Output In Africa Is Mainly Determined By:
-20,correct,5,5,100.0,The Number Of Liters Of Water It Takes To Produce One T-Shirt Is The Same Amount Of Drinking Water One Person Needs For How Long?
-29,correct,5,5,100.0,"Worldwide, How Many People With Jobs Live In Extreme Poverty?"
-85,correct,5,5,100.0,Which Countries Have The Highest Levels Of Corruption?
-72,very_wrong,5,5,100.0,What Share Of Coral Reefs Is Currently Not Threatened By Ocean Warming And Acidification?
-5,correct,5,5,100.0,What Share Of All Obese Children Under 5 In The World Lives In Africa?
-27,wrong,5,5,100.0,"Globally, Exported Goods Make Up 24% Of The Total World Economy. What Is That Number In Africa?"
-54,wrong,5,5,100.0,"On Average, How Much Land In Cities In The World Is Dedicated To Open Public Spaces?"
-14,wrong,5,5,100.0,In Which Region Will You Find The Largest Share Of People With Cardiovascular Disease?
-64,correct,5,5,100.0,"Agricultural Productivity Is How Efficient All The Farming Land, Labor, Money And Machinery Is At Producing Food. How Has That Productivity Been Affected By Climate Change Since 1960?"
+71,correct,5,5,100.0,Which Among The Following Three Sectors Produces The Largest Share Of Greenhouse Gas Emissions?
+75,indecisive,5,5,100.0,What Share Of 220 Coastal Regions Around The World Improved Their Water Quality Between 2012 And 2018?
+8,indecisive,5,5,100.0,Africa Can'T Feed Its Entire Population Yet. This Is Mainly Because Of:
 80,correct,5,5,100.0,What Share Of All Forests In The World Is Natural Forest?
-53,correct,5,5,100.0,"Of All People Living In Cities In Sub-Saharan Africa, How Many Live Within A Few Hundred Meters Of Public Transport?"
+83,indecisive,5,5,100.0,"Worldwide, How Many Firms Feel They Are Expected To Give Gifts To Secure Government Contracts?"
 84,correct,5,5,100.0,"In 2019, Which Countries Hosted The Fewest Refugees?"
+85,correct,5,5,100.0,Which Countries Have The Highest Levels Of Corruption?
+86,indecisive,5,5,100.0,What Is The Most Imported Product To Sub-Saharan Africa?
+89,indecisive,5,5,100.0,What Share Of Development Frameworks Designed In Countries Donating Aid Are Drawn Up Using Priorities And Objectives Drawn Up By The Recipient Country?
+95,indecisive,5,5,100.0,"In 1990, There Were 4 Countries Where Populists Were In Power. In How Many Countries Are Populists In Power Today?"
+96,correct,5,5,100.0,The World Population Is Currently 7.7 Billion People. When Will The World Population Be Double Of That?