📊 agriculture: Update datasets using FAOSTAT

[pabloarosado]

• Update all agriculture and animal welfare steps that rely on the old faostat steps.
• Archive unused steps.
• A few improvements and fixes in the process that archives steps.

[owidbot]

Quick links (staging server):

Site Dev	Site Preview	Admin	Wizard	Docs

Login: ssh owid@staging-site-data-faostat-update-datasets

chart-diff: ✅

• 24/24 reviewed charts
• Modified: 24/24
• New: 0/0
• Rejected: 0
• Data changes: 0
• Metadata changes: 0

data-diff: ✅ No differences found

+ Dataset garden/agriculture/2026-03-02/attainable_yields
+ + Table attainable_yields
+   + Column abaca__manila_hemp__raw_yield
+   + Column agave_fibres__raw__n_e_c__yield
+   + Column almond_yield
+   + Column apples_yield
+   + Column apricots_yield
+   + Column areca_nuts_yield
+   + Column artichokes_yield
+   + Column asparagus_yield
+   + Column avocados_yield
+   + Column bambara_beans__dry_yield
+   + Column banana_yield
+   + Column barley_yield
+   + Column bean_yield
+   + Column beans__green_yield
+   + Column blueberries_yield
+   + Column broad_beans_and_horse_beans__green_yield
+   + Column broad_beans_yield
+   + Column buckwheat_yield
+   + Column cabbages_yield
+   + Column canary_seed_yield
+   + Column carrots_and_turnips_yield
+   + Column cashew_nuts_yield
+   + Column cashewapple_yield
+   + Column cassava_leaves_yield
+   + Column cassava_yield
+   + Column castor_oil_seed_yield
+   + Column cauliflowers_and_broccoli_yield
+   + Column cereals_n_e_c__yield
+   + Column cereal_yield
+   + Column cherries_yield
+   + Column chestnut_yield
+   + Column chickpeas_yield
+   + Column chicory_roots_yield
+   + Column chillies_and_peppers__green__capsicum_spp__and_pimenta_spp__yield
+   + Column chillies_and_peppers_yield
+   + Column cinnamon_and_cinnamon_tree_flowers__raw_yield
+   + Column citrus_fruit_yield
+   + Column cloves__whole_stems__raw_yield
+   + Column cocoa_yield
+   + Column coconuts_yield
+   + Column coffee_yield
+   + Column cow_peas_yield
+   + Column cranberries_yield
+   + Column cucumbers_and_gherkins_yield
+   + Column currants_yield
+   + Column dates_yield
+   + Column edible_roots_and_tubers_with_high_starch_or_inulin_content__n_e_c__fresh_yield
+   + Column eggplants_yield
+   + Column fibre_crops__fibre_equivalent_yield
+   + Column fibre_crops_yield
+   + Column figs_yield
+   + Column flax__raw_or_retted_yield
+   + Column fonio_yield
+   + Column fruit_yield
+   + Column garlic_yield
+   + Column ginger__raw_yield
+   + Column gooseberries_yield
+   + Column grapefruit_yield
+   + Column grapes_yield
+   + Column green_maize_yield
+   + Column groundnut_yield
+   + Column hazelnuts_yield
+   + Column hempseed_yield
+   + Column herbs_yield
+   + Column hop_cones_yield
+   + Column jojoba_seeds_yield
+   + Column jute_yield
+   + Column kapok_fruit_yield
+   + Column karite_nuts_yield
+   + Column kenaf__and_other_textile_bast_fibres__raw_or_retted_yield
+   + Column kiwi_yield
+   + Column kola_nuts_yield
+   + Column leeks_yield
+   + Column lemons_and_limes_yield
+   + Column lentils_yield
+   + Column lettuce_yield
+   + Column linseed_yield
+   + Column locust_beans__carobs__yield
+   + Column lupins_yield
+   + Column maize_yield
+   + Column mangoes_yield
+   + Column mate_leaves_yield
+   + Column melon_yield
+   + Column melonseed_yield
+   + Column millet_yield
+   + Column mixed_grains_yield
+   + Column mustard_seed_yield
+   + Column natural_rubber_in_primary_forms_yield
+   + Column nutmeg__mace__cardamoms__raw_yield
+   + Column oats_yield
+   + Column oilcrops__cake_equivalent_yield
+   + Column oilcrops__oil_equivalent_yield
+   + Column okra_yield
+   + Column olives_yield
+   + Column onions_and_shallots__green_yield
+   + Column onions_yield
+   + Column orange_yield
+   + Column other_berries_and_fruits_of_the_genus_vaccinium_n_e_c__yield
+   + Column other_citrus_fruit__n_e_c__yield
+   + Column other_fruits__n_e_c__yield
+   + Column other_nuts__excluding_wild_edible_nuts_and_groundnuts__in_shell__n_e_c__yield
+   + Column other_oil_seeds__n_e_c__yield
+   + Column other_pome_fruits_yield
+   + Column other_pulses_n_e_c__yield
+   + Column other_stimulant__spice_and_aromatic_crops__n_e_c__yield
+   + Column other_stone_fruits_yield
+   + Column other_sugar_crops_n_e_c__yield
+   + Column other_tropical_fruits__n_e_c__yield
+   + Column other_vegetables__fresh_n_e_c__yield
+   + Column oilpalm_yield
+   + Column papayas_yield
+   + Column peaches_and_nectarines_yield
+   + Column pears_yield
+   + Column pea_yield
+   + Column peas__green_yield
+   + Column pepper_yield
+   + Column peppermint__spearmint_yield
+   + Column persimmons_yield
+   + Column pigeon_peas_yield
+   + Column pineapples_yield
+   + Column pistachios_yield
+   + Column plantains_yield
+   + Column plums_yield
+   + Column poppy_seeds_yield
+   + Column potato_yield
+   + Column pulses_yield
+   + Column pumpkins__squash_and_gourds_yield
+   + Column pyrethrum__dried_flowers_yield
+   + Column quinces_yield
+   + Column quinoa_yield
+   + Column ramie__raw_or_retted_yield
+   + Column rapeseed_yield
+   + Column raspberries_yield
+   + Column rice_yield
+   + Column roots_and_tubers_yield
+   + Column rye_yield
+   + Column safflower_seed_yield
+   + Column cotton_yield
+   + Column sesame_seed_yield
+   + Column sisal__raw_yield
+   + Column sorghum_yield
+   + Column sour_cherries_yield
+   + Column soybean_yield
+   + Column spinach_yield
+   + Column strawberries_yield
+   + Column string_beans_yield
+   + Column sugarbeet_yield
+   + Column sugarcane_yield
+   + Column sugar_crops_yield
+   + Column sunflower_yield
+   + Column sweet_potatoes_yield
+   + Column tallowtree_seeds_yield
+   + Column tangerines_yield
+   + Column taro_yield
+   + Column tobacco_yield
+   + Column tomato_yield
+   + Column treenuts_yield
+   + Column triticale_yield
+   + Column true_hemp__raw_or_retted_yield
+   + Column tung_nuts_yield
+   + Column vanilla__raw_yield
+   + Column vegetables_yield
+   + Column vetches_yield
+   + Column walnuts_yield
+   + Column watermelons_yield
+   + Column wheat_yield
+   + Column yams_yield
+   + Column yautia_yield
+   + Column barley_attainable_yield
+   + Column cassava_attainable_yield
+   + Column cotton_attainable_yield
+   + Column groundnut_attainable_yield
+   + Column maize_attainable_yield
+   + Column millet_attainable_yield
+   + Column oilpalm_attainable_yield
+   + Column potato_attainable_yield
+   + Column rapeseed_attainable_yield
+   + Column rice_attainable_yield
+   + Column rye_attainable_yield
+   + Column sorghum_attainable_yield
+   + Column soybean_attainable_yield
+   + Column sugarbeet_attainable_yield
+   + Column sugarcane_attainable_yield
+   + Column sunflower_attainable_yield
+   + Column wheat_attainable_yield
+   + Column barley_yield_gap
+   + Column cassava_yield_gap
+   + Column cotton_yield_gap
+   + Column groundnut_yield_gap
+   + Column maize_yield_gap
+   + Column millet_yield_gap
+   + Column oilpalm_yield_gap
+   + Column potato_yield_gap
+   + Column rapeseed_yield_gap
+   + Column rice_yield_gap
+   + Column rye_yield_gap
+   + Column sorghum_yield_gap
+   + Column soybean_yield_gap
+   + Column sugarbeet_yield_gap
+   + Column sugarcane_yield_gap
+   + Column sunflower_yield_gap
+   + Column wheat_yield_gap
+ Dataset garden/agriculture/2026-03-02/corn_yields
+ + Table corn_yields
+   + Column yield
+ Dataset garden/agriculture/2026-03-02/daily_calories_per_person
+ + Table daily_calories_per_person
+   + Column daily_calories
+   + Column source
+ Dataset garden/agriculture/2026-03-02/long_term_crop_yields
+ + Table long_term_crop_yields
+   + Column abaca__manila_hemp__raw_yield
+   + Column agave_fibres__raw__n_e_c__yield
+   + Column almonds_yield
+   + Column apples_yield
+   + Column apricots_yield
+   + Column areca_nuts_yield
+   + Column artichokes_yield
+   + Column asparagus_yield
+   + Column avocados_yield
+   + Column bambara_beans__dry_yield
+   + Column bananas_yield
+   + Column barley_yield
+   + Column beans__dry_yield
+   + Column beans__green_yield
+   + Column blueberries_yield
+   + Column broad_beans_and_horse_beans__green_yield
+   + Column broad_beans_yield
+   + Column buckwheat_yield
+   + Column cabbages_yield
+   + Column canary_seed_yield
+   + Column carrots_and_turnips_yield
+   + Column cashew_nuts_yield
+   + Column cashewapple_yield
+   + Column cassava_leaves_yield
+   + Column cassava_yield
+   + Column castor_oil_seed_yield
+   + Column cauliflowers_and_broccoli_yield
+   + Column cereals_n_e_c__yield
+   + Column cereals_yield
+   + Column cherries_yield
+   + Column chestnut_yield
+   + Column chickpeas_yield
+   + Column chicory_roots_yield
+   + Column chillies_and_peppers__green__capsicum_spp__and_pimenta_spp__yield
+   + Column chillies_and_peppers_yield
+   + Column cinnamon_and_cinnamon_tree_flowers__raw_yield
+   + Column citrus_fruit_yield
+   + Column cloves__whole_stems__raw_yield
+   + Column cocoa_beans_yield
+   + Column coconuts_yield
+   + Column coffee__green_yield
+   + Column cow_peas_yield
+   + Column cranberries_yield
+   + Column cucumbers_and_gherkins_yield
+   + Column currants_yield
+   + Column dates_yield
+   + Column edible_roots_and_tubers_with_high_starch_or_inulin_content__n_e_c__fresh_yield
+   + Column eggplants_yield
+   + Column fibre_crops__fibre_equivalent_yield
+   + Column fibre_crops_yield
+   + Column figs_yield
+   + Column flax__raw_or_retted_yield
+   + Column fonio_yield
+   + Column fruit_yield
+   + Column garlic_yield
+   + Column ginger__raw_yield
+   + Column gooseberries_yield
+   + Column grapefruit_yield
+   + Column grapes_yield
+   + Column green_maize_yield
+   + Column groundnuts_yield
+   + Column hazelnuts_yield
+   + Column hempseed_yield
+   + Column herbs_yield
+   + Column hop_cones_yield
+   + Column jojoba_seeds_yield
+   + Column jute_yield
+   + Column kapok_fruit_yield
+   + Column karite_nuts_yield
+   + Column kenaf__and_other_textile_bast_fibres__raw_or_retted_yield
+   + Column kiwi_yield
+   + Column kola_nuts_yield
+   + Column leeks_yield
+   + Column lemons_and_limes_yield
+   + Column lentils_yield
+   + Column lettuce_yield
+   + Column linseed_yield
+   + Column locust_beans__carobs__yield
+   + Column lupins_yield
+   + Column maize_yield
+   + Column mangoes_yield
+   + Column mate_leaves_yield
+   + Column melon_yield
+   + Column melonseed_yield
+   + Column millet_yield
+   + Column mixed_grains_yield
+   + Column mustard_seed_yield
+   + Column natural_rubber_in_primary_forms_yield
+   + Column nutmeg__mace__cardamoms__raw_yield
+   + Column oats_yield
+   + Column oilcrops__cake_equivalent_yield
+   + Column oilcrops__oil_equivalent_yield
+   + Column okra_yield
+   + Column olives_yield
+   + Column onions_and_shallots__green_yield
+   + Column onions_yield
+   + Column oranges_yield
+   + Column other_berries_and_fruits_of_the_genus_vaccinium_n_e_c__yield
+   + Column other_citrus_fruit__n_e_c__yield
+   + Column other_fruits__n_e_c__yield
+   + Column other_nuts__excluding_wild_edible_nuts_and_groundnuts__in_shell__n_e_c__yield
+   + Column other_oil_seeds__n_e_c__yield
+   + Column other_pome_fruits_yield
+   + Column other_pulses_n_e_c__yield
+   + Column other_stimulant__spice_and_aromatic_crops__n_e_c__yield
+   + Column other_stone_fruits_yield
+   + Column other_sugar_crops_n_e_c__yield
+   + Column other_tropical_fruits__n_e_c__yield
+   + Column other_vegetables__fresh_n_e_c__yield
+   + Column palm_fruit_oil_yield
+   + Column papayas_yield
+   + Column peaches_and_nectarines_yield
+   + Column pears_yield
+   + Column peas__dry_yield
+   + Column peas__green_yield
+   + Column pepper_yield
+   + Column peppermint__spearmint_yield
+   + Column persimmons_yield
+   + Column pigeon_peas_yield
+   + Column pineapples_yield
+   + Column pistachios_yield
+   + Column plantains_yield
+   + Column plums_yield
+   + Column poppy_seeds_yield
+   + Column potatoes_yield
+   + Column pulses_yield
+   + Column pumpkins__squash_and_gourds_yield
+   + Column pyrethrum__dried_flowers_yield
+   + Column quinces_yield
+   + Column quinoa_yield
+   + Column ramie__raw_or_retted_yield
+   + Column rapeseed_yield
+   + Column raspberries_yield
+   + Column rice_yield
+   + Column roots_and_tubers_yield
+   + Column rye_yield
+   + Column safflower_seed_yield
+   + Column seed_cotton_yield
+   + Column sesame_seed_yield
+   + Column sisal__raw_yield
+   + Column sorghum_yield
+   + Column sour_cherries_yield
+   + Column soybeans_yield
+   + Column spinach_yield
+   + Column strawberries_yield
+   + Column string_beans_yield
+   + Column sugar_beet_yield
+   + Column sugar_cane_yield
+   + Column sugar_crops_yield
+   + Column sunflower_seed_yield
+   + Column sweet_potatoes_yield
+   + Column tallowtree_seeds_yield
+   + Column tangerines_yield
+   + Column taro_yield
+   + Column tobacco_yield
+   + Column tomatoes_yield
+   + Column treenuts_yield
+   + Column triticale_yield
+   + Column true_hemp__raw_or_retted_yield
+   + Column tung_nuts_yield
+   + Column vanilla__raw_yield
+   + Column vegetables_yield
+   + Column vetches_yield
+   + Column walnuts_yield
+   + Column watermelons_yield
+   + Column wheat_yield
+   + Column yams_yield
+   + Column yautia_yield
+ + Table long_term_crop_yields_smoothed
+   + Column barley_yield_smoothed
+   + Column oats_yield_smoothed
+   + Column potatoes_yield_smoothed
+   + Column pulses_yield_smoothed
+   + Column rye_yield_smoothed
+   + Column sugar_beet_yield_smoothed
+   + Column wheat_yield_smoothed
+ Dataset garden/agriculture/2026-03-02/long_term_food_and_agriculture_trends
+ + Table long_term_food_and_agriculture_trends
+   + Column population
+   + Column food_supply
+   + Column agricultural_land
+ Dataset garden/agriculture/2026-03-02/long_term_wheat_yields
+ + Table long_term_wheat_yields
+   + Column wheat_yield
+ Dataset garden/agriculture/2026-03-02/uk_long_term_yields
+ + Table uk_long_term_yields
+   + Column barley_yield
+   + Column oats_yield
+   + Column potatoes_yield
+   + Column pulses_yield
+   + Column rye_yield
+   + Column sugar_beet_yield
+   + Column wheat_yield
+ Dataset garden/animal_welfare/2026-03-03/animals_used_for_food
+ + Table animals_used_for_food
+   + Column n_wild_fish_per_capita
+   + Column n_farmed_fish_per_capita
+   + Column n_farmed_crustaceans_per_capita
+   + Column n_wild_fish
+   + Column n_farmed_fish
+   + Column n_farmed_crustaceans
+   + Column n_wild_fish_low_per_capita
+   + Column n_farmed_fish_low_per_capita
+   + Column n_farmed_crustaceans_low_per_capita
+   + Column n_wild_fish_low
+   + Column n_farmed_fish_low
+   + Column n_farmed_crustaceans_low
+   + Column n_wild_fish_high_per_capita
+   + Column n_farmed_fish_high_per_capita
+   + Column n_farmed_crustaceans_high_per_capita
+   + Column n_wild_fish_high
+   + Column n_farmed_fish_high
+   + Column n_farmed_crustaceans_high
+   + Column n_animals_killed
+   + Column n_animals_killed_per_capita
+   + Column n_animals_alive
+   + Column n_animals_alive_per_capita
= Dataset garden/usda_ers/2025-03-26/food_expenditure
  = Table food_expenditure
+ Dataset garden/usda_ers/2026-03-02/food_expenditure_in_us
+ + Table food_expenditure_in_us
+   + Column food_as_share_of_disposable_income__at_and_away_from_home
+   + Column food_consumed_at_home_as_share_of_disposable_income
+   + Column food_consumed_away_from_home_as_share_of_disposable_income
+   + Column food_eaten_away_from_home_as_share_of_total_food_expenditure
+   + Column food_expenditure_at_home__constant_1988_prices
+   + Column food_expenditure_at_home__current_prices
+   + Column food_expenditure_away_from_home__constant_1988_prices
+   + Column food_expenditure_away_from_home__current_prices
+   + Column food_expenditure_total__constant_1988_prices
+   + Column food_expenditure_total__current_prices
+ Dataset garden/usda_nass/2026-03-02/us_corn_yields
+ + Table us_corn_yields
+   + Column corn_yield


Legend: +New  ~Modified  -Removed  =Identical  Details
Hint: Run this locally with etl diff REMOTE data/ --include yourdataset --verbose --snippet

Automatically updated datasets matching excess_mortality|covid|fluid|flunet|country_profile|garden/ihme_gbd/2019/gbd_risk are not included

Edited: 2026-03-03 17:21:16 UTC
Execution time: 5.54 seconds

[Copilot]

Pull request overview

This PR updates agriculture and animal welfare pipeline steps to use the latest FAOSTAT dataset version (2026-02-25), replacing the previous 2025-03-17 version. Several new snapshots, meadow, garden, and grapher steps are introduced with updated dates, while old steps are archived.

Changes:

• Updates all agriculture/animal welfare steps to reference the new FAOSTAT 2026-02-25 datasets.
• Archives previously active steps (moved from active DAGs to archive DAGs).
• Improves the step-archiving logic to block archiving of steps with active dependents unless --include-usages is specified.

Reviewed changes

Copilot reviewed 47 out of 47 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
snapshots/usda_nass/2026-03-02/us_corn_yields.csv.dvc	New snapshot DVC file for updated US corn yields data
snapshots/usda_ers/2026-03-02/food_expenditure_in_us.xlsx.dvc	New snapshot DVC file for updated US food expenditure data
snapshots/usda_ers/2025-03-26/food_expenditure_since_2019.xlsx.dvc	Adds a comment noting the data has not been updated as of 2026-03-03
snapshots/agriculture/2026-03-02/corn_yields.xls.dvc	New snapshot DVC file for updated USDA/FAS corn yields data
etl/steps/data/meadow/usda_nass/2026-03-02/us_corn_yields.py	New meadow step for US corn yields
etl/steps/data/meadow/usda_ers/2026-03-02/food_expenditure_in_us.py	New meadow step for US food expenditure data
etl/steps/data/meadow/agriculture/2026-03-02/corn_yields.py	New meadow step for USDA/FAS corn yields
etl/steps/data/grapher/usda_ers/2026-03-02/food_expenditure_in_us.py	New grapher step for US food expenditure data
etl/steps/data/grapher/animal_welfare/2026-03-03/land_animals_killed_for_food.py	New grapher step for land animals killed for food
etl/steps/data/grapher/animal_welfare/2026-03-03/land_animals_killed_for_food.meta.yml	Metadata for land animals killed for food grapher step
etl/steps/data/grapher/animal_welfare/2026-03-03/animals_used_for_food.py	New grapher step for animals used for food
etl/steps/data/grapher/agriculture/2026-03-02/*.py	New grapher steps for various agriculture datasets
etl/steps/data/garden/*/2026-03-02/*.py	New garden steps for agriculture/welfare data using updated FAOSTAT
etl/steps/data/garden/*/2026-03-02/*.meta.yml	Metadata for new garden steps
etl/dag_helpers.py	Bug fix: prevents partial step name matches when removing steps from DAG
dag/faostat.yml	Removes old additional_variables steps from active DAG
dag/archive/faostat.yml	Archives old additional_variables steps
dag/archive/animal_welfare.yml	Archives old animal welfare steps
dag/archive/agriculture.yml	Archives old agriculture steps
dag/animal_welfare.yml	Updates active animal welfare steps to 2026-03-03 versions
dag/agriculture.yml	Updates active agriculture steps to 2026-03-02 versions
apps/step_update/cli.py	Improves archiving logic to block archiving steps with active dependents