Skip to content

fix: Add bounding box to nutrition extraction#1545

Closed
abderhman487 wants to merge 1 commit intoopenfoodfacts:mainfrom
abderhman487:bounding_box
Closed

fix: Add bounding box to nutrition extraction#1545
abderhman487 wants to merge 1 commit intoopenfoodfacts:mainfrom
abderhman487:bounding_box

Conversation

@abderhman487
Copy link

@abderhman487 abderhman487 commented Apr 12, 2025

What

  • Added bounding box computation for nutrition extraction using OCR word positions
  • Added bounding_box column to both prediction and product_insight tables
  • Included bounding box in nutrition extraction insights
  • Created an endpoint to allow users to propose a new bounding box during validation
  • Modified the image cropping function to use the stored bounding box information

Implementation Details

  • Enhanced NutrientPrediction and NutritionExtractionPrediction classes to include bounding box information
  • Added compute_bounding_box function to calculate the overall bounding box from individual word positions
  • Updated the NutrientExtractionImporter to include bounding box data in generated insights
  • Added support for updating bounding box through annotation in NutrientExtractionAnnotator

How This Improves User Experience

This enhancement significantly improves the visualization of nutrition extraction results during validation. Previously, the bounding box was only used internally for cropping, but was not exposed in the insight data. Now, users validating nutrition insights can see exactly which part of the image was analyzed, and even propose corrections to the bounding box if needed.

Fixes

Add bounding box to nutrition extraction #1539

@abderhman487
Copy link
Author

@alexgarel

@alexgarel
Copy link
Member

@abderhman487 so sorry, I only see this PR now, I will try to review it soon !

@abderhman487
Copy link
Author

@alexgarel @raphael0202 Any updates?

@raphael0202
Copy link
Collaborator

Hello @abderhman487,
I'm sorry for the review delay. I will review your PR this week!

@teolemon
Copy link
Member

@abderhman487 could you solve the small merge conflict so that @raphael0202 can review afterwards ?

@raphael0202
Copy link
Collaborator

I'm sorry it took me so long to review your PR.
One major caveat is that we only detect in the nutrition model the values (ex: 10 g) from the nutrition table, not the mentions (ex: carbohydrates, sugars, proteins,...). It means that the bounding box won't include important information for the user to understand the nutrition table.

We have an object detection model to detect the position of the nutrition table, and we can use the prediction of this model to provide the bounding box of the nutrition table.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

4 participants