Spacy and working with fractions as text

[robertpfaff]

I am working on an application that reads culinary recipes.

I have discovered I need to work around how spaCy normally tags food, units of measurement and numbers. I have used Entity Rulers to resolve the first two issues with no problems. I am really pleased with the accuracy.

The problem lies in how spaCy tags fractions.

It recognizes some numbers in recipes as QUANTITIES (e.g. "2 1/2 teaspoons" as a single entity) as some as CARDINALS (e.g. 2 cups, recognized separately as a "CARDINAL" plus a noun. I tried creating a special entity ruler to recognize all numbers and fractions "amts" as a new entity. But this does not work. The slash symbol "" in the textual representation of a fraction seems to force spaCy to recognize the numerators and denominators just as separate numbers. I played with using escape characters "/" knowing full well that probably would not work. I was right about that at least.

So my question is... now that I have entity rulers that recognize ingredients, and units of measurement accurately as ents, how can I ask spaCy to recognize fractions as they appear in many recipes?

I thought maybe I could do this using token.pos_ "NUM," but it doesn't work either.

Any suggestions?

Thanks in advance.

Robert

[rspeer]

I'd like to see how you're writing the Entity Rule for this and how you're adding the patterns to your pipeline, so I know what you've tried so far.

In the examples I've trying, numbers with slashes are consistently recognized as tokens, even when the phrase as a whole isn't recognized as a QUANTITY by the default NER. This should allow you to write an Entity Rule that matches a NUM followed by certain NOUNs, and labels them as QUANTITY.

For example, this example has no problem with the slash:

>>> nlp = spacy.load("en_core_web_lg")
>>> [(tok, tok.pos_) for tok in nlp("1 1/2 cups of water")]
[(1, 'NUM'), (1/2, 'NUM'), (cups, 'NOUN'), (of, 'ADP'), (water, 'NOUN')]

The forward slash (/) does not need to be escaped in Python.

I haven't come across any examples where the numerator and denominator come out as separate NUM tokens, which is why I'd like to see an example of what you've tried.

[robertpfaff]

I don't mind sharing what I found and what finally worked.

SpaCy NER would sometimes recognize some amounts and units of measurement, as quantities (e.g 1 1/2 teaspoons) sometimes not (2 cups - two was recognized as a cardinal, not a quantity. I don't know why it does that, but that was/is an issue on my end.

The code you show above uses token.pos_, not ent.label_. I was referencing the latter. I tried mixing the two. It may be possible. I played with it. But I could not make it work.

The period in decimal amounts, once I converted the fraction, definitely needed to be escaped in Python regex with a slash.

If I said the forward slash itself needed to be escaped, then I misspoke.

The real challenge turned out to be dealing with "uniciode vulgar fractions," something I knew nothing about before I started working on the problem. I ended up writing code that would convert the unicode character into a decimal amount, but...the real pain was that it still needed to find the number preceding the decimal amount, if one existed, regardless of white spaces between them, and add them together.

That was the real regex challenge once I finally dug into it.

I am new to Python (six months) and brand new to SpaCY and just purchased Prodigy.

I don''t mind sharing the finished code if you still want to see it.

Unfortunately, I can no longer locate the version of the code that produced the problem.

Here is the working code with the three entity rulers:

nlp = spacy.load("en_core_web_lg")

rulerIngs = nlp.add_pipe("entity_ruler", name="rulerIngs", before="ner")
rulerUms = nlp.add_pipe("entity_ruler", name="rulerUms", before="ner")
rulerAmt = nlp.add_pipe("entity_ruler", name="rulerAmt", before="ner")

rulerIngs.from_disk("patterns/ing_patterns2021-10-11.jsonl")
rulerUms.from_disk("patterns/ums_patterns2021-10-11.jsonl")
rulerAmt.from_disk("patterns/amt_patterns2021-10-12.jsonl")

print("")
print("")
print("This is the decoded recipe:")
print("")
print("")

for line in index:
	doc = nlp(line)
	for ent in doc.ents:
		print(ent.text, ent.label_)

Though it has nothing to do with SpaCy, here is the regex code that deals with unicode vulgar fractions and the subsequent problems that resulted from working with them. As I said, it is unedited, but it works.

import csv
import re

with open ("recipes/minestrone.txt", "r", encoding='utf-8') as recipe:
	recipe = recipe.readlines()

fractions = {
     u'\u00BD': '.50',
     u'\u00BC': '.25',
     u'\u00BE': '.75',
     u'\u2153': '.33',
     u'\u2154': '.67',
     u'\u2155': '.20',
     u'\u2156': '.40',
     u'\u2157': '.60',
     u'\u2158': '.80',
     u'\u2159': '.17',
     u'\u215A': '.83',
     u'\u215B': '.13',
     u'\u215C': '.38',
     u'\u215D': '.63',
     u'\u215E': '.88'
     }


new_recipe = []

# Is the unicode fraction in the dictionary?
     
words = line.split()
     for word in words:
          for key, value in fractions.items():

# If found, then replace it with its decimal value:

               if word == key:
                    line = line.replace(word, value)
                    print("Word: ", word)
                    print("Word: ", value)

# For all matches found, is there a preceding number?

                    **matches = re.findall(r"\d\s*(?=" + re.escape(value) + ")", line)**
                    print("Matches :", matches)

# If found, multiply that number by 1+ the decimal value.

                    for match in matches:
                         number = (float(match))
                         new_value = (float(value))
                         print("Number: ", number)
                         print("New value: ", new_value)
                         multiplier = (float(1 + new_value))
                         print("Multiplier:" , multiplier)
                         result = (number * multiplier)
                         print("Result: ", result)
                         string_result = str(result)

# Now replace the number + decimal value for the number and replace the original decimal amount with nothing.

                         line = line.replace(match, string_result)
                         line = line.replace(value, '')
                         print(line)

# If the word is not in the dictionary key, then just continue:

               elif word != key:
                    continue
               
# United revised and unrevised lines into a new recipe list.

     new_recipe.append(line)

print(" ")
print("This recipe should be free of vulgar fractions:")
print(" ")
for line in new_recipe:
     print(line, end='')

Let's say, you have a recipe like this one:

4 tablespoons extra-virgin olive oil, divided
1 medium yellow onion, chopped
2 medium carrots, peeled and chopped
2 medium ribs celery, chopped
¼ (e.g. vulgar fraction) cup tomato paste
2 cups chopped seasonal vegetables (potatoes, yellow squash, zucchini, butternut squash, green beans or peas all work)
4 cloves garlic, pressed or minced
½ teaspoon dried oregano
½ teaspoon dried thyme
1 large can (28 ounces) diced tomatoes, with their liquid (or 2 small 15-ounce cans)
4 cups (32 ounces) vegetable broth
2 cups water
1 teaspoon fine sea salt
2 bay leaves
Pinch of red pepper flakes
Freshly ground black pepper
1 cup whole grain orecchiette, elbow or small shell pasta
1 can (15 ounces) Great Northern beans or cannellini beans, rinsed and drained, or 1 ½ cups cooked beans
2 cups baby spinach, chopped kale or chopped collard greens
2 teaspoons lemon juice
Freshly grated Parmesan cheese, for garnishing (optional)

It now looks like this, so I can now perform the conversions that return a nutritional profile:

4 tablespoons extra-virgin olive oil, divided
1 medium yellow onion, chopped
2 medium carrots, peeled and chopped
2 medium ribs celery, chopped
.25 cup tomato paste
2 cups chopped seasonal vegetables (potatoes, yellow squash, zucchini, butternut squash, green beans or peas all work)
4 cloves garlic, pressed or minced
.50 teaspoon dried oregano
.50 teaspoon dried thyme
1 large can (28 ounces) diced tomatoes, with their liquid (or 2 small 15-ounce cans)
4 cups (32 ounces) vegetable broth
2 cups water
1 teaspoon fine sea salt
2 bay leaves
Pinch of red pepper flakes
Freshly ground black pepper
1 cup whole grain orecchiette, elbow or small shell pasta
1 can (15 ounces) Great Northern beans or cannellini beans, rinsed and drained, or 1.5 cups cooked beans
2 cups baby spinach, chopped kale or chopped collard greens
2 teaspoons lemon juice
Freshly grated Parmesan cheese, for garnishing (optional)