Spacy Debugger 's log gives different results from Assemble in spacy-llm

[innocent-charles]

Hello ! i need assistance on spacy-llm. when I use it with GPT-3.5, if the same entity.text is already extracted it won't shown or extracted again.

Here is example of results from logging :
DEBUG:spacy_llm:LLM response for doc: Magdalena Gerson Mtera ACCADEMIC BACK GROUND PROFESSIONAL DEVELOPMENT I have an ability to interact with different kind of people easily Hard working and committed to achieve expected results on time I can work under non or minimum level of supervision Also I am cooperative and a good team player AWARDS. Trophy of the Best Netball player of the Year (2017) from the Institute of Social Work Dar es Salaam. A certificate of Restless Development Program from the Institute of Social Work Dar es Salaam (2014). A Certificate of Computer Applications in Microsoft Office (Word Excel and Publisher) Internet and E-mails from South and East Africa Minerals and Geosciences Centre (SEAMGC) Dar es Salaam Tanzania (2012). A Certificate of Recognition Youth of United Nations of Tanzania (YUNA). From the UN Club at Morogoro Secondary School (2011). A Certificate of Nursing from Morogoro Secondary School (2011). Bachelor of Human Resource Management Institute of Social Work 2015 to 2017 Diploma of Human Resource Management Institute of Social Work 2013 to 2015 Certificate of Social Work Practice Institute of Social Work 2012 to 2013 Certificate of Secondary Education Morogoro Secondary School 2008 to 2011 Magdalena Gerson Mtera Address South and Eastern Mineral Centre P.O. Box 9573 Dar es Salaam Tanzania Mobile +255 716 665 162 +255 752 720 741 E-mail [email protected] INTERESTS Sports and Volunteering Art and Culture Travelling REFERIES. Ricky Emmanuel Loan Manager Moraiah Financial Services Ltd [email protected] Tel +(255) 758 811217 Salome Doreen Magoha Human Resources Officer Murphy Ben International Tanzania Tel + (255) 687 246425 Mr. Dickson Ndege Lecturer at the Institute of Social Work Dar es Salaam Department of Human Resources Management Tel + (255)715 011234
Education_Institution Level_One: Bachelor
Institution Name_One: Institute of Social Work
Course Name_One: Human Resource Management
Start Date_Institution_One: 2015
End Date_Institution_One: 2017
Education_Institution Level_Two: Diploma
Institution Name_Two: Institute of Social Work
Course Name_Two: Human Resource Management
Start Date_Institution_Two: 2013
End Date_Institution_Two: 2015
Education_Institution Level_Three: Certificate
Institution Name_Three: Institute of Social Work
Course Name_Three: Social Work Practice
Start Date_Institution_Three: 2012
End Date_Institution_Three: 2013
Education_Institution Level_Four: Certificate
Institution Name_Four: Morogoro Secondary School
Course Name_Four: Secondary Education
Start Date_Institution_Four: 2008
End Date_Institution_Four: 2011

There above there is entity like Course Name_Two.

But when I use:

from spacy_llm.util import assemble

nlp = assemble("config.cfg")
doc = nlp("Magdalena Gerson Mtera ACCADEMIC BACK GROUND PROFESSIONAL DEVELOPMENT I have an ability to interact with different kind of people easily Hard working and committed to achieve expected results on time I can work under non or minimum level of supervision Also I am cooperative and a good team player AWARDS. Trophy of the Best Netball player of the Year (2017) from the Institute of Social Work Dar es Salaam. A certificate of Restless Development Program from the Institute of Social Work Dar es Salaam (2014). A Certificate of Computer Applications in Microsoft Office (Word Excel and Publisher) Internet and E-mails from South and East Africa Minerals and Geosciences Centre (SEAMGC) Dar es Salaam Tanzania (2012). A Certificate of Recognition Youth of United Nations of Tanzania (YUNA). From the UN Club at Morogoro Secondary School (2011). A Certificate of Nursing from Morogoro Secondary School (2011). Bachelor of Human Resource Management Institute of Social Work 2015 to 2017 Diploma of Human Resource Management Institute of Social Work 2013 to 2015 Certificate of Social Work Practice Institute of Social Work 2012 to 2013 Certificate of Secondary Education Morogoro Secondary School 2008 to 2011 Magdalena Gerson Mtera Address South and Eastern Mineral Centre P.O. Box 9573 Dar es Salaam Tanzania Mobile +255 716 665 162 +255 752 720 741 E-mail [email protected] INTERESTS Sports and Volunteering Art and Culture Travelling REFERIES. Ricky Emmanuel Loan Manager Moraiah Financial Services Ltd [email protected] Tel +(255) 758 811217 Salome Doreen Magoha Human Resources Officer Murphy Ben International Tanzania Tel + (255) 687 246425 Mr. Dickson Ndege Lecturer at the Institute of Social Work Dar es Salaam Department of Human Resources Management Tel + (255)715 011234")

for x in doc.ents:
  print(x.label_,x.text)

I got this :
End Date_Institution_One 2017
Institution Name_One Institute of Social Work
Education_Institution Level_Three certificate
Institution Name_One Institute of Social Work
Education_Institution Level_Three Certificate
Start Date_Institution_Three 2012
Education_Institution Level_Three Certificate
Institution Name_Four Morogoro Secondary School
End Date_Institution_Four 2011
Education_Institution Level_Three Certificate
Institution Name_Four Morogoro Secondary School
End Date_Institution_Four 2011
Education_Institution Level_One Bachelor
Course Name_One Human Resource Management
Institution Name_One Institute of Social Work
Start Date_Institution_One 2015
End Date_Institution_One 2017
Education_Institution Level_Two Diploma
Course Name_One Human Resource Management
Institution Name_One Institute of Social Work
Start Date_Institution_Two 2013
Start Date_Institution_One 2015
Education_Institution Level_Three Certificate
Course Name_Three Social Work Practice
Institution Name_One Institute of Social Work
Start Date_Institution_Three 2012
Start Date_Institution_Two 2013
Education_Institution Level_Three Certificate
Course Name_Four Secondary Education
Institution Name_Four Morogoro Secondary School
Start Date_Institution_Four 2008
End Date_Institution_Four 2011
Institution Name_One Institute of Social Work

There above the entity Course Name_Two is missing , why ? ... and this happens for all entity that appear to carry the same value.

[svlandeg]

Hi @innocent-charles,

Thanks for the report! I've been able to reproduce this with spacy-llm 0.4.0, using a slightly smaller example:

[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = LOCATION, PERSON

[components.llm.model]
@llm_models = "spacy.GPT-3-5.v1"
name = "gpt-3.5-turbo"
config = {"temperature": 0.3}

and then

    from spacy_llm.util import assemble
    import logging
    import spacy_llm

    spacy_llm.logger.addHandler(logging.StreamHandler())
    spacy_llm.logger.setLevel(logging.DEBUG)

    nlp = assemble("config.cfg")
    text = "The woman Paris was walking around in Paris"
    doc = nlp(text)
    for ent in doc.ents:
        print(ent.label_, ent.text, ent.start, ent.end)

which gives as logging output:

LLM response for doc: The woman Paris was walking around in Paris
LOCATION: Paris
PERSON: Paris

and as stdout output reflecting the (wrong) data in doc.ents:

LOCATION Paris 2 3
LOCATION Paris 7 8

This must be due to the relatively simple parsing/alignment procedure that "spacy.NER.v2" applies. Let me look into this more to see whether we can fix this.

[innocent-charles]

Thanks @svlandeg for noticing this too, that is a severe problem. I am currently working on one of the NER projects, but unfortunately, I have been stuck on that problem... So please, help me out...

[svlandeg]

Hi @innocent-charles,

With spacy-llm, it should be relatively straightforward to implement your own task definition according to your needs, cf here.

We're also working on improving the built-in NER task in the meantime, and will take this feedback on board.