llm for relation extraction

[AGudden]

I am using the model for enrity and relation extraction. I adjusted the spacy model a little so that I can upload sentences in a csv file. I did some testing and notices that for some sentences I got following error in my terminal: IndexError: tuple index out of range
Closer inspection showed that this is properly caused by sentences that have only 1 entity. I also noticed that the model extraxt relations between entities that are not in my example.json.file.

So my question is: For the model to work do you need to preprocess input text based on the relarion you want to extract? So if I want to extract the relation between PERSON and GPE/LOC do I only need to feed sentences that contains a PERSON and a GPE/LOC?

And how does the model deal with sentences where entities co-occure but do not have a relation?

Thanks,
Arno

[rmitsch]

Hi @AGudden!

...for some sentences I got following error in my terminal: IndexError: tuple index out of range
Closer inspection showed that this is properly caused by sentences that have only 1 entity. I also noticed that the model extraxt relations between entities that are not in my example.json.file.

Could you post your stack trace (and ideally the data you're using)?

So my question is: For the model to work do you need to preprocess input text based on the relarion you want to extract? So if I want to extract the relation between PERSON and GPE/LOC do I only need to feed sentences that contains a PERSON and a GPE/LOC?

While this depends on the model, I think it's safe to say it's not necessary to only feed sentences with the entity types whose relations to each other you're looking for. What I would recommend, to increase accuracy, is to provide some few-shot examples for those entity types.

And how does the model deal with sentences where entities co-occure but do not have a relation?

This depends entirely on the used LLM, spacy-llm has no impact on this. I recommend trying out some examples to get a feel for how well this case is handled.

[rmitsch]

Alright, I looked into this. This is a bug on our side. The issue is that the LLM returns incorrect information here - '{"dep": 0, "dest": 1, "relation": "WorksIn"}' despite there being only one entity in the prompt text. We should check for the indices to be consistent with the number of entities, but we aren't doing this in the current implementation.

We'll release a patch (v0.4.2) for this soon. Probably end of this or beginning of next week. I'll update here once this has been done.

As a note - your examples are currently annoted incorrectly:

Laura[ENT0:PERSON] heeft een huis gekocht in Amste[ENT1:GPE]rdam.
Michael[ENT0:PERSON] is op een stedentrip in Barcel[ENT1:GPE]ona.
Peter[ENT0:PERSON] werk momenteel in Groni[ENT1:GPE]ngen.

You'll want to change your examples.json to

{"text": "Laura heeft een huis gekocht in Amsterdam.", "ents": [{"start_char": 0, "end_char": 5, "label": "PERSON"}, {"start_char": 28, "end_char": 41, "label": "GPE"}], "relations": [{"dep": 0, "dest": 1, "relation": "LivesIn"}]}
{"text": "Michael is op een stedentrip in Barcelona.", "ents": [{"start_char": 0, "end_char": 7, "label": "PERSON"}, {"start_char": 29, "end_char": 41, "label": "GPE"}], "relations": [{"dep": 0, "dest": 1, "relation": "Visits"}]}
{"text": "Peter werk momenteel in Groningen.", "ents": [{"start_char": 0, "end_char": 5, "label": "PERSON"}, {"start_char": 20, "end_char": 43, "label": "GPE"}], "relations": [{"dep": 0, "dest": 1, "relation": "WorksIn"}]}

[rmitsch]

FYI: the corresponding PR has been merged.

[AGudden]

Hi Mitsch,
Thanks for your anwer. I will keep an eye on the new release. For now I can use a workaround by uploading sentence that contains 2 entities.
I will change my example.json file, thanks for that.

[rmitsch]

spacy-llm==0.4.2 is out and should avoid this issue (LLM responses pointing at entity indices higher than the number of available entities are discarded).