NER plus Relationship Extraction (RE) on document

[KMayank29]

I have scrapped few news articles that talk about not very famous personalities.
The task is to extract structured information out of these articles. The structured information is as follows:

• Get NER tags for PER, ORG and GPE
• Find Relationship among PER, ORG and GPE

The first task is quite easy. I have achieved that thanks to spaCy :slight_smile: .
The second part of the problem is a bit tricky. I can do RE for a sentence which is not very complex using dependency parsing but for complex sentences and in an article that contains so many sentences doing same (RE) is not that easy 😦 .

Could you guys share me your industrial experience and guide me a little bit.
Thank you so much for your time.

[polm]

What kind of relations do you want to extract?

If you just want to take whatever's there, what you're doing could be defined as "Open Information Extraction". That can be of limited usefulness though because it's hard to tell when things are the same, or one relation implies another.

You might want to look at the example project on relation extraction or at Chapter 17 on Information Extraction in the Jurafsky and Martin book for an overview of the field.

[KMayank29]

Thank you @polm for your quick response.

Given a news article, I want to extract PER, ORG and GPE entities and find the relationship.
For example, John Smith being a serial entrepreneur founded a fintech startup Razorpay based out of Seattle.......

I want to find --> (John Smith, Founder, Razorpay), (Razorpay, in, Seattle)

[polm]

So basically you can either

1. Extract everything that exists
2. Decide what specific relationships you care about and extract them

If you take approach 1, which is what it sounds like you want, this is open information extraction. Some easy things you can do in this situation is take the span of words between any two entities, or take the verb. The problem is that you end up with different words for the same thing, like "based out of" and "in".

How did you know "based out of" and "in" were the same thing? Can you make a list of all possible equivalent phrases? If you can't, how can you communicate to the computer which sentences indicate that relationship and not another one? This is a very hard problem, and is a subject of research.

If you take approach 2, you can use specific verbs or textual patterns and build from them. If you have a small variety of relationships you care about, you can classify entities and sentences to see if they fall into one of a few buckets (your relationships) or not. This doesn't capture all relations but it's possible to make steady progress.

Because approach 1 is very hard, and would often be hard even for humans to get useful data out of it if you don't have a specific use case in mind, I recommend starting with approach 2 by deciding what relationships you actually care about. I strongly suggest you read the Jurafsky and Martin chapter, which has an overview of these issues.