Named Entity Recognition and Extraction Relation

[imhans33]

Hi

I have a specific use case for identifying some specific url's from a web page source data like "Book Tickets". Following is an example of the data i am dealing

<a href="#modal-window-0"><a href="#modal-window-1">Buy Tickets</a></a>
Great choice! But why stop there?
Do you want to save up to 20% by making this a Create-Your-Own subscription?
Click here to learn more.
<a href="/cyo"><a href="/cyo">Yes</a></a>
<a href="https://thebroadstage.secure.force.com/ticket/#/instances/a0F5G00000L12hAUAR"><a href="https://thebroadstage.secure.force.com/ticket/#/instances/a0F5G00000L12hKUAR">Not Right Now</a></a>
SAT, FEB 19
$105/$75/$45
7:30 PM
Buy Tickets
Great choice! But why stop there?

These two urls <a href="https://thebroadstage.secure.force.com/ticket/#/instances/a0F5G00000L12hAUAR"><a href="https://thebroadstage.secure.force.com/ticket/#/instances/a0F5G00000L12hKUAR"> are related to "Buy Tickets" . So Is it possible to detect those urls which are associated with "Buy Tickets". When searched through the features of spacy i found NER + Relation Entity Extractor will solve this kind of problems. If so how we can create the dataset, train the custom model. Any documentation available for custom modelling these specific use cases.
Also there can be several <a href tags in one web page source data(as all knows it) but only these tags which are closer to Buy Ticket are required ones. So in these cases should we tag all the <a href as entities or only the required ones to be tagged.

[polm]

The relation extraction sample project has documentation on how to use it. Unfortunately I don't think relation extraction will actually help with your problem - it's used to find facts implied by sentence structure, like:

• JFK airport is in New York. → located(JFK airport, New York)
• Narita is located in Chiba Prefecture → located(Narita, Chiba Prefecture)

Here the model learns that "is in" and "is located in" are equivalent. This is a simple case but the idea is to learn more complicated ones. However it relies on well formed sentences, or a certain similarity in surrounding text. I don't think there is any of that in the example document you had?

You could probably make training data and train a model but I wouldn't expect it to work very well.

[imhans33]

Thanks for the reply. The similarity in the document will be like Book Tickets, Register Here etc can be present very close to the required <a href ..> tag. It can be varying too but there will be words similar to that. I understood that it will be very difficult for that to work but it seem worth trying as it can be the only one solution which can solve this. In normal NER tagging, data and entity format accepted by Spacy is like

[('JFK airport is in New York', {'entities': [(0, 3, 'airport'),(18,26,'place')]}),
 ('Narita is located in Chiba Prefecture', {'entities': [(0,6, 'airport'),(21,37,'place')]})]

So when Entity linking component located is also included how will the above format look like for the doc bin file to be created.

[polm]

You can see how the data is created in this script.

[tdubon]

@imhans33 You may want to try out the crawler module on Haystack. In particular, see the filter_urls. Documentation is here: https://haystack.deepset.ai/reference/crawler

What kind of documentation are you looking for?

[imhans33]

I think i am not looking for a crawler module to pick all the urls that appear in web page data source. I just need few urls that appear which has a particular meaning. I am not sure whether normal crawling will solve this. But i can see haystack also provides NLP solution which is worth looking. Thanks