Extract links while extracting text?

[CaiSamuelsFSA]

I currently have a script that extracts text from a PDF file and converts it into HTML, but I'd also like to extract links.

I know that you can extract links from a page using first_link, but how do you output the links while extracting the text so that they are outputted at the same time?

The goal is to output the link and the text as a HTML hyperlink: <a href="url">link text</a>

[JorjMcKie]

Links have nothing to do with text. For sure, the picture is blurred because it mostly is text that is covered by some link rectangle. But it remains a coincidence: You can cover an image or also nothing by a link.
In practice, this means that you should extract all links via page.get_links() first. Then, while extracting text (including its coordinates) try to match text bboxes with link rectangles (the "from" keys in the dictionaries in the get_links() list) to generate the links in HTML syntax.

With this approach, chances are that not all links in the links list will be eaten up by fitting text ...

[CaiSamuelsFSA]

Ah okay that makes sense.

How do you match text bboxes with link rectangles?

It seems like the bbox of the span is not an exact match to the bbox of the link.

[JorjMcKie]

It seems like the bbox of the span is not an exact match to the bbox of the link.

Exactly! That is the best proof for my above argument!
You might take the link rect and check, which words (from the "words" output) significantly overlap it. E.g. intersection of word and link rectangle are > 90% of the word rectangle - something like that.
Or you take the "rawdict" output and do a similar check for each character.
Then put matching words / characters in between <a> and </a>.
In any case: tedious and ugly.
Much depends on the quality if your input PDF

[CaiSamuelsFSA]

I am trying to export the link text using page.get_textbox(link.rect), however occasionally this is also returning some text from the line below the link. This doesn't seem to happen with page.get_text(clip=link.rect), however instead it returns either y\ng\n, y\ny\n, g\ny\n or g\ng\n at the start of links that are within tables. What could be causing this?

[JorjMcKie]

The logic between the two is slightly different:

• get_textbox() includes all characters that overlap in any way and finally removes an ending "\n".
• get_text() has the goal to provide plain text output and therefore uses a general logic that also delivers line breaks where MuPDF sees them.

I suggest to use get_textbox() with a slightly reduced rectangle. E.g. add (0, x, 0, -x) to the link rectangle with a suitably chosen "x", for instance 10% of the rectangle height or similar.

[JorjMcKie]

Something like this:

rect = link["from"]  # the link "hot area"
h = rect.height * 0.1  # 10% of the link rectangle's height
smaller = rect + (0, h, 0, -h)  # this rectangle has a that is 20% smaller but same width
text = page.get_textbox(smaller)

The hope is that no "foreign" characters are included any longer, but still all relevant link text.

[CaiSamuelsFSA]

I tried matching them up by checking if the link bbox is within the span bbox but most of the time the y1 coordinate of the link was greater than the y1 coordinate of the span so it didn't work. I then figured out that the y0 coordinate of the link always seems to be within the bbox of the span so I just use that to match them up now and it seems to work.

[JorjMcKie]

PDFs are created sloppily most of the time:
It is an illusion to expect that link rectangles ("from") exactly cover the text for which they seem to be made.
Therefore my suggestion to check whether a text bbox intersects a "from" rectangle. This is a simple check and fairly safe.

[CaiSamuelsFSA]

I'm not entirely sure what you mean. Do you mind explaining how this is different to my current method and give and example on how to implement it please?

[JorjMcKie]

Via trial and error you found that y0 of the "from" rectangle suffices to ensure that it meant for the text bbox.
I am saying that bbox.insersects(link["from"]) is safer because more general.

[CaiSamuelsFSA]

That works much better, thank you.