Extract text from region in reading order

[dalanicolai]

Some very short background: I am trying to create a pymupdf backend for the amazing emacs pdf-tools package. Currently, pdf-tools uses a server written in C using the 'poppler' pdf-library. Creating a server using 'pymupdf' would make it much more hackable, and would make it much easier to add features like line annotations, pdf forms support etc. (I have already added a dirty hack for pymupdf support in the pymupdf-mode package, but that is no clean solution).
I have the pymupdf-server/backend already working b.t.w. (including annotations, but not yet for annotating text in reading order)

Now to get to the point: I would like to be able to select text regions in reading order (over multiple lines). Checking out the pdf-tools epdfinfo server, I have found that poppler 'simply' provides a function (poppler_page_get_selected_region) for this, but I have found no equivalent in pymupdf and neither in your pymupdf-utilities.

However, checking out the mupdf-gl reader (part of the mupdf library) annotation capabilities, I have found that that reader perfectly supports selecting text in reading order, so I would guess that the mupdf library somehow also already provides a function for this (which would probably be much faster than any hack written in python). Therefore, I was wondering if you are already aware of this, and if it would be possible to include this functionality in pymupdf (I hope you could determine the responsible function yourself, as I am no c-developer, but I would be happy to investigate it for you). It would be great if this would be possible (I figured you might be interested yourself also). Additionally, if pdf-tools would switch to the pymupdf library, this would be a great showcase for pymupdf as pdf-tools is really one of the most beautiful and most powerful, and wiht pymupdf for sure the most hackable, pdf readers available.

Here follow two screenshots of the functionality I would like to see, the first is of Emacs with pdf-tools using the current epdfinfo server, the second is of the 'mupdf' library its own mupdf-gl reader (in both images, the active areas are obtained using a single selection/mouse-drag). I only need to obtain the rectangles, coloring the active area I simply achieve (already) using the Pillow pdf library.

PDF-TOOLS

MUPDF-GL

Thank you!

[JorjMcKie]

There have been a number of text processing enhancements lately. For you purposes, let rect be the minimum rectangle containing start point A, end point B and which has the full page width, resp column text width. Then do words = page.get_text("words", sort=True, clip=rect).
From the start of the resulting list remove items as long as word[0] < A.x and remove words from the the end of the list as long as word[2] > B.x.
In your attached example, A corresponds to top-left of word "It", and B to bottom-right of word "editor.".
The resulting cleaned list then has the text " ".join([w[4] for w in words]).

[dalanicolai]

Thank you for your quick reply. Your solution works, but it is not ideal for the following two (or three) reasons:

1. It requires users to drag from before and above the upper left corner to after and below the lower right corner of the to be annotated text. In most other editors (like current pdf-tools and mupdf-gl), the user can just drag from left to right, from somewhere within the vertical location of the starting line to somewhere in the vertical location of the final line (this can be 'fixed' by writing some extra python code).
2. As far as I know, even with this trick to get the rectangles, it seems currently not possible to create a single annotation over multiple lines if the 'final markup' is not a single exact rectangle (the annotations in the examples of my initial message require multiple rectangles for the markup definition). Looking at the current code of pdf-tools, I have found that the annotations have a rectangle (named edges) definition and additionally a markup-edges definition, which can contain multiple rectangles for the exact markup. I guess it maps to the QuadPoints of the markup annotations as described on page 506 of the pdf specs (although it could be the AP streams as described on page 491 too). I could not find an option to set these using pymupdf. (I don't see how to fix this by using only the currently available pymupdf functions)
3. The selected area does not look very nice, although this is not very important, and also this is probably only because I am not used to the disconnected word-regions.

Here follows a screenshot of some active area using your trick (I did not yet bother to remove the 'items' from the left and the right, as I do not see how to use the result anyway, i.e. create the single annotation using multiple rectangles for markup)

[JorjMcKie]

Maybe we talked a little past each other:
If you were referring to adding highlight annotations between 2 given points (as opposed to extracting text from that region), things are easier text marking.
The call pattern is this add_highlight_annot(quads=None, start=None, stop=None, clip=None).
Just specify page.add_highlight_annot(start=A, stop=B) and you will get something like this:

Where the red dots are A resp. B.
For a multi-column page you would have to also specify clip to prevent the logic extending lines across columns.

[dalanicolai]

Thanks again for the answer, and sorry for the late reply. I have been on a short trip, visiting family.

Indeed, this looks like what I was looking for, but I would like the same function also for 'marking' some 'active region' first. Looking at the code, I have found that add_highlight_annot simply uses get_highlight_selection, so I figured that I could use this function directly for the 'marking' instead of the 'trick' that you informed me about in your previous message.

However, after using the above functions for marking and highlighting, I have found that defining the correct 'start' and 'stop' points to select/highlight some desired region is quite tricky (using the mouse). The behavior seems somewhat 'unpredictable', and it seems not suited for easy selection with the mouse. But as you mention in your message below, maybe I should determine the 'start' and 'stop' points via some algorithm.

To clarify... what I would like to achieve is to have a feature to select some (text) region with the mouse, but in a similar way as in most other PDF reader/editors, i.e. for text on a single line, by simply dragging horizontally over the text and preferably the ability to select per letter instead of per word. This kind of /selecting/highlighting is what is offered by pdf-tools and also by 'mupdf-gl'.

It would be great if the algorithm that 'mupdf-gl' uses would be available in pymupdf (I am not sure if the algorithm is part of the reader or the mupdf code, but if it is part of the mupdf code then my original request was if that functionality could be added to pymupdf).

[JorjMcKie]

So the effort reduces to find the A, B coordinates - and, potentially, the left and right borders of the page text columns we should restrict the highlight to.

BTW: the above works the same way for underline, strikethrough and squiggly annotations.

[JorjMcKie]

I am going to move this thread to the Discussions thread.

[JorjMcKie]

I have found that defining the correct 'start' and 'stop' points to select/highlight some desired region is quite tricky (using the mouse).

Well, somehow the user selection must be communicated to your app. If you control the GUI, then you should be able to get this info. You could instruct the user to mark the start, and, potentially as a separeate interaction, the end of the region to mark. Could be just one character each.
The app code will receive a rectangle per each mark. Take the first rect's top-left for A and the bottom-right for the second rect as B. Done.

There is new rectangle method torect() which can be used to transform between page and image (pixmap) coordinates. Computes a matrix e.g. via instruction mat = page.rect.torect(pix.irect).
Then, given a point A in pixmap coordinates, the point page_A = A * ~mat is the corresponding point in page coordinates. Works also for rectangles. Or for transforming page coordinates to pixmap coordinates: just use mat.

[dalanicolai]

Finding the correct start and stop points is a pretty simple job indeed, I simply pass the start point to the following function:

def point_to_word(point):
    i = 0
    while point[1] > current_page_text[i][3]:
        i += 1
    while point[0] > current_page_text[i][2]:
        i += 1
    return current_page_text[i]

where current_page_text is the page (words) list.

However, the function get_highlight_selection (and therefore also add_highlight_annot) seems to function incorrectly. Namely, the highlighting stops at the correct location, but it always starts at the beginning of the line.

Thanks for pointing out the torect() function. It will make the code a little simpler indeed...

[JorjMcKie]

Well, I don't know what happens in detail on your side, but this is my approach, which works just fine:

>>> import fitz
>>> doc=fitz.open("v110-changes.pdf")
>>> page=doc[0]
>>> rl1 = page.search_for("pixmaps coming from")
>>> len(rl1)
1
>>> A = rl1[0].tl
>>> rl2=page.search_for("needs to be checked.")
>>> len(rl2)
1
>>> B = rl2[0].br
>>> page.add_highlight_annot(start=A, stop=B)
'Highlight' annotation on page 0 of v110-changes.pdf
>>> doc.ez_save("x.pdf")
>>> 

Before:

After:

[dalanicolai]

Ah that's interesting. So I have tested it again, and it turns out that the previously described 'incorrect' behavior happens in my particular document (i.e. a document created using latex). So I tried on a different document (for uploading it here), but in that document it works correctly.

Also, I have tried to create a smaller, simpler, latex test document, but on that document things work fine also.

So it seems that things work fine on most documents. If you are interested to receive the document that 'shows faulty behavior' then let me know. It would be no problem to send it.

Thank you very much for your help, and the nice work on PyMuPDF!

I will notify you as soon as the PyMuPDF backend is ready. Of course it requires Emacs, but if you use GNU/linux, and if you are familiar with using vim, then you could simply try it via Spacemacs which really is a marvelous editor...

[JorjMcKie]

If it doesn't work for some PDFs only, then chances are that there exist unclear / sloppy "geometry changes" as explained in at the of the recipes chapter.
It would probably help to do a page.clean_contents() before inserting anything in it - including annotations.