How to convert text elements coordinates in images after OCR (pdfocr_save) to hosting PDF coordinates

[dannydocrxiv]

I tried few options

0. I need PDF coordinates not pymupdf coordinates at the end
1. Get the bounding box of the image and calculate x offset and y offset from hosting PDF (using get_image_info())
2. Open the pdf as memory object - fitz.open("pdf", outbytes)
3. Get the coordinate of the text element inside the image OCR.
4. Here I tried few options a) multiply by inverse "transform" matrix from image data and adding offsets of x, y from hosted PDF borders b) adding offsets and then multiply by hosted page inverse matrix

All above did not help to get correct coordinates

Thanks
Danny

[JorjMcKie]

more detail please: how did you make the "image OCR" and which coordinates do you need: text, image ...?

[JorjMcKie]

Aha - these text elements of course use coordinates compatible with their image page - let's call it pixpage.
If you had used the inbuilt feature of OCRing a PDF page, the problem would have been solved for you automatically:

tp = page.get_textpage_ocr(dpi=nnn, full=True, flags=nnn)  # force a full OCR
output = page.get_text(option, textpage=tp)  # will deliver coordinates relative to 'page'

What you did instead, now requires coordinate transformation. You can use the new method torect of the Rect class:

mat = page.rect.torect(pixpage.rect)

This matrix transforms page coordinates to coordinates of the page made from the OCRed pixmap.
Accordingly, ~mat transforms point and rectangle coordinates of pixpage back to ones of page.

As an aside: if you use the default full=False when making the TextPage, a selective OCR will take place: only any images on the page will be OCRed, the rest of the text will be accepted as is. The overall result of normal and OCR text is afterwards available in text extractions and text search.

[dannydocrxiv]

I am still struggling with this
I need PDF coordinates (bottom up) and not PyMuPDF coordinates
Should I multiply by Page transformation after multiplying by ~mat? like:
r = fitz.Rect(word_bbox)
coords = list((r * ~mat) * ~page.transformationMatrix)

BTW: I tried also tp = page.get_textpage_ocr() but the result of the OCR were not good like the pdfocr_save() I am using currently
I am still investigating

Thx

[dannydocrxiv]

Page means the hosting page when I do transformationMatrix

[JorjMcKie]

• You have a page in a PDF that you would like to OCR. Do this:

tp = page.get_textpage_ocr(dpi=300, full=True)  # use excellent resolution and the full page
# for example
print(page.get_text("text", textpage=tp))

Use the dpi parameter until you are satisfied ...

---

Or:

pix = page.get_pixmap(dpi=300)
pdfbytes = pix.pdfocr_tobytes()
ocrdoc = fitz.open("pdf", pdfbytes)
pixpage = ocrdoc[0]
mat = page.rect.torect(pixpage.rect)
# or:
mat = page.rect.torect(pix.irect)
# whatever seems to deliver better precision ...

If you then have a point or rect on pixpage, its corresponding coordinates / objects on page are point * ~mat or rect * ~mat.

[dannydocrxiv]

In my case pix is a pixmap of image raw data
pix = fitz.Pixmap(img_info['image'])

pix.pdfocr_save(outbytes, language=ocr_lang)
doc2 = fitz.open("pdf", outbytes)

The host page is page that host this image I am working on
mat = hostpage.rect.torect(doc2[0].rect)

Then I have a word in Image I want to get True PDF coordinate for (not PyMupdf coordinates)
r = fitz.Rect(word_bbox)

coords = list(r * ~ma)
OR
coords = list((r * ~img_matrix) * ~Hostpage.transformationMatrix)

Both dont work to coordinates of text in image

[JorjMcKie]

The image blocks of get_text("dict") etc. also contain a transformation matrix, key "transform" (matrix like). As documented here, this matrix can be used to transform image coordinates to page coordinates.

Once you have those (they are MuPDF coordinates) simply apply ~page.transformation_matrix to get the PDF equivalent.

[dannydocrxiv]

Tried these matrixes with no success
Here are the details of my case

// Word I am looking inside image : pymupdf coordinates
[26.25, 72.75, 34.125, 91.5091552734375]

#Multiplying by transform matrix : Why so big values?
list(r * img_matrix)
[10808.8505859375, 21519.958984375, 14020.9052734375, 26976.623046875]

#Multiplying by transform and then by page inverse matrix
list((r * img_matrix) * ~hosting_page.transformationMatrix)
[10808.8505859375, -26184.623046875, 14020.9052734375, -20727.958984375]

#Transform matrix of image
Matrix(407.8800048828125, 0.0, -0.0, 290.8800048828125, 102.0, 358.44000244140625)

#When I just look at the core coordinates I get + offset of bbox from hosting page I get the exact coordinates, I am looking for

#Offsets (x & y) ... I calculate from bbox of image inside page
0 = {float} 102.0
1 = {float} 358.44000244140625
2 = {float} 509.8800048828125
3 = {float} 649.3200073242188

// Word I am looking inside image : pymupdf coordinates
[26.25, 72.75, 34.125, 91.5091552734375]

so 102+26, 72 + 358, .. are the one I was looking for ...
So I am not sure how to use these matrixes and why the core values are correct without any transformation...for this specific case

Thx
Danny

[dannydocrxiv]

Also you said

What you did instead, now requires coordinate transformation. You can use the new method torect of the Rect class:

mat = page.rect.torect(pixpage.rect)
This matrix transforms page coordinates to coordinates of the page made from the OCRed pixmap.
Accordingly, ~mat transforms point and rectangle coordinates of pixpage back to ones of page.

I need the coordinates of the PDF HOSTING the image , not the image itself
The hosted PDF can contain multiple images & text

[JorjMcKie]

bboxing.zip

This script is even somewhat simpler as it uses the new Rect method torect.

[JorjMcKie]

The main logic therefore is:

• if imgrect is the rectangle of the PDF page made from pix.pdfocr_tobytes(),
• and bbox the location of the image on the source page,
• then for every text_bbox (or point or whatever) in coordinates of the image page,
• the corresponding rectangle on the source page is text_bbox * mat
• where mat = imgrect.torect(bbox).

As mentioned, to obtain PDF coordinates, compute text_rect * mat * ~page.transformation_matrix, where page is the source page.

[dannydocrxiv]

Works now.
Thanks very much for your help