Decipher get_texttrace

[sgthuth]

Good day everyone,

Problem:
Long story short, I have contend with a program that converts PDF files to another format. During this conversion, text that was previously not seen on the PDF, is now visible within the new format. Since I do not have the ability to alter how the other program works, I thought Python would be a good means to solution and I've found a love for PyMuPDF.
Solutioning:
At first thought, I tried to convert each page of the pdf to an image; then, build a new pdf with those images. Works great excluding the file size. The file size is or at least will be a problem especially with hitting a DPI of 300. I thought about compression and if compressing the images prior to building the new pdf would work. I started down that road and of course, I started doing some more crazy things like cutting each page's image into individual pieces to a puzzle but looking at the sizes... it appears all the little image sizes added together equals a size greater than just the single image. I didn't continue with that concept nor work with the images compressed, but I began searching for another solution. (If anyone knows if the compression approach would work OR knows of a better way of doing that, please let me know).
Because the text that becomes visible is under other layers, I started searching for something that could distinguish layers on each page. That's when I came across get_texttrace HERE. I wrote up a short script just to see what the output would look like in a dump.txt file. What I see is similar to a JSON file format with key value pairs. I know there are settings for this but I wasn't able to find anything specific to put other than comparisons to the other things like words, blocks, dicts, etc.

Any advice is much appreciated.

[JorjMcKie]

You seem to be resourceful in finding new ideas. I recommend reading the documentation about get_texttrace() to find the meaning of the dictionary items.
What else do you need?

[sgthuth]

Hmmm... The main reason I thought the imaging would be best is because the converted format is just being printed. There's some validation steps that occur in that other program but it is minimal. I'm scraping the document prior to creating new PDFs based on images. Any information needed on the document is being retained by me or placed back on there but all in all... it's just going to be printed. When I was on the idea, I found THIS. I'm assuming with the age of that discussion, some methods have been updated (getPixmap -> get_pixmap). I tried ppm = pix.getImageData('ppm') but couldn't get it to take.

[JorjMcKie]

Ah ok. But if printing is the only purpose:

1. The logic needed to make a page appearing as it should is unbeatably easy with making pixmaps of course
2. Why does PDF size matter so much then - the output appears to be just a through-away product!
3. You just must be careful with choosing the resolution
4. You seem to have a thorough dislike to read documentation (😉): there is an extra chapter / section dealing with the major renaming effort two years ago.
5. This jpeg = pix.pil_tobytes("JPEG", ... more pillow parameters ...) will create bytes in JPEG image format using PIL / Pillow internally. The pillow documentation will also have hints on how to optimize / minimize the resulting binary. Using "ppm" output is about the largest (although fastest) alternative.

[sgthuth]

Well as much as the throw away concept should apply, size is a major consideration. The secondary application that converts PDFs is also concatenating them in a single file. It could be just a few PDF files that range from 1 to 5 pages each. It could be upwards 5000 PDF files but on average, it's about 500 PDF files at a time. I experimented with the image concept with our considering compression just to see. The resulting culmination of PDFs was about 40 times the original size and the converted format was about 10 times. I'd be happy if I can just get it under 5 times. I like to read and learn. Just a matter of finding the right pieces to read and tie them together to create the best solution. One day I'll read through all the documentation. I'd like to build something similar to manipulate AFP files which is the resulting format in this process that I have to contend with. Needless to say about a month ago, I'd only used Python in lambda functions for AWS and had never thought about the structure of a PDF much-less heard of PyMuPDF. You guys have done an amazing job... and yes, it has been well documented. I guess I'll get to reading.

[JorjMcKie]

Well the documentation admittedly is a huge book in v1.20.x already: 420 Letter pages.
But the packages on the other hand is a large one too.

Just looking at an arbitrary PDF page with text and one image:

len(page.get_pixmap().tobytes())
202299
len(page.get_pixmap().pil_tobytes("jpeg",optimize=True))
81306

[sgthuth]

Oh wow. That is very helpful! Thank you @JorjMcKie . I'll come back over here and post what I've done but that's plenty to go in that direction. I'll read up, too! Thanks for all the help once again.

[sgthuth]

Alright... Either my brain is fried or I'm just having a bad day...
I've gotten 95% of my code written out but I can't recall or find (again) how to append the current page from my template with no alterations 'as is' to my final pdf. I think my brain is just not working. A bit of my code is below.

with fitz.open(path_and_filename_of_original_pdf) as original_pdf:
   final_pdf = fitz.open()
   for p, page in enumerate(original_pdf, 0):
      rdt = [] # list of redactions
      adt = [] # list of additions
      blocks = page.get_text('blocks')
      blocks.sort(key=lambda block: block[1])
      for b in blocks:
         if b[4].contains(x):
            rdt.append(b[4].strip())
            if c:
               cpl = True # Cripple PDF
         if b[4].contains(y):
            adt.append(fitz.Point(b[0], b[1]), b[4].strip())
      for r in rdt:
         draft = page.search_for(r)
         for fnd in draft:
            page.add_redact_annot(fnd)
            page.apply_redactions()
            page.apply_redactions(images=fitz.PDF_REDACT_IMAGE_NONE)
      if cpl:
         if len(blocks) > 0:
            pix = page.get_pixmap(matrix=fitz.Matrix(4.16, 4.16))
            jpeg = pix.pil_tobytes("JPEG", optimize=True, mode="L")
            imgPDF = fitz.open("pdf", jpeg)
            imgPage = final_pdf.new_page(width=rect.width, height=rect.height)
            imgPage.show_pdf_page(rect, imgPDF, 0)
         else:
            final_pdf.new_page(width=rect.width, height=rect.height)
      else:
         # PASS PAGE AS-IS TO FINAL PDF
      for a in adt:
         page.insert_text(a[0], a[1], color=(), fontsize=7, fontname="Times-Roman")
   final_pdf.save(path_and_filename_of_final_pdf, garbage=4, deflate=True, clean=True)

A little hint to pass the current state of a page to the final pdf?

[sgthuth]

If iterating over the pages in the template and making changes, will the following capture the page's current state?

final_pdf.insert_pdf(template_pdf, from_page=p, to_page=p)

[sgthuth]

Was just trying to experiment and see if the insertion would work but I can't get passed...

imgPDF = fitz.open("pdf", jpeg)

Exception Thrown:
cannot open broken document

[sgthuth]

Got it passed imgPDF... had to change a bit...

# Cripple Page
pix = page.get_pixmap(matrix=fitz.Matrix(4.16, 4.16))
jpeg = pix.pil_tobytes(format='jpeg', optimize=True, mode='L')
newPage = final_pdf.new_page(width=rect.width, height=rect.height)
newPage.insert_image(rect, stream=jpeg, overlay=False)

is mode "L" not grayscale for pillow? Somehow my resulting PDF has been updated to images but it's still in color. Lol.

Moving on to processing a PDF that doesn't need crippling to determine if insertion will work or not to pass page 'as-is'

[JorjMcKie]

you must use colorspace gray already when making the pixmap: pix=page.get_pixmap(colorspace=fitz.csGRAY, ...).

[sgthuth]

Thanks @JorjMcKie!

I got it late last night. All things cripple are working as anticipated. Only issue now is when crippling is not required... I'm going to try the insert_pdf method. Hoping there's a simple way to appending the current page and it's current state within the original pdf's page iteration to the end of the final pdf.

final_pdf.insert_pdf(original_pdf, from_page=p, to_page=p)