increased file size when using apply_redactions()

[lbr991]

I am getting a huge increase in filesize and have pinpointed to just one line (!):
pdf_page.apply_redactions()
If I save the file without calling apply_redactions() the size is 55KB, if I do the size is 800KB.

Why is this happening? It only draws 4 small black rectangles.

[JorjMcKie]

What were the save options you used?

[lbr991]

Just doc.save(output)

Regarding the annotation:

    rect = fitz.Rect(
        bbox['Left'] * page.rect.width, 
        bbox['Top'] * page.rect.height, 
        (bbox['Left'] + bbox['Width']) * page.rect.width, 
        (bbox['Top'] + bbox['Height']) * page.rect.height
    )
    annot = page.add_redact_annot(rect)
    annot.set_colors(stroke=(0, 0, 0), fill=(0, 0, 0))
    annot.set_opacity(1)
    annot.update()

[JorjMcKie]

Just doc.save(output)

Well, this will not physically delete stuff that you only think you have deleted. All old objects are still there as "ghosts" and could be accessed by hackers.
Use at least "garbage=3, deflate=True", or equivalently doc.ez_save(...).

[lbr991]

Thanks, it works!
Does doc.save(output, garbage=3, deflate=True) have the same effect as doc.ez_save(...)?
Also why do you mention "at least"? Do you recommend to do anything more for redactions?

[JorjMcKie]

At least means no less than garbage=3. There are (rare) occasions, where garbage=4 compresses even better than 3: it scans binary streams too to discover equality. Such a situation should however not be caused by applying redactions.
doc.ez_save() (pronounce as "easy save") is just an abbreviation for the most commonly used save with reasonable compression - meaning a good balance between compression result and processing required to do it.

[lbr991]

In Adobe when the user redacts a pdf, it shows:

https://www.adobe.com/acrobat/resources/how-to-redact-a-pdf.html

What is "hidden data and metadata" in this case?

Is hidden data the "ghosts" handled by garbage=3 and metadata is title, author, subject etc?
Are there any other measures I can take for redacting data?

[JorjMcKie]

Is hidden data the "ghosts" handled by garbage=3 and metadata is title, author, subject etc?

Yes, it means all the stuff removed by garbage collection in PyMuPDF.

Are there any other measures I can take for redacting data?

Normally, there are two different organizational units that (1) enter the redaction amd (2) apply it.
Obviously, the second one could decide to reject the redaction. Or change how intersecting images should be handled. Rejecting it altogether will restore earlier data for example.
I can't image what sense it could make to not remove the ghosts.

[lbr991]

Rejecting a redaction, must be an organizational process or a capability offered to the rejector, such that he can simply delete the redact annot. If he does so, the PDF will be unchanged, and no black rectanle drawn.

Not sure what you mean, I thought "organizational process" was something defined in the PDF format?

[JorjMcKie]

Adding a redact annot means suggesting a change.
Applying it means accepting it.
Deleting a redact annot means rejecting the change.
That's all.

[lbr991]

Just to confirm, if I add a redact annot and apply it, then use garbage=3, a malicious user would not be able to delete the redact annotation from output.pdf, correct?

[JorjMcKie]

Not quite:
After applying the readctions, their changes will be permanent part of the PDF and cannot be reverted (if not saving incrementally).
Reasonable garbage collection will recover file space by deleting objects that have become ghosts during the process. This will also prevent to access those previous data - which is important, because one of redaction motivations is data protection.
The redaction annotations themselves however will be deleted.

[JorjMcKie]

Just to confirm, if I add a redact annot and apply it, then use garbage=3, a malicious user would not be able to delete the redact annotation from output.pdf, correct?

When you execute page.apply_redactions() then the redacted data and all redaction annotations are removed.
Logically at first. Physically also if saving with a sufficient garbage option.
Physically also means that nothing in the world can recover things removed by the redaction - as well as the redaction annot itself.

[lbr991]

Thanks for that clarification but meant this earlier question:

Can you list an example case where a duplicate stream can exist? The way I understand this (please correct me if I'm wrong) is that if I'm prioritizing "safety" (confidence that the data has been redacted) over speed I should opt for garbage=4?

[JorjMcKie]

E.g. some careless PDF creator may have inserted the same image twice under different names and different xrefs. This does not happen under the control of the redaction algorithm.
But of course: who knows where that PDF has come from, it may have contained grease before you started dealing with it.

[luchux]

@JorjMcKie Would write(garbage=4, deflate=True) be expected to also clear redactions and release memory?

[luchux]

with those options and ez_save, or save and those options, I am still seeing a significantly increase in size (from 4mb to 21mb) when adding 10 redaction annotation and applying them.

[JorjMcKie]

As part of the redaction process, images will be changed that intersect a redact rectangle - if the default option images=fitz.PDF_REDACT_IMAGE_PIXELS is used. These changes lead to converting affected images to PNG format ... but only for the page where redactions are being processed!
Some implications are:

• Previous other image formats are converted to PNG format. Depending on the previous format, a larger image size will result.
• If the same image is used on multiple pages, then these other pages will still use their original image. So file size will increase b/o of an additional image.

Example
A PDF with 4 pages all showing the same JPG image shows this initially:

for page in doc:
    print(page.get_images())

[(5, 0, 439, 501, 8, 'DeviceRGB', '', 'fzImg0', 'DCTDecode')]  # all the same JPG image
[(5, 0, 439, 501, 8, 'DeviceRGB', '', 'fzImg0', 'DCTDecode')]
[(5, 0, 439, 501, 8, 'DeviceRGB', '', 'fzImg0', 'DCTDecode')]
[(5, 0, 439, 501, 8, 'DeviceRGB', '', 'fzImg0', 'DCTDecode')]

After applying a redaction on page 0 (intersecting image at xref 5):

for page in doc:
    print(page.get_images())

[(12, 0, 439, 501, 8, 'DeviceRGB', '', 'Im1', 'FlateDecode')]  # PNG image
[(5, 0, 439, 501, 8, 'DeviceRGB', '', 'fzImg0', 'DCTDecode')]  # same JPG image for the other pages
[(5, 0, 439, 501, 8, 'DeviceRGB', '', 'fzImg0', 'DCTDecode')]
[(5, 0, 439, 501, 8, 'DeviceRGB', '', 'fzImg0', 'DCTDecode')]

So in spite of using best compression, the image at xref 12 is (1) larger (PNG) than the JPG original, and (2) comes on top in terms of size.