In some case the annotation using saveIncr breaks the signature of the original document

[pulse-mind]

Describe the bug (mandatory)

• I open a pdf, the document contains a signature
• I add a freetext annotation
• I save using incremental save
• The esign is broken when acrobat reader is validating the signature.

To Reproduce (mandatory)

pdf = fitz.open(pdf_output_file_with_path)
        for page in pdf:
            #  Annot prototype.
            r = fitz.Rect(10, 10, 300, 20)
            t1 = u"1a8c9b3c-d3cb-4eeb-bfb6-a225d8553f69"
            annot = page.add_freetext_annot(
                r,
                t1,
                fontsize=10,
                rotate=0,
                text_color=(0, 0, 0),
                align=fitz.TEXT_ALIGN_LEFT,
            )
            annot.update(text_color=(0, 0, 0))

        pdf.saveIncr()

In some other cases (with some others document/signature) it works fine.

Expected behavior (optional)

The signature is not broken, acrobat validate the signature like it validates it on the original document.

Screenshots

Your configuration (mandatory)

• Ubuntu 22.04 same on MACOS
• Python 3.10
• PyMuPDF version 1.21.0 installed using pip

For example, the output of print(sys.version, "\n", sys.platform, "\n", fitz.__doc__) would be sufficient (for the first two bullets).

>>> print(sys.version, "\n", sys.platform, "\n", fitz.__doc__)
3.10.6 (main, Nov  2 2022, 18:53:38) [GCC 11.3.0]
 linux

PyMuPDF 1.21.0: Python bindings for the MuPDF 1.21.0 library.
Version date: 2022-11-08 00:00:01.
Built for Python 3.10 on linux (64-bit).

[JorjMcKie]

As you stated yourself, this situation occurs intermittently.
So we cannot handle this issue without a reproducing file.
Please provide a document where your code fails.

To keep signatures valid, incremental saving is required - but it is no guarantee either.

[pulse-mind]

Hi thank you for your answer. Can I send it on your email define on github?

[JorjMcKie]

Hi thank you for your answer. Can I send it on your email define on github?

Sure, but please be aware that I may have to share the file with colleagues from Artifex.

[pulse-mind]

Yes you can share it.
I sent you the file.

Thanks a lot.

[JorjMcKie]

I have reviewed the file and made a few tests. Here are the findings:
Method doc.saveIncr() works exactly as it should:

• adding an annot, or even a new page, leaves the start of the file intact - on bitlevel!
• verified this by deleting the appended data again with a text editor: Adobe accepts the resulting file
  as being signed, as before.

So, the reason for loosing the signed status must be, that there happens some check whether the file has been changed at all.

I am not sure which part of the signature field specification is responsible for this type of check: /Filter /Adobe.PPKLite?, /SubFilter /ETSI.CAdES.detached?
It definitely is something outside (Py-) MuPDF's responsibility. There is no bug here.

BTW some other viewers (Foxit, Nitro Reader) do not reognize the signer as being validated - in contrast to Adobe, but that may be another problem.

[JorjMcKie]

As per the message ("byte range invalid"):
Indeed the byte range in the signature is [0 37841 56787 773], which includes bytes of the full old file, excluding any data appended b/o incremental changes. The old file length is 56787 + 773 = 57560.

[pulse-mind]

Thanks a lot for the time spent on that. I will have a check with the person in charge of the signature...

[pulse-mind]

There is something strange because I have done the same thing with pdftron library using a test licence and also pdfbox (Java) and the signature was still valid.

[JorjMcKie]

There is something strange because I have done the same thing with pdftron library using a test licence and also pdfbox (Java) and the signature was still valid.

To see what actually has been happening when using incremental save, I compared the before / after versions of your nice example file with a "diff" generating program. This clearly proved that the changes exclusively happened by appending data after the original's final %%EOF\n line - which is what should be happening.

[pulse-mind]

Thank you
I tried to use diff but it says that the files are binary files
Binary files 221113_DOS-POLLOS_Feuille-de-presence.pdf and 221113_DOS-POLLOS_Feuille-de-presence_annot.pdf differ
Are you doing something before to extract something from the files ?

I want to do the same as you and then compare with the results provided by others tools llike pdftron of pdfbox in order may be to give the right information to the signature server provider or to you. I understand what you are saying but I do not understand why with other tools it works fine in Acrobat and not when I use pymupdf.

Best regards,
Fred

[pulse-mind]

I have done a less of the files and copy and past the final part after the %%EOF and I got these files.
1-pymupdf.txt
2-pdfbox.txt
3-pdftron.txt
I do not really know how to manage that :D

[JorjMcKie]

Let us convert this to a "Discussions" item first.
Then please attach the complete PDF of the pdfbox output, and I will compare that with the pymupdf output pdf.
Thanks.

[pulse-mind]

Thanks again.
I sent you an email with the files.

[JorjMcKie]

Sorry to have to ask you again:
Can you please send me one more time a simple example file:

1. original file signed
2. that signed file with an incremental change by PyMuPDF invalidating the sgignature
3. that file with (same) incremental change with maintained validation status made with another tool (pdftron?).

Thank you, and again sorry about that.

[JorjMcKie]

Ah - no, forget about it: found your initial files!
Thanks!

[JorjMcKie]

Hi, good news: the problem has been identified and will be fixed in the first patch version of next year!

[pulse-mind]

Great thank you Le ven. 23 déc. 2022, 19:03, Jorj X. McKie ***@***.***> a écrit :

…

Hi, good news: the problem has been identified and will be fixed in the first patch version of next year! — Reply to this email directly, view it on GitHub <#2068 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFJMVS6JOSI5WPZ6ZSNXVTTWOXSNPANCNFSM6AAAAAASF67EJE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[JorjMcKie]

It was hard to find:
There was no actual error and the Adobe message on wrong /ByteRange was wrong - the byte range has never been touched.

It was easy to fix:
There are two ways how to code the XREF table - in straight ASCII format or compressed as a stream.
When appending incremental changes, the relevant new XREF table piece must not be a stream, if the preceding XREF table was no stream either.
I have never seen this being documented!

[pulse-mind]

Interesting so it is available in the snapshot version? Or the only way is it wait for the release? Le ven. 23 déc. 2022, 20:05, Jorj X. McKie ***@***.***> a écrit :

…

It was hard to find: There was no actual error and the Adobe message on wrong /ByteRange was wrong - the byte range has never been touched. It was easy to fix: There are two ways how to code the XREF table - in straight ASCII format or compressed as a stream. When appending incremental changes, the relevant new XREF table piece must not be a stream, if the preceding XREF table was no stream either. I have never seen this being documented! — Reply to this email directly, view it on GitHub <#2068 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFJMVSYKJO3GWVVFQS3CY6LWOXZYHANCNFSM6AAAAAASF67EJE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[JorjMcKie]

Next release actually, but I can make a 3.10 preliminary wheel for Ubuntu 22.04 if you want to test something.

[pulse-mind]

Hello Thanks As you like but I can test only on macos (apple cpu). There is no urgency. Le sam. 24 déc. 2022, 03:13, Jorj X. McKie ***@***.***> a écrit :

…

Next release actually, but I can make a 3.10 preliminary wheel for Ubuntu 22.04 if you want to test something. — Reply to this email directly, view it on GitHub <#2068 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFJMVS4O33RSK2WNGSUNXPLWOZL4JANCNFSM6AAAAAASF67EJE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[JorjMcKie]

Sorry - have no MAC machine, so you will have to wait for the official version early next year.

[pulse-mind]

Hi, do you know when the new official version will be released? Le sam. 24 déc. 2022 à 11:30, Jorj X. McKie ***@***.***> a écrit :

…

Sorry - have no MAC machine, so you will have to wait for the official version early next year. — Reply to this email directly, view it on GitHub <#2068 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFJMVSZ7YHNCYUL4RQVET5TWO3GFXANCNFSM6AAAAAASF67EJE> . You are receiving this because you authored the thread.Message ID: ***@***.***>