Using custom decoding for NameObject

[rassie]

What would be the best way to get a raw representation of a NameObject? I'm seeing PDFs with NameObjects encoded in MacRoman instead of UTF-8 on regular basis and would like to have a bit of flexibility in how to decode them. Currently, pypdf tries to decode UTF-8 and failing that falls back to gbk, i.e. unified Chinese, which is not what I want or expect, however I cannot access the raw string to implement some conversion myself.

[stefan6419846]

Although it is no clean solution, you should probably be able to monkey-patch the corresponding method with custom logic.

How would you expect a clean solution to look like for this?

[pubpub-zz]

Name Objects specification is available in Pdf Reference 1.7 §.3.2.4 p56-58

If you have examples some pdf with Name Objects encoded in MacRoman. please share for analysis

[rassie]

The files are confidential, I'm afraid. The NameObject I'm getting is /strukturpr#8agung (which is supposed to be strukturprägung). I'm well aware that NameObjects are supposed to be encoded in UTF-8 according to specification, but in this case, it's not. The byte string b'strukturpr\x8agung' is not decodable by the utf-8 decoder, which is fine, however, I don't want to get strukturpr奼ung, which is what the fallback gbk is returning.

My whole point is not that I want pypdf to correctly decode wrong data, but that I would like to have some control over it, e.g. retrieve the raw string and decode it myself empirically. Should I try a PR and we'll discuss there?

[pubpub-zz]

no problem to make a proposal

[rassie]

So I've tried to come up with a patch and failed to create something simple, working and non-breaking, but I had time to think about the problem and I now have a different suggestion going forward.

In general, my problem is identical to the problem which prompted the original patch which introduced the gbk encoding as fallback -- the PDF spec only recommends NameObjects to be encoded in UTF-8, but apparently does not require it. Judging by my files and by that pull request, non-UTF-8 encoded strings are somewhat common. As a parser, pypdf has no way of recognizing a different encoding if it doesn't decode as UTF-8, so the gbk fallback is as good or as bad as MacRoman, cp1251, ucs2 or any other encoding, especially since it's basically random whether a particular codec manages to decode a particular string to anything.

I'd argue that there shouldn't be a fallback in the first place. If the name is not decodable by UTF-8, use charmap (as the code does now after the gbk fallback) or return a bytestring (which would probably be a breaking change and more cumbersome to implement). In this case, library user can apply custom logic to convert that charmap coded string (or a bytestring) to something legible. For example, in my case I'd try to find out whether the particular file has been created on a Mac Illustrator and then apply MacRoman to the string, otherwise WinAnsi and in some other cases PDFDoc.

I'd like to propose to remove the gbk fallback completely. This would probably be something for 4.0, even though I'd appreciate it going in earlier.

[rassie]

For the record and for anyone who'll find this: I've fallen back to monkey-patching for now. Would still appreciate a somewhat more correct solution.

import pypdf
from pypdf._utils import StreamType, logger_warning, read_until_regex
from pypdf.errors import PdfReadError
from pypdf.generic import NameObject
from pypdf.generic import decode_pdfdocencoding


def read_from_stream(stream: StreamType, pdf: Any) -> "NameObject":
    name = stream.read(1)
    if name != NameObject.surfix:
        raise PdfReadError("name read error")

    name += read_until_regex(stream, NameObject.delimiter_pattern)
    try:
        # Name objects should represent irregular characters
        # with a '#' followed by the symbol's hex number
        name = NameObject.unnumber(name)
        ret = name.decode("utf-8")
        return NameObject(ret)

    except (UnicodeEncodeError, UnicodeDecodeError) as e:
        if not pdf.strict:
            logger_warning(f"Illegal character in Name Object ({name!r})", __name__)
            return NameObject(name.decode("charmap"))

        msg = f"Illegal character in Name Object ({name!r})"
        raise PdfReadError(msg) from e


pypdf.generic.NameObject.read_from_stream = staticmethod(read_from_stream)