Implementation of get_contents

[j-t-1]

pypdf/pypdf/_page.py

Lines 964 to 983 in f5c5747

	def get_contents(self) -> Optional[ContentStream]:
	"""
	Access the page contents.
	Returns:
	The ``/Contents`` object, or ``None`` if it does not exist.
	``/Contents`` is optional, as described in §7.7.3.3 of the PDF Reference.
	"""
	if PG.CONTENTS in self:
	try:
	pdf = cast(IndirectObject, self.indirect_reference).pdf
	except AttributeError:
	pdf = None
	obj = self[PG.CONTENTS]
	if is_null_or_none(obj):
	return None
	resolved_object = obj.get_object()
	return ContentStream(resolved_object, pdf)
	return None

Are there advantages to the above implementation? Why not do similar to _get_contents_as_bytes (below), but join as a string and cast to a ContentStream? Further, only this handles when the content stream is an array?

pypdf/pypdf/_page.py

Lines 949 to 962 in f5c5747

	def _get_contents_as_bytes(self) -> Optional[bytes]:
	"""
	Return the page contents as bytes.
	Returns:
	The ``/Contents`` object as bytes, or ``None`` if it doesn't exist.
	"""
	if PG.CONTENTS in self:
	obj = self[PG.CONTENTS].get_object()
	if isinstance(obj, list):
	return b"".join(x.get_object().get_data() for x in obj)
	return cast(EncodedStreamObject, obj).get_data()
	return None

[stefan6419846]

I always suggest to use Git blaming to identify why these changes were done and dig through the code.

In this specific case, _get_contents_as_bytes has been introduced in 94f23f9 to speed up inline image extraction. It is an internal method only.

The regular public method is get_contents. Here the array handling is moved to the constructor of the ContentStream class.

Section 7.7.3.3 (table 31) indicates that our list handing might be wrong:

The value shall be either a single stream or an array of streams. If the value is an array, the effect shall be as if all of the streams in the array were concatenated with at least one white-space character added between the streams’ data, in order, to form a single stream.

[j-t-1]

Yes get_contents will need slightly amending. Will it need the get_data method, as in _get_contents_as_bytes?

Is the design decision for ContentStream having a pdf argument so that we can tell if it is part of a PDF or not (if the argument is None) and also to tell which PDF it is part of if pypdf is dealing with more than one pdf (say when merging)?

[stefan6419846]

get_data() is accessing the (decoded) byte data of the corresponding content stream, thus this only relevant when reading from the original PDF file.

I have not been involved in the design of the class, but skimming through the usages of self.pdf in ContentStream seems to indicate that this is used to avoid cloning if the content streams are both targeting the same PDF reference.

[Apfelkuchenbemme]

This might be a bit of a weird question, but what's the point of the get_contents method, exactly? I'm looking into how to extract the layout from PDF files, read the explanation on why text extraction is hard in the documentation, figured this part with the missing semantic layer explains it quite well, and decided to try other libs.

Just for the hell of it, I poked through the API reference anyways, tried PageObject.get_contents on the title page of a test PDF to see if I could extract a rectangle, but lo and behold, it returned an empty dictionary. Next thing I now, I ran a thousand PDFs through pypdf and literally EVERY SINGLE ONE of the 23something thousand pages returned an empty dictionary for get_contents(). Well okay, aside from one page that actually didn't contain a "content entry", because the PDF had been created with LaTeX and that page was the result of a \cleardoublepage command, sooo "it actually was an empty page", I guess. Even the page of a PDF I just created like this returned an empty dictionary for get_contents():

• open LibreOffice Writer
• don't add anything, just print the thing as .pdf

I'm not going to upload a thousand PDFs, of course, but this example PDF from the docs should be enough to reproduce this. Just tested it with python 3.10.0, Windows 10 and pypdf 5.7.0, all pages of this PDF return an empty dictionary for get_contents().

eh, might aswell post the code I used, while I'm at it

import pypdf
reader = pypdf.PdfReader("GeoBase_NHNC1_Data_Model_UML_EN.pdf")
for page in reader.pages:
    print(page.get_contents())

I looked through the issues before posting this and also found this PR #1847 which supposedly fixes the issue with get_contents() not returning a datastream. Which, looking at the thousand PDFs I just tried, I'm inclined to argue still doesn't work.

[stefan6419846]

Simply printing PageObject.get_contents() always looks like there is nothing to see. Have you tried running PageObject.get_contents().get_data() instead? This should usually return you the plain bytes content of the page.

[Apfelkuchenbemme]

Ahhhh, sorry, no I didn't. I ended up testing reader.pages[0]["/Contents"][0].get_object().get_data(), which returns the same thing as reader.pages[0]._get_contents_as_bytes(), after I saw the page object still had a "/Contents" key. reader.pages[0].get_contents().get_data() returns almost the same thing, albeit with a b"\n" appended at the end.

Because I scrolled right past the note at the top here, which mentions using page.get_contents().get_data(), I missed that method at first. However, part of the reason I missed this note, is that it doesn't show up in the search docs function when you look for "get_contents".

I poked around a bit more and found that with PDFs from arXiv, e.g. this one, reader.pages[0]["/Contents"] returns an ArrayObject with two indirect objects, while reader.pages[1]["/Contents"] returns a single stream, that looks like this when printed: {'/Filter': '/FlateDecode'}. The reason is that on the first page, there's a vertical watermark of sorts on the left, which is the 2nd element of the ArrayObject; the first object is the "normal content" of the page, which is the only object for the other pages, so they only have a single stream.

The fact that reader.pages[1]["/Contents"] prints {'/Filter': '/FlateDecode'} is also the reason I thought page.get_contents() has no data, since it prints an empty dictionary.

If I wanted to do something with that vertical watermark, I could identify it using reader.pages[0]["/Contents"][1].get_object().get_data() because it's tm_matrix starts with 0 1 -1 0, i.e. it's rotated by 90° counterclockwise. Hooowever page.get_contents().get_data() returns a single bytes array of like 11k characters, instead of several objects. Is this a result of §7.7.3.3 of the PDF Reference, which states: "If the value is an array, the effect shall be as if all of the streams in the array were concatenated, in order, to form a single stream." ?

Is this also why reader.pages[0].extract_text(rotations=(90)) returns the entire text of the page from arXiv, instead of just the watermark on the left, because the entire text is read as a single stream?

[stefan6419846]

Yes, this might be confusing and I have fallen for it as well in the past. The search is generated automatically by Sphinx/RTD, thus we do not really have any influence on this.

Depending on your use-case, you might indeed want to try different methods like accessing the dictionary entries of the PDF manually instead of relying on the processed variant. Just avoid _get_contents_as_bytes() and other internal functionality which might break without further notice unless you are sure what you are doing and do not have problems adapting your code if needed.

Regarding the joining: Yes, this most likely is the case (I have not implemented it, but it makes sense). As you mentioned, all array elements should be considered as one big blob/stream. For extracting the text (and other features of interest), the whole page content stream is parsed into its single operations accordingly and processed as required. The rotation handling might be a bug you are of course invited to investigate and propose a PR for, but I cannot definitely argue about this without digging deeper myself as I have honestly never used the rotation support, while extracting the text correctly is complex in theory and in the code as well.