tabs returned as linefeeds by page.get_text()

[Jacek11]

Please provide all mandatory information!

Describe the bug (mandatory)

PDF downloaded from EDGAR. The page.get_text() method is treating tabs as line feeds, causing linefeeds between the currency symbol and amount, for example.

To Reproduce (mandatory)

f = fitz.open(pdf_path)
for page in f:
page_text = page.get_text()

The returned text has many extra '\n's.
pypdf reads the doc correctly.

Expected behavior (optional)

Describe what you expected to happen (if not obvious).
I expected to see spaces instead of \n

Screenshots (optional)

If applicable, add screenshots to help explain your problem.

Your configuration (mandatory)

3.10.11 (tags/v3.10.11:7d4cc5a, Apr 5 2023, 00:38:17) [MSC v.1929 64 bit (AMD64)]
win32

PyMuPDF 1.23.4: Python bindings for the MuPDF 1.23.2 library.
Version date: 2023-09-26 00:00:01.
Built for Python 3.10 on win32 (64-bit).

Additional context (optional)

Add any other context about the problem here.
sonos_q2_2023_10q.pdf

[JorjMcKie]

The document has 60 pages - please pick an example page.

[Jacek11]

Happens with any page with tables. The 29th page f[28] for example.

[JorjMcKie]

As a matter of fact, the page contains no tab characters (\t) at all - neither \n (line breaks). All of these are generated by the respective extractor software.
For example, between the "$" and the amount here

there is nothing - no tab and no line break.

MuPDF works as designed. Its algorithm that generates the block -> line -> span hierarchy is based on various influencing factors like font size and inter-character distances. A decision is made as to whether text pieces ("spans") belong to the same line or not.
If a distance is larger than some threshold, a new linbe is started - independent from whether also the bottom coordinate changed. In this process, tabulator characters \t are never generated.
This is a design decision.

What causes your irritation happens when tables are present as you noticed.
Extracting plain (naive) text from a page that contains tables does not make much sense in most cases anyway. An adequate solution would be to identify and extract tables separately from other text.
PyMuPDF allows you to do that.

You can also extract the page's word strings, sort them accordingly and then re-arrange them line-wise. Something like that:

words = page.get_text("words", sort=True)
y = words[0][3]  # bottom of first word
text = words[0][4]  # text of first word
for w in words[1:]:  # walk through remaining words
    if w[3] == y:  # if same line
        text += " " + w[4]  # append word text
    else:  # line has changed
        print(text)  # print the finished line
        text=w[4]  # start a new line with current word
        y=w[3]  # and its bottom y
print(text)  # remaining word

The first few lines that this snippet will deliver look like so:

Table of contents
Comparison of the six months ended April 1, 2023, and April 2, 2022
Six Months Ended Change
April 1, April 2,
2023 2022 $ %
(In thousands, except percentages)
Sonos speakers $ 780,377 $ 819,620 $ (39,243 ) (4.8 )%
Sonos system products 158,525 195,965 (37,440 ) (19.1 )
...

[Jacek11]

Thanks for the reply. I noticed that table detection was added in a recent release but haven't tried it out yet.

Is there another export type that would preserve the layout better?

[JorjMcKie]

Is there another export type that would preserve the layout better?

Did you see my code snippet at the end of my post? Might be a decent approximation.
But re-iterating: if tables are present, then don't try plain text extraction.

BTW there also exists layout-preserving text extraction via the "fitz as a module".

Going to move this issue to the "Discussions" tab now.

[Jacek11]

HTML export also has the phantom line breaks. Maybe that's the expected behavior as well. I think that I understand why linefeeds may be warranted in some cases when the text is on the same y coordinate; in cases of multi columns text, as an example.

When extra linefeeds are not inserted, some LLMs can accurately understand tables from plain text. With extra linefeeds, they're far worse at it.

Since the library can already detect tables, any chance that you could include an option in a future release to treat tables differently in the get_text() call?