Repeated word out of place ruins text

[brucenielson]

Karl Popper A World of Propensities

I'm using version 1.24.13.

Attached is a pdf I'm trying to load as markdown. The first page reads as follows:

I shall begin with some personal memories...

It
was 54 years ago, in Prague in August 1934, that I first
attended an International Congress of Philosophy. I found it
uninspiring. But the Congress was preceded by another meeting
in Prague, organized by Otto Neurath, who had kindly invited
me to attend a 'Preliminary Conference' ('Vorkonferenz' as he
called it) which he organized on behalf of the Vienna Circle.

I came to Prague with the corrected page proofs of my book,
It
Logik der Forschung. was published three months later...essentially an Aristotelian theory at which, it appears,
Tarski and Godel arrived, independently at almost the same
It
time. was first published by Tarski in 1930, whereupon
It
Godel, of course, accepted Tarski's priority.

Note the weird newline after the first "It" and then that word "It" becomes an unwanted artifact that breaks up the text after that several times. This problem repeats on the next page with some other word.

I tried loading the same PDF using PyPDF and the problem goes away. (But obviously PyPDF isn't trying to convert to markdown.)

See: https://bugs.ghostscript.com/show_bug.cgi?id=708129

I can get you the file I used if desired.

[brucenielson]

Try this link:

[JorjMcKie]

I have no time to fight my way through that website. I don't see an obvious link to the PDF in questions.
Do me a favor and download (parts of) the file.

[brucenielson]

[JorjMcKie]

Thanks for the file.
Cannot reproduce your problem however:
My output looks quite good given the fact that this is an OCR-ed book with its usual share of misreading ...
Uploading test.zip…

[brucenielson]

Do you have the resulting file I can look at? It surely doesn't work well for me. And a regular PyPDF extract works perfectly.

[JorjMcKie]

Here you are ... test.zip

[brucenielson]

I copied your code and I still get the issue.

I'm on windows with these versions:

PyMuPDF==1.24.13
pymupdf4llm==0.0.17

Is it possibly a version or environment related issue?

[brucenielson]

I have confirmed that if I get the text page by page out of pymupdf directly there is no issue:

Ladies and Gentlemen,
I shall begin with some personal memories and a personal
confession of faith, and only then turn to the topic of my
lecture.
It was 54 years ago, in Prague in August 1934, that I first
attended an International Congress of Philosophy. I found it
uninspiring. But the Congress was preceded by another meeting
in Prague, organized by Otto Neurath, who had kindly invited
me to attend a 'Preliminary Conference' ('Vorkonferenz' as he
called it) which he organized on behalf of the Vienna Circle.
I came to Prague with the corrected page proofs of my book,
Logik der Forschung. It was published three months later in
Vienna, and in English 25 years later as The Logic of Scientific
Discovery. In Prague it was read by two Polish philosophers,
Alfred Tarski and Janina Hosiasson-Lindenbaum, the wife of
Tarski's friend and collaborator, Adolf Lindenbaum. Janina
Hosiasson and her husband were murdered when, 5 years later,
the Nazis invaded Poland and systematically exterminated what
they described as its 'Fuhrerschicht': its 'intellectual elite'. Tarski
went from Prague to Vienna where he stayed for a year and
where we became friends. Philosophically, it was the most
important friendship of my life. For I learnt from Tarski the
logical defensibility and the power of absolute and objective
truth: essentially an Aristotelian theory at which, it appears,
Tarski and Godel arrived, independently at almost the same
time. It was first published by Tarski in 1930, whereupon
Godel, of course, accepted Tarski's priority. It is a theory of
objective truth - truth as the correspondence of a statement
with the facts - and of absolute truth: if an unambiguously
formulated statement is true in one language, then any correct
translation of it into any other language is also true. This
theory is the great bulwark against relativism and
3

[brucenielson]

My code:

import pymupdf4llm, pymupdf
import pathlib

doc = pymupdf.open(r"D:\Documents\AI\BookSearchArchive\documents\A World of Propensities by Karl Popper (1997).pdf")
for page_num in range(len(doc)):
    page = doc.load_page(page_num)
    text = page.get_text("text")
    print(text)
md = pymupdf4llm.to_markdown(doc)
path = pathlib.Path("test.txt")
path.write_text(md, encoding="utf-8")

But the to_markdown call introduces the problem.

To be clear this call has no bug:

doc = pymupdf.open(r"D:\Documents\AI\BookSearchArchive\documents\A World of Propensities by Karl Popper (1997).pdf")

This call creates the problem of the repeated 'It':
md = pymupdf4llm.to_markdown(doc)

[JorjMcKie]

Then this issue does not belong in this repository. It is not even a bug probably because there is no provision in standard get_text() to look at page layout and other complications.

[brucenielson]

I don't follow why you are saying it is not a bug. Clearly it doesn't work as intended for some reason. Or rather it does for you for your environment and version and does not for me. Even though we're using the same code.

It must be a bug in pymupdf4llm's to_markdown call. I traced it to 'write_text'.

I take it pymupdf4llm is not part of this repo? What is the correct repo? Pymupdf/RAG is the repo the specifically mentions pymupdf4llm.

[JorjMcKie]

I tried it again and still cannot reproduce the problem. Here is my result original.zip.
Environment Windows 3.11, Python 3.9, pymupdf 1.24.13, pymupdf4llm 0.0.17.

[brucenielson]

Here is my result with the same code you are using.