Can I force mapping of pages to lines to be the same using get_text("words") and get_text("rawdict")?

[maspotts]

Hi: I've noticed that the breakdown of a document into lines (mapping of page to an array of lines) is slightly different when I use get_text("words") vs get_text("rawdict"): in my current document I see the first several 100 lines are identical, but further down the document they get out of sync. Is there any way to force the line breakdown to be the same irrespective of whether I specify "words" or "rawdict"? (The reason is: I've created a number of blacklists of line numbers (using get_text("words")) which I now want to exclude from processing: this worked fine when the processing also used get_text("words"), but now I've had to switch to get_text("rawdict") (to get access to span-level metadata), and so as it stands my line-number based blacklists are suddenly usable because they (eventually) don't match the lines coming out of get_text("rawdict")). (I realize that going forward I should use "rawdict" to extract line numbers to create new blacklists, but I don't want to have to re-create my existing blacklists (there are a lot of them!)).

Thanks!

Mike

[JorjMcKie]

There actually should be no deviation between the line number within a word item (x0, y0, x1, y1, "word", block_no, line_no, word_no) and the line within which the word's span occurs.
I would like to see an example where this happens.

[JorjMcKie]

For the testfile you gave me, there is no such deviation.

[JorjMcKie]

Aha! Found an error: the "words" in this context are defined by strings surrounded by spaces. If a line contains only spaces, then I obviousl do not increase the line count ...
Let me confirm ...

[JorjMcKie]

Okay found a few minor issues:

1. My program to test line number congruence had an error and issued false alarms.
2. The words extractor currently ignores the second / third components of ligatures such that "specifed" is produced instead of "specified" in cases where the "fi" ligature is being used.
3. The words extractor currently may output word bboxes that are empty.

After fixing these issues things are working as expected ...

[maspotts]

Thanks very much!! Are these fixes isolated enough that I could apply them myself now, by creating local copies of the relevant functions? If not, any idea when the next release might be due?

[JorjMcKie]

The changes are in both, Python and C code. So you will need a new wheel. I can send you a prelim one when I am done. What was your platform again? Holen Sie sich Outlook für Android<https://aka.ms/AAb9ysg>

…

________________________________ From: maspotts ***@***.***> Sent: Saturday, November 6, 2021 10:40:30 AM To: pymupdf/PyMuPDF ***@***.***> Cc: Jorj X. McKie ***@***.***>; Comment ***@***.***> Subject: Re: [pymupdf/PyMuPDF] Can I force mapping of pages to lines to be the same using get_text("words") and get_text("rawdict")? (Discussion #1375) Thanks very much!! Are these fixes isolated enough that I could apply them myself now, by creating local copies of the relevant functions? If not, any idea when the next release might be due? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#1375 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AB7IDIS5AKE2LRJAEZPX2MLUKVD65ANCNFSM5HOKSOUA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[maspotts]

That would be great: thanks!

print(sys.version, "\n", sys.platform, "\n", fitz.doc)
3.9.5 (default, Jun 24 2021, 15:50:05)
[Clang 12.0.0 (clang-1200.0.32.29)]
darwin

PyMuPDF 1.19.1: Python bindings for the MuPDF 1.19.0 library.
Version date: 2021-10-23 00:00:01.
Built for Python 3.9 on darwin (64-bit).

[JorjMcKie]

The wheel is here. Look at "Artifacts" at the bottom. There is a ZIP file whih should contain your wheel.

[maspotts]

Thanks: got it. Verified bug #1364 is fixed (thanks!).

I (embarrassedly) also noticed some remaining differences due to my having applied strip() to the output of get_text() (copied from an example!): apparently using strip() causes line breaks that cross a page boundary to be lost, so I was getting the last line of each page appended to first line of its successor. eg: without strip() I get these two lines (last line of page 2 and first line of page 3 from the paper attached to issue #1364):

American Journal of Therapeutics (2021) 28(4)
for settings with limited resources.24 If demonstrated

and with strip() I get:

American Journal of Therapeutics (2021) 28(4)for settings with limited resources.24 If demonstrated

and similarly for every page transition. Not sure why strip() is having this effect (my understanding is that it should not strip newlines or linefeeds). I guess this is not your problem, but if the cause is obvious then I'll be curious to know what is is! Here's the output of 'od -c' (with line numbers prepended by own code):

With strip():

0022340    c   s   .   c   o   m  \n   1   4   2   :       A   m   e   r
0022360    i   c   a   n       J   o   u   r   n   a   l       o   f    
0022400    T   h   e   r   a   p   e   u   t   i   c   s       (   2   0
0022420    2   1   )       2   8   (   4   )   f   o   r       s   e   t
0022440    t   i   n   g   s       w   i   t   h       l   i   m   i   t
0022460    e   d       r   e   s   o   u   r   c   e   s   .   2   4    
0022500    I   f       d   e   m   o   n   s   t   r   a   t   e   d  \n

Without strip():

0022340    a   p   e   u   t   i   c   s   .   c   o   m  \n   1   4   3
0022360    :       A   m   e   r   i   c   a   n       J   o   u   r   n
0022400    a   l       o   f       T   h   e   r   a   p   e   u   t   i
0022420    c   s       (   2   0   2   1   )       2   8   (   4   )  \n
0022440    1   4   4   :       f   o   r       s   e   t   t   i   n   g
0022460    s       w   i   t   h       l   i   m   i   t   e   d       r
0022500    e   s   o   u   r   c   e   s   .   2   4       I   f       d
0022520    e   m   o   n   s   t   r   a   t   e   d  \n   1   4   5   :

So it seems pretty clear that strip() is removing the intervening newline in the former case. Happily this means that I can still use my pre-determined line-number blacklists by accounting for the merged lines across each page boundary, so please feel free to close this discussion! Thanks awfully for chasing this down despite my massive red herring (and for the other fixes!).

Mike

[JorjMcKie]

The strip() method removes white space if no (None) argument. This includes newline stuff and more:

>>> "\nHello\t".strip()
'Hello'
>>> 

[maspotts]

Yup, that's my mistake! Luckily I can (with a little effort) offset my line-number lists to account for that (and remove it from any further experiments!). Thanks again!