Detect Weird texts which is not properly extractable

[Soumadip-Saha]

Hi @JorjMcKie. I am having problems in identifying texts which are not properly extractable. Here is a sample of the PDF and I have 1000s of similar type of PDFs. Some of them have proper text which can be extracted and some of them are not. I have tried to extract the text using this method

doc = fitz.open("FR-2019-03.pdf")
page = doc[0]
text = page.get_text(sort=True)
print(text)

and the output was this

�������������	
����
������������
��������������� �!"#�$!#%�%
�
�' ()	
��*�+,,�,,-����.�/0	
��*�+��10��,�	���
,�.�3344�567)	
���8�9�
)�,,�)�0�;�2
<�������=������>�??�<�@�A����=���
&+B5C�D��(DE&CE&7
FG4
CHC(I&�8C�7I::+�JGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGK
KG4
DBLC(&�8CGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGK
MG4
7(D2CGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGM
3G4
NC��E�&�DE7GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGM
OG4
2�D(C77��5DP�N�+9�+:GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGO
QG4
(��&�(+5�&J�C8+5I+&�DE�D��/I+5�&J�+&&��BI&C7GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGQ
RG4
2�D(C77�NC7(��2&�DEGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGFQ
SG4
(DE(5I7�DEGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGQ3
TG4
�C�C�CE(C7GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGQO

For other documents this process is working to extract the texts and I would like to skip the pages which has similar kind of garbage output based on a logic. Can you please help me with that? I have attached two sample PDFs which, one garbage PDF and the other Proper PDF. I am a newbie here so it would help me a lot. Thanks a lot in advance.

[JorjMcKie]

Your garbage example indeed is not made at all for text extraction.
There low-level technical ways to predict the chances of a successful extraction, but probably not worth considering here.
You could simply check whether there are many "invalid Unicode" characters (chr(0xfffd) - �) in the extraction.
The first page gives for instance this:

text=page.get_text()
text.count(chr(0xfffd))
53

You could of course try to OCR the page in such cases, but I noticed the watermark across the page, so these results may be unpleasant as well. Example page:

tp=page.get_textpage_ocr(dpi=150, full=True)
        
print(page.get_text(textpage=tp))
        
Independent Final Report
Page1 of 66
Process Parameters of 4400 L-Scale IVIG
Manufacturing Process
TABLE OF CONTENTS
1.0
EXECUTIVE SUMMARY.........cccscessssssrsssssesesesssssesnsesesseeesnseseansetesnsnsessessanscaeacsettansneceeasessearanaeeeensel,
5.0
PROCESS FLOW DIAGRAM........c.sssssssssssssssssesssssessenenssessersserersceransnensasssensesransttenseseeseasesseenanseeeesseD)
6.0
CRITICALITY EVALUATION OF QUALITY ATTRIBUTEG........scssssssssssssrsesssesesessssseensesenseeterseeeeetsO
7.0
PROCESS DESCRIPTION. ......s.scsssssssssssssessseresssssessessessesserseererscersnssensasesenseeransettasseteentesesseereraceeeeTO