No whitespace in text?

[JBGruber]

I tried to parse a few complex PDFs, which worked really well. Now I put in a simpler one and was suprised to see that the result contains no whitespace. Not sure if I'm doing something wrong or if this might be a bug:

import openparse
import urllib.request
urllib.request.urlretrieve("https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0207996&type=printable", "test.pdf")

basic_doc_path = "test.pdf"
parser = openparse.DocumentParser()
parsed_basic_doc = parser.parse(basic_doc_path)

print(parsed_basic_doc.nodes[2].text)
#> Abstract<br><br>a1111111111
#> a1111111111
#> a1111111111
#> a1111111111
#> a1111111111<br><br>**Introduction**<br><br>Exploitinginformationinhealth-relatedsocialmediaservicesisofgreatinterestforpatients,
#> researchersandmedicalcompanies.Thechallengeis,however,toprovideeasy,quickand
#> relevantaccesstothevastamountofinformationthatisavailable.Onesteptowardsfacili-
#> tatinginformationaccesstoonlinehealthdataisopinionmining.Eventhoughtheclassifica-
#> tionofpatientopinionsintopositiveandnegativehasbeenpreviouslytackled,mostworks
#> makeuseofmachinelearningmethodsandbagsofwords.Ourfirstcontributionisanexten-
#> siveevaluationofdifferentfeatures,includinglexical,syntactic,semantic,network-based,
#> sentiment-basedandwordembeddingsfeaturestorepresentpatient-authoredtextsfor
#> polarityclassification.Thesecondcontributionofthisworkisthestudyofpolarfacts(i.e.
#> objectiveinformationwithpolarconnotations).Traditionally,thepresenceofpolarfactshas
#> beenneglectedandresearchinpolarityclassificationhasbeenboundedtoopinionated
#> texts.Wedemonstratetheexistenceandimportanceofpolarfactsforthepolarityclassifica-
#> tionofhealthinformation.
#> **Received:**January30,2018

Using copy and paste in a PDF reader, it looks like this:

Exploiting information in health-related social media services is of great interest for patients,
researchers and medical companies. The challenge is, however, to provide easy, quick and
relevant access to the vast amount of information that is available. One step towards facili-
tating information access to online health data is opinion mining. Even though the classifica-
tion of patient opinions into positive and negative has been previously tackled, most works
make use of machine learning methods and bags of words. Our first contribution is an exten-
sive evaluation of different features, including lexical, syntactic, semantic, network-based,
sentiment-based and word embeddings features to represent patient-authored texts for
polarity classification. The second contribution of this work is the study of polar facts (i.e.
objective information with polar connotations). Traditionally, the presence of polar facts has
been neglected and research in polarity classification has been bounded to opinionated
texts. We demonstrate the existence and importance of polar facts for the polarity classifica-
tion of health information.

[Filimoa]

This is related to #48 . Should be merged this weekend!

[JBGruber]

I was thinking the same. But:

$ env/bin/pip install git+https://github.com/cipherCOM/open-parse.git
#> [omitted for clarity]
$ env/bin/python test.py 
#> Abstract<br><br>a1111111111
#> a1111111111
#> a1111111111
#> a1111111111
#> a1111111111<br><br>**Introduction**<br><br>Exploitinginformationinhealth-relatedsocialmediaservicesisofgreatinterestforpatients,
#> researchersandmedicalcompanies.Thechallengeis,however,toprovideeasy,quickand
#> relevantaccesstothevastamountofinformationthatisavailable.Onesteptowardsfacili-
#> tatinginformationaccesstoonlinehealthdataisopinionmining.Eventhoughtheclassifica-
#> tionofpatientopinionsintopositiveandnegativehasbeenpreviouslytackled,mostworks
#> makeuseofmachinelearningmethodsandbagsofwords.Ourfirstcontributionisanexten-
#> siveevaluationofdifferentfeatures,includinglexical,syntactic,semantic,network-based,
#> sentiment-basedandwordembeddingsfeaturestorepresentpatient-authoredtextsfor
#> polarityclassification.Thesecondcontributionofthisworkisthestudyofpolarfacts(i.e.
#> objectiveinformationwithpolarconnotations).Traditionally,thepresenceofpolarfactshas
#> beenneglectedandresearchinpolarityclassificationhasbeenboundedtoopinionated
#> texts.Wedemonstratetheexistenceandimportanceofpolarfactsforthepolarityclassifica-
#> tionofhealthinformation.
#> **Received:**January30,2018

test.py being the lines of code above.

[Filimoa]

Are you checking out his PR? Do you have a sample PDF you'd be able to share?

[JBGruber]

https://github.com/cipherCOM/open-parse.git is the PR, as far as I can see. The code above downloads and parses this PDF: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0207996&type=printable

[Filimoa]

Issues should be fixed. I tested with your pdf.