S2ORC vs Arxiv vs PMC

[lvwerra]

Currently we have four datasets containing S2ORC, Arxiv, and PMC data:

• lm_en_s2orc_ai2_pdf_parses
• lm_en_s2orc_ai2_abstracts
• lm_en_arxiv
• lm_en_pmc

There are a few concerns:

1. Overlap between abstracts and pdf parses of S2ORC. Since there are many more abstracts than full pdf parses we probably don't want to discard all abstracts. Currently investigating if we can match on paper_id to discard abstracts of papers that have pdf parses.
2. There is probably significant overlap between Arxiv, PMC <-> S2ORC pdf parses but the former are probably larger. So it would make sense to exclude the Arxiv/PMC sources from S2ORC. The source info exists in principal in the S2ORC dataset but seems not to be present in the datasets above. Asked Kyle if there is a way to get that info.
3. The Arxiv/PMC sources are less preprocessed and e.g. references should be removed. This is requires a custom filter/map.

[lvwerra]

Update on 1): 85% of the paper_id in lm_en_s2orc_ai2_pdf_parses also appear in lm_en_s2orc_ai2_abstracts. So I think we should definitely remove those from the abstracts. I think in any case we probably need a separate script to do these steps before we apply the other filters.

cc @SaulLu @thomasw21

[thomasw21]

How many are there? We can have a json_file with all the ids we need to remove from abstract and have a remove_duplicate_s2orc function or something like that that loads that json file and remove all elements that match from the abstract dataset?

[lvwerra]

How many are there? We can have a json_file with all the ids we need to remove from abstract and have a remove_duplicate_s2orc function or something like that that loads that json file and remove all elements that match from the abstract dataset?

In principle yes, but we would need to load the json outside the function, otherwise it would be loaded for every batch which does not sound very efficient :)

[thomasw21]

Yeah you can do a trick about this

def build_dedup_function():
    with open(blabla, "r") as fi:
         duplicate_ids = json.load(fi)
     def f(batch):
          return [paper_id in duplicate_ids for paper_id in blablabla]
    return f
  
# In clean.py
FILTERS = {
   "dedup_filter": build_dedup_function()
}

[SaulLu]

Looks great! 😄

[lvwerra]

Although we dropped arxiv and pmc, here Kyle's response how we could have filtered them out of S2ORC:

gotcha thanks! removing those documents is not difficult. the metadata is stored with the _abstracts , you can see things like pmc_id and arxiv_id fields. if we identify the paper_id in the _abstracts collection that are pmc & arxiv, then we can remove the _pdf_parses entries that have those same paper_id.
in short -- the paper_id acts as an index to align documents between _abstracts and _pdf_parses.