Merge with pubextract?

[adelavega]

https://github.com/litmining/pubextract is a plugin for pubget developed by @jeromedockes

The original goal was for it to be a place to store extractors that can be applied to pubget outputs, natively as plugins.

However, there is a concern of lack of maintenance and redundancy with this neurostore-text-extraction.

JB is also interested in having a generic place to store text extractors.

It's not clear if these two projects are fully mergable (as neurostore-text-extraction's goal is primarily on production level extractors), but there are some things we could do to make this library for generically useful, such as:

• Allowing Dataset to read in pubget outputs (rather than only the neurostore-data-pond)
• Making outputs more generic

[jdkent]

future thinking:

• splitting this up into two libraries
  • pipelining logic
  • repository of extractors

independent versus dependent pipelines

How we format the extractors:

• this is a collection of plugins

make the extractors directly applicable to the pubget outputs, not just our idiosyncratic data-pond.
^ make this a more proximal goal, being able to read pubget outputs/labelbuddy files as inputs into a dataset

like being able to read in labelbuddy jsonl files.

[adelavega]

Apply to labelbuddy file as well (for development)