Adding more tools to the benchmark?

[adbar]

Hi,

Thanks for your contribution, it's really useful to see evaluations on real-world data! There are further extraction tools for Python which this repository doesn't feature yet and which could be more efficient than some of the ones you're mentioning. You might have a look at

• goose3
• jusText (especially with a custom configuration)
• inscriptis (html-to-txt conversion)
• trafilatura (disclaimer: I'm the author).

Or is there a reason why you didn't use them in the first place? I'd be curious to hear about it.

For more details please refer to the evaluation I've performed. The code including baselines is available here.

[lopuhin]

hi @adbar thanks for the pointers of the tools and evaluation. Another tool which was referenced elsewhere by @saippuakauppias was https://github.com/go-shiori/go-readability. It would be great to add them, we only need to write a script which outputs results in JSON. PRs are welcome, and I hope to have time to add more tools soon as well, it would be great to have more tools evaluated.

[adbar]

Thanks for your answer, I've added JSON to trafilatura and will check if I can write a straightforward PR.

[adbar]

Hi @lopuhin, here is another tool that could be added: Mercury Parser.
(source: adbar/trafilatura#114)

[adbar]

Hi @lopuhin, just a quick follow-up: the benchmark could also be updated using the latest versions of the tools, see for instance the issue adbar/trafilatura#156.

[Seirdy]

Another tool to consider is Azure Immersive Reader, used in Microsoft Edge.

[BradKML]

Seconded this, but also would like to see:

1. which ones are better (F1/precision/accuracy/recall) relative to speed in the same vein as Squash Benchmark or Matt Mahoney for compression algorithms (since there will always a tradeoff between performance and speed)
2. bigger datasets for re-evaluating the benchmark since having a larger diversity of articles from blogs may how a stronger use case

[BradKML]

With the current advancement in RAGs with LLMs I think these benchmarks would be paramount to help in gathering information, and is really due for an update.
P.S. DragNet has a new fork now https://github.com/currentslab/extractnet

[mislav]

Hi all, I was wondering how would the benchmark score today, so I've updated all the tools to their latest versions and re-generated the scores here: master...mislav:article-extraction-benchmark:master

Here are the results sorted by their F₁ score (i.e. the harmonic mean of Precision and Recall):

tool	version	F1	precision	recall	accuracy
trafilatura	2.0.0	0.958 ± 0.006	0.938 ± 0.009	0.978 ± 0.006	0.293 ± 0.033
go-trafilatura	ae7ea06	0.957 ± 0.007	0.940 ± 0.008	0.974 ± 0.008	0.287 ± 0.032
newspaper4k	0.9.3.1	0.949 ± 0.008	0.964 ± 0.008	0.934 ± 0.011	0.326 ± 0.033
news_please	1.6.16	0.948 ± 0.008	0.964 ± 0.008	0.933 ± 0.011	0.326 ± 0.034
readability_js	0.6.0	0.947 ± 0.005	0.914 ± 0.008	0.982 ± 0.003	0.166 ± 0.028
readeck/go-readability	fb0fbc5	0.944 ± 0.005	0.914 ± 0.008	0.976 ± 0.006	0.166 ± 0.028
go-readability	9f5bf5c	0.931 ± 0.009	0.899 ± 0.011	0.965 ± 0.010	0.188 ± 0.030
go-DomDistiller	25b8d04	0.925 ± 0.007	0.900 ± 0.010	0.950 ± 0.011	0.061 ± 0.017
readability-lxml	0.8.4.1	0.922 ± 0.013	0.913 ± 0.014	0.931 ± 0.015	0.315 ± 0.034
goose3	3.1.20	0.896 ± 0.015	0.940 ± 0.013	0.856 ± 0.020	0.232 ± 0.031
beautifulsoup	4.13.5	0.860 ± 0.016	0.850 ± 0.016	0.870 ± 0.020	0.006 ± 0.006
justext	3.0.2	0.804 ± 0.018	0.858 ± 0.016	0.756 ± 0.027	0.088 ± 0.021
inscriptis	2.6.0	0.679 ± 0.015	0.517 ± 0.018	0.992 ± 0.001	0.000 ± 0.000
html2text	2025.4.15	0.662 ± 0.015	0.499 ± 0.017	0.983 ± 0.002	0.000 ± 0.000

I've added go-trafilatura, the faithful Go implementation of the excellent Trafilatura package, and Readeck's go-readability fork of which I am the maintainer.

There is now a Makefile so that all the results can be set up and regenerated with just make run-all.

I did not evaluate AutoExtract nor Diffbot due to being hosted, closed-source services; nor did I evaluate Boilerpipe nor Dragnet due to the difficulty of installing them.

If anyone from Scrapinghub is interested in maintaining this benchmark with these updates, I'd be happy to submit a PR. For now, I'm sharing my findings for anyone who might stumble on this thread where there has been excellent discussion already. I learned a lot about article extraction from this repository and subsequent comments, so thank you all.

Suggestions for bigger datasets that this could be run against are welcome!

[BradKML]

Better Recall: readability_js, inscriptis, html2text
Better Precision/Accuracy: newspaper4k, news_please
What other services can you think of that might be good for the benchmark? Also any benchmark that tests the accuracy of article extraction better than the current one? (It's 5 years old now)

[lopuhin]

hi @mislav wow that's fantastic, yes we're interested in maintaining this. I just checked the changes you quoted and would be happy to merge, the only thing I'd like to change is to preserve the original results in the README, and add new results you obtained as a new section above old results, and reference the commit used. So we'd only keep the current code and versions, but full history of results.