Log settings changes and ignores

[JustAnotherArchivist]

Example output from grab-site --concurrency 1 --delay 5000 https://www.example.org/:

2022-05-07 02:15:25,835 - wpull.application.tasks.plugin - INFO - Found plugin wpull_hooks_py from /app/libgrabsite/wpull_hooks.py.
2022-05-07 02:15:25,839 - grab_site.wpull_plugin - INFO - Adding ignore: /bxSlider/bxSlider/
2022-05-07 02:15:25,839 - grab_site.wpull_plugin - INFO - Adding ignore: ^https?://(www|px\.srvcs)\.tumblr\.com/(impixu\?|share(/link/?)?\?|reblog/)
<snip>
2022-05-07 02:15:25,847 - grab_site.wpull_plugin - INFO - Adding ignore: ^https?://www\.warnerbros\.com/\d+$
2022-05-07 02:15:26,036 - wpull.processor.web - INFO - Fetching ‘https://www.example.org/’.
2022-05-07 02:15:26,633 - grab_site.wpull_plugin - INFO - Settings change: max_content_length = -1
2022-05-07 02:15:26,642 - wpull.processor.web - INFO - Fetched ‘https://www.example.org/’: 200 OK. Length: 1256 [text/html; charset=UTF-8].
2022-05-07 02:15:26,652 - grab_site.wpull_plugin - INFO - Settings change: delay = 5000
2022-05-07 02:15:26,652 - grab_site.wpull_plugin - INFO - Settings change: concurrency = 1
2022-05-07 02:15:31,668 - wpull.processor.web - INFO - Fetching ‘https://www.example.org/robots.txt’.
2022-05-07 02:15:31,812 - wpull.processor.web - INFO - Fetched ‘https://www.example.org/robots.txt’: 404 Not Found. Length: 445 [text/html; charset=UTF-8].
2022-05-07 02:15:36,825 - grab_site.wpull_plugin - INFO - Adding ignore: asdfasdfasdf
2022-05-07 02:15:36,879 - wpull.processor.web - INFO - Fetching ‘https://www.example.org/sitemap.xml’.
2022-05-07 02:15:37,023 - wpull.processor.web - INFO - Fetched ‘https://www.example.org/sitemap.xml’: 404 Not Found. Length: 445 [text/html; charset=UTF-8].
2022-05-07 02:15:42,034 - wpull.processor.web - INFO - Fetching ‘https://www.iana.org/domains/example’.
2022-05-07 02:15:42,505 - wpull.processor.web - INFO - Fetched ‘https://www.iana.org/domains/example’: 301 Moved Permanently. Length: 244 [text/html; charset=iso-8859-1].
2022-05-07 02:15:47,511 - wpull.processor.web - INFO - Fetching ‘http://www.iana.org/domains/reserved’.
2022-05-07 02:15:47,833 - wpull.processor.web - INFO - Fetched ‘http://www.iana.org/domains/reserved’: 200 OK. Length: 10914 [text/html; charset=UTF-8].
2022-05-07 02:15:52,866 - wpull.processor.web - INFO - Fetching ‘http://www.iana.org/_js/jquery.js’.
2022-05-07 02:15:53,574 - wpull.processor.web - INFO - Fetched ‘http://www.iana.org/_js/jquery.js’: 200 OK. Length: 89501 [application/javascript].
2022-05-07 02:15:58,607 - grab_site.wpull_plugin - INFO - Adding ignore: www\.iana
2022-05-07 02:15:58,607 - grab_site.wpull_plugin - INFO - Removing ignore: asdfasdfasdf
2022-05-07 02:15:58,660 - grab_site.wpull_plugin - INFO - Ignoring ‘http://www.iana.org/_js/iana.js’: [ignore pattern match]
2022-05-07 02:15:58,668 - grab_site.wpull_plugin - INFO - Ignoring ‘http://www.iana.org/_css/2022/iana_website.css’: [ignore pattern match]
2022-05-07 02:15:58,674 - grab_site.wpull_plugin - INFO - Ignoring ‘http://www.iana.org/_img/bookmark_icon.ico’: [ignore pattern match]
2022-05-07 02:15:58,679 - grab_site.wpull_plugin - INFO - Ignoring ‘http://www.iana.org/_img/2022/iana-logo-header.svg’: [ignore pattern match]
2022-05-07 02:15:58,684 - wpull.application.tasks.stats - INFO - FINISHED.
2022-05-07 02:15:58,684 - wpull.application.tasks.stats - INFO - Duration: 0:00:32. Speed: 3.6 KiB/s.
2022-05-07 02:15:58,684 - wpull.application.tasks.stats - INFO - Downloaded: 3 files, 99.3 KiB.

[TheTechRobo]

I tested this a few months ago, and it worked fine.

[ivan]

I don't think I want to log every single URL that is ignored by default, but I would merge this if it were activated by some option.

[ivan]

I am open to being convinced that I must log every URL that is ignored, but most users seem to have been fine without it, and it increases both disk usage and possibly makes the logs very noisy on some sites.