Skip to content

chore: Further reduce retry error summary in log #1370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Pijukatel merged 1 commit into from
Aug 25, 2025
Merged

chore: Further reduce retry error summary in log #1370

Pijukatel merged 1 commit into from
Aug 25, 2025

Conversation

@Pijukatel
Copy link
Collaborator

@Pijukatel Pijukatel commented Aug 22, 2025 •
edited
Loading

Description

  • Further reduce retry error summary in log - no more multiline
  • Ignore details of Playwrigth internal errors in warning summary

Issues

@Pijukatel
Further reduce retry error summary in log.
ad76522 
Especially playwrigth internal errors
@Pijukatel Pijukatel changed the title feat: Further reduce retry error summary in log. feat: Further reduce retry error summary in log Aug 22, 2025
@github-actions github-actions bot added this to the 121st sprint - Tooling team milestone Aug 22, 2025
@github-actions github-actions bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Aug 22, 2025
@Pijukatel Pijukatel marked this pull request as ready for review August 22, 2025 13:08
@Pijukatel Pijukatel requested a review from vdusek August 22, 2025 13:08
vdusek
vdusek reviewed Aug 22, 2025
View reviewed changes
Copy link
Collaborator

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feat?

could you show me how it looked before and how it looks now on an example? thanks

@Pijukatel
Copy link
Collaborator Author

Pijukatel commented Aug 25, 2025 •
edited
Loading

could you show me how it looked before and how it looks now on an example? thanks

Asyncio timeout example:

Before:

[crawlee.crawlers._basic._basic_crawler] WARN  Retrying request to https://a.placeholder.com due to: Request handler timed out after 1.0 seconds 
File "/home/pijukatel/repos/crawlee-python/tests/unit/crawlers/_basic/test_basic_crawler.py", line 1360, in handler,     await asyncio.sleep(10)  # INJECTED DELAY
[crawlee.crawlers._basic._basic_crawler] WARN  Retrying request to https://a.placeholder.com due to: Request handler timed out after 1.0 seconds 
File "/home/pijukatel/repos/crawlee-python/tests/unit/crawlers/_basic/test_basic_crawler.py", line 1360, in handler,     await asyncio.sleep(10)  # INJECTED DELAY
[crawlee.crawlers._basic._basic_crawler] WARN  Retrying request to https://a.placeholder.com due to: Request handler timed out after 1.0 seconds 
File "/home/pijukatel/repos/crawlee-python/tests/unit/crawlers/_basic/test_basic_crawler.py", line 1360, in handler,     await asyncio.sleep(10)  # INJECTED DELAY

After:

[crawlee.crawlers._basic._basic_crawler] WARN  Retrying request to https://a.placeholder.com due to: Request handler timed out after 1.0 seconds,  File "/home/pijukatel/repos/crawlee-python/tests/unit/crawlers/_basic/test_basic_crawler.py", line 1360, in handler,     await asyncio.sleep(10)  # INJECTED DELAY
[crawlee.crawlers._basic._basic_crawler] WARN  Retrying request to https://a.placeholder.com due to: Request handler timed out after 1.0 seconds,  File "/home/pijukatel/repos/crawlee-python/tests/unit/crawlers/_basic/test_basic_crawler.py", line 1360, in handler,     await asyncio.sleep(10)  # INJECTED DELAY
[crawlee.crawlers._basic._basic_crawler] WARN  Retrying request to https://a.placeholder.com due to: Request handler timed out after 1.0 seconds,  File "/home/pijukatel/repos/crawlee-python/tests/unit/crawlers/_basic/test_basic_crawler.py", line 1360, in handler,     await asyncio.sleep(10)  # INJECTED DELAY

Playwright error example:
Before:

[crawlee.crawlers._playwright._playwright_crawler] WARN  Retrying request to https://totally-non-existing-site.com/blablablba due to: Page.goto: net::ERR_NAME_NOT_RESOLVED at https://totally-non-existing-site.com/blablablba
Call log:
  - navigating to "https://totally-non-existing-site.com/blablablba", waiting until "load"
 
File "/home/pijukatel/repos/crawlee-python/.venv/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 132, in _inner_send,     result = next(iter(done)).result()
[crawlee.crawlers._playwright._playwright_crawler] WARN  Retrying request to https://totally-non-existing-site.com/blablablba due to: Page.goto: net::ERR_NAME_NOT_RESOLVED at https://totally-non-existing-site.com/blablablba
Call log:
  - navigating to "https://totally-non-existing-site.com/blablablba", waiting until "load"
 
File "/home/pijukatel/repos/crawlee-python/.venv/lib/python3.10/site-packages/playwright/_impl/_connection.py", line 132, in _inner_send,     result = next(iter(done)).result()
[crawlee.crawlers._playwright._playwright_crawler] WARN  Retrying request to https://totally-non-existing-site.com/blablablba due to: Page.goto: net::ERR_NAME_NOT_RESOLVED at https://totally-non-existing-site.com/blablablba
Call log:
  - navigating to "https://totally-non-existing-site.com/blablablba", waiting until "load"

After:

[crawlee.crawlers._playwright._playwright_crawler] WARN  Retrying request to https://totally-non-existing-site.com/blablablba due to: Page.goto: net::ERR_NAME_NOT_RESOLVED at https://totally-non-existing-site.com/blablablba
[crawlee.crawlers._playwright._playwright_crawler] WARN  Retrying request to https://totally-non-existing-site.com/blablablba due to: Page.goto: net::ERR_NAME_NOT_RESOLVED at https://totally-non-existing-site.com/blablablba
[crawlee.crawlers._playwright._playwright_crawler] WARN  Retrying request to https://totally-non-existing-site.com/blablablba due to: Page.goto: net::ERR_NAME_NOT_RESOLVED at https://totally-non-existing-site.com/blablablba

@Pijukatel Pijukatel changed the title feat: Further reduce retry error summary in log chore: Further reduce retry error summary in log Aug 25, 2025
vdusek
vdusek approved these changes Aug 25, 2025
View reviewed changes
Copy link
Collaborator

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Pijukatel Pijukatel merged commit 80b5fa8 into master Aug 25, 2025
22 checks passed
@Pijukatel Pijukatel deleted the futher-reduce-retry-error-summaries branch August 25, 2025 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vdusek vdusek vdusek approved these changes

Assignees

@Pijukatel Pijukatel

Labels

t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.

Projects

None yet

Milestone

121st sprint - Tooling team

Development

Successfully merging this pull request may close these issues.

Further reduce Playwright error summary in retry log

3 participants

@Pijukatel @vdusek