Conversation
optparse is not used in Scrapy>=2.7
sh_scrapy/crawl.py
Outdated
| Client(_sentry_dsn).captureException() | ||
| except Exception as err: | ||
| print(datetime.datetime.utcnow().isoformat(), | ||
| print(dt.now(timezone.utc).isoformat(), |
There was a problem hiding this comment.
Technically not exactly the same, the new version adds +00:00:
In [19]: dt.now(timezone.utc).isoformat()
Out[19]: '2026-02-25T16:09:38.206121+00:00'
In [20]: dt.utcnow().isoformat()
<ipython-input-20-cac1a3b575b4>:1: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
dt.utcnow().isoformat()
Out[20]: '2026-02-25T16:09:39.256359'It could easily be updated to remove that, however I don't think that's a big deal.
There was a problem hiding this comment.
I think removing deprecation warnings is kind of a big deal, I would go for it.
Going forward, I think we should eventually make tests running with the latest version of everything (deps and Python) fail in case of warnings.
There was a problem hiding this comment.
I meant that having the additional +00:00 is not a big deal
| dist = None | ||
| for ep in entry_points(group='scrapy'): | ||
| if ep.name == 'settings': | ||
| dist = ep.dist | ||
| break |
There was a problem hiding this comment.
Current minimum Python is 3.10, which always includes importlib.metadata
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #104 +/- ##
==========================================
+ Coverage 88.65% 89.25% +0.59%
==========================================
Files 14 14
Lines 873 856 -17
==========================================
- Hits 774 764 -10
+ Misses 99 92 -7 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| super().add_options(parser) | ||
| parser.add_argument( |
There was a problem hiding this comment.
Current minimum Scrapy is 2.7, which always uses argparse.
sh_scrapy/crawl.py
Outdated
| import warnings | ||
| from contextlib import contextmanager | ||
| from importlib.metadata import PathDistribution | ||
| from datetime import timezone, datetime as dt |
There was a problem hiding this comment.
I read somewhere, cannot find it now, a recommendation to import datetime as dt and datetime.datetime as datetime (or use dt.datetime, which would avoid confusion). What do you think?
There was a problem hiding this comment.
Yeah I think I read the same thing and just implemented it wrong, import datetime as dt makes for better namespacing with dt.datetime 😄
There was a problem hiding this comment.
Some of these changes could have been addressed automatically by Ruff’s UP rules. Maybe we should do something like scrapinghub/spidermon#468 in this repository soon in the future.
Several unrelated cleanups: