Commit 4df835a
Merge #356
356: Try to fix Scrapy ERROR: Spider error processing r=alallema a=alallema
Currently when running doc-scraper this error occur:
```sh
2023-03-14 12:10:56 [scrapy.core.scraper] ERROR: Spider error processing <GET https://docs.meilisearch.com/learn/advanced/geosearch.html> (referer: https://docs.meilisearch.com/sitemap.xml)
Traceback (most recent call last):
File "/Users/amelielallemand/.local/share/virtualenvs/docs-scraper-vWaWSN46/lib/python3.9/site-packages/twisted/internet/defer.py", line 892, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/documentation_spider.py", line 170, in parse_from_sitemap
self.add_records(response, from_sitemap=True)
File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/documentation_spider.py", line 151, in add_records
records = self.strategy.get_records_from_response(response)
File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/strategies/default_strategy.py", line 44, in get_records_from_response
records = self.get_records_from_dom(response.url)
File "/Users/amelielallemand/Projects/meili/repo/docs-scraper/scraper/src/strategies/default_strategy.py", line 67, in get_records_from_dom
sys.exit('DefaultStrategy.dom is not defined')
SystemExit: DefaultStrategy.dom is not defined
```
This PR try to fix it
Co-authored-by: alallema <[email protected]>1 file changed
+1
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | | - | |
| 42 | + | |
43 | 43 | | |
44 | | - | |
45 | 44 | | |
46 | 45 | | |
47 | 46 | | |
| |||
0 commit comments