DaZhongDianPing

[stutys]

您好非常感谢您的分享
最近在用爬取大众点评的评论数据，
想请问一下，您的代码里边 DaZhongDianPing->Spyder->Xiuxian->spiders comment_spider.py 的一点问题。

1. 这个是怎么用的呢？直接把 CommentSpider(scrapy.Spider)实例化吗？还是有个main是可以把所有的spider都run起来呢？
2. from scrapy.conf import settings 不知道您用的scrapy版本是多少，我的一直显示这个有问题：
   Module scrapy.conf is deprecated, use crawler.settings attribute instead
   Thanks♪(･ω･)ﾉ

[preke]

你之前用过scrapy吗？
我这边是没有用main函数串起来的，每次运行就scrapy crawl _spider.py这个命令就可以了
至于你说的另一个问题？是和版本有关吗，我不太清楚，我的是1.1.1，建议去stackoverflow找找看是否有你问题的答案.我直接把这个报错信息百度了一下，也出现了很多解决方案，建议逐个尝试一下

[stutys]

您好感谢您的回复
现在跑通了 url_spider.py （我看见您这个代码是需要先把url写入mongo, 在其他_spider.py里边通过查询这个url来爬取其他内容的，不知道理解对不？）
下边是代码显示结果，不知道是没爬出来东西？还是没有写到mongo里边去？我在mongodb里边没有看到相关记录·~~
2017-09-27 21:53:38 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (30
2) to <GET http://1.1.1.3:80/disable/disable.htm?url_type=%E8%AE%BF%E9%97%AE%E7%BD
%91%E7%AB%99/%E8%BD%AF%E4%BB%B6%E4%B8%8B%E8%BD%BD&plc_name=%E5%8A%9E%E5%85%AC%E7%B
D%91%E7%AD%96%E7%95%A5> from <GET http://1.1.1.3:80/disable/disable.htm?url_type=%
E8%AE%BF%E9%97%AE%E7%BD%91%E7%AB%99/%E8%BD%AF%E4%BB%B6%E4%B8%8B%E8%BD%BD&plc_name=
%E5%8A%9E%E5%85%AC%E7%BD%91%E7%AD%96%E7%95%A5>
2017-09-27 21:53:38 [scrapy.downloadermiddlewares.redirect] DEBUG: Discarding <GET
http://1.1.1.3:80/disable/disable.htm?url_type=%E8%AE%BF%E9%97%AE%E7%BD%91%E7%AB%
99/%E8%BD%AF%E4%BB%B6%E4%B8%8B%E8%BD%BD&plc_name=%E5%8A%9E%E5%85%AC%E7%BD%91%E7%AD
%96%E7%95%A5>: max redirections reached
2017-09-27 21:53:38 [scrapy.core.engine] INFO: Closing spider (finished)
2017-09-27 21:53:38 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 22228,
'downloader/request_count': 43,
'downloader/request_method_count/GET': 43,
'downloader/response_bytes': 11168,
'downloader/response_count': 43,
'downloader/response_status_count/200': 1,
'downloader/response_status_count/302': 42,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 9, 27, 13, 53, 38, 949000),
'log_count/DEBUG': 44,
'log_count/INFO': 7,
'log_count/WARNING': 1,
'response_received_count': 1,
'scheduler/dequeued': 21,
'scheduler/dequeued/memory': 21,
'scheduler/enqueued': 21,
'scheduler/enqueued/memory': 21,
'start_time': datetime.datetime(2017, 9, 27, 13, 53, 23, 965000)}
2017-09-27 21:53:38 [scrapy.core.engine] INFO: Spider closed (finished)
再次感谢您的回复~~
@preke

[preke]

你的理解是正确的，不过你遇到了302重定向问题，需要查找一下解决方案，可能是大众点评更新了反爬策略？

[stutys]

好的，非常感谢~
@preke

[stutys]

不好意思，我想再问一下，那个url是怎么写到数据库的呢？在url_spider.py里边没有看到连接数据库的代码~~~ 其他_spider里边貌似都有
@preke

[stutys]

对了，您好您用的python版本是2.X吗？
我用的3.6，middlewares.py里边 proxyAuth = "Basic " + base64.b64encode(proxyUser + ":" + proxyPass) 总是有问题，找一堆办法貌似没啥用不知道这个在3.6里边应该怎么改呢？

[stutys]

您好，上面的问题都已经解决，也爬出来了一些数据，非常感谢~~~
现在遇到一个问题是，正爬着的时候爬虫就关了，再次运行就出现下面的问题，是不是被封了呢？需要怎么修改才行呢?Thanks♪(･ω･)ﾉ
2017-09-28 18:03:04 [scrapy.utils.log] INFO: Scrapy 1.4.0 started (bot: Xiuxian)
2017-09-28 18:03:04 [scrapy.utils.log] INFO: Overridden settings: {'BOT_NAME': 'Xiuxian', 'CLOSESPIDER_ERRORCOUNT': 20, 'CONCURRENT_REQUESTS':
32, 'COOKIES_ENABLED': False, 'DOWNLOAD_DELAY': 0.3, 'NEWSPIDER_MODULE': 'Xiuxian.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['Xiuxian
.spiders']}
2017-09-28 18:03:04 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.closespider.CloseSpider',
'scrapy.extensions.logstats.LogStats']
***************************** ['http://www.dianping.com/shop/38212967']
2017-09-28 18:03:04 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'Xiuxian.middlewares.RotateUserAgentMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-09-28 18:03:04 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-09-28 18:03:04 [scrapy.middleware] INFO: Enabled item pipelines:
['Xiuxian.pipelines.MongoDBPipeline']
2017-09-28 18:03:04 [scrapy.core.engine] INFO: Spider opened
2017-09-28 18:03:04 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-09-28 18:03:04 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-09-28 18:03:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.dianping.com/robots.txt> (referer: None)
2017-09-28 18:03:05 [scrapy.core.engine] DEBUG: Crawled (403) <GET http://www.dianping.com/shop/38212967> (referer: None)
2017-09-28 18:03:05 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 http://www.dianping.com/shop/38212967>: HTTP status code
is not handled or not allowed
2017-09-28 18:03:05 [scrapy.core.engine] INFO: Closing spider (finished)
2017-09-28 18:03:05 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 588,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 975,
'downloader/response_count': 2,
'downloader/response_status_count/200': 1,
'downloader/response_status_count/403': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2017, 9, 28, 10, 3, 5, 520099),
'httperror/response_ignored_count': 1,
'httperror/response_ignored_status_count/403': 1,
'log_count/DEBUG': 3,
'log_count/INFO': 8,
'response_received_count': 2,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'start_time': datetime.datetime(2017, 9, 28, 10, 3, 4, 947156)}
2017-09-28 18:03:05 [scrapy.core.engine] INFO: Spider closed (finished)
@preke