You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
HTTP server which provides API for scheduling Scrapy spiders and
18
28
making requests with spiders.
19
29
20
-
Allows you to easily add HTTP API to your existing Scrapy project. All Scrapy project
21
-
components (e.g. middleware, pipelines, extensions) are supported out of the box. You
22
-
simply run Scrapyrt in Scrapy project directory and it starts HTTP server allowing you
23
-
to schedule your spiders and get spider output in JSON format.
30
+
Features
31
+
========
32
+
* Allows you to easily add HTTP API to your existing Scrapy project
33
+
* All Scrapy project components (e.g. middleware, pipelines, extensions) are supported out of the box.
34
+
* You simply run Scrapyrt in Scrapy project directory and it starts HTTP server allowing you to schedule your spiders and get spider output in JSON format.
35
+
36
+
Note
37
+
====
38
+
* Project is not a replacement for `Scrapyd <https://scrapyd.readthedocs.io/en/stable/>`_ or `Scrapy Cloud <https://www.zyte.com/scrapy-cloud/>`_ or other infrastructure to run long running crawls
39
+
* Not suitable for long running spiders, good for spiders that will fetch one response from some website and return response
40
+
41
+
Getting started
42
+
===============
43
+
44
+
To install Scrapyrt::
45
+
46
+
pip install scrapyrt
47
+
48
+
Now you can run Scrapyrt from within Scrapy project by just typing::
49
+
50
+
scrapyrt
51
+
52
+
in Scrapy project directory.
53
+
54
+
Scrapyrt will look for ``scrapy.cfg`` file to determine your project settings,
55
+
and will raise error if it won't find one. Note that you need to have all
56
+
your project requirements installed.
57
+
58
+
Scrapyrt supports endpoint ``/crawl.json`` that can be requested
59
+
with two methods.
60
+
61
+
To run sample `toscrape-css spider`_ from `Scrapy educational quotesbot project`_
Documentation is available here: http://scrapyrt.readthedocs.org/en/latest/index.html
30
-
77
+
`Documentation is available on readthedocs <http://scrapyrt.readthedocs.org/en/latest/index.html>`_.
31
78
32
79
Support
33
80
=======
@@ -40,16 +87,11 @@ Commercial support is also available by `Scrapinghub`_.
40
87
.. _create a question issue: https://github.com/scrapinghub/scrapyrt/issues/new?labels=question
41
88
.. _Scrapinghub: http://scrapinghub.com
42
89
43
-
Development
44
-
===========
45
-
46
-
Release
47
-
-------
48
-
49
-
Use `bumpversion`_ tool, e.g. to release minor version do::
90
+
License
91
+
=======
92
+
ScrapyRT is offered under `BSD 3-Clause license <https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22)>`_.
0 commit comments