Skip to content

Commit 93f47c3

Browse files
authored
Merge pull request #121 from scrapinghub/docs-for-0.12
[readme.rst] edits, icons, quickstart
2 parents de1f7c3 + 38609cc commit 93f47c3

File tree

2 files changed

+61
-18
lines changed

2 files changed

+61
-18
lines changed

LICENSE

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
1-
Copyright (c) Scrapy developers.
1+
Copyright (c) 2015 Scrapinghub
2+
23
All rights reserved.
34

45
Redistribution and use in source and binary forms, with or without modification,

README.rst

Lines changed: 59 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -14,20 +14,67 @@ Scrapyrt (Scrapy realtime)
1414
.. image:: https://img.shields.io/pypi/l/scrapyrt.svg
1515
:target: https://pypi.python.org/pypi/scrapyrt
1616

17+
.. image:: https://img.shields.io/pypi/dm/scrapyrt.svg
18+
:target: https://pypistats.org/packages/scrapyrt
19+
:alt: Downloads count
20+
21+
.. image:: https://readthedocs.org/projects/scrapyrt/badge/?version=latest
22+
:target: https://scrapyrt.readthedocs.io/en/latest/api.html
23+
24+
Introduction
25+
============
26+
1727
HTTP server which provides API for scheduling Scrapy spiders and
1828
making requests with spiders.
1929

20-
Allows you to easily add HTTP API to your existing Scrapy project. All Scrapy project
21-
components (e.g. middleware, pipelines, extensions) are supported out of the box. You
22-
simply run Scrapyrt in Scrapy project directory and it starts HTTP server allowing you
23-
to schedule your spiders and get spider output in JSON format.
30+
Features
31+
========
32+
* Allows you to easily add HTTP API to your existing Scrapy project
33+
* All Scrapy project components (e.g. middleware, pipelines, extensions) are supported out of the box.
34+
* You simply run Scrapyrt in Scrapy project directory and it starts HTTP server allowing you to schedule your spiders and get spider output in JSON format.
35+
36+
Note
37+
====
38+
* Project is not a replacement for `Scrapyd <https://scrapyd.readthedocs.io/en/stable/>`_ or `Scrapy Cloud <https://www.zyte.com/scrapy-cloud/>`_ or other infrastructure to run long running crawls
39+
* Not suitable for long running spiders, good for spiders that will fetch one response from some website and return response
40+
41+
Getting started
42+
===============
43+
44+
To install Scrapyrt::
45+
46+
pip install scrapyrt
47+
48+
Now you can run Scrapyrt from within Scrapy project by just typing::
49+
50+
scrapyrt
51+
52+
in Scrapy project directory.
53+
54+
Scrapyrt will look for ``scrapy.cfg`` file to determine your project settings,
55+
and will raise error if it won't find one. Note that you need to have all
56+
your project requirements installed.
57+
58+
Scrapyrt supports endpoint ``/crawl.json`` that can be requested
59+
with two methods.
60+
61+
To run sample `toscrape-css spider`_ from `Scrapy educational quotesbot project`_
62+
parsing page about famous quotes::
63+
64+
curl "http://localhost:9080/crawl.json?spider_name=toscrape-css&url=http://quotes.toscrape.com/"
65+
66+
67+
To run same spider only allowing one request and parsing url
68+
with callback ``parse_foo``::
69+
70+
curl "http://localhost:9080/crawl.json?spider_name=toscrape-css&url=http://quotes.toscrape.com/&callback=parse_foo&max_requests=1"
71+
2472

2573

2674
Documentation
2775
=============
2876

29-
Documentation is available here: http://scrapyrt.readthedocs.org/en/latest/index.html
30-
77+
`Documentation is available on readthedocs <http://scrapyrt.readthedocs.org/en/latest/index.html>`_.
3178

3279
Support
3380
=======
@@ -40,16 +87,11 @@ Commercial support is also available by `Scrapinghub`_.
4087
.. _create a question issue: https://github.com/scrapinghub/scrapyrt/issues/new?labels=question
4188
.. _Scrapinghub: http://scrapinghub.com
4289

43-
Development
44-
===========
45-
46-
Release
47-
-------
48-
49-
Use `bumpversion`_ tool, e.g. to release minor version do::
90+
License
91+
=======
92+
ScrapyRT is offered under `BSD 3-Clause license <https://en.wikipedia.org/wiki/BSD_licenses#3-clause_license_(%22BSD_License_2.0%22,_%22Revised_BSD_License%22,_%22New_BSD_License%22,_or_%22Modified_BSD_License%22)>`_.
5093

51-
bumpversion minor --verbose
52-
git push origin master
53-
git push origin <new_version_tag>
5494

55-
.. _bumpversion: https://pypi.python.org/pypi/bumpversion
95+
Development
96+
===========
97+
Development taking place on `Github <https://github.com/scrapinghub/scrapyrt>`_.

0 commit comments

Comments
 (0)