Skip to content

Commit 7a505c7

Browse files
committed
Provide links to API in Overview
1 parent 7ea58a6 commit 7a505c7

File tree

1 file changed

+43
-35
lines changed

1 file changed

+43
-35
lines changed

docs/client/overview.rst

Lines changed: 43 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11
Overview
22
========
33

4-
The ``scrapinghub.ScrapinghubClient`` is a new Python client for communicating
5-
with the `Scrapinghub API`_. It takes best from ``scrapinghub.Connection`` and
6-
``scrapinghub.HubstorageClient`` and combines it under single interface.
4+
The :class:`~scrapinghub.client.ScrapinghubClient` is a new Python client for
5+
communicating with the `Scrapinghub API`_.
6+
It takes best from :class:`~scrapinghub.legacy.Connection` and
7+
:class:`~scrapinghub.hubstorage.HubstorageClient`, and combines it under single
8+
interface.
79

810
First, you instantiate new client::
911

@@ -12,7 +14,8 @@ First, you instantiate new client::
1214
>>> client
1315
<scrapinghub.client.ScrapinghubClient at 0x1047af2e8>
1416

15-
Client instance has ``projects`` field for access to client projects.
17+
Client instance has :attr:`~scrapinghub.client.ScrapinghubClient.projects` field
18+
for access to client projects.
1619

1720
Projects
1821
--------
@@ -46,32 +49,29 @@ And select a particular project to work with::
4649

4750
.. tip:: The above is a shortcut for ``client.projects.get(123)``.
4851

52+
4953
Project
5054
-------
5155

52-
Project instance has ``jobs`` field to work with the project jobs.
56+
:class:`~scrapinghub.client.projects.Project` instance has
57+
:attr:`~scrapinghub.client.projects.Project.jobs` field to work with
58+
the project jobs.
5359

54-
Jobs instance is described well in :ref:`Jobs <jobs>` section below.
60+
:class:`~scrapinghub.client.jobs.Jobs` instance is described well in
61+
:ref:`Jobs <jobs>` section below.
5562

56-
For example, to schedule a spider run (it returns a job object)::
63+
For example, to schedule a spider run (it returns a
64+
:class:`~scrapinghub.client.jobs.Job` object)::
5765

5866
>>> project.jobs.run('spider1', job_args={'arg1': 'val1'})
5967
<scrapinghub.client.Job at 0x106ee12e8>>
6068

61-
Project instance also has the following fields:
62-
63-
- **activity** - access to :ref:`project activity <project-activity>` records
64-
- **collections** - work with :ref:`project collections <project-collections>`
65-
- **frontiers** - using :ref:`project frontiers <project-frontiers>`
66-
- **settings** - interface to :ref:`project settings <project-settings>`
67-
- **spiders** - access to :ref:`spiders collection <project-spiders>`
68-
69-
70-
.. _project-settings:
7169

7270
Settings
7371
--------
7472

73+
You can work with project settings via :class:`~scrapinghub.client.projects.Settings`.
74+
7575
To get a list of the project settings::
7676

7777
>>> project.settings.list()
@@ -91,11 +91,12 @@ Or update a few project settings at once::
9191
>>> project.settings.update({'default_job_units': 1,
9292
... 'job_runtime_limit': 20})
9393

94-
.. _project-spiders:
9594

9695
Spiders
9796
-------
9897

98+
Spiders collection is accessible via :class:`~scrapinghub.client.spiders.Spiders`.
99+
99100
To get the list of spiders of the project::
100101

101102
>>> project.spiders.list()
@@ -119,7 +120,8 @@ To select a particular spider to work with::
119120
Spider
120121
------
121122

122-
Like project instance, spider instance has ``jobs`` field to work with the spider's jobs.
123+
Like project instance, :class:`~scrapinghub.client.spiders.Spider` instance has
124+
``jobs`` field to work with the spider's jobs.
123125

124126
To schedule a spider run::
125127

@@ -133,7 +135,8 @@ Note that you don't need to specify spider name explicitly.
133135
Jobs
134136
----
135137

136-
Jobs collection is available on project/spider level.
138+
:class:`~scrapinghub.client.jobs.Jobs` collection is available on project/spider
139+
level.
137140

138141
get
139142
^^^
@@ -290,7 +293,8 @@ Note that there can be a lot of spiders, so the method above returns an iterator
290293
Job
291294
---
292295

293-
Job instance provides access to a job data with the following fields:
296+
:class:`~scrapinghub.client.jobs.Job` instance provides access to a job data
297+
with the following fields:
294298

295299
- metadata
296300
- items
@@ -311,7 +315,8 @@ To delete a job::
311315
Metadata
312316
^^^^^^^^
313317

314-
Job details can be found in jobs metadata and it's scrapystats::
318+
:class:`~scrapinghub.client.jobs.JobMeta` details can be found in jobs metadata
319+
and it's scrapystats::
315320

316321
>>> job.metadata.get('version')
317322
'5123a86-master'
@@ -338,7 +343,8 @@ Anything can be stored in metadata, here is example how to add tags::
338343
Items
339344
^^^^^
340345

341-
To retrieve all scraped items from a job::
346+
To retrieve all scraped items from a job use
347+
:class:`~scrapinghub.client.items.Items`::
342348

343349
>>> for item in job.items.iter():
344350
... # do something with item (it's just a dict)
@@ -348,7 +354,7 @@ To retrieve all scraped items from a job::
348354
Logs
349355
^^^^
350356

351-
To retrieve all log entries from a job::
357+
To retrieve all log entries from a job use :class:`~scrapinghub.client.logs.Logs`::
352358

353359
>>> for logitem in job.logs.iter():
354360
... # logitem is a dict with level, message, time
@@ -364,7 +370,7 @@ To retrieve all log entries from a job::
364370
Requests
365371
^^^^^^^^
366372

367-
To retrieve all requests from a job::
373+
To retrieve all requests from a job there's :class:`~scrapinghub.client.requests.Requests`::
368374

369375
>>> for reqitem in job.requests.iter():
370376
... # reqitem is a dict
@@ -384,18 +390,21 @@ To retrieve all requests from a job::
384390
Samples
385391
^^^^^^^
386392

387-
To retrieve all samples for a job::
393+
:class:`~scrapinghub.client.samples.Samples` is useful to retrieve all samples
394+
for a job::
388395

389396
>>> for sample in job.samples.iter():
390397
... # sample is a list with a timestamp and data
391398
>>> sample
392399
[1482233732452, 0, 0, 0, 0, 0]
393400

394-
.. _project-activity:
395401

396402
Activity
397403
--------
398404

405+
:class:`~scrapinghub.client.activity.Activity` provides a convenient interface
406+
to project activity events.
407+
399408
To retrieve all activity events from a project::
400409

401410
>>> project.activity.iter()
@@ -419,14 +428,13 @@ Or post multiple events at once::
419428
>>> project.activity.add(events)
420429

421430

422-
.. _project-collections:
423431

424432
Collections
425433
-----------
426434

427435
As an example, let's store hash and timestamp pair for foo spider.
428436

429-
Usual workflow with `Collections`_ would be::
437+
Usual workflow with :class:`~scrapinghub.client.collections.Collections` would be::
430438

431439
>>> collections = project.collections
432440
>>> foo_store = collections.get_store('foo_store')
@@ -447,12 +455,11 @@ Usual workflow with `Collections`_ would be::
447455

448456
Collections are available on project level only.
449457

450-
.. _project-frontiers:
451458

452459
Frontiers
453460
---------
454461

455-
Typical workflow with `Frontier`_::
462+
Typical workflow with :class:`~scrapinghub.client.frontiers.Frontiers`::
456463

457464
>>> frontiers = project.frontiers
458465

@@ -466,7 +473,7 @@ List all frontiers::
466473
>>> frontiers.list()
467474
['test', 'test1', 'test2']
468475

469-
Get a frontier by name::
476+
Get a :class:`~scrapinghub.client.frontiers.Frontier` instance by name::
470477

471478
>>> frontier = frontiers.get('test')
472479
>>> frontier
@@ -482,7 +489,7 @@ List all slots::
482489
>>> frontier.list()
483490
['example.com', 'example.com2']
484491

485-
Get a frontier slot by name::
492+
Get a :class:`~scrapinghub.client.frontiers.FrontierSlot` by name::
486493

487494
>>> slot = frontier.get('example.com')
488495
>>> slot
@@ -507,7 +514,9 @@ Add a fingerprint only to the slot::
507514
>>> slot.fingerprints.add(['fp1', 'fp2'])
508515
>>> slot.flush()
509516

510-
There are convenient shortcuts: ``f`` for ``fingerprints`` and ``q`` for ``queue``.
517+
There are convenient shortcuts: ``f`` for ``fingerprints`` to access
518+
:class:`~scrapinghub.client.frontiers.FrontierSlotFingerprints` and ``q`` for
519+
``queue`` to access :class:`~scrapinghub.client.frontiers.FrontierSlotQueue`.
511520

512521
Add requests with additional parameters::
513522

@@ -583,7 +592,6 @@ Exceptions
583592

584593

585594
.. _Scrapinghub API: http://doc.scrapinghub.com/api.html
586-
.. _Collections: http://doc.scrapinghub.com/api/collections.html
587595
.. _Frontier: http://doc.scrapinghub.com/api/frontier.html
588596
.. _count endpoint: https://doc.scrapinghub.com/api/jobq.html#jobq-project-id-count
589597
.. _list endpoint: https://doc.scrapinghub.com/api/jobq.html#jobq-project-id-list

0 commit comments

Comments
 (0)