11Overview
22========
33
4- The ``scrapinghub.ScrapinghubClient `` is a new Python client for communicating
5- with the `Scrapinghub API `_. It takes best from ``scrapinghub.Connection `` and
6- ``scrapinghub.HubstorageClient `` and combines it under single interface.
4+ The :class: `~scrapinghub.client.ScrapinghubClient ` is a new Python client for
5+ communicating with the `Scrapinghub API `_.
6+ It takes best from :class: `~scrapinghub.legacy.Connection ` and
7+ :class: `~scrapinghub.hubstorage.HubstorageClient `, and combines it under single
8+ interface.
79
810First, you instantiate new client::
911
@@ -12,7 +14,8 @@ First, you instantiate new client::
1214 >>> client
1315 <scrapinghub.client.ScrapinghubClient at 0x1047af2e8>
1416
15- Client instance has ``projects `` field for access to client projects.
17+ Client instance has :attr: `~scrapinghub.client.ScrapinghubClient.projects ` field
18+ for access to client projects.
1619
1720Projects
1821--------
@@ -46,32 +49,29 @@ And select a particular project to work with::
4649
4750.. tip :: The above is a shortcut for ``client.projects.get(123)``.
4851
52+
4953Project
5054-------
5155
52- Project instance has ``jobs `` field to work with the project jobs.
56+ :class: `~scrapinghub.client.projects.Project ` instance has
57+ :attr: `~scrapinghub.client.projects.Project.jobs ` field to work with
58+ the project jobs.
5359
54- Jobs instance is described well in :ref: `Jobs <jobs >` section below.
60+ :class: `~scrapinghub.client.jobs.Jobs ` instance is described well in
61+ :ref: `Jobs <jobs >` section below.
5562
56- For example, to schedule a spider run (it returns a job object)::
63+ For example, to schedule a spider run (it returns a
64+ :class: `~scrapinghub.client.jobs.Job ` object)::
5765
5866 >>> project.jobs.run('spider1', job_args={'arg1': 'val1'})
5967 <scrapinghub.client.Job at 0x106ee12e8>>
6068
61- Project instance also has the following fields:
62-
63- - **activity ** - access to :ref: `project activity <project-activity >` records
64- - **collections ** - work with :ref: `project collections <project-collections >`
65- - **frontiers ** - using :ref: `project frontiers <project-frontiers >`
66- - **settings ** - interface to :ref: `project settings <project-settings >`
67- - **spiders ** - access to :ref: `spiders collection <project-spiders >`
68-
69-
70- .. _project-settings :
7169
7270Settings
7371--------
7472
73+ You can work with project settings via :class: `~scrapinghub.client.projects.Settings `.
74+
7575To get a list of the project settings::
7676
7777 >>> project.settings.list()
@@ -91,11 +91,12 @@ Or update a few project settings at once::
9191 >>> project.settings.update({'default_job_units': 1,
9292 ... 'job_runtime_limit': 20})
9393
94- .. _project-spiders :
9594
9695Spiders
9796-------
9897
98+ Spiders collection is accessible via :class: `~scrapinghub.client.spiders.Spiders `.
99+
99100To get the list of spiders of the project::
100101
101102 >>> project.spiders.list()
@@ -119,7 +120,8 @@ To select a particular spider to work with::
119120Spider
120121------
121122
122- Like project instance, spider instance has ``jobs `` field to work with the spider's jobs.
123+ Like project instance, :class: `~scrapinghub.client.spiders.Spider ` instance has
124+ ``jobs `` field to work with the spider's jobs.
123125
124126To schedule a spider run::
125127
@@ -133,7 +135,8 @@ Note that you don't need to specify spider name explicitly.
133135Jobs
134136----
135137
136- Jobs collection is available on project/spider level.
138+ :class: `~scrapinghub.client.jobs.Jobs ` collection is available on project/spider
139+ level.
137140
138141get
139142^^^
@@ -290,7 +293,8 @@ Note that there can be a lot of spiders, so the method above returns an iterator
290293Job
291294---
292295
293- Job instance provides access to a job data with the following fields:
296+ :class: `~scrapinghub.client.jobs.Job ` instance provides access to a job data
297+ with the following fields:
294298
295299- metadata
296300- items
@@ -311,7 +315,8 @@ To delete a job::
311315Metadata
312316^^^^^^^^
313317
314- Job details can be found in jobs metadata and it's scrapystats::
318+ :class: `~scrapinghub.client.jobs.JobMeta ` details can be found in jobs metadata
319+ and it's scrapystats::
315320
316321 >>> job.metadata.get('version')
317322 '5123a86-master'
@@ -338,7 +343,8 @@ Anything can be stored in metadata, here is example how to add tags::
338343Items
339344^^^^^
340345
341- To retrieve all scraped items from a job::
346+ To retrieve all scraped items from a job use
347+ :class: `~scrapinghub.client.items.Items `::
342348
343349 >>> for item in job.items.iter():
344350 ... # do something with item (it's just a dict)
@@ -348,7 +354,7 @@ To retrieve all scraped items from a job::
348354Logs
349355^^^^
350356
351- To retrieve all log entries from a job::
357+ To retrieve all log entries from a job use :class: ` ~scrapinghub.client.logs.Logs ` ::
352358
353359 >>> for logitem in job.logs.iter():
354360 ... # logitem is a dict with level, message, time
@@ -364,7 +370,7 @@ To retrieve all log entries from a job::
364370Requests
365371^^^^^^^^
366372
367- To retrieve all requests from a job::
373+ To retrieve all requests from a job there's :class: ` ~scrapinghub.client.requests.Requests ` ::
368374
369375 >>> for reqitem in job.requests.iter():
370376 ... # reqitem is a dict
@@ -384,18 +390,21 @@ To retrieve all requests from a job::
384390Samples
385391^^^^^^^
386392
387- To retrieve all samples for a job::
393+ :class: `~scrapinghub.client.samples.Samples ` is useful to retrieve all samples
394+ for a job::
388395
389396 >>> for sample in job.samples.iter():
390397 ... # sample is a list with a timestamp and data
391398 >>> sample
392399 [1482233732452, 0, 0, 0, 0, 0]
393400
394- .. _project-activity :
395401
396402Activity
397403--------
398404
405+ :class: `~scrapinghub.client.activity.Activity ` provides a convenient interface
406+ to project activity events.
407+
399408To retrieve all activity events from a project::
400409
401410 >>> project.activity.iter()
@@ -419,14 +428,13 @@ Or post multiple events at once::
419428 >>> project.activity.add(events)
420429
421430
422- .. _project-collections :
423431
424432Collections
425433-----------
426434
427435As an example, let's store hash and timestamp pair for foo spider.
428436
429- Usual workflow with ` Collections `_ would be::
437+ Usual workflow with :class: ` ~scrapinghub.client.collections. Collections ` would be::
430438
431439 >>> collections = project.collections
432440 >>> foo_store = collections.get_store('foo_store')
@@ -447,12 +455,11 @@ Usual workflow with `Collections`_ would be::
447455
448456Collections are available on project level only.
449457
450- .. _project-frontiers :
451458
452459Frontiers
453460---------
454461
455- Typical workflow with ` Frontier `_ ::
462+ Typical workflow with :class: ` ~scrapinghub.client.frontiers.Frontiers ` ::
456463
457464 >>> frontiers = project.frontiers
458465
@@ -466,7 +473,7 @@ List all frontiers::
466473 >>> frontiers.list()
467474 ['test', 'test1', 'test2']
468475
469- Get a frontier by name::
476+ Get a :class: ` ~scrapinghub.client.frontiers.Frontier ` instance by name::
470477
471478 >>> frontier = frontiers.get('test')
472479 >>> frontier
@@ -482,7 +489,7 @@ List all slots::
482489 >>> frontier.list()
483490 ['example.com', 'example.com2']
484491
485- Get a frontier slot by name::
492+ Get a :class: ` ~scrapinghub.client.frontiers.FrontierSlot ` by name::
486493
487494 >>> slot = frontier.get('example.com')
488495 >>> slot
@@ -507,7 +514,9 @@ Add a fingerprint only to the slot::
507514 >>> slot.fingerprints.add(['fp1', 'fp2'])
508515 >>> slot.flush()
509516
510- There are convenient shortcuts: ``f `` for ``fingerprints `` and ``q `` for ``queue ``.
517+ There are convenient shortcuts: ``f `` for ``fingerprints `` to access
518+ :class: `~scrapinghub.client.frontiers.FrontierSlotFingerprints ` and ``q `` for
519+ ``queue `` to access :class: `~scrapinghub.client.frontiers.FrontierSlotQueue `.
511520
512521Add requests with additional parameters::
513522
@@ -583,7 +592,6 @@ Exceptions
583592
584593
585594.. _Scrapinghub API : http://doc.scrapinghub.com/api.html
586- .. _Collections : http://doc.scrapinghub.com/api/collections.html
587595.. _Frontier : http://doc.scrapinghub.com/api/frontier.html
588596.. _count endpoint : https://doc.scrapinghub.com/api/jobq.html#jobq-project-id-count
589597.. _list endpoint : https://doc.scrapinghub.com/api/jobq.html#jobq-project-id-list
0 commit comments