Skip to content

Commit a546e75

Browse files
janbucharvdusek
janbuchar
and
vdusek
authored
docs: Add PPE guide (#416)
- closes #414 - should be similar to https://docs.apify.com/sdk/js/docs/guides/pay-per-event, except for a minor difference in Local development - in the Python version, we always enable the charging log dataset when running locally --------- Co-authored-by: Vlada Dusek <[email protected]>
1 parent ade53dc commit a546e75

File tree

9 files changed

+357
-0
lines changed

9 files changed

+357
-0
lines changed

‎docs/03_concepts/12_pay_per_event.mdx‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
id: pay-per-event
3+
title: Pay-per-event monetization
4+
description: Monetize your Actors using the pay-per-event pricing model
5+
---
6+
7+
import ActorChargeSource from '!!raw-loader!./code/actor_charge.py';
8+
import ConditionalActorChargeSource from '!!raw-loader!./code/conditional_actor_charge.py';
9+
import ApiLink from '@site/src/components/ApiLink';
10+
import CodeBlock from '@theme/CodeBlock';
11+
12+
Apify provides several [pricing models](https://docs.apify.com/platform/actors/publishing/monetize) for monetizing your Actors. The most recent and most flexible one is [pay-per-event](https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event), which lets you charge your users programmatically directly from your Actor. As the name suggests, you may charge the users each time a specific event occurs, for example a call to an external API or when you return a result.
13+
14+
To use the pay-per-event pricing model, you first need to [set it up](https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event) for your Actor in the Apify console. After that, you're free to start charging for events.
15+
16+
## Charging for events
17+
18+
After monetization is set in the Apify console, you can add <ApiLink to="class/Actor#charge">`Actor.charge`</ApiLink> calls to your code and start monetizing!
19+
20+
<CodeBlock language="python">
21+
{ActorChargeSource}
22+
</CodeBlock>
23+
24+
Then you just push your code to Apify and that's it! The SDK will even keep track of the max total charge setting for you, so you will not provide more value than what the user chose to pay for.
25+
26+
If you need finer control over charging, you can access call <ApiLink to="class/Actor#get_charging_manager">`Actor.get_charging_manager()`</ApiLink> to access the <ApiLink to="class/ChargingManager">`ChargingManager`</ApiLink>, which can provide more detailed information - for example how many events of each type can be charged before reaching the configured limit.
27+
28+
## Transitioning from a different pricing model
29+
30+
When you plan to start using the pay-per-event pricing model for an Actor that is already monetized with a different pricing model, your source code will need support both pricing models during the transition period enforced by the Apify platform. Arguably the most frequent case is the transition from the pay-per-result model which utilizes the `ACTOR_MAX_PAID_DATASET_ITEMS` environment variable to prevent returning unpaid dataset items. The following is an example how to handle such scenarios. The key part is the <ApiLink to="class/ChargingManager#get_pricing_info">`ChargingManager.get_pricing_info()`</ApiLink> method which returns information about the current pricing model.
31+
32+
<CodeBlock language="python">
33+
{ConditionalActorChargeSource}
34+
</CodeBlock>
35+
36+
## Local development
37+
38+
It is encouraged to test your monetization code on your machine before releasing it to the public. To tell your Actor that it should work in pay-per-event mode, pass it the `ACTOR_TEST_PAY_PER_EVENT` environment variable:
39+
40+
```shell
41+
ACTOR_TEST_PAY_PER_EVENT=true python -m youractor
42+
```
43+
44+
If you also wish to see a log of all the events charged throughout the run, the Apify SDK keeps a log of charged events in a so called charging dataset. Your charging dataset can be found under the `charging_log` name (unless you change your storage settings, this dataset is stored in `storage/datasets/charging_log/`). Please note that this log is not available when running the Actor in production on the Apify platform.
45+
46+
Because pricing configuration is stored by the Apify platform, all events will have a default price of $1.

‎docs/03_concepts/code/actor_charge.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
from apify import Actor
2+
3+
4+
async def main() -> None:
5+
async with Actor:
6+
# highlight-start
7+
# Charge for a single occurence of an event
8+
await Actor.charge(event_name='init')
9+
# highlight-end
10+
11+
# Prepare some mock results
12+
result = [
13+
{'word': 'Lorem'},
14+
{'word': 'Ipsum'},
15+
{'word': 'Dolor'},
16+
{'word': 'Sit'},
17+
{'word': 'Amet'},
18+
]
19+
# highlight-start
20+
# Shortcut for charging for each pushed dataset item
21+
await Actor.push_data(result, 'result-item')
22+
# highlight-end
23+
24+
# highlight-start
25+
# Or you can charge for a given number of events manually
26+
await Actor.charge(
27+
event_name='result-item',
28+
count=len(result),
29+
)
30+
# highlight-end

‎docs/03_concepts/code/conditional_actor_charge.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
from apify import Actor
2+
3+
4+
async def main() -> None:
5+
async with Actor:
6+
# Check the dataset because there might already be items
7+
# if the run migrated or was restarted
8+
default_dataset = await Actor.open_dataset()
9+
dataset_info = await default_dataset.get_info()
10+
charged_items = dataset_info.item_count if dataset_info else 0
11+
12+
# highlight-start
13+
if Actor.get_charging_manager().get_pricing_info().is_pay_per_event:
14+
# highlight-end
15+
await Actor.push_data({'hello': 'world'}, 'dataset-item')
16+
elif charged_items < (Actor.config.max_paid_dataset_items or 0):
17+
await Actor.push_data({'hello': 'world'})
18+
charged_items += 1

‎website/src/components/ApiLink.jsx‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
import React from 'react';
2+
import Link from '@docusaurus/Link';
3+
4+
const ApiLink = ({ to, children }) => {
5+
return (
6+
<Link to={`/reference/${to}`}>{children}</Link>
7+
);
8+
};
9+
10+
export default ApiLink;

‎website/src/components/Gradients.jsx‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
import React from 'react';
2+
3+
export default function Gradients() {
4+
return (
5+
<svg xmlns="http://www.w3.org/2000/svg" width="0" height="0" viewBox="0 0 0 0" fill="none">
6+
<defs>
7+
<linearGradient id="gradient-1" x1="26.6667" y1="12" x2="14.2802" y2="34.5208"
8+
gradientUnits="userSpaceOnUse">
9+
<stop offset="0%" stop-color="#9dceff"/>
10+
<stop offset="70%" stop-color="#4584b6"/>
11+
<stop offset="100%" stop-color="#4584b6"/>
12+
</linearGradient>
13+
<linearGradient id="gradient-2" x1="29.6667" y1="0" x2="-1.80874" y2="26.2295"
14+
gradientUnits="userSpaceOnUse">
15+
<stop offset="0%" stop-color="#4584b6"/>
16+
</linearGradient>
17+
</defs>
18+
</svg>
19+
);
20+
}

‎website/src/components/Highlights.jsx‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
import React from 'react';
2+
import clsx from 'clsx';
3+
import styles from './Highlights.module.css';
4+
import Gradients from './Gradients';
5+
6+
const FeatureList = [
7+
{
8+
title: 'Python with type hints',
9+
Svg: require('../../static/img/features/runs-on-py.svg').default,
10+
description: (
11+
<>
12+
Crawlee for Python is written in a modern way using type hints, providing code completion in your IDE
13+
and helping you catch bugs early on build time.
14+
</>
15+
),
16+
},
17+
// {
18+
// title: 'HTTP scraping',
19+
// Svg: require('../../static/img/features/fingerprints.svg').default,
20+
// description: (
21+
// <>
22+
// Crawlee makes HTTP requests that <a href="https://crawlee.dev/docs/guides/avoid-blocking"><b>mimic browser headers and TLS fingerprints</b></a>.
23+
// It also rotates them automatically based on data about real-world traffic. Popular HTML
24+
// parsers <b><a href="https://crawlee.dev/docs/guides/cheerio-crawler-guide">Cheerio</a>&nbsp;
25+
// and <a href="https://crawlee.dev/docs/guides/jsdom-crawler-guide">JSDOM</a></b> are included.
26+
// </>
27+
// ),
28+
// },
29+
{
30+
title: 'Headless browsers',
31+
Svg: require('../../static/img/features/works-everywhere.svg').default,
32+
description: (
33+
<>
34+
Switch your crawlers from HTTP to a <a href="https://crawlee.dev/python/api/class/PlaywrightCrawler">headless browser</a> in 3 lines of code.
35+
Crawlee builds on top of <b>Playwright</b> and adds its own features. Chrome, Firefox and more.
36+
</>
37+
),
38+
39+
// TODO: this is not true yet
40+
// Crawlee builds on top of <b>Playwright</b> and adds its own <b>anti-blocking features and human-like fingerprints</b>. Chrome, Firefox and more.
41+
},
42+
{
43+
title: 'Automatic scaling and proxy management',
44+
Svg: require('../../static/img/features/auto-scaling.svg').default,
45+
description: (
46+
<>
47+
Crawlee automatically manages concurrency based on <a href="https://crawlee.dev/python/api/class/AutoscaledPool">available system resources</a> and&nbsp;
48+
<a href="https://crawlee.dev/python/api/class/ProxyConfiguration">smartly rotates proxies</a>.
49+
Proxies that often time-out, return network errors or bad HTTP codes like 401 or 403 are discarded.
50+
</>
51+
),
52+
},
53+
// {
54+
// title: 'Queue and Storage',
55+
// Svg: require('../../static/img/features/storage.svg').default,
56+
// description: (
57+
// <>
58+
// You can <a href="https://crawlee.dev/docs/guides/result-storage">save files, screenshots and JSON results</a> to disk with one line of code
59+
// or plug an adapter for your DB. Your URLs are <a href="https://crawlee.dev/docs/guides/request-storage">kept in a queue</a> that ensures their
60+
// uniqueness and that you don't lose progress when something fails.
61+
// </>
62+
// ),
63+
// },
64+
// {
65+
// title: 'Helpful utils and configurability',
66+
// Svg: require('../../static/img/features/node-requests.svg').default,
67+
// description: (
68+
// <>
69+
// Crawlee includes tools for <a href="https://crawlee.dev/api/utils/namespace/social">extracting social handles</a> or phone numbers, infinite scrolling, blocking
70+
// unwanted assets <a href="https://crawlee.dev/api/utils">and many more</a>. It works great out of the box, but also provides&nbsp;
71+
// <a href="https://crawlee.dev/api/basic-crawler/interface/BasicCrawlerOptions">rich configuration options</a>.
72+
// </>
73+
// ),
74+
// },
75+
];
76+
77+
function Feature({ Svg, title, description }) {
78+
return (
79+
<div className={clsx('col col--4')}>
80+
<div className="padding-horiz--md padding-bottom--md">
81+
<div className={styles.featureIcon}>
82+
{Svg ? <Svg alt={title}/> : null}
83+
</div>
84+
<h3>{title}</h3>
85+
<p>{description}</p>
86+
</div>
87+
</div>
88+
);
89+
}
90+
91+
export default function Highlights() {
92+
return (
93+
<section className={styles.features}>
94+
<Gradients />
95+
<div className="container">
96+
<div className="row">
97+
{FeatureList.map((props, idx) => (
98+
<Feature key={idx} {...props} />
99+
))}
100+
</div>
101+
</div>
102+
</section>
103+
);
104+
}

‎website/src/components/Highlights.module.css‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
.features {
2+
display: flex;
3+
align-items: center;
4+
width: 100%;
5+
font-size: 18px;
6+
line-height: 32px;
7+
color: #41465d;
8+
}
9+
10+
html[data-theme="dark"] .features {
11+
color: #b3b8d2;
12+
}
13+
14+
.feature svg {
15+
height: 60px;
16+
width: 60px;
17+
}
18+
19+
.features svg path:nth-child(1) {
20+
fill: url(#gradient-1) !important;
21+
}
22+
23+
.features svg path:nth-child(n + 1) {
24+
fill: url(#gradient-2) !important;
25+
}
26+
27+
html[data-theme="dark"] .featureIcon {
28+
background: #272c3d;
29+
}
30+
31+
.featureIcon {
32+
display: flex;
33+
justify-content: center;
34+
align-items: center;
35+
margin-bottom: 24px;
36+
border-radius: 8px;
37+
background-color: #f2f3fb;
38+
width: 48px;
39+
height: 48px;
40+
}
41+
42+
.features h3 {
43+
font-weight: 700;
44+
font-size: 18px;
45+
line-height: 32px;
46+
}

‎website/src/components/RunnableCodeBlock.jsx‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
import React from 'react';
2+
import clsx from 'clsx';
3+
import CodeBlock from '@theme/CodeBlock';
4+
import Link from '@docusaurus/Link';
5+
import styles from './RunnableCodeBlock.module.css';
6+
7+
const EXAMPLE_RUNNERS = {
8+
playwright: '6i5QsHBMtm3hKph70',
9+
puppeteer: '7tWSD8hrYzuc9Lte7',
10+
cheerio: 'kk67IcZkKSSBTslXI',
11+
};
12+
13+
const RunnableCodeBlock = ({ children, actor, hash, type, ...props }) => {
14+
hash = hash ?? children.hash;
15+
16+
if (!children.code) {
17+
throw new Error(`RunnableCodeBlock requires "code" and "hash" props
18+
Make sure you are importing the code block contents with the roa-loader.`);
19+
}
20+
21+
if (!hash) {
22+
return (
23+
<CodeBlock {...props}>
24+
{ children.code }
25+
</CodeBlock>
26+
);
27+
}
28+
29+
const href = `https://console.apify.com/actors/${actor ?? EXAMPLE_RUNNERS[type ?? 'playwright']}?runConfig=${hash}&asrc=run_on_apify`;
30+
31+
return (
32+
<div className={clsx(styles.container, 'runnable-code-block')}>
33+
<Link href={href} className={styles.button} rel="follow">
34+
Run on
35+
<svg width="91" height="25" viewBox="0 0 91 25" fill="none" xmlns="http://www.w3.org/2000/svg" className="apify-logo-light alignMiddle_src-theme-Footer-index-module"><path d="M3.135 2.85A3.409 3.409 0 0 0 .227 6.699l2.016 14.398 8.483-19.304-7.59 1.059Z" fill="#97D700"></path><path d="M23.604 14.847 22.811 3.78a3.414 3.414 0 0 0-3.64-3.154c-.077 0-.153.014-.228.025l-3.274.452 7.192 16.124c.54-.67.805-1.52.743-2.379Z" fill="#71C5E8"></path><path d="M5.336 24.595c.58.066 1.169-.02 1.706-.248l12.35-5.211L13.514 5.97 5.336 24.595Z" fill="#FF9013"></path><path d="M33.83 5.304h3.903l5.448 14.623h-3.494l-1.022-2.994h-5.877l-1.025 2.994h-3.384L33.83 5.304Zm-.177 9.032h4.14l-2-5.994h-.086l-2.054 5.994ZM58.842 5.304h3.302v14.623h-3.302V5.304ZM64.634 5.304h10.71v2.7h-7.4v4.101h5.962v2.632h-5.963v5.186h-3.309V5.303ZM82.116 14.38l-5.498-9.076h3.748l3.428 6.016h.085l3.599-6.016H91l-5.56 9.054v5.569h-3.324v-5.548ZM51.75 5.304h-7.292v14.623h3.3v-4.634h3.993a4.995 4.995 0 1 0 0-9.99Zm-.364 7.417h-3.628V7.875h3.627a2.423 2.423 0 0 1 0 4.846Z" className="apify-logo" fill="#000"></path></svg>
36+
</Link>
37+
<CodeBlock {...props} className={clsx(styles.codeBlock, 'code-block', props.title != null ? 'has-title' : 'no-title')}>
38+
{ children.code }
39+
</CodeBlock>
40+
</div>
41+
);
42+
};
43+
44+
export default RunnableCodeBlock;

‎website/src/components/RunnableCodeBlock.module.css‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
.button {
2+
display: inline-block;
3+
padding: 3px 10px;
4+
position: absolute;
5+
top: 9px;
6+
right: 9px;
7+
z-index: 1;
8+
font-size: 16px;
9+
line-height: 28px;
10+
background: var(--prism-background-color);
11+
color: var(--prism-color);
12+
border: 1px solid var(--ifm-color-emphasis-300);
13+
border-radius: var(--ifm-global-radius);
14+
opacity: 0.7;
15+
font-weight: 600;
16+
width: 155px;
17+
}
18+
19+
@media screen and (max-width: 768px) {
20+
.button {
21+
display: none;
22+
}
23+
}
24+
25+
.button svg {
26+
height: 20px;
27+
position: absolute;
28+
top: 7.5px;
29+
right: 0;
30+
}
31+
32+
.button:hover {
33+
opacity: 1;
34+
color: var(--prism-color);
35+
}
36+
37+
.container {
38+
position: relative;
39+
}

0 commit comments

Comments
 (0)