-
Notifications
You must be signed in to change notification settings - Fork 164
feat: dynamic Actor memory #2145
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 6 commits
35307a4
0305a25
6f1618d
90f48e9
e203dce
b3e4628
133af16
52099e7
9e853c8
7ab0ccc
2f7561c
50b78ba
87fba63
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,198 @@ | ||||||
| --- | ||||||
| title: Dynamic Actor memory | ||||||
| description: Learn how to automatically adjust your Actor's memory based on input size and run options, so you can optimize performance and reduce costs without manual configuration.. | ||||||
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
| slug: /actors/development/actor-definition/dynamic-actor-memory | ||||||
| --- | ||||||
|
|
||||||
| **Learn how to automatically adjust your Actor's memory based on input size and run options, so you can optimize performance and reduce costs without manual configuration.** | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| Dynamic Actor memory allows Actors to _automatically adjust its memory allocation based on the input and run options_. Instead of always using a fixed memory value, Actor can use just the right amount of memory for each run. | ||||||
|
||||||
|
|
||||||
| This helps: | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (nit) This is kind of overlapping with |
||||||
|
|
||||||
| - _Optimize performance_ for large inputs (more memory for bigger tasks). | ||||||
| - _Reduce costs_ for small runs (less memory when it’s not needed). | ||||||
| - _Provide better user experience_, so users get optimal performance without having to manually configure memory. | ||||||
|
|
||||||
| :::info | ||||||
| _Important_: The memory calculated by dynamic expression is used as the default memory for the run. Users can still override it manually for each run. | ||||||
| ::: | ||||||
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
|
|
||||||
| --- | ||||||
|
|
||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (opt) I'm kind of missing here a section How dynamic memory works. First of all, we should double-down on the fact that dynamic does not mean that the Actor can on-the-fly ask for more memory. IMHO this is easily confused (we have seen it even internally). Second, it should explain the whole "memory configuration pipeline", as in, how is memory for a run determined (memory expression vs Actor defaults vs run overrides).
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Good point, refactored it 👍 |
||||||
| ## Why dynamic memory matters | ||||||
|
|
||||||
| Currently, Actors often use a _static default memory_, but the optimal memory usually depends on the input: | ||||||
|
|
||||||
| - A small input (e.g., 10 URLs) might run fine on 512 MB. | ||||||
| - A large input (e.g., 1,000 URLs) could require 4 GB+ to run efficiently. | ||||||
|
|
||||||
| Setting a single default either _wastes resources or slows down execution_. Dynamic memory solves this by calculating memory just before the run starts. | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ## How to define dynamic memory expression | ||||||
|
|
||||||
| You can define a dynamic memory expression in your `actor.json`: | ||||||
|
|
||||||
| ```json | ||||||
| { | ||||||
| "defaultMemoryMbytes": "get(input, 'startUrls.length' * 1024)" | ||||||
| } | ||||||
| ``` | ||||||
|
|
||||||
| - The expression is evaluated _before the run starts_. | ||||||
|
||||||
| - The expression can reference variables from input and run options to calculate memory (e.g., `input.startUrls`, `runOptions.maxItems`). | ||||||
| - The result becomes the default memory for the run, but users can still override it. | ||||||
|
|
||||||
| --- | ||||||
|
|
||||||
| ### Writing Expressions | ||||||
|
||||||
|
|
||||||
| Expressions are based on [MathJS](https://mathjs.org/), extended with custom helper function `get`. | ||||||
|
|
||||||
| #### Variable Access | ||||||
|
||||||
| #### Variable Access | |
| #### Variable access |
This is forbidden by our style-guide. If you use LLMs to write the docs, let it follow the guide 😝
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's make it clear that this is not always rounding up.
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
| urls = get(input, 'startUrls.length', 0); | |
| urlCount = get(input, 'startUrls.length', 0); |
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can use Tabs component here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, refactored it
danpoletaev marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is not correct language declaration for code block. IIRC it's based on Prism.js and in such cases bash is generally used (this is valid for whole document)
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any reason why there are these lines to break the page? we do it once to at the beginning usually to provide TL:DR of what the page is about but do not use it later for page break
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removed
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(opt) Is this needed? Is such a section something that we usually do?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not really, and even if we would do it it shouldn't have been in Title case rather in Sentence case
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed key takeaways
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -39,6 +39,9 @@ As part of the input, you can also specify run options such as [Build](../develo | |||||
| | Timeout | Timeout for the Actor run in seconds. Zero value means there is no timeout. | | ||||||
| | Memory | Amount of memory allocated for the Actor run, in megabytes. | | ||||||
|
|
||||||
| :::info Dynamic memory | ||||||
| If the Actor is configured by developer to use [dynamic memory](../development/actor_definition/dynamic_actor_memory/index.md), the system will calculate the optimal memory allocation based on your input. In this case, the **Memory** option acts as an override - if you set it, the calculated value will be ignored. | ||||||
|
||||||
| If the Actor is configured by developer to use [dynamic memory](../development/actor_definition/dynamic_actor_memory/index.md), the system will calculate the optimal memory allocation based on your input. In this case, the **Memory** option acts as an override - if you set it, the calculated value will be ignored. | |
| If the Actor is configured by developer to use [dynamic memory](../development/actor_definition/dynamic_actor_memory/index.md), the system will calculate the optimal memory allocation based on your input. In this case, the **Memory** option acts as an override — if you set it, the calculated value will be ignored. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This component is only used in the
Buildcomponent, which is only returned from "get build", "build actor" and "abort actor build" endpoints - its that correct?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, correct