-
Notifications
You must be signed in to change notification settings - Fork 1.3k
(Actions) Add Python expressions support for workflow automation #34148
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from 14 commits
69b8433
ce76d63
9646d93
6731e35
d75efff
6ec0dff
cf9da85
450f593
560ade5
334677b
67cfacd
72fc15b
3a32b8c
c7652d4
27217c7
85c858d
bffc609
9b4fb2e
cd3fec1
3f93261
51b122d
6ad20c6
5b62929
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| --- | ||
| title: Expressions | ||
| description: Use inline expressions and dedicated function/expression actions for custom data transformations in workflows. | ||
| disable_toc: false | ||
| aliases: | ||
| - /service_management/workflows/actions/data_transformation/ | ||
| - /service_management/workflows/expressions | ||
| further_reading: | ||
| - link: "/service_management/workflows/variables/" | ||
| tag: "Documentation" | ||
| text: "Variables and parameters" | ||
| --- | ||
|
|
||
| To learn about language-specific expressions, choose one of the following: | ||
|
|
||
| {{< partial name="actions/expressions.html" >}} | ||
|
|
||
| <br> | ||
|
|
||
| ## Further reading | ||
|
|
||
| {{< partial name="whats-next/whats-next.html" >}} | ||
|
|
||
| <br>Do you have questions or feedback? Join the **#workflows** channel on the [Datadog Community Slack][2]. | ||
|
|
||
| [1]: https://lodash.com/ | ||
| [2]: https://chat.datadoghq.com/ | ||
| [3]: /service_management/workflows/test_and_debug/#test-a-step | ||
| [4]: /service_management/workflows/actions/set_variables |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,90 @@ | ||
| --- | ||
| title: JavaScript Expressions | ||
| description: Capabilities and limits of JavaScript expressions in App Builder | ||
| code_lang: javascript | ||
| type: multi-code-lang | ||
| code_lang_weight: 10 | ||
| --- | ||
|
|
||
| JavaScript (JS) is available in Workflows using inline expressions or through the dedicated JS **Function** and **Expression** actions. | ||
|
|
||
| ## Inline JavaScript expressions | ||
|
|
||
| You can use JS expressions directly in workflow steps to perform a wide range of data transformations without needing to include dedicated JS steps. | ||
|
|
||
| To use an inline expression in your workflow, enclose the expression in `${}`. For example, to convert a string ID (`Trigger.stringId`) to an integer, use `${ parseInt(Trigger.stringId) }`. | ||
|
|
||
| The [Lodash][1] utility library is available in inline expressions. The Lodash underscore prefix (`_`) is optional. For example, both `${ _.toNumber("1") }` and `${ toNumber("1") }` are valid inline expressions. | ||
|
|
||
| ### Examples | ||
|
|
||
| #### Retrieve a timestamp | ||
|
|
||
| The following example uses the Lodash `now()` function inside a **Get hosts total** step to get the count of hosts in the last minute. | ||
|
|
||
| The action uses the following inline expression in the **From** field: | ||
| ```js | ||
| ${ Math.floor(now() / 1000) - 60 } | ||
| ``` | ||
|
|
||
| {{< img src="/service_management/workflows/timestamp.png" alt="An inline expression using the now() lowdash function" style="width:90%;" >}} | ||
|
|
||
| #### Increment a value | ||
|
|
||
| The following example increments the desired capacity inside a **Set desired capacity** step by 1. | ||
|
|
||
| The action uses the following inline expression in the **Desired capacity** field: | ||
| ```js | ||
| ${ Steps.Describe_auto_scaling_group.autoScalingGroup.DesiredCapacity + 1 } | ||
| ``` | ||
|
|
||
| {{< img src="/service_management/workflows/increment.png" alt="An inline expression that increments the desired capacity by one" style="width:90%;" >}} | ||
|
|
||
| ## JavaScript expression actions | ||
|
|
||
| The [Expression](#expression-step) and [Function](#function-step) actions perform custom data transformations within your workflows using JavaScript. Use the values of any context variables available within your workflow as inputs for your JavaScript expressions and functions with the syntax `$.Steps.<step_name>.<variable>`. | ||
|
|
||
| The data returned by these actions can then be referenced in subsequent steps of the workflow. | ||
|
|
||
| You can use an underscore (`_`) to make use of [Lodash][1] in your expression and function steps. For example, to reference the HTTP request status variable (`status`) from the HTTP request step (`Make_request`), you'd use the following context variable: | ||
|
|
||
| ``` | ||
| $.Steps.Make_request.status | ||
| ``` | ||
|
|
||
| And to determine if an array returned by `Array_function` includes the name `Bits`, apply the `_.includes` Lodash function with the following syntax: | ||
|
|
||
| ``` | ||
| _.includes($.Steps.Array_function.data, "Bits") | ||
| ``` | ||
| ### Function step | ||
|
|
||
| The function action allows for variable assignments and complex data transformations requiring multiple expressions. | ||
|
|
||
| To add a function action: | ||
| - In a new workflow, click **Add step** and search for `function`. Select the **Function** action to add it to your workflow. | ||
| - In an existing workflow, click **+** and search for `function`. Select the **Function** action to add it to your workflow. | ||
|
|
||
| #### Write function steps with AI | ||
|
|
||
| You can use Bits AI to help you write the JavaScript for a **Function** step. To use this feature, perform the following steps: | ||
|
|
||
| 1. Add a **Function** step to your workflow. | ||
| 1. Under **General**, in the **Script** field, click **<i class="icon-bits-ai"></i> Write with Bits AI**. | ||
| 1. In the **Describe your transformation script** field, enter a description of what you want your script to do. Click the up arrow (**↑**) to submit your description. | ||
| 1. Choose an option to **Replace script**, **Insert in script**, or **Copy to clipboard**. | ||
| 1. Check the script and change it as necessary to fit your needs. | ||
|
|
||
| ### Expression step | ||
|
|
||
| In most cases, use an inline expression instead of a dedicated expression step. Expression actions accept a single line of code. For example, `[1, 2, 3].filter(x => x < 3)`. Variable assignments are not available in expressions. | ||
|
|
||
| To add an expression action: | ||
| - In a new workflow, click **Add step** and search for `expression`. Select the **Expression** action to add it to your workflow. | ||
| - In an existing workflow, click **+** and search for `expression`. Select the **Expression** action to add it to your workflow. | ||
|
|
||
| In an expression step, execution uses _copies_ of all available variables. Mutating a variable within a step has no effect on the variable's value outside of the step. To assign the result of an expression to a variable, see [Set variables][4]. | ||
|
|
||
| ## Test expressions and functions | ||
|
|
||
| See the test and debug page to learn how to [test a workflow step][3]. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,124 @@ | ||
| --- | ||
| title: Python Expressions | ||
| description: Capabilities and limits of Python expressions in App Builder | ||
| code_lang: python | ||
| type: multi-code-lang | ||
| code_lang_weight: 20 | ||
| --- | ||
|
|
||
| The Python function action allows you to write custom Python scripts for data transformations, parsing, and payload enrichment within your workflows. Python provides additional capabilities beyond JavaScript for certain use cases. | ||
|
|
||
| ## Python environment | ||
|
|
||
| The Python function action runs in a restricted execution environment with the following characteristics: | ||
| • Python version: 3.12.8 | ||
| • Available libraries: In addition to the Python standard library, the following packages are available: | ||
| – rsa (version 4.9) | ||
| – python-dateutil (version 2.8.2) | ||
| • Network access: Restricted | ||
|
|
||
| ## Script structure | ||
| All Python scripts must define a main function that accepts a ctx parameter of type Context. | ||
| Example structure: | ||
|
|
||
| ```python | ||
| from execution_context import Context | ||
OliviaShoup marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| def main(*, ctx: Context): | ||
| # Use ctx to access Trigger or Steps data | ||
| workflow_name = ctx["WorkflowName"] | ||
| return f"Running workflow {workflow_name!r}" | ||
| ``` | ||
|
|
||
| The ctx object provides access to all workflow context variables, similar to the `$` variable in JavaScript expressions. Use dictionary-style access (for example, `ctx["Steps"]["Step_name"]["variable"]`) to reference values from previous steps. | ||
OliviaShoup marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Add a Python function action | ||
|
|
||
| In the workflow canvas: | ||
| 1. Click **+** to add a workflow step. | ||
| 1. Search for `Python`. | ||
| 1. Select the Python action to add it to your workflow. | ||
|
|
||
| ## Write Python scripts with AI | ||
|
|
||
| You can use Bits AI to help write Python scripts inside a workflow step. | ||
|
|
||
| To write a script with Bits AI: | ||
|
|
||
| 1. Add a Python step to your workflow. | ||
| 1. In the **Inputs** section, click **Write with Bits AI**. | ||
| 1. Enter a custom prompt or select one of the sample prompts. | ||
| 1. Optionally, click **Test script** to generate a preview of the workflow step. | ||
| 1. To save the script, click **Accept changes**. To continue editing the script, click **Reject changes**. | ||
| 1. Click the **X** to close the AI module. | ||
| 1. Enter a **Description**. | ||
| 1. Click **Save**. | ||
|
|
||
| ## Script examples | ||
|
|
||
| ### Parse and transform JSON data | ||
|
|
||
| This example parses a JSON string from a previous step and extracts specific fields. | ||
|
|
||
| ```python | ||
| from execution_context import Context | ||
| import json | ||
|
|
||
| def main(*, ctx: Context): | ||
| # Get JSON string from previous step | ||
| json_string = ctx["Steps"]["Get_data"]["output"] | ||
|
|
||
| # Parse and transform | ||
| data = json.loads(json_string) | ||
| return { | ||
| "user_ids": [user["id"] for user in data["users"]], | ||
| "total_count": len(data["users"]) | ||
| } | ||
| ``` | ||
|
|
||
| ### Work with dates and timestamps | ||
|
|
||
| This example uses the python-dateutil library to perform date calculations. | ||
|
|
||
| ```python | ||
| from execution_context import Context | ||
| from dateutil import parser, relativedelta | ||
| from datetime import datetime | ||
|
|
||
| def main(*, ctx: Context): | ||
| # Parse a date string | ||
| start_date = parser.parse(ctx["Trigger"]["date_string"]) | ||
|
|
||
| # Calculate date 30 days in the future | ||
| future_date = start_date + relativedelta.relativedelta(days=30) | ||
|
|
||
| return { | ||
| "start": start_date.isoformat(), | ||
| "end": future_date.isoformat(), | ||
| "days_difference": 30 | ||
| } | ||
| ``` | ||
|
|
||
| ### Cryptographic operations | ||
|
|
||
| This example uses the rsa library to encrypt a message. | ||
|
|
||
| ```python | ||
| from execution_context import Context | ||
| import rsa | ||
| import base64 | ||
|
|
||
| def main(*, ctx: Context): | ||
| # Get message from workflow context | ||
| message = ctx["Steps"]["Compose_message"]["text"] | ||
|
|
||
| # Generate RSA key pair | ||
| (public_key, private_key) = rsa.newkeys(512) | ||
|
|
||
| # Encrypt message | ||
| encrypted = rsa.encrypt(message.encode(), public_key) | ||
|
|
||
| return { | ||
| "encrypted_message": base64.b64encode(encrypted).decode(), | ||
| "public_key": public_key.save_pkcs1().decode() | ||
| } | ||
| ``` | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| {{ $dot := . }} | ||
| <div class="apm-languages"> | ||
| <div class="container cards-dd"> | ||
| <div class="row row-cols-2 row-cols-sm-3 g-3 justify-content-center"> | ||
| <div class="col"> | ||
| <a class="card h-100" href="javascript"> | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These cards feels a bit big and less nice in the doc. Wondering if we could have something shorter. The main issue is that because the cards are large, the icons displayed inside are kinda blurry. Also on hover, the shadow seems a bit overwhelming.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. great point, thanks for catching this! i'll ask and find out
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i've adjusted them 🫡 |
||
| <div class="card-body text-center py-2 px-1"> | ||
| {{ partial "img.html" (dict "root" . "src" "integrations_logos/javascript.png" "class" "img-fluid" "alt" "JavaScript" "width" "400") }} | ||
| </div> | ||
| </a> | ||
| </div> | ||
| <div class="col"> | ||
| <a class="card h-100" href="python"> | ||
| <div class="card-body text-center py-2 px-1"> | ||
| {{ partial "img.html" (dict "root" . "src" "integrations_logos/python.png" "class" "img-fluid" "alt" "Python" "width" "400") }} | ||
| </div> | ||
| </a> | ||
| </div> | ||
| <!-- Conditional Android and iOS cards (no compatibility or library_config pages for these as of 12/22/23) --> | ||
| {{ $currentURL := .Page.Permalink }} | ||
| {{ if and (not (in $currentURL "/compatibility/")) (not (in $currentURL "/library_config/")) }} | ||
| {{ end }} | ||
| </div> | ||
| </div> | ||
| </div> | ||
OliviaShoup marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Uh oh!
There was an error while loading. Please reload this page.