Skip to content

Commit eba1318

Browse files
ammeekammeek
committed
docs: Add page for kestra integration.
1 parent 00b13f0 commit eba1318

File tree

3 files changed

+131
-0
lines changed

3 files changed

+131
-0
lines changed

‎sources/platform/integrations/index.mdx‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,12 @@ If you use one of the main integration platforms, Apify's support is here for yo
134134
imageUrl="/img/platform/integrations/n8n.svg"
135135
smallImage
136136
/>
137+
<Card
138+
title="Kestra"
139+
to="/platform/integrations/kestra"
140+
imageUrl="/img/platform/integrations/kestra.svg"
141+
smallImage
142+
/>
137143
<Card
138144
title="IFTTT"
139145
to="/platform/integrations/ifttt"

‎sources/platform/integrations/workflows-and-notifications/kestra.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
---
2+
title: Kestra integration
3+
description: Connect Apify with Kestra to orchestrate workflows — run Flows, extract structured data, and react to Actor or task events in real time.
4+
sidebar_label: Kestra
5+
sidebar_position: 7
6+
slug: /integrations/kestra
7+
---
8+
9+
**Connect Apify with Kestra to orchestrate workflows — run Flows, extract structured data, and react to Actor or task events in real time.**
10+
11+
---
12+
13+
[Kestra](https://kestra.io/) is an open-source, event-driven orchestration platform that unifies workflows for all engineers.
14+
With the [Apify plugin for Kestra](https://github.com/apify/kestra-plugin-apify), you can seamlessly connect Apify Actors and storage to your workflows.
15+
Run scrapers, extract structured data — all defined declaratively in YAML and orchestrated directly from the UI.
16+
In this guide, you'll learn how to set up authentication and incorporate it into a Kestra flows.
17+
18+
## Prerequisites
19+
20+
Before you begin, make sure you have:
21+
22+
- An [Apify account](https://console.apify.com/)
23+
- An [Kestra instance](https://kestra.io/docs/getting-started/quickstart) (self‑hosted or cloud)
24+
25+
Once installed, the next step is authentication.
26+
27+
## Authentication
28+
29+
The Apify plugin for Kestra uses **API Key authentication** to securely connect with your Apify account.
30+
This method works for both self-hosted and cloud instances.
31+
You can store and manage your API key directly in [Kestra Secrets](https://kestra.io/docs/concepts/secret) in the UI or through environment variables, ensuring
32+
credentials stay secure and flows remain declarative.
33+
34+
With authentication configured, you can now define flows that integrate the Apify plugin — orchestrating scrapers, data extraction, and event-driven tasks directly from Kestra.
35+
36+
## Create a flow with the Apify Plugin
37+
38+
Start by building a basic flow in Kestra,
39+
then add any Apify Task to handle jobs like running Actors
40+
or fetching data.
41+
42+
1. Create a new flow in Kestra.
43+
1. Select **Add Node**, search for **Apify**, and select it.
44+
1. Choose the desired **Resource** and **Operation**.
45+
1. In the node's **Credentials** dropdown, choose the Apify credential you configured earlier. If you haven't configured any credentials, you can do so in this step. The process will be the same.
46+
1. You can now use Apify node as a trigger or action in your workflow.
47+
48+
## Use Apify Tasks as an action
49+
50+
Tasks allow you to perform operations like running an Actor within a workflow.
51+
52+
1. Create a new flow.
53+
1. Inside the **Flow code** tab change the hello tasks type to be **io.kestra.plugin.apify.actor.Run**.
54+
1. Change the tasks id to be **run_apify_actor**
55+
1. Remove the message property.
56+
1. Configure the **run_apify_actor** task by adding your required values for the properties listed below:
57+
- **actor id**: Actor ID or a tilde-separated owner's username and Actor name.
58+
- **apiToken**: A reference to the secret value you set up earlier. For example "\{\{secret(namespace=flow.namespace, key='APIFY_API_KEY')\}\}"
59+
1. Add a new task below the **run_apify_actor** with an ID of **get_dataset** and a type of **io.kestra.plugin.apify.dataset.Get**.:
60+
1. Configure the **get_dataset** to fetch the dataset generated by the **run_apify_actor** task by configuring the following values:
61+
- **datasetId**: The Id of the dataset to fetch. You can use the value from the previous task using the following syntax: "\{\{secret(namespace=flow.namespace, key='APIFY_API_KEY')\}\}"
62+
- **apiToken**: A reference to the secret value you set up earlier. For example '\{\{secret(namespace=flow.namespace, key='APIFY_API_KEY')\}\}'
63+
- **input**: Input for the Actor run. The input is optional and can be used to pass data to the Actor. For our example we will add 'hashtags: ["fyp"]'
64+
- **maxItems**: The maximum number of items to fetch from the dataset. For our example we will set this to 5.
65+
1. Now add the final task to log the output of the dataset. Add a new task below the **log_output** with an ID of **log_output** and a type of **io.kestra.plugin.core.log.Log**.
66+
1. Configure the **log_output** task to log the output of the dataset by configuring the following values:
67+
- **message** The message to log. You can use the value from the previous task using the following syntax: '\{\{outputs.get_dataset.dataset\}\}'
68+
1. Now save and run your flow.
69+
70+
Your completed template should match the template below.
71+
```yaml
72+
id: get_last_actor_run_and_get_dataset
73+
namespace: company.team
74+
75+
tasks:
76+
- id: run_actor
77+
type: io.kestra.plugin.apify.actor.Run
78+
actorId: GdWCkxBtKWOsKjdch
79+
maxItems: 5
80+
input:
81+
hashtags: ["fyp"]
82+
apiToken: "{{secret(namespace=flow.namespace, key='APIFY_API_KEY')}}"
83+
- id: log_get_last_run_results
84+
type: io.kestra.plugin.core.log.Log
85+
message: '{{outputs.run_actor}}'
86+
- id: get_data_set_raw
87+
type: io.kestra.plugin.apify.dataset.Get
88+
datasetId: '{{outputs.run_actor.defaultDatasetId}}'
89+
apiToken: "{{secret(namespace=flow.namespace, key='APIFY_API_KEY')}}"
90+
- id: log_get_data_set_raw_results
91+
type: io.kestra.plugin.core.log.Log
92+
message: '{{outputs.get_data_set_raw}}'
93+
```
94+
95+
## Resources
96+
97+
- [Kestra Apify Plugin](https://kestra.io/plugins/plugin-apify)
98+
- [Apify API Documentation](https://docs.apify.com)
99+
- [Kestra Documentation](https://kestra.io/docs)
100+
101+
## Troubleshooting
102+
103+
If you encounter issues, start by double-checking basics.
104+
105+
- **Authentication errors**: Verify your API token in [Secrets](https://kestra.io/docs/concepts/secret).
106+
- **Operation failures**: Check input parameters, JSON syntax, and resource IDs in your Apify account.
107+
108+
Feel free to explore other resources and contribute to the integration on [GitHub](https://github.com/kestra-io/plugin-apify).

‎static/img/platform/integrations/kestra.svg‎

Lines changed: 17 additions & 0 deletions
Loading

0 commit comments

Comments
 (0)