Skip to content
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
1c210e1
Initial commit of workflow invocation docs
dylburger Sep 21, 2024
97c5839
Merge remote-tracking branch 'origin/master' into connect/workflow-sd…
dylburger Sep 21, 2024
a9626a6
Merge remote-tracking branch 'origin/master' into connect/workflow-sd…
dylburger Sep 25, 2024
e06e919
removing old docs, adding IP clarification
dylburger Sep 25, 2024
bb5a9e2
Removing broken link checker
dylburger Sep 25, 2024
63ab997
Adding eslint-config-next to try to fix ESLint linting errors
dylburger Sep 25, 2024
cc1a987
Merge remote-tracking branch 'origin/master' into connect/workflow-sd…
dylburger Oct 16, 2024
8cfb804
Adding pnpm lock again
dylburger Oct 16, 2024
c087633
Work so far
dylburger Oct 17, 2024
72921fe
Many more fixes
dylburger Oct 18, 2024
b73b177
Many more docs changes
dylburger Oct 20, 2024
28417e6
Merge remote-tracking branch 'origin/master' into connect/workflow-sd…
dylburger Oct 20, 2024
9b798d5
Many more updates
dylburger Oct 22, 2024
f3bac44
Merge remote-tracking branch 'origin/master' into connect/workflow-sd…
dylburger Oct 22, 2024
b83ada2
Adding the PD SDK
dylburger Oct 24, 2024
cd7ce90
Updating Connect Link quickstart
dannyroosevelt Oct 24, 2024
86e7b51
Some more docs updates
dannyroosevelt Oct 24, 2024
bb694e8
Merge remote-tracking branch 'origin/master' into connect/workflow-sd…
dylburger Oct 28, 2024
3082b0c
Merge branch 'connect/workflow-sdk-docs' of github.com:PipedreamHQ/pi…
dylburger Oct 28, 2024
5d3206d
Adding Apps, Users, Webhooks docs
dylburger Oct 28, 2024
0a12942
Small tweaks
dannyroosevelt Oct 28, 2024
1e2ef10
Adding info re: OAuth clients to Connect docs
dannyroosevelt Oct 29, 2024
d5b256b
More info re: OAuth clients
dannyroosevelt Oct 30, 2024
4516cf0
Updating quickstart
dannyroosevelt Oct 30, 2024
7cfd4e9
Adding /apps APIs, new /accounts params, changing include_credentials…
dylburger Oct 31, 2024
3e2b4f5
Adding /components/search endpoint
dylburger Oct 31, 2024
f5d7c0c
Updating more client references
dylburger Nov 1, 2024
8f7acfe
getAccounts changes
dylburger Nov 1, 2024
0a5c3b6
Small quickstart fix
dylburger Nov 1, 2024
7cb7338
Merge remote-tracking branch 'origin/master' into connect/workflow-sd…
dylburger Nov 1, 2024
6bdfb2c
Docs tweaks
dannyroosevelt Nov 1, 2024
6e1d464
s/oauth app/oauth client
dannyroosevelt Nov 1, 2024
d5df964
Update quickstart.mdx
dannyroosevelt Nov 1, 2024
ce0bdeb
Adding migration guide
dylburger Nov 4, 2024
5c1160c
Merge remote-tracking branch 'origin/master' into connect/workflow-sd…
dylburger Nov 4, 2024
ea3f967
Merge remote-tracking branch 'origin/master' into connect/workflow-sd…
dylburger Nov 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
37 changes: 0 additions & 37 deletions .github/workflows/docs.yaml

This file was deleted.

2 changes: 1 addition & 1 deletion components/ai_textraction/ai_textraction.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion components/air/air.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion components/attractwell/attractwell.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion components/deepimage/deepimage.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion components/fluidforms/fluidforms.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion components/gainsight_nxt/gainsight_nxt.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion components/gainsight_px/gainsight_px.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion components/nerv/nerv.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion components/runpod/runpod.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion components/runware/runware.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion components/teach_n_go/teach_n_go.app.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ export default {
console.log(Object.keys(this.$auth));
},
},
};
};
2 changes: 1 addition & 1 deletion docs-v2/pages/_meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,14 +10,14 @@
"connected-accounts": "Connected Accounts",
"apps": "Integrations",
"connect": {
"title": "Pipedream Connect"

Check failure on line 13 in docs-v2/pages/_meta.json

View workflow job for this annotation

GitHub Actions / Lint Code Base

Missing trailing comma
},
"components": "Components",
"sources": "Sources",
"event-history": "Event History",
"http": "HTTP",
"environment-variables": "Environment Variables",
"rest-api": "API Reference",
"rest-api": "REST API",
"cli": "CLI",
"destinations": "Destinations",
"user-settings": "User and Billing Settings",
Expand All @@ -30,19 +30,19 @@
"title": "Support",
"type": "page",
"href": "https://pipedream.com/support",
"newWindow": true

Check failure on line 33 in docs-v2/pages/_meta.json

View workflow job for this annotation

GitHub Actions / Lint Code Base

Missing trailing comma
},
"pricing-page": {
"title": "Pricing",
"href": "https://pipedream.com/pricing",
"newWindow": true,
"type": "page"

Check failure on line 39 in docs-v2/pages/_meta.json

View workflow job for this annotation

GitHub Actions / Lint Code Base

Missing trailing comma
},
"statuspage": {
"title": "Status",
"type": "page",
"href": "https://status.pipedream.com",
"newWindow": true

Check failure on line 45 in docs-v2/pages/_meta.json

View workflow job for this annotation

GitHub Actions / Lint Code Base

Missing trailing comma
},
"version": {
"title": "Version",
Expand All @@ -50,19 +50,19 @@
"items": {
"v3": {
"title": "Current",
"href": "https://pipedream.com/docs"

Check failure on line 53 in docs-v2/pages/_meta.json

View workflow job for this annotation

GitHub Actions / Lint Code Base

Missing trailing comma
},
"v2": {
"title": "Linear (classic)",
"href": "https://pipedream.com/docs/v2"

Check failure on line 57 in docs-v2/pages/_meta.json

View workflow job for this annotation

GitHub Actions / Lint Code Base

Missing trailing comma
}

Check failure on line 58 in docs-v2/pages/_meta.json

View workflow job for this annotation

GitHub Actions / Lint Code Base

Missing trailing comma
}

Check failure on line 59 in docs-v2/pages/_meta.json

View workflow job for this annotation

GitHub Actions / Lint Code Base

Missing trailing comma
},
"abuse": {
"display": "children"

Check failure on line 62 in docs-v2/pages/_meta.json

View workflow job for this annotation

GitHub Actions / Lint Code Base

Missing trailing comma
},
"airtable": {
"display": "children"

Check failure on line 65 in docs-v2/pages/_meta.json

View workflow job for this annotation

GitHub Actions / Lint Code Base

Missing trailing comma
},
"examples": {
"display": "children"
Expand Down
51 changes: 31 additions & 20 deletions docs-v2/pages/connect/api.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ or a specific version:

## Authentication

### OAuth

TO DO

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

OAuth section needs to be completed.

The OAuth section is currently marked as "TO DO". This important authentication method should be documented to provide a complete reference for users.

Please complete the OAuth section with relevant details and examples.

### TypeScript SDK (Server)

Most of your interactions with the Connect API will happen on the server, to protect API requests and user credentials. You'll use the SDK in [your frontend](#typescript-sdk-browser) to let users connect accounts. Once connected, you'll use the SDK on the server to retrieve credentials, invoke workflows on their behalf, and more.
Expand Down Expand Up @@ -100,21 +104,6 @@ export default function Home() {
}
```

### REST API

You authenticate to the Connect API using **Basic Auth**. Send your project public key as the username and the project secret key as the password. When you make API requests, pass an
`Authorization` header of the following format:

```shell
Authorization: Basic base_64(public_key:secret_key)
```

Clients like `cURL` will often make this easy. For example, here's how you list all accounts on a project:

```shell
curl 'https://api.pipedream.com/v1/connect/accounts' -u public_key:secret_key
```

## External users

When you use the Connect API, you'll pass an `external_id` parameter when initiating account connections and retrieving credentials. This is your user's ID, in your system — whatever you use to uniquely identify them.
Expand All @@ -134,6 +123,32 @@ If you need higher rate limits, please [reach out](https://pipedream.com/support

## API Reference

### Invoke workflows

You can use the SDK to invoke workflows on behalf of your users. Here's an example of invoking a workflow with the SDK:

```typescript
import { createClient } from "@pipedream/sdk";

const pd = createClient({
oauthClientId: "your-oauth-client-id",
oauthClientSecret: "your-oauth-client-secret",
});

const response = await pd.invokeWorkflow(
"https://your-workflow-url.m.pipedream.net",
{
body: {
foo: 123,
bar: "abc",
},
// pass any other fetch options here
},
);
```

`invokeWorkflow` uses Node's `fetch` API, so you can pass any standard options in that object.

### Tokens

Your app will initiate the account connection flow for your end users in your frontend. But you can't expose your project keys in the client, since they'd be accessible to anyone. Instead, on your server, **you exchange your project keys for a short-lived token that allows a specific user to connect a specific app**, and return that token to your frontend.
Expand Down Expand Up @@ -576,11 +591,7 @@ response = client.connect_token_create(connect_token_opts[:external_id])

#### List all accounts

List all connected accounts for all end users within your project

<Callout type="warning">
This endpoint is not currently paginated, so we'll attempt to return all connected accounts for all users within your project. We intend to add pagination soon.
</Callout>
List all connected accounts for all end users within a project.

```
GET /accounts/
Expand Down
11 changes: 3 additions & 8 deletions docs-v2/pages/connect/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,6 @@ import VideoPlayer from "@/components/VideoPlayer";
Please reach out at `[email protected]` or our [Slack community](https://pipedream.com/support) to let us know how you're using it, what's not working, and what else you'd like to see.
</Callout>

<VideoPlayer
src="https://www.youtube.com/embed/xhUagMsogkQ"
title="Pipedream Connect Public Preview"
/>

Pipedream Connect is the easiest way for your users to connect to [over {process.env.PUBLIC_APPS}+ APIs](https://pipedream.com/apps), **right in your product**. You can build in-app messaging, CRM syncs, AI-driven products, [and much more](/connect/use-cases), all in a few minutes. Visit [the quickstart](/connect/quickstart) to build your first integration.

Connect lets your users authorize access to any API, directly in your app. You can then retrieve fresh credentials for any account, making requests on their behalf. Pipedream handles the security of credentials and the whole OAuth flow — **no need to manage authorization grants or token refresh yourself.**
Expand Down Expand Up @@ -55,7 +50,9 @@ Please let us know if you have any feedback on the value of Connect and how you'

## Security

Pipedream takes the security of our products seriously. Please [review our security docs](/privacy-and-security) and send us any questions or [suspected vulnerabilities](/privacy-and-security#reporting-a-vulnerability). You can also get a copy of our [SOC 2 Type 2 report](/privacy-and-security#soc-2), [sign HIPAA BAAs](/privacy-and-security#hipaa), and get information on other practices and controls.
Pipedream takes the security of our products seriously. See details on the security of the Connect product [here](/privacy-and-security#pipedream-connect).

Please also [review our general security docs](/privacy-and-security) and send us any questions or [suspected vulnerabilities](/privacy-and-security#reporting-a-vulnerability). You can also get a copy of our [SOC 2 Type 2 report](/privacy-and-security#soc-2), [sign HIPAA BAAs](/privacy-and-security#hipaa), and get information on other practices and controls.

### Storing user credentials, token refresh

Expand All @@ -71,9 +68,7 @@ All credentials and tokens are sent to Pipedream securely over HTTPS, and encryp
## Product roadmap for Connect

- Address bugs and feedback during the preview phase
- Use Pipedream OAuth clients while in development, to make it easier to get started
- Invoke Pipedream workflows on behalf of your end users
- Improve error handling for Connect developers and end users
- And more!

## Glossary of terms
Expand Down
7 changes: 0 additions & 7 deletions docs-v2/pages/connect/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,6 @@ import VideoPlayer from "@/components/VideoPlayer";

Pipedream Connect is the easiest way for your users to connect to [over {process.env.PUBLIC_APPS}+ APIs](https://pipedream.com/apps), **right in your product**. You can build in-app messaging, CRM syncs, AI-driven products, [and much more](/connect/use-cases), all in a few minutes.

If you prefer videos to text, this demo walks through this quickstart:

<VideoPlayer
src="https://www.youtube.com/embed/xhUagMsogkQ"
title="Pipedream Connect Public Preview"
/>

## Visual overview

Here's a high-level overview of how Connect works with your app:
Expand Down
6 changes: 5 additions & 1 deletion docs-v2/pages/connected-accounts/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,11 @@ If you use a secrets store like [Pipedream Connect](/connect) or [HashiCorp Vaul

## Connecting to apps with IP restrictions

If you're connecting to an app that enforces IP restrictions, you may need to whitelist Pipedream's IP addresses:
<Callout type="info">
These IP addresses are tied to **app connections only**, not workflows or other Pipedream services. To whitelist requests from Pipedream workflows, [use VPCs](/workflows/vpc).
</Callout>

If you're connecting to an app that enforces IP restrictions, you may need to whitelist the Pipedream API's IP addresses:

<pre className="mt-4 nx-bg-primary-700/5 nx-mb-4 nx-overflow-x-auto nx-rounded-xl nx-subpixel-antialiased dark:nx-bg-primary-300/10 nx-text-[.9em] contrast-more:nx-border contrast-more:nx-border-primary-900/20 contrast-more:nx-contrast-150 contrast-more:dark:nx-border-primary-100/40 nx-py-4">
{process.env.PD_EGRESS_IP_RANGE}
Expand Down
11 changes: 5 additions & 6 deletions docs-v2/pages/destinations/http.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -112,13 +112,12 @@ Currently, Pipedream will not retry any failed request. If your HTTP destination

## IP addresses for Pipedream HTTP requests

<Callout type="info">
These IP addresses are tied to **requests sent with `$.send.http` only, not other HTTP requests made from workflows**. To whitelist standard HTTP requests from Pipedream workflows, [use VPCs](/workflows/vpc).
</Callout>

When you make an HTTP request using `$.send.http()`, the traffic will come from one of the following IP addresses:

<PublicIPs />

This list may change over time. If you've previously whitelisted these IP addresses and are having trouble sending HTTP requests to your target service, please check to ensure this list matches your firewall rules.

<Callout type="warning">
These IP addresses are tied specifically to the `$.send.http()` service. If you send traffic directly from a workflow, it will be sent from one of Pipedream's general range of IP addresses. [See our hosting docs for more information](/privacy-and-security/#hosting-details).
</Callout>

This list may change over time. If you've previously whitelisted these IP addresses and are having trouble sending HTTP requests to your target service, please check to ensure this list matches your firewall rules.
16 changes: 7 additions & 9 deletions docs-v2/pages/privacy-and-security/best-practices.mdx
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Security Best Practices

Pipedream implements a range of [privacy and security measures](/privacy-and-security/) meant to protect your data from unauthorized access. Since Pipedream [workflows](/workflows/), [event sources](/sources/), and other resources can run any Node.js code and process any event data, you also have a responsibility to ensure you handle that code and data securely. We've outlined a handful of best practices for that below.
Pipedream implements a range of [privacy and security measures](/privacy-and-security/) meant to protect your data from unauthorized access. Since Pipedream [workflows](/workflows/), [event sources](/sources/), and other resources can run any code and process any event data, you also have a responsibility to ensure you handle that code and data securely. We've outlined a handful of best practices for that below.

## Store secrets as Pipedream connected accounts or environment variables

Even if your workflow code is private, you should never store secrets like API keys in code. These secrets should be stored in one of two ways:
Never store secrets like API keys directly in code. These secrets should be stored in one of two ways:

- [If Pipedream integrates with the app](https://pipedream.com/apps), use [connected accounts](/connected-accounts/) to link your apps / APIs.
- If you need to store credentials for an app Pipedream doesn't support, or you need to store arbitrary configuration data, use [environment variables](/environment-variables/).
Expand All @@ -13,21 +13,19 @@ Read more about how Pipedream secures connected accounts / environment variables

## Deliver data to Pipedream securely

Whenever possible, encrypt data in transit to Pipedream. For example, use HTTPS endpoints when sending HTTP traffic to a workflow.
Always send data over HTTPS to Pipedream endpoints.

## Send data out of Pipedream securely

When you connect to APIs in a workflow, or deliver data to third-party destinations, encrypt that data in transit. For example, use HTTPS endpoints when sending HTTP traffic to third parties.

## Add authentication to incoming event data
## Require authorization for HTTP triggers

You can add many public-facing triggers to your workflows. For example, when you add an HTTP trigger to your workflow, anyone with the associated trigger URL can invoke it. You should protect your workflow with an authentication mechanism like [Basic Auth](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication), JWT, or others.
HTTP triggers are public by default, and require no authorization or token to invoke.

The easiest way to do this is to use the [Validate Webhook Auth action](https://pipedream.com/apps/http/actions/validate-webhook-auth). This supports common auth options, and you don't have to write any code to configure it.
For many workflows, you should [configure authorization](/workflows/triggers#authorizing-http-requests) to ensure that only authorized parties can invoke your HTTP trigger.

If you need to implement custom logic in code, see [this workflow](https://pipedream.com/new?h=tch_OaJfNv) for a shared API key example. This reads the header `x-api-key` and compares it to the [environment variable](/environment-variables/) called `YOUR_WEBHOOK_API_KEY`. If the `x-api-key` header does not match this variable, it returns a `401 Unauthorized` error and exits the workflow early.

This pattern is typical for protecting workflows: add the authentication logic in a step at the top of your workflow, ending early if auth fails. If auth succeeds, Pipedream runs the remaining steps of your workflow.
For third-party services like webhooks, that authorize requests using their own mechanism, use the [Validate Webhook Auth action](https://pipedream.com/apps/http/actions/validate-webhook-auth). This supports common auth options, and you don't have to write any code to configure it.

## Validate signatures for incoming events, where available

Expand Down
Loading
Loading