Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,48 @@ export default defineComponent({
});
```

### Setting expiration (TTL) for records

You can set an expiration time for a record by passing a TTL (Time-To-Live) option as the third argument to the `set` method. The TTL value is specified in seconds:

```javascript
export default defineComponent({
props: {
data: { type: "data_store" },
},
async run({ steps, $ }) {
// Store a temporary value that will expire after 1 hour (3600 seconds)
await this.data.set("temporaryToken", "abc123", { ttl: 3600 });

// Store a value that will expire after 1 day
await this.data.set("dailyMetric", 42, { ttl: 86400 });
},
});
```

When the TTL period elapses, the record will be automatically deleted from the data store.

### Updating TTL for existing records

You can update the TTL for an existing record using the `setTtl` method:

```javascript
export default defineComponent({
props: {
data: { type: "data_store" },
},
async run({ steps, $ }) {
// Update an existing record to expire after 30 minutes
await this.data.setTtl("temporaryToken", 1800);

// Remove expiration from a record
await this.data.setTtl("temporaryToken", null);
},
});
```

This is useful for extending the lifetime of temporary data or removing expiration from records that should now be permanent.

## Retrieving keys

Fetch all the keys in a given Data Store using the `keys` method:
Expand Down Expand Up @@ -242,6 +284,46 @@ export default defineComponent({
});
```

## TTL use case: temporary caching and rate limiting

TTL functionality is particularly useful for implementing temporary caching and rate limiting. Here's an example of a simple rate limiter that prevents a user from making more than 5 requests per hour:

```javascript
export default defineComponent({
props: {
data: { type: "data_store" },
},
async run({ steps, $ }) {
const userId = steps.trigger.event.userId;
const rateKey = `ratelimit:${userId}`;

// Try to get current rate limit counter
let requests = await this.data.get(rateKey);

if (requests === undefined) {
// First request from this user in the time window
await this.data.set(rateKey, 1, { ttl: 3600 }); // Expire after 1 hour
return { allowed: true, remaining: 4 };
}

if (requests >= 5) {
// Rate limit exceeded
return { allowed: false, error: "Rate limit exceeded", retryAfter: "1 hour" };
}

// Increment the counter
await this.data.set(rateKey, requests + 1);
return { allowed: true, remaining: 4 - requests };
},
});
```

This pattern can be extended for various temporary caching scenarios like:
- Session tokens with automatic expiration
- Short-lived feature flags
- Temporary access grants
- Time-based promotional codes

## Data store limitations

Data Stores are only currently available in Node.js and Python steps. They are not yet available [Bash](/workflows/building-workflows/code/bash/) or [Go](/workflows/building-workflows/code/go/).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,14 @@ def handler(pd: "pipedream"):
data_store["last_ran_at"] = datetime.now().isoformat()
```

### Setting expiration (TTL) for records

<Callout type="info">
TTL functionality is currently only available in Node.js code steps. If you need to set expiration on records from Python steps, you'll need to use a Node.js step to set or update the TTL.
</Callout>

For temporary data that should automatically expire, you can use a Node.js step with TTL functionality. See the [Node.js documentation](/workflows/building-workflows/code/nodejs/using-data-stores/#setting-expiration-ttl-for-records) for details on how to use TTL with data stores.

## Retrieving keys

Fetch all the keys in a given data store using the `keys` method:
Expand Down
21 changes: 20 additions & 1 deletion docs-v2/pages/workflows/data-management/data-stores.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,10 @@ import VideoPlayer from "@/components/VideoPlayer";
Data stores are useful for:

- Storing and retrieving data at a specific key
- Setting automatic expiration times for temporary data (TTL)
- Counting or summing values over time
- Retrieving JSON-serializable data across workflow executions
- Caching
- Caching and rate limiting
- And any other case where you'd use a key-value store

You can connect to the same data store across workflows, so they're also great for sharing state across different services.
Expand All @@ -41,6 +42,7 @@ Configure the action:
1. **Select or create a Data Store** — create a new data store or choose an existing data store.
2. **Key** - the unique ID for this data that you'll use for lookup later
3. **Value** - The data to store at the specified `key`
4. **Time to Live (TTL)** - (Optional) The number of seconds until this record expires and is automatically deleted. Leave blank for records that should not expire.

![Configure the action](/images/data-stores/configuring-data-store-update-action.png)

Expand Down Expand Up @@ -74,6 +76,23 @@ Configure the action:

![Get record action](/images/data-stores/configure-data-store-retrieve-record.png)

### Setting or updating record expiration (TTL)

You can set automatic expiration times for records using the **Update record expiration** action:

1. Add a new step to your workflow.
2. Search for the **Data Stores** app and select it.
3. Select the **Update record expiration** pre-built action.

Configure the action:

1. **Select a Data Store** - select the data store containing the record to modify
2. **Key** - the key for the record you want to update the expiration for
3. **Expiration Type** - choose from preset expiration times (1 hour, 1 day, 1 week, etc.) or select "Custom value" to enter a specific time in seconds
4. **Custom TTL (seconds)** - (only if "Custom value" is selected) enter the number of seconds until the record expires

To remove expiration from a record, select "No expiration" as the expiration type.

### Deleting Data

To delete a single record from your data store, use the **Delete a single record** action in a step:
Expand Down
9 changes: 3 additions & 6 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading