Skip to content

Commit d72c712

Browse files
Add documentation for Data Store TTL functionality
1 parent bfccd6c commit d72c712

File tree

3 files changed

+110
-1
lines changed

3 files changed

+110
-1
lines changed

docs-v2/pages/workflows/building-workflows/code/nodejs/using-data-stores.mdx

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,48 @@ export default defineComponent({
5151
});
5252
```
5353

54+
### Setting expiration (TTL) for records
55+
56+
You can set an expiration time for a record by passing a TTL (Time-To-Live) option as the third argument to the `set` method. The TTL value is specified in seconds:
57+
58+
```javascript
59+
export default defineComponent({
60+
props: {
61+
data: { type: "data_store" },
62+
},
63+
async run({ steps, $ }) {
64+
// Store a temporary value that will expire after 1 hour (3600 seconds)
65+
await this.data.set("temporaryToken", "abc123", { ttl: 3600 });
66+
67+
// Store a value that will expire after 1 day
68+
await this.data.set("dailyMetric", 42, { ttl: 86400 });
69+
},
70+
});
71+
```
72+
73+
When the TTL period elapses, the record will be automatically deleted from the data store.
74+
75+
### Updating TTL for existing records
76+
77+
You can update the TTL for an existing record using the `setTtl` method:
78+
79+
```javascript
80+
export default defineComponent({
81+
props: {
82+
data: { type: "data_store" },
83+
},
84+
async run({ steps, $ }) {
85+
// Update an existing record to expire after 30 minutes
86+
await this.data.setTtl("temporaryToken", 1800);
87+
88+
// Remove expiration from a record
89+
await this.data.setTtl("temporaryToken", null);
90+
},
91+
});
92+
```
93+
94+
This is useful for extending the lifetime of temporary data or removing expiration from records that should now be permanent.
95+
5496
## Retrieving keys
5597

5698
Fetch all the keys in a given Data Store using the `keys` method:
@@ -242,6 +284,46 @@ export default defineComponent({
242284
});
243285
```
244286
287+
## TTL use case: temporary caching and rate limiting
288+
289+
TTL functionality is particularly useful for implementing temporary caching and rate limiting. Here's an example of a simple rate limiter that prevents a user from making more than 5 requests per hour:
290+
291+
```javascript
292+
export default defineComponent({
293+
props: {
294+
data: { type: "data_store" },
295+
},
296+
async run({ steps, $ }) {
297+
const userId = steps.trigger.event.userId;
298+
const rateKey = `ratelimit:${userId}`;
299+
300+
// Try to get current rate limit counter
301+
let requests = await this.data.get(rateKey);
302+
303+
if (requests === undefined) {
304+
// First request from this user in the time window
305+
await this.data.set(rateKey, 1, { ttl: 3600 }); // Expire after 1 hour
306+
return { allowed: true, remaining: 4 };
307+
}
308+
309+
if (requests >= 5) {
310+
// Rate limit exceeded
311+
return { allowed: false, error: "Rate limit exceeded", retryAfter: "1 hour" };
312+
}
313+
314+
// Increment the counter
315+
await this.data.set(rateKey, requests + 1);
316+
return { allowed: true, remaining: 4 - requests };
317+
},
318+
});
319+
```
320+
321+
This pattern can be extended for various temporary caching scenarios like:
322+
- Session tokens with automatic expiration
323+
- Short-lived feature flags
324+
- Temporary access grants
325+
- Time-based promotional codes
326+
245327
## Data store limitations
246328
247329
Data Stores are only currently available in Node.js and Python steps. They are not yet available [Bash](/workflows/building-workflows/code/bash/) or [Go](/workflows/building-workflows/code/go/).

docs-v2/pages/workflows/building-workflows/code/python/using-data-stores.mdx

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,14 @@ def handler(pd: "pipedream"):
4242
data_store["last_ran_at"] = datetime.now().isoformat()
4343
```
4444

45+
### Setting expiration (TTL) for records
46+
47+
<Callout type="info">
48+
TTL functionality is currently only available in Node.js code steps. If you need to set expiration on records from Python steps, you'll need to use a Node.js step to set or update the TTL.
49+
</Callout>
50+
51+
For temporary data that should automatically expire, you can use a Node.js step with TTL functionality. See the [Node.js documentation](/workflows/building-workflows/code/nodejs/using-data-stores/#setting-expiration-ttl-for-records) for details on how to use TTL with data stores.
52+
4553
## Retrieving keys
4654

4755
Fetch all the keys in a given data store using the `keys` method:

docs-v2/pages/workflows/data-management/data-stores.mdx

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,10 @@ import VideoPlayer from "@/components/VideoPlayer";
1313
Data stores are useful for:
1414

1515
- Storing and retrieving data at a specific key
16+
- Setting automatic expiration times for temporary data (TTL)
1617
- Counting or summing values over time
1718
- Retrieving JSON-serializable data across workflow executions
18-
- Caching
19+
- Caching and rate limiting
1920
- And any other case where you'd use a key-value store
2021

2122
You can connect to the same data store across workflows, so they're also great for sharing state across different services.
@@ -41,6 +42,7 @@ Configure the action:
4142
1. **Select or create a Data Store** — create a new data store or choose an existing data store.
4243
2. **Key** - the unique ID for this data that you'll use for lookup later
4344
3. **Value** - The data to store at the specified `key`
45+
4. **Time to Live (TTL)** - (Optional) The number of seconds until this record expires and is automatically deleted. Leave blank for records that should not expire.
4446

4547
![Configure the action](/images/data-stores/configuring-data-store-update-action.png)
4648

@@ -74,6 +76,23 @@ Configure the action:
7476

7577
![Get record action](/images/data-stores/configure-data-store-retrieve-record.png)
7678

79+
### Setting or updating record expiration (TTL)
80+
81+
You can set automatic expiration times for records using the **Update record expiration** action:
82+
83+
1. Add a new step to your workflow.
84+
2. Search for the **Data Stores** app and select it.
85+
3. Select the **Update record expiration** pre-built action.
86+
87+
Configure the action:
88+
89+
1. **Select a Data Store** - select the data store containing the record to modify
90+
2. **Key** - the key for the record you want to update the expiration for
91+
3. **Expiration Type** - choose from preset expiration times (1 hour, 1 day, 1 week, etc.) or select "Custom value" to enter a specific time in seconds
92+
4. **Custom TTL (seconds)** - (only if "Custom value" is selected) enter the number of seconds until the record expires
93+
94+
To remove expiration from a record, select "No expiration" as the expiration type.
95+
7796
### Deleting Data
7897

7998
To delete a single record from your data store, use the **Delete a single record** action in a step:

0 commit comments

Comments
 (0)