|
| 1 | +--- |
| 2 | +title: 'Persistent Storage and 97% Faster Cold Starts for Edge Functions' |
| 3 | +description: 'Mount S3-compatible buckets as persistent file storage in Edge Functions with up to 97% faster cold start times.' |
| 4 | +categories: |
| 5 | + - product |
| 6 | + - launch-week |
| 7 | + - edge-functions |
| 8 | +tags: |
| 9 | + - launch-week |
| 10 | + - edge-functions |
| 11 | + - storage |
| 12 | +date: '2025-07-18:00:00' |
| 13 | +toc_depth: 3 |
| 14 | +author: laktek,nyannyacha |
| 15 | +image: launch-week-15/day-5-persistent-storage-for-functions/og.jpg |
| 16 | +thumb: launch-week-15/day-5-persistent-storage-for-functions/thumb.png |
| 17 | +launchweek: '15' |
| 18 | +--- |
| 19 | + |
| 20 | +Today, we are introducing Persistent Storage and up to 97% faster cold start times for Edge Functions. Previously, Edge Functions only supported ephemeral file storage by writing to `/tmp` directory. Many common libraries for performing tasks, such as zipping/unzipping files and image transformations, are built to work with persistent file storage, so making them work with Edge Functions required extra steps. |
| 21 | + |
| 22 | +The persistent storage option is built on top of the S3 protocol. It allows you to mount any [S3-compatible bucket](https://supabase.com/docs/guides/storage/s3/compatibility), including [Supabase Storage Buckets](https://supabase.com/docs/guides/storage), as a directory for your Edge Functions. You can perform operations such as reading and writing files to the mounted buckets as you would in a POSIX file system. |
| 23 | + |
| 24 | +```tsx |
| 25 | +// read from S3 bucket |
| 26 | +const data = await Deno.readFile('/s3/my-bucket/results.csv') |
| 27 | + |
| 28 | +// make a directory |
| 29 | +await Deno.mkdir('/s3/my-bucket/sub-dir') |
| 30 | + |
| 31 | +// write to S3 bucket |
| 32 | +await Deno.writeTextFile('/s3/my-bucket/demo.txt', 'hello world') |
| 33 | +``` |
| 34 | + |
| 35 | +<div className="video-container mb-8"> |
| 36 | + <iframe |
| 37 | + className="w-full" |
| 38 | + src="https://www.youtube-nocookie.com/embed/h3mQrDC4g14" |
| 39 | + title="Persistent Storage and Faster Boot Times for Edge Functions" |
| 40 | + allow="accelerometer; autoplay; clipboard-write; encrypted-media; fullscreen; gyroscope; picture-in-picture; web-share" |
| 41 | + allowfullscreen |
| 42 | + /> |
| 43 | +</div> |
| 44 | + |
| 45 | +## How to configure |
| 46 | + |
| 47 | +To access an S3 bucket from Edge Functions, you must set the following as environment variables in Edge Function Secrets. |
| 48 | + |
| 49 | +- `S3FS_ENDPOINT_URL` |
| 50 | +- `S3FS_REGION` |
| 51 | +- `S3FS_ACCESS_KEY_ID` |
| 52 | +- `S3FS_SECRET_ACCESS_KEY` |
| 53 | + |
| 54 | +If you are using Supabase Storage, [follow this guide](https://supabase.com/docs/guides/storage/s3/authentication) to enable and create an access key and id. |
| 55 | + |
| 56 | +## Use Case: SQLite in Edge Functions |
| 57 | + |
| 58 | +The S3 File System simplifies workflows that involve reading and transforming data stored in an S3 bucket. |
| 59 | + |
| 60 | +For example, imagine you are building an IoT app where a device backs up its SQLite database to S3. You can set up a scheduled Edge Function to read this data and then push the data to your primary Postgres database for aggregates and reporting. |
| 61 | + |
| 62 | +```tsx |
| 63 | +// Following example is simplified for readability |
| 64 | + |
| 65 | +import { DB } from "https://deno.land/x/[email protected]/mod.ts"; |
| 66 | +import { supabase } from '../shared/client.ts' |
| 67 | + |
| 68 | +const today = new Date().toISOString().split('T')[0] |
| 69 | +const backupDBPath = `backups/backup-${today}.db` |
| 70 | + |
| 71 | +// Use S3 FS to read the Sqlite DB |
| 72 | +const data = Deno.readFileSync(`/s3/${backupDBPath}`); |
| 73 | + |
| 74 | +// Create an in-memory SQLite from the data downloaded from S3 |
| 75 | +// This is faster than directly reading from S3 |
| 76 | +const db = new DB(); |
| 77 | +db.deserialize(data); |
| 78 | + |
| 79 | +function calculateStats(rows: IoTData[], date: string): StatsSummary { |
| 80 | + // .... |
| 81 | +} |
| 82 | + |
| 83 | +Deno.serve(async (req)=>{ |
| 84 | + // Assuming IoT data is stored in a table called 'sensor_data' |
| 85 | + const rows = db.queryEntries<IoTData>(` |
| 86 | + SELECT * FROM sensor_data |
| 87 | + WHERE date(timestamp) = date('now', 'localtime') |
| 88 | + `) |
| 89 | + |
| 90 | + // Calculate statistics |
| 91 | + const stats = calculateStats(rows, today) |
| 92 | + |
| 93 | + // Insert stats into Supabase |
| 94 | + const { data, error } = await supabase |
| 95 | + .from('iot_daily_stats') |
| 96 | + .insert([stats]) |
| 97 | + |
| 98 | + return new Response("OK); |
| 99 | +}); |
| 100 | + |
| 101 | +``` |
| 102 | +
|
| 103 | +## 97% Faster Function Boot Times, Even Under Load |
| 104 | +
|
| 105 | +Previously, Edge Functions with large dependencies or doing preparation work at the start (e.g., parsing/loading configs, initializing AI models) would incur a noticeable boot delay. Sometimes, these slow neighbors can impact other functions running on the same machine. All JavaScript _workers_ in the Supabase Edge Functions Runtime were cooperatively scheduled on the same [**Tokio thread pool**](https://github.com/tokio-rs/tokio). If one worker had heavy startup logic, such as parsing JavaScript modules or running synchronous operations, it could delay every worker scheduled after. This led to occasional long‑tail latency spikes in high-traffic projects. |
| 106 | +
|
| 107 | +To address this issue, we moved workers which are still performing initial script evaluation onto a dedicated blocking pool. This approach prevents heavy initialization tasks from blocking the Tokio thread, significantly reducing boot time spikes for other functions. |
| 108 | +
|
| 109 | +### The result |
| 110 | +
|
| 111 | +Boot times are now more predictable and wait times for cold starts are now much faster. Here’s a result of a [benchmark](https://github.com/supabase/edge-runtime/blob/develop/k6/specs/mixed.ts) we did to compare boot times before and after these changes. |
| 112 | +
|
| 113 | +| Metric | Before | After | (Delta) | |
| 114 | +| ---------------- | ---------- | --------- | --------- | |
| 115 | +| **Avg** | **870 ms** | **42 ms** | **95 %** | |
| 116 | +| **P95** | 8,502 ms | 86 ms | **99 %** | |
| 117 | +| **P99** | 15,069 ms | 460 ms | **97 %** | |
| 118 | +| **Worst** | 24,300 ms | 1 630 ms | **93 %** | |
| 119 | +| **Spikes > 1 s** | 47 % | 4 % | **43 pp** | |
| 120 | +
|
| 121 | +## Support for Synchronous APIs |
| 122 | +
|
| 123 | +By offloading expensive compute at function boot time onto a separate pool, we were able to enable the use of synchronous File APIs during function boot time. Some libraries only support synchronous File APIs (eg, SQLite), and this would allow you to set them up on Edge Functions before it starts processing requests. |
| 124 | +
|
| 125 | +You can now safely use the following synchronous Deno APIs (and their Node counterparts) _during_ initial script evaluation: |
| 126 | +
|
| 127 | +- Deno.statSync |
| 128 | +- Deno.removeSync |
| 129 | +- Deno.writeFileSync |
| 130 | +- Deno.writeTextFileSync |
| 131 | +- Deno.readFileSync |
| 132 | +- Deno.readTextFileSync |
| 133 | +- Deno.mkdirSync |
| 134 | +- Deno.makeTempDirSync |
| 135 | +- Deno.readDirSync |
| 136 | +
|
| 137 | +**Keep in mind** that the sync APIs are available only during initial script evaluation and aren’t supported in callbacks like HTTP handlers or setTimeout. |
| 138 | +
|
| 139 | +```tsx |
| 140 | +Deno.statSync('...') // ✅ |
| 141 | + |
| 142 | +setTimeout(() => { |
| 143 | + Deno.statSync('...') // 💣 ERROR! Deno.statSync is blocklisted on the current context |
| 144 | +}) |
| 145 | + |
| 146 | +Deno.serve(() => { |
| 147 | + Deno.statSync('...') // 💣 ERROR! Deno.statSync is blocklisted on the current context |
| 148 | +}) |
| 149 | +``` |
| 150 | +
|
| 151 | +## Try it on Preview Today |
| 152 | +
|
| 153 | +These changes will be rolled out along with the Deno 2 upgrade to all clusters within the next 2 weeks. Meanwhile, you can use the Preview cluster if you'd like to try them out today. Please see [this guide](https://github.com/orgs/supabase/discussions/36814) on how to test your functions in Preview cluster. |
0 commit comments