Remove task snapshots from persistence #89412

lukesandberg · 2026-02-02T20:35:57Z

Don't clone TaskStorage objects when snapshotting the database

What

Instead of cloning TaskStorage objects when preparing to snapshot we directly encode the reference in the backend storage map.

Why?

From a performance profile of save_snapshot i observed that ~22% of the time was spent in clone_meta_snapshot or clone_data_snapshot, while ~28% of the time was spent in encode_task_data. The whole reason we are cloning is so we can drop the lock on the dashmap shard while encoding. In principle this makes sense, however:

we are only holding read locks on the dashmap so we don't block other encoding threads
holding the read locks for slightly longer will just slow down racing writing threads, but those are rare and already incur a high cost due to the extra cloning in track_modification. So slowing them down isn't too risky.

For a site like vercel-site there are ~8M tasks and 65K shards, so each shard has ~128 items. So the worst case would be waiting for them all to flush and 128 isn't so many that the task would be blocked for a very long time.

So this should be a pure win, reduced allocations in the encoding path.

lukesandberg · 2026-02-02T20:36:18Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

Remove task snapshots from persistence #89412 👈 (View in Graphite)
Optimize how we track data for persistence #89370
canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

turbopack/crates/turbo-tasks-macros/src/derive/task_storage_macro.rs

codspeed-hq · 2026-02-02T20:56:22Z

CodSpeed Performance Report

Merging this PR will not alter performance

_{Comparing better_task_storage_snapshotting (5e371ae) with optimized_change_tracking (b3d4105)¹}

Summary

✅ 17 untouched benchmarks
⏩ 3 skipped benchmarks²

No successful run was found on optimized_change_tracking (4dada51) during the generation of this report, so d6591b6 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
3 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

nextjs-bot · 2026-02-02T23:44:04Z

Stats from current PR

✅ No significant changes detected

📊 All Metrics

📖 Metrics Glossary

Dev Server Metrics:

Listen = TCP port starts accepting connections
First Request = HTTP server returns successful response
Cold = Fresh build (no cache)
Warm = With cached build artifacts

Build Metrics:

Fresh = Clean build (no .next directory)
Cached = With existing .next directory

Change Thresholds:

Time: Changes < 50ms AND < 10%, OR < 2% are insignificant
Size: Changes < 1KB AND < 1% are insignificant
All other changes are flagged to catch regressions

⚡ Dev Server

Metric	Canary	PR	Change	Trend
Cold (Listen)	455ms	455ms	✓	▁▁▁▁▁
Cold (Ready in log)	439ms	437ms	✓	▂▂▂▂▁
Cold (First Request)	1.159s	1.193s	✓	▅▅▅▄▁
Warm (Listen)	457ms	457ms	✓	▁▁▁▁▁
Warm (Ready in log)	441ms	443ms	✓	▁▁▁▁▁
Warm (First Request)	343ms	341ms	✓	▁▁▂▂▁

📦 Dev Server (Webpack) (Legacy)

📦 Dev Server (Webpack)

Metric	Canary	PR	Change	Trend
Cold (Listen)	455ms	455ms	✓	▁█▅▅█
Cold (Ready in log)	438ms	437ms	✓	▇▆█▇█
Cold (First Request)	1.767s	1.781s	✓	▄▃▅▄▆
Warm (Listen)	457ms	456ms	✓	▅▅▅▅█
Warm (Ready in log)	438ms	436ms	✓	▇▆▆▆█
Warm (First Request)	1.769s	1.766s	✓	▅▄▆▄▇

⚡ Production Builds

Metric	Canary	PR	Change	Trend
Fresh Build	3.964s	4.004s	✓	▁▁▁▁▃
Cached Build	4.008s	3.868s	✓	▁▁▁▁▃

📦 Production Builds (Webpack) (Legacy)

📦 Production Builds (Webpack)

Metric	Canary	PR	Change	Trend
Fresh Build	13.876s	13.800s	✓	▁▁▃▁▅
Cached Build	13.883s	13.910s	✓	▁▁▃▁▅
node_modules Size	464 MB	464 MB	✓	▁▁▁▁▁

📦 Bundle Sizes

Bundle Sizes

⚡ Turbopack

Client

Main Bundles: **434 kB** → **435 kB** ⚠️ +131 B

81 files with content-based hashes (individual files not comparable between builds)

Server

Middleware

	Canary	PR	Change
middleware-b..fest.js gzip	763 B	764 B	✓
Total	763 B	764 B	⚠️ +1 B

Build Details

Build Manifests

	Canary	PR	Change
_buildManifest.js gzip	449 B	451 B	✓
Total	449 B	451 B	⚠️ +2 B

📦 Webpack

Client

Main Bundles

	Canary	PR	Change
5528-HASH.js gzip	5.47 kB	N/A	-
6280-HASH.js gzip	54.5 kB	N/A	-
6335.HASH.js gzip	169 B	N/A	-
912-HASH.js gzip	4.53 kB	N/A	-
e8aec2e4-HASH.js gzip	62.5 kB	N/A	-
framework-HASH.js gzip	59.7 kB	59.7 kB	✓
main-app-HASH.js gzip	254 B	253 B	✓
main-HASH.js gzip	39 kB	39.1 kB	✓
webpack-HASH.js gzip	1.68 kB	1.68 kB	✓
262-HASH.js gzip	N/A	4.52 kB	-
2889.HASH.js gzip	N/A	169 B	-
5602-HASH.js gzip	N/A	5.48 kB	-
6948ada0-HASH.js gzip	N/A	62.5 kB	-
9544-HASH.js gzip	N/A	55.2 kB	-
Total	228 kB	229 kB	⚠️ +796 B

Polyfills

	Canary	PR	Change
polyfills-HASH.js gzip	39.4 kB	39.4 kB	✓
Total	39.4 kB	39.4 kB	✓

Pages

	Canary	PR	Change
_app-HASH.js gzip	194 B	194 B	✓
_error-HASH.js gzip	183 B	180 B	🟢 3 B (-2%)
css-HASH.js gzip	331 B	330 B	✓
dynamic-HASH.js gzip	1.81 kB	1.81 kB	✓
edge-ssr-HASH.js gzip	256 B	256 B	✓
head-HASH.js gzip	351 B	352 B	✓
hooks-HASH.js gzip	384 B	383 B	✓
image-HASH.js gzip	580 B	581 B	✓
index-HASH.js gzip	260 B	260 B	✓
link-HASH.js gzip	2.49 kB	2.49 kB	✓
routerDirect..HASH.js gzip	320 B	319 B	✓
script-HASH.js gzip	386 B	386 B	✓
withRouter-HASH.js gzip	315 B	315 B	✓
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Total	7.97 kB	7.97 kB	✅ -1 B

Server

Edge SSR

	Canary	PR	Change
edge-ssr.js gzip	126 kB	126 kB	✓
page.js gzip	249 kB	249 kB	✓
Total	375 kB	375 kB	⚠️ +390 B

Middleware

	Canary	PR	Change
middleware-b..fest.js gzip	616 B	613 B	✓
middleware-r..fest.js gzip	156 B	155 B	✓
middleware.js gzip	33.1 kB	33.1 kB	✓
edge-runtime..pack.js gzip	842 B	842 B	✓
Total	34.7 kB	34.8 kB	⚠️ +33 B

Build Details

Build Manifests

	Canary	PR	Change
_buildManifest.js gzip	732 B	736 B	✓
Total	732 B	736 B	⚠️ +4 B

Build Cache

	Canary	PR	Change
0.pack gzip	3.8 MB	3.81 MB	🔴 +14.4 kB (+0%)
index.pack gzip	102 kB	104 kB	🔴 +1.87 kB (+2%)
index.pack.old gzip	102 kB	102 kB	✓
Total	4 MB	4.02 MB	⚠️ +16 kB

🔄 Shared (bundler-independent)

Runtimes

	Canary	PR	Change
app-page-exp...dev.js gzip	311 kB	311 kB	✓
app-page-exp..prod.js gzip	166 kB	166 kB	✓
app-page-tur...dev.js gzip	311 kB	311 kB	✓
app-page-tur..prod.js gzip	166 kB	166 kB	✓
app-page-tur...dev.js gzip	308 kB	308 kB	✓
app-page-tur..prod.js gzip	164 kB	164 kB	✓
app-page.run...dev.js gzip	308 kB	308 kB	✓
app-page.run..prod.js gzip	164 kB	164 kB	✓
app-route-ex...dev.js gzip	70.4 kB	70.5 kB	✓
app-route-ex..prod.js gzip	48.9 kB	49 kB	✓
app-route-tu...dev.js gzip	70.4 kB	70.5 kB	✓
app-route-tu..prod.js gzip	49 kB	49 kB	✓
app-route-tu...dev.js gzip	70 kB	70.1 kB	✓
app-route-tu..prod.js gzip	48.7 kB	48.8 kB	✓
app-route.ru...dev.js gzip	70 kB	70.1 kB	✓
app-route.ru..prod.js gzip	48.7 kB	48.7 kB	✓
dist_client_...dev.js gzip	324 B	324 B	✓
dist_client_...dev.js gzip	326 B	326 B	✓
dist_client_...dev.js gzip	318 B	318 B	✓
dist_client_...dev.js gzip	317 B	317 B	✓
pages-api-tu...dev.js gzip	43.1 kB	43.2 kB	✓
pages-api-tu..prod.js gzip	32.9 kB	32.9 kB	✓
pages-api.ru...dev.js gzip	43.1 kB	43.2 kB	✓
pages-api.ru..prod.js gzip	32.8 kB	32.9 kB	✓
pages-turbo....dev.js gzip	52.4 kB	52.5 kB	✓
pages-turbo...prod.js gzip	39.4 kB	39.4 kB	✓
pages.runtim...dev.js gzip	52.4 kB	52.5 kB	✓
pages.runtim..prod.js gzip	39.3 kB	39.4 kB	✓
server.runti..prod.js gzip	62.6 kB	62.6 kB	✓
Total	2.77 MB	2.78 MB	⚠️ +1.01 kB

📝 Changed Files (25 files)

Files with changes:

app-page-exp..ntime.dev.js
app-page-exp..time.prod.js
app-page-tur..ntime.dev.js
app-page-tur..time.prod.js
app-page-tur..ntime.dev.js
app-page-tur..time.prod.js
app-page.runtime.dev.js
app-page.runtime.prod.js
app-route-ex..ntime.dev.js
app-route-ex..time.prod.js
app-route-tu..ntime.dev.js
app-route-tu..time.prod.js
app-route-tu..ntime.dev.js
app-route-tu..time.prod.js
app-route.runtime.dev.js
app-route.ru..time.prod.js
pages-api-tu..ntime.dev.js
pages-api-tu..time.prod.js
pages-api.runtime.dev.js
pages-api.ru..time.prod.js
... and 5 more

View diffs

app-page-exp..ntime.dev.js

failed to diff

app-page-exp..time.prod.js

Diff too large to display

app-page-tur..ntime.dev.js

failed to diff

app-page-tur..time.prod.js

Diff too large to display

app-page-tur..ntime.dev.js

failed to diff

app-page-tur..time.prod.js

Diff too large to display

app-page.runtime.dev.js

failed to diff

app-page.runtime.prod.js

Diff too large to display

app-route-ex..ntime.dev.js

Diff too large to display

app-route-ex..time.prod.js

Diff too large to display

app-route-tu..ntime.dev.js

Diff too large to display

app-route-tu..time.prod.js

Diff too large to display

app-route-tu..ntime.dev.js

Diff too large to display

app-route-tu..time.prod.js

Diff too large to display

app-route.runtime.dev.js

Diff too large to display

app-route.ru..time.prod.js

Diff too large to display

pages-api-tu..ntime.dev.js

Diff too large to display

pages-api-tu..time.prod.js

Diff too large to display

pages-api.runtime.dev.js

Diff too large to display

pages-api.ru..time.prod.js

Diff too large to display

pages-turbo...ntime.dev.js

Diff too large to display

pages-turbo...time.prod.js

Diff too large to display

pages.runtime.dev.js

Diff too large to display

pages.runtime.prod.js

Diff too large to display

server.runtime.prod.js

Diff too large to display

nextjs-bot · 2026-02-02T23:53:22Z

Tests Passed

We snapshot tasks to ensure that we only hold the the dashmap locks for a short amount of time. However, profiling has revealed that we spend ~22% of the time in `save_snapshot` calling these clone methods, and 28% of the time encoding task storage values. So by dropping the clones we save a lot of time and end up holding locks a little bit longer. These are read locks on the dashmap so we will block contending writers (which would be about every turbo task execution that races with persistence in dev). This of course was already the case but now we do it for slightly longer.

nextjs-bot added created-by: Turbopack team PRs by the Turbopack team. Turbopack Related to Turbopack with Next.js. labels Feb 2, 2026

lukesandberg mentioned this pull request Feb 2, 2026

Optimize how we track data for persistence #89370

Open

vercel bot reviewed Feb 2, 2026

View reviewed changes

turbopack/crates/turbo-tasks-macros/src/derive/task_storage_macro.rs Outdated Show resolved Hide resolved

lukesandberg force-pushed the better_task_storage_snapshotting branch from f71fd0f to c3343d0 Compare February 2, 2026 21:00

lukesandberg added 4 commits February 2, 2026 21:20

fix modification tracking

f53f723

Fix clippy and simplify a condition

362c948

a debug assert and a comment

5e371ae

lukesandberg force-pushed the better_task_storage_snapshotting branch from 3f34bea to 5e371ae Compare February 3, 2026 05:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove task snapshots from persistence #89412

Remove task snapshots from persistence #89412

lukesandberg commented Feb 2, 2026 •

edited

Loading

Uh oh!

lukesandberg commented Feb 2, 2026

Uh oh!

Uh oh!

codspeed-hq bot commented Feb 2, 2026 •

edited

Loading

Uh oh!

nextjs-bot commented Feb 2, 2026 •

edited

Loading

⚡ Dev Server

📦 Dev Server (Webpack)

⚡ Production Builds

📦 Production Builds (Webpack)

Bundle Sizes

⚡ Turbopack

📦 Webpack

🔄 Shared (bundler-independent)

Uh oh!

nextjs-bot commented Feb 2, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Remove task snapshots from persistence #89412

Are you sure you want to change the base?

Remove task snapshots from persistence #89412

Conversation

lukesandberg commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Don't clone TaskStorage objects when snapshotting the database

What

Why?

Uh oh!

lukesandberg commented Feb 2, 2026

Uh oh!

Uh oh!

codspeed-hq bot commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging this PR will not alter performance

Summary

Footnotes

Uh oh!

nextjs-bot commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Stats from current PR

✅ No significant changes detected

⚡ Dev Server

📦 Dev Server (Webpack)

⚡ Production Builds

📦 Production Builds (Webpack)

Bundle Sizes

⚡ Turbopack

📦 Webpack

🔄 Shared (bundler-independent)

Uh oh!

nextjs-bot commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tests Passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lukesandberg commented Feb 2, 2026 •

edited

Loading

codspeed-hq bot commented Feb 2, 2026 •

edited

Loading

nextjs-bot commented Feb 2, 2026 •

edited

Loading

nextjs-bot commented Feb 2, 2026 •

edited

Loading