Skip to content

Commit 64cf12e

Browse files
committed
(Disclosure: I am the lead engineer of Cloudflare Workers.)
This fixes two issues with the test: * The next.js benchmark used force-dynamic on Vercel but not Cloudflare. It should use force-dynamic on both. Ironically, this discrepency should have given Cloudflare an advantage, but we found that Open Next had significant performance bugs in the non-dynamic code path that actually made it worse. Among other things, streaming of the response body was essentially disabled in this mode. We are fixing those bugs, but obviously it's most fair for both platforms to use the same dynamic setting. * We also fixed several other performance issues in Open Next, so this bumps the version to get some of those fixes. This work is ongoing: we expect to land more improvements in the future. Open Next is not as mature as Next.js, it seems. Separately, Cloudflare has made some changes to our production environment which should significantly improve performance. In particular: * We corrected a problem where CPU-heavy requests would tend to queue up on a single worker instance per colo, causing excess latency when running concurrent CPU-heavy requests driven from a single client location. (That said, it is still possible for requests to be randomly assigned to the same isolate and block each other, but this should be less common now.) * We found that we had tuned the V8 garbage collector too far in the direction of favoring memory usage over execution speed. A small adjustment made a big difference in performance, especially in these tests which do a lot of memory allocation. (We'll have a blog post about these changes later.) We also have a few suggestions about how to run and interpret these benchmarks: * The "shitty sine benchmark" is indeed suffering from a missing optimization in Node, penalizing Vercel. [We are fixing it](nodejs/node#60153), but it will presumably take some time for this Node change to find its way to Vercel. In the meantime, we recommend excluding this test, as it unfairly benefits Cloudflare. * We think it is more appropriate to test with a Vercel instance using 1vcpu rather than 2. [The CTO of Vercel argues there should be no difference since the workload is fundamentally single-threaded](https://x.com/cramforce/status/1975656443954274780), and [he is publishing pricing comparisons on the assumption that only 1 vcpu was actually used](https://x.com/cramforce/status/1975652040195084395). These pricing comparisons are only fair if the assumption is correct. We honestly think he is correct, so we think to avoid any questions the test should be run with 1vcpu. (I realize this sounds like some sort of trick, but it isn't. We haven't had a chance to test the difference ourselves. I just honestly think the 2vcpu thing creates confusion that would be nice to avoid.) * This benchmark still contains a singificant "luck" factor in terms of what hardware you get assigned to. Cloudflare has several different generations of hardware in our fleet, and we would expect Vercel / AWS does as well. Different CPUs may have different single-threaded performance. Noisy neighbors can also have significant impact by consuming memory bandwidth that is shared by all tenants on the machine. We have seen runs of the test where Cloudflare wins across the board, and others where Vercel wins across the board, presumably as a result of this noise -- and it's not just Cloudflare's performance that varies, but also Vercel's. Note, though, that simply running more iterations of the benchmark does not correct for this "luck", because once instances are assigned to machines, they tend to stay on those machines. Additionally, noisy neighbor effects can be driven by other factors like time of day, regional load imbalances, etc., that don't go away with additional iterations.
1 parent 5bf5d94 commit 64cf12e

File tree

2 files changed

+3
-1
lines changed

2 files changed

+3
-1
lines changed

next-bench/cf-edition/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
"cf-typegen": "wrangler types --env-interface CloudflareEnv ./cloudflare-env.d.ts"
1313
},
1414
"dependencies": {
15-
"@opennextjs/cloudflare": "^1.3.0",
15+
"@opennextjs/cloudflare": "^1.10.1",
1616
"next": "15.4.6",
1717
"react": "19.1.0",
1818
"react-dom": "19.1.0"

next-bench/cf-edition/src/app/bench/page.tsx

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
import ComplexComponent from "./complex-component";
22

3+
export const dynamic = "force-dynamic";
4+
35
export default async function ComplexPage() {
46
console.log("rendering", Date.now());
57

0 commit comments

Comments
 (0)