NextJS Source Maps Upload Slow

[Averylamp]

Problem Statement

We're experiencing slow upload times for source maps in our Next.js application likely due to a large number of chunk files generated during the build process and potentially that the upload isn't done in parallel or batching the files together. Currently NextJS generates a few thousand chunk files, which sentry source map adds multiple minutes to upload.

I've tried widenClientFileUpload: false, but it doesnt seem to make it too much faster.

I am not certain if it's because of sequential uploading or if the source maps can be merged before uploading to resolve the slow source map upload.

Our vercel build logs show a couple thousand lines of uploading individual chunk files

    ~/5fe05a97-ee5c-483e-8b22-3ffaf1e8b653-745.js (sourcemap at 55649.js.map, debug id 5fe05a97-ee5c-483e-8b22-3ffaf1e8b653)
    ~/602b546a-3b21-436f-9513-d0ef34677811-1373.js (sourcemap at 96984.js.map, debug id 602b546a-3b21-436f-9513-d0ef34677811)
    ~/6037b97b-c678-43cc-a820-f2524ae09286-791.js (sourcemap at 5919.js.map, debug id 6037b97b-c678-43cc-a820-f2524ae09286)
    ~/607a5fb8-7126-4215-b650-afcb3e8dfb6d-1405.js (sourcemap at 99593.js.map, debug id 607a5fb8-7126-4215-b650-afcb3e8dfb6d)
    ~/609731b9-93da-490c-92ae-e6790feeb690-547.js (sourcemap at 429.js.map, debug id 609731b9-93da-490c-92ae-e6790feeb690)
    ~/60bbc6fd-910a-45f3-b660-a70a1bd98d6a-276.js (sourcemap at 26763.js.map, debug id 60bbc6fd-910a-45f3-b660-a70a1bd98d6a)
    ~/60d88009-4959-4a15-bea7-b01c3a04c039-582.js (sourcemap at 4546.js.map, debug id 60d88009-4959-4a15-bea7-b01c3a04c039)
    ~/60f4a3df-1282-41ab-b35d-3f1f66b97fbc-896.js (sourcemap at 65118.js.map, debug id 60f4a3df-1282-41ab-b35d-3f1f66b97fbc)
    ~/60f694cb-8af2-4533-a270-2c8ce141b905-268.js (sourcemap at 2628.js.map, debug id 60f694cb-8af2-4533-a270-2c8ce141b905)

Solution Brainstorm

I think either parallelizing source map uploads (if it isn't already) or merging the chunks into larger files will speed up source maps upload significantly.

[Averylamp]

This issue seems related as well, where others are reporting similar upload times for NextJS
getsentry/sentry-cli#309

[Lms24]

Hey @Averylamp thanks for writing in! We are aware of longer upload times for a large number of files although were only getting sporadic reports. We have some ideas how to speed things up but I can't give you exact numbers ir ETAs yet. I'll tag @lforst - in case you want to add something.

[lforst]

I think either parallelizing source map uploads (if it isn't already) or merging the chunks into larger files will speed up source maps upload significantly.

We're already doing both of these things. We looked at some numbers recently and came to the loosely held conclusion that the upload speed is already relatively fast compared to how much data is being uploaded - which obviously sucks because that makes it hard to fix. The reality is that sourcemaps can get very very big (think that they basically include all node_modules that you import) and upload speeds are often not that high.

Maybe we're missing something really bad and obvious though. Can you share concrete numbers on how long your uploads take?

[Averylamp]

I think it adds ~2 minutes to our build time in CI. I checked the total size of all of my source maps which is ~34MB across 2848 files. 34MB to me doesn't feel like it should account for the time uploading. All of the node_modules source maps is ~196MB, which maybe makes sense. Is there a way to skip node modules source mapping uploads? They're mostly pinned anyways and I wouldn't care to not have them.
I think either way 225MB on a Gbps still should only take a few seconds.

If that's not it then maybe an IOPs issue then which maybe tarring them will help. It is significantly faster on my local machine which has an NVME drive versus the CI machines so that's the other hypothesis

Thanks for the quick reply!