`web-ext sign` intermittently fails with 503 on upload

[ridhwanc]

When invoking web-ext sign, to upload our build to AMO to be signed, the build is still uploaded to AMO, however the the invocation quite frequently throws out a 503 error code shown below:

Waiting for validation...
Submitting source to version
Upload of source failed: Error: response status was 503.
WebExtError: Uploading source failed
    at /node_modules/.pnpm/web-ext@8.9.0_body-parser@1.20.3_express@4.21.2/node_modules/web-ext/lib/cmd/sign.js:101:13
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async Program.execute (/node_modules/.pnpm/web-ext@8.9.0_body-parser@1.20.3_express@4.21.2/node_modules/web-ext/lib/program.js:270:7)
    at async /node_modules/.pnpm/web-ext@8.9.0_body-parser@1.20.3_express@4.21.2/node_modules/web-ext/bin/web-ext.js:13:1

Has anyone else ran into this or something similar?

[Rob--W]

This error is rare. Does the issue happen consistently, or do you have an occasional successful upload on retry?

If you have multiple versions of the extension, where one consistently succeeds, and the other consistently fails, do you see any difference in characteristics between the submissions? For example, the size of the source code and/or xpi file that you're trying to submit.

[ridhwanc]

We've been seeing this error pretty consistently starting July 31st, 2025. This occurs on uploading the nightly version of the extension. As an example, on our past 4 uploads, 2 of them were successful and other 2 failed on the 503 (yet still uploaded the extension). This has been the intermittent cadence since we've started to see the issue.

There aren't any notable differences between submissions of other versions of our extensions (beta/stable). Our beta and stable extension source code differ by less than ~1 Mb in size.

[abyrne-moz]

Hi @ridhwanc - are you able to please provide some specific dates and times that you see this error so we can take a look at our server side logs for further troubleshooting?

[ridhwanc]

Hey @abyrne-moz, here are a few timestamps I was able to find around the time of submitting this issue:

• August 6th - 2:34 AM EDT
• August 10th - 2:32 AM EDT
• August 12th - 2:39 AM EDT
• August 13th - 2:33 AM EDT
• August 13th - 1:45 PM EDT

[willdurand]

Thanks, that's very helpful. Despite the 503s, the versions have been successfully submitted and signed as far as I can tell. We identified the root cause that led to that situation, and we're working on a bunch of mitigations. I'll update this issue when I have more information.

[ridhwanc]

Despite the 503s, the versions have been successfully submitted and signed as far as I can tell

Though this may be true, this is far less than ideal since these 503 errors are false positives that lead to failures in our CI. Our nightlies are self-hosted, we use AMO to sign the extension for this channel. So in turn, this halts the automation of our nightlies being published.

We appreciate you looking into this for us and hope to hear an update on this soon!

[wagnerand-moz]

Though this may be true, this is far less than ideal since these 503 errors are false positives that lead to failures in our CI. Our nightlies are self-hosted, we use AMO to sign the extension for this channel. So in turn, this halts the automation of our nightlies being published.

Perhaps you should not automatically submit a new version every night. This seems hardly necessary, such a testing version could easily be installed as a temporary extension.
Also, keep in mind that all versions submitted to Mozilla are applicable to Mozilla's add-on policies.

[willdurand]

@ridhwanc hey, I deployed a change on prod this morning, which should prevent this from happening ever again. Could you confirm that you're not seeing 503s anymore? Thanks!

[ridhwanc]

Hi @willdurand, we've just ran into the 503 again on October 20th @ 2:31 AM EDT.

Upload of source failed: Error: response status was 503.
WebExtError: Uploading source failed
    at file:///web-ext-v8/lib/cmd/sign.js:101:13
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Program.execute (file:///web-ext-v8/lib/program.js:270:7)
    at async file:///web-ext-v8/bin/web-ext.js:13:1

[ridhwanc]

@willdurand Do we need to update the version on our side to use latest version of web-ext? We're currently using a v8 fork of the project.

[willdurand]

No, it's a server side issue.

[ridhwanc]

@willdurand do you have an update on this? We're still running into this issue.

[diox]

@ridhwanc we're actively working on this. There are several things at play in this case (large, complex source code archive, code and infrastructure changes on our end) and we're eliminating/fixing them one by one.