fix: using GET for candidate checks - sitemaps

[nikitachapovskii-dev]

New case discovered.

This fix parallelizes sitemap candidate probing in discoverValidSitemaps for sitemaps.

Candidate checks are now executed concurrently and collected, so a slow/failing candidate no longer blocks others. This removes avoidable discovery delays while preserving sitemap detection semantics.

Closes #3463

[barjin]

Imo urlExists() should be a lightweight check, which is in-line with HEAD semantics. Fetching with GET will download the entire resource before resolving the call (and sitemaps can be hefty).

As per the HTTP Semantics RFC, general-purpose HTTP servers must support GET and HEAD methods. Imo the right solution for misbehaving servers is stricter timeouts or parallel processing, so the failing response doesn't fully block the ones that could pass.

[nikitachapovskii-dev]

Imo urlExists() should be a lightweight check, which is in-line with HEAD semantics. Fetching with GET will download the entire resource before resolving the call (and sitemaps can be hefty).

As per the HTTP Semantics RFC, general-purpose HTTP servers must support GET and HEAD methods. Imo the right solution for misbehaving servers is stricter timeouts or parallel processing, so the failing response doesn't fully block the ones that could pass.

Agreed, thanks for the feedback @barjin

I reverted the GET. The implemented fix is parallel candidate probing with Promise.allSettled, so one slow/misbehaving candidate does not serialize discovery for the rest.

Note: because candidate probing is parallel, emission order between domains is no longer strictly deterministic; tests were updated accordingly.

[barjin]

lgtm, thank you @nikitachapovskii-dev !

I'm wondering how we managed to make sitemap discovery so complicated, but that's not a problem of this PR (previous state wasn't any better 😅 )

[barjin]

nit

Suggested change

	candidateSitemapUrls.map((candidateSitemapUrl) => urlExists(candidateSitemapUrl)),
	candidateSitemapUrls.map(urlExists),

[nikitachapovskii-dev]

There was a bunch of issues where we solve problems which are already addressed in V4, so when we switch to it on most of actors the sitemap drama(/comedy?) should be over
Thanks for review!

[B4nan]

Next time please adjust the PR title to match what the PR actually does before you merge it. Looking at the diff, we didn't really change the HEAD method in the end, but now using GET for candidate checks - sitemaps will end up in the changelog.