Auth server metadata fallbacks abort on non-404 failures

[LucaButBoring]

Initial Checks

• I confirm that I'm using the latest version of MCP Python SDK
• I confirm that I searched for my issue in https://github.com/modelcontextprotocol/python-sdk/issues before opening this issue

Description

During auth server metadata discovery, the client SDK attempts to query the four supported metadata endpoints sequentially (#1061). However, it only falls back from one to another on 404 responses (source). This is stricter than the specification requires, and happens to break compatibility with Cognito, which returns a 400 at any endpoints it doesn't support (example).

In the case of Cognito, the SDK attempts to discover metadata at /.well-known/oauth-authorization-server, gets a 400, and immediately falls back to the legacy behavior of calling /authorize on the RS.

Given that the fallback conditions are not precisely defined by the specification, I believe the SDK should fall back on any 4XX error response. Arguably, it should fall back on 5XX responses as well, but I'm limiting this to 4XX responses to avoid the possibility of a load-sensitive 5XX being quadrupled in scale by fallback behaviors.

Example Code

Python & MCP Python SDK

Python 3.12.9, SDK v1.12.1

[LucaButBoring]

I plan to put out a PR to propose a fix for this soon.