Implement RFC9728 - Support WWW-Authenticate header by MCP client

[yurikunash]

Resolves #1054
This PR implements proper WWW-Authenticate header parsing for protected resource metadata URL discovery as required by RFC9728 and the MCP specification. This change ensures full compliance with MCP authentication requirements and includes several related improvements to the authentication flow.

Changes Made

Primary Changes

• ✅ WWW-Authenticate Header Parsing: Implemented parsing of WWW-Authenticate headers for protected resource metadata URL discovery per RFC9728
• ✅ MCP Specification Compliance: Now properly handles HTTP 401 Unauthorized responses as required by the MCP specification

Indirect Improvements

• Optimized Authorization Flow: Stop making unnecessary Authorize requests when the initial request doesn't respond with 401 status code
• Code Cleanup: Removed code duplication in authentication flow logic

Motivation and Context

As per MCP specification:

MCP clients MUST be able to parse WWW-Authenticate headers and respond appropriately to HTTP 401 Unauthorized responses from the MCP server.

This implementation was missing from the current MCP client, creating a gap in specification compliance that has real-world implications. The absence of proper WWW-Authenticate header handling prevents hosting multiple MCP servers that share the same domain but function as separate OAuth resources.

How Has This Been Tested?

The changes were verified by the comprehensive unit-test coverage for different scenarios, as well as simulating the Authorization scenario by executing the client locally.

Breaking Changes

Authorization Flow Change: The client now follows proper OAuth flow by sending an initial request to the resource and only performing authorization upon receiving a 401 status code. This aligns with OAuth specifications and security best practices.
Impact: Clients that relied on the previous behavior (where authorization was attempted regardless of initial response) may need to be updated. However, this change improves security and compliance with OAuth standards.
Migration: Most properly configured MCP servers should work without changes. Servers that expect immediate authorization without sending a 401 response should be updated to follow OAuth specifications.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• [] Documentation update

Checklist

• I have read the MCP Documentation
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling
• I have added or updated documentation as needed

Additional context

[dead8309]

Hi, Any updates on it ? @ihrpr

we rely on correct parsing of Auth header

[q815101630]

@ihrpr Can we raise attention to this bug? It blocks the MCP client from correctly finding the resource metadata, leading to incorrect AS. Thank you

[pcarleton]

I'm not sure I agree with this part... if we know our current tokens are invalid, and we know we can refresh them, shouldn't we do that before sending a request we know is going to 401?

I think that would mean moving this if/else block ahead of line 526 where we yield request.

[yurikunash]

Thank you for the suggestion @pcarleton
I was hesitant about it, so I implemented the "try first" approach, as it seemed more reliable.
As this article states, it's also perfectly fine to have that kind of optimization and refresh the token preemptively, so I updated the code as you proposed.

[pcarleton]

LGTM

Thanks for the thorough PR with tests, and being open to feedback!

Note to self: we'll still need to follow up to fix the path-based URL issue #1052