Investigate compliance issues with the current a2a-tck

[Blackhex]

Description

The https://github.com/a2aproject/a2a-tck has been updated to support v0.3.0 of the specification and many new checks were added. a2a-dotnet is not passing even mandatory set of checks now.

Within this tasks, identify particular a2a-dotnet deficiencies and report individual issues to the backlog that are preventing being compliant with a2a-tck.

Relates to #141.

Results (WIP)

• [AGENT] Test suite initialization fails because getting agent card expects preferredTransport field set. Fix is WIP.
• [FALSE_POSITIVE] transport_get_agent_card fails on missing agent/authenticatedExtendedCard. This is rather false positive as the test might use unauthenticated agent card. Otherwise, this test is skipped if agent declares push notification support. Fix submitted as fix: Remove usage of non-existent agent/card method a2a-tck#64.
• [FALSE_POSITIVE] Multiple tests from various categories fail on inner Message object missing kind property. Fix submitted as fix: Message inside MessageSendParams should contain kind attribute a2a-tck#63.
• mandatory
  • authentication
    • test_auth_compliance_v030.py
      • [AGENT] test_security_scheme_structure_compliance is skipped due to missing securitySchemes in agent card. (fixed by fix: Add security field to AgentSkill class #168)
    • test_auth_enforcement.py
      • [FEATURE v0.3.0] test_authentication_required_when_declared fails on authentication enforcement.
      • [AGENT/FEATURE] test_invalid_credentials_rejected fails because invalid authentication request is not rejected.
      • [AGENT] test_authentication_scheme_consistency fails on missing security field on agent card. (fixed by Add security to the AgentSkill #165)
  • jsonrpc
    • test_a2a_error_codes_enhanced.py
      • [BUG/FEATURE] test_push_notification_not_supported_error_32003_enhanced fails because the tests checks for push notification support and expects apropriate error message when they are declared as not supported. It fails on missing tasks/pushNotificationConfig/delete endpoint when they are declared supported.
      • [IMPROVEMENT] test_unsupported_operation_error_32004_enhanced is skipped because agent do not trigger error when sending unsupported format in acceptedOutputModes field of configuration param to message/send endpoint.
      • [IMPROVEMENT] test_content_type_not_supported_error_32005_enhanced is skipped because agent do not trigger error when sending file part with unsupported MIME type to message/send endpoint.
      • [FALSE_NEGATIVE] test_invalid_agent_response_error_32006_enhanced is skipped because the test fails to produce invalid request.
    • test_a2a_error_codes.py
      • [IMPROVEMENT] test_push_notification_not_supported_error_32003 is skipped when agent card does not declare push notifications support while tasks/pushNotificationConfig/set endpoint is implemented.
      • [IMPROVEMENT] test_unsupported_operation_error_32004 is skipped because agent do not trigger error when sending unsupported format in acceptedOutputModes field of configuration param to message/send endpoint.
      • [IMPROVEMENT] test_content_type_not_supported_error_32005 is skipped because agent do not trigger error when sending file part with unsupported MIME type to message/send endpoint.
    • test_json_rpc_compliance.py
      • [BUG] test_rejects_invalid_json_rpc_requests fails becase agent return error -32600 instead -32602 when message/send RPC request params contains "" as object key.
      • [BUG] test_rejects_invalid_params fails becase agent return error -32600 instead -32602 when message/send RPC request params is string instead of expected object.
  • protocol
    • test_a2a_v030_new_methods.py
      • [FALSE_NEGATIVE/FEATURE] test_authenticated_extended_card_method_exists fails because of wrong implementation of Advanced Agent Card retrieval on a2a-tck side, issue submitted as [Bug]: Is agent/card method defined by the specification? a2a-tck#61, fix submitted as fix: Remove usage of non-existent agent/card method a2a-tck#64. Then implementation of Advanced Agent Card on a2a-dotnet side will be needed.
      • [FALSE_NEGATIVE/FEATURE] test_authenticated_extended_card_without_auth same as above.
      • [FALSE_NEGATIVE/FEATURE] test_authenticated_extended_card_with_auth same as above.
      • [AGENT] test_tasks_list_method_availability is skipped because of check for JSON-RPC transport.
      • [AGENT] test_tasks_list_with_existing_tasks is skipped because of check for JSON-RPC transport.
      • [AGENT/FEATURE] test_grpc_specific_features is skipped becase transport is not gRPC.
      • [AGENT] test_rest_specific_features is skipped becase transport is not REST.
    • test_a2a_v030_transport_compliance.py
      • [FEATURE] test_multi_transport_method_mapping is skipped because not all transports are supported.
    • test_agent_card_mandatory.py
      • [FALSE_POSITIVE] test_agent_card_skills_mandatory fails because the test mandates skills array not being empty in the agent card. Reported as [Bug]: Skills array in agent card cannot be empty? a2a-tck#65.
    • test_extended_agent_card.py
      • [FEATURE v0.3.0] test_extended_agent_card_endpoint_exists fails because agent/getAuthenticatedExtendedCard method is not implemented.
      • [FEATURE v0.3.0] test_extended_agent_card_authentication_required fails because agent/getAuthenticatedExtendedCard method is not implemented.
      • [FEATURE v0.3.0] test_extended_agent_card_response_format is skipped because agent/getAuthenticatedExtendedCard method is not implemented.
    • test_message_send_method.py
      • [BUG] test_message_send_invalid_params fails becase agent returns error -32600 instead of -32602 when message/send RPC request is missing required properties of message object.
  • quality
    • test_error_validation_enhanced.py
      • [BUG] test_invalid_method_error_validation failse because agent returns -32603 instead of -32601 when non-existent method is requested.
  • securtity
    • test_agent_card_security.py
      • [A2A_BUG/FALSE_POSITIVE/BUG] test_public_agent_card_access_control fails on miss alignment what should be the url property in AgentCard. The test tries to access the url directly to get the card wich is wrong because it should be at least <url>/.well-known/agent-card.json. On the other hand, the SpecComplianceAgent returns http://localhost:5048/speccompliance in the url which is also wrong because http://localhost:5048/speccompliance/.well-known/agent-card.json is not a valid location for the card. This is basically another consequence in the design of how agent cards should be served in the specification.
      • [A2A_BUG/FALSE_POSITIVE/FEATURE] test_extended_card_access_controls fails because the tests construct the extended authenticated agent card URL wrong but there is no clarity on where is should be in the specification either.
      • [A2A_BUG/FALSE_NEGATIVE/FEATURE] test_authentication_scheme_validation tests passes but there are validation errors with the same cause as test_extended_card_access_controls.
      • [FALSE_POSITIVE/AGENT] test_sensitive_information_protection failse because the test checks for certain patterns in the public agent card on deserialized structure converted to string which contains fields for secrects even if they were not present in the agent response. Nevertheless, the SpecComplianceAgent does not have any special handling that no credentials are exposed on public card and extended authenticated card is not supported.
    • test_certificate_validation.py
      • [UNDECICIVE] All tests in this file are skipped because they require agent running on HTTPS with a domain name.
    • test_in_task_authentication.py
      • [UNDECICIVE/AGENT/FEATURE] test_in_task_authentication_workflow is skipped becuase the test is trying to authenticate using LLM prompt submitted as task while the SpecComplianceAgent has no LLM.
      • [FALSE_NEGATIVE/AGENT/FEATURE] test_authentication_challenge_headers passes even if it tries to request non-existing task and check if auhentication was enforced. This is not implemented both in the SpecComplianceAgent and the library.
      • [FEATURE] test_invalid_authentication_handling fails because agent do not respond with correct errors on requests with ivalid authentication.
      • [AGENT/FEATURE] test_auth_state_transitions is skipped because SpecComplianceAgent do not implement tasks state management and authentication.
    • test_tls_configuration_enhanced.py
      • [UNDECICIVE/FEATURE] All tests in this file are skipped because they require agent running on HTTPS with a domain name and TLS support.
    • test_transport_security.py
      • [UNDECICIVE/FEATURE] All tests in this file are skipped because they require agent running on HTTPS with a domain name and TLS support.
  • transport
    • test_multi_transport_equivalence.py
      • [FALSE_POSITIVE/FALSE_NEGATIVE] Tests in this file fails because they are unfinished / regressed. Some of them pass but should fail for the same reason.
• unit
  • [FALSE_POSITIVE] Some unit test fails on usage of removed send_json_rpc method of SUTClient. Fix submitted as fix: Fix tests failing on missing send_json_rpc method a2a-tck#66.
  • [FALSE_POSITIVE] test_transport_error_with_original fails because of signature of TransportError has changed. Fix submitted as fix: Fix test failing on TransportError constructor signature change a2a-tck#67.
  • [FALSE_POSITIVE] test_get_task_interface and test_cancel_task_interface fail on invalid assert of task ID. Fix submitted as fix: Fix tests failing due to get_task and cancel_task params change a2a-tck#68.

Explanation

• [A2A_BUG]: Issue in A2A specification.
• [AGENT]: Issue in SpecComplianceAgent that can be resolved without change to a2a-dotnet
• [BUG]: Issue in a2a-dotnet in already implemented feature.
• [FALSE_POSITIVE]: Issue in a2a-tck where test is failing but shouln't.
• [FALSE_NEGATIVE]: Issue in a2a-tck where test is passing but shouln't.
• [FEATURE ]: Unimplemented feature in a2a-dotnet for the specified protocol version.
• [FEATURE]: Unimplemented feature in a2a-dotnet, the protocol version is unknown.
• [IMPROVEMENT]: Not a bug but improvement in a2a-dotnet can resolve.
• [UNDECICIVE]: The test cannot be run for a reason.