feat: Add A2A/ADK Evaluation with Vertex AI Notebook

[holtskinner]

Redo of a2aproject/a2a-python#300

[gemini-code-assist]

Summary of Changes

Hello @holtskinner, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request adds a comprehensive Jupyter Notebook that serves as a practical guide for evaluating A2A and ADK agents within the Google Cloud Vertex AI ecosystem. It provides a concrete example of an agent, integrates it into the A2A framework, and demonstrates how to leverage Vertex AI's evaluation capabilities to measure the agent's performance and quality.

Highlights

• New Evaluation Notebook: Introduced a new Jupyter Notebook (notebooks/a2a_evaluation.ipynb) dedicated to demonstrating end-to-end evaluation of A2A/ADK agents using Vertex AI Evaluation services.
• Reimbursement Agent Implementation: The notebook includes a complete, in-memory implementation of a sample 'Reimbursement Agent' built using the google.adk framework, showcasing its capabilities for handling reimbursement requests with tool usage.
• A2A Integration and Execution: An AgentExecutor is implemented to integrate the ReimbursementAgent with the a2a framework, allowing it to process messages, manage tasks, and handle structured form inputs/outputs.
• Vertex AI Evaluation Setup: The notebook configures and utilizes Vertex AI Evaluation, defining helper functions to parse agent outcomes and setting up an EvalTask to assess agent responses based on metrics like 'safety' and 'coherence'.
• Evaluation Data and Visualization: A sample evaluation dataset is provided, and functions are included to display summary metrics and row-wise results from the Vertex AI evaluation, facilitating quick analysis of agent performance.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

The pull request introduces a new notebook for demonstrating end-to-end evaluation of an A2A/ADK agent using Vertex AI. Several critical issues were identified, including missing await calls and the use of a hardcoded session ID, along with other high and medium severity issues related to correctness, robustness, and maintainability.

[gemini-code-assist]

Missing await call for updater.update_status(...). This will cause the status to not be updated.

                    await updater.update_status(
                        TaskState.failed,
                        new_agent_text_message(
                            'Reaching an unexpected state',
                            task.contextId,
                            task.id,
                        ),
                        final=True,
                    )

[gemini-code-assist]

Using a hardcoded contextId for all evaluation runs is a critical issue. Generate a unique contextId for each call to a2a_parsed_outcome.

                    "contextId": f"colab-session-{get_id()}",

[gemini-code-assist]

Missing await call for event_queue.enqueue_event(task). This will cause the task to not be executed.

            await event_queue.enqueue_event(task)

[gemini-code-assist]

Missing await call for updater.update_status(...). This will cause the status to not be updated.

                await updater.update_status(
                    TaskState.working,
                    new_agent_text_message(
                        item['updates'], task.contextId, task.id
                    ),
                )

[gemini-code-assist]

Missing await call for updater.update_status(...). This will cause the status to not be updated.

                    await updater.update_status(
                        TaskState.input_required,
                        new_agent_parts_message(
                            [Part(root=DataPart(data=data))],
                            task.contextId,
                            task.id,
                        ),
                        final=True,
                    )

[gemini-code-assist]

predicted_trajectory_list is initialized but never populated, resulting in an empty predicted_trajectory field. Extract tool calls from the rpc_response_send_msg.

[gemini-code-assist]

Calling min() or max() on an empty sequence will raise a ValueError. Add a check to handle this case gracefully.

    if not summary_metrics.values():
        print("No summary metrics to plot.")
        return

    min_val = min(summary_metrics.values())
    max_val = max(summary_metrics.values())

[gemini-code-assist]

This line modifies the input dictionary output directly. Load the JSON into a new variable to avoid modifying the original data structure.

    if output["predicted_trajectory"]:
        predicted_trajectory = json.loads(output["predicted_trajectory"])
        markdown += "### Function Calls\n"
        for call in predicted_trajectory:
            markdown += f"- **Function**: `{call['tool_name']}`\n"
            markdown += "  - **Arguments**:\n"
            for key, value in call["tool_input"].items():
                markdown += f"    - `{key}`: `{value}`\n"

[gemini-code-assist]

The function return_form is type-hinted to return a dict[str, Any], but it actually returns a string from json.dumps(form_dict) on line 353. This mismatch can be misleading. Update the return type hint to str.

) -> str:

[gemini-code-assist]

The condition any([True for p in event.content.parts if p.function_response]) is verbose. Simplify using a generator expression.

elif (
                    event.content
                    and event.content.parts
                    and any(p.function_response for p in event.content.parts)
                ):