New MCP tool for running mobile tests (via app distribution). #9250

jrothfeder · 2025-10-02T22:48:53Z

Description

New MCP tool for running mobile tests. Extracted business logic from appdistribution CLI so that it can be used by both that CLI and this tool.

Scenarios Tested

Ran the equivalent of

firebase appdistribution:distribute app/build/outputs/apk/debug/app-debug.apk --app <gmp-app-id> --test-devices="model=MediumPhone.arm,version=34,locale=en,orientation=portrait" --test-case-ids crash-app

via Gemini cli. This can be initiated by typing appdistribution in the prompt, gemini cli will then walk you through providing the data needed to upload a new distribution binary and run tests.

Sample Commands

initiated by typing appdistribution in the prompt, gemini cli will then walk you through providing the data needed to upload a new distribution binary and run tests.

… be used by an MCP tool.

gemini-code-assist · 2025-10-02T22:49:10Z

Summary of Changes

Hello @jrothfeder, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a new Mobile Cloud Platform (MCP) tool to streamline the process of running mobile tests via app distribution. It achieves this by refactoring and centralizing the core app distribution and testing logic into a reusable function. This new function is now utilized by both the existing appdistribution command-line interface and the newly integrated MCP tool, simplifying the codebase and enhancing the overall workflow for uploading app binaries, managing release details, and executing automated tests.

Highlights

New MCP Tool for Mobile Tests: Introduced a new Mobile Cloud Platform (MCP) tool specifically designed for running mobile tests through app distribution, enhancing the testing workflow.
Business Logic Extraction: Extracted core business logic for app distribution from the existing appdistribution CLI into a reusable function. This function is now shared between the CLI and the new MCP tool, promoting code reuse and maintainability.
Centralized Distribution Logic: The distribute function in src/appdistribution/distribution.ts now encapsulates the entire process of uploading releases, handling AAB (Android App Bundle) information, updating release notes, distributing to testers, and initiating automated tests.
CLI Simplification: The appdistribution:distribute command has been refactored to leverage the new centralized distribute function, significantly reducing its internal complexity and making it more streamlined.
Flexible Release Notes and Parsing: The updateReleaseNotes method in src/appdistribution/client.ts now accepts optional release notes, and utility functions like parseIntoStringArray and parseTestDevices in src/appdistribution/options-parser-util.ts have been updated to allow optional file parameters, improving flexibility.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a new MCP tool for running mobile tests and refactors the app distribution logic for better reusability. The extraction of business logic from the CLI command into a shared distribute function is a great improvement for maintainability.

I've found a couple of issues. There's a critical issue regarding a potential runtime error due to an unsafe non-null assertion. I've also pointed out some redundant code in the new MCP tool.

Overall, this is a solid contribution that improves the codebase structure. Addressing the feedback will make it even more robust.

src/appdistribution/distribution.ts

src/mcp/tools/appdistribution/tests.ts

lfkellogg · 2025-10-03T14:51:24Z

src/mcp/tools/appdistribution/tests.ts

+          'model=<model-id>,version=<os-version-id>,locale=<locale>,orientation=<orientation>'. Run 'gcloud firebase test android|ios models list' to see
+          available devices.`,
+      ),
+      testCaseIds: z.string().describe(`A comma-separated list of test case IDs.`),


Since running existing test cases is out of scope for this effort (right?), I figured instead of using test case IDs you would add the ability to run a test by calling CreateReleaseTest with the AiInstructions set: http://google3/google/firebase/appdistro/v1alpha/firebase_app_distribution.proto?l=783-785

That way you would be running a one-off test without have to call CreateTestCase first.

I agree that it's technically out of scope, however, it is far simpler to do so I figured we'd support both.

I was thinking we'd get this functionality in and then add the ability for one-off.

lfkellogg · 2025-10-03T14:58:26Z

src/mcp/tools/appdistribution/tests.ts

+      testDevices: z.string().describe(
+        `Semicolon-separated list of devices to run automated tests on, in the format
+          'model=<model-id>,version=<os-version-id>,locale=<locale>,orientation=<orientation>'. Run 'gcloud firebase test android|ios models list' to see
+          available devices.`,


I would suggest also including some suggested device configurations, so Gemini can easily run a test without having to have the gcloud CLI installed. Something like:

"model=tokay,version=36,locale=en,orientation=portrait;model=e1q,version=34,locale=en,orientation=portrait" (Pixel 9 and Galaxy S24)

Side note: we could consider adding a tool to get the catalog, so we wouldn't have to rely on the gcloud CLI. Here's the API: https://firebase.google.com/docs/test-lab/reference/testing/rest/v1/testEnvironmentCatalog/get?apix_params=%7B%22environmentType%22%3A%22ANDROID%22%7D

I'll add this as the default.

Getting the catalog is a good idea. Maybe we'll flesh this out a bit more when we wire this up in to the prompt (that @tagboola is working on)?

I'm happy to add it as a TODO, don't know if we'll add it in the short term though.

tagboola · 2025-10-03T14:56:59Z

src/appdistribution/client.ts

  }

-  async updateReleaseNotes(releaseName: string, releaseNotes: string): Promise<void> {
+  async updateReleaseNotes(releaseName: string, releaseNotes?: string): Promise<void> {


Why would you ever call updateReleaseNotes with releaseNotes set to null?

I agree but this is how the code already worked.

The code around parameter processing needs to be refactored to simplify all of this. All I did was move it to a shared-library. Would you be comfortable with addressing the parameter processing in a separate PR?

I think leave this the same i.e. don't make releaseNotes nullable and then in the code where you call it, only call it if releaseNotes is defined. I feel like that isn't too much of a change and can easily be addressed here.

src/appdistribution/distribution.ts

tagboola · 2025-10-03T14:59:58Z

src/appdistribution/options-parser-util.ts

 * Value takes precedent over file.
 */
-export function parseIntoStringArray(value: string, file: string): string[] {
+export function parseIntoStringArray(value: string, file = ""): string[] {


Why not default to null or undefined?

Is null or undefined preferable to the empty string in TS?

Good question. I guess my Kotlin mind just assumed null was prefered, but maybe empty string is in TS.

tagboola · 2025-10-03T15:00:16Z

src/appdistribution/options-parser-util.ts

 * Value takes precedent over file.
 */
-export function parseTestDevices(value: string, file: string): TestDevice[] {
+export function parseTestDevices(value: string, file = ""): TestDevice[] {


yeah, again I think we should revisit all of this. The existing code is a bit overcomplicated with how it handles parameters.

tagboola · 2025-10-03T15:02:28Z

src/mcp/tools/appdistribution/tests.ts

+          'model=<model-id>,version=<os-version-id>,locale=<locale>,orientation=<orientation>'. Run 'gcloud firebase test android|ios models list' to see
+          available devices.`,
+      ),
+      testCaseIds: z.string().describe(`A comma-separated list of test case IDs.`),


Since we're running on-off tests, I don't think we want to require testCaseIds since that would mean we are supporting running existing tests. If you're just keeping this here for now for simplicity and plan to update it in a future change I'm good with that.

Yeah, I was thinking we could support both on-off and existing and this would be a good start. If you're comfortable, I'd like to see if we can get this PR in and then build the on-off functionality separately.

I'm not sure it makes sense to give Gemini access to a tool that requires test case IDs unless we also give it the ability to at least list test cases, and preferably to create test cases as well.

Fair enough.

tagboola · 2025-10-03T15:02:54Z

src/mcp/tools/appdistribution/tests.ts

+    inputSchema: z.object({
+      appId: ApplicationIdSchema,
+      releaseBinaryFile: z.string().describe("Path to the binary release (APK)."),
+      testDevices: z.string().describe(


Can we make this optional and provide a default? I don't think we want to require the user to specify a device.

Makes sense. I'll update the tool to send the following by default: model=MediumPhone.arm,version=34,locale=en,orientation=portrait

I think with zod you can set a default value

https://zod.dev/api?id=defaults

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

tagboola · 2025-10-03T15:35:52Z

src/mcp/tools/appdistribution/index.ts

+import type { ServerTool } from "../../tool";
+import { run_tests } from "./tests";
+
+export const appdistributionTools: ServerTool[] = [run_tests];


Can we place these behind the mcpalpha experiment?

tagboola · 2025-10-06T16:41:31Z

src/appdistribution/client.ts

  }

-  async updateReleaseNotes(releaseName: string, releaseNotes: string): Promise<void> {
+  async updateReleaseNotes(releaseName: string, releaseNotes?: string): Promise<void> {


I think leave this the same i.e. don't make releaseNotes nullable and then in the code where you call it, only call it if releaseNotes is defined. I feel like that isn't too much of a change and can easily be addressed here.

tagboola · 2025-10-06T17:06:04Z

src/appdistribution/options-parser-util.ts

 * Value takes precedent over file.
 */
-export function parseIntoStringArray(value: string, file: string): string[] {
+export function parseIntoStringArray(value: string, file = ""): string[] {


Good question. I guess my Kotlin mind just assumed null was prefered, but maybe empty string is in TS.

tagboola · 2025-10-06T17:08:30Z

src/mcp/tools/appdistribution/tests.ts

+    inputSchema: z.object({
+      appId: ApplicationIdSchema,
+      releaseBinaryFile: z.string().describe("Path to the binary release (APK)."),
+      testDevices: z.string().describe(


I think with zod you can set a default value

https://zod.dev/api?id=defaults

tagboola · 2025-10-06T17:09:15Z

src/mcp/tools/appdistribution/tests.ts

+      testDevices: z.string().describe(
+        `Semicolon-separated list of devices to run automated tests on, in the format
+          'model=<model-id>,version=<os-version-id>,locale=<locale>,orientation=<orientation>'. Run 'gcloud firebase test android|ios models list' to see
+          available devices.`,


I'm happy to add it as a TODO, don't know if we'll add it in the short term though.

Jamie Rothfeder added 6 commits October 2, 2025 11:29

Scaffolding for new appdistribution MCP tool.

b9d24b1

Refactor business logic out of the appdistribution CLI so that it can…

46f06fd

… be used by an MCP tool.

Merge branch 'master' into jr/mcp/mobile-testing

8b028c2

Wire new appdistribution tool up to the business logic.

844b3a2

Merge branch 'master' into jr/mcp/mobile-testing

6b0b3de

Fix linting errors.

1771da9

github-project-automation bot added this to [Cloud] Extensions + Functions Oct 2, 2025

gemini-code-assist bot reviewed Oct 2, 2025

View reviewed changes

src/appdistribution/distribution.ts Outdated Show resolved Hide resolved

src/mcp/tools/appdistribution/tests.ts Show resolved Hide resolved

lfkellogg reviewed Oct 3, 2025

View reviewed changes

tagboola reviewed Oct 3, 2025

View reviewed changes

Jamie Rothfeder and others added 2 commits October 3, 2025 12:13

Merge branch 'master' into jr/mcp/mobile-testing

3392c6c

Update src/appdistribution/distribution.ts

81cda3c

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

jrothfeder changed the base branch from master to feature-branch/mcp/mobile-testing October 6, 2025 16:12

tagboola reviewed Oct 6, 2025

View reviewed changes

tagboola approved these changes Oct 6, 2025

View reviewed changes

github-project-automation bot moved this to Approved [PR] in [Cloud] Extensions + Functions Oct 6, 2025

jrothfeder merged commit 18c29f8 into feature-branch/mcp/mobile-testing Oct 6, 2025
26 of 31 checks passed

jrothfeder deleted the jr/mcp/mobile-testing branch October 6, 2025 19:09

github-project-automation bot moved this from Approved [PR] to Done in [Cloud] Extensions + Functions Oct 6, 2025

New MCP tool for running mobile tests (via app distribution). #9250

New MCP tool for running mobile tests (via app distribution). #9250

Conversation

jrothfeder commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Scenarios Tested

Sample Commands

Uh oh!

gemini-code-assist bot commented Oct 2, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrothfeder Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrothfeder Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jrothfeder Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

jrothfeder commented Oct 2, 2025 •

edited

Loading

jrothfeder Oct 3, 2025 •

edited

Loading

jrothfeder Oct 3, 2025 •

edited

Loading

jrothfeder Oct 3, 2025 •

edited

Loading