Google Home/Assistant AI Test Automation Framework

Team 9 - CMPE 187 - Software Quality Engineering

Project Overview

This project is an automated testing framework designed to evaluate the Natural Language Understanding (NLU) capabilities of Google Assistant on Android. Unlike traditional API testing, this framework performs Black-Box UI Testing using Appium. It simulates real user interactions by triggering the native Google Assistant overlay, inputting natural language commands via the keyboard interface, and verifying the AI's response through screen scraping.

Core Features

• Native Assistant Interaction: Triggers the actual Google Assistant overlay using Android KeyCodes.
• Hybrid Input Strategy: Supports both text input simulation and fallback mechanisms (Send Button vs. Enter Key).
• Data-Driven Testing: Reads test scenarios (Lamp, TV, Music, Message) dynamically from a CSV file.
• Robust Error Handling: Handles dynamic UI elements, timeouts, and emulator lags.

Prerequisites

Before running the tests, ensure the following tools are installed on your machine (macOS M-Series recommended):

1. Python 3.9+
2. Node.js & npm (Required for Appium Server)
3. Android Studio (For Android SDK and Emulator)
4. Appium Inspector (For UI debugging)

Installation & Setup Guide

1. Python Environment Setup

It is recommended to use a virtual environment to manage dependencies.

#1. Create a virtual environment

python3 -m venv venv

#2. Activate the virtual environment

source venv/bin/activate

#3. Install required Python libraries

pip install Appium-Python-Client selenium

2. Appium Server Setup

Install the Appium server and the UiAutomator2 driver for Android.

#1. Install Appium via npm

npm install -g appium

Then, in Appium Inspector, write this JSON for constructing Appium Server with this UI debugger

{
  "platformName": "Android",
  "appium:automationName": "UiAutomator2",
  "appium:deviceName": "Android Emulator",
  "appium:appPackage": "com.google.android.googlequicksearchbox",
  "appium:appActivity": "com.google.android.googlequicksearchbox.SearchActivity",
  "appium:noReset": true,
  "appium:ensureWebviewsHavePages": true,
  "appium:nativeWebScreenshot": true,
  "appium:newCommandTimeout": 3600,
  "appium:connectHardwareKeyboard": true
}

#2. Install the Android Driver

appium driver install uiautomator2

#3. Verify installation

appium driver list

3. Android Emulator Setup (Crucial Step)

1. Open Android Studio > Virtual Device Manager.
2. Create a new device: Pixel 7 (or Pixel 6).
3. System Image: Select a release that includes Google Play (e.g., Android 13/14 Tiramisu/Upside DownCake).
4. IMPORTANT: Once the emulator starts, open the Play Store app and Log in with a valid Google Account.
   • Why? Google Assistant requires an active account to process commands like "Turn on lights" or "Send message".
5. Initialize Assistant: Long-press the Home button on the emulator once to trigger the "Welcome" screen and complete the initial setup until the listening interface appears.

Environment Configuration (macOS/Linux)

Appium requires access to the Android SDK. You must export the ANDROID_HOME variable. Run the following commands in your terminal (or add them to ~/.zshrc):

export ANDROID_HOME=~/Library/Android/sdk
export ANDROID_SDK_ROOT=~/Library/Android/sdk
export PATH=$PATH:$ANDROID_HOME/platform-tools
export PATH=$PATH:$ANDROID_HOME/tools
export PATH=$PATH:$ANDROID_HOME/tools/bin

To verify, run adb devices. It should list your connected emulator.

Project Structure

Directory: Team9_TestFramework/

File/Folder	Description
venv/	Virtual Environment
home_client.py	[CORE] Appium Client Wrapper (UI Logic)
home_test_runner.py	[RUNNER] Data-Driven Test Engine
test_data.csv	[DATA] Test Cases (Input Commands & Expected Outputs)
README.md	Documentation

File Descriptions:

• home_client.py: Contains the HomeClient class. It manages the Appium driver, handles the logic to trigger KeyCode 219 (Assistant), switches input modes, and scrapes the screen for results.
• home_test_runner.py: The main entry point. It loads rows from the CSV, executes tests sequentially, calculates Pass/Fail rates, and generates a report.
• test_data.csv: Stores the test scenarios.

How to Run the Tests

Step 1: Start the Android Emulator

Launch your Pixel device from Android Studio or via terminal:

emulator -avd Pixel_7_API_34 # Example name

Step 2: Start the Appium Server

Open a new terminal window and run:

# Ensure ANDROID_HOME is set before running this
appium

Keep this terminal window open.

Step 3: Run the Test Automation Script

Open your main terminal (with venv activated) and run:

python home_test_runner.py

Test Data Strategy (Simulation)

Since the Emulator does not have physical IoT devices (Lights, TV) connected, we use Keyword Validation to verify that Google Assistant correctly interpreted the Intent.

Functionality	Command	Expected Keyword	Logic
Lamp	"Turn on the living room light"	lights	Google replies "Cannot find lights..." -> Intent recognized.
Telecom	"Call 123456789"	123456789	Google shows "Calling 123456789..." -> Intent recognized.
Music	"Sing a song"	song	Google replies "I can sing a song..." -> Intent recognized.
Message	"Send message to Alex..."	message	Google asks "Ready to send message?" -> Intent recognized.

Troubleshooting

1. Error: Neither ANDROID_HOME nor ANDROID_SDK_ROOT...

• Fix: You forgot to export the environment variables in the terminal running the Appium Server. Stop the server (Ctrl+C), run source ~/.zshrc, and restart appium.

2. Error: NoSuch Element or Timeout

• Fix: Google often updates element IDs. Use Appium Inspector to verify the Resource ID of the target element.
• Note: Ensure you Close the Appium Inspector Session before running the Python script, otherwise they will conflict.

3. Test fails to submit (Text stays in box)

• Fix: The framework uses a hybrid submit strategy. It tries to click the blue Send Button first. If not found, it falls back to the Enter Key (66).

---

© 2025 Team 9. All Rights Reserved.