Skip to content

gtests.md

Latest commit

 

History

History
228 lines (179 loc) · 9.7 KB

gtests.md

File metadata and controls

228 lines (179 loc) · 9.7 KB

Deploying and running gtests on Fuchsia

[TOC]

Fuchsia gtest binaries are deployed and executed via scripts that are automatically generated by the test() GN target. For each test, a wrapper scripts is created named run_<test_target_name> for running the Fuchsia package on the device. These executables are found in ${OUTPUT_DIR}/bin.

The aforementioned devices can be either emulators started by the scripts themselves, an existing emulator instance, or a physical device. To build a gtest binary, check this documentation.

For the sake of this example, we will be using base_unittests as the package we wish to install and/or execute.

If using anything other than the default Fuchsia boot image, see Supported Fuchsia Product configurations.

For details of the implementation of run_<test_target_name> scripts, see here.

Hermetic emulation

The test script brings up an emulator, runs the tests on it, and shuts the emulator down when finished.

$ out/fuchsia/bin/run_base_unittests

The flag --custom-image can be used to specify the Fuchsia boot image used by the emulator. For instance, setting --custom-image=workstation.qemu-x64-release would run the test on a workstation image.

Run on a persistent emulator

You can also run tests on physical devices or start a persistent emulator from the Chromium tree. To start a persistent emulator, run:

$ ./build/fuchsia/test/start_emulator.py

Part of the output should be like the following:

INFO:root:Emulator successfully started. You can now run Chrome Fuchsia tests with --target-id=fuchsia-emulator-198 to target this emulator.

You can run tests on persistent devices by adding the command line arguments indicated above. For example, to run base_unittests you would run the following command:

$ out/fuchsia/bin/run_base_unittests --target-id fuchsia-emulator-198

If only one Fuchsia device is connected, you can use the following command:

$ out/fuchsia/bin/run_base_unittests -d

Run on a device paved with Fuchsia built from source

Make sure that the CPU architecture of your Chromium output directory matches the architecture of the Fuchsia output directory (x64==x64, arm64==arm64, etc.).

$ out/fuchsia/bin/run_base_unittests -d --repo [FUCHSIA_OUT_DIR]/amber-files --no-repo-init

Note that you should not have fx serve running as Chromium will handle serving the repository.

Run on a device the host is connected to remotely via ssh

$ out/fuchsia/bin/run_base_unittests --target-id=[::1]:8022

Troubleshooting a test

To troubleshoot a specific test, consider a combination of the following arguments to the test runner script:

  • --gtest_filter="[TestSuite.TestName]" to only run a specific test, or a comma-separated list to run a set of tests. Wildcards can also be used to run a set of tests or an entire test suite, e.g. --gtest_filter="[TestSuite.*]".
  • --test-launcher-jobs=1 to only have one batch of tests running at a time. This bypasses the test launcher buffering of test log output, making it possible to access the log output from successful test runs.
  • --single-process-tests to run all the tests in the same process. Unlike the above option, this will run the tests directly in the test launcher process, making it easier to attach a debugger.
  • "--logs-dir=[/path/to/log/directory]` to specify the directory to write logs to. By default, Chromium logs are written to the "system_log" file in that directory.
  • --gtest_repeat=[number] --gtest_break_on_failure to run a test or test suite a certain number of times until it fails. This is useful to investigate flaky tests.

Supported Fuchsia Product configurations

The prebuilt Terminal and Workstation images downloaded by gclient sync are specifically configured to support running Chromium tests. It is also possible to run Chromium tests on other Product configurations, such as Core, using a custom build.

Use the default unless you have a specific reason to run on a different Product. For example, if you need to reproduce an issue occurring on a bot using that Product.

Terminal

Chromium gtests are primarily supported (i.e., pass all tests) on the Terminal Fuchsia Product configuration. This is the default image used for hermetic emulation and when starting a persistent emulator.

Workstation

Workstation configurations also generally support most tests.

Core

By default, Fuchsia Core Product configurations do not support running Chromium tests. Similarly, the Core images provided with the Fuchsia SDK are unable to run Chromium tests. (In the future, it may be possible to run them on such images with a corresponding TUF repo.)

When working with a local Fuchsia Core build, there are have two options for running Chromium tests.

  1. Combined repo: Follow the instructions in Run on a device paved with Fuchsia built from source.
  2. Packages in base: Add the additional required packages to the base packages.

Using method (1) with the default configuration for core.qemu-x64, all base_unittests pass. The same tests pass using method (2) with modifications to core.qemu-x64's configuration.

Most other test suites will require additional packages used (indirectly) by those tests. Some of these packages may not even be in Core's default set of universe packages and need to be explicitly added as described in Adding packages.

Additional required packages

At a minimum, test_manager is required in order to run tests. This is already available when using a combined repo as in option (1).

Another required package is //src/developer/build_info/testing:fake-build-info, which is not available in Core by default.

Other useful packages include:

  • intl_property_manager - avoids many warnings when the test shard tries to launch it.
  • audio_core - avoids warnings related to fuchsia.media.ProfileProvider.

This is sufficient to pass base_unittests and eliminate most warnings.

Adding packages

To add packages, find the path to the package's build target in the Fuchsia code and add --with[-base] followed by //path_from_fuchsia_root:target_name to the fx set command for the Fuchsia build.

Combined repo

When using a combined repo as in option (1), packages that are already part of the Core configuration (including cached and universe packages), are available. Thus, only packages not included in Core's base, cached, or universe packages, need to be added. Similarly, --with, --with-cache, and --with-base are all acceptable ways to add such packages.

The following makes all additional required packages available to the Chromium tests.

$ fx set core.qemu-x64 --auto-dir --release --with //src/developer/build_info/testing:fake-build-info --with //src/testing/fidl/intl_property_manager --with //src/media/audio/audio_core
Chromium-only repo: packages in base

By default - and specifically when not using option (1) - Chromium's test scripts only serve packages from the Chromium workspace, meaning that Fuchsia provided packages not in Core's base packages cannot be resolved. Thus, all Fuchsia packages needed during the tests must be added to base.

In this case --with-base is the only acceptable method of specifying packages, and test_manager must be explicitly added to base packages by including --with-base //src/sys/test_manager. The following is the minimum fx set command line for this case.

$ fx set core.qemu-x64 --auto-dir --release --with-base //src/sys/test_manager

For the purposes of successfully running Chromium tests, the following is equivalent to the command line in Combined repo. It adds all additional required packages to base packages. It also adds //sdk/lib/sys/cpp to base packages to enable base_unittests's TestComponentContextForProcessTest.ProvideSystemService test to pass.

$ fx set core.qemu-x64 --auto-dir --release  --with-base //src/sys/test_manager  --with-base //src/developer/build_info/testing:fake-build-info --with-base //src/testing/fidl/intl_property_manager --with-base //src/media/audio/audio_core --with-base //sdk/lib/sys/cpp

Other Products

Other Fuchsia Product configurations may not support running Chromium tests by default and require some subset of the additional required packages needed for Core as well as others for specific test suites.