Add Firebase Test Lab Support (#355)

* Add Android Firebase Action + Helper

* Move Firebase Device out of the Helper

* Move log file parsing out of the FirebaseHelper

The Firebase Helper had a lot of pieces doing a variety of things – splitting this into model objects makes it easier to change and test.

* Add CHANGELOG note

* Cleanup

* Apply suggestions from code review

Co-authored-by: Olivier Halligon <olivier.halligon@automattic.com>

* Fix tests

* Don’t authenticate in constructor

* Verify gcloud binary on FirebaseTestRunner.new

* Make CI happy

* Remove the explicit verify_has_gloud_binary call

* Simplify more_details_url

Co-authored-by: Olivier Halligon <olivier.halligon@automattic.com>

* Simplify raw_results_paths

Co-authored-by: Olivier Halligon <olivier.halligon@automattic.com>

* Adjust `require`

* Add docs

* Fix broken success? method

* Fix `passed?`

* Fix Google Cloud Storage gem pinning

* Apply suggestions from code review

Co-authored-by: Olivier Halligon <olivier.halligon@automattic.com>

* Fix tests

* Simplify parameters

* Indentation Fix

Co-authored-by: Olivier Halligon <olivier.halligon@automattic.com>

* Fix an oops

* Re-add default_value_dynamic

* Fix unclear errors when gcloud binary missing

* Escape all the things

* Improve Firebase Test Runner Errors

* Remove constants

* Fix lint issues

* Fix rubocop error

* Document the `env_name`

* Remove key file environment variable setting

* Improve argument documentation

* Avoid double escaping

* Apply suggestions from code review

Co-authored-by: Olivier Halligon <olivier.halligon@automattic.com>

* Fix crashes

* Check params early to avoid late prompts

* Add Action Validation Tests

* Fix params

* Add Firebase Login Action

* Remove login logic from Firebase_Test Action

* Put back key file parameter

* Make FirebaseTestRunner static

* Manually specify the project_id field

Co-authored-by: Olivier Halligon <olivier.halligon@automattic.com>

Author: jkmassel

.rubocop.yml

@@ -67,7 +67,7 @@ Metrics/ClassLength:
   Max: 300
 
 Metrics/MethodLength:
-  Max: 100
+  Max: 150
 
 Metrics/ModuleLength:
   Max: 300

CHANGELOG.md

@@ -11,6 +11,7 @@
 ### New Features
 
 * Introduce new `ios_send_app_size_metrics` and `android_send_app_size_metrics` actions. [#364] [#365]
+* Add the ability to run Firebase Test Lab tests. [#355]
 
 ### Bug Fixes
 

Gemfile.lock

@@ -8,6 +8,7 @@ PATH
       chroma (= 0.2.0)
       diffy (~> 3.3)
       git (~> 1.3)
+      google-cloud-storage (~> 1.31)
       jsonlint (~> 0.3)
       nokogiri (~> 1.11)
       octokit (~> 4.18)

fastlane-plugin-wpmreleasetoolkit.gemspec

@@ -38,6 +38,10 @@ Gem::Specification.new do |spec|
   spec.add_dependency 'progress_bar', '~> 1.3'
   spec.add_dependency 'rake', '>= 12.3', '< 14.0'
   spec.add_dependency 'rake-compiler', '~> 1.0'
+
+  # `google-cloud-storage` is required by fastlane, but we pin it in case it's not in the future
+  spec.add_dependency 'google-cloud-storage', '~> 1.31'
+
   # Some of the upstream code uses `BigDecimal.new` which version 2.0 of the
   # `bigdecimal` gem removed. Until we'll find the time to identify the
   # dependencies and see if we can move them to something compatible with

lib/fastlane/plugin/wpmreleasetoolkit.rb

@@ -2,9 +2,9 @@
 
 module Fastlane
   module Wpmreleasetoolkit
-    # Return all .rb files inside the "actions" and "helper" directory
+    # Return all .rb files inside the "actions", "helper" and "models" directories
     def self.all_classes
-      Dir[File.expand_path('**/{actions,helper}/**/*.rb', File.dirname(__FILE__))]
+      Dir[File.expand_path('**/{actions,helper,models}/**/*.rb', File.dirname(__FILE__))]
     end
   end
 end

lib/fastlane/plugin/wpmreleasetoolkit/actions/android/android_firebase_test.rb

@@ -0,0 +1,187 @@
+require 'securerandom'
+
+module Fastlane
+  module Actions
+    module SharedValues
+      FIREBASE_TEST_RESULT = :FIREBASE_TEST_LOG_FILE
+      FIREBASE_TEST_LOG_FILE_PATH = :FIREBASE_TEST_LOG_FILE_PATH
+    end
+
+    class AndroidFirebaseTestAction < Action
+      def self.run(params)
+        validate_options(params)
+
+        UI.user_error!('You must be logged in to Firebase prior to calling this action. Use the `FirebaseLogin` Action to log in if needed') unless Fastlane::FirebaseAccount.authenticated?
+
+        # Log in to Firebase (and validate credentials)
+        run_uuid = params[:test_run_id] || SecureRandom.uuid
+        test_dir = params[:results_output_dir] || File.join(Dir.tmpdir(), run_uuid)
+
+        # Set up the log file and output directory
+        FileUtils.mkdir_p(test_dir)
+        Fastlane::Actions.lane_context[:FIREBASE_TEST_LOG_FILE_PATH] = File.join(test_dir, 'output.log')
+
+        device = Fastlane::FirebaseDevice.new(
+          model: params[:model],
+          version: params[:version],
+          locale: params[:locale],
+          orientation: params[:orientation]
+        )
+
+        result = FirebaseTestRunner.run_tests(
+          project_id: params[:project_id],
+          apk_path: params[:apk_path],
+          test_apk_path: params[:test_apk_path],
+          device: device,
+          type: params[:type]
+        )
+
+        # Download all of the outputs from the job to the local machine
+        FirebaseTestRunner.download_result_files(
+          result: result,
+          destination: test_dir,
+          project_id: params[:project_id],
+          key_file_path: params[:key_file]
+        )
+
+        FastlaneCore::UI.test_failure! "Firebase Tests failed – more information can be found at #{result.more_details_url}" unless result.success?
+
+        UI.success 'Firebase Tests Complete'
+      end
+
+      # Fastlane doesn't eagerly validate options for us, so we'll do it first to have control over
+      # when they're evalutated.
+      def self.validate_options(params)
+        available_options
+          .reject { |opt| opt.optional || !opt.default_value.nil? }
+          .map(&:key)
+          .each { |k| params[k] }
+      end
+
+      #####################################################
+      # @!group Documentation
+      #####################################################
+
+      def self.description
+        'Runs the specified tests in Firebase Test Lab'
+      end
+
+      def self.details
+        description
+      end
+
+      def self.available_options
+        [
+          FastlaneCore::ConfigItem.new(
+            key: :project_id,
+            # `env_name` comes from the Google Cloud default: https://cloud.google.com/functions/docs/configuring/env-var
+            env_name: 'GCP_PROJECT',
+            description: 'The Project ID to test in',
+            type: String
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :key_file,
+            description: 'The key file used to authorize with Google Cloud',
+            type: String,
+            verify_block: proc do |value|
+              UI.user_error!('The `:key_file` parameter is required') if value.empty?
+              UI.user_error!("No Google Cloud Key file found at: #{value}") unless File.exist?(value)
+            end
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :apk_path,
+            description: 'Path to the application APK on the local machine',
+            type: String,
+            verify_block: proc do |value|
+              UI.user_error!('The `:apk_path` parameter is required') if value.empty?
+              UI.user_error!("Invalid application APK: #{value}") unless File.exist?(value)
+            end
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :test_apk_path,
+            description: 'Path to the test bundle APK on the local machine',
+            type: String,
+            verify_block: proc do |value|
+              UI.user_error!('The `:test_apk_path` parameter is required') if value.empty?
+              UI.user_error!("Invalid test APK: #{value}") unless File.exist?(value)
+            end
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :model,
+            description: 'The device model to use to run the test',
+            type: String,
+            verify_block: proc do |value|
+              UI.user_error!('The `:model` parameter is required') if value.empty?
+              FirebaseTestRunner.verify_has_gcloud_binary!
+              model_names = Fastlane::FirebaseDevice.valid_model_names
+              UI.user_error!("Invalid Model Name: #{value}. Valid Model Names: #{model_names}") unless model_names.include?(value)
+            end
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :version,
+            description: 'The Android version (API Level) to use to run the test',
+            type: Integer,
+            verify_block: proc do |value|
+              FirebaseTestRunner.verify_has_gcloud_binary!
+              version_numbers = Fastlane::FirebaseDevice.valid_version_numbers
+              UI.user_error!("Invalid Version Number: #{value}. Valid Version Numbers: #{version_numbers}") unless version_numbers.include?(value)
+            end
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :locale,
+            description: 'The locale code to use when running the test',
+            type: String,
+            default_value: 'en',
+            verify_block: proc do |value|
+              FirebaseTestRunner.verify_has_gcloud_binary!
+              locale_codes = Fastlane::FirebaseDevice.valid_locales
+              UI.user_error!("Invalid Locale: #{value}. Valid Locales: #{locale_codes}") unless locale_codes.include?(value)
+            end
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :orientation,
+            description: 'Which orientation to run the device in',
+            type: String,
+            default_value: 'portrait',
+            verify_block: proc do |value|
+              orientations = Fastlane::FirebaseDevice.valid_orientations
+              UI.user_error!("Invalid Orientation: #{value}. Valid Orientations: #{orientations}") unless orientations.include?(value)
+            end
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :type,
+            description: 'The type of test to run (e.g. `instrumentation` or `robo`)',
+            type: String,
+            default_value: 'instrumentation',
+            verify_block: proc do |value|
+              types = Fastlane::FirebaseTestRunner::VALID_TEST_TYPES
+              UI.user_error!("Invalid Test Type: #{value}. Valid Types: #{types}") unless types.include?(value)
+            end
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :test_run_id,
+            description: 'A unique ID used to identify this test run',
+            default_value_dynamic: true,
+            optional: true,
+            type: String
+          ),
+          FastlaneCore::ConfigItem.new(
+            key: :results_output_dir,
+            description: 'The path to the folder where we will store the results of this test run',
+            default_value_dynamic: true,
+            optional: true,
+            type: String
+          ),
+        ]
+      end
+
+      def self.authors
+        ['Automattic']
+      end
+
+      def self.is_supported?(platform)
+        platform == :android
+      end
+    end
+  end
+end

lib/fastlane/plugin/wpmreleasetoolkit/actions/android/android_send_app_size_metrics.rb

@@ -152,7 +152,6 @@ def self.details
         DETAILS
       end
 
-      # rubocop:disable Metrics/MethodLength
       def self.available_options
         [
           FastlaneCore::ConfigItem.new(
@@ -259,7 +258,6 @@ def self.available_options
           ),
         ]
       end
-      # rubocop:enable Metrics/MethodLength
 
       def self.return_type
         :integer

lib/fastlane/plugin/wpmreleasetoolkit/actions/common/firebase_login.rb

@@ -0,0 +1,44 @@
+require 'securerandom'
+
+module Fastlane
+  module Actions
+    class FirebaseLoginAction < Action
+      def self.run(params)
+        Fastlane::FirebaseAccount.activate_service_account_with_key_file(params[:key_file])
+      end
+
+      #####################################################
+      # @!group Documentation
+      #####################################################
+      def self.description
+        'Logs the local machine into Google Cloud using the provided key file'
+      end
+
+      def self.details
+        description
+      end
+
+      def self.available_options
+        [
+          FastlaneCore::ConfigItem.new(
+            key: :key_file,
+            description: 'The key file used to authorize with Google Cloud',
+            type: String,
+            verify_block: proc do |value|
+              UI.user_error!('The `:key_file` parameter is required') if value.empty?
+              UI.user_error!("No Google Cloud Key file found at: #{value}") unless File.exist?(value)
+            end
+          ),
+        ]
+      end
+
+      def self.authors
+        ['Automattic']
+      end
+
+      def self.is_supported?(platform)
+        true
+      end
+    end
+  end
+end

lib/fastlane/plugin/wpmreleasetoolkit/models/firebase_account.rb

@@ -0,0 +1,19 @@
+module Fastlane
+  class FirebaseAccount
+    def self.activate_service_account_with_key_file(key_file_path)
+      Fastlane::Actions.sh('gcloud', 'auth', 'activate-service-account', '--key-file', key_file_path)
+    end
+
+    def self.authenticated?
+      auth_status = JSON.parse(auth_status_data)
+      auth_status.any? do |account|
+        account['status'] == 'ACTIVE'
+      end
+    end
+
+    # Lookup the current account authentication status
+    def self.auth_status_data
+      Fastlane::Actions.sh('gcloud', 'auth', 'list', '--format', 'json', log: false)
+    end
+  end
+end

lib/fastlane/plugin/wpmreleasetoolkit/models/firebase_device.rb

@@ -0,0 +1,62 @@
+module Fastlane
+  class FirebaseDevice
+    attr_reader :model, :version, :locale, :orientation
+
+    def initialize(model:, version:, orientation:, locale: 'en')
+      raise 'Invalid Model' unless FirebaseDevice.valid_model_names.include? model
+      raise 'Invalid Version' unless FirebaseDevice.valid_version_numbers.include? version
+      raise 'Invalid Locale' unless FirebaseDevice.valid_locales.include? locale
+      raise 'Invalid Orientation' unless FirebaseDevice.valid_orientations.include? orientation
+
+      @model = model
+      @version = version
+      @locale = locale
+      @orientation = orientation
+    end
+
+    def to_s
+      "model=#{@model},version=#{@version},locale=#{@locale},orientation=#{@orientation}"
+    end
+
+    class << self
+      @locale_data = nil
+      @model_data = nil
+      @version_data = nil
+
+      def valid_model_names
+        JSON.parse(model_data).map { |device| device['codename'] }
+      end
+
+      def valid_version_numbers
+        JSON.parse(version_data).map { |version| version['apiLevel'].to_i }
+      end
+
+      def valid_locales
+        JSON.parse(locale_data).map { |locale| locale['id'] }
+      end
+
+      def valid_orientations
+        %w[portrait landscape]
+      end
+
+      def locale_data
+        FirebaseDevice.verify_logged_in!
+        @locale_data ||= Fastlane::Actions.sh('gcloud', 'firebase', 'test', 'android', 'locales', 'list', '--format="json"', log: false)
+      end
+
+      def model_data
+        FirebaseDevice.verify_logged_in!
+        @model_data ||= Fastlane::Actions.sh('gcloud', 'firebase', 'test', 'android', 'models', 'list', '--format="json"', log: false)
+      end
+
+      def version_data
+        FirebaseDevice.verify_logged_in!
+        @version_data ||= Fastlane::Actions.sh('gcloud', 'firebase', 'test', 'android', 'versions', 'list', '--format="json"', log: false)
+      end
+
+      def verify_logged_in!
+        UI.user_error!('You must call `firebase_login` before creating a FirebaseDevice object') unless FirebaseAccount.authenticated?
+      end
+    end
+  end
+end