Skip to content

Un-ignore screenshot tests and exclude them properly instead #17063

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AliSoftware merged 5 commits into from
Aug 22, 2022
Merged

Un-ignore screenshot tests and exclude them properly instead #17063

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
37503ba
Revert "Disabled `jPScreenshotTest`"
AliSoftware Aug 22, 2022
03e1246
Exclude screenshots tests from Connected Tests
AliSoftware Aug 22, 2022
393ab1a
Update `release-toolkit` version requirement to `~>5.4`
AliSoftware Aug 22, 2022
2939197
Un-ignore `wPScreenshotTest` completely
AliSoftware Aug 22, 2022
dbf7496
Fix checkstyle violation (unused import)
AliSoftware Aug 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Gemfile.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ PLATFORMS

DEPENDENCIES
fastlane (~> 2)
fastlane-plugin-wpmreleasetoolkit (~> 5.2)
fastlane-plugin-wpmreleasetoolkit (~> 5.4)
nokogiri
rmagick (~> 4.1)

Expand Down
2 changes: 0 additions & 2 deletions WordPress/src/androidTest/java/org/wordpress/android/ui/screenshots/JPScreenshotTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@
import com.google.android.libraries.cloudtesting.screenshots.ScreenShotter;

import org.junit.ClassRule;
import org.junit.Ignore;
import org.junit.Test;
import org.wordpress.android.BuildConfig;
import org.wordpress.android.R;
Expand Down Expand Up @@ -96,7 +95,6 @@ public static String buildScreenshotName(Screenshots screen) {
}
}

@Ignore("Ignored until there's a way to exclude tests on FTL properly, via `--test-targets` option.")
@Test
public void jPScreenshotTest() {
if (BuildConfig.IS_JETPACK_APP) {
Expand Down
3 changes: 1 addition & 2 deletions WordPress/src/androidTest/java/org/wordpress/android/ui/screenshots/WPScreenshotTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,7 @@ public class WPScreenshotTest extends BaseTest {


private DemoModeEnabler mDemoModeEnabler = new DemoModeEnabler();

@Ignore("Ignored until there's a way to exclude tests on FTL properly, via `--test-targets` option.")

Copy link
Contributor Author

@AliSoftware AliSoftware Aug 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that commit 789eb20 that changed this line in #16970 and that we reverted in this PR… still had the @Ignore annotation already (just without an explanation message until #16970).

I'm not sure why the test was already @Ignore'd even before we had to apply the workaround for the last of test_target: parameter in our FTL action. Probably because the WPScreenshotTest was in a bad place and failing at that time so someone decided to ignore it (and forgot to document why)?

But since the whole goal of this PR is to be in a good place to be able to work on Android screenshot tests again, so without them being @Ignore'd when we need to run them explicitly and in isolation, I figured it made more sense to remove the @Ignore completely here, instead of just reverting #16970 and keeping the @Ignore that we had before the workaround.

See: 2939197

[EDIT: Previous message linked to wrong PR; edited since.]

Copy link
Contributor

@pachlava pachlava Aug 22, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why the test was already @Ignore'd even before we had to apply the workaround for the last of test_target: parameter in our FTL action. Probably because the WPScreenshotTest was in a bad place and failing at that time so someone decided to ignore it

Exactly, the WPScreenshotTest was already ignored before #16970 because it was failing. #16970 really only disables the JP one, but adds a comment for both tests.

@Test
public void wPScreenshotTest() {
if (!BuildConfig.IS_JETPACK_APP) {
Expand Down
2 changes: 1 addition & 1 deletion fastlane/Pluginfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@ group :screenshots, optional: true do
end

# gem 'fastlane-plugin-wpmreleasetoolkit', git: '[email protected]:wordpress-mobile/release-toolkit.git', branch: 'trunk'
gem 'fastlane-plugin-wpmreleasetoolkit', '~> 5.2'
gem 'fastlane-plugin-wpmreleasetoolkit', '~> 5.4'
1 change: 1 addition & 0 deletions fastlane/lanes/test.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
version: 28,
test_apk_path: File.join(apk_dir, 'androidTest', 'wordpressVanilla', 'debug', 'org.wordpress.android-wordpress-vanilla-debug-androidTest.apk'),
apk_path: File.join(apk_dir, 'wordpressVanilla', 'debug', 'org.wordpress.android-wordpress-vanilla-debug.apk'),
test_targets: 'notPackage org.wordpress.android.ui.screenshots',
results_output_dir: File.join(PROJECT_ROOT_FOLDER, 'build', 'instrumented-tests')
)
end
Expand Down