Skip to content

Commit 886158d

Browse files
JoseAlcerrecaSimonMarquis
JoseAlcerreca
and
SimonMarquis
authored
Adds Screenshot testing with Roborazzi (#876)
* Adds screenshot tests using Roborazzi (Robolectric Native Graphics) - Adds Roborazzi to convention plugins - Adds Screenshot helper in :core-testing - Creates screenshot suites for :app and :feature-foryou * CI and spotless * Moves :app tests to testDemo and makes NiaAppScreenSizesScreenshotTests prettier * CI: Moves local tests to their own step * CI: Adds --rerun to screenshot task * CI: Moves screenshots before local tests * CI: Fixes wrong if statement in workflow * CI WIP: trying to trigger the push step * CI: Re-enables roborazzi verification * Fixes flaky screenshot tests by setting LocalInspectionMode on * CI: screenshot commits now use the original author intead of bot account * CI: Disables globbing because file_pattern didn't work * CI: Trying new file pattern for png files * CI: Adds a check for forks * 🤖 Updates screenshots * Code review: toml cleanup, comments * Use new github.event.pull_request.head.repo.fork Co-authored-by: Simon Marquis <[email protected]> * Uses Robolectric qualifiers to set the dpi, adds section to README * Spotless * Delegates creation of repository to Hilt in test * Revert "Use new github.event.pull_request.head.repo.fork" * 🤖 Updates screenshots * Empty commit to trigger GHA on main branch * Makes time zones deterministic in screenshot tests * Increases GMD timeout to 90m, but it has to be reduced --------- Co-authored-by: Simon Marquis <[email protected]>
1 parent 4716c7f commit 886158d

File tree

35 files changed

+600
-11
lines changed

35 files changed

+600
-11
lines changed

‎.github/workflows/Build.yaml‎

Lines changed: 62 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ concurrency:
1313
jobs:
1414
build:
1515
runs-on: ubuntu-latest
16-
timeout-minutes: 60
16+
timeout-minutes: 90
1717

1818
steps:
1919
- name: Checkout
@@ -43,9 +43,6 @@ jobs:
4343
- name: Build all build type and flavor permutations
4444
run: ./gradlew assemble
4545

46-
- name: Run local tests
47-
run: ./gradlew testDemoDebug testProdDebug
48-
4946
- name: Upload build outputs (APKs)
5047
uses: actions/upload-artifact@v3
5148
with:
@@ -59,6 +56,65 @@ jobs:
5956
name: lint-reports
6057
path: '**/build/reports/lint-results-*.html'
6158

59+
test:
60+
runs-on: ubuntu-latest
61+
62+
permissions:
63+
contents: write
64+
65+
timeout-minutes: 60
66+
67+
steps:
68+
- name: Checkout
69+
uses: actions/checkout@v3
70+
71+
- name: Validate Gradle Wrapper
72+
uses: gradle/wrapper-validation-action@v1
73+
74+
- name: Copy CI gradle.properties
75+
run: mkdir -p ~/.gradle ; cp .github/ci-gradle.properties ~/.gradle/gradle.properties
76+
77+
- name: Set up JDK 17
78+
uses: actions/setup-java@v3
79+
with:
80+
distribution: 'zulu'
81+
java-version: 17
82+
83+
- name: Setup Gradle
84+
uses: gradle/gradle-build-action@v2
85+
86+
- name: Run all local screenshot tests (Roborazzi)
87+
id: screenshotsverify
88+
continue-on-error: true
89+
run: ./gradlew verifyRoborazziDemoDebug
90+
91+
- name: Prevent pushing new screenshots if this is a fork
92+
id: checkfork
93+
continue-on-error: false
94+
if: steps.screenshotsverify.outcome == 'failure' && github.event.pull_request.head.repo.full_name != github.repository
95+
run: |
96+
echo "::error::Screenshot tests failed, please create a PR in your fork first." && exit 1
97+
98+
# Runs if previous job failed
99+
- name: Generate new screenshots if verification failed and it's a PR
100+
id: screenshotsrecord
101+
if: steps.screenshotsverify.outcome == 'failure' && github.event_name == 'pull_request'
102+
run: |
103+
./gradlew recordRoborazziDemoDebug
104+
105+
- name: Push new screenshots if available
106+
uses: stefanzweifel/git-auto-commit-action@v4
107+
if: steps.screenshotsrecord.outcome == 'success'
108+
with:
109+
file_pattern: '*/*.png'
110+
disable_globbing: true
111+
commit_message: "🤖 Updates screenshots"
112+
113+
# Run local tests after screenshot tests to avoid wrong UP-TO-DATE. TODO: Ignore screenshots.
114+
- name: Run local tests
115+
if: always()
116+
run: ./gradlew testDemoDebug testProdDebug
117+
62118
- name: Upload test results (XML)
63119
if: always()
64120
uses: actions/upload-artifact@v3
@@ -77,7 +133,7 @@ jobs:
77133
steps:
78134
- name: Checkout
79135
uses: actions/checkout@v3
80-
136+
81137
- name: Copy CI gradle.properties
82138
run: mkdir -p ~/.gradle ; cp .github/ci-gradle.properties ~/.gradle/gradle.properties
83139

@@ -113,7 +169,7 @@ jobs:
113169
androidTest-GMD:
114170
needs: build
115171
runs-on: macOS-latest # enables hardware acceleration in the virtual machine
116-
timeout-minutes: 55
172+
timeout-minutes: 90
117173

118174
steps:
119175
- name: Checkout

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,13 @@ Examples:
109109
manipulate the state of the `Test` repository and verify the resulting behavior, instead of
110110
checking that specific repository methods were called.
111111

112+
## Screenshot tests
113+
114+
**Now In Android** uses [Roborazzi](https://github.com/takahirom/roborazzi) to do screenshot tests
115+
of certain screens and components. To run these tests, run the `verifyRoborazziDemoDebug` or
116+
`recordRoborazziDemoDebug` tasks. Note that screenshots are recorded on CI, using Linux, and other
117+
platforms might generate slightly different images, making the tests fail.
118+
112119
# UI
113120
The app was designed using [Material 3 guidelines](https://m3.material.io/). Learn more about the design process and
114121
obtain the design files in the [Now in Android Material 3 Case Study](https://goo.gle/nia-figma) (design assets [also available as a PDF](docs/Now-In-Android-Design-File.pdf)).

‎app/build.gradle.kts‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,4 +120,16 @@ dependencies {
120120
implementation(libs.androidx.profileinstaller)
121121
implementation(libs.kotlinx.coroutines.guava)
122122
implementation(libs.coil.kt)
123+
124+
// Core functions
125+
testImplementation(project(":core:testing"))
126+
testImplementation(project(":core:datastore-test"))
127+
testImplementation(project(":core:data-test"))
128+
testImplementation(project(":core:network"))
129+
testImplementation(libs.androidx.navigation.testing)
130+
testImplementation(libs.accompanist.testharness)
131+
testImplementation(kotlin("test"))
132+
implementation(libs.work.testing)
133+
kaptTest(libs.hilt.compiler)
134+
123135
}

‎app/src/testDemo/java/com/google/samples/apps/nowinandroid/ui/NiaAppScreenSizesScreenshotTests.kt‎

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
/*
2+
* Copyright 2022 The Android Open Source Project
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License");
5+
* you may not use this file except in compliance with the License.
6+
* You may obtain a copy of the License at
7+
*
8+
* https://www.apache.org/licenses/LICENSE-2.0
9+
*
10+
* Unless required by applicable law or agreed to in writing, software
11+
* distributed under the License is distributed on an "AS IS" BASIS,
12+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
* See the License for the specific language governing permissions and
14+
* limitations under the License.
15+
*/
16+
17+
package com.google.samples.apps.nowinandroid.ui
18+
19+
import android.util.Log
20+
import androidx.compose.foundation.layout.BoxWithConstraints
21+
import androidx.compose.material3.windowsizeclass.ExperimentalMaterial3WindowSizeClassApi
22+
import androidx.compose.material3.windowsizeclass.WindowSizeClass
23+
import androidx.compose.runtime.CompositionLocalProvider
24+
import androidx.compose.ui.platform.LocalInspectionMode
25+
import androidx.compose.ui.test.junit4.createAndroidComposeRule
26+
import androidx.compose.ui.test.onRoot
27+
import androidx.compose.ui.unit.Dp
28+
import androidx.compose.ui.unit.DpSize
29+
import androidx.compose.ui.unit.dp
30+
import androidx.test.platform.app.InstrumentationRegistry
31+
import androidx.work.Configuration
32+
import androidx.work.testing.SynchronousExecutor
33+
import androidx.work.testing.WorkManagerTestInitHelper
34+
import com.github.takahirom.roborazzi.captureRoboImage
35+
import com.google.accompanist.testharness.TestHarness
36+
import com.google.samples.apps.nowinandroid.core.data.repository.TopicsRepository
37+
import com.google.samples.apps.nowinandroid.core.data.repository.UserDataRepository
38+
import com.google.samples.apps.nowinandroid.core.data.repository.UserNewsResourceRepository
39+
import com.google.samples.apps.nowinandroid.core.data.util.NetworkMonitor
40+
import com.google.samples.apps.nowinandroid.core.testing.util.DefaultRoborazziOptions
41+
import com.google.samples.apps.nowinandroid.uitesthiltmanifest.HiltComponentActivity
42+
import dagger.hilt.android.testing.BindValue
43+
import dagger.hilt.android.testing.HiltAndroidRule
44+
import dagger.hilt.android.testing.HiltAndroidTest
45+
import dagger.hilt.android.testing.HiltTestApplication
46+
import kotlinx.coroutines.flow.first
47+
import kotlinx.coroutines.runBlocking
48+
import org.junit.Before
49+
import org.junit.Rule
50+
import org.junit.Test
51+
import org.junit.rules.TemporaryFolder
52+
import org.junit.runner.RunWith
53+
import org.robolectric.RobolectricTestRunner
54+
import org.robolectric.annotation.Config
55+
import org.robolectric.annotation.GraphicsMode
56+
import org.robolectric.annotation.LooperMode
57+
import java.util.TimeZone
58+
import javax.inject.Inject
59+
60+
/**
61+
* Tests that the navigation UI is rendered correctly on different screen sizes.
62+
*/
63+
@OptIn(ExperimentalMaterial3WindowSizeClassApi::class)
64+
@RunWith(RobolectricTestRunner::class)
65+
@GraphicsMode(GraphicsMode.Mode.NATIVE)
66+
// Configure Robolectric to use a very large screen size that can fit all of the test sizes.
67+
// This allows enough room to render the content under test without clipping or scaling.
68+
@Config(application = HiltTestApplication::class, qualifiers = "w1000dp-h1000dp-480dpi", sdk = [33])
69+
@LooperMode(LooperMode.Mode.PAUSED)
70+
@HiltAndroidTest
71+
class NiaAppScreenSizesScreenshotTests {
72+
73+
/**
74+
* Manages the components' state and is used to perform injection on your test
75+
*/
76+
@get:Rule(order = 0)
77+
val hiltRule = HiltAndroidRule(this)
78+
79+
/**
80+
* Create a temporary folder used to create a Data Store file. This guarantees that
81+
* the file is removed in between each test, preventing a crash.
82+
*/
83+
@BindValue
84+
@get:Rule(order = 1)
85+
val tmpFolder: TemporaryFolder = TemporaryFolder.builder().assureDeletion().build()
86+
87+
/**
88+
* Use a test activity to set the content on.
89+
*/
90+
@get:Rule(order = 2)
91+
val composeTestRule = createAndroidComposeRule<HiltComponentActivity>()
92+
93+
@Inject
94+
lateinit var networkMonitor: NetworkMonitor
95+
96+
@Inject
97+
lateinit var userDataRepository: UserDataRepository
98+
99+
@Inject
100+
lateinit var topicsRepository: TopicsRepository
101+
102+
@Inject
103+
lateinit var userNewsResourceRepository: UserNewsResourceRepository
104+
105+
@Before
106+
fun setup() {
107+
val config = Configuration.Builder()
108+
.setMinimumLoggingLevel(Log.DEBUG)
109+
.setExecutor(SynchronousExecutor())
110+
.build()
111+
112+
// Initialize WorkManager for instrumentation tests.
113+
WorkManagerTestInitHelper.initializeTestWorkManager(
114+
InstrumentationRegistry.getInstrumentation().context,
115+
config,
116+
)
117+
118+
hiltRule.inject()
119+
120+
// Configure user data
121+
runBlocking {
122+
userDataRepository.setShouldHideOnboarding(true)
123+
124+
userDataRepository.setFollowedTopicIds(
125+
setOf(topicsRepository.getTopics().first().first().id),
126+
)
127+
}
128+
}
129+
130+
@Before
131+
fun setTimeZone() {
132+
// Make time zone deterministic in tests
133+
TimeZone.setDefault(TimeZone.getTimeZone("UTC"))
134+
}
135+
136+
private fun testNiaAppScreenshotWithSize(width: Dp, height: Dp, screenshotName: String) {
137+
composeTestRule.setContent {
138+
CompositionLocalProvider(
139+
LocalInspectionMode provides true,
140+
) {
141+
TestHarness(size = DpSize(width, height)) {
142+
BoxWithConstraints {
143+
NiaApp(
144+
windowSizeClass = WindowSizeClass.calculateFromSize(
145+
DpSize(maxWidth, maxHeight),
146+
),
147+
networkMonitor = networkMonitor,
148+
userNewsResourceRepository = userNewsResourceRepository,
149+
)
150+
}
151+
}
152+
}
153+
}
154+
155+
composeTestRule.onRoot()
156+
.captureRoboImage(
157+
"src/testDemo/screenshots/$screenshotName.png",
158+
roborazziOptions = DefaultRoborazziOptions,
159+
)
160+
}
161+
162+
@Test
163+
fun compactWidth_compactHeight_showsNavigationBar() {
164+
testNiaAppScreenshotWithSize(
165+
610.dp,
166+
400.dp,
167+
"compactWidth_compactHeight_showsNavigationBar",
168+
)
169+
}
170+
171+
@Test
172+
fun mediumWidth_compactHeight_showsNavigationRail() {
173+
testNiaAppScreenshotWithSize(
174+
610.dp,
175+
400.dp,
176+
"mediumWidth_compactHeight_showsNavigationRail",
177+
)
178+
}
179+
180+
@Test
181+
fun expandedWidth_compactHeight_showsNavigationRail() {
182+
testNiaAppScreenshotWithSize(
183+
900.dp,
184+
400.dp,
185+
"expandedWidth_compactHeight_showsNavigationRail",
186+
)
187+
}
188+
189+
@Test
190+
fun compactWidth_mediumHeight_showsNavigationBar() {
191+
testNiaAppScreenshotWithSize(
192+
400.dp,
193+
500.dp,
194+
"compactWidth_mediumHeight_showsNavigationBar",
195+
)
196+
}
197+
198+
@Test
199+
fun mediumWidth_mediumHeight_showsNavigationRail() {
200+
testNiaAppScreenshotWithSize(
201+
610.dp,
202+
500.dp,
203+
"mediumWidth_mediumHeight_showsNavigationRail",
204+
)
205+
}
206+
207+
@Test
208+
fun expandedWidth_mediumHeight_showsNavigationRail() {
209+
testNiaAppScreenshotWithSize(
210+
900.dp,
211+
500.dp,
212+
"expandedWidth_mediumHeight_showsNavigationRail",
213+
)
214+
}
215+
216+
@Test
217+
fun compactWidth_expandedHeight_showsNavigationBar() {
218+
testNiaAppScreenshotWithSize(
219+
400.dp,
220+
1000.dp,
221+
"compactWidth_expandedHeight_showsNavigationBar",
222+
)
223+
}
224+
225+
@Test
226+
fun mediumWidth_expandedHeight_showsNavigationRail() {
227+
testNiaAppScreenshotWithSize(
228+
610.dp,
229+
1000.dp,
230+
"mediumWidth_expandedHeight_showsNavigationRail",
231+
)
232+
}
233+
234+
@Test
235+
fun expandedWidth_expandedHeight_showsNavigationRail() {
236+
testNiaAppScreenshotWithSize(
237+
900.dp,
238+
1000.dp,
239+
"expandedWidth_expandedHeight_showsNavigationRail",
240+
)
241+
}
242+
}

‎app/src/testDemo/screenshots/compactWidth_compactHeight_showsNavigationBar.png‎

57.8 KB
Loading

‎app/src/testDemo/screenshots/compactWidth_expandedHeight_showsNavigationBar.png‎

105 KB
Loading

‎app/src/testDemo/screenshots/compactWidth_mediumHeight_showsNavigationBar.png‎

51 KB
Loading

‎app/src/testDemo/screenshots/expandedWidth_compactHeight_showsNavigationRail.png‎

75.7 KB
Loading

‎app/src/testDemo/screenshots/expandedWidth_expandedHeight_showsNavigationRail.png‎

201 KB
Loading

‎app/src/testDemo/screenshots/expandedWidth_mediumHeight_showsNavigationRail.png‎

112 KB
Loading

0 commit comments

Comments
 (0)