Screenshot tests for a Design system with Roborazzi

by Martinblancopablo and Jeprubio

In Telefónica we must deal with distinct brands and maintain consistent styles across different apps in different countries. For this reason, a Design system called Mística is used. Mística provides User Interface elements with the look & feel expected for each brand.

To check the correct appearance of these components, we use screenshots tests which check their final appearance when rendered in a phone. To avoid the flakiness and the time/resources consumption of emulators we made use of Roborazzi library which gives us the advantage of making screenshot tests without the need of an emulator. We discovered this library because it is used by Now in Android app and the Automotive Design for Compose app, both by Google.

The main advantage of Roborazzi over more commonly used Paparazzi is that Paparazzi doesn’t allow interaction with the tested app (like a button click) and is not compatible with Robolectric which we use in our projects. Roborazzi allows us to capture screenshots with Roboelectric (by using Roboelectric native graphics).

Note that we use these tests in a complementary way to the End-to-end tests that do use emulators. The screenshot tests focus exclusively on the appearance while the End-to-end tests check the operation of the app flows.

Our approach

Instead of comparing full screens screenshots which can lead to storage problems because of its size (screenshots change overtime but they are kept in the Git history) we make screenshots of single UI components such as buttons, cards, tags, …. We want to verify the UI of the component of each brand with its different variations (styles), dark mode etc. To be able to handle that many combinations we use parameterized tests which allows us to define a set of inputs to be applied to each screenshot test.

@JvmStatic
@ParameterizedRobolectricTestRunner.Parameters(name = " Button {1} brand {0} darkTheme={2} ")
fun brands(): List<Array<Any>> {
    return brands.flatMap { brand ->
        buttonStyles.flatMap { buttonStyle ->
            themes.map { darkTheme ->
                arrayOf(brand, buttonStyle, darkTheme)
            }
        }
    }
}

Since we have implementations with Jetpack Compose and xml layouts, we need to provide screenshot tests for both. In the case of Compose the component is just created and added to the screenshot.

composeTestRule.setContent {
    MisticaTheme(brand = brand, darkTheme = darkTheme) {
        Surface(
            color = if (style.isInverse()) {
                colors.backgroundBrand
            } else {
                colors.background
            }
        ) {
            Button(
                text = "textValue",
                buttonStyle = style,
                icon = android.R.drawable.ic_lock_power_off.takeIf { icon },
                onClickListener = { },
                modifier = Modifier.padding(16.dp)
            )
        }
    }
}
...
compareScreenshot( 
    composeTestRule.onRoot(), 
    component = "Button", 
    style = style.toString(), 
    brand = brand, 
    darkTheme = darkTheme, 
    extra = iconSuffix 
)

In the case of xml views implementation, we need a layout which includes the element to test in isolation. To achieve that we added a dummy activity with a layout and add the component programmatically at runtime:

rule.scenario.onActivity { activity ->
    val wrapper: FrameLayout = activity.findViewById(R.id.dummy_activity_wrapper)
    val textInput = TextInput(activity)
    wrapper.addView(textInput)
    compareScreenshot(
        Espresso.onView(ViewMatchers.withId(R.id.dummy_activity_wrapper)),
        component = "XMLButton",
        style = style.toString(),
        brand = brand,
        darkTheme = darkTheme,
        extra = iconSuffix
    )
}

Finally, we created a function to provide a meaningful name to the screenshots using the name of the component, style, brand and dark theme since by default Roborazzi assigns the name of the function to the screenshot file.

fun compareScreenshot(...) { 
    node.captureRoboImage(screenshotName(component, style, brand, darkTheme, extra)) 
}

After executing the tests, we have a set of screenshots files with meaningful names:

XMLButton_DANGER_MovistarBrand_dark.png 
XMLButton_DANGER_MovistarBrand_icon_dark.png 
XMLButton_DANGER_MovistarBrand_icon.png 
XMLButton_DANGER_MovistarBrand.png 
Button_DANGER_MovistarBrand_dark.png 
Button_DANGER_MovistarBrand_icon_dark.png 
Button_DANGER_MovistarBrand_icon.png 
Button_DANGER_MovistarBrand.png 
....

Way of Working

Update Baseline

To ensure consistency when generating the screenshot baseline (those screenshots with expected appearance with which we compare the current UI) we use Github Actions. This way we can execute the update of the baseline always under the same conditions. This action has two main steps: first it executes the Gradle task which captures screenshots replacing the current baseline and then it commits them in the current branch.

steps:

...
  - name: Run Roborazzi Record 
    run: 'bash ./gradlew clean recordRoborazziDebug' 
  
  - name: Commit and push screenshots baseline 
    id: commitAndPushScreenshotsBaseline 
    uses: EndBug/add-and-commit@v7 
    with: 
      message: 'Updated screenshots baseline' 
      add: './**/screenshots/*'

Compare Screenshots

The screenshot comparison is done also with a Github Action for the same reason. It is triggered with each PR and with each commit of it.

on: 
  pull_request: 

jobs: 
  CompareScreenshots: 
  
    runs-on: ubuntu-latest 
  
    steps: 
      ... 

      - name: Verify Screenshots (roborazzi) 
        run: 'bash ./gradlew verifyRoborazziDebug'

In case one or more comparison fails Roborazzi creates some _compare.png images with the previous screenshot, the current one and the changed pixels in between. We use these files to create our own report which shows failed screenshots tests in a usual way showing the file name and the comparison files of what has failed.

In order to do that a bash script is executed on ci getting those files and creating the corresponding HTML after a verification task has failed.

Current version of Roborazzi also offers a similar report but the report that it creates is a bit different.

The report it generates has the html file which contains all the screenshots of all the tests plus the screenshots of the failed tests all in different directories. This would mean that to save that report it would take much more space on our cloud servers as it would contain all those screenshots that are still the same and that are not necessary for us to find out what has changed.

With a few adaptations Roborazzi allows us to have a set of screenshots tests of many UI components and its variations efficiently