Skip to content

Android Test Reports

Setup

Start by including the library:

androidTestImplementation("com.rubensousa.carioca:report-android:1.0.0-beta01")

Then create your own report rule that extends InstrumentedReportRule:

class TestReportRule : InstrumentedReportRule()

Note

The goal of extending InstrumentedReportRule is to apply consistent report rules throughout your test suite. It contains settings regarding screen recording and screenshot that you can customize.

Now apply the rule in your tests:

class SampleTest {

    @get:Rule
    val report = TestReportRule()

}

Note

Use different orders for your rules in case you have multiple of them and assign the lowest value to TestReportRule. This ensures that it starts before all other rules you have in your test suite. Example: @get:Rule(order = 0) for the report rule and order = 1 for the next rule

This basic setup will achieve this out of the box:

  1. Automatic screen recordings for every test
  2. Automatic screenshot when the test fails
  3. Automatic dumps of the view hierarchy if the test fails

Visualize reports

The test reports are generated automatically after running any task like connectedDebugAndroidTest and can be found in build/outputs/connected_android_test_additional_output/**/carioca-report.

By default, those reports are in json format and are not really easily readable. To visualize them properly, this library ships with an Allure plugin that can be used to generate test reports based on the metadata collected through each test execution. Check it out in this page.

Test structure

Test body

You can decorate your tests with individual reports for every execution step:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
@Test
fun testHomeIsDisplayedAfterQuickSettings() = report {

    step("Open quick settings") {
        device.openQuickSettings()
        screenshot("Quick settings displayed")
    }

    step("Press home") {
        device.pressHome()
    }

    step("Home is displayed") {
        screenshot("Launcher displayed")
        assertLauncherIsDisplayed()
    }

}

Optionally, Given, When, Then statements from BDD are also available to describe your tests:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
@Test
fun testHomeIsDisplayedAfterQuickSettings() = report {

    Given("User opens quick settings") {
        device.openQuickSettings()
        screenshot("Quick settings displayed")
    }

    When("User presses home") {
        device.pressHome()
    }

    Then("Home is displayed") {
        assertLauncherIsDisplayed()
    }

}

Before and after

If you have re-usable logic in @Before or @After that you want to include in your reports, just use the following APIs:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
@Before
fun before() = report.before {
    step("Press home") {
        device.pressHome()
    }
    step("Set device orientation to natural") {
        device.setOrientationNatural()
    }
}

@After
fun after() = report.after {
    step("Unfreeze orientation") {
        device.unfreezeRotation()
    }
}

Scenario

The library includes an InstrumentedScenario which allows you to re-use a set of steps across multiple tests:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
class ClickNotification : InstrumentedScenario("Click Notification") {

    private val device = UiDevice.getInstance(
        InstrumentationRegistry.getInstrumentation()
    )

    override fun run(scope: InstrumentedStageScope) = with(scope) {
        screenshot("Before opening notifications")

        step("User opens notifications") {
            device.openNotification()
        }

        step("Wait for animation") {
            Thread.sleep(1000L)
        }

        screenshot("After opening notifications")

        step("Click notification") {
            device.click()
        }

        screenshot("After clicking notification")

    }

}

Then, in your tests, can use it like so:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
@Test
fun testAppOpensHomeAfterClickingNotification() = report {

    step("Trigger notification") {
        sendNotificationIntent()
    }

    // Or When(ClickNotification())
    scenario(ClickNotification())

    step("Home screen is visible") {
        assertHomeScreenDisplayed()
    }

}

Extra metadata

Using @TestReport allows you to describe your tests in more detail:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
@TestReport(
    id = "TicketID",
    title = "App opens home after notification click",
    description = "Our app needs to show the home screen " +
            "whenever our notifications are clicked",
    links = [
        "https://board.issue.tracker/id",
        "https://developer.android.com/training/testing/other-components/ui-automator"
    ]
)
@Test
fun testAppOpensHomeAfterClickingNotification() = report {
        // Test body
    }

Recording options

To override the recording options for individual tests, use @TestRecording:

1
2
3
4
5
6
7
// Disables screen recording for this test only
@TestRecording(
    enabled = false
)
@Test
fun fastTestThatShouldFinishInLessThan1Second() {
}

Or also:

1
2
3
4
5
6
7
8
9
// Records this test in landscape mode 
// and keeps the recording file even if the test passes
@TestRecording(
    keepOnSuccess = true,
    orientation = RecordingOrientation.LANDSCAPE
)
@Test
fun testInLandscape() {
}

This configuration will replace the RecordingOptions from InstrumentedReportRule

Screenshot options

To override the screenshot options for individual tests, use @TestScreenshot:

1
2
3
4
5
6
7
@TestScreenshot(
    scale = 1.0f,
    format = Bitmap.CompressFormat.PNG,
)
@Test
fun testSomething() {
}

This configuration will replace the ScreenshotOptions from InstrumentedReportRule

Compose

com.rubensousa.carioca:report-android-compose contains a DumpComposeHierarchyInterceptor that can be used to inspect the compose hierarchy when tests fail.

To use it, create your rule and pass it in the list of interceptors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
class SampleScreenTest {

    @get:Rule
    val report = InstrumentedReportRule(
        interceptors = listOf(
            TakeScreenshotOnFailureInterceptor(),
            DumpComposeHierarchyInterceptor(),
            DumpViewHierarchyInterceptor(),
        )
    )
}

The output of the hierarchy looks similar to this:

Node #1 at (left=0.0, top=0.0, right=1920.0, bottom=1080.0)px
 |-Node #2 at (left=0.0, top=0.0, right=1920.0, bottom=1080.0)px
   IsTraversalGroup = 'true'
    |-Node #14 at (left=835.0, top=500.0, right=1085.0, bottom=580.0)px
    | Focused = 'false'
    | Role = 'Button'
    | Actions = [OnClick, RequestFocus]
    | MergeDescendants = 'true'
    |  |-Node #16 at (left=883.0, top=520.0, right=1037.0, bottom=560.0)px
    |    Text = '[Main button]'
    |    Actions = [SetTextSubstitution, ShowTextSubstitution, ClearTextSubstitution, GetTextLayoutResult]
    |-Node #7 at (left=1728.0, top=936.0, right=1888.0, bottom=1048.0)px
      Focused = 'false'
      Role = 'Button'
      Actions = [OnClick, RequestFocus]
      MergeDescendants = 'true'
       |-Node #10 at (left=1782.0, top=972.0, right=1834.0, bottom=1012.0)px
         Text = '[FAB]'
         Actions = [SetTextSubstitution, ShowTextSubstitution, ClearTextSubstitution, GetTextLayoutResult]

Tracking ignored tests

To include ignored tests in the reports, you need to attach the library's instrumentation listener:

android {
    defaultConfig {
        testInstrumentationRunnerArguments listener: 'com.rubensousa.carioca.report.android.CariocaInstrumentedListener'
    }
}

Or in kotlin DSL:

android {
    defaultConfig {
        testInstrumentationRunnerArguments["listener"] =
                "com.rubensousa.carioca.report.android.CariocaInstrumentedListener"
    }
}

Note

If you use other listeners, just separate them with a comma in the same string