How To Implement Automated UI Testing In Roku

UI testing is a form of integration testing that focuses on interacting with the application in the same way a real user would, through manual inputs. Similarly, assertions are not made against the internal state of the application but against the UI elements that the user can directly see.

For example, a regular unit test would look like if input=X then function(X)=Y. It’s simple and effective, but it doesn’t consider the surrounding state of the application.

A UI test would look like this: From the main screen, pressing back, up, up, OK pauses the current video and takes you to the profile options. Every step of the test will programatically mimic what the user is doing (i.e: send the same signal to the device as the remote would do), and every assertion would be done against the UI elements.

Why is this important?

UI testing is vital for the Roku platform because of the quality it assures before deploying a new release. Releasing new versions can be very time-consuming. So if you make a mistake, you might have to wait a whole month before you can release again. Plus, only getting your release certified can take up to 5 business days.This means you really need a solid process for checking everything works correctly before releasing.

Learning how to do UI testing can also save you time and money. It allows QA Engineers to concentrate on what they do best: figuring out and documenting the best ways to test the system, instead of wasting time running their test cases manually.

The Roku UI Testing Journey

This post is the first of a three-part series about UI testing in Roku.

1. UI Testing Fundamentals (You are here)
2. Next post: Testing deeplinks, best practices and troubleshooting tips
   • Launching deeplinks effectively
   • Tips and tricks for UI testing in Roku
   • The top 3 reasons why your tests are failing (and how to fix them)
3. Case Study: Implementing UI tests in a real world application (Jellyfin for Roku)
   • Integrating RTA into a BrighterScript Project
   • Replacing network responses with custom payloads
   • Running tests in parallel using multiple devices

Environment setup

The best way to start any new Roku project is using create-roku-app. Make sure you have npm installed and run the following command:

npx create-roku-app

For this introduction we’ll use the vanilla configuration for the Roku project: no BrighterScript and no linter.

Confirm that the application was initialized correctly by setting the password field in the bsconfig.json file and launching the first option in the Run & Debug panel from VSCode. You should see a screen that says "Hello from create-roku-app".

Now we will need to install the dependencies for UI testing. We will use mocha as our testing library and chai as the assertion engine. You can use other options like Jest if you wish so.

The most important dependency is roku-test-automation (RTA), which contains the utility classes to send key presses, get field values, observe field changes and much more.

npm install -D mocha chai roku-test-automation

RTA requires a configuration file with at least your device’s credentials. For this example we’ll use the path ui-tests/rta-config.json. We will separate the credentials from the rest of the configuration, so that we can git commit the base configuration and ignore the file with our credentials.

// ui-tests/rta-config.base.json
{
	"ECP": {
		"default": {
			"launchChannelId": "dev"
		}
	},
	"OnDeviceComponent": {
		"logLevel": "info"
	}
}

The file below is the one that RTA will use, and the one we need to git ignore.

// ui-tests/rta-config.json
{
	"$schema": "https://raw.githubusercontent.com/triwav/roku-test-automation/master/client/rta-config.schema.json",
	"extends": "./ui-tests/rta-config.base.json",
	"RokuDevice": {
		"devices": [
			{
				"host": "your-roku-ip",
				"password": "your-roku-password"
			}
		]
	}
}

Additionally, you should add the following configuration to your package.json file:

{
	"type": "module", // will allow us to use `import` statements in our .js files
	"mocha": {
		"timeout": 72000 // default is 2000ms, which is too small for ui tests
	}
}

Our First test

Before we start, we need to inject a special component called RTA_OnDeviceComponent to our application. This component will execute the actions we send from our tests, from inside of the application. Add the following code to the start of src/components/MainScene.brs

sub init()
    #if ENABLE_RTA
        m.odc = CreateObject("roSGNode", "RTA_OnDeviceComponent")
    #end if

	' ...
end sub

The ENABLE_RTA manifest constant will help us inject the component only if we’re running a test. Manifest constants don’t have default values, so we have to set it to false in our src/manifest file. When running a test, our UI testing library will set it to true before deploying.

bs_const=ENABLE_RTA=false

Now let’s write our first UI test. As it was mentioned in the introduction, in most cases we’ll assert what the user can directly see. The create-roku-app command creates a Label in the scene component, so we’ll assert that the label is present and that the text is correct.

Create a new file ui-tests/hello-world.spec.js and add the following code:

import { expect } from "chai";
import { device, odc, utils } from "roku-test-automation";

describe('hello world', () => {
    it('says hello world', async () => {
        // Load configuration
        utils.setupEnvironmentFromConfigFile('./ui-tests/rta-config.json');

        // Deploy application
        await device.deploy({ project: './bsconfig.json' });

        // Wait for application load
        await utils.sleep(3000);

        // Get text from Label with id "welcome"
        let textResponse = await odc.getValue({
            keyPath: '#welcome.text'
        });

        // Text value assertion
        expect(textResponse.value).to.eql("Hello from create-roku-app");

        // End the in-app task that enables the `getValue()` method.
        await odc.shutdown();
    });
});

Loading the configuration is necessary because it includes the device host and password that will be used by the device.deploy() method to launch your application.

// Load configuration
utils.setupEnvironmentFromConfigFile('./ui-tests/rta-config.json');

Remember that, unless we wait for things to happen, the script will continue to the next statement. To avoid a false negative for the label lookup while the app is opening, we can use the utils.sleep() method to wait for 3 seconds before continuing.

// Wait for application load
await utils.sleep(3000);

The method odc.getValue() allows us to define a key path to look for a field value. In this case, #welcome.text indicates to look for “the text field inside the node with id welcome”. By default, odc.getValue() starts traversing the node tree starting from the scene component.

// Get text from Label with id "welcome"
let textResponse = await odc.getValue({
	keyPath: '#welcome.text'
});

Now that we have a reference to the text value, we assert that it’s equal to the expected text using Chai’s expect()

// Text value assertion
expect(textResponse.value).to.eql("Hello from create-roku-app");

Finally, we need to call odc.shutdown() to end the in-app task that responds to getValue() calls (and others) from inside of the application. This step is important because otherwise the test will be left hanging, even if all previous assertions have passed.

// End the in-app task that enables the `getValue()` method.
await odc.shutdown();

To run the test we can use the following command. Make sure to insert the values for host and password in ui-tests/rta-config.json beforehand:

npx mocha ui-tests/hello-world.spec.js

Refining Our First Test

Let’s improve our test to get it closer to a real life case. First of all, waiting three seconds for the application load is an arbitrary action that can cost us precious time when running a bigger test suite. On a real application, to pass Roku’s certification criteria (3.3), you should fire an AppLaunchComplete beacon when your first screen is fully rendered. This usually happens at the end of the MainScene’s init() method. Let’s add that, plus a new field for the scene that will tell us if the app has finished loading.

' src/components/MainScene.brs
sub init()
	' ...

	m.top.signalBeacon("AppLaunchComplete")
	m.top.addField("appLaunchFinished", "boolean", false)
	m.top.appLaunchFinished = true
end sub

Now instead of just sleeping for 3 seconds, we can use the odc.onFieldChangeOnce method to wait for the value of appLaunchFinished to be true:

// ui-tests/hello-world.spec.js
it('says hello world', async () => {
	// ...

	// Wait for application load
	await odc.onFieldChangeOnce({
		keyPath: '.appLaunchFinished',
		match: true
	})
});

As you can see, key paths are very flexible. The lookup always starts from the scene node, so to get one of it’s fields we use the .fieldName key path directly. We can also look for a node’s ID + field as we did before with #welcome.text. For elements with multiple children, we can access them by index using #rowList.1.title.

Finally, for nodes that support it, we can use the following methods (e.g: #label.boundingRect())

• getParent()
• count()
• keys()
• len()
• getChildCount()
• threadinfo()
• getFieldTypes()
• subtype()
• boundingRect()
• localBoundingRect()
• sceneBoundingRect()

Here’s the before and after run times of the test:

BEFORE:
  hello world
    ✔ says hello world (8308ms)

AFTER:
  hello world
    ✔ says hello world (6287ms)

Testing User Inputs

So far we’ve only tested what is already present on the screen, without user input. Let’s add a button that changes the default text when pressed and test the result of pressing it.

' src/components/MainScene.brs
sub init()
	' ...

	button = m.top.createChild("Button")
    button.update({
        id: "button"
        translation: [200, 200],
        text: "Press me"
    })
    button.observeField("buttonSelected", "onButtonSelected")
    button.setFocus(true)

	m.top.signalBeacon("AppLaunchComplete")
	' ...
end sub

sub onButtonSelected()
    label = m.top.findNode("welcome")
    label.text = "Button was pressed!"
end sub

Our test will need to perform the user input (press OK) and check for a change in the label’s text.

// ui-tests/hello-world.spec.js
import { device, odc, utils, ecp } from "roku-test-automation";

describe('hello world', () => {
    it('says hello world', async () => {
        // ...

        // Text value assertion
        expect(textResponse.value).to.eql('Hello from create-roku-app');

        // Check that the button is focused
        expect(await odc.isInFocusChain({ keyPath: '#button' })).to.be.true;

        // Press the button
        await ecp.sendKeypress(ecp.Key.Ok);
        
        // Verify that the label changed
        textResponse = await odc.getValue({ keyPath: '#welcome.text' });
        expect(textResponse.value).to.eql('Button was pressed!');
    });
});

Now we’re testing that the button is focused by default, and that pressing it changes the label in screen. Great! That’s our first semi-realistic test. Let’s take a look at the basics we’ve learned:

1. We can get Node field values using keypaths and the odc.getValue() method.
2. We can observe a field value change with the odc.onFieldChangeOnce() method.
3. Sending key presses is easy using the ecp.sendKeypress() method.

These basics will be useful throughout our whole journey to automate UI testing, so it’s very important that we understand them.

What’s Next?

This post is the first of a three-part series about UI testing in Roku.

1. UI Testing Fundamentals (You are here)
2. Next post: Testing deeplinks, best practices and troubleshooting tips
   • Launching deeplinks effectively
   • Tips and tricks for UI testing in Roku
   • The top 3 reasons why your tests are failing (and how to fix them)
3. Case Study: Implementing UI tests in a real world application (Jellyfin for Roku)
   • Integrating RTA into a BrighterScript Project
   • Replacing network responses with custom payloads
   • Running tests in parallel using multiple devices

Subscribe to get notified when the next post comes out!