0 Comments

Xamarin Test Cloud Logo

Automatically testing apps is not only a huge time saver but it also ensures that bugs introduced into the system get quickly caught. This leads to a shorter timespan to fixing these bugs since the memory of what has changed is still fresh in the head of the developers so overall shorter time means less money spent on hunting down bugs and more time to implement the next juicy feature. When developing a Xamarin application one can choose to run the Unit- and Integration Tests on a device/simulator to get a feeling how the logic will behave and perform in the real world. But with Xamarin Test Cloud we can go one step further and test the UI i.e. the entire stack of the application automatically. If you are interested into a general introduction into the topic I recommend you read this post before diving into the technical details of the lines to follow.

For writing a Xamarin Test Cloud (XTC) test the Xamarin Toolchain has to be installed. This post was written using Visual Studio 2015 Update 2.

Having a look at the app

Since this post focuses on writing UI tests the app is rather simple but will show the basic steps required to write maintainable UI tests. The app under test is a basic app with an input field, a button and a button that displays the entry of the input field. The app is based on the MVVM pattern (using MVVM Light) and is based on Xamarin.Forms.

Screenshot of app that is going to be tested.

When a user enters a message and submits it, it will appear on the label bellow. Now that the basic outline is defined, lets have a look at how we can implement a UI test for the app.

Writing UI Tests

If you created your Xamarin.Android and iOS projects with UI tests from the beginning you can skip the next steps and just start writing the tests. If not do not despair a few easy steps will allow you start writing those UI tests in no-time.

Setup

To start writing UI tests we first have to add a UI testing project to an existing project. Simply selecting the UI Test Project will create the Project in the solution. What is still left to do is associate the Project we want to test with the test project. Right click onto References, select the Android and/or iOS Project you want to test and simply add them:

Adding iOS And Android App Projects to a UITestProject

If you take the default UI testing project from Visual Studio you might want to check if you have the latest version for Xamarin Test Cloud. Update XTC to 1.1.0 or higher or else you will get an error when executing the tests which is asking for an API key.

Starting with Xamarin Test Cloud 1.1.0 or higher you no longer need a subscription for Xamarin Test Cloud to execute the tests locally.

iOS and XTC

To run UI tests on iOS there is an additional library required that is included in the app - only the debug version, this will not be deployed to the store. Add the Xamarin Test Cloud Agent package to your iOS Project, then edit the App Delegate to look the following:

Next open the Properties of your iOS project and ensure that Debug build configuration under the Build tab defines the Symbol: ENABLE_TEST_CLOUD

iOSProjectConfiguration

Now this symbol must only be set for Debug builds. Trying to submit an app with the calabash package included will lead to a rejection when trying to submit the app to the store.

Writing tests

Enough chit chat, lets start writing those tests already. In the UI test project the Tests.cs class file contains the basic setup for a test. Lets change the AppLaunches method to this:

Now when we start the test for i.e. Android the App will be started and a Read Eval Print-Loop (Repl) console will open. One of the most used commands when using the Repl is treenow if we execute the command the following output will be printed:

Shows the XTC Repl tree command when a UI has no Ids set.

Now we can identify some of the controls, but it would be a lot better if we could just identify the elements over Ids. So lets just do that. Lets start by adding an AutomationId to all of our controls:

Next we will have to add a couple of lines of code to the AppDelegate.cs in the iOS Project:

And the MainActivity.cs of the Android project:

If we rerun the test again and execute the command tree we will now see the following output:

XTC Repl showing Ids after executing the tree command

Working against Ids is considered best practice. For instance if the app is required to be available in multiple languages the UI test will work when the device is set to a different locale and can be used to acquire screenshots

In the Repl, we can now type in the following commands which will write a text in the Entry control, then submit the message and read out the text value of the label displaying the last submitted message:

Repl with the entered commands.

One can use the copy Repl command to copy all of the executed commands to the clipboard (the tree command will be discarded from copying). We can create a new test method and simply paste the copied commands from the Repl. Add an Assert for the Text input et voila, we have created our first basic UI test. Smile

Now the test has been added with some additional lines that capture a screenshot of the app. These screenshots can be of great value when not observing the test while it is running. And since automation is all about freeing up the human to do other tasks you should enable your tests to take a screen shot after any meaning full step.

Please note that taking screenshots does increase the runtime of your tests slightly so you might want to enable a global constant which allows you to easily en- i.e. disable the screenshots.

Further when running your tests on the Xamarin Test Cloud the screen shot will allow you to get a lot more information about each step such as CPU & RAM usage at the time of the screenshot. So it also allows you to get more telemetry of the app when the screenshot is taken. The string you pass to the method will also appear as under the test.

Executing Tests

After creating the test we can run it on either Android or iOS. Per default the IDE settings are taken for executing the test so it is possible to select different emulators e.g. in Visual Studio you can select different Android Emulators to execute the tests on. If one hooks a device to the computer it is even possible to execute the test on the device. Again for all this there are no licenses required. The tests are base on NUnit, so as long as a NUnit Testrunner is installed the tests should just simply execute.

Showing test run result with the app in the background. The resharper NUnit Testrunner was used to perform the testrun.

At the time of writing this blog post the iOS execution is only possible on a Mac.

A small side note regarding the execution time of the test. Due to the fact that the app has to be deployed and then executed on the device the test takes a couple of seconds to run through. If we would think about an Integration Test performing the same task we would have a much shorter runtime, but would not be able to see if the information is correctly displayed on the screen for the user. Just keep in mind that UI Tests in general take a while longer to perform the same test that a integration test would need and that it is worth to choose beforehand which tests to execute.

Running in the cloud

As stated before it is possible to execute the tests locally. This could even be integrated into an automated build process. But there are some limitations when executing the tests locally. For instance if we decide to attach a device we have to maintain that device. Ever heard of bloated batteries, broken cables and the like? Well those things can/will happen, further it is not possible to parallelise the tests locally out of the box. Another aspect is easily adding devices to the test set, well all this is what is the great strength of running your tests on Xamarin Test Cloud. Running the tests on the Xamarin Test Cloud can easily be integrated into a build server and allows to easily run the app on multiple devices (in parallel) and even allows to parallelize the execution of tests on multiple devices of the same type. So in short Xamarin Test Cloud offers a wide collection of devices with different OS versions that do not have to maintained and choosing which devices to use during the tests can be easily configured and changed. The results are displayed on a dashboard:

Screenshot showing the dashboard of XTC results for a testrun.

There is even an API that allows to control the execution and collect the results of a test run.

Currently there are still some details missing from being available for the extraction. Such as the snapshots taking during a test run. If you would like the team to add this or other features please feel free to rate it up or tell them here.

Conclusion

In this blog post we saw how a UI Test based on Xamarin Test Cloud is created. How the UI for Xamarin.Forms should be adapted to make writing of UI tests more robust. Further we saw how we can configure the tests to capture screenshots from steps performed during a test run. Capturing screenshots of a test run not only allows to see more easily where a functional error occurred but also allows to see how the screen looks for a given device and if the layout is behaving as expected.

You can find the sample project and the test code on GitHub.

HTH

Further Reading

Getting started with iOS.

0 Comments

testcloud_devices

When developing an app e.g. a mobile app for iOS, Android and Windows 10 or a web app what initially stands in the spotlight is a core idea that brings a core value to it’s users. As soon as the scope is better known one ideally starts developing a Minimal Viable Product which demonstrates the core ideas and let’s the creators get first feedback from potential users. With the initial ideas and the feedback the team starts to add more and more functionality to the app and all is well. Until users start reporting some odd behaviour and the well intended changes start having side effects on other features. So every release gets thoroughly tested before it is sent out. The time it takes to check on all features starts to increase, but holding off of going through all the tests may lead to a new bug that snuck into the app. So what to do?

Bring The Fairy Dust - Automated UI tests

This is often the point where one person e.g. a non-technical manager hears about a UI testing framework – which solves exactly the problem of automating manual tests. Plus one great thing about a UI testing framework usually is that there are no changes required in the app. The coverage of a test is very high since one works against the entire stack.

Automated UI tests simulate user interaction on the UI layer of an app. Depending on what kind of app there are different frameworks that one can choose from. For mobile apps on iOS and Android the solution we will be looking at is Xamarin Test Cloud (XTC) which is based on the Calabash testing framework. XTC allows to create UI test that can be used across platforms. They can be run locally i.e. read “phones attached to the Computer of the tester”. Far more interesting though is the opportunity of running the same tests in Xamarins Test Cloud which offers hundreds of devices (with different OS versions) on which the app and tests can be run on.

But as so many things in life - UI tests come at a cost. Compared to unit- or integration tests they are rather slow and like integration tests they tend to be more on the brittle side i.e. a test fails due to changes to the app. This is especially true if the UI is still under a lot of change. That being said they can be a great tool to bring down manual testing time. But solely relying on UI tests for your mobile tests will most probably not result in the result you were hoping for. So what are the sweet spots where automated UI tests will bring you the largest benefit?

Well lets start by looking at what kind of tests you can actually implement when considering mobile applications:

TestDistribution

As the graphic indicates the wider the pyramid the more tests you will probably want to have in your project. So lets dive into what the main benefits are for the different areas and let’s also look at how they could be implemented.

Unit Tests

Testing your basic logic of a class is where Unit Tests come into play. A Unit Test should not rely on any dependencies a class may have and therefore requires Stubs and Mocks to evaluate the correct working of business logic. Unit Tests should always pass without an error since they check the basic logical implementations of an app.

UI Tests should always pass i.e. be green since they only test logic without any dependencies.

Further Unit Tests are fast. In fact a Unit Tests is considered slow if it runs longer than100 milliseconds. For larger projects it is not uncommon to have a large number of these tests checking that all the small building blocks are working according to specification/user story. The additional work required for stubbing out dependencies may result in tedious work for certain parts of an app that are mainly coordinating workflows. Further since they only test logic in isolation the Unit Tests do not give any feedback how the components work together. But when all Unit Tests are green the confidence in the logic of an is very high and allows to shift the focus on search for an issue to the interaction of the building blocks which is where Integration Tests come into play.

Characteristics of a Unit Test

What is testedLogic Blocks, single Classes, Parts of Classes
Execution Time / Test< 100 ms
ReliabilityVery High
Impact on ArchitectureHigh

Integration Tests

When multiple parts e.g. classes of an app want to be tested an integration test is usually the way to go. Integration tests can range from testing a couple of classes that work closely together to system tests that require multiple parts of a system up and running. For external services that are not part of the developed app stubs and mocks may be used to ensure that tests do not fail due to services failing outside of the teams scope. Integration tests usually take more time to run since they run the real life scenarios. Due to the number of different parts and sub-systems an integration test may be running through the tests tend to be more brittle than a unit test. Further it can get harder to pin point errors to a specific area of a project, so unit tests are a perfect completion to integration tests as they allow to test complex logical parts in separation. If we look at a standard app built according to the MVVM pattern, the Integration tests are usually run from the View Model or lower in the stack. Integration Tests do not only allow to test for functional correctness of an app but also allow to get a feel for how performant an app is. If you are interested to run your tests on an actual device you can read more about this topic here.

One issue with integration tests is that the View is not being tested. Even though may argument that when the MVVM pattern is applied correctly there is only minimal logic remaining in the View, the user will still use the app via the view. This leaves the possibility that the app will still have issues when operating it through the view. Here (automated) UI tests come into play.

Generally speaking if a project has no automated tests at all, integration tests tend to give a bigger bang for the buck. Since the architectural requirements are not equally thorough.

Characteristics of an Integration Test

What is testedInteraction between componentens, Integration with other System parts, Network communications, Long running tests, memory consumption
Execution Time / TestSeconds to hours
ReliabilityMedium - High
Impact on ArchitectureMedium

UI Tests – Xamarin Test Cloud

Now it might be tempting to avoid testing your app altogether when having a sophisticated test suite of Unit- and Integration Tests, but in the end the user will interact via the UI with the app. And generally speaking they do not care if the logic beneath is sound, when the app crashes on their Device due to some incompatibility with a UI element that is used in the app. So the minimum UI testing every app project should be performing is the manual test. A person taps through the app and ensures that the UI responds as expected and follows the intended workflow. Now as stated before a manual test takes a lot of time and the resource performing the test I.e. a human might feel a bit underwhelmed. So generally speaking where can we utilise automated UI tests to improve test coverage:

  • Smoke Testing
  • Workflows
  • Testing interaction
  • Presence of Elements
  • Localization
  • Multi device tests

It might be tempting to think one can eliminate manual testing altogether, but there are some areas that UI tests just won’t be able to cover automatically and will always need a human:

  • Layout (Elements not rendering correctly)
  • System Interactions
    • Camera
    • Photo Album
    • Sharing
  • UI “Snappiness”
  • Exploratory Testing
  • Offline Testing
  • Bluetooth

Even though the Layout can be covered by an automated test insofar, that it takes pictures which can be verified by a user/MD5 Hash compared. A computer (at least at the time of writing) does not have the capability to analyse the layout of an app for its correctness. So a human eye and brain is still needed for some tests, this is something to keep in mind when automating all the things. Other limitations come either from the setup e.g. there must be a WiFi connection to the phone to run the automated UI tests. In general testing interconnectivity to other devices i.e. proprietary hardware or plainly some other phone that has to be paired, UI tests often become quite cumbersome to setup and the XTC does generally not support these kind of scenarios. So yes there are limitations to UI tests but using them properly will not only greatly enhance the coverage of automated tests but also free up valuable tester time that can then be used to put the human brain to more advanced testing tasks then numbly tapping through an app.

Characteristics of a UI Test

What is testedTests entire app software stack, Smoke Testing, Workflow Testing
Execution Time / TestMinutes
ReliabilityMedium – Low
Impact on ArchitectureLow

Conclusion

There are multiple different ways how one can approach automated testing of a mobile app. On one end there are tests verifying the logic of the single building blocks an application is built upon. Integration tests allow to verify the correct interworking of parts of the app, this can extend over the border of a an app and also include calls to a backend for verifying correct integration. The highest level tests available to mobile applicaitons are automated UI tests. The Xamarin Test Cloud allows create UI tests that can be executed on many hundred of devices. Testing apps on so many devices before the app ships can lead to valuable insights and automating these tests allows to easily perform regression tests not only on logic and integration layers but also for the UI. Think about an upcoming OS release and you want to check if there any issues with your app, given automated UI tests you can easily run a test suite that will quickly inform you if there is any concern with the upcoming update.