0 Comments
 

Zurich Xamarin User Group Logo

I had a great time on Thursday the 15. September 2016 at the Xamarin User Group in Zurich. Though having a little struggle with the latest updates which seemed to have a “minor” impact on my UI Tests to run on Windows i.e. Visual Studio I had a great time presenting and some interesting questions to chew on during answering.

Image of my talk at our great location at the impact hub.

After the Meetup there was the typical Pizza, Sweets, Beer, Water discussions that always spawn some new insights on topics. I want to thank all those who came to the talk and I’m looking forward hopefully seeing you dear reader at one of the upcoming Xamarin Meetups in Zurich.

Image of a Pizza - which made me hungry while just looking at it... well done to past-mark...

Plus I am thrilled to announce that I’m now a Co-Organizer of the Meetup together with Thomas Charrière and Reto Senn. So if you have any topics you would like to present or see presented be sure to reach out to me via  Twitter or just write a comment down bellow. If you simply wish to support our effort by providing some beverages do neither hesitate to contact me or one of my co-organizers Winking smile

Talk Content

You can find my slides here and the code on GitHub.

0 Comments
 

image

In this post we will look at how we can write UI tests for a Xamarin.Android app. The app which we are testing is a basic app based on the MVVM pattern. You can find a detailed blog post on the in and outs of the app under test here.

Assuming you already have a Xamarin Test Cloud (XTC) project added to your solution. As described in this former post. Lets get started by preparing the app for UI testing.

Enabling your Android app for the XTC tests

You do not have to do anything to enable your app to be compatible with XTC. That being said when writing a test with Xamarin Test Cloud. The test should not recognize a control by the content it is displaying. Rather it identifies a control via an ID which is not visible to the end user but rather a invisible ID. Setting this ID is best done via the accessibility label attribute. You can set this in the AXML code of your view i.e. control:

This ID will not have to change if the locale changes. Even if the location of the UI element changes it will still be found by the test. Thus IDs should be generally used to identify controls as they allow for higher resilient test code.

Writing Test(s)

After tuning our the UI code we can start writing the test. One way is to use the XTC recorder. While this tool brings a great benefit when starting off with or smaller apps (as it would be the case in this post). I usually prefer to use the REPL which is easy to use and can be invoked at any point in the test code.

When the UI is prepared accordingly we can start writing the test. One way is to use the XTC recorder. While this tool brings a great benefit when starting of with or smaller apps (as it would be the case in this post), I usually prefer to use the REPL which is easy to use and can be inserted to be started at any point in your code.

The REPL is especially useful once you have a large testing framework. Having a framework i.e. helpers in place you can run those before firing up the REPL. So you can start exploring and interacting with the UI right were you need to.

XtcTreeWithIds_thumb

Starting the REPL is as easy as adding the following line to your test method:

You can see the entire apps visual tree outlined.

Now we can define the steps we want to perform in our test:

Note how the test takes screenshots at every stage. This allows to easily identify the steps on the Xamarin test cloud and give them a label. While this might seem to be a bit of an overkill for such a small sample. When writing larger tests naming your steps i.e. screen shots will help you narrowing down where an error is happening.
One good thing about the tests is that they also run well on your local machine. So using e.g. the al Studio we can execute the tests over the NUnit Runner:

shows resharper testrunner in visual studio after executing a test

You can choose to run the tests on an emulator or device as you like. Just select the desired target as you would when starting the application. Or submit it to the test cloud i.e. integrate it in our build process.

Conclusion

This post described how to get started writing automated UI tests with Xamarin Test Cloud for a Xamarin.Android. Showing you how to adopt your UI for writing resilient test. Based on NUnit the tests are running on a stable and battle proven foundation.

Keep in mind that UI tests should not be the only testing you rely on for testing your app. But in combination with Unit and Integration tests they can provide great value to your project. Running them on Xamarin Test Cloud allows you to run them on a sea of devices. Along with different versions of Android i.e. vendor flavours.

You can find the entire sample on GitHub.

0 Comments
 

Title image showing the xamarin test cloud and ios logo

In this post we will look at how we can test a Xamarin.iOS app which is based on a storyboard. The app which we are testing is a basic app based on the MVVM pattern. You can find a detailed blog post on the in and outs of the app under test here.

Assuming you already have a Xamarin Test Cloud (XTC) project added to your solution as described in this former post. Lets get started by preparing the app to be tested.

Preparing the iOS for the XTC tests

When writing a test with Xamarin Test Cloud the test ideally does not recognize a control by the content it is displaying but rather via an ID which is not visible to the end user but rather a invisible ID. Setting this ID is best done via the accessibility label attribute which can be set in the storyboard under properties:

Shows where under properties the label can be enabled.

This ID will not have to change if the locale changes or the UI element is moved to a different location and therefore will allow you to write more resilient test code.

Writing Test(s)

When the UI is prepared accordingly we can start writing the test. One way is to use the XTC recorder. While this tool brings a great benefit when starting of with or smaller apps (as it would be the case in this post), I usually prefer to use the REPL which is easy to use and can be inserted to be started at any point in your code.

The REPL is especially useful once you have a substantial testing framework written because it allows you to execute it after executing some prearranged steps to get to a certain point which you would like to test.

Shows repl output of the app

Starting the REPL is as easy as adding the following line to your test method:

You can see the entire apps visual tree nicely outlined.

Now we can define the steps we want to perform in our test:

Note how the test takes screenshots at every stage. This allows to easily identify the steps on the Xamarin test cloud and give them a label. While this might be a bit overkill for such a small sample on larger tests when named right the steps and screen shots will greatly help you narrowing down where an error is happening.
One good thing about the tests is that they also run well on your local machine. So using Xamarin Studio on a Mac we can simply execute the tests over the NUnit Runner:

image showing the nunit test runner of xamarin studio.

Or submit it to the test cloud i.e. integrate it in our build process.

Conclusion

This post described how to get started writing automated UI tests with Xamarin Test Cloud for a Xamarin.iOS application. After some minor adjustments to the UI a test can be written using the familiar NUnit testing framework. Keeping in mind that UI tests should not be the only testing you apply to your app since they are rather slow, they do bring a lot of value when used properly since one can easily run the tests on different devices and iOS versions with the Xamarin Test Cloud.

You can find the entire sample on GitHub.

0 Comments
 

title image showing 1s and 0s in a green console style.

Running your tests on Xamarin Test Cloud (XTC) is as easy as right clicking onto tests and select run in XTC. As easy as this approach is – as limited it becomes when more sophisticated demands arise. Those might be running a group of tests, running on preconfigured device groups or integrating into an automated Continuous Integration (CI) and Continuous Deployment (CD) process. But there are also some additional useful features and information which we shall see in the follow up of this blog post.

Thereby the focus will be on the following:

  • Creating and Running the basic Command
  • Setting Categories and using them in the Command
  • Recording your Android XTC test runs
  • Using the XTC API to retrieve even more information

So lets get started Smile 

The basics Command

The easiest way to get a test script is if you login to the Xamarin Test Cloud dashboard website. Click on New Test Run, now you can either select an existing app or just choose to create a new App, select a team, devices that should be included in the test run and some additional information such as series, device locale and parallelisation. In the last step a sample script is generated which is the basic recipe for running the tests from the command line.

Now this script has to be adopted at the various points to reflect the setup of the app under test but from there on simply navigate to the root e.g. the solution folder and execute the script from the command line, power shell or the shell of your choosing. The tests will be uploaded and run on the selected devices. Once the tests are completed the results can be viewed in the XTC dashboard.

Running specific categories of your test suite

XTC allows to set categories when declaring a test as follows:

Categories allow to choose a subset of the tests to be run when executing them on the XTC. This can be a great way to create fast running smoke tests that give instant feedback after a build or running focused tests on a area of the app. The command can be extended by adding the following parameter:

If applicable one choose multiple categories for one run which makes mix and matching an easy undertaking. So a command using e.g. the categories SmokeTest and Purchase would be the following command:

Okay now to becoming more insights into tests, which is especially interesting when they are failing…

Recording your test runs

The basic option to get a better sense of what is going on during a test run is taking screenshots. This can be done by adding the following line to your tests as follows:

Note taking screenshots might increase the duration of a test run, in case of large suites one might want to reduce the overall time by using a constant within the UI test project that enables/disables detailed reporting of the test runs.

Now screenshots are already a great help. But in case of Android you are able to record the test runs. How awesome is that?! Enabling recordings is as easy as adding an additional parameter to the command line:

The recording can be viewed on the test dashboard with the very usable feature of speeding up the playback so you wont be bored by watching how your test fights it’s way to the juicy part. And as of date of writing recordings are only available for Android and can be solely enabled by adding the parameter to the Command Line Interface (CLI). If you try to use the command on an iOS run there will be no exceptions but the recording will just simply be missing.

Also don’t forget to add a good failure message to the Assert. Or you will be left with the reason of a failed test because an expected value was false instead of true (not helpful for me…).

Accessing the XTC API

Invoking tests via the command line is a great way to integrate UI tests running on XTC to an automated DevOps build pipeline. By default adding the following parameter to the command will allow to receive an NUnit Test report, it gives a very high level report of the run i.e. will show which tests have failed but not on which device etc.. If you desire to have a richer interaction with the data you can start interacting with the Xamarin Test Cloud API. There is a ton of information you can retrieve about a test run when using the API. Unfortunately at the current time it is not possible to retrieve images, recordings and some other information. If you are interested in giving the team an initiative to retrieve this information you can vote for this feature here.

All in all if you are using an on premise or other third party dashboard for displaying your test results, make sure you check out the API as it may just provide the more detailed information you are looking for and do not find within the NUnit test-report.

Conclusion

Starting test runs for Xamarin Test Cloud from the command line not only lets you integrate it into a DevOps toolchain but also allows you to run sub-sets of your tests, or record your test runs for Android. Last but not least if you are interested on feeding your test results into a dashboard the Xamarin Test Cloud API might just provide you with the information you are looking for.

The command line option is not only valuable for integration into automated toolchains but also for a more granular control when executing the tests locally. Be it from Windows or OSX the Xamarin Test Cloud toolchain has you covered.

0 Comments
 

Xamarin Test Cloud Logo

Automatically testing apps is not only a huge time saver but it also ensures that bugs introduced into the system get quickly caught. This leads to a shorter timespan to fixing these bugs since the memory of what has changed is still fresh in the head of the developers so overall shorter time means less money spent on hunting down bugs and more time to implement the next juicy feature. When developing a Xamarin application one can choose to run the Unit- and Integration Tests on a device/simulator to get a feeling how the logic will behave and perform in the real world. But with Xamarin Test Cloud we can go one step further and test the UI i.e. the entire stack of the application automatically. If you are interested into a general introduction into the topic I recommend you read this post before diving into the technical details of the lines to follow.

For writing a Xamarin Test Cloud (XTC) test the Xamarin Toolchain has to be installed. This post was written using Visual Studio 2015 Update 2.

Having a look at the app

Since this post focuses on writing UI tests the app is rather simple but will show the basic steps required to write maintainable UI tests. The app under test is a basic app with an input field, a button and a button that displays the entry of the input field. The app is based on the MVVM pattern (using MVVM Light) and is based on Xamarin.Forms.

Screenshot of app that is going to be tested.

When a user enters a message and submits it, it will appear on the label bellow. Now that the basic outline is defined, lets have a look at how we can implement a UI test for the app.

Writing UI Tests

If you created your Xamarin.Android and iOS projects with UI tests from the beginning you can skip the next steps and just start writing the tests. If not do not despair a few easy steps will allow you start writing those UI tests in no-time.

Setup

To start writing UI tests we first have to add a UI testing project to an existing project. Simply selecting the UI Test Project will create the Project in the solution. What is still left to do is associate the Project we want to test with the test project. Right click onto References, select the Android and/or iOS Project you want to test and simply add them:

Adding iOS And Android App Projects to a UITestProject

If you take the default UI testing project from Visual Studio you might want to check if you have the latest version for Xamarin Test Cloud. Update XTC to 1.1.0 or higher or else you will get an error when executing the tests which is asking for an API key.

Starting with Xamarin Test Cloud 1.1.0 or higher you no longer need a subscription for Xamarin Test Cloud to execute the tests locally.

iOS and XTC

To run UI tests on iOS there is an additional library required that is included in the app - only the debug version, this will not be deployed to the store. Add the Xamarin Test Cloud Agent package to your iOS Project, then edit the App Delegate to look the following:

Next open the Properties of your iOS project and ensure that Debug build configuration under the Build tab defines the Symbol: ENABLE_TEST_CLOUD

iOSProjectConfiguration

Now this symbol must only be set for Debug builds. Trying to submit an app with the calabash package included will lead to a rejection when trying to submit the app to the store.

Writing tests

Enough chit chat, lets start writing those tests already. In the UI test project the Tests.cs class file contains the basic setup for a test. Lets change the AppLaunches method to this:

Now when we start the test for i.e. Android the App will be started and a Read Eval Print-Loop (Repl) console will open. One of the most used commands when using the Repl is treenow if we execute the command the following output will be printed:

Shows the XTC Repl tree command when a UI has no Ids set.

Now we can identify some of the controls, but it would be a lot better if we could just identify the elements over Ids. So lets just do that. Lets start by adding an AutomationId to all of our controls:

Next we will have to add a couple of lines of code to the AppDelegate.cs in the iOS Project:

And the MainActivity.cs of the Android project:

If we rerun the test again and execute the command tree we will now see the following output:

XTC Repl showing Ids after executing the tree command

Working against Ids is considered best practice. For instance if the app is required to be available in multiple languages the UI test will work when the device is set to a different locale and can be used to acquire screenshots

In the Repl, we can now type in the following commands which will write a text in the Entry control, then submit the message and read out the text value of the label displaying the last submitted message:

Repl with the entered commands.

One can use the copy Repl command to copy all of the executed commands to the clipboard (the tree command will be discarded from copying). We can create a new test method and simply paste the copied commands from the Repl. Add an Assert for the Text input et voila, we have created our first basic UI test. Smile

Now the test has been added with some additional lines that capture a screenshot of the app. These screenshots can be of great value when not observing the test while it is running. And since automation is all about freeing up the human to do other tasks you should enable your tests to take a screen shot after any meaning full step.

Please note that taking screenshots does increase the runtime of your tests slightly so you might want to enable a global constant which allows you to easily en- i.e. disable the screenshots.

Further when running your tests on the Xamarin Test Cloud the screen shot will allow you to get a lot more information about each step such as CPU & RAM usage at the time of the screenshot. So it also allows you to get more telemetry of the app when the screenshot is taken. The string you pass to the method will also appear as under the test.

Executing Tests

After creating the test we can run it on either Android or iOS. Per default the IDE settings are taken for executing the test so it is possible to select different emulators e.g. in Visual Studio you can select different Android Emulators to execute the tests on. If one hooks a device to the computer it is even possible to execute the test on the device. Again for all this there are no licenses required. The tests are base on NUnit, so as long as a NUnit Testrunner is installed the tests should just simply execute.

Showing test run result with the app in the background. The resharper NUnit Testrunner was used to perform the testrun.

At the time of writing this blog post the iOS execution is only possible on a Mac.

A small side note regarding the execution time of the test. Due to the fact that the app has to be deployed and then executed on the device the test takes a couple of seconds to run through. If we would think about an Integration Test performing the same task we would have a much shorter runtime, but would not be able to see if the information is correctly displayed on the screen for the user. Just keep in mind that UI Tests in general take a while longer to perform the same test that a integration test would need and that it is worth to choose beforehand which tests to execute.

Running in the cloud

As stated before it is possible to execute the tests locally. This could even be integrated into an automated build process. But there are some limitations when executing the tests locally. For instance if we decide to attach a device we have to maintain that device. Ever heard of bloated batteries, broken cables and the like? Well those things can/will happen, further it is not possible to parallelise the tests locally out of the box. Another aspect is easily adding devices to the test set, well all this is what is the great strength of running your tests on Xamarin Test Cloud. Running the tests on the Xamarin Test Cloud can easily be integrated into a build server and allows to easily run the app on multiple devices (in parallel) and even allows to parallelize the execution of tests on multiple devices of the same type. So in short Xamarin Test Cloud offers a wide collection of devices with different OS versions that do not have to maintained and choosing which devices to use during the tests can be easily configured and changed. The results are displayed on a dashboard:

Screenshot showing the dashboard of XTC results for a testrun.

There is even an API that allows to control the execution and collect the results of a test run.

Currently there are still some details missing from being available for the extraction. Such as the snapshots taking during a test run. If you would like the team to add this or other features please feel free to rate it up or tell them here.

Conclusion

In this blog post we saw how a UI Test based on Xamarin Test Cloud is created. How the UI for Xamarin.Forms should be adapted to make writing of UI tests more robust. Further we saw how we can configure the tests to capture screenshots from steps performed during a test run. Capturing screenshots of a test run not only allows to see more easily where a functional error occurred but also allows to see how the screen looks for a given device and if the layout is behaving as expected.

You can find the sample project and the test code on GitHub.

HTH

Further Reading

Getting started with iOS.