0 Comments
 

Processed with VSCO with j2 preset

In the previous post we saw how methods i.e. functions can be implemented with PowerShell. Functions are not only a great way to structure code for reuse but also allow to create larger scripts. Whenever writing code that ends up in production it is always a must to ensure that the code runs as expected. When the Script is small or does not invoke any long running services we can do this quite simply by executing the script. However the more complex a script becomes or is integrated into long running processes the harder it becomes to ensure that the script is still running as intended after changes.

This is where automated Testing allows to ensure functional correctness without demanding any great user interaction to execute multiple test scenarios.

Writing automated Tests

PowerShell comes with it’s own testing framework called Pester. Which is the de facto standard PowerShell testing framework and comes with every installation of Windows 10. Pester follows a Behavior Driven Development (BDD) test-development style. I’ll leave it up to you to follow up on the difference between Test Driven Development (TDD) and BDD, for a framework it usually shows in a more human friendly description of the tests.
Let’s get started by having a look at what we want to do in a C# context.

For the C# samples we will be using the xUnit.net framework which follows the xUnit pattern. Let’s assume we have the following class we would like to test:

We could verify the functionality with the following test:

For PowerShell the same function could look like this:

And the corresponding test function looks like this:

From the structure there is a lot in common.

Note that Pester strongly nudges the developer to go along with a certain naming pattern for files. When a function is in a file called Something.ps1, the corresponding tests should be in the file Something.Tests.ps1.

When we execute the PowerShell test from the PowerShell IDE console pane we get the following result in the PowerShell Window.

Showing console output of the testrunner (executed in VS Code)

There are multiple assertions you can use within pester as one can see in the following overview:

  • Should Be
  • Should BeExactly
  • Should BeNullOrEmpty
  • Should Match
  • Should MatchExactly
  • Should Exist
  • Should Contain
  • Should ContainExactly
  • Should Throw


Mocking external dependencies

When writing automated tests one topic quickly arises and that is how to manage dependencies to the outside world. When writing automated tests we can separate them into different categories:

CategoryDescription
Unit TestsTesting the core logic of code. Tests that logic has been implemented according to specs and ensures that the core building blocks are working correctly. No dependencies to other code e.g. Modules in PowerShell. Should make up the majority of automated tests in a project.
Integration TestsTest a part of an application i.e. Script. Ensuring that the tested parts work together which may involve calling services outside of the script itself.
These tests are usually the easiest to implement but are more fragile than unit tests. An error might be caused by an other system or a logic error in the script. In large projects you should have more Unit Tests than Integration Tests.
System TestsTest entire use case scenarios end to end. These tests should run scenarios of the script e.g. “Setup new Web Server Instance”, “Setup new DB instance”. They usually take a long time to run and may require upfront work regarding infrastructure i.e. a staging/testing environment.
In general you should only test your core scenarios with System Tests. They usually take a lot of upfront investment to create the environments and also require a lot of maintenance since they have to be adapted once a change to the system is made.

Since PowerShell was mainly created for automating infrastructure tasks, most of the code you would write is interacting with one or multiple systems in one kind or the other. As long as our scripts are only reading we might not feel the need for mocking our services i.e. as long as they are not overly slow. But when we start writing to production systems we surely do want to be certain that our scripts run as intended. And this is where mocking comes in. Under C# there are multiple libraries for mocking external dependencies of a class. A very popular one (and my personal favorite) is Moq.

So let’s assume we want to mock an external dependency which returns us all the entries in a directory. We would write it something like this in C# (a bit lenthy I know...):

Now in PowerShell we would implment the function as follows:

And Pester provides a handy mocking functionality:

Not only is this a great way to test our code without having to actually be on the filesystem but it also shows how we could mock other dependencies such as a system clock (midnight testing anyone?), network calls etc. which for one will ensure that the script does not actually invoke any external sources. Another benefit you will most probably see is a speed up in execution time. Since everything is running from in memory (aka fast Winking smile) there is no network or disk delay. For further information and options on mocking check out the official documentation of Pester.

Executing the Tests

We can also execute the test from the command line with the following command:

Invoke-Pester

The command can be extended with parameters described here. While skimming through the parameters you may notice that they would provide the means to enable Continuous Integration (CI) for PowerShell Scripts. This is the case, we could add a Step to a Visual Studio Team Service (VSTS) or Team Foundation Server (TFS) build configuration by adding a PowerShell step with the following parameters:

If you already have a build server setup (you can get VSTS for free within minutes….), it is easy to setup a CI job for your scripts. And CI can be a real cure to sleep problems – so if I can have it this cheap the answer is yes please Smile

Summary

In this post we went through how to write tests with PowerShell and execute them. Further we saw how we can mock dependencies. We further looked at how tests can not only be executed during development but also how we can execute the tests in an automated build process i.e. Continuous Integration (CI) environment.

References

Check out my previous posts on how to get started with PowerShell for the C# developer:

You can find more detailed information about Pester on their Project Page.

External References

0 Comments
 

title image showing 1s and 0s in a green console style.

Running your tests on Xamarin Test Cloud (XTC) is as easy as right clicking onto tests and select run in XTC. As easy as this approach is – as limited it becomes when more sophisticated demands arise. Those might be running a group of tests, running on preconfigured device groups or integrating into an automated Continuous Integration (CI) and Continuous Deployment (CD) process. But there are also some additional useful features and information which we shall see in the follow up of this blog post.

Thereby the focus will be on the following:

  • Creating and Running the basic Command
  • Setting Categories and using them in the Command
  • Recording your Android XTC test runs
  • Using the XTC API to retrieve even more information

So lets get started Smile 

The basics Command

The easiest way to get a test script is if you login to the Xamarin Test Cloud dashboard website. Click on New Test Run, now you can either select an existing app or just choose to create a new App, select a team, devices that should be included in the test run and some additional information such as series, device locale and parallelisation. In the last step a sample script is generated which is the basic recipe for running the tests from the command line.

Now this script has to be adopted at the various points to reflect the setup of the app under test but from there on simply navigate to the root e.g. the solution folder and execute the script from the command line, power shell or the shell of your choosing. The tests will be uploaded and run on the selected devices. Once the tests are completed the results can be viewed in the XTC dashboard.

Running specific categories of your test suite

XTC allows to set categories when declaring a test as follows:

Categories allow to choose a subset of the tests to be run when executing them on the XTC. This can be a great way to create fast running smoke tests that give instant feedback after a build or running focused tests on a area of the app. The command can be extended by adding the following parameter:

If applicable one choose multiple categories for one run which makes mix and matching an easy undertaking. So a command using e.g. the categories SmokeTest and Purchase would be the following command:

Okay now to becoming more insights into tests, which is especially interesting when they are failing…

Recording your test runs

The basic option to get a better sense of what is going on during a test run is taking screenshots. This can be done by adding the following line to your tests as follows:

Note taking screenshots might increase the duration of a test run, in case of large suites one might want to reduce the overall time by using a constant within the UI test project that enables/disables detailed reporting of the test runs.

Now screenshots are already a great help. But in case of Android you are able to record the test runs. How awesome is that?! Enabling recordings is as easy as adding an additional parameter to the command line:

The recording can be viewed on the test dashboard with the very usable feature of speeding up the playback so you wont be bored by watching how your test fights it’s way to the juicy part. And as of date of writing recordings are only available for Android and can be solely enabled by adding the parameter to the Command Line Interface (CLI). If you try to use the command on an iOS run there will be no exceptions but the recording will just simply be missing.

Also don’t forget to add a good failure message to the Assert. Or you will be left with the reason of a failed test because an expected value was false instead of true (not helpful for me…).

Accessing the XTC API

Invoking tests via the command line is a great way to integrate UI tests running on XTC to an automated DevOps build pipeline. By default adding the following parameter to the command will allow to receive an NUnit Test report, it gives a very high level report of the run i.e. will show which tests have failed but not on which device etc.. If you desire to have a richer interaction with the data you can start interacting with the Xamarin Test Cloud API. There is a ton of information you can retrieve about a test run when using the API. Unfortunately at the current time it is not possible to retrieve images, recordings and some other information. If you are interested in giving the team an initiative to retrieve this information you can vote for this feature here.

All in all if you are using an on premise or other third party dashboard for displaying your test results, make sure you check out the API as it may just provide the more detailed information you are looking for and do not find within the NUnit test-report.

Conclusion

Starting test runs for Xamarin Test Cloud from the command line not only lets you integrate it into a DevOps toolchain but also allows you to run sub-sets of your tests, or record your test runs for Android. Last but not least if you are interested on feeding your test results into a dashboard the Xamarin Test Cloud API might just provide you with the information you are looking for.

The command line option is not only valuable for integration into automated toolchains but also for a more granular control when executing the tests locally. Be it from Windows or OSX the Xamarin Test Cloud toolchain has you covered.

0 Comments
 

Blog Title Image showing ios logo and Beta Testing with a ruber stamp effect.

When developing your latest and greatest (Xamarin) iOS app there will come a time when you are ready to ship the app to users for some beta testing. This allows you to get some feedback before shipping it to the official store. First thing needed is the application package also known by it’s file ending the IPA. If you are using Xamarin.iOS you can find a more details on building the IPA and setting up the certificates here. Assuming that you have an IPA ready for distribution the most common instruction found on the internet is to use iTunes to side load the app. Now this has a bit of an indirection to it, since the user must be sitting in front of a Laptop which allows him to install the app. In case of a company Laptop with IT policies this may end up being rather complicated. But there is an easier way, you can send a link which the user can select on an iOS Device which will install the app directly onto the device.

For the user to be able to use the app the certificates have to be correctly configured. If not, the user will be able to install the app but unable to launch it, which shows itself with by either closing right after the splash screen or a greyed out app icon.

Sharing your app via link

Now since it would be way to easy to just simply send a link to a file share which allows you to install the app, we jump through an additional hoop to comply with Apples “process”. We need to send the user(s) a link to a plist file that includes a link to the actual app i.e. IPA file:

Now lets go through this step by step:

  1. Upload IPA to the file share of your choosing and get a public URL for the IPA
  2. Edit the plist template above to match the requirements of your app (note the comments)
  3. Upload the plist file to Fileshare and get a public URL to the plist-File
  4. Send link to plist-File to whomever you would like to have the app installed

Since I was met with a confused look the first time I explained this to a co-worker see the graphic bellow and just think of the link to the plist file as a pointer of a pointer Winking smile:

Shows image where beta users taps on a link that is pointed towards the plist file (lying on a file share). The plist itself contains a link to the ipa file which will get installed on the users device.

This is a super easy way to share your app with testers. Then again these steps have to be performed for each update of the software and there is no user monitoring included with this approach. So you can not see if the app crashed nor can you know if a user already installed the latest version of the app. Or simply inform the user while he is using the app that there is an update. Now all this can be included in an app but it requires quite some effort to do so and then again we are talking about code that is only used during beta testing. So if you are developing a larger app with a longer lifetime or multiple beta releases you might want to consider a third party service e.g. Hockey App.

Conclusion

Setting yourself or your company up for running beta tests for iOS is as easy as sharing a link to a file (with some overhead for the publisher). No additional accounts needed or third party services. You could even share this thing from a NAS. That being said I would recommend you check out Hockey App as they do provide a free tier for up to two apps for free (at time of writing) and the process of adding it to your app is a couple of lines of code and a NuGet package.

HTH

0 Comments
 

testcloud_devices

When developing an app e.g. a mobile app for iOS, Android and Windows 10 or a web app what initially stands in the spotlight is a core idea that brings a core value to it’s users. As soon as the scope is better known one ideally starts developing a Minimal Viable Product which demonstrates the core ideas and let’s the creators get first feedback from potential users. With the initial ideas and the feedback the team starts to add more and more functionality to the app and all is well. Until users start reporting some odd behaviour and the well intended changes start having side effects on other features. So every release gets thoroughly tested before it is sent out. The time it takes to check on all features starts to increase, but holding off of going through all the tests may lead to a new bug that snuck into the app. So what to do?

Bring The Fairy Dust - Automated UI tests

This is often the point where one person e.g. a non-technical manager hears about a UI testing framework – which solves exactly the problem of automating manual tests. Plus one great thing about a UI testing framework usually is that there are no changes required in the app. The coverage of a test is very high since one works against the entire stack.

Automated UI tests simulate user interaction on the UI layer of an app. Depending on what kind of app there are different frameworks that one can choose from. For mobile apps on iOS and Android the solution we will be looking at is Xamarin Test Cloud (XTC) which is based on the Calabash testing framework. XTC allows to create UI test that can be used across platforms. They can be run locally i.e. read “phones attached to the Computer of the tester”. Far more interesting though is the opportunity of running the same tests in Xamarins Test Cloud which offers hundreds of devices (with different OS versions) on which the app and tests can be run on.

But as so many things in life - UI tests come at a cost. Compared to unit- or integration tests they are rather slow and like integration tests they tend to be more on the brittle side i.e. a test fails due to changes to the app. This is especially true if the UI is still under a lot of change. That being said they can be a great tool to bring down manual testing time. But solely relying on UI tests for your mobile tests will most probably not result in the result you were hoping for. So what are the sweet spots where automated UI tests will bring you the largest benefit?

Well lets start by looking at what kind of tests you can actually implement when considering mobile applications:

TestDistribution

As the graphic indicates the wider the pyramid the more tests you will probably want to have in your project. So lets dive into what the main benefits are for the different areas and let’s also look at how they could be implemented.

Unit Tests

Testing your basic logic of a class is where Unit Tests come into play. A Unit Test should not rely on any dependencies a class may have and therefore requires Stubs and Mocks to evaluate the correct working of business logic. Unit Tests should always pass without an error since they check the basic logical implementations of an app.

UI Tests should always pass i.e. be green since they only test logic without any dependencies.

Further Unit Tests are fast. In fact a Unit Tests is considered slow if it runs longer than100 milliseconds. For larger projects it is not uncommon to have a large number of these tests checking that all the small building blocks are working according to specification/user story. The additional work required for stubbing out dependencies may result in tedious work for certain parts of an app that are mainly coordinating workflows. Further since they only test logic in isolation the Unit Tests do not give any feedback how the components work together. But when all Unit Tests are green the confidence in the logic of an is very high and allows to shift the focus on search for an issue to the interaction of the building blocks which is where Integration Tests come into play.

Characteristics of a Unit Test

What is testedLogic Blocks, single Classes, Parts of Classes
Execution Time / Test< 100 ms
ReliabilityVery High
Impact on ArchitectureHigh

Integration Tests

When multiple parts e.g. classes of an app want to be tested an integration test is usually the way to go. Integration tests can range from testing a couple of classes that work closely together to system tests that require multiple parts of a system up and running. For external services that are not part of the developed app stubs and mocks may be used to ensure that tests do not fail due to services failing outside of the teams scope. Integration tests usually take more time to run since they run the real life scenarios. Due to the number of different parts and sub-systems an integration test may be running through the tests tend to be more brittle than a unit test. Further it can get harder to pin point errors to a specific area of a project, so unit tests are a perfect completion to integration tests as they allow to test complex logical parts in separation. If we look at a standard app built according to the MVVM pattern, the Integration tests are usually run from the View Model or lower in the stack. Integration Tests do not only allow to test for functional correctness of an app but also allow to get a feel for how performant an app is. If you are interested to run your tests on an actual device you can read more about this topic here.

One issue with integration tests is that the View is not being tested. Even though may argument that when the MVVM pattern is applied correctly there is only minimal logic remaining in the View, the user will still use the app via the view. This leaves the possibility that the app will still have issues when operating it through the view. Here (automated) UI tests come into play.

Generally speaking if a project has no automated tests at all, integration tests tend to give a bigger bang for the buck. Since the architectural requirements are not equally thorough.

Characteristics of an Integration Test

What is testedInteraction between componentens, Integration with other System parts, Network communications, Long running tests, memory consumption
Execution Time / TestSeconds to hours
ReliabilityMedium - High
Impact on ArchitectureMedium

UI Tests – Xamarin Test Cloud

Now it might be tempting to avoid testing your app altogether when having a sophisticated test suite of Unit- and Integration Tests, but in the end the user will interact via the UI with the app. And generally speaking they do not care if the logic beneath is sound, when the app crashes on their Device due to some incompatibility with a UI element that is used in the app. So the minimum UI testing every app project should be performing is the manual test. A person taps through the app and ensures that the UI responds as expected and follows the intended workflow. Now as stated before a manual test takes a lot of time and the resource performing the test I.e. a human might feel a bit underwhelmed. So generally speaking where can we utilise automated UI tests to improve test coverage:

  • Smoke Testing
  • Workflows
  • Testing interaction
  • Presence of Elements
  • Localization
  • Multi device tests

It might be tempting to think one can eliminate manual testing altogether, but there are some areas that UI tests just won’t be able to cover automatically and will always need a human:

  • Layout (Elements not rendering correctly)
  • System Interactions
    • Camera
    • Photo Album
    • Sharing
  • UI “Snappiness”
  • Exploratory Testing
  • Offline Testing
  • Bluetooth

Even though the Layout can be covered by an automated test insofar, that it takes pictures which can be verified by a user/MD5 Hash compared. A computer (at least at the time of writing) does not have the capability to analyse the layout of an app for its correctness. So a human eye and brain is still needed for some tests, this is something to keep in mind when automating all the things. Other limitations come either from the setup e.g. there must be a WiFi connection to the phone to run the automated UI tests. In general testing interconnectivity to other devices i.e. proprietary hardware or plainly some other phone that has to be paired, UI tests often become quite cumbersome to setup and the XTC does generally not support these kind of scenarios. So yes there are limitations to UI tests but using them properly will not only greatly enhance the coverage of automated tests but also free up valuable tester time that can then be used to put the human brain to more advanced testing tasks then numbly tapping through an app.

Characteristics of a UI Test

What is testedTests entire app software stack, Smoke Testing, Workflow Testing
Execution Time / TestMinutes
ReliabilityMedium – Low
Impact on ArchitectureLow

Conclusion

There are multiple different ways how one can approach automated testing of a mobile app. On one end there are tests verifying the logic of the single building blocks an application is built upon. Integration tests allow to verify the correct interworking of parts of the app, this can extend over the border of a an app and also include calls to a backend for verifying correct integration. The highest level tests available to mobile applicaitons are automated UI tests. The Xamarin Test Cloud allows create UI tests that can be executed on many hundred of devices. Testing apps on so many devices before the app ships can lead to valuable insights and automating these tests allows to easily perform regression tests not only on logic and integration layers but also for the UI. Think about an upcoming OS release and you want to check if there any issues with your app, given automated UI tests you can easily run a test suite that will quickly inform you if there is any concern with the upcoming update.

0 Comments
 

TitleImage

Writing Unit, Component and Integration tests allows to test functionality of software modules. In other words letting developers sleep tightly without worries at night. These tests run against certain parts of the code verifying the logic and interaction between modules of the app. All this is usually performed on the developers machine powered by a beefy processor and often unheard amount of RAM when it comes to mobile devices. Wouldn’t it be great if we could get a bit closer to the real thing? Well we can…

In this blog post we will look at how we can run xUnit.net tests on a device. The app will be a Xamarin(.Forms) app, which consists of three platform projects and a shared Portable Class Library which can be used to write logic that can be consumed on every platform.

Setting up xUnit.net

The typical xUnit.net tests are created in a standard .Net library. All that is required is the xUnit.net NuGet package:

PM> Install-Package xunit -Version 2.0.0

Now we can create start writing unit tests, for this blog post let’s stick to the basic calculator example in the BasicMathServiceTest.cs:

public class BasicMathServiceTest
{
    private BasicMathService _basicMathService;

    public BasicMathServiceTest()
    {
        _basicMathService = new BasicMathService();
    }

    [Fact]
    public void Add_GivenTwoNumbers_TheSumIsReturned()
    {
        var x = 35;
        var y = 7;
        var expectedResult = 42;

        var result = _basicMathService.Add(x, y);

        Assert.Equal(expectedResult, result);
    }

    [Fact]
    public void Subtract_GivenTwoNumbers_TheSubtractedResultIsReturned()
    {
        var x = 456;
        var y = 123;
        var expectedResult = 333;

        var result = _basicMathService.Subtract(x, y);

        Assert.Equal(expectedResult, result);
    }

    [Fact]
    public void Mulitply_GivenTwoNumbers_TheResultIsReturned()
    {
        var x = 2;
        var y = 2;
        var expectedResult = 4;

        var result = _basicMathService.Multiply(x, y);

        Assert.Equal(expectedResult, result);
    }

    [Fact]
    public void Divide_GivenTwoNumbers_TheSumIsReturned()
    {
        var x = 848;
        var y = 8;
        var expectedResult = 106;

        var result = _basicMathService.Divide(x, y);

        Assert.Equal(expectedResult, result);
    }

    [Fact]
    public void TheFailingTest()
    {
        var x = 4;
        var y = 2;
        var expectedResult = 8;

        var result = _basicMathService.Add(x, y);

        Assert.Equal(expectedResult, result);
    }
}

The code that we test against lives in a PCL and performs the calculations. Now let’s take these existing tests and execute them on the simulator or even a real device.

Running Tests on devices

For this first add the platforms on which the project will run. Now add the following xUnit.net NuGet package:

PM> Install-Package xunit.runner.devices -Version 1.0.0

Written by Oren Novotny whom we have to thank for enabling xUnit.net on Android, iOS and Windows Phone! This package will automatically add the xUnit.net and other dependencies required for running tests on your devices according to the platform to which it’s being added. Depending on the platform the xUnit.net for Devices package adds different template files as text files:

  • Android
    • MainActivity.cs.txt
  • iOS
    • AppDelegate.cs.txt
  • Windows Phone
    • MainPage.xaml.txt
    • MainPage.xaml.cs.txt

Simply replace the content of their default counterparts e.g. for iOS replace the AppDelegate.cs with the content of the App.Delegate.cs.txt (and thanks to some extra effort by Oren Novotny even the namespaces will just simply match Smile). Now you could simply start adding new class files and write out tests. The runner will automatically find them via reflection (even if you place them in subfolders).

Using existing tests

What is probably a more useful approach is to simply reuse existing tests. This is why in a next step lets see how we can add the unit test file BasicMathServiceTest.cs without creating a duplicate dopy. Simply right click on the Project then Add/Existing Item… browse to the unit test file and add it as link.

Shows the dropdown on the Add button that can be used to add a file as a link.

As a link to the file is simply a pointer to the original we only have to maintain one unit test file and all the links will automatically “update” as they simply reference the original.

Now the Project can be set as startup project and executed on the device. Bellow you see some sample screenshots from an iOS Emulator.

TestOverview

You can even dig into the error messages and see what went wrong during a test.

TestDetails

Conclusion

In this post we saw how to create and reuse xUnit.net tests on devices for iOS, Android and Windows. This can be very powerful when we have to test certain features on a device for example does this work with AOT on iOS? Or simply running integration tests on a real device, giving valuable insights and further allowing to tests performance, latency and so on.

This post focuses on xUnit.net and also shows one of it’s many strengths on the capabilities that can be easily added. The same can be done with NUnit which may be your testing framework of choice and which Xamarin provides project templates to run the tests on a device.

You can find the sample code on GitHub.