UIAutomation in Xcode

Testing UI automatically

As you know, UI testing is tricky to do automatically, especially on mobile. On the desktop and in web development, there’s Selenium for instance,  but smartphones, being embedded devices, are hard to automate. Even test driven development leaves some to be desired when it comes to the user interface. Most people have to rely on manual testing. However, there is hope! There are frameworks, such as FoneMonkey that allow programmers to write scripts for functionally testing their applications. But more importantly, though many people don’t know it, such a tool is already integrated into Xcode itself – you can find it in the Automation section of the Xcode profiling tool.

What is UIAutomation?

UIAutomation is part of the “Instruments” application that comes bundled with Xcode. It allows developers to use JavaScript in order to write scripts that simulate actions on the user interface. This allows a large portion of the UI testing to become automatic. Using UIAutomation’s language, users can create touch events (both taps and complex multi-touch gestures), locate visual components such as views and buttons, examine the view hierarchy, create screenshots, check error conditions using predicates and log test results. While it does not provide the functionality of a unit test, it can definitely help you achieve a considerably higher degree of automation. As I mentioned earlier, there are other tools for automating user interface testing, but they cannot achieve the amount of integration and ease of use provided by UIAutomation. FoneMonkey, for instance, requires you to create a separate build target containing all of it’s libraries, whereas with UIAutomation you only hit the “Profile” button (instead of the “Run” button) and you’re all set. Now, in FoneMonkey’s defense, it is clear that Apple will never allow third parties to have the amount of access needed to create a fully integrated testing suite, so needless to say, the stock option, provided by Xcode will always be the winner in this category.

How does UIAutomation work?

This section is dedicated to some core UIAutomation principles. Though you can skip it if you only want to see some examples, I would recommend reading it so that you know how all parts fit together.

The first thing we should cover is the way UIAutomation identifies component in the user interface. A major strength of this tool is that your script doesn’t normally consist of  taps at specific coordinates. In fact, most of the time (when you’re not working with complex touch gestures) you don’t have to bother with coordinates at all. What you do is, you specify a view component, and you simulate a touch event to it. This makes your scripts very flexible in the sense that, if you change the layout of your screens or maybe modify it to fit another resolution like the iPad’s, you don’t have to rewrite your scripts. As long as a given component is still present on the screen, the script will find it and continue going. So how does UIAutomation achieve that?

All components in an UIAutomation script are identified by their accessibility label. These can be set both in Interface Builder and programatically. Normally, this label is used when providing accessibility features to an application, but here it doubles as the “name” of your view in the automation script’s scope.

Note: You don’t always need to have the accessibility label set. There are other ways to identify a view. The label just makes it more consistent.

Setting accessibility labels in Interface Builder

In Interface Builder, each view’s accessibility label can be easily set in the “identity inspector” – it is the third tab in the right assistant editor:

Accessibility options in Interface Builder
Accessibility options in Interface Builder

It is important to make sure the “Enabled” option is set. Afterwards, you can choose an appropriate name for your view. You should also think of something that you can be actually useful for accessibility, while you’re at it.

Setting accessibility labels programmatically

If you’re not using Interface builder, you can add accessibility labels to your views with the following code:

testedButton.accessibilityEnabled = YES;
testedButton.accessibilityLabel = @"Big red button";

That’s it.

How to start UIAutomation?

As I said, UIAutomation is part of Xcode’s Instruments tool. In order to start it, you need to run your application using the “Profile option” (accessed in Product -> Profile, in the toolbar or by pressing CMD + I). This will rebuild your application and launch the Instruments app. You should be presented with something like this:

Instruments
Figure 1: The UIAutomation “New” window

As you can see, the Instruments app provides a wide range of tools for tweaking your application. If you haven’t already, I suggest that you take the time to get to know (and love) them. But for now, let’s choose the “Automation” option and start writing scripts.

UIAutomation's home screen
Figure 2: UIAutomation’s home screen

This is where the magic happens. By the time you see this screen, the application should have started running. UIAutomation can be used on both the simulator and on a real device, though the simulator has a few limitations (it cannot take screenshots) but more on that later. In the screenshot above we can see that Apple’s GenericKeychain example has been running for 7 seconds now, and there is no script currently automating. It even tells as that an error has occurred while trying to run it. This is just Xcode’s fancy way of saying that there is no script to start currently.

As you can see, there is quite a lot going on in UIAutomation’s window. There are timers, timelines, recording buttons and a lot more. However, at least for now, all you’re going to need is the trace log, where you’ll be looking at your script running (and failing miserably), and the source code editor, where you’ll be writing your JavaScript. If that scares you as much as it scared me, don’t worry – the Javascript we will be using is fairly straight-forward and self-explanatory.

How to create a script in UIAutomation?

In my opinion, this is where UIAutomation starts to fall short. It’s user interface is just not intuitive enough. It supports running only a single script, several open files are not indicated well enough in the view and switching between the source editor, trace log and editor log is a mess. It’s not a big deal, but being bundled with the Xcode IDE, I expected a lot more.

Anyway, let’s create a new script. In the middle of the left pane, you will see an “Add” button. Clicking it will show a drop down menu allowing you to create, import or open a recent script. Figure 4 depicts just that:

The UIAutomation window. "1" creates a new script and "2" runs the selected one
Figure 4: The UIAutomation window. “1” creates a new script and “2” runs the selected one
Show the slightly unintuitive way to navigate between script, editor and trace log perspective
Figure 5: Show the slightly unintuitive way to navigate between script, editor and trace log perspective

After you do that, you should be presented with a source code editor and the following statement already inserted for you

var target = UIATarget.localTarget();

Figure 4 also shows the way scripts are run. Press the “Play” button at the bottom of the source editor and UIAutomation will immediately start executing. It is important to note that your application will NOT be rebuilt or re-run – the script will start from wherever state you left your app at. So make sure you navigate the user interface to whatever screen you expect in your JavaScript.

How to write UIAutomation scripts?

The reality is that UIAutomation is not as well documented as other iOS frameworks and tools. Apple’s documentation only shows a few generic examples and does not go into great detail explaining UIAutomation’s features. However, using this article and the Javascript API’s reference, you should be able to learn quickly. Here, we will go over all important principles of writing UIAutomation scripts and after that you should be able to start creating your own by looking up the classes in the API reference.

Let’s start from the beginning. What is that strange piece of code UIAutomation put at the start of my newly created script? “UIATarget.localTarget();”, most commonly seen as “UIATarget.localTarget().frontMostApp().mainWindow()” is the way you get a reference to the screen, currently displayed on the device. It should be the the view controller you plan on testing.

Accessing elements in the view hierarchy

The most important task for your script will be accessing different UI components. As mentioned earlier, UIAutomation is not about tapping on certain coordinates, but identifying views and changing their properties. If you are in doubt where a certain component is located within the view structure, you can use the following code to print it in a convenient manner:

element.logElementTree();

This will print all components in a tree-like hierarchy inside the trace log and even create a screenshot of each view. This makes debugging your script very visual and user-friendly. Unfortunately, creating screenshots is not available when using the simulator.

You can access an element’s “children” using several methods provided by the element’s class. Here’s some examples of such methods:

  • tableViews()
  • cells()
  • textFields()
  • secureTextFields()
  • buttons()
  • tabBar()
  • etc.

Since you have a reference to the currently displayed view controller, you can start going down the view hierarchy. You can also go upwards, accessing an element’s superview:

element.parent();

As you can see, most of these methods return an array, containing all components that match the requested criteria (for instance, all buttons in the view). If you are unfamiliar with JavaScript, here’s how to access an element of this array:

element.buttons()[2];

Now, you might say that just hardcoding that “2” in not a very flexible thing to do. What if a new button is added… or the others rearranged? Well, remember that thing about the accessibility labels above? You can also refer to components by them:

element.buttons()["Login Button"];

Here, we specifically said that we wanted the button, named “Login button”. In my experience, the name does not necessarily have to be the accessibility label. It can be the button’s title label value. In fact, if you don’t want to modify your existing source code, you might just use that. However, button titles are subject to localization and possibly branding and you might be better off using the accessibility (if it’s not localized as well).

I would like to emphasis something that caused me frustration a while ago. There is a difference between a text field and a secure text field. You cannot access a secure text field by calling “element.textFields()”, you have to use “element.secureTextFields()”.

Generating touch events

Naturally, the most important job of a user interface automation framework is to generate input events. Fortunately,  this is made easy with UIAutomation. I guess generating tap gestures is good enough for most people. Here’s how it’s done:

UIATarget.localTarget().frontMostApp().mainWindow().tabBar().buttons()[4].tap();

As you can see, it is just a matter of calling the “tap” method of an element. The code example above sends a single finger tap event to the fifth button in a tab bar.

UIAutomation can do a lot more than simple taps. It can perform complex multitouch gestures. I’m not going to go into details about them, but you can read about them from Apple’s documentation.

Introducing delays between UIAutomation actions

First of all, maybe you should know a little more about the way UIAutomation schedules actions. It is actually very simple – tries to execute an action and if your application is not in a state that can accept this sort of event, it will wait a specified amount of time. If it is unable to complete the action and this timeout expires, the test fails.

This timeout can be changed during runtime using the following code:

UIATarget.localTarget().pushTimeout(15);

And to revert that change, all you do is:

UIATarget.localTarget().popTimeout();

In the example, we changed the timeout between actions to 15 seconds and then reverted to the default value, which should be 5 seconds. Five seconds, in my experience seems to be good enough for most cases, but I guess that if you will be waiting for data coming from the network, you might want to consider increasing that value.

Logging information in UIAutomation’s trace log

Naturally, UIAutomation provides support for logging arbitrary information in the trace log. Most actions that your scripts do are already logged, but it is always nice to be able to provide additional information for debugging. Especially since UIAutomation does not allow you to run multiple scripts, you will need a way to log which test you are running in order to know what went wrong if the script fails. Available here is a reference of all logging methods supported. Overall you would be using “logFail” and “logPass” to indicate the test result, “logStart” to mark the beginning of a new test and “logMessage” (or any other method with different log level) to write important data into the trace. Unfortunately, unlike it’s counterpart from Objective-C, NSLog, UIALogger does not provide a printf-like signature, but you can use the following to, at least, concatenate objects into the log message:

UIALogger.logPass("Hello " + "world");

I just had to include a “Hello world” reference. Also, the Javascript language includes a “sprintf” function, but I guess it is not available here.

Another useful feature of UIALogger is the ability to create screenshot. This only works on iOS devices and not the simulator. Try this:

UIATarget.localTarget().captureScreenWithName("ScreenShot");

An image of the currently displayed screen should appear right into the trace log. Very useful if a test fails and you want to save whatever state the user interface was in while it happened.

Figure 6 show what a typical trace log looks like.

 

A trace log showing several failed tests
Figure 6. A trace log showing several failed tests

Verifying test results

Since UIAutomation is a testing platform, obviously it is going to need a way to verify that the application being tested is acting the way it should. Unfortunately, there isn’t an integrated way to verify test results. For the most part you will have to come up with your own pass criteria and perform checks manually.

Example: Did the application login? Well, how do you define a successful login? The isLoggedIn method returns YES? It does, but you’re not debugging and you don’t have access to the isLoggedIn method in your script. Hmmm…

In the example above your verification will have to check that your user interface shows the screen that is displayed right after login. Something like “The client zone screen appears” or “the tab bar is now visible”. Choosing a criteria is not always easy. You have to choose something that will ALWAYS be true for every successful login, but false for every failed attempt. Also it has to be something that will still work after same UI changes are made. For the most part test result verification looks something like this:

var logoutButton = UIATarget.localTarget().frontMostApp().mainWindow().buttons()["Logout"];

if (logoutButton()) {
    UIALogger.logPass("Application was able to login");
}
else {
    UIALogger.logFail("Login failed");
}

Here, the script tries to get a reference to a button named “Logout” inside the screen’s view hierarchy. The test is considered to be have passed if such a button exists.

To make your life easier, UIAutomation has support for using predicates. It also uses the same syntax used for NSPredicate:

var textField = screen.textFields()[0].firstWithPredicate("value is ‘Username’");

Here, predicates are used in order to find the right text field. Of course, the example is overly simplistic but using predicates, you will be able to verify complex conditions.

Handling alerts

While testing on a real devices, execution might be interrupted by unexpected events such as receiving a call, a local notification or even a low battery warning. Also, your application will probably be using some alerts of it’s own. UIAutomation provides functionality for handling these alerts. Intercepting an alert can be done in the following way:

UIATarget.onAlert = function onAlert(alert){
    var title = alert.name();
    UIALogger.logWarning("Alert with title ’" + title + "’ encountered!");
    return false;
}

Every time an alert is shown during the execution of your script, this function will be called. If it returns false, the alert will be dismissed automatically by tapping the cancel button, and you can perform some additional actions provided that you require some additional processing:

UIATarget.onAlert = function onAlert(alert) {
    var title = alert.name();
    if (title == "This error") {
        alert.buttons()["Ignore"].tap();
        return true;
    }
    return false;
}

This demonstrates that you can decide not to ignore the alert altogether, but decide which option to choose and tap that button.

Conclusion

UIAutomation is available with Xcode 4.5 and later and is a tool that definitely deserves your attention. Use the comment section to let me know – will you be using UIAutomation in the future and do you plan on teaching your QA team write automation scripts?

Thanks for reading!

Using the debugger console lldb in Xcode

 

Let’s take a moment to talk about the Xcode debugger. I’m sure each and every one of you knows how to debug, but here, we are going to be talking about some “more advanced” usage of lldb in Xcode. This post is not going to be about continuing or stepping over or setting a breakpoint – it’s about getting more information from the program while we are at a breakpoint. These are features of the lldb debugger that I use on daily bases and I thought I’d share my experience.

Using lldb in Xcode

This one’s easy – you probably already know how to do that, but let’s mention it real quick just for reference. The debug console is located on the bottom of the console area in Xcode:

debugConsole

 

Typing commands there is done by clicking next to the blue (lldb) text. When you do that, a cursor will appear on the right and if you start typing, text should appear on the screen like in the screenshot above.

Printing backtraces in the lldb console

Have you ever received crash reports from another developer? When they sent you a back trace, did it look like this:

debugbt

If this has happened to you… I feel your pain. It pains me to see something like this. The RIGHT way to get a backtrace while debugging is by typing the following:

(lldb) bt

The bt command (short for backtrace) will print the current thread’s stack in the console. You can then copy it and send it to a colleague. Sure beats launching the Grab application to make a screenshot.

Printing objects in the lldb console

While you are at a breakpoint, we can see the values of all variables in the current scope by looking at the debugging area in Xcode. This is extremely helpful, but you will agree that it doesn’t always show what you’re looking for. Consider this – you have some NSData coming from an NSURLConnection and you want to see what’s written there. If you try to do that in the debugging area, you will only see some addresses and hex values. You’d much rather see it as a string. Even the “print description” option will be useless in this case. You need to get a string representation of your data – you need

[[NSString alloc] initWithData:data:usingEncoding:NSUTF8StringEncoding]

We are going to do just that using lldb. The “po” (print object) command takes an arbitrary expression, that returns an object and prints it’s description in the console. So the following statement:

(lldb) po [[NSString alloc] initWithData:data:usingEncoding:NSUTF8StringEncoding]

Is going to print the string representation of data. Well, actually, it will print:

(lldb) po [[NSString alloc] initWithData:data encoding:NSUTF8StringEncoding]
error: use of undeclared identifier 'NSUTF8StringEncoding'
error: 1 errors parsing expression

Unfortunately, lldb doesn’t seem to be able to recognize enums. So, instead of using NSUTF8StringEncoding, we will have to use it’s integer representation – 4.

(lldb) po [[NSString alloc] initWithData:data encoding:4]
I love chicken!

So that’s printing objects covered. But not everything in Objective-C is an object. You also have primitive data like int or char for example. What’s more, C++ objects are not the same as Objective-C objects. How do you print those?

The lldb print command

The print command can be used to print primitive data types. It’s syntax is the same as po:

(lldb) print array.count
5

Printf in lldb

When it comes to printing debug information, you can’t beat printf. For example, I use it whenever I want to print a C++ object. Of course, it needs to have a method like “toString” that can generate a string representation of it. Here’s how you do it:

(lldb) print (int) printf("%s", cObject->toString())

So printf allows you to display any information (currently in scope) that you can using the printf command.

Another trick I use with Xcode’s debugger is that when I want to extract some application-wide information, I pause the program’s execution using the “Pause” button in the debug area and then use “po” or “print” to get some data. For example, I needed to extract the token used for communicating with a server. It was stored in a singleton object named LoginManager. I was able to copy the access token using the following command:

(lldb) po [[LoginManager instance] token]

gcc vs. lldb

I know some of you might already know how to use the gcc console and are transitioning into lldb.  This table will show you all gcc commands and their lldb counterparts.

Conclusion

It might not seem like that at first, but once you start using these lldb commands (and maybe others) you will notice that they can help a lot. They are a powerful tool in any developer’s repertoire. I suggest you give it a shot and see for yourself.

Once again, thanks for reading.