Xcode UI Tests problems

The problem with Xcode UI Tests

A little background

Ah, yes, automated UI tests. You cannot consider yourself a good developer and not love them. Or at least do them. Have tried them? Know about them?

To me, they are a lot like urban legends. Everyone knows about them, but not so many people claim to have seen them in action. And even fewer have actually seen them in action. But why is that?

Naturally, most often it’s the case of “Ain’t nobody got time for dat!”. But beyond… why do they tend to be overlooked. My theory is that they are simply too frustrating and tedious. There’s a major effort in making them easy to write and reliable, but really? Are they?

Lets look at Appium, for instance. It was supposed to be this great new tool that makes testing a breeze. And cross-platform too! But in reality it is (in my opinion at least) fragile. It can be slow, you have to wait for updates after a major Xcode release, and it’s not exactly cross platform in the real world. It comes with so much pain and maintenance costs, that it is often impractical to use.

And what about Xcode UI Tests? It was a huge deal when it was announced several years ago at WWDC. I mean, even before, we had UIAutomation, but surely, Xcode UI Tests are better, right? And yes, they are, but to be fair, it wasn’t that difficult to be better than UIAutomation.

The problem with UI Tests

Don’t get the wrong idea, I love the prospect of UI Tests. It would be an ideal way to test applications. It’s not manual, it’s relatively fast and it’s (almost) black box. Also, it’s really fun watching the simulator or your test device come to life and start tapping on buttons on its own.

That being said, why did I say it would be an ideal way to test?
It’s just because it often proves unreliable. There are just too many variables. So many moving parts need to come together in order for a test to run and execute to completion. Moreover, if you really want a black box test, you will have to live with your application accessing remote and/or persistent data. And by doing that, you become exposed to network issues and inconsistent state between test runs.

Additionally, if you look at the tools themselves, they can also be a bit unreliable. Just like I wrote about Appium above, it would be unrealistic to expect it to just work. You have to fight it all the time. And unfortunately, after many long fights, you will be tempted to abandon your tests.

The problem with Xcode UI Tests

With the release of Xcode’s UI Tests, one would expect that things would get better. And not because everything Apple touches magically turns into gold as many people believe, but because only they have the power to break through all the sandboxing and private API barriers and make something good happen. But I still feel there’s plenty to be desired.

Limited API

A major problem with the new Xcode UI Tests is that the programming interface is quite limited. Also, the documentation is seriously lacking. Now this is really unlike Apple. Usually, you’d get an API reference and a programming guide. But with UI tests, we get close to nothing. It almost feels like it’s incomplete.

Basically, all Apple provides is the User Interface Testing and the XCTest reference. Even doing a web search for it, doesn’t yield as many results as expected. I wonder why that is. Do people not understand it and avoid writing about it? Or they just ignore it?

Unintuitive classes and methods

While writing UI tests, you are never working with the classes you are familiar with from development. It’s always a facade (or proxy) of the real thing. Whether it’s XCUIElement, XCUIApplication or XCUIElementQuery, you will never be able to use the methods you are used to.

Text fields, switches, labels and buttons… they all have the same interface – they are hiding behind XCUIElement.

One nice example of how primitive the interface is, would be typing text in a text field. You’d expect there would be a property for that. But there isn’t. You have to type the text yourself. At least you don’t have to do it one character at a time. However, you DO need to make sure the text field has a first responder status by tapping on it. If you fail to do so, you’d get an exception. Additionally, if you want to set the text, not just append, you need to clear any previous characters manually. For that, you need to find the code for the backspace button (“\u{8}”, you’re welcome) AND also calculate how many times to type it in order to clear the whole field.

So, to do something so trivial and basic as setting the text on a text field, you need to:
* Tap on the field to make it a first responder
* See how many characters it already has
* Type backspace just as many times
* Type the desired text

And even though it takes 3 minutes to write a utility function for that, it just shows how unrefined the API really is.

Poor asynchronous operations

Sure, being asynchronous is probably not the highest priority item on a UI testing framework’s agenda. But shouldn’t it at least be in the top 5? Lets face it, some tasks take longer than others. And in the app scene nowadays, a server-side component is almost a must-have. So it should come to no surprise that tests needs to wait sometimes.

To be honest, Xcode UI Tests do. In a way… Sort of… sometimes. It’s of course quite blurry, since the resources are lacking, but here’s what I understood:

Waiting to idle

Yes, Xcode UI Tests “wait”. As far as I understand, waiting “for the app to idle” means that it will detect that there are animations still performing and wait for them to complete. But after they all finish, and some element you need is still not there, (boom). Assertion failure. Oh, well!

Waiting for long running tasks

Xcode UI Tests will not wait for several seconds in case something changes (by default). I think that was the case with other testing platforms, but not Xcode UI Tests.

Waiting for expectations

That being said, Xcode UI Tests support some sort of asynchronous tasks using expectations. And by “some sort of”, I mean that most cases are possible to achieve but sometimes quite annoying to implement. Spoiler alert – waiting for element to disappear or waiting an explicit time interval.

The basic idea is to create an “Expectation” with a predicate (most often the predicate is that an element exist, or a table has more than 0 cells). Then you wait until those expectations are fulfilled with a given timeout. If the predicate doesn’t evaluate to true after the timeout expires… bad news. There’s also a completion handler that gets called after the timeout but the annoying part is that there is nothing you can do there to avoid the test failure.

Most of the time, an expectation would look like this (in Swift 2):

_ = self.expectationForPredicate(NSPredicate(format: predicate), evaluatedWithObject: uiElement, handler: nil)
self.waitForExpectationsWithTimeout(timeout) { (error:NSError?) in
  if error != nil {
    print("*** Failed expectation for test: \(message)")
  }
}

For now, expectations don’t look too bad. The problems emerges when you realize how often you have to write that same code. As I said, every operation that might take a non-trivial amount of time, needs to have that. Normally, pushing or presenting a new screen wouldn’t require it. But then again, sometimes it does. At some point you stop caring and put it there as well.

On that note, what really gets on my nerves is that this is hardly explained in the documentation. Even more, using the famous recording mode, such code will never be generated. It will only generate taps making you assume you don’t need the expectations. Then you start running your tests only to find they often fail. I wonder how many people got too frustrated at that point and gave up.

Waiting specific time interval

As hinted above, this is something that’s especially frustrated about Xcode UI Tests. I don’t think it’s an edge case that developers might want to wait an explicit amount of time and then proceed. Even in UIAutomation, you could achieve that using action timeouts:

UIATarget.localTarget().pushTimeout(15);
UIATarget.localTarget().popTimeout();

It’s not ideal, because that’s just the time UIAutomation waits between actions, but still, it worked. However, Xcode UI Tests don’t have that…
If you want to wait explicitly, you have two options (at least I know two ways).

One would be a good old “sleep” on the thread. But lets face it – the intersection between programmers that “sleep” ( :) ) and programmers that automate tests, is not that big.

NSThread.sleepForTimeInterval(15)

Alternatively, some GCD magic can be employed. It’s a bit messy, though. But it’s a cool hack you can use to brag about when you meet your nerd friends:

func checkFor(predicate pred:NSPredicate, onElement:XCUIElement, afterDelay:NSTimeInterval, timeout:NSTimeInterval = 3) -> Bool {
  let waitExpectation = expectationWithDescription("Delaying for predicate \(pred) on element \(onElement)")
  let waitTime = dispatch_time(DISPATCH_TIME_NOW, Int64(UInt64(afterDelay) * NSEC_PER_SEC))
  dispatch_after(waitTime, dispatch_get_main_queue()) {
    waitExpectation.fulfill()
  }
  self.waitForExpectationsWithTimeout(afterDelay + 0.2, handler: nil)
  return pred.evaluateWithObject(onElement)
}

That’s not the easiest code to understand so lets explain it a little bit. We make an expectation and start waiting for it because we need to block the process for a while. And the trick is to automatically fulfill that expectation just before it “expires”. So we use grand central dispatch to fulfill the expectation after the timeout we want to wait. At that point, the process resumes because there is no expectation to wait for anymore. Finally, on the last line we evaluate the predicate directly, knowing the timeout has already passed.

Note: Another limitation about expectations is that there can only be one at the same time. That means you cannot wait for several things at the same time.

In general

I complained a lot in this article but I still want to end on a positive note. I actually like Xcode’s UI Tests, and I do use them. I’m convinced that they have a lot of potential. But on the other hand, they still need a lot of work until they become easy to use and reliable. it is all too easy to hit a brick wall with them and quit. After you go over the initial hurdle, it gets easier and you start seeing more and more potential with them.

I’m planning on writing a complete guide on using Xcode UI Tests in order to share my findings and hopefully spare some people a lot of hair pulling. But until then, only this rant about them will be available. Stay tuned!

Leave a Reply

Your email address will not be published. Required fields are marked *