Snapshot testing with FBSnapshotTestCase

If you had asked me two weeks ago about snapshot testing, I’d definitely tell you it’s stupid. And I think I had my reasons… I don’t regret it. But I guess today’s opposite day, because my oppinion turned 180 degres. I haven’t already made my mind up, but it seems like snapshot testing is quite promising.

What is snapshot testing

I’m sure you’ve already heard about it, probably from your hipster friend. Basically, snapshot testing is a way to automatically test the aesthetic part of your UI. It consists of creating a view from your app and comparing it against a reference image. It they match, all is good and people are happy. But if even one pixel is off, women scream and children cry.

I think that’s exactly what designers expect to happen when something is one pixel off.

Man, snapshot testing is lame

Just a few sentences in and many of you probably already scream at the monitor and have foam forming in the corner of your mouths. And I know what you’re thinking.

  1. How can the screen be exactly the same as the picture?
  2. How can I generate an image that’s exactly identical to my UI?
  3. How can I make these tests repeatable with a pixel-to-pixel precision?
  4. Isn’t comparing images totally slow?
  5. How do I know what exactly is different between the two images?
  6. How do I live with myself knowing my test fail for no reason… which they will?
  7. When I change my UI’s appearance tomorrow, who’s going to give me new reference images? Not me!

Ofcourse, these are all good points. I feel the same way :). But let me show you how one little framework attempts to answer all those quiestions.

Introducing FBSnapshotTestCase

FBSnapshotTestCase is a framework developed by a company you might have heard of. It’s called Facebook.

Maaaan, snapshot testing… Facebook… This article sucks!

It probably does… but not because of FBSnapshotTestCase.

Aaaaaanyway, what the FBSnapshotTestCase library does is not a lot, but it’s enough.

An XCTestCase subclass

The first thing that deserves credit is the fact that the FBSnapshotTestCase class inherits from XCTestCase, essentially making comparing snapshots a unit test. And yes, I know this totally violates many of the core principals of unit testing and TDD, but still it’s useful to get all the goodies from it. So what kind of goodies are we talking about:

Xcode integration

To Xcode, your FBSnapshotTestCase suite is just another unit test collection. So you get all the IDE support for it like:

  • CMD+U to run tests
  • UI for showing test results
  • Easy continuous integration

Reliability

Since you are not running your app with real data, you can isolate different screens and even views. That makes it easy to test different components in isolation using hardcoded data. This sagnificantly reduces the possibility of random failures. Go Facebook!

Isolation

I’m repeating myself a little here, but I cannot stress enough how useful is to test components in isolation. You are not limited to testing the whole screen. Just instantiate a view and see if it looks like it should. Allocate a view controller, inflate a Xib, load a storyboard. It’s all on the table.

Recording mode

One question I’ve always asked myself when thinking about snapshot testing is… Who the hell creates all those reference images and how come they match the app 100%?. The answer is recording mode…

FBSnapshotTestCase lets the app generate reference images for itself. The class has a property named recordingMode. If set to YES, instead of comparing images, it will make a snapshot of your view and store in the in the reference images folder. Essentially, whenever you’ve verified that your view looks good, you can run your test in recording mode and it will become a reference for all subsequent runs.

Well, it still sucks that you need to change some property in order to generate reference image. And then, you will probably forget to remove the recording mode and suddenly tests start failing… But anyway, life’s not perfect.

Playing nice with animations?

So what if you’ve done some animations that go along with your views – a snapshot will only capture the initial position.

Not necessarily. Most of the time, animations are fine. Since they mostly work on layers, UIView’s drawViewHierarchyInRect method is smart enough to render the view’s real… uhm… final position. Thus most of your animations are safe and you will be able to get an image of your UI after they are done.

performSelector: ?

Oh, you got me there… If your views rely on NSObject’s performSelector for delaying actions, you might (and will) run into problems with FBSnapshotTestCase. Well, theoretically you can try to work around it…

Runloops?

For the above mentioned performSelector and other time-based issues you might try the following:

[[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:1.0f]];

This will effectively activate the runloop for 1 second and force it to execute everything scheduled on it. However, I wouldn’t recommend using it unless you have no other choice. And let’s be honest… you always have another choice :).

The problem with running the runloop is that it will autimatically increase your tests in length. For the above example, that’s one extra second. And it’s a secred TDD rule to have tests that are fast… really, really fast. The idea behind is that developers should be running test all the time during implementation. And if tests take a minute to complete, they’re are just not going to do that.

Even though FBSnapshotTestCase is not exactly unit testing and running snapshot tests every 2 minutes during development is not that useful, it still helps to follow TDD rules and keep execution times to a minimim. Spoiler alert – 5-10 seconds should be more than enough for all your FBSnapshotTestCase suits.

Changing code to make it more… testable

As you can see, most of your UI classes can be snapshot tested without modification. But if some of them fall victom to the symptoms we talked about in the last chapter, you will be faced with a decision:

  1. Change your production code to allow snapshot testing
  2. Skip testing for that particular item

The second option might be tempting, and depending on how important and risky that component is, it may be the best solution. But still, the first option is worth a try!

To change production… or not to change production

It’s an old phylosophical question… Should you change production code in order to make it more testable, or should test code try to adapt to production code. Some developers agree to the former, some to the latter. Test-driven developement for one thing has the core value of shaping your production code according to your tests. On the other hand, many programmers would argue that if a testing platform forces you to change your application, it’s obviouly not doing its job.

So the truth is out there (X-files anyone?). If you are dumb enough to ask me, the answer depends on some additional factors.
First of all, I agree that the testing “harness” shouldn’t interfere with production code. That being said… there’s an exception! So:

The testing “harness” shouldn’t interfere with production code… unless it forces you to change it for the better!

And that’s exactly where TDD comes in. Yes, it forces you to change the way you write production code, BUT only to make you write better structured programs.

I like to consider non-testable or hard to test classes as [code smells][Code smell definition]. If you find it hard to put into a test harness, it’s probably because it’s too tightly coupled to something or it tries to be more than it should ([Single responsibility principle][Single responsibility principle definition] violation). So it might be worth fixing that, instead of trying to make your tests work around it.

Granted, this holds true more about TDD and less about Snapshot testing with FBSnapshotTestCase. If you’re not able to create the reference image you want, that doesn’t mean your class is not well structured.

Coping with unwieldy animations

This is going to sound terribly underwhelming but a universal way of doing that is to… not have animations in the first place. I would like to bring a common pattern from UIKit to your attention. For example, the method for changing the value of a UISwitch:

- (void)setOn:(BOOL)on animated:(BOOL)animated;

You can specify if you want the animation or not. Normally, the purpose of this is that if you want to change the value while your view is not on screen and the user is not going to see it, you might choose to have it not animated.

So being able to turn off animations can be a cool thing to have. Even if you don’t use it right for now. (And yes, I do realize writing code you’re not using violates some core principals of TDD, but hey, I also like to live life dangerously :) ).

Adding FBSnapshotTestCase to a project

Get the framework

There are two ways to add FBSnapshotTestCase to your project. The easy way… and the way I did it. The easy way is CocoaPods. Yes, FBSnapshotTestCase supports CocoaPods so chill. So if you’re one of those guys that use CocoaPods, you’re all set. But the thing is, I don’t love myself enough to use it. What I did was to compile FBSnapshotTestCase myself. In fact, that was kind of simple. The folks in Facebook already have a target in their Xcode project that generates a .framework file. So all I did was build it. The trick is that you might want to compile for multiple architectures. But even that is easy. If you are nerdy enough to be interested in that, you can read my Compiling libraries article.

Reference directory

One thing you need to do in order to setup FBSnapshotTestCase correctly is to set the FB_REFERENCE_IMAGE_DIR variable. Again, there are a couple of options. The lazy way is to set it directly in code somewhere. But lets face it – that’s not how the cool kids do it. Instead (and this is what the authors recommend) they would set it as an environment variable. As most of you probably already know, that’s done in the “Edit Scheme screen”.

Scheme settings image

One important note here is that you need to make sure the ” alt=”” /> @implementation LoginSnapshotTests

- (void)setUp {
    [super setUp];
    self.recordMode = NO;
}

- (void)tearDown {
    // Put teardown code here. This method is called after the invocation of each test method in the class.
    [super tearDown];
}

- (void)testLogin {
    UIStoryboard* loginStoryboard = [UIStoryboard storyboardWithName:@"Login" bundle:nil];
    LoginViewController* loginController = [loginStoryboard instantiateViewControllerWithIdentifier:@"myLoginVC"];
    UIView* loginView = loginController.view;
    loginController.usernameField.text = @"dev@monologue.com";
    FBSnapshotVerifyView(loginView, nil);
}

I hope that the setUp and tearDown methods need no introduction? They are run before and after each test in order to prepare the environment for the test. In OCUnit, all methods returning void and starting with the word test are considered tests, hence the name testLogin. What follows is a function body so simple even the most hardcode TDD fans wouldn’t complain (well, maybe we can move the UIStoryboard stuff in the setUp method).

As discussed above, you can use self.recordMode = (NO|YES); to toggle between generating reference images and actual testing.

First of all, we are opening the storyboard to create an instance of the login view controller. Nothing interesting there, but what happens afterward is really important. We are (somewhat artificially) calling the getter for the view controller’s view property. This will force the instance to call the famous loadView and viewDidLoad methods to fully initialize its UI elements. If you fail to do so… well, lets just say you’d be disappointed with your reference images.

With that out of the way, you can do whatever you like in order to make your view controller display some data. Even if you don’t, I guess it would still be a valid and useful test, but I like to fully supply my UI with data in order to increase test coverage (inless I have time to make a separate test for every situation).

Lastly, we do the actual comparison. It’s most of the time the same line:

FBSnapshotVerifyView(yourView, nil);

Common Snapshot testing with FBSnapshotTestCase issues

Table view cells in a storyboard

Going back to our testLogin method, we can conclude that what a typical snapshot test involves is:

  • Creating a view using a storyboard, xib or good old fashioned code
  • Making sure subviews are created, positioned and layed out
  • Filling the view with contents
  • Making a photo (Cheeeeeeese!)

One case where this is not as clear is with table view cells. In the port-storyboard era, many cell are created directly in their table view controller, making it unclear how to get to programatically.

But I’ve figured it out for you. There are several ways to get a reference to the cell you want, and some of them result in a view that’s not properly layed out. So to save you a little time, here’s the code that does it for me:

- (void)testStoryboardCell {
    UIStoryboard* myStoryboard = [UIStoryboard storyboardWithName:@"MyStoryboard" bundle:nil];
    SomeTableViewController* tableController = [myStoryboard instantiateViewControllerWithIdentifier:@"RandomTableViewController"];
    MyTableViewCell* cellISoDesire = [tableController.tableView cellForRowAtIndexPath [NSIndexPath indexPathForItem:0 inSection:0]];
    FBSnapshotVerifyView(cellISoDesire, nil);
}

So what I found to be a good way to test storyboard cells is to initialize the whole controller (I doubt you can get around that) and call cellForRowAtIndexPath: to make it create the cell itself.

Another option would be to make the table dequeue a cell, but that doesn’t work reliably. Particularly, sometimes the layout is incorrect even if you call layoutIfNeeded. I assume that’s a way to force the view to apply its constraint, but I haven’t found it yet.

Getting back to my solution, the drawback is that the table view will start calling it’s data source methods so you need to implement them in your test harness. Well, you might get around that if you are using static cells. But most of the time you’ll need some test data anyway.

Keeping reference images in sync

Creating and modifying reference images is dead simple with FBSnapshotTestCase. But you still need to remember to keep them in sync. If you change your UI even a little, you need new images. And sometimes this is a little subtle. Sometimes, you don’t even know the aesthetics changed. Actually, having a test fail in a case when you change your app’s appearance without knowing is a really good thing. That’s what snapshot tests are for. But it can get annoying if something moves just one pixel but suddently several tests fail. And if you rely on CI, you builds start failing.

So, apart from always remembering to run tests locally before commiting I don’t know what to recommend. It’s hard to admit, but even after being properly setup, Snapshot testing with FBSnapshotTestCase still require some maintainance.

Jenkins… Hudson… or whoever that guy was

A real cool thing to do with FBSnapshotTestCase is to add it to your CI jobs. Every time someone commits, the project is built and snapshot tests run. If they fail, another job starts that prints that “someone’s” resignation letter and denies him/her access to the building. Oh, and also publishes a tweet that they suck. Or is it just my company that does that?

There is one problem that might occur though… Since snapshot testing with FBSnapshotTestCase is for the most part only available for the simulator (because your reference images typically reside on your Mac and the iOS device cannot access them unless you copy them into the bundle), your CI server needs to run the iOS simulator. Typically this is OK, but I personally incountered a problem with our Jenkins.

It would run the tests fine for a while but when you try again in a few hours, the job would fail, saying that the simulator failed to respond in time.
The problem is that the Simulator is a GUI application and services (daemons) have a little trouble starting those. Now, I really struggled fixing this, but in the end, I fould a solution. It turns out you have to change Jenkins from being a daemon to being an Launch Agent. The idea is that a Launch Agent has the permission to start GUI apps.

In order to setup Jenkins as a Launch Agent you can follow this article. Basically you need to move Jenkin’s plist from /Library/LaunchDaemons/ to /Library/LaunchAgents/. And I suspect this is not specific to Jenkins, so you can apply it to whichever CI you’re using.

Testing on multiple devices

What I really find useful about snapshot testing is running the cases for multiple screen sizes. I don’t know about you guys, but I personally mostly use a single device/simulator while developing so I always risk having layout issues on the iPhone 6+ ot 4s for example. To tackle this, I’ve setup a build machine task that runs my snapshot testing with FBSnapshotTestCase on several simulators.

This is actually fairly easy using either xcodebuild or xctool:

xcodebuild -project MyProject.xcodeproj -scheme "SnapshotTests" -destination name="iPhone 5s" -destination name="iPhone 6 Plus" test

What I’m trying to say is… Yes, you can chain destinations to execute on several devices one after another.

Try it out, it’s free!

And just like that, you’ve learned everything I know about snapshot testing with FBSnapshotTestCase. I hope I’ve convinced you it’s not lame to test using images. And I hope that just like every teenage girl and every hipster, you’ll start taking pictures of your UI and showing them to your frinds. You can even apply filters, I guess.

Seriously though, in the past I’ve always considered snapshot testing terribly unefficient and useless. But with a simple framework like FBSnapshotTestCase, it has become a breeze and by far the easiest way to automate UI testing. So stop acting prejudice and try it out!

Unit testing in Swift

If you haven’t realized it already, I’m a fan of test driven developement. I’m still learning, but I’m trying to introduce it into my coding practice more and more. And for that reason, one of the first things I thought about when I learned about Swift was

But wait, how are we going to do unit testing in Swift???

I wasn’t sure if they planned on porting OCUnit from Objective-C. And since you are reading this, I assume you are interested in finding out too. So there you go… they did!

I’m going to keep this post short as there isn’t much to talk about. Apple did a really good job making sure we don’t have to learn another framework – unit testing in Swift is as close to it’s Objective-C equivilent as humanly possible. If you are not familiar with OCUnit, check out the series of posts I wrote about it.

Unit testing in Swift via OCUnit

If you’ve already used OCUnit, very little of your workflow will change. Let’s see what you need to do.

If you are setting up a new project, you will, ofcourse, have a testing target already configured by Xcode. On the other hand, if you are working on an existing one, you will be able to use the target that you already have.

Let’s try to create an new test case class. As always, open the “New file” wizard in Xcode and choose “Test Case Class”.

New File Template

After that, we are presented with some unit test specific options. Choose a name for your new file, leave the superclass as XCTTestCase and in the language field, select “Swift”

New Swift Test Case File

Once you do that, you will be presented with this beautiful template:

import XCTest

class SwiftUnitTest: XCTestCase {

    override func setUp() {
        super.setUp()
        // Put setup code here. This method is called before the invocation of each test method in the class.
    }

    override func tearDown() {
        // Put teardown code here. This method is called after the invocation of each test method in the class.
        super.tearDown()
    }

    func testExample() {
        // This is an example of a functional test case.
        XCTAssert(true, "Pass")
    }

    func testPerformanceExample() {
        // This is an example of a performance test case.
        self.measureBlock() {
        // Put the code you want to measure the time of here.
        }
    }

}

Looks familiar? Well, it should. It has everything an Objective-C unit test had – an XCTestCase superclass, a setUp and tearDown method and actual testing methods. Additionally, all assertion functions that you know have been ported to Swift.

  • XCTAssert
  • XCTAssertEqual
  • XCTAssertEqualObjects
  • XCTAssertEqualWithAccuracy
  • XCTAssertFalse
  • XCTAssertGreaterThan
  • XCTAssertGreaterThanOrEqual
  • XCTAssertLessThan
  • XCTAssertLessThanOrEqual
  • XCTAssertNil
  • XCTAssertNotEqual
  • XCTAssertNotEqualObjects
  • XCTAssertNotEqualWithAccuracy
  • XCTAssertNotNil
  • XCTAssertTrue

So, don’t be intimidated when it comes to unit testing in Swift. Setting it up is really easy if you have previous experience with OCUnit. Most of the frustration I had was related to the language itself – every new begining is hard. But once you come into terms with Swift’s synthax, you should be alright.

Finally, let’s finish this article with a demonstration. You will be able to get a glimpse of the very first unit test I ever wrote in Swift. It’s designed to test an applications’s AppDelegate. I like to keep functionality in it to a minimum (as should you). In fact, the only thing it does is handle opening URLs from other applications and copying the referenced file in the documents directory within the sandbox.

First, let’s look at the old Objective-C code:

@interface DMAppDelegateTests : XCTestCase
{
    DMAppDelegate* appDelegate;
}

@end

@implementation DMAppDelegateTests

- (void)setUp
{
    [super setUp];
    appDelegate = [[DMAppDelegate alloc] init];
    [appDelegate application:[UIApplication sharedApplication]
    didFinishLaunchingWithOptions:nil];
}

- (void)tearDown
{
    appDelegate = nil;
    [super tearDown];
}

- (void)testAbilityToOpenFilesFromOtherApps
{
    DMTestableFileSystemManager* fileSystemManager = [DMTestableFileSystemManager new];
    appDelegate.fileManager = fileSystemManager;
    XCTAssertNoThrow([appDelegate application:[UIApplication sharedApplication]
                                      openURL:[NSURL URLWithString:@"file://path/to/resource"] 
                            sourceApplication:@"test.app"
                                   annotation:nil], @"Should respond to the openURL method");
}

- (void)testAbilityToCopyOpenedFilesInTheSandbox
{
    DMTestableFileSystemManager* fileSystemManager = [DMTestableFileSystemManager new];
    appDelegate.fileManager = fileSystemManager;
    NSURL* openedURL = [NSURL URLWithString:@"file://path/to/resource"];
    [appDelegate application:[UIApplication sharedApplication]
                     openURL:openedURL
          sourceApplication:@"test.app"
                 annotation:nil];
    XCTAssertEqual(fileSystemManager.copiedURL, openedURL, @"The app delegate should ask the file manager to copy the file in the documents dir");
}

- (void)testAbilityToCopyOpenedFilesInTheRightDestination
{
    DMTestableFileSystemManager* fileSystemManager = [DMTestableFileSystemManager new];
    appDelegate.fileManager = fileSystemManager;
    NSURL* openedURL = [NSURL URLWithString:@"file://path/to/resource.epub"];
    [appDelegate application:[UIApplication sharedApplication]
                     openURL:openedURL
           sourceApplication:@"test.app"
                  annotation:nil];
    NSArray* paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
    NSString* documentsDirectory = [paths objectAtIndex:0];
    documentsDirectory = [documentsDirectory stringByAddingPercentEscapesUsingEncoding:NSUTF8StringEncoding];
    NSString* destinationPath = [documentsDirectory stringByAppendingPathComponent:@"resource.epub"];
    XCTAssertEqualObjects(fileSystemManager.destinationURL.absoluteString, destinationPath, @"The app delegate should ask the file manager to copy the file in the documents dir");
}

@end

Several hundred pages worth of Swift’s book and some fiddling around with the synthax and you come up with it’s Swift equivilent:

import XCTest
import UIKit

class DMAppDelegateTests: XCTestCase {

var appDelegate: AppDelegate = AppDelegate();

override func setUp() {
    super.setUp()
    appDelegate.application(UIApplication.sharedApplication(), didFinishLaunchingWithOptions: nil)
}

override func tearDown() {
    super.tearDown()
}

func testAbilityToCopyOpenedFilesInTheSandbox() {
    var fileSystemManager = DMTestableFileSystemManager()
    appDelegate.fileManager = fileSystemManager as DMFileSystemManager;
    let openedURL = NSURL(string: "file://path/to/resource");
    appDelegate.application(UIApplication.sharedApplication(), openURL: openedURL, sourceApplication: "test.app", annotation: nil)
    XCTAssertEqual(fileSystemManager.copiedURL!, openedURL, "The app delegate should ask the file manager to copy the file in the documents dir");
}

func testAbilityToCopyOpenedFilesInTheRightDestination() {
    var fileSystemManager = DMTestableFileSystemManager()
    appDelegate.fileManager = fileSystemManager as DMFileSystemManager;
    let openedURL = NSURL(string: "file://path/to/resource.epub");
    appDelegate.application(UIApplication.sharedApplication(), openURL: openedURL, sourceApplication: "test.app", annotation: nil)
    let documentsDirectoryRaw = NSHomeDirectory() + "Documents";
    let documentsDirectory = documentsDirectoryRaw.stringByAddingPercentEscapesUsingEncoding(NSUTF8StringEncoding)
    let destinationPath = documentsDirectory.stringByAppendingPathComponent("resource.epub")
    XCTAssertEqualObjects(fileSystemManager.destinationURL!.absoluteString, destinationPath, "The app delegate should ask the file manager to copy the file in the documents dir");
}

}

And that’s how you get unit testing in Swift. As always, thanks for reading!

Writing Objective-C unit tests

This post is the third part of the Test Driven Development series, guiding you in the process of integrating test driven development in your iOS projects. Previously, in parts one and two, we covered the mechanics of setting up Xcode with OCUnit, as well as some general concepts of TDD. So now it is time to dig a little bit deeper and discuss how to write meaningful tests.

If you’ve read the previous articles in the series, you should be able to setup Xcode and OCUnit for test driven development. However, if you have tried it, you might have felt a bit lost when it comes to writing an actual unit test. In my experience, TDD requires some practice before you are able to do it without struggling. And that’s what this article is about – it will give you some pointers on how to go about writing Objective-C unit tests – how to approach the problem and what techniques to use.

How to write Objective-C unit tests

Test a single piece of functionality

It is very important that your unit tests check for only one thing. Don’t be tempted to verify several properties at once. For instance, if you are writing a calculator application and want to test your multiplication algorithm, your most important unit test will be the one that proves that you are able to multiply two numbers. But you might also want to try going beyond the bound of the integer data type, or using negative numbers. However, you shouldn’t do that at the same time. These are three distinct pieces of functionality and should be handled separately. Write one test for multiplying simple numbers like 2 and 4 so that you know your algorithm is working in the most straight-forward case and after that you can write another one using negative values, zero and big numbers that go outside the bounds of integer.

But why is this so important? First of all, it results in simpler and easier to read unit tests. Remember that one of the key advantages of TDD is that other developers can read your tests to quickly learn how your code is supposed to be used. Secondly, if you wrote only one unit test in the example above, and it happened to fail for some reason, you would have had a hard time discovering what caused the failure. You would know that the multiplication algorithm failed, but was it the part that handles negative numbers, the checks for going beyond the integer max value or the core functionality? You wouldn’t have had that problem if you had three separate tests.

Write appropriate unit test names

You should take the time to write descriptive names for your test methods. This is helpful when one of them fails – Xcode will print it’s name and if it hints the functionality that is incorrect, it is going to help you find the problem quickly. And also, it makes the code easier to understand.

The name should describe the exact scenario executed in the test so that other developers don’t have to read the method’s body to understand what it does – something like “testCalulatorMultipliesNegativeNumberWithZero”. It will not be very pleasant to look at but it can help a lot.

Write tests from the bottom up

Another great tip I can give you is related to the way you write your actual test method. Once you decide what functionality you are going to verify and have created an empty method, instead of writing your code from the beginning towards the end, try doing the opposite. A unit test has three distinct parts. The first one sets up the environment for the test – it initializes objects, sets values and overall prepares the program for executing the code under test. The second part is the actual invocation to the tested functionality and the third part verifies that the program worked correctly.

So, instead of writing these three steps in sequence, try implementing the error checking one first. Once that’s done, add the first two parts accordingly.

The idea here is that if you think of a sample value to test with, it isn’t always easy to find the right solution for it (after all, that’s why you are writing an application to do it for you). The reverse, on the other hand, is often a lot easier. Pick a solution and figure out what problem it solves.

An example that illustrates this concept is finding a string in text. Looking for all occurrences can be difficult and error-prone thing to do by hand. But if you start with the final result (let’s way three occurrences at indexes 23, 54 and 346) it will be a lot easier to add random data between these indexes to come up with the initial text.

Use mock objects

Mock objects are instances of classes that you made specifically to help in testing. Let’s say you have to test some code that depends on the file system to get it’s job done. Since you shouldn’t rely on the actual file system for your unit test, you need to replace that functionality with your own. Essentially, you need to override all methods that interact with the “real” file system and replace them with hardcoded values. So, when the class, being tested asks for the contents of the file at a given path, instead of actually reading it, your overridden methods return a hardcoded piece of data.

There are two ways you can achieve this. The easiest is to identify all methods that read file system data and create a subclass that overrides them and returns test data. However, there might be many such methods and they might have additional functionality (not only interacting with the file system). Overriding all of them is not always a good idea and ideally, you don’t want to replace much of the production code to avoid something that works in your tests break in production because you have overridden some functionality.

The second (and better) way to override all file system interactions is to not have any in your production class in the first place. Instead of having the class under test interact with the file system directly, have a separate class that does it and let the tested class have an instance of it. Then, you can replace that new class with a mock implementation that doesn’t actually read any files, but rather returns a predefined set of data. Such a scenario is depicted in the sample below:

/**

* unit test

*/

- (void)testBeingAbleToDoSomethingImportant

{

ClassUnderTest* testClass = [ClassUnderTest new];

XCTAssertNoThrow([testClass methodUnderTest], @"methodUnderTest shouldn't throw and exception");

}
/**

* unit test with mock objects

*/

- (void)testBeingAbleToDoSomethingImportant

{

MockClassUnderTest* testClass = [ClassUnderTest new];

testClass.fileInteractionManager = [MockFileManager new];

XCTAssertNoThrow([testClass methodUnderTest], @"methodUnderTest shouldn't throw and exception");

}
/**

* Production classes

*/

@inteface ClassUnderTest : NSObject

{

FileManager* fileManager;

}

- (void)methodUnderTest;

@end

@implementation ClassUnderTest

- (void)methodUnderTest

{

NSData* someFileData = [fileManager readSomeFileFromPath:@"/test/path"];

// do something with the data

}

@end

@interface FileManager : NSObject

- (NSData*)readSomeFileFromPath:(NSString*)path;

@end

@implementation FileManager

- (NSData*)readSomeFileFromPath:(NSString*)path

{

return [NSData dataWithContentOfFile:path];

}
/**

* Mock classes

*/

@inteface MockClassUnderTest : ClassUnderTest

@property(nonatomic, strong) FileManager* fileInteractionManager;

@end

@implementation MockClassUnderTest

- (void)setFileInteractionManager:(FileManager*)fileInteractionManager

{

fileManager = fileInteractionManager;

}

- (FileManager*)fileInteractionManager

{

return fileManager;

}

@end

@interface MockFileManager : FileManager

@end

@implementation MockFileManager

- (NSData*)readSomeFileFromPath:(NSString*)path

{

// return empty data for simplicity.

// A more useful implementation might have a property that

// can be used to set the NSData that will be returned from

// this method

return [NSData data];

}

@end

In the code snippet above, two mock objects are created – MockFileManager and MockClassUnderTest. MockFileManager is used in order to override the “readSomeFileFromPath:” method and return a predefined value from it that does not depend on the real file system on the device. MockClassUnderTest is used in order to replace the standard instance of FileManager with the mock one. It just adds a property that can be used to set the “fileManager” member variable that is used by MockClassUnderTest. You might get away with adding that property in MockClassUnderTest itself, but it is generally not recommended because functionality shouldn’t be added to a production class solely for the purpose of unit testing. If a property should be private for the production code, it needs to stay that way even though access to it is needed for testing.

The second solution might be more complex, but in reality it is the right way to do it. It is an example of how test driven development forces you to write better code. Since it is difficult to test code with a lot of dependancies, you instinctively write code with greater structural (and functional) quality.

So even though it is considerably easier to have all file system interaction code in the classes that need it, you take the time to extract it into another layer of abstraction because it will make it easier to unit test.

Singleton vs. single instance

The singleton is a popular design pattern where a class is only allowed to have one instance. This so called “shared instance” is used throughout the application and every other class has access to it. On the other hand, a single instance refers to a class that is not a singleton (and does not actively enforce that you cannot create several instances of it) but is only instantiated once.

Even though a singleton is more convenient for developers in the sense that they don’t have to worry about getting a reference to the object when they need it (they just call the “sharedInstance” class method and the instance is returned), it takes away a but of flexibility and can lead to some difficulties when using test driven development. Imagine that in the code example above, the file manager class was a singleton (which can be the case with NSFileManager if using the “defaultManager”). In the unit tests, we need to replace that object with a mock one. However, it is not easy because the class directly calls the class method that returns the singleton instance. In Cocoa, we can use the Objective-C runtime in order to change what that class method does and make it exchange the singleton instance with a mock one, but other languages don’t have that. And this is hardly an elegant solution.

That’s why it is recommended to choose single instance, instead of singleton. You can create an object of the class and pass it along to the classes that need it.

I think that’s it for now. It is by no means a complete set of rules to follow when using test driven development but it is something that has helped me a lot. Make sure to come back for the next TDD article. Once again, thanks for reading.

Test Driven Development with OCUnit

This post is the second part of the Test Driven Development series. It will guide you in the process of integrating test driven development with OCUnit in your iOS projects. Last time, in part one, we covered some test driven development basics and now it’s time to actually prepare a project for unit testing.

Xcode and TDD

It is a common misconception that test driven development is not suitable for mobile applications. Far from it! Xcode has unit testing integration since version 2.1. Do you remember Xcode 2.1? I certainly don’t, because I hadn’t started developing for iOS. Neither had most of you, I assume. So nowadays, TDD is just a few clicks away. And that’s exactly today’s topic.

 There are several unit testing platforms available to iOS (and Mac) developers:

  • OCUnit
  • GHUnit
  • CATCH

Disclaimer: I’ve only used OCUnit :)

Let me introduce them to you:

OCUnit:

OCUnit is the single, most popular TDD framework for iOS development. Why? Well… uhm… it’s integrated in Xcode.

GHUnit:

GHUnit is another popular unit testing platform for iOS. I can only pretend to know anything about it other than that it compensates for not being integrated in Xcode by providing a user interface, where it shows all tests that are executed and the results from them. While both OCUnit and GHUnit require a new target to be created in Xcode solely for testing, GHUnit needs you to add it’s own application files along with your testing classes and mock objects, making the setup process a little more complicated. If you are interested, you might consider visiting it’s github page.

CATCH:

CATCH is a framework that is written in C++ and hence, it can be easily used to test your C/C++ code. If that’s important to you, visit it’s github page.

In this article, we’re going to focus on OCUnit because it’s easiest to setup and works best with Xcode. Later on, I might consider writing a separate post about it’s alternatives.

A philosophical question one might ask is “Are a unit testing frameworks unit tested during development and what with?”. The answer I found was “Yes” and “With the same framework”. Please refer to GHUnit’s GHUnit tests and CATCH’s CATCH tests.

Creating a new Xcode project with unit tests

It is now time to show you how to setup your new projects with test driven development in mind. More specifically, I was going to show you multiple screenshots of me essentially clicking an “Include Unit Tests” checkbox like it’s rocket science. But as I went through Xcode’s new project wizard, I realized that there is no such option. Apparently, you cannot opt to exclude unit testing from your new project in the latest version of Xcode. Either way, I’m going to add a few images because an article without pictures seems boring:

The image shows Xcode's new project template, where developers can choose a template for their new application
New Project Wizard – choosing application template

Every tutorial I’ve ever read starts with the single view template. So I’m going to do something radical and use the empty application. Next, in Figure 2, I choose the name for my new project – Dev Monologue – an application that allows my readers to enjoy the blog on the go (though I wouldn’t get too excited that it’s going to hit the AppStore any time soon). Depending on the version of Xcode you use, you might see an “Include Unit Tests” checkbox. In the latest update, it is missing, but I remember that previously it was there. So in reality, those of you who update their macs, should be able to get a TDD-ready project without any extra effort.

 

The image shown the section of Xcode's new project wizard where the project name is chosen and some application-wide parameters are set - Core Data usage and Unit testing
New Project Wizard – choosing project name

After all that, we should be presented with our newly created project. It is almost empty right now, but there are still some goodies, generated for us by Xcode.

The image shown the newly created project opened in Xcode
The new project opened in Xcode

Alongside the usual AppDelegate, Info.plist and project file, you will also notice that in the project navigator there is a group called “Dev MonogolueTests”. That’s where our unit tests and mock objects go, but more on that later on. Also, there is a second target configured in our project file.

If you are unfamiliar with targets, intuitively, they are different ways to build your application. Two targets in a single projects share most of their code and have roughly the same configuration, but have some differences. Targets are commonly used for separating “lite” and “premium” versions of an application, or even the iPhone from the iPad version. For more information, please refer to Apple’s documentation.

So, we have a target called “Dev Monologue” that is used to build the application itself, and another one called “Dev MonologueTests” that deals with unit testing. The first one contains (or will eventually contain) all “production” source files in the project – all the functionality needed for the application we are developing to work as intended. The second target should contain all test files, mock objects (classes created to assist testing) and classes that are subject to testing. Notice that the classes that are going to be tested and not necessarily all classes in the production code. You might decide not to test some modules, though it would be better if you did. For instance, the user interface is famous for being difficult for TDD because it is very subjective – how can you define that a certain component is in the right place or visualizes properly? That being said, there are many properties of your UI that can be unit tested. You can check if a button fires the right even when it receives a touch event, or that a view controller is popped from the navigation stack once the user dismisses it. Of course, you still need manual testing in order to be sure everything works fine. Also, as far as user interfaces are concerned, Xcode provides a handy tool for automating it – UIAutomation. I even wrote an article about it here.

Running Unit tests

Normally, when having several build target, one would choose which one to run. But in this case, it’s a little bit different. You cannot select the testing target from the toolbar. Instead, you run your application normally by selecting “Run” from the menu (or CMD+R) and you start your unit test with Product → Test, or using the CMD+U shortcut.

The image shows the Product menu in Xcode expanded, exposing the Test option that is used for running the unit tests
Starting unit tests in Xcode

Now let’s see what happens when we run our new project:

The image shows the Xcode window just after running all unit test. It displays one failed test in the issue navigator and some comment inline in the source code editor
Unit testing results in Xcode

Our testing fails. That’s because Xcode’s template created a single test for us that just instructs the OCUnit framework to fail, saying that we haven’t implemented any unit tests yet. But that gives me the opportunity to explain how Xcode’s TDD integration works. As you can see, failures in any unit test shows up in the issue navigator, just like compiler errors and warnings do. Which test failed and why is also described in the message. Additionally, most of this information is displayed inline in the source code editor, making it easy to see which line in the code caught the error.

That’s right about it. That’s all you need to know about the OCUnit integration in Xcode in order to start using it. You write code, press CMD+U, see what fails, fix it and repeat until all tests pass.

Anatomy of a unit test

The last section of this article is devoted to the syntax of writing unit tests with the OCUnit framework. It will not teach you to write meaningful tests, but rather, it will explain the mechanics of a test case class.

Since you already know how to setup a test driven development environment with Xcode, it is now time to replace the class Xcode generated with some real unit tests.

All test case classes in OCUnit are derived from XCTestCase. If the test classes that you create are not XCTestCase subclasses, they will not be run. Also, all of your methods have to have no return value (void) and take no parameters:

- (void)nameOfTheTest;

The two rules described above are the only conditions that have to be met in order for your functions to be executed as unit tests. If you wonder how is this possible, the answer is that OCUnit uses Objective-C Runtime to get all XCTestCase subclasses and all of their methods that classify as tests.

Every unit testing class should contain two special functions – setUp and tearDown.

SetUp:

SetUp is called before the beginning of every unit test. It contains code, common to all tests in the class, that performs some initial configurations in order to prepare the environment for the validations that will follow. These include initializing the objects that are going to be tested, setting parameters and variables.

TearDown:

TearDown is called after each unit test in order to reset the environment and clean up. If some network connection have been initiated, for instance, this method might cancel and close the socket to avoid callback from being called during a subsequent test.

Note, that each test is executed independently from all others. Before it, all variables and states are reset in order to avoid side effects from another test to influence the next one.

So what does an actual test method look like? Well, let’s look at what it has to accomplish. First of all, it has to create a setup for testing a particular scenario. Consequently, before anything else, the method has to initialize some variables in accordance to the desired functionality. After that, it needs to execute the code it is trying to verify. And lastly, the test needs to compare the result to a pre-calculated answer in order to determine whether to fail or succeed. This is done via assert methods. If you are unfamiliar with assert methods, they are functions that check a condition and if it is not true, cause the program to terminate. OCUnit provides the following assertion methods:

  • XCTFail(format…)
  • XCTAssertNil(a1, format…)
  • XCTAssertNotNil(a1, format…)
  • XCTAssert(expression, format…)
  • XCTAssertTrue(expression, format…)
  • XCTAssertFalse(expression, format…)
  • XCTAssertEqualObjects(a1, a2, format…)
  • XCTAssertNotEqualObjects(a1, a2, format…)
  • XCTAssertEqual(a1, a2, format…)
  • XCTAssertNotEqual(a1, a2, format…)
  • XCTAssertEqualWithAccuracy(a1, a2, accuracy, format…)
  • XCTAssertNotEqualWithAccuracy(a1, a2, accuracy, format…)
  • XCTAssertThrows(expression, format…)
  • XCTAssertThrowsSpecific(expression, specificException, format…)
  • XCTAssertThrowsSpecificNamed(expression, specificException, exception_name, format…)
  • XCTAssertNoThrow(expression, format…)
  • XCTAssertNoThrowSpecific(expression, specificException, format…)
  • XCTAssertNoThrowSpecificNamed(expression, specificException, exception_name, format…)

They should be fairly self explanatory so I’m not going to go into details on each of them. I’ll just mention that all of the method above allow programmers to provide a human readable message that will be displayed whenever the assertion fails. The syntax is the same as with NSLog. In fact, we already saw the XCTFail method in action – it was a part of Xcode’s template:


XCTFail(@"No implementation for \"%s\"", __PRETTY_FUNCTION__);

It caused our tests to fail and show the following explanation:

No implementation for “-[Dev_MonologueTests testExample]”

Putting everything in this section together, we can write a quick sample of how a unit test would look like using OCUnit:

@implementation StringAppendTests

- (void)setUp
{
    [super setUp];
    firstString = @"one";
}

- (void)tearDown
{
    firstString = nil;
    [super tearDown];
}

- (void)testNSStringAbleToAppendString
{
	NSString* secondString = @" two";
	NSString* appendString = [firstString stringByAppendingString:secondString];
	XCTAssertEqualObjects(appendString, @"one two", @"NSString is not able to append strings");
}

@end

This example creates a new class called “StringAppendTests”. It is always good to name your test case classes in a way that hints what functionality it will check – in this sample – appending strings. Of course, there are many ways to append two strings and this class would probably check all of them – a test case class is by no means limited to a single test. The two special methods, setUp and tearDown, are really simple – they make sure the “firstString” variable is initialized. It is not mandatory to set the pointer to nil in the tearDown function – it will work fine without it, but it is included in the example as a reminder that you have to clean up after each test. Finally, let’s look at the actual unit test. As already discussed, it has to have no return value and take no parameters. It is a good practice to have a potentially very long, but descriptive name for it, so that readers can easily understand what kind of scenario it is checking. The method’s body holds no surprises either. It first initializes a second string variable, which is then appended to the first one. The last line check whether the result from the appending is the expected string “one two”.

Conclusion

Congratulations! You now know how to use the OCUnit framework in Xcode. As you can probably see, it is really simple and almost everything is already configured by the new project wizard. Introducing test driven development to existing project can be a little more difficult and I will probably cover that in a separate post.

So that’s it for the second part of the Test Driven Development series. I hope you found it useful and you will stay tuned for the third part where we will deep deeper into OCUnit and discussed some more advanced unit testing concepts.

Up next:

Once again, thanks for reading!

Introduction to Test Driven iOS development

This post will be the first part of a TDD series. It will guide you in the process of integrating test driven development in your projects. In this first article, we will be discussing what test driven iOS development is and how it can be used when developing for Mac or iPhone/iPad.

What is Test-Driven Development (or TDD)?

Test-Driven Development is a section of extreme programming where programmers write tests for the code that they are developing. These tests should prove that the code is working correctly in a given circumstance. In the purest form of TDD, these test are written before the real (production) code. This forces developers to first think about how their code is going to be used before they even attempt to write it. It ensures that a new functionality is written according to the requirements, and not according to the details of implementation. Also, it ensures that functionality that is not going to be used, is not added in the first place[1]. But why is this important? What if it’s not used – someday it might be? Well, that’s exactly the problem. Since this functionality is not used at first, it is probably not tested well enough and might not be working properly. And when, later on, someone decides to use it, he/she will run into problems. This problem is often referred to as YAGNI (Ya Ain’t Gonna Need It).

Another core concept in test driven development is the so called “Red, Green, Refactor”. It refers to the process you follow while you are using TDD.

Red:

You write a new test that proves that the next functionality you plan on implementing is working. At this point, if you run your code, this test will fail, because you don’t have that functionality yet. Nevertheless, you should do it to prove that the test does not pass with the current code. If you wonder, this stage is called “Red” because when you run the project, your IDE will flash red due to the failing test.

Green:

After the first stage is complete, you are ready to write the code that actually implements the feature. It is important to write just the bare minimum that will make the test pass. You don’t need to provide a complete implementation. For example, lets say you have to implement a method that multiplies two numbers. You are testing this new method by calling it with arguments 2 and 4. Using the “Red, Green, Refactor” technique, it is perfectly fine to implement a function that just returns 8 (return 8;). This is the absolute minimum needed in order for the test to pass. It might seem insane at first, but that’s how test driven development works. The idea here is that after this test, you will create another one (lets say that it will test if 0 * 13 = 0 to be sure that our implementation handles zero correctly) and at that point our initial implementation of the method will no longer be able to pass the test and we will be forced to replace it with a real one. This concept introduces incremental changes in your code. With every new test that you add, another small change is introduced in the production code.

Refactor:

You might wonder – “If I make all those incremental and minimalistic changes to my code, without ever thinking about the big picture, wont my code be doomed to have poor structural properties?”. That’s what I thought anyway. Well, it is true that thinking only locally might result in a code that is not all that good. But that’s where the “Refactor” stage come in handy. Once you complete the “Red” and “Green” stages, it is time to think about what you’ve created, see where you can improve. Maybe you realized that it would be better to extract something into a method or there is a better way to implement something. If you do, you are free to makes whatever refactoring you want (as long as it is covered by the tests). And since you have all those tests that verify that your code is still working, refactoring should be safe. If you introduced some regressions, testing would fail and you will be able to fix your mistake before it causes any harm.

This is the most powerful benefit of using test driven development – you no longer have to fear that if you change a piece of code, something else will break. Of course, it does not guarantee a bug free software. In fact, code is only as good as its tests. If your tests have good coverage and check all possible error scenarios, you might be relatively confident that the code is working properly. However, you certainly wont have the time and resources to write tests for everything. There is a trade-off between how much you want to ensure your application works and how much are you willing to “pay” for it. It all depends on what your project is and whatever works for you. Intuitively, you should always write more tests for code that is critical to your application and is vulnerable to regressions and fewer tests in places that you feel confident.

If my calculations are correct, you should be starting to wonder if test driven development is actually worth it. It certainly seems that it slows you down a lot. And you end up writing much more code. Well, it is true, and I cannot be sure how it compares to conventional programming techniques. All I know is that the experts say it does pay off in the end. It’s supposed to save 20% (as far as I remember) development time when considering time for bug fixes. As you should know, the sooner a bug is discovered, the cheaper it is to fix, and since you don’t have to do as much bug fixing upon completion, if a project is developed using TDD, the overall process becomes more efficient. My personal opinion is that, if quality is important to you as a developer, and there is not much pressure to complete a project, test driven development is always worthwhile.

Test driven development on existing projects

Lastly, lets discuss test driven development on existing projects. Not everybody has the luxury of starting a project from scratch. In such case, it looks as if TDD is out of the question. It maybe is, in it’s purest form, but you can still start testing your code. The first way to do it that comes to mind is to start writing tests for all currently available code. Well, that’s generally not a good idea, because, first of all, it violates the principal of writing tests first. But the more important consideration is that you lose the perspective that TDD gives you while you write. As we discussed previously, TDD forces you to think about requirements, rather than the details of implementation while you are writing a new feature. And if you start writing a bunch of test for existing code, you will be designing them around your implementation, whereas it should be the other way around. What good is it to start writing test that are designed so that they pass. And last, but certainly not least, writing tests for all existing code in the project would be terribly boring. :)

So what is the right way to start using test driven development in an existing project? The answer is that you will have to start from somewhere – just write tests for all new features and fixes that you develop in the future. That would mean that some parts will use TDD and others wont, but that’s the price you will have to pay.

Unfortunately, even like that you will most likely run into problems unless your existing project is really well written. Test driven development requires code with little dependancies – in order for a unit test to check only one functionality, the production code that implements it will have to do that (and only that) – hence the Single Responsibility Principle has to be in place. This is often not the case.

In order to fix that, you will have to break the tight coupling and start testing the new code. A common technique is to think if a place where the parts of your application can be separated (for instance, the line between the controller and the view is a Model-View-Controller environment like Cocoa). Think if the classes that sit on either side of that line and revise their functionality for something that does not belong and can be moved. Do this until you are satisfied that everything is in it’s right place. If you have done this properly, this would be the place you could start using TDD.

I think that’s enough for the first part of the Test-Driven-Development series. We got acquainted with TDD and we covered some core principles and techniques so that you know what you are getting into. This description was by no means complete – there is still a lot to learn. If you are interested, I will probably add some links to additional resources.

Next time we will be covering several frameworks for test driven development in iOS that will guide you to some specifics in introducing TDD for your Xcode projects.

Thanks for reading!

[1.] How many times have you written something because you thought it could be useful, but when integrating your code, discovered that you don’t actually need it? TDD makes sure you never end up in a situation like this, because you already know how you’re going to be using your code.

References

  1. http://www.sunetos.com/items/2011/10/24/tdd-ios-part-1/
  2. iOS Test driven development — Graham Lee