Testing with TestFlight

Testing an application is indispensable if you plan to deliver a robust and reliable product to your customers. This article will take a closer look at beta testing iOS applications with TestFlight, a free web service that makes it easy to distribute Ad Hoc builds and monitor beta testing usage.

Introduction

Testing software isn’t limited to beta testing. There are several techniques to test software, such as unit testing, integration testing, stress testing, etc. Each of these methodologies has its benefit and place in the development cycle. Beta testing is a process in which a pre-release or beta version is distributed to a limited audience that is not part of the development team.

In the early days of iOS development, it wasn’t trivial to distribute test builds, gather feedback, and collect crash reports. However, in recent years a handful of services have emerged that make beta testing not only easier, but trivial. In this article, I will discuss one such service: TestFlight. TestFlight allows developers to distribute test builds over-the-air. Fiddling with .ipa files and provisioning profiles is a thing of the past if you decide to team up with TestFlight.

The ease of Ad Hoc distribution is not the only advantage of TestFlight. TestFlight also offers an SDK that gives you a number of great features with surprisingly little overhead. With the SDK installed and configured, crash reports are sent to TestFlight and automatically symbolicated. Another great feature of the TestFlight SDK is checkpoints. You can set checkpoints at specific locations in your application to see if a particular feature is actually used. Checkpoints tie in neatly with sessions. Whenever a user opens the application, a session is automatically started. The TestFlight dashboard shows how long a session lasts and which checkpoints the tester passed during a session. The list of features doesn’t stop there. A few other useful features include: in-app feedback, in-app updates, and remote logging.

Step 1: Getting Started

To follow the steps in this tutorial, you will need a TestFlight account. Head over to the TestFlight website and sign up for a free account. After signing in to your TestFlight account for the first time, you are asked to create a team. What are teams? A team is just a categorization for grouping builds and testers. Chances are that you work on several applications for different clients or projects. A team allows you to group the builds and testers for each application or client easily. In other words, it is a convenient way to keep the builds and testers of different projects separated and organized.

The next step is to upload a test build to TestFlight. However, before we do that, we need to create an application that is properly set up and configured for TestFlight. This includes integrating the TestFlight SDK to take advantage of the features I described earlier.

Of course, TestFlight really shines if you have a group of committed testers (preferably people that are not part of the development team). Adding testers to TestFlight is as simple as sending an invitation. With TestFlight, it is no longer cumbersome to obtain a device’s UDID as you will see a bit later. I explain how to invite beta testers a bit later in this tutorial.

Step 2: Create a New Project

The application that we are about to build will be simple. The primary goal of this tutorial is show you how to get up to speed with TestFlight and not so much building a feature rich application. The features of the application are simple, (1) implement TestFlight checkpoints, (2) ask the user for feedback while using the application, and (3) crashing the application and let TestFlight collect the crash report.

Create a new project in Xcode by selecting the Single View Application template from the list of templates (figure 3). Name your application Take-Off, enter a company identifier, set iPhone for the device family, and check Use Automatic Reference Counting. Make sure to uncheck the remaining checkboxes for this project. Tell Xcode where you want to save your project and hit the Create button (figure 4).

Step 3: Add the TestFlight SDK

Start by downloading the latest stable release of the TestFlight SDK (1.1 at the time of writing). Extract the archive and add libTestFlight.a and TestFlight.h, located in the TestFlightx.x folder, to your project. Make sure to copy both files to your project by checking the checkbox labeled Copy items into destination group’s folder (if needed) and don’t forget to add both files to the Take-Off target (figure 5). To keep everything organized, place libTestFlight.a and TestFlight.h in a separate group named TestFlight.

A few more steps are necessary to finalize the integration with TestFlight. Select your project in the Project Navigator and click the Take-Off target in the list of targets. Select the Build Phases tab at the top and open the Link Binary With Libraries drawer. If all went well, libTestFlight.a should be present in the list of libraries. Drag libTestFlight.a into the list of linked libraries if it isn’t present in the list (figure 6).

TestFlight also makes use of the libz library to do some of its work so we need to link the project against this library as well. Click the plus button at the bottom of the list of libraries, search for libz.dylib, and add it to the list of linked libraries.

The next step is optional, but recommended if you plan to use TestFlight throughout your application. Instead of importing TestFlight.h in every file that makes use of the TestFlight SDK, it is more convenient to add it to the project’s Prefix.pch file. Take a look at the complete Prefix.pch file below for clarification.

//
// Prefix header for all source files of the 'Take-Off' target in the 'Take-Off' project
//
#import <Availability.h>
#ifndef __IPHONE_4_0
#warning "This project uses features only available in iOS SDK 4.0 and later."
#endif
#ifdef __OBJC__
    #import <UIKit/UIKit.h>
    #import <Foundation/Foundation.h>
    #import "TestFlight.h"
#endif

TestFlight installs an uncaught exception handler to report crashes and collect crash reports. If you want to make use of this feature then it is recommended to slightly modify the build settings of your project. Select your project from the Project Navigator and choose the Take-Off target from the list of targets. Select the Build Settings tab and scroll to the Deployment settings (figure 8). Three deployment settings need to be set to NO.

1. Deployment Postprocessing
2. Strip Debug Symbols During Copy
3. Strip Linked Product

Settings in bold indicate that the default value is overridden. You can revert any changes you made by selecting a bold setting and hitting backspace on your keyboard. Make sure that the effective value of the build setting is set to Combined (figure 9).

Step 4: Setup TestFlight

TestFlight doesn’t do anything in your application just yet. To make use of its features, we need to initialize TestFlight when the application launches. The ideal place for setting up TestFlight is in the application delegate’s application:didFinishLaunchingWithOptions: method. Setting up TestFlight is surprisingly easy. All we need to do is call takeOff: on the TestFlight class and pass the team token of the team we set up earlier in this tutorial.

To find your team token, head over to TestFlight’s Dashboard, select the correct team from the dropdown menu at the top right, and choose Edit Info from that same menu. Copy the team token and pass it as the parameter in the takeOff: method.

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {
    // Initialize View Controller
    self.viewController = [[MTViewController alloc] initWithNibName:@"MTViewController" bundle:nil];
    // Initialize Window
    self.window = [[UIWindow alloc] initWithFrame:[[UIScreen mainScreen] bounds]];
    [self.window setRootViewController:self.viewController];
    [self.window makeKeyAndVisible];
    // Initialize TestFlight
    [TestFlight takeOff:@"TEAM_TOKEN_GOES_HERE"];
    return YES;
}

Step 5: Upload a Build

TestFlight is now set up, but we still need to upload a build to TestFlight. The following steps are no different than the steps you would normally take to prepare a test or ad hoc build for an application. I have listed the necessary steps below.

1. Create Distribution Certificate in iOS Provisioning Portal
2. Create App ID in iOS Provisioning Portal
3. Add Devices in iOS Provisioning Portal
4. Create Provisioning Profile in iOS Provisioning Portal
5. Create Ad Hoc Configuration in Xcode
6. Create Ad Hoc Scheme in Xcode
7. Archive Build for Ad Hoc Distribution in Xcode
8. Upload Ad Hoc Build to TestFlight
9. Distribute Ad Hoc Build to Testers in TestFlight

This might seem like a lot of work, but most of these steps only need to be done once for a new application. In addition, much of this can be automated. In addition to TestFlight’s SDK, TestFlight also has an upload API that allows developers to automatically upload builds to TestFlight. I won’t cover the upload API in this tutorial as this is a more advanced topic.

Since you are reading this tutorial, I will assume that you are already familiar with beta testing and the steps involved to prepare a test build for ad hoc distribution. In this article, I will limit myself to the steps that involve TestFlight.

When distributing builds using TestFlight, it is important to properly version your builds. By doing so, keeping track of different test builds will become much easier.

Before uploading your application, make sure to set the version number of your application to 0.1.0 to indicate that this is a a pre-release version. Take a look at this question on Stack Overflow for more information about version numbers.

To manually upload a build to TestFlight, click the second button from the right at the top of the TestFlight Dashboard.

Adding a new build is as easy as dragging the .ipa file into the appropriate field, adding a short description, also known as release notes, and clicking the Upload button. Release notes are much more useful than most people think. Release notes should contain information about the changes made to the test build, but they should also contain known bugs (if necessary) and potential workarounds.

After uploading a build of your application, you are taken to the Permissions view of your new test build. The permissions of a build determine who has access to the new test build, that is, who can install the test build on their device(s). For example, if you want to test a critical build only internally and prevent external testers from accessing the build then you can restrict the permissions of that build to only include members of your development team.

To make the distribution of test builds easier, TestFlight has a feature appropriately named distribution lists. A distribution list is a list or group of people within a TestFlight team. Instead of manually selecting members of a TestFlight team every time you upload a new build, you tell TestFlight which distribution lists have access to the new build.

Step 6: Crash Reports, Checkpoints, and Feedback

One of the best features of TestFlight is the ability to collect and automatically symbolicate crash reports. Implementing checkpoints and asking for user feedback is easy as well. Let’s modify the project to see how all this works.

Open your Xcode project and head over to the view controller’s implementation file (MTViewController.m). In the viewDidLoad method, create three buttons as shown below. The code shouldn’t be difficult to grasp.

- (void)viewDidLoad {
    [super viewDidLoad];
    // Create Crash Button
    UIButton *crashButton = [[UIButton alloc] initWithFrame:CGRectMake(20.0, 20.0, 280.0, 44.0)];
    [crashButton setTitle:@"Crash" forState:UIControlStateNormal];
    [crashButton setBackgroundColor:[UIColor blueColor]];
    [crashButton addTarget:self action:@selector(crash:) forControlEvents:UIControlEventTouchUpInside];
    [self.view addSubview:crashButton];
    // Create Checkpoint Button
    UIButton *checkpointButton = [[UIButton alloc] initWithFrame:CGRectMake(20.0, 72.0, 280.0, 44.0)];
    [checkpointButton setTitle:@"Checkpoint" forState:UIControlStateNormal];
    [checkpointButton setBackgroundColor:[UIColor blueColor]];
    [checkpointButton addTarget:self action:@selector(checkpoint:) forControlEvents:UIControlEventTouchUpInside];
    [self.view addSubview:checkpointButton];
    // Create Feedback Button
    UIButton *feedbackButton = [[UIButton alloc] initWithFrame:CGRectMake(20.0, 124.0, 280.0, 44.0)];
    [feedbackButton setTitle:@"Feedback" forState:UIControlStateNormal];
    [feedbackButton setBackgroundColor:[UIColor blueColor]];
    [feedbackButton addTarget:self action:@selector(feedback:) forControlEvents:UIControlEventTouchUpInside];
    [self.view addSubview:feedbackButton];
}

The idea is simple. The application should crash when the user taps the first button. Crashing an application is easy. Right? Take a look at the implementation of the crash: method to see how it is implemented. We create an array with one element and then ask for the second object in the array. This throws an NSRangeException since there is only one element in the array.

- (void)crash:(id)sender {
    NSArray *array = @[@"one"];
    NSLog(@"%@", [array objectAtIndex:1]);
}

The implementation of the checkpoint: method is surprisingly easy thanks to the TestFlight SDK. As I mentioned earlier, checkpoints are a means to track if certain features of your application are used by your testers. In other words, checkpoints tell you when a user has done something that is of interest to you. As I said, checkpoints tell you (among other things) which features are used and, even more important, which features are not. Some features are difficult to find even though this might not be obvious to the developer.

- (void)checkpoint:(id)sender {
    [TestFlight passCheckpoint:@"User did click checkpoint button."];
}

There are various ways to collect feedback from your testers. The easiest way to collect feedback, however, is to use TestFlight’s feedback integration. Take a look at the implementation of the feedback: method to see how this works. When the user taps the feedback button, a modal view shows up and lets the user enter feedback (figure 13).

- (void)feedback:(id)sender {
    [TestFlight openFeedbackView];
}

After adding these changes to your application, update the version number of your application to 0.2.0 and archive the project. It is good practice to always clean your project before you prepare a build for distribution, for the App Store as well as for ad hoc distribution. Upload the new .ipa file to TestFlight, set the appropriate permissions, and update the installation on your device with the new build by visiting the TestFlight dashboard on your device. If you followed the steps, you should see the three buttons and tapping each button will trigger the functionality in the application.

TestFlight sends information to the TestFlight servers whenever it can, that is, if a network connection is available and the operating system doesn’t kill the application before it is finished sending the data to the TestFlight servers. This means that TestFlight is a great tool to collect live data from your testers. You can try this yourself by tapping the three buttons in your application and taking a look at the TestFlight dashboard a few minutes later.

TestFlight shows you which testers installed the update and on what device. It shows you the number of sessions, which checkpoints they passed, and how many crashes occurred. As I mentioned earlier, the crash reports are automatically symbolicated, which is one of the features I love most.

It is also possible to explore individual sessions by clicking the sessions tab on the left (figure 14), selecting a user from the list, and clicking on one of the sessions. This gives you a detailed outline of the session of the respective user (figure 15).

Step 7: Adding Additional Testers

Beta testing is only useful if you can rely on a group of committed testers who truly want to put your application through its paces. Adding testers to TestFlight can be done in two ways. (1) Open the TestFlight dashboard of the team to which you’d like to add a new tester. Click the button with the tiny plus sign in the top right corner and fill out the form. It is recommended that the user clicks the accept link on the test device. Even though this isn’t strictly necessary, it makes the process much easier, because the device the user is using will be automatically added to their account as a test device.

(2) A second option to add testers, is to use a recruitment URL. This is a form that allows anyone to sign up as a tester. This makes it easier if you have a fairly large group of testers you’d like to add to TestFlight.

More?

A few months ago, TestFlight was acquired by Burstly and this has resulted in the creation of TestFlight Live. TestFlight Live is another addition to the TestFlight platform and it gives developers the means to not only use TestFlight for development and testing, but also when the application is live in the App Store. You can read more about it on TestFlight’s blog.

Conclusion

Even though the idea behind TestFlight is simple, over-the-air distribution of beta versions, TestFlight has a lot more to offer. After having used TestFlight for a few weeks, you will notice that the team behind TestFlight has done a great job in terms of which features to include and how all the different pieces fit together.

There are many more features that I did not discuss in this article so I encourage you to visit TestFlight’s website and browse the outstanding documentation. TestFlight is still growing rapidly and I am curious to see how it continues to evolve and improve.