How to Run a Great iPhone Beta Test
The App Store has over 100,000 apps available for purchase.
That’s a big number.
I can’t tell you the secret to success in the App Store. I haven’t discovered any magic formulas. The one thing I can tell you with 100% certainty is this: if you ship a product with bugs that genuinely piss off your customers or, worse, make them unable to use some portion of your hard work, the road toward success just got much longer and a lot more miserable. Even with “prioritized review” requests, once your buggy app is on the store, the review queue means you could be waiting a long time before your customers can enjoy your bug fix.
You will not sleep well at night.
The good news is that this problem has been solved a thousand times over. A thorough beta test can save you from this terrible outcome and the sleepless nights that come with it. With a bit of diligence, you can ship an app with awesome stability to match your great ideas.
I’ll share how I run my betas. This is about process — the technical side of Ad Hoc distribution is thoroughly documented by Apple and others. The tools I use are simple and free. They work. If you have suggestions for better ones, I want to know about them before my next test! Please share in the comments.
Gather Your Testers
You’re going to need testers. If you use a volunteer testing pool, figure that 80% of testing will be done by 20% of your volunteers. That means to have any hope of gathering meaningful feedback, you’ll need at least 10 volunteers. 15 or 20 would be the ideal minimum.
Next, you’ll need to start early — but not so early that your testers’ enthusiasm cools by the time you ship your first beta build. Try to get the word out between three to six weeks before you’re ready to test. If you have a particularly strong network, you may need less time. If you’re relying more on the kindness of strangers, though, make sure to give yourself plenty of lead time. Use your website, your blog and your Twitter to solicit volunteers but don’t be a pest.
Most importantly: if you have contact with existing customers from previous versions or perhaps other products entirely, send a quick, polite email explaining what you’re going to test and encouraging them to join up. Customers who have emailed you in the past with feedback or feature requests may be especially keen on seeing your latest work.
Understand Your Testing Pool
You need to know things about your testers. What kind of hardware will they be using? What version of iPhone OS? What’s their geographic region? (International testers are crucial for finding your screwups with localized formatting and other geographic issues.) Most importantly, you’re going to need to collect their UUID.
I use Forms for Google Docs to collect most of my feedback. It’s an easy-to-configure way to store structured data. Since you can designate required fields, you can ensure your testers don’t forget to include anything crucial when reporting issues. Best of all, you can embed the forms into existing web pages, allowing you to include them alongside testing guidance, version history or other information that might be useful.
Here’s the form I use to collect beta tester info:
It’s not short. The other benefit to using a form is to ensure you only get testers who are reasonably skilled at providing detailed information. Everyone who makes it past this filter has a decent shot at actually telling you everything you need to know later in the beta.
Betas Worth Testing
If your app doesn’t look somewhat close to the final product, don’t distribute it for testing. If you’re relying on a volunteer testing pool, you want them eager to use your stuff. If your overall UI is incomplete, you’re not ready to test. You’ve got a limited amount of time and attention with your testers. Don’t squander it on early builds you could easily test yourself. The closer you think your app is to shipping, the more useful your testers will be to you.
You do yourself no favors if the last bit of functionality you add affects previously-solid features that now need a whole new round of testing.
Distributing Your Binaries
So you’ve toiled away and you’ve finally got an app you feel is ready to be tested.
Don’t email the binary.
Depending on the size of your app, the attachment may not be properly delivered to the recipient, especially if they’re using Exchange-hosted email. Instead, host your app and the necessary provisioning profile on the web.
I also recommend a simple checkout system. When you’ve got a new build ready for testing, send your testers a link to a one-question form that asks for their email address. On the form confirmation screen, you can provide a link to the binary. This lets you get an idea of who is participating. It also lets you answer an important question: am I getting zero feedback because there are no bugs or because there are no active testers? With a checkout process, you’ll know for sure.
Use clear versioning with every distributed build. You need to be certain you and your users are talking about the same build when they give you feedback. Beta phases, version numbers, dates, whatever you need to make things clear for everyone involved. Make sure that somewhere in your app, your code pulls the version number out of your app’s Info.plist file and displays it in the UI. Here’s some code to get that string:
NSString *currentVersion = [[[NSBundle mainBundle] infoDictionary] objectForKey:@"CFBundleVersion"];
Update your Info.plist’s version number religiously before distribution. Even better, update it automatically.
Communication
You need to communicate with your testing pool. Keeping them regularly updated, without being spammy, is crucial to keeping them excited about what’s going on.
When you send out a new build, explain what’s new. Give specific testing guidance so that your testers know where they should focus their efforts. New method of rendering cells you strongly suspect has a bug or two? Share your hunches. Built a new import subsystem you’re suspicious might break with legacy data? Provide an example data set or encourage existing users of the old version to give it a whirl. If you’re manipulating data that lives only on the user’s device, tell them to back up their iPhone before running your beta. And tell them why. Don’t be a douche on this: never take for granted the importance of your testers’ application data, especially if you’re lucky enough to have your existing customers helping you. Yes, we all know, betas are risky. Still, the iPhone isn’t the desktop and if you’re testing an upgrade to an existing app, your customers may have data they created before accepting the risks of a beta.
If it has been more than seven to ten days since your last build, send a quick update to the pool. Call out awesome testers and let them know they’re making a difference. Share the score of resolved issues you were able to fix thanks to the pool’s feedback. Most of all, thank them. Their efforts are helping you to not look like an idiot.
Sandboxing Helps
If you’re making an all-new app, you can skip this section. If you’re testing an upgrade to an existing app, read on.
Your early beta builds should use a different app identifier than your shipping app. This way, your beta will be installed as a distinct application alongside your currently shipping version on the user’s device, instead of overwriting it. This lets you ensure that no shenanigans happen to legacy data if you’ve got customers from previous versions in your testing pool. Toward the end of your beta, after much internal testing, you can start distributing builds that overwrite previous shipping versions. You’ll want to ensure that the app properly imports legacy data and preferences and, of course, ensure nothing otherwise disastrous happens when iTunes distributes your updated app to existing customers with existing data.
Gather Useful Feedback
You need to know things about how your application is failing. The forms I use are inspired heavily by what I saw once upon a time during the iPhone OS 2.0 Enterprise AppleSeed beta test. I think it covers the basics. Of course, don’t be afraid to add other questions specific to your application’s circumstances. For bugs, measure how well you can reproduce the bug and its overall severity while capturing the reproduction steps. For feature or enhancement requests, measure the tester’s impression of the request’s importance along with what they’re asking for.
However you gather your feedback, ensure that every single message you send to your testing pool includes brief instructions on how your testers should report their bugs. Never make them hunt around for it.
Ship
You haven’t added or changed anything for a couple of builds. You’ve fixed plenty of bugs and no one has reported regressions. You’ve sent emails to your top-performing testers and they’re giving you a thumbs-up.
Give it one last test yourself. Thirty minutes, minimum, actually doing things your users are likely to do. Is it kosher?
Then ship it.
Then, while you wait for approval from Apple, test it some more, informally, yourself. If somehow a show-stopping bug pops up while you await approval, you can reject your binary and submit a fix before it ruins anyone’s day.
Add Yours
YOU