How to Run a Great iPhone Beta Test
The App Store has over 100,000 apps available for purchase.
That’s a big number.
I can’t tell you the secret to success in the App Store. I haven’t discovered any magic formulas. The one thing I can tell you with 100% certainty is this: if you ship a product with bugs that genuinely piss off your customers or, worse, make them unable to use some portion of your hard work, the road toward success just got much longer and a lot more miserable. Even with “prioritized review” requests, once your buggy app is on the store, the review queue means you could be waiting a long time before your customers can enjoy your bug fix.
You will not sleep well at night.
The good news is that this problem has been solved a thousand times over. A thorough beta test can save you from this terrible outcome and the sleepless nights that come with it. With a bit of diligence, you can ship an app with awesome stability to match your great ideas.
I’ll share how I run my betas. This is about process — the technical side of Ad Hoc distribution is thoroughly documented by Apple and others. The tools I use are simple and free. They work. If you have suggestions for better ones, I want to know about them before my next test! Please share in the comments.
Gather Your Testers
You’re going to need testers. If you use a volunteer testing pool, figure that 80% of testing will be done by 20% of your volunteers. That means to have any hope of gathering meaningful feedback, you’ll need at least 10 volunteers. 15 or 20 would be the ideal minimum.
Next, you’ll need to start early — but not so early that your testers’ enthusiasm cools by the time you ship your first beta build. Try to get the word out between three to six weeks before you’re ready to test. If you have a particularly strong network, you may need less time. If you’re relying more on the kindness of strangers, though, make sure to give yourself plenty of lead time. Use your website, your blog and your Twitter to solicit volunteers but don’t be a pest.
Most importantly: if you have contact with existing customers from previous versions or perhaps other products entirely, send a quick, polite email explaining what you’re going to test and encouraging them to join up. Customers who have emailed you in the past with feedback or feature requests may be especially keen on seeing your latest work.
Understand Your Testing Pool
You need to know things about your testers. What kind of hardware will they be using? What version of iPhone OS? What’s their geographic region? (International testers are crucial for finding your screwups with localized formatting and other geographic issues.) Most importantly, you’re going to need to collect their UUID.
I use Forms for Google Docs to collect most of my feedback. It’s an easy-to-configure way to store structured data. Since you can designate required fields, you can ensure your testers don’t forget to include anything crucial when reporting issues. Best of all, you can embed the forms into existing web pages, allowing you to include them alongside testing guidance, version history or other information that might be useful.
Here’s the form I use to collect beta tester info:
It’s not short. The other benefit to using a form is to ensure you only get testers who are reasonably skilled at providing detailed information. Everyone who makes it past this filter has a decent shot at actually telling you everything you need to know later in the beta.
Betas Worth Testing
If your app doesn’t look somewhat close to the final product, don’t distribute it for testing. If you’re relying on a volunteer testing pool, you want them eager to use your stuff. If your overall UI is incomplete, you’re not ready to test. You’ve got a limited amount of time and attention with your testers. Don’t squander it on early builds you could easily test yourself. The closer you think your app is to shipping, the more useful your testers will be to you.
You do yourself no favors if the last bit of functionality you add affects previously-solid features that now need a whole new round of testing.
Distributing Your Binaries
So you’ve toiled away and you’ve finally got an app you feel is ready to be tested.
Don’t email the binary.
Depending on the size of your app, the attachment may not be properly delivered to the recipient, especially if they’re using Exchange-hosted email. Instead, host your app and the necessary provisioning profile on the web.
I also recommend a simple checkout system. When you’ve got a new build ready for testing, send your testers a link to a one-question form that asks for their email address. On the form confirmation screen, you can provide a link to the binary. This lets you get an idea of who is participating. It also lets you answer an important question: am I getting zero feedback because there are no bugs or because there are no active testers? With a checkout process, you’ll know for sure.
Use clear versioning with every distributed build. You need to be certain you and your users are talking about the same build when they give you feedback. Beta phases, version numbers, dates, whatever you need to make things clear for everyone involved. Make sure that somewhere in your app, your code pulls the version number out of your app’s Info.plist file and displays it in the UI. Here’s some code to get that string:
NSString *currentVersion = [[[NSBundle mainBundle] infoDictionary] objectForKey:@"CFBundleVersion"];
Update your Info.plist’s version number religiously before distribution. Even better, update it automatically.
Communication
You need to communicate with your testing pool. Keeping them regularly updated, without being spammy, is crucial to keeping them excited about what’s going on.
When you send out a new build, explain what’s new. Give specific testing guidance so that your testers know where they should focus their efforts. New method of rendering cells you strongly suspect has a bug or two? Share your hunches. Built a new import subsystem you’re suspicious might break with legacy data? Provide an example data set or encourage existing users of the old version to give it a whirl. If you’re manipulating data that lives only on the user’s device, tell them to back up their iPhone before running your beta. And tell them why. Don’t be a douche on this: never take for granted the importance of your testers’ application data, especially if you’re lucky enough to have your existing customers helping you. Yes, we all know, betas are risky. Still, the iPhone isn’t the desktop and if you’re testing an upgrade to an existing app, your customers may have data they created before accepting the risks of a beta.
If it has been more than seven to ten days since your last build, send a quick update to the pool. Call out awesome testers and let them know they’re making a difference. Share the score of resolved issues you were able to fix thanks to the pool’s feedback. Most of all, thank them. Their efforts are helping you to not look like an idiot.
Sandboxing Helps
If you’re making an all-new app, you can skip this section. If you’re testing an upgrade to an existing app, read on.
Your early beta builds should use a different app identifier than your shipping app. This way, your beta will be installed as a distinct application alongside your currently shipping version on the user’s device, instead of overwriting it. This lets you ensure that no shenanigans happen to legacy data if you’ve got customers from previous versions in your testing pool. Toward the end of your beta, after much internal testing, you can start distributing builds that overwrite previous shipping versions. You’ll want to ensure that the app properly imports legacy data and preferences and, of course, ensure nothing otherwise disastrous happens when iTunes distributes your updated app to existing customers with existing data.
Gather Useful Feedback
You need to know things about how your application is failing. The forms I use are inspired heavily by what I saw once upon a time during the iPhone OS 2.0 Enterprise AppleSeed beta test. I think it covers the basics. Of course, don’t be afraid to add other questions specific to your application’s circumstances. For bugs, measure how well you can reproduce the bug and its overall severity while capturing the reproduction steps. For feature or enhancement requests, measure the tester’s impression of the request’s importance along with what they’re asking for.
However you gather your feedback, ensure that every single message you send to your testing pool includes brief instructions on how your testers should report their bugs. Never make them hunt around for it.
Ship
You haven’t added or changed anything for a couple of builds. You’ve fixed plenty of bugs and no one has reported regressions. You’ve sent emails to your top-performing testers and they’re giving you a thumbs-up.
Give it one last test yourself. Thirty minutes, minimum, actually doing things your users are likely to do. Is it kosher?
Then ship it.
Then, while you wait for approval from Apple, test it some more, informally, yourself. If somehow a show-stopping bug pops up while you await approval, you can reject your binary and submit a fix before it ruins anyone’s day.
iPhone Development: Force UITableView to show only complete cells
Tallymander uses UITableViewCells not only to display data but also to manipulate it. That means that partially-visible cells — that is, those that are cut off at the top or bottom of the view — aren’t terribly useful.
Better, I thought, would be to always ensure that when a partially-visible cell crops up, the UITableViewController will quietly nudge things to be completely visible.
vs. 
In the first image, you can see the only part of the top and bottom cell. In the second image, the cells have been nudged so that everything visible is completely visible. This took a little noodling around with math but it wasn’t hard to do. Put this method into your UITableViewController subclass:
- (void)snapBottomCell { NSInteger cellHeight = 62; //Cells for my view are 62px tall. Sub your own height here NSInteger offsetOverage = (NSInteger) self.tableView.contentOffset.y % cellHeight; //Use the tableview's contentOffset property and the cell height to determine how much is being cut off if (offsetOverage > 0) { //If the overage is more than 0, we should figure out what the new offset needs to be NSInteger newOffset; if (offsetOverage >= (cellHeight/2)) { newOffset = self.tableView.contentOffset.y + (cellHeight - offsetOverage); //If the overage is greater than or equal to half the height of a cell, pull the cell up so it's fully visible } else { newOffset = self.tableView.contentOffset.y - offsetOverage; //Else, push the cell out of view } //With the new offset determined, animate the movement: [UIView beginAnimations:nil context:NULL]; [UIView setAnimationDuration:0.5]; [UIView setAnimationCurve:UIViewAnimationCurveEaseOut]; [self.tableView setContentOffset:CGPointMake(0, newOffset) animated:NO]; [UIView commitAnimations]; } }
You’ll need to implement these two delegate methods as well:
- (void)scrollViewDidEndDecelerating:(UIScrollView *)scrollView { [self snapBottomCell]; } - (void)scrollViewDidEndDragging:(UIScrollView *)scrollView willDecelerate:(BOOL)decelerate { if (decelerate == NO) { [self snapBottomCell]; } }
And that’s it. If more than half of the bottom cell is visible, the cell gets nudged completely into view. If less than half is visible, it’s pushed out of view.
You could change snapBottomCell to accept cellHeight as an argument if you need to accommodate cells with variable heights. You might sort out a particular cell’s height like so:
NSIndexPath *indexPath = [self.tableView indexPathForRowAtPoint:CGPointMake(0, (self.tableView.frame.size.height - 1))]; //Get the index path of the cell currently visible at the bottom edge of the tableview UITableViewCell *cell = [self.tableView cellForRowAtIndexPath:indexPath]; [self snapBottomCell:cell.frame.size.height];
There are cases where this isn’t useful: long lists whose primary function is selecting data for loading in another view, for example. Still, any time you’re using UITableViewCells to house controls or chunky bits of data, it might be good to ensure that only whole cells are displayed.
Tallymander 2.0 Post-Mortem
Tallymander 2.0 has been in the App Store for a few weeks. Here’s the story on my favorite product, so far.
Once upon a time…
Tallymander was born because once upon a time, I worked for the man and that meant wearing fancy pants and shirts that buttoned (the better to establish my image as a professional and advance my career). About once a month I rounded up my finery, counted it, and took it to the dry cleaners.
My mental math abilities could be bested by a fruit fly so I wondered if the App Store had anything to make the counting easier. They did, but it was either crap or too limited for my needs. The idea for Tallymander was born. I wanted to make something simple but useful.
I expected to make about $40 off of it, but I knew I wanted the challenge so I plowed ahead anyway. 1.0 shipped and people liked it. Really liked it! I got a handful of emails with kudos and feature requests. I wanted to get cracking on GlobeJot but decided I would spend a little time adding some customer requests to Tallymander first. A couple more weeks of development and Tallymander 1.1 was submitted. I forgot all about it.
Until one night, around midnight, when I got the “Ready for Sale” email that iPhone developers live for. Tallymander 1.1 was live.
And an App Store Staff Favorite. I peed myself a little. There followed a slew of copycat apps following this prominence. So it goes on the App Store.
The future
Tthanks to a handful of enthusiastic users and a little help from Apple, Tallymander managed to make me a tidy bit more money than the $40 I had expected. About $4,000 to date, in fact. Not bad for an app that started as a two week UX trial.
SwedeShop
I remember the first time I went to IKEA.
It blew my brain open. It’s the coolest retail experience in the history of the world.
There’s only one problem: you gotta keep track of a lot of data if you want to get out of there with everything you wanted to take home.
Enter SwedeShop, my unofficial IKEA shopping list app. Find out more at the main site:
The Gravest Pain of an iPhone Developer
It’s a chattery time for App Store problems. Apple rejected Google Voice, then neutered Ninjawords and still presents an utterly opaque face to developers.
There are a laundry list of problems facing the growth of the App Store. I won’t bother to rehash them here. Let’s focus on the one that most thoroughly jeopardizes the future of developer businesses: Customer Service. Every other problem can be overcome or worked around but without the power of caring for your customers, your business has no reason to exist.
In an aside to a link last month, John Gruber muses:
I’m wondering how much of the problem is that the App Store is built on the foundation and framework of the iTunes Music Store, which was designed from the outset specifically as a venue for selling 99-cent downloads.
This is the most crucially important point: the iTunes Store was never designed to sell software. Among other things, Craig Hockenberry enumerates all the ways in which the App Store is hobbled by this historical truth. It’s a good, important post that you should read if you care about this kind of stuff. But it doesn’t address long-term outcomes related to customer service that will doom the developer community.
As an iPhone developer, I have no control over my storefront – Apple manages it for me, with basic data I provide. On the one hand, this is incredible news: access to a huge pool of customers, a complete distribution infrastructure and – best of all – I never have to worry about payment processing.
There’s just one issue: Apple doesn’t give a damn about my relationship with my customers.
Generous, attentive, impassioned customer service is an important piece of any successful business. My customers mean the world to me. Unfortunately, iTunes does not provide a clear, encouraging feedback channel.
User Reviews
When you’re selling music, user reviews are a simple tool. Much is subjective, but overall quality will be reflected in the reviews.
With software, the reviews have become more complicated. The most tantalizing way for a customer to speak out about software that is giving them problems is to write a review. And that’s what they do. Bug reports, feature requests and anything else that comes into their minds gets dumped into the reviews. And why not? The ability to write a review is prominently featured and uses a built-in, official form. It’s infinitely more seductive than leaving iTunes to write an email to the support contact. It’s also a venue provided by the same service that is taking the customer’s money, so it feels more intimately linked to their purchase than anything they can do on an external website or in their email client.
This is infuriating since the communication is strictly one-sided. There’s no way for the developer to follow up on these reviews to ask for more information. Without that information, acting on a bug report is often impossible. The worst part is that without dialogue, it’s impossible for the customer to learn more about their problem, discover workarounds and discover that there’s a living, breathing person who truly cares about the quality of the software they’ve just purchased.
Like it or not, the iTunes user review becomes the support form of last resort.
The Consequence
There are ways around this. Tap4Help is an interesting example, providing a built-in feedback and support request system. Developers, like Lucius Kwok, report some success explicitly declaring their email right in their application description with a call to action encouraging its use. I do this, too, but it doesn’t catch them all.
Why not? Nothing will ever come close to the power and authority of iTunes itself. I theorize that part of the reason so many customers prefer the review form to using a support email or link is that they know that iTunes will provide them satisfaction. No matter what, iTunes will show their review. They will be heard.
By keeping these customers so thoroughly at arm’s length, Apple retards the formation of relationships that will build developers’ business. I’ve turned angry emails into loyal customers through the power of honesty and genuine interest in customer issues. I’d desperately love to provide that dialogue for every customer, ever, but iTunes, under the current system, will continue to siphon off some portion of those opportunities into its black hole of customer reviews.
Having good conversations with your customers is as essential and non-negotiable as having an engine in your car. When Zappos tweets at me in thanks for my praise, I feel as though my relationship with the company has been further validated. When Netflix gives me complete and generous support when I have trouble with their service, I feel respect for them, since their conduct conveys respect for my business.
It’s all about how the customer feels. If you never get to talk with them, you’ll never get to impact that feeling.
Let’s Do It Better
This is not a hard problem to solve. If you happen to work on the iTunes Store infrastructure team, you may feel differently, but the company you work for is in the business of accomplishing the impossible on a fairly regular basis. My sympathy is limited.
Developer Review Replies
This is the easiest part. Let the developer reply to user reviews. This isn’t groundbreaking and I’m the eight thousandth developer to suggest it. So make it happen. The developer can join the conversation and solicit additional information so that bug reports that go into the reviews can actually be productive. Notify whomever left the review that they have a response via email. For bonus points, let the customer reply directly to that notification to reach the developer.
Feedback/Support Form
Let the user provide feedback or support requests through an official, iTunes-embedded form. Send the feedback to the developer via email, with an anonymized reply-to address, like craigslist uses, so Apple can cover their ass on privacy concerns. For bonus points, provide a rating for each application that states how responsive each developer is to requests sent via this form.
There is no step three. With those two provisions, an open dialogue has been created for anyone who bothers to seek one. Software, even for the iPhone, is not music. The one-sided echo-chamber conversation of the iTunes Music Store does not work in the App Store. With the two modest tools I’ve described, developers will have an infinitely easier time creating the relationships they need to build their business.
I’m not going to hold my breath. Hopefully Apple is working on this stuff, but in the meantime, I need to figure out better ways to put myself in easy reach of my customers.